Skip to main content
Fachgebiete
Automobil + Motoren Bauwesen + Immobilien Business IT + Informatik Elektrotechnik + Elektronik Energie + Nachhaltigkeit Finance + Banking Management + Führung Marketing + Vertrieb Maschinenbau + Werkstoffe Versicherung + Risiko
DE
EN
Bücher
Zeitschriften
Themenseiten
Organisationspsychologie Projektmanagement Marketing Smart Manufacturing
Weitere Formate
Podcasts Webinare Technik Webinare Wirtschaft Kongresse Veranstaltungskalender Awards
Jetzt Einzelzugang starten
Zugang für Unternehmen
Referenzkunden
SLX-Digitalkonferenz: Zukunftswerkstatt 2024

Mehr Umsatz mit Top-Kontakten

Am 7. Mai erfahren Sie, wie Vertriebsteams Leadgenerierung, Leadnurturing und Leadstrategien effizient steuern können.
Erweiterte Suche
Anmelden
nach oben
Erschienen in:
Automated Detection of Central Retinal Vein Occlusion Using Convolutional Neural Network Kapitel lesen Erstes Kapitel lesen

2017 | OriginalPaper | Buchkapitel

Big Sensor Data Acquisition and Archiving with Compression

verfasst von : Dongeun Lee

Erschienen in: Big Data and Visual Analytics

Verlag: Springer International Publishing

Einloggen

Aktivieren Sie unsere intelligente Suche, um passende Fachinhalte oder Patente zu finden.

search-config
loading …

Abstract

Machine-generated data such as sensor data now comprise major portion of available information, which raises two important problems: efficient acquisition of sensor data and the storage of massive sensor data collection. These data sources generate so much data quickly that data compression is essential to reduce storage requirement or transmission capacity of devices. This work first discusses a low complexity sensing framework which enables to reduce computation and communication overheads of devices without much compromising the accuracy of sensor readings. Then a new class of compression algorithm based on statistical similarity is presented that can be effectively used in many applications where an order of data sequence could be relaxed. Next, a quality-adjustable sensor data archiving is discussed, which compresses an entire collection of sensor data efficiently without compromising key features. Considering data aging aspect, this archiving scheme is capable of decreasing data fidelity gradually to secure more storage space.

Sie haben noch keine Lizenz? Dann Informieren Sie sich jetzt über unsere Produkte:

Springer Professional "Wirtschaft+Technik"

Online-Abonnement

Mit Springer Professional "Wirtschaft+Technik" erhalten Sie Zugriff auf:

  • über 102.000 Bücher
  • über 537 Zeitschriften

aus folgenden Fachgebieten:

  • Automobil + Motoren
  • Bauwesen + Immobilien
  • Business IT + Informatik
  • Elektrotechnik + Elektronik
  • Energie + Nachhaltigkeit
  • Finance + Banking
  • Management + Führung
  • Marketing + Vertrieb
  • Maschinenbau + Werkstoffe
  • Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

Jetzt informieren

Springer Professional "Technik"

Online-Abonnement

Mit Springer Professional "Technik" erhalten Sie Zugriff auf:

  • über 67.000 Bücher
  • über 390 Zeitschriften

aus folgenden Fachgebieten:

  • Automobil + Motoren
  • Bauwesen + Immobilien
  • Business IT + Informatik
  • Elektrotechnik + Elektronik
  • Energie + Nachhaltigkeit
  • Maschinenbau + Werkstoffe




 

Jetzt Wissensvorsprung sichern!

Jetzt informieren

Springer Professional "Wirtschaft"

Online-Abonnement

Mit Springer Professional "Wirtschaft" erhalten Sie Zugriff auf:

  • über 67.000 Bücher
  • über 340 Zeitschriften

aus folgenden Fachgebieten:

  • Bauwesen + Immobilien
  • Business IT + Informatik
  • Finance + Banking
  • Management + Führung
  • Marketing + Vertrieb
  • Versicherung + Risiko




Jetzt Wissensvorsprung sichern!

Jetzt informieren
Vorheriges Kapitel Scalable Gene Sequence Analysis on Spark
Nächstes Kapitel Advanced High Performance Computing for Big Data Local Visual Meaning
Fußnoten
1
This can also be seen as inner product operations.
 
2
The two bases Φ and Ψ are (maximally) incoherent when the largest correlation between any two elements of Φ and Ψ is \(1/\sqrt {N}\) where N is the order of two square matrices.
 
3
This can be generated by a random permutation.
 
4
When the random indices for the spatial sampling are not explicitly synchronized between encoder and decoder, the spatio-temporal measurement no longer has a matrix form since the number of spatial sampling can vary between time instants. However, the decoding process does not impose the matrix form on the spatio-temporal measurement.
 
5
This value is also called the significance level.
 
6
These analytical models are convex by virtue of the trade-off relationship between data fidelity and compression ratio. The sum of these convex functions is also convex.
 
Literatur
1.
2.
Ahmed, N., Natarajan, T., Rao, K.R.: Discrete cosine transform. IEEE Trans. Comput. C-23(1), 90–93 (1974)
3.
Bajwa, W., Haupt, J., Sayeed, A., Nowak, R.: Compressive wireless sensing. In: Proceedings of the International Conference on Information Processing in Sensor Networks (IPSN ’06), pp. 134–142 (2006)
4.
Baraniuk, R.G.: Compressive sensing [lecture notes]. IEEE Signal Process. Mag. 24(4), 118–121 (2007)
5.
Baraniuk, R.G., Cevher, V., Duarte, M.F., Hegde, C.: Model-based compressive sensing. IEEE Trans. Inf. Theory 56(4), 1982–2001 (2010)
6.
Baron, D., Duarte, M.F., Wakin, M.B., Sarvotham, S., Baraniuk, R.G.: Distributed compressive sensing (2009). arXiv preprint arXiv:0901.3403
7.
Barr, K.C., Asanović, K.: Energy-aware lossless data compression. ACM Trans. Comput. Syst. 24(3), 250–291 (2006)
8.
Berinde, R., Indyk, P.: Sparse recovery using sparse random matrices. Tech. Rep. MIT-CSAIL-TR-2008–001, Massachusetts Institute of Technology (2008)
9.
Burke, J.A., Estrin, D., Hansen, M., Parker, A., Ramanathan, N., Reddy, S., Srivastava, M.B.: Participatory sensing. Center for Embedded Network Sensing (2006)
10.
11.
Candès, E.J., Wakin, M.B.: An introduction to compressive advanced sampling. IEEE Signal Process. Mag. 25(2), 21–30 (2008)
12.
Cao, H., Wolfson, O., Trajcevski, G.: Spatio-temporal data reduction with deterministic error bounds. VLDB J. 15(3), 211–228 (2006)
13.
Choi, J., Hu, K., Sim, A.: Relational dynamic Bayesian networks with locally exchangeable measures. Tech. Rep. LBNL-6341E, Lawrence Berkeley National Laboratory (2013)
14.
Coding of audiovisual objects - Part 10: advanced video coding (2003)
15.
Cohen, E., Kaplan, H.: Aging through cascaded caches: performance issues in the distribution of web content. In: Proceedings of the 2001 ACM Conference on Special Interest Group on Data Communication, pp. 41–53 (2001)
16.
Cover, T.M., Thomas, J.A.: Elements of Information Theory, 2nd edn. Wiley, Hoboken (2006)
17.
Do, T.T., Gan, L., Nguyen, N.H., Tran, T.D.: Fast and efficient compressive sensing using structurally random matrices. IEEE Trans. Signal Process. 60(1), 139–154 (2012)
18.
Duarte, M.F., Wakin, M.B., Baraniuk, R.G.: Fast reconstruction of piecewise smooth signals from incoherent projections. In: Proceedings of the Workshop Signal Processing with Adaptive Sparse Structured Representations (SPARS ’05) (2005)
19.
Duarte, M.F., Wakin, M.B., Baron, D., Baraniuk, R.G.: Universal distributed sensing via random projections. In: Proceedings of the International Conference on Information Processing in Sensor Networks (IPSN ’06), pp. 177–185 (2006)
20.
Engmann, S., Cousineau, D.: Comparing distributions: the two-sample Anderson-Darling test as an alternative to the Kolmogorov-Smirnoff test. J. Appl. Quant. Methods 6(3), 1–17 (2011)
21.
Esmaeilzadeh, H., Sampson, A., Ceze, L., Burger, D.: Architecture support for disciplined approximate programming. In: Proceedings of the International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS ’12), pp. 301–312 (2012)
22.
Foucart, S., Rauhut, H.: A Mathematical Introduction to Compressive Sensing. Springer, New York (2013)
23.
Ganesan, D., Estrin, D., Heidemann, J.: Dimensions: why do we need a new data handling architecture for sensor networks? SIGCOMM Comput. Commun. Rev. 33(1), 143–148 (2003)
24.
Ganesan, D., Greenstein, B., Estrin, D., Heidemann, J., Govindan, R.: Multiresolution storage and search in sensor networks. Trans. Storage 1(3), 277–315 (2005)
25.
Gantz, J.F., Chute, C., Manfrediz, A., Minton, S., Reinsel, D., Schlichting, W., Toncheva, A.: The diverse and exploding digital universe: an updated forecast of worldwide information growth through 2011. White Paper (2008)
26.
Hang, H.M., Chen, J.J.: Source model for transform video coder and its application. I. Fundamental theory. IEEE Trans. Circuits Syst. Video Technol. 7(2), 287–298 (1997)CrossRef
27.
Hilbert, M., López, P.: The world’s technological capacity to store, communicate, and compute information. Science 332(6025), 60–65 (2011)CrossRef
28.
29.
Lee, D., Choi, J.: Low complexity sensing for big spatio-temporal data. In: Proceedings of the International Conference on Big Data (BigData ’14), pp. 323–328 (2014)
30.
Lee, D., Choi, J.: Learning compressive sensing models for big spatio-temporal data. In: Proceedings of the International Conference on Data Mining (SDM ’15), pp. 667–675 (2015)
31.
Lee, D., Lee, Y., Lee, H., Lee, J., Shin, H.: Determining efficient bit stream extraction paths in H.264/AVC scalable video coding. In: Proceedings of the Asilomar Conference on Signals, on Systems, and Computers (ACSSC ’08), pp. 2233–2237 (2008)
32.
Lee, D., Choi, J., Shin, H.: Low-complexity compressive sensing with downsampling. IEICE Electron. Express 11(3), 20130947 (2014)CrossRef
33.
Lee, D., Choi, J., Shin, H.: A scalable and flexible repository for big sensor data. IEEE Sensors J. 15(12), 7284–7294 (2015)CrossRef
34.
Lee, D., Ryu, J., Shin, H.: Scalable management of storage for massive quality-adjustable sensor data. Computing 97(8), 769–793 (2015)MathSciNetCrossRef
35.
Lee, D., Lima, R., Choi, J.: Improving imprecise compressive sensing models. In: Proceedings of the Thirty-Second Conference on Uncertainty in Artificial Intelligence (UAI ’16), pp. 397–406 (2016)
36.
Lee, D., Sim, A., Choi, J., Wu, K.: Novel data reduction based on statistical similarity. In: Proceedings of the International Conference on Scientific and Statistical Database Management (SSDBM ’16), pp. 21:1–21:12 (2016)
37.
Luo, C.,Wu, F., Sun, J., Chen, C.W.: Compressive data gathering for large-scale wireless sensor networks. In: Proceedings of the Mobile Computing and Networking (MobiCom ’09), pp. 145–156 (2009)
38.
Marcelloni, F., Vecchio, M.: Enabling energy-efficient and lossy-aware data compression in wireless sensor networks by multi-objective evolutionary optimization. Inf. Sci. 180(10), 1924–1941 (2010)CrossRef
39.
Massey, F.J. Jr.: The Kolmogorov-Smirnov test for goodness of fit. J. Am. Stat. Assoc. 46(253), 68–78 (1951)CrossRefMATH
40.
Moffat, A.: Implementing the PPM data compression scheme. IEEE Trans. Commun. 38(11), 1917–1921 (1990)CrossRef
41.
Noh, D., Lee, D., Shin, H.: Mission-oriented selective routing for wireless sensor networks. In: Proceedings of the International Conference on Communications and Networking in China (CHINACOM ’07), pp. 809–813 (2007)
42.
Noh, D., Lee, D., Shin, H.: QoS-aware geographic routing for solar-powered wireless sensor networks. IEICE Trans. Commun. 90(12), 3373–3382 (2007)CrossRef
43.
Palmer, M.: Seven principles of effective RFID data management (2005)
44.
Palpanas, T., Vlachos, M., Keogh, E., Gunopulos, D., Truppel, W.: Online amnesic approximation of streaming time series. In: Proceedings of the International Conference on Data Engineering (ICDE ’04), pp. 339–349 (2004)
45.
Quer, G., Masiero, R., Munaretto, D., Rossi, M., Widmer, J., Zorzi, M.: On the interplay between routing and signal representation for compressive sensing in wireless sensor networks. In: Proceedings of the Information Theory and Applications Workshop (ITA ’09), pp. 206–215 (2009)
46.
Quer, G., Masiero, R., Pillonetto, G., Rossi, M., Zorzi, M.: Sensing, compression, and recovery for WSNs: sparse signal modeling and monitoring framework. IEEE Trans. Wireless Commun. 11(10), 3447–3461 (2012)CrossRef
47.
Quinsac, C., Basarab, A., Girault, J.M., Kouamé, D.: Compressed sensing of ultrasound images: sampling of spatial and frequency domains. In: Proceedings of the International Workshop on Signal Processing Systems (SiPS ’10), pp. 231–236 (2010)
48.
Richardson, I.E.: The H.264 Advanced Video Compression Standard, 2nd edn. Wiley, Hoboken (2010)
49.
Sadler, C.M., Martonosi, M.: Data compression algorithms for energy-constrained devices in delay tolerant networks. In: Proceedings of the International Conference on Embedded Network Sensor Systems (SenSys ’06), pp. 265–278 (2006)
50.
Sampson, A., Nelson, J., Strauss, K., Ceze, L.: Approximate storage in solid-state memories. In: Proceedings of the International Symposium on Microarchitecture (MICRO ’46), pp. 25–36 (2013)
51.
Sayood, K.: Introduction to Data Compression, 4th edn. Morgan Kaufmann, Burlington (2012)MATH
52.
Schwarz, H., Marpe, D., Wiegand, T.: Overview of the scalable video coding extension of the H.264/AVC standard. IEEE Trans. Circuits Syst. Video Technol. 17(9), 1103–1120 (2007)
53.
Seabra, J., Sanches, J.: Modeling log-compressed ultrasound images for radio frequency signal recovery. In: Proceedings of the International Conference on Engineering in Medicine and Biology Society (EMBC ’08), pp. 426–429 (2008)
54.
55.
Srisooksai, T., Keamarungsi, K., Lamsrichan, P., Araki, K.: Practical data compression in wireless sensor networks: a survey. J. Netw. Comput. Appl. 35(1), 37–59 (2012)CrossRef
56.
57.
Vuran, M.C., Akan, Ö.B., Akyildiz, I.F.: Spatio-temporal correlation: theory and applications for wireless sensor networks. Comput. Netw. 45(3), 245–259 (2004)CrossRefMATH
58.
Wang, Y.C., Hsieh, Y.Y., Tseng, Y.C.: Multiresolution spatial and temporal coding in a wireless sensor network for long-term monitoring applications. IEEE Trans. Comput. 58(6), 827–838 (2009)MathSciNetCrossRefMATH
59.
60.
Wien, M., Cazoulat, R., Graffunder, A., Hutter, A., Amon, P.: Real-time system for adaptive video streaming based on SVC. IEEE Trans. Circuits Syst. Video Technol. 17(9), 1227–1237 (2007)CrossRef
61.
Xiong, Z., Ramchandran, K., Orchard, M.T., Zhang, Y.Q.: A comparative study of DCT-and wavelet-based image coding. IEEE Trans. Circuits Syst. Video Technol. 9(5), 692–695 (1999)CrossRef
62.
Yu, J., Ongarello, S., Fiedler, R., Chen, X., Toffolo, G., Cobelli, C., Trajanoski, Z.: Ovarian cancer identification based on dimensionality reduction for high-throughput mass spectrometry data. Bioinformatics 21(10), 2200–2209 (2005)CrossRef
Metadaten
Titel
Big Sensor Data Acquisition and Archiving with Compression
verfasst von
Dongeun Lee
Copyright-Jahr
2017
Buch
Big Data and Visual Analytics

Print ISBN: 978-3-319-63915-4

Electronic ISBN: 978-3-319-63917-8

Copyright-Jahr: 2017

DOI
https://doi.org/10.1007/978-3-319-63917-8_7

Premium Partner

    Bildnachweise