Skip to main content

2017 | OriginalPaper | Buchkapitel

A Distributed, Scalable Computing Facility for Big Data Analytics in Atmospheric Physics

verfasst von : Reena Bharathi, S. C. Shirwaikar, Vilas Kharat

Erschienen in: Advances in Computing and Data Sciences

Verlag: Springer Singapore

Aktivieren Sie unsere intelligente Suche, um passende Fachinhalte oder Patente zu finden.

search-config
loading …

Abstract

Technological advancements in computing and communication have led to a flood of data from different domains like healthcare, social networks, Internet commerce and finance. Over the past few years a larger chunk of data comes from the domain of scientific applications, using simulated experiments or collected using sensors. This development calls for new architectural models for data acquisition, storage, and large-scale data analytics.
In this paper, we present a distributed and scalable computing facility, using low cost machines, which support analytics of large scientific data sets, constituting three sequential modules, namely data pre-processing, data analytics and data post-processing. These three modules together form a big data value chain which is illustrated through a case study related to Atmospheric physics.

Sie haben noch keine Lizenz? Dann Informieren Sie sich jetzt über unsere Produkte:

Springer Professional "Wirtschaft+Technik"

Online-Abonnement

Mit Springer Professional "Wirtschaft+Technik" erhalten Sie Zugriff auf:

  • über 102.000 Bücher
  • über 537 Zeitschriften

aus folgenden Fachgebieten:

  • Automobil + Motoren
  • Bauwesen + Immobilien
  • Business IT + Informatik
  • Elektrotechnik + Elektronik
  • Energie + Nachhaltigkeit
  • Finance + Banking
  • Management + Führung
  • Marketing + Vertrieb
  • Maschinenbau + Werkstoffe
  • Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

Springer Professional "Technik"

Online-Abonnement

Mit Springer Professional "Technik" erhalten Sie Zugriff auf:

  • über 67.000 Bücher
  • über 390 Zeitschriften

aus folgenden Fachgebieten:

  • Automobil + Motoren
  • Bauwesen + Immobilien
  • Business IT + Informatik
  • Elektrotechnik + Elektronik
  • Energie + Nachhaltigkeit
  • Maschinenbau + Werkstoffe




 

Jetzt Wissensvorsprung sichern!

Springer Professional "Wirtschaft"

Online-Abonnement

Mit Springer Professional "Wirtschaft" erhalten Sie Zugriff auf:

  • über 67.000 Bücher
  • über 340 Zeitschriften

aus folgenden Fachgebieten:

  • Bauwesen + Immobilien
  • Business IT + Informatik
  • Finance + Banking
  • Management + Führung
  • Marketing + Vertrieb
  • Versicherung + Risiko




Jetzt Wissensvorsprung sichern!

Literatur
1.
Zurück zum Zitat Cuzzocrea, A., Song, I.-Y., Davis, K.C.: Analytics over large-scale multidimensional data: the big data revolution!. In: Proceedings of the ACM 14th International Workshop on Data Warehousing and OLAP, pp. 101–104. ACM (2011) Cuzzocrea, A., Song, I.-Y., Davis, K.C.: Analytics over large-scale multidimensional data: the big data revolution!. In: Proceedings of the ACM 14th International Workshop on Data Warehousing and OLAP, pp. 101–104. ACM (2011)
2.
Zurück zum Zitat Hu, H., Wen, Y., Chua, T.-S., Li, X.: Toward scalable systems for big data analytics: a technology tutorial. IEEE Access 2, 652–687 (2014)CrossRef Hu, H., Wen, Y., Chua, T.-S., Li, X.: Toward scalable systems for big data analytics: a technology tutorial. IEEE Access 2, 652–687 (2014)CrossRef
3.
Zurück zum Zitat Chen, H., Chiang, R.H.L., Storey, V.C.: Business intelligence and analytics: from big data to big impact. MIS Q. 36(4), 1165–1188 (2012) Chen, H., Chiang, R.H.L., Storey, V.C.: Business intelligence and analytics: from big data to big impact. MIS Q. 36(4), 1165–1188 (2012)
4.
Zurück zum Zitat Boyd, D., Crawford, K.: Critical questions for big data: provocations for a cultural, technological, and scholarly phenomenon. Inf. Commun. Soc. 15(5), 662–679 (2012)CrossRef Boyd, D., Crawford, K.: Critical questions for big data: provocations for a cultural, technological, and scholarly phenomenon. Inf. Commun. Soc. 15(5), 662–679 (2012)CrossRef
5.
Zurück zum Zitat Gorton, I., Greenfield, P., Szalay, A., Williams, R.: Data-intensive computing in the 21st century. Computer 41(4), 30–32 (2008)CrossRef Gorton, I., Greenfield, P., Szalay, A., Williams, R.: Data-intensive computing in the 21st century. Computer 41(4), 30–32 (2008)CrossRef
6.
Zurück zum Zitat Srirama, S.N., Jakovits, P., Vainikko, E.: Adapting scientific computing problems to clouds using MapReduce. Future Gener. Comput. Syst. 28(1), 184–192 (2012)CrossRef Srirama, S.N., Jakovits, P., Vainikko, E.: Adapting scientific computing problems to clouds using MapReduce. Future Gener. Comput. Syst. 28(1), 184–192 (2012)CrossRef
7.
Zurück zum Zitat Tudoran, R., Costan, A., Antoniu, G., Bougé, L.: A performance evaluation of azure and nimbus clouds for scientific applications. In: Proceedings of the 2nd International Workshop on Cloud Computing Platforms, p. 4. ACM (2012) Tudoran, R., Costan, A., Antoniu, G., Bougé, L.: A performance evaluation of azure and nimbus clouds for scientific applications. In: Proceedings of the 2nd International Workshop on Cloud Computing Platforms, p. 4. ACM (2012)
8.
Zurück zum Zitat Wang, L., Tao, J., Kunze, M., Castellanos, A.C., Kramer, D., Karl, W.: Scientific cloud computing: early definition and experience. In: HPCC, vol. 8, pp. 825–830 (2008) Wang, L., Tao, J., Kunze, M., Castellanos, A.C., Kramer, D., Karl, W.: Scientific cloud computing: early definition and experience. In: HPCC, vol. 8, pp. 825–830 (2008)
9.
Zurück zum Zitat Ramakrishnan, L., Zbiegel, P.T., Campbell, S., Bradshaw, R., Canon, R.S., Coghlan, S., Sakrejda, I., Desai, N., Declerck, T., Liu, A.: Magellan: experiences from a science cloud. In: Proceedings of the 2nd International Workshop on Scientific Cloud Computing, pp. 49–58. ACM (2011) Ramakrishnan, L., Zbiegel, P.T., Campbell, S., Bradshaw, R., Canon, R.S., Coghlan, S., Sakrejda, I., Desai, N., Declerck, T., Liu, A.: Magellan: experiences from a science cloud. In: Proceedings of the 2nd International Workshop on Scientific Cloud Computing, pp. 49–58. ACM (2011)
10.
Zurück zum Zitat Grossman, R.L., Gu, Y., Mambretti, J., Sabala, M., Szalay, A., White, K.: An overview of the open science data cloud. In: Proceedings of the 19th ACM International Symposium on High Performance Distributed Computing, pp. 377–384. ACM (2010) Grossman, R.L., Gu, Y., Mambretti, J., Sabala, M., Szalay, A., White, K.: An overview of the open science data cloud. In: Proceedings of the 19th ACM International Symposium on High Performance Distributed Computing, pp. 377–384. ACM (2010)
11.
Zurück zum Zitat Ekanayake, J., Pallickara, S., Fox, G.: Mapreduce for data intensive scientific analyses. In: IEEE Fourth International Conference on eScience, eScience 2008, pp. 277–284. IEEE (2008) Ekanayake, J., Pallickara, S., Fox, G.: Mapreduce for data intensive scientific analyses. In: IEEE Fourth International Conference on eScience, eScience 2008, pp. 277–284. IEEE (2008)
12.
Zurück zum Zitat Dean, J., Ghemawat, S.: MapReduce: a flexible data processing tool. Commun. ACM 53(1), 72–77 (2010)CrossRef Dean, J., Ghemawat, S.: MapReduce: a flexible data processing tool. Commun. ACM 53(1), 72–77 (2010)CrossRef
13.
Zurück zum Zitat Pawar, G.V., Devara, P.C.S., Aher, G.R.: Identification of aerosol types over an urban site based on air-mass trajectory classification. Atmos. Res. 164, 142–155 (2015)CrossRef Pawar, G.V., Devara, P.C.S., Aher, G.R.: Identification of aerosol types over an urban site based on air-mass trajectory classification. Atmos. Res. 164, 142–155 (2015)CrossRef
14.
Zurück zum Zitat Jorba, O., Pérez, C., Rocadenbosch, F., Baldasano, J.: Cluster analysis of 4-day back trajectories arriving in the Barcelona area, Spain, from 1997 to 2002. J. Appl. Meteorol. 43(6), 887–901 (2004)CrossRef Jorba, O., Pérez, C., Rocadenbosch, F., Baldasano, J.: Cluster analysis of 4-day back trajectories arriving in the Barcelona area, Spain, from 1997 to 2002. J. Appl. Meteorol. 43(6), 887–901 (2004)CrossRef
Metadaten
Titel
A Distributed, Scalable Computing Facility for Big Data Analytics in Atmospheric Physics
verfasst von
Reena Bharathi
S. C. Shirwaikar
Vilas Kharat
Copyright-Jahr
2017
Verlag
Springer Singapore
DOI
https://doi.org/10.1007/978-981-10-5427-3_54