Skip to main content
Fachgebiete
Automobil + Motoren Bauwesen + Immobilien Business IT + Informatik Elektrotechnik + Elektronik Energie + Nachhaltigkeit Finance + Banking Management + Führung Marketing + Vertrieb Maschinenbau + Werkstoffe Versicherung + Risiko
DE
EN
Themenseiten
Organisationspsychologie Projektmanagement Marketing Smart Manufacturing
Bücher
Zeitschriften
Weitere Formate
Podcasts Webinare Technik Webinare Wirtschaft Kongresse Veranstaltungskalender Awards
Jetzt Einzelzugang starten
Zugang für Unternehmen
Referenzkunden
MTZ-Fachkongress

Antriebe und Energiesysteme von morgen


Austausch von Automobil- und Energiewirtschaft

Am 14./15. Mai geht es in Chemnitz um die Mobilitätswende durch Elektro- und Wasserstofffahrzeuge.
Erweiterte Suche
Anmelden
nach oben
Erschienen in:
Enterprise Systems for Networked Smart Cities Kapitel lesen Erstes Kapitel lesen

2020 | OriginalPaper | Buchkapitel

23. Big Data and HPC Convergence for Smart Infrastructures: A Review and Proposed Architecture

verfasst von : Sardar Usman, Rashid Mehmood, Iyad Katib

Erschienen in: Smart Infrastructure and Applications

Verlag: Springer International Publishing

Einloggen

Aktivieren Sie unsere intelligente Suche, um passende Fachinhalte oder Patente zu finden.

search-config
loading …

Abstract

The world has seen exponential data growth due to social media, mobility, E-commerce, and other factors. The issues related to avalanche of data being produced are immense and cover variety of challenges that need a careful consideration. The use of HPDA (High Performance Data Analytics) is increasing at brisk speed in many industries and has resulted in expansion of HPC market in many new territories. HPC (High Performance Computing) and big data are different systems, not only at the technical level, but also have different ecosystems. HPC systems are mainly developed for computationally intensive applications but recently data intensive applications are also among the major workload in HPC environment. Big data analytics have grown in different perspectives and have separate developer communities. As we head towards the exascale and smart infrastructure era, the necessary integration of big data and HPC is currently a hot topic of research but still at very infant stages. Both systems have different architecture and their integration brings many challenges. The aim of this work is to identify the driving forces, challenges, current and future trends associated with the integration of HPC and big data. This paper is an extension of our earlier work. We have reviewed programming models and frameworks of big data and HPC. The big data and HPC challenges in the exascale-computing era are discussed. Additional elaborations are provided on HPC and big data convergence research efforts and future directions are provided. The HPC-big data convergence architecture proposed in our earlier paper has been enhanced.

Sie haben noch keine Lizenz? Dann Informieren Sie sich jetzt über unsere Produkte:

Springer Professional "Wirtschaft+Technik"

Online-Abonnement

Mit Springer Professional "Wirtschaft+Technik" erhalten Sie Zugriff auf:

  • über 102.000 Bücher
  • über 537 Zeitschriften

aus folgenden Fachgebieten:

  • Automobil + Motoren
  • Bauwesen + Immobilien
  • Business IT + Informatik
  • Elektrotechnik + Elektronik
  • Energie + Nachhaltigkeit
  • Finance + Banking
  • Management + Führung
  • Marketing + Vertrieb
  • Maschinenbau + Werkstoffe
  • Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

Jetzt informieren

Springer Professional "Technik"

Online-Abonnement

Mit Springer Professional "Technik" erhalten Sie Zugriff auf:

  • über 67.000 Bücher
  • über 390 Zeitschriften

aus folgenden Fachgebieten:

  • Automobil + Motoren
  • Bauwesen + Immobilien
  • Business IT + Informatik
  • Elektrotechnik + Elektronik
  • Energie + Nachhaltigkeit
  • Maschinenbau + Werkstoffe




 

Jetzt Wissensvorsprung sichern!

Jetzt informieren

Springer Professional "Wirtschaft"

Online-Abonnement

Mit Springer Professional "Wirtschaft" erhalten Sie Zugriff auf:

  • über 67.000 Bücher
  • über 340 Zeitschriften

aus folgenden Fachgebieten:

  • Bauwesen + Immobilien
  • Business IT + Informatik
  • Finance + Banking
  • Management + Führung
  • Marketing + Vertrieb
  • Versicherung + Risiko




Jetzt Wissensvorsprung sichern!

Jetzt informieren
Vorheriges Kapitel Open Source and Open Data Licenses in the Smart Infrastructure Era: Review and License Selection Frameworks
Nächstes Kapitel Towards a Runtime Testing Framework for Dynamically Adaptable Internet of Things Networks in Smart Cities
Literatur
1.
Singh, K., Kaur, R.: Hadoop: addressing challenges of big data. In: 2014 IEEE International Advance Computing Conference (IACC), pp. 686–689. IEEE (2014)
2.
Mehmood, R., Faisal, M.A., Altowaijri, S.: Future networked healthcare systems: a review and case study. In: Boucadair, M., Jacquenet, C. (eds.) Handbook of research on redesigning the future of internet architectures, pp. 531–558. IGI Global, Hershey (2015)CrossRef
3.
4.
Alzahrani, S., Ikbal, M.R., Mehmood, R., Fayez, M., Katib, I.: Performance evaluation of Jacobi iterative solution for sparse linear equation system on multicore and manycore architectures. In: Mehmood, R., Bhaduri, B., Katib, I., Chlamtac, I. (eds.) Smart Societies, Infrastructure, Technologies and Applications. SCITA 2017. Lecture Notes of the Institute for Computer Sciences, Social Informatics and Telecommunications Engineering, pp. 296–305. Springer, Cham (2018)
5.
Alyahya, H., Mehmood, R., Katib, I.: Parallel sparse matrix vector multiplication on Intel MIC: performance analysis. In: Mehmood, R., Bhaduri, B., Katib, I., Chlamtac, I. (eds.) Smart Societies, Infrastructure, Technologies and Applications. SCITA 2017. Lecture Notes of the Institute for Computer Sciences, Social Informatics and Telecommunications Engineering, pp. 306–322. Springer, Cham (2018)
6.
Kwiatkowska, M., Mehmood, R.: Out-of-core solution of large linear systems of equations arising from stochastic modelling. In: Hermanns, H., S.R. (eds.) Process Algebra and Probabilistic Methods: Performance Modeling and Verification. PAPM-PROBMIV, pp. 135–151. Springer, Berlin (2002)CrossRef
7.
8.
Mehmood, R., Crowcroft, J.: Parallel iterative solution method for large sparse linear equation systems. Technical Report Number UCAM-CL-TR-650, Computer Laboratory, University of Cambridge, Cambridge, UK (2005)
9.
Kwiatkowska, M., Mehmood, R., Norman, G., Parker, D.: A symbolic out-of-core solution method for Markov Models. Electron. Notes Theor. Comput. Sci. 68, 589–604 (2002)CrossRef
10.
Mehmood, R., Lu, J.A.: Computational Markovian analysis of large systems. J. Manuf. Technol. Manag. 22, 804–817 (2011)CrossRef
11.
Kwiatkowska, M., Parker, D., Yi Zhang, Y., Mehmood, R.: Dual-processor parallelisation of symbolic probabilistic model checking. In: The IEEE Computer Society’s 12th Annual International Symposium on Modeling, Analysis, and Simulation of Computer and Telecommunications Systems, 2004 (MASCOTS 2004). Proceedings, pp. 123–130. IEEE (2004)
12.
Mehmood, R.: Serial disk-based analysis of large stochastic models. In: Baier, C., Haverkort, B.R., Hermanns, H., Katoen, J.P., S.M. (eds.) Validation of Stochastic Systems, pp. 230–255. Springer, Berlin (2004)CrossRef
13.
Mehmood, R., Crowcroft, J., Elmirghani, J.M.H.: A parallel implicit method for the steady-state solution of CTMCs. In: 14th IEEE International Symposium on Modeling, Analysis, and Simulation, pp. 293–302. IEEE (2006)
14.
15.
Usman, S., Mehmood, R., Katib, I.: Big data and HPC convergence: The cutting edge and outlook. In: Mehmood, R., Bhaduri, B., Katib, I., Chlamtac, I. (eds.) Smart Societies, Infrastructure, Technologies and Applications. SCITA 2017. Lecture Notes of the Institute for Computer Sciences, Social Informatics and Telecommunications Engineering, pp. 11–26. Springer, Cham (2018)
16.
Padhy, R.P.: Big data processing with Hadoop-MapReduce in cloud systems. IJ-CLOSER Int. J. Cloud Comput. Serv. Sci. 2, 233–245 (2012)
17.
18.
19.
20.
Jost, G., Jin, H.-Q., anMey, D., Hatay, F.F.: Comparing the OpenMP, MPI, and hybrid programming paradigm on an SMP cluster. In: European Workshop on OpenMP and Applications. Aachen Germany (2003)
21.
De Wael, M., Marr, S., De Fraine, B., Van Cutsem, T., De Meuter, W.: Partitioned global address space languages. ACM Comput. Surv. 47, 1–27 (2015)CrossRef
22.
Calin, G., Derevenetc, E., Majumdar, R., Meyer, R.: A theory of partitioned global address spaces (2013)
23.
24.
Geist, A., Lucas, R.: Whitepaper on the Major Computer Science Challenges at Exascale (2009)CrossRef
25.
Krishnan, S., Tatineni, M., Baru, C.: myHadoop-Hadoop-on-demand on traditional HPC resources (2011)
26.
Xuan, P., Denton, J., Ge, R., Srimani, P.K., Luo, F.: Big data analytics on traditional HPC infrastructure using two-level storage (2015)
27.
28.
Katal, A., Wazid, M., Goudar, R.H.: Big data: issues, challenges, tools and good practices. In: 2013 Sixth International Conference on Contemporary Computing (IC3), pp. 404–409. IEEE (2013)
29.
Philip Chen, C.L., Zhang, C.Y.: Data-intensive applications, challenges, techniques and technologies: a survey on big data. Inf. Sci. (Ny). 275, 314–347 (2014)CrossRef
30.
Chen, M., Mao, S., Liu, Y.: Big data: a survey. Mob. Netw. Appl. 19, 171–209 (2014)CrossRef
31.
Xu, Z., Shi, Y.: Exploring big data analysis: fundamental scientific problems. Ann. Data Sci. 2(4), 363–372 (2015)CrossRef
32.
Hashem, I.A.T., Yaqoob, I., Badrul Anuar, N., Mokhtar, S., Gani, A., Ullah Khan, S.: The rise of “Big Data” on cloud computing: review and open research issues. Inf. Syst. 47, 98–115 (2014)CrossRef
33.
Reed, D.A., Dongarra, J.: Exascale computing and big data. Commun. ACM. 58, 56–68 (2015)CrossRef
34.
Asch, M., Moore, T., Badia, R., Beck, M., Beckman, P., Bidot, T., Bodin, F., Cappello, F., Choudhary, A., de Supinski, B., Deelman, E., Dongarra, J., Dubey, A., Fox, G., Fu, H., Girona, S., Gropp, W., Heroux, M., Ishikawa, Y., Keahey, K., Keyes, D., Kramer, W., Lavignon, J.-F., Lu, Y., Matsuoka, S., Mohr, B., Reed, D., Requena, S., Saltz, J., Schulthess, T., Stevens, R., Swany, M., Szalay, A., Tang, W., Varoquaux, G., Vilotte, J.-P., Wisniewski, R., Xu, Z., Zacharov, I.: Big data and extreme-scale computing. Int. J. High Perform. Comput. Appl. 32, 435–479 (2018)CrossRef
35.
Islam, N.S., Lu, X., Wasi-ur-Rahman, M., Shankar, D., Panda, D.K.: Triple-H: a hybrid approach to accelerate HDFS on HPC clusters with heterogeneous storage architecture. In: 2015 15th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing, pp. 101–110. IEEE (2015)
36.
Ranger, C., Raghuraman, R., Penmetsa, A., Bradski, G., Kozyrakis, C.: Evaluating MapReduce for multi-core and multiprocessor systems. In: 2007 IEEE 13th International Symposium on High Performance Computer Architecture, pp. 13–24. IEEE (2007)
37.
Schwan, P., Schwan, P.: Lustre: building a file system for 1000-node clusters. In: Proceedings of 2003 LINUX Symposium (2003)
38.
Ghemawat, S., Gobioff, H., Leung, S.-T., Ghemawat, S., Gobioff, H., Leung, S.-T.: The Google file system. In: Proceedings of the Nineteenth ACM Symposium on Operating Systems Principles—SOSP’03, p. 29. ACM Press, New York (2003)
39.
Owre, S., Shankar, N., Rushby, J.M., Stringer-Calvert, D.W.J.: PVS System Guide (2001)
40.
Shvachko, K., Kuang, H., Radia, S., Chansler, R.: The Hadoop Distributed File System. In: 2010 IEEE 26th Symposium on Mass Storage Systems and Technologies (MSST), pp. 1–10. IEEE (2010)
41.
42.
Martinec, J., Rango, A., Major, E.: The Snowmelt-Runoff Model (SRM) user’s manual (1983)
43.
Rajasekar, A., Moore, R., Hou, C.-Y., Lee, C.A., Marciano, R., de Torcy, A., Wan, M., Schroeder, W., Chen, S.-Y., Gilbert, L., Tooby, P., Zhu, B.: iRODS primer: integrated rule-oriented data system. In: Synth. Lect. Inf. Concepts, Retrieval, Serv. 2, 1–143 (2010)CrossRef
44.
Plimpton, S.J., Devine, K.D.: MapReduce in MPI for large-scale graph algorithms. Parallel Comput. 37, 610–632 (2011)CrossRef
45.
Mantha, P.K., Luckow, A., Jha, S.: Pilot-MapReduce. In: Proceedings of Third International Workshop on MapReduce and Its Applications Date—MapReduce’12, p. 17. ACM Press, New York (2012)
46.
Tiwari, N., Sarkar, S., Bellur, U., Indrawan, M.: An empirical study of Hadoop’s energy efficiency on a HPC cluster. Procedia Comput. Sci. 29, 62–72 (2014)CrossRef
47.
48.
Veiga, J., Exp, R.R., Taboada, G.L., Touri, J.: Analysis and evaluation of big data computing solutions in an HPC environment (2015)
49.
Wang, Y., Jiao, Y., Xu, C., Li, X., Wang, T., Que, X., Cira, C., Wang, B., Liu, Z., Bailey, B., Yu, W.: Assessing the performance impact of high-speed interconnects on MapReduce. In: Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics) LNCS, vol. 8163, pp. 148–163 (2014)CrossRef
50.
Islam, N.S., Lu, X., Wasi-ur-Rahman, M., Panda, D.K.: Can parallel replication benefit Hadoop distributed file system for high performance interconnects? In: 2013 IEEE 21st Annual Symposium on High-Performance Interconnects, pp. 75–78. IEEE (2013)
51.
Moore, J., Chase, J., Ranganathan, P., Sharma, R.: Making scheduling cool: temperature-aware workload placement in data centers (2005)
52.
Rajovic, N., Puzovic, N., Vilanova, L., Villavieja, C., Ramirez, A.: The low-power architecture approach towards exascale computing. In: Proceedings of the Second Workshop on Scalable Algorithms for Large-Scale Systems—ScalA’11, p. 1. ACM Press, New York (2011)
53.
Cappello, F.: Fault tolerance in petascale/exascale systems: current knowledge, challenges and research opportunities. Int. J. High Perform. Comput. Appl. 23, 212–226 (2009)CrossRef
54.
55.
56.
Willard, C.G., Snell, A., Segervall, L.: Top six predictions for HPC in 2015 (2015)
57.
58.
Ahmed, W., Khan, M., Khan, A.A., Mehmood, R., Algarni, A., Albeshri, A., Katib, I.: A framework for faster porting of scientific applications between heterogeneous clouds. In: Lecture Notes of the Institute for Computer Sciences, Social-Informatics and Telecommunications Engineering, LNICST, pp. 27–43. Springer, Cham (2018)
59.
Alotaibi, S., Mehmood, R.: Big data enabled healthcare supply chain management: opportunities and challenges. In: Mehmood, R., Bhaduri, B., Katib, I., Chlamtac, I. (eds.) Smart Societies, Infrastructure, Technologies and Applications. SCITA 2017. Lecture Notes of the Institute for Computer Sciences, Social Informatics and Telecommunications Engineering, pp. 207–215. Springer, Cham (2018)
60.
Alamoudi, E., Mehmood, R., Albeshri, A., Gojobori, T.: DNA profiling methods and tools: a review. In: Mehmood, R., Bhaduri, B., Katib, I., Chlamtac, I. (eds.) Smart Societies, Infrastructure, Technologies and Applications. SCITA 2017. Lecture Notes of the Institute for Computer Sciences, Social Informatics and Telecommunications Engineering, pp. 216–231. Springer, Cham (2018)
61.
Al Shehri, W., Mehmood, R., Alayyaf, H.: A smart pain management system using big data computing. In: Mehmood, R., Bhaduri, B., Katib, I., Chlamtac, I. (eds.) Smart Societies, Infrastructure, Technologies and Applications. SCITA 2017. Lecture Notes of the Institute for Computer Sciences, Social Informatics and Telecommunications Engineering, pp. 232–246. Springer, Cham (2018)
62.
Khanum, A., Alvi, A., Mehmood, R.: Towards a semantically enriched computational intelligence (SECI) framework for smart farming. In: Mehmood, R., Bhaduri, B., Katib, I., Chlamtac, I. (eds.) Smart Societies, Infrastructure, Technologies and Applications. SCITA 2017. Lecture Notes of the Institute for Computer Sciences, Social Informatics and Telecommunications Engineering, pp. 247–257. Springer, Cham (2018)
63.
Suma, S., Mehmood, R., Albeshri, A.: Automatic event detection in smart cities using big data analytics. In: Mehmood, R., Bhaduri, B., Katib, I., Chlamtac, I. (eds.) Smart Societies, Infrastructure, Technologies and Applications. SCITA 2017. Lecture Notes of the Institute for Computer Sciences, Social Informatics and Telecommunications Engineering, pp. 111–122. Springer, Cham (2018)
64.
Aqib, M., Mehmood, R., Albeshri, A., Alzahrani, A.: Disaster management in smart cities by forecasting traffic plan using deep learning and GPUs. In: Mehmood, R., Bhaduri, B., Katib, I., Chlamtac, I. (eds.) Smart Societies, Infrastructure, Technologies and Applications. SCITA 2017. Lecture Notes of the Institute for Computer Sciences, Social Informatics and Telecommunications Engineering, pp. 139–154. Springer, Cham (2018)
65.
Alam, F., Mehmood, R., Katib, I.: D2TFRS: an object recognition method for autonomous vehicles based on RGB and spatial values of pixels. In: Mehmood, R., Bhaduri, B., Katib, I., Chlamtac, I. (eds.) Smart Societies, Infrastructure, Technologies and Applications. SCITA 2017. Lecture Notes of the Institute for Computer Sciences, Social Informatics and Telecommunications Engineering, pp. 155–168. Springer, Cham (2018)
66.
Muhammed, T., Mehmood, R., Albeshri, A.: Enabling reliable and resilient IoT based smart city applications. In: Mehmood, R., Bhaduri, B., Katib, I., Chlamtac, I. (eds.) Smart Societies, Infrastructure, Technologies and Applications. SCITA 2017. Lecture Notes of the Institute for Computer Sciences, Social Informatics and Telecommunications Engineering, pp. 169–184. Springer, Cham (2018)
67.
Al-Dhubhani, R., Mehmood, R., Katib, I., Algarni, A.: Location privacy in smart cities era. In: Mehmood, R., Bhaduri, B., Katib, I., Chlamtac, I. (eds.) Smart Societies, Infrastructure, Technologies and Applications. SCITA 2017. Lecture Notes of the Institute for Computer Sciences, Social Informatics and Telecommunications Engineering, pp. 123–138. Springer, Cham (2018)
68.
Alomari, E., Mehmood, R.: Analysis of Tweets in Arabic Language for detection of road traffic conditions. In: Mehmood, R., Bhaduri, B., Katib, I., Chlamtac, I. (eds.) Smart Societies, Infrastructure, Technologies and Applications. SCITA 2017. Lecture Notes of the Institute for Computer Sciences, Social Informatics and Telecommunications Engineering, pp. 98–110. Springer, Cham (2018)
69.
Arfat, Y., Aqib, M., Mehmood, R., Albeshri, A., Katib, I., Albogami, N., Alzahrani, A.: Enabling smarter societies through mobile big data fogs and clouds. Procedia Comput. Sci. 109, 1128–1133 (2017)CrossRef
70.
Schlingensiepen, J., Nemtanu, F., Mehmood, R., McCluskey, L.: Autonomic transport management systems—enabler for smart cities, personalized medicine, participation and industry grid/industry 4.0. In: Intelligent Transportation Systems—Problems and Perspectives, pp. 3–35. Springer, Cham (2016)
71.
Arfat, Y., Mehmood, R., Albeshri, A.: Parallel shortest path graph computations of United States Road Network Data on Apache Spark. In: Mehmood, R., Bhaduri, B., Katib, I., Chlamtac, I. (eds.) Smart Societies, Infrastructure, Technologies and Applications. SCITA 2017. Lecture Notes of the Institute for Computer Sciences, Social Informatics and Telecommunications Engineering, pp. 323–336. Springer, Cham (2018)
72.
El Baz, D.: IoT and the need for high performance computing. In: 2014 International Conference on Identification, Information and Knowledge in the Internet of Things, pp. 1–6. IEEE (2014)
73.
Conway, S: High performance data analysis (HPDA): HPC—big data convergence—insideHPC (2017)
74.
Keutzer, K., Tim, M.: Our pattern language_our pattern language, file:///Users/abdulmanan/Desktop/Our Pattern Language_Our Pattern Language.htm (2016)
75.
Bodkin, R., Bodkin, R.: Big data patterns, pp. 1–23 (2017)
76.
Metadaten
Titel
Big Data and HPC Convergence for Smart Infrastructures: A Review and Proposed Architecture
verfasst von
Sardar Usman
Rashid Mehmood
Iyad Katib
Copyright-Jahr
2020
Buch
Smart Infrastructure and Applications

Print ISBN: 978-3-030-13704-5

Electronic ISBN: 978-3-030-13705-2

Copyright-Jahr: 2020

DOI
https://doi.org/10.1007/978-3-030-13705-2_23

Neuer Inhalt

    Bildnachweise