nach oben

Erschienen in:

2021 | OriginalPaper | Buchkapitel

Data Processing and Development of Big Data System: A Survey

verfasst von : Shuyan Yu

Erschienen in: Advances in Artificial Intelligence and Security

Verlag: Springer International Publishing

Einloggen

Aktivieren Sie unsere intelligente Suche, um passende Fachinhalte oder Patente zu finden.

search-config

KI-gestützte Suche

Aus

Abstract

At present, data are generated all the time in the world, and these data contain great value. However, these data have problems such as huge scale, complex type, fast flow rate and low value density. Therefore, the research of big data analysis means and tools has increasingly become a research hotspot. On the basis of research at home and abroad, this paper attempts to analyze the definition, framework and typical big data processing systems of big data, especially focuses on the analysis and comparison of batch data processing system, stream data processing system, hybrid processing system and graph processing system, and obtains the characteristics, processing mechanism and applicable occasions of each system. The paper hopes to provide some references for understanding big data systems, solving problems in the process of big data processing and developing big data applications, and provide reference for improving the effectiveness and efficiency of data processing.

Sie haben noch keine Lizenz? Dann Informieren Sie sich jetzt über unsere Produkte:

Springer Professional "Wirtschaft+Technik"

Online-Abonnement

Mit Springer Professional "Wirtschaft+Technik" erhalten Sie Zugriff auf:

über 102.000 Bücher
über 537 Zeitschriften

aus folgenden Fachgebieten:

Automobil + Motoren
Bauwesen + Immobilien
Business IT + Informatik
Elektrotechnik + Elektronik
Energie + Nachhaltigkeit
Finance + Banking
Management + Führung
Marketing + Vertrieb
Maschinenbau + Werkstoffe
Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

Jetzt informieren

Springer Professional "Technik"

Online-Abonnement

Mit Springer Professional "Technik" erhalten Sie Zugriff auf:

über 67.000 Bücher
über 390 Zeitschriften

aus folgenden Fachgebieten:

Automobil + Motoren
Bauwesen + Immobilien
Business IT + Informatik
Elektrotechnik + Elektronik
Energie + Nachhaltigkeit
Maschinenbau + Werkstoffe

Jetzt Wissensvorsprung sichern!

Jetzt informieren

Springer Professional "Wirtschaft"

Online-Abonnement

Mit Springer Professional "Wirtschaft" erhalten Sie Zugriff auf:

über 67.000 Bücher
über 340 Zeitschriften

aus folgenden Fachgebieten:

Bauwesen + Immobilien
Business IT + Informatik
Finance + Banking
Management + Führung
Marketing + Vertrieb
Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

Jetzt informieren

Vorheriges Kapitel A Tag Recommendation Method for OpenStreetMap Based on FP-Growth and Improved Markov Process

Nächstes Kapitel Current PHM Surveys for Mechanical Engineering

Canaj, E., Xhuvani, A.: Big data in cloud computing: a review of key technologies and open issues. In: Barolli, L., Xhafa, F., Javaid, N., Spaho, E., Kolici, V. (eds.) EIDWT 2018. LNDECT, vol. 17, pp. 504–513. Springer, Cham (2018). https://doi.org/10.1007/978-3-319-75928-9_45CrossRef

Beniamino, D.M., Rocco, A., Giuseppina, C., Antonio, E., Joanna, K.: Big data (lost) in the cloud. Int. J. Big Data Intell. 1(1), 3–17 (2014)CrossRef

Chen, M., Mao, S., Liu, Y.: Big data: a survey. Mob. Netw. Appl. 19(2), 171–209 (2014). https://doi.org/10.1007/s11036-013-0489-0CrossRef

Bello-Orgaz, G., Jung, J., Camacho, D.: Social big data: recent achievements and new challenges. Inf. Fusion 28, 45–59 (2016)CrossRef

The Digitization of the World: From Edge to Core. https://resources.moredrect.com/white-papers/idc-report-the-digitization-of-the-world-from-edge-to-core. Accessed 27 Nov 2018

Big data: the next frontier for innovation, competition, and productivity. https://www.mckinsey.com/business-functions/mckinsey-digital/our-insights/big-data-the-next-frontier-for-innovation. Accessed 1 May 2011

Extracting Value from Chaos. https://www.smartdatacollective.com/digital-universe-study-extracting-value-chaos/. Accessed 27 June 2011

Liu, C.: Market research and forecast. Mech. Ind. press 8(1), 84–85 (2017)

Zikopoulos, P., Eaton, C.: Understanding Big Data: Analytics for Enterprise Class Hadoop and Streaming Data, 1st edn. McGraw-Hill Osborne Media, New York (2011)

10.

3D Data Management: Controlling Data Volume, Velocity, and Variety. https://www.bibsonomy.org/bibtex/742811cb00b303261f79a98e9b80bf49. Accessed 1 Feb 2001

11.

Kumar, A., Maskara, S., Chiang, I.: Identifying semantic in high-dimensional web data using latent semantic manifold. J. Data Anal. Inf. Process. 3(4), 136–152 (2015)

12.

Li, X., Gong, H.: Overview of big data system. Sci. China 45(1), 1–44 (2015)MathSciNetCrossRef

13.

Notice on Printing and Distributing the Action Plan for Promoting Big Data Development. http://www.gov.cn/zhengce/content/2015-09-05/content_10137.html Accessed 05 Sept 2015

14.

The digital universe decade-are you ready. https://www.researchgate.net/publication/229124759_The_Digital_Universe_Decade_-_Are_You_Ready. Accessed 05 Jan 2010

15.

Zaharia, M., Chowdhury, M., Franklin, M., Shenker, S. and Stoica, I.: Spark: cluster computing with working sets. In: Proceedings of the 2nd USENIX conference on Hot topics in cloud computing, pp. 10–17. USENIX Association, USA (2010)

16.

Apache Spark. http://spark.apache.org/. Accessed 18 Apr 2020

17.

Apache Flink. https://flink.apache.org/. Accessed 26 May 2020

18.

Yong, T.: Comparative study on calculation models of spark and flink. Comput. Products Circ. 2(4), 152–153 (2019)

19.

Dai, M., Gao, S.: Performance evaluation of large-scale data analysis based on hadoop, spark and flink. J. Chin. Acad. Electron. Sci. 13(2), 149–155 (2018)

20.

We knew the web was big. https://googleblog.blogspot.com/2008/07/we-knew-web-was-big.html. Accessed 25 July 2008

21.

Pregel. http://kowshik.github.io/Jpregel/. Accessed 25 July 2009

22.

Brief History of Graph Computing System Development. https://zhuanlan.zhihu.com/p/79169412 . Accessed 21 Oct 2019

23.

Comparison of GOD_WAR, GraphX, GraphLab, Pregel. https://blog.csdn.net/young_0609/article/details/101021094Accessed 19 Sept 2019

24.

Sun, L., Yu, Q., Peng, D., Subramani, S., Wang, X.: a fog-based framework for disease prognosis based medical sensor data streams. Comput. Mater. Continua 66(1), 603–619 (2021)CrossRef

25.

Alhroob, A., Alzyadat, W., Imam, A.T., Jaradat, G.M.: The genetic algorithm and binary search technique in the program path coverage for improving software testing using big data. Intell. Autom. Soft Comput. 26(4), 725–733 (2020)CrossRef

26.

Mirarab, A., Mirtaheri, S.L., Asghari, S.A.: A model to create organizational value with big data analytics. Comput. Syst. Sci. Eng. 35(2), 69–79 (2020)CrossRef

Titel: Data Processing and Development of Big Data System: A Survey
verfasst von: Shuyan Yu
Verlag: Springer International Publishing
Buch: Advances in Artificial Intelligence and Security
Print ISBN: 978-3-030-78617-5

Electronic ISBN: 978-3-030-78618-2

Copyright-Jahr: 2021
DOI: https://doi.org/10.1007/978-3-030-78618-2_34

Springer Professional

Abstract

Bitte loggen Sie sich ein, um Zugang zu Ihrer Lizenz zu erhalten.

Sie haben noch keine Lizenz? Dann Informieren Sie sich jetzt über unsere Produkte:

Springer Professional "Wirtschaft+Technik"

Springer Professional "Technik"

Springer Professional "Wirtschaft"

Premium Partner