Top

Published in:

2019 | OriginalPaper | Chapter

18. Hadoop: A Standard Framework for Computer Cluster

Authors : Eljar Akhgarnush, Lars Broeckers, Thorsten Jakoby

Published in: The Impact of Digital Transformation and FinTech on the Finance Professional

Publisher: Springer International Publishing

Activate our intelligent search to find suitable subject content or patents.

search-config

AI-assisted search

Off

Abstract

Hadoop has become a standard for processing big data in a clustered environment. This article provides an introduction to Hadoop/HDSF and other important Apache projects including Spark, Hive and HBase. The basic concepts like worker nodes and cluster manager are also introduced here.

Dont have a licence yet? Then find out more about our products and how to get one now:

Springer Professional "Wirtschaft+Technik"

Online-Abonnement

Mit Springer Professional "Wirtschaft+Technik" erhalten Sie Zugriff auf:

über 102.000 Bücher
über 537 Zeitschriften

aus folgenden Fachgebieten:

Automobil + Motoren
Bauwesen + Immobilien
Business IT + Informatik
Elektrotechnik + Elektronik
Energie + Nachhaltigkeit
Finance + Banking
Management + Führung
Marketing + Vertrieb
Maschinenbau + Werkstoffe
Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

inform now

Springer Professional "Wirtschaft"

Online-Abonnement

Mit Springer Professional "Wirtschaft" erhalten Sie Zugriff auf:

über 67.000 Bücher
über 340 Zeitschriften

aus folgenden Fachgebieten:

Bauwesen + Immobilien
Business IT + Informatik
Finance + Banking
Management + Führung
Marketing + Vertrieb
Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

inform now

previous chapter Deep Learning: An Introduction

next chapter In-Memory Databases and Their Impact on Our (Future) Organizations

RC—record columnar files.

ORC—optimized RC files.

Abyson, J. (2018, July 18). sap.com (SAP). Retrieved December 23, 2018, from sap.com https://blogs.sap.com/2017/07/18/understanding-the-basics-of-big-data-hadoop-and-sap-vora/.

Archive.org. (2018). archive.org (Firma). Retrieved December 22, 2018, from archive.org http://archive.apache.org/dist/hadoop/core/.

Aunkofer, B. (2016, August 3). data-science-blog.com. Retrieved December 23, 2018, from data-science-blog.com https://data-science-blog.com/blog/2016/08/03/was-ist-eigentlich-apache-spark/.

Bappalige, S. P. (2014, August 26). opensource.com. Retrieved December 22, 2018, from An introduction to Apache Hadoop for big data https://opensource.com/life/14/8/intro-apache-hadoop-big-data.

Baskaran, A. (2014, October 9). quora.com. Retrieved December 23, 2018, from quora.com https://www.quora.com/What-are-the-Apache-Spark-concepts-around-its-DAG-Directed-Acyclic-Graph-execution-engine-and-its-overall-architecture.

Burns, E. (2015, 9). searchenterprisesoftware.de. Retrieved December 23, 2018, from searchenterprisesoftware.de.

Capriolo, E., Wampler, D., & Rutherglen, J. (2012). Programming Hive: Data warehouse and query language for Hadoop. Sebastopol: O’Reilly Media, Inc.

Das, T. (2015, November 30). datanami.com. Retrieved December 23, 2018, from datanami.com https://www.datanami.com/2015/11/30/spark-streaming-what-is-it-and-whos-using-it/.

DataFlair Team. (2017, April 8). data-flair.training (Data Flair). Retrieved December 23, 2018, from data-flair.training https://data-flair.training/blogs/dag-in-apache-spark/.

Dean, J., & Ghemawat, S. (2008). MapReduce: Simplified data processing on large clusters. Communications of the ACM, 51(1), 107–113.

deRoos, D. (2013, April). dummies.com. Retrieved December 23, 2018, from dummies.com https://www.dummies.com/programming/big-data/hadoop/data-blocks-in-the-hadoop-distributed-file-system-hdfs/.

Dvorský, M. (2016, February 18). History of massive-scale sorting experiments at Google (Google). Retrieved December 22, 2018, from cloud.google.com https://cloud.google.com/blog/products/gcp/history-of-massive-scale-sorting-experiments-at-google.

Gerecke, K., & Poschke, K. (2010). IBM system storage-Kompendium. Ehningen, Germany: IBM.

Haas, K. (2017, November 7). hitachivantara.com. Retrieved December 23, 2018, from hitachivantara.com https://community.hitachivantara.com/community/products-and-solutions/pentaho/blog/2017/11/07/hadoop-file-formats-its-not-just-csv-anymore.

Kumar, A. (2017, September 14). quora.com. Retrieved December 23, 2018, from quora.com https://www.quora.com/What-is-NameNode-in-Hadoop.

Laskowski, J. (2018, December 4). jaceklaskowski.gitbooks.io. Retrieved December 23, 2018, from jaceklaskowski.gitbooks.io.

Litzel, N. (2016, September 1). bigdata-insider.de. Retrieved December 23, 2018, from bigdata-insider.de https://www.bigdata-insider.de/was-ist-spark-a-572706/.

Meerasahib, A. (2018, June 22). tdwi.org. Retrieved December 23, 2018, from tdwi.org https://tdwi.org/articles/2018/06/22/ta-all-apache-spark-and-big-data-whats-ahead.aspx.

Meng, X., & Bradley, J. (2016). MLlib: Machine learning in Apache Spark. Journal of Machine Learning Research, 1, 1235–1241.

Rouse, M., & Burns, E. (2017, June). techtarget.com. Retrieved December 23, 2018, from techtarget.com https://searchdatamanagement.techtarget.com/definition/Apache-Spark.

Schuler, F. (2017, November 28). sap.com (SAP). Retrieved December 23, 2018, from sap.com https://blogs.sap.com/2017/11/28/connect-your-sap-data-hub-to-sap-vora-and-hadoop/.

Shankar, N. (2017, April 5). sap.com (SAP). Retrieved December 23, 2018, from sap.com https://blogs.sap.com/2017/04/05/sap-meets-big-data-sap-hana-and-hadoop-ecosystem/.

Shreya. (2017, November 14). quora.com. Retrieved December 23, 2018, from quora.com https://www.quora.com/What-is-the-use-of-SparkContext-in-Apache-Spark.

The Apache Software Foundation. (2018). apache.org. Retrieved from apache.org https://parquet.apache.org/documentation/latest/.

Title: Hadoop: A Standard Framework for Computer Cluster
Authors: Eljar Akhgarnush
Lars Broeckers
Thorsten Jakoby
Publisher: Springer International Publishing
Book: The Impact of Digital Transformation and FinTech on the Finance Professional
Print ISBN: 978-3-030-23718-9

Electronic ISBN: 978-3-030-23719-6

Copyright Year: 2019
DOI: https://doi.org/10.1007/978-3-030-23719-6_18