Harsh Mishra
ABSTRACT
Over the last few decades, there has been a tremendous increase in the volume of data available for analysis in various domains. Although processing power has scaled up as well, it is well known that the rate of increase of data far supersedes the higher processing capabilities of modern processors. The natural consequence to the advent of big data was distribution of data across multiple nodes to facilitate not only storage but also parallel processing. The advent of the age of large volumes of data came to be known as the era of big data. The distribution of data among various machines posed a fundamental problem in big data as well as distributed computing: The impact of data skew. We worked on a project to profile data skew on a multi-computing cluster. This paper summarizes our efforts and findings. We use HPCC Systems, a modern big data management and analysis tool. In this project, we analyze the impact of differently skewed data distributions on the most common database operations, namely, NORMALIZE, DENORMALIZE, JOIN, SORT, TABLE, and PROJECT using a set of queries, and analyzing their runtimes.
- L. Xu, E. Muharemagic, F. Villanustre and A. Apon, "IEEE2941 ECL-Watch: A Big Data Application Performance Tuning Tool in the HPCC Systems Platform" in 2017 IEEE International Conference on Big Data (BIGDATA)Google Scholar
- H. Jagadish, J. Gehrke, A. Labrinidis, Y. Papakonstantinou, J. M. Patel, R. Ramakrishnan, and C. Shahabi, "Big Data and its Technical Challenges," Communications of the ACM, vol. 57, no. 7, pp. 86--94, 2014. Google ScholarDigital Library
- M. Anthony and C. Arjuna. (2011) Introduction to HPCC(High-PerformanceComputing Cluster).{Online}.Available:http://cdn.hpccsystems.com/whitepapers/wp introduction HPCC.pdf.Google Scholar
- M. Anthony and C. Arjuna. (2017) Using ECL-Watch. {Online}. Available: http://cdn.hpccsystems.com/releases/CE-Candidate-6.4.2/docs/The ECL Watch Manual-6.4.2--1.pdfGoogle Scholar
- HPCCSystems.(2017)HPCCSystemAdministrator'sGuide.{Online}.Available:http://cdn.{hpcc}systems.com/releases/CE-Candidate-6.4.2/docs/HPCCSystemAdministratorsGuide-6.4.2--1.pdfGoogle Scholar
- K. Shvachko, H. Kuang, S. Radia, and R. Chansler, "TheHadoop distributed file system," in Mass storage systems andtechnologies (MSST), 2010 IEEE 26th symposium on. IEEE, 2010, pp. 1--10 Google ScholarDigital Library
- Yufei Gao, Yanjie Zhou, Bing Zhou, Lei Shi, and Jiacai Zhang, "Handling Data Skew in MapReduce Cluster by Using Partition Tuning," Journal of Healthcare Engineering, vol. 2017, Article ID 1425102, 12 pages, 2017.Google Scholar
- HPCC Systems vs Hadoop Detailed Comparison. Available: https://hpccsystems.com/about/hpcc-hadoop-comparisonGoogle Scholar
- IMDBDataset.Available:ftp://ftp.fu-berlin.de/pub/misc/movies/database/Google Scholar
- NYC Taxi dataset. Available: https://github.com/toddwschneider/nyc-taxi-data/tree/master/dataGoogle Scholar
Index Terms
- Data Skew Profiling using HPCC Systems
Recommendations
A Brief Survey on Big Data in Healthcare
This article presents a brief introduction to big data and big data analytics and also their roles in the healthcare system. A definite range of scientific researches about big data analytics in the healthcare system have been reviewed. The definition ...
Responsible Big Data Analytics for E-Business Services
ICBDR '21: Proceedings of the 5th International Conference on Big Data ResearchThis paper examines responsible big data analytics for e-business services and looks at how to use responsible big data analytics to obtain responsible e-business services. It addresses why responsibility matters to big data analytics and e-business ...
A Major Threat to Big Data: Data Security
ICTCS '16: Proceedings of the Second International Conference on Information and Communication Technology for Competitive StrategiesBig Data has nowadays become the most talked latest IT trends. The fact that it can handle all the forms of data which includes unstructured data, big data has now become the preferred choice for analysis of huge amount of data over the Relational ...
Comments