Advertisement
Published in:
2019 | OriginalPaper | Chapter
On Development of Data Science and Machine Learning Applications in Databricks
Authors : Wenhao Ruan, Yifan Chen, Babak Forouraghi
Published in: Services – SERVICES 2019
Publisher: Springer International Publishing
Activate our intelligent search to find suitable subject content or patents.
Select sections of text to find matching patents with Artificial Intelligence. powered by
Select sections of text to find additional relevant content using AI-assisted search. powered by
Abstract
Databricks is a unified analytics engine that allows rapid development of data science applications using machine learning techniques such as classification, linear and nonlinear regression, clustering, etc. Existence of myriad sophisticated computational options, however, can become overwhelming for designers as it may not always be clear what choices can produce the best predictive model given a specific data set. Further, the mere high dimensionality of big data sets is a challenge for data scientists to gain a deep understanding of the results obtained by a utilized model.
This paper provides general guidelines for utilizing a variety of machine learning algorithms on the cloud computing platform, Databricks. Visualization is an important means for users to understand the significance of the underlying data. Therefore, it is also demonstrated how graphical tools such as Tableau can be used to efficiently examine results of classification or clustering. The dimensionality reduction techniques such as Principal Component Analysis (PCA), which help reduce the number of features in a learning experiment, are also discussed.
To demonstrate the utility of Databricks tools, two big data sets are used for performing clustering and classification. A variety of machine learning algorithms are applied to both data sets, and it is shown how to obtain the most accurate learning models employing appropriate evaluation methods.