Anzeige
Erschienen in:
2020 | OriginalPaper | Buchkapitel
Latency Estimation of Big Data Processing Under the MapReduce Framework with Coupling Effects
verfasst von : Di Lin, Lingshuang Cai, Xiaofeng Zhang, Xiao Zhang, Jiazhi Huo
Erschienen in: Artificial Intelligence in China
Verlag: Springer Singapore
Aktivieren Sie unsere intelligente Suche, um passende Fachinhalte oder Patente zu finden.
Wählen Sie Textabschnitte aus um mit Künstlicher Intelligenz passenden Patente zu finden. powered by
Markieren Sie Textabschnitte, um KI-gestützt weitere passende Inhalte zu finden. powered by
Abstract
MapReduce is a model of processing large-scaled data with parallel and distributed algorithms at a cluster, and it is composed of two stages: a map stage for filtering and sorting data and a reduce stage for the operation of summary. We develop a model with two connected queues: one upstream queue for the data flow to access the mappers and one downstream queue for the data flow to access the reducers. Also, we analyze the latency of processing a large scale of data using queueing models, in consideration of the coupling effects between these two queues for map and reduce, respectively. Our analysis results on various datasets and with various algorithms show that the MapReduce framework can almost linearly speed up with increasingly more processors, and adding mappers is usually more efficient than adding reducers to reduce the latency when processing a large-scaled dataset.