Anzeige
Erschienen in:
2017 | OriginalPaper | Buchkapitel
A Lightweight Elastic Queue Middleware for Distributed Streaming Pipeline
verfasst von : Weiping Qu, Stefan Dessloch
Erschienen in: Big Data Analytics and Knowledge Discovery
Aktivieren Sie unsere intelligente Suche, um passende Fachinhalte oder Patente zu finden.
Wählen Sie Textabschnitte aus um mit Künstlicher Intelligenz passenden Patente zu finden. powered by
Markieren Sie Textabschnitte, um KI-gestützt weitere passende Inhalte zu finden. powered by
Abstract
We introduce an elastic queue middleware (EQM) in a distributed streaming processing architecture to handle drastically growing input streams at peak times and maintain resource utilization at off-peak times. EQM serves as a scalable stream buffer to solve bottlenecks of stream processing on the fly. With spikes in data rates, the stream buffer which holds the input tuples for a bottleneck operator scales out in EQM to immediately alleviate back pressure and the streaming engines can thus gradually deploy additional replicas of the bottleneck operator to cope with the increasing data rates. This differs from general elastic streaming processing where bottleneck operators scale out first and then the stream buffers are allocated. To implement a scalable buffer, EQM utilizes existing scalable data stores (e.g. HBase) to avoid re-inventing the same elasticity and scalability logic and meanwhile ensures load balancing performance. Experiment results show that stable throughput is achieved at varying data rates using EQM.