Advertisement
Published in:
2017 | OriginalPaper | Chapter
A Lightweight Elastic Queue Middleware for Distributed Streaming Pipeline
Authors : Weiping Qu, Stefan Dessloch
Published in: Big Data Analytics and Knowledge Discovery
Publisher: Springer International Publishing
Activate our intelligent search to find suitable subject content or patents.
Select sections of text to find matching patents with Artificial Intelligence. powered by
Select sections of text to find additional relevant content using AI-assisted search. powered by
Abstract
We introduce an elastic queue middleware (EQM) in a distributed streaming processing architecture to handle drastically growing input streams at peak times and maintain resource utilization at off-peak times. EQM serves as a scalable stream buffer to solve bottlenecks of stream processing on the fly. With spikes in data rates, the stream buffer which holds the input tuples for a bottleneck operator scales out in EQM to immediately alleviate back pressure and the streaming engines can thus gradually deploy additional replicas of the bottleneck operator to cope with the increasing data rates. This differs from general elastic streaming processing where bottleneck operators scale out first and then the stream buffers are allocated. To implement a scalable buffer, EQM utilizes existing scalable data stores (e.g. HBase) to avoid re-inventing the same elasticity and scalability logic and meanwhile ensures load balancing performance. Experiment results show that stable throughput is achieved at varying data rates using EQM.