2009 | OriginalPaper | Chapter
Job Admission and Resource Allocation in Distributed Streaming Systems
Authors : Joel Wolf, Nikhil Bansal, Kirsten Hildrum, Sujay Parekh, Deepak Rajan, Rohit Wagle, Kun-Lung Wu
Published in: Job Scheduling Strategies for Parallel Processing
Publisher: Springer Berlin Heidelberg
Activate our intelligent search to find suitable subject content or patents.
Select sections of text to find matching patents with Artificial Intelligence. powered by
Select sections of text to find additional relevant content using AI-assisted search. powered by
This paper describes a new and novel scheme for job admission and resource allocation employed by the
SODA
scheduler in
System S
. Capable of processing enormous quantities of streaming data,
System S
is a large-scale, distributed stream processing system designed to handle complex applications. The problem of scheduling in distributed, stream-based systems is quite unlike that in more traditional systems. And the requirements for
System S
, in particular, are more stringent than one might expect even in a “standard” stream-based design. For example, in
System S
, the offered load is expected to vastly exceed system capacity. So a careful job admission scheme is essential. The jobs in
System S
are essentially directed graphs, with software “processing elements” (
PE
s) as vertices and data streams as edges connecting the PEs. The jobs themselves are often heavily interconnected. Thus resource allocation of individual PEs must be done carefully in order to balance the flow. We describe the design of the
SODA
scheduler, with particular emphasis on the component, known as
macroQ
, which performs the job admission and resource allocation tasks. We demonstrate by experiments the natural trade-offs between job admission and resource allocation.