2010 | OriginalPaper | Chapter
FLEX: A Slot Allocation Scheduling Optimizer for MapReduce Workloads
Authors : Joel Wolf, Deepak Rajan, Kirsten Hildrum, Rohit Khandekar, Vibhore Kumar, Sujay Parekh, Kun-Lung Wu, Andrey Balmin
Published in: Middleware 2010
Publisher: Springer Berlin Heidelberg
Activate our intelligent search to find suitable subject content or patents.
Select sections of text to find matching patents with Artificial Intelligence. powered by
Select sections of text to find additional relevant content using AI-assisted search. powered by
Originally, MapReduce implementations such as
Hadoop
employed
First In First Out
(
fifo
) scheduling, but such simple schemes cause job starvation. The
Hadoop Fair Scheduler
(
hfs
) is a slot-based MapReduce scheme designed to ensure a degree of fairness among the jobs, by guaranteeing each job at least some minimum number of allocated slots. Our prime contribution in this paper is a different,
flexible
scheduling allocation scheme, known as
flex
. Our goal is to optimize any of a variety of standard scheduling theory metrics (response time, stretch, makespan and
Service Level Agreements
(
sla
s), among others) while ensuring the same minimum job slot guarantees as in
hfs
, and maximum job slot guarantees as well. The
flex
allocation scheduler can be regarded as an add-on module that works synergistically with
hfs
. We describe the mathematical basis for
flex
, and compare it with
fifo
and
hfs
in a variety of experiments.