A Hybrid Resource Scheduling Strategy in Speculative Execution Based on Non-cooperative Game Theory

Hadoop is a well-known parallel computing framework for processing large-scale data, but there is such a task in the Hadoop framework called the “Straggling task” and has a serious impact on Hadoop. Speculative execution is an efficient method of processing “Straggling Tasks” by monitoring the real-time rate of running tasks and backing up “Straggler” on another node to increase the chance of an early completion of a backup task. The proposed speculative execution strategy has many problems, such as misjudgement of “Straggling task” and improper selection of backup nodes, which leads to inefficient implementation of speculative execution. This paper proposes a hybrid resource scheduling strategy in speculative execution based on non-cooperative game theory (HRSE), which transforms the resource scheduling of backup task in speculative execution into a multi-party non-cooperative game problem. The backup task group is the game participant and the game strategy is the computing node, the utility function is the overall task execution time of the cluster. When the game reaches the Nash equilibrium state, the final resource scheduling scheme is obtained. Finally, we implemented the strategy in Hadoop-2.6.0, experimental results show that the scheduling scheme can guarantee the efficiency of speculative execution and improve the fault-tolerant performance of the computation under the condition of high cluster load.