Joint Optimization of Computation and Communication Resources for GPU Allocation in Heterogeneous Clusters

This chapter delves into the complexities of GPU resource allocation in heterogeneous clusters, particularly in the context of distributed training for deep learning models. It highlights the limitations of existing resource allocation schemes, which either prioritize computational efficiency at the cost of significant communication overhead or vice versa. The chapter introduces HetSpeed, a novel resource allocation method designed to optimize both computational efficiency and communication overhead. HetSpeed formulates the job placement problem as an integer linear programming optimization problem and provides a submodular optimization-based solution. Experimental results demonstrate that HetSpeed can minimize the total cross-rack traffic of the cluster, shorten the average job completion time by 27.8%, and improve the cluster speedup value by 21.3% compared to state-of-the-art solutions. The chapter also discusses the NP-hardness of the problem and provides a detailed analysis of the performance of HetSpeed in various scenarios. Overall, this chapter offers valuable insights into the challenges of GPU resource allocation in heterogeneous clusters and presents a promising solution to address these challenges.