Optimizing communication performance in scale-out storage system

Ceph is an object-based scale-out storage system that is widely used in the cloud computing environment due to its scalable and reliable characteristics. Although there are many factors to affect the performance of scale-out storage systems, the design of a communication subsystem plays an important role in determining the overall performance of these systems. In this paper, we first conduct an extensive analysis of communication subsystem in Ceph, which uses asynchronous messenger framework, called async messenger, for inter-component communication in the storage cluster. Then, we propose three optimization techniques to improve the performance of Ceph messenger. These include (i) deploying load balancing algorithm among worker threads based on the amount of workloads, (ii) assigning multiple worker threads (we call dual worker) per single connection to maximize the overlapping activity among threads, and (iii) using multiple connections between storage servers to maximize bandwidth usage. The experimental results show that the optimized Ceph messenger outperforms the original messenger implementation by up to 40% in random writes with 4 K messages. Moreover, Ceph with optimized communication subsystem shows up to 13% performance improvement as compared to original Ceph.