Communication Bandwidth of Parallel Programming Models on Hybrid Architectures

Most HPC systems are clusters of shared memory nodes. Parallel programming must combine the distributed memory parallelization on the node inter-connect with the shared memory parallelization inside of each node. This paper introduces several programming models for hybrid systems. It focuses on programming methods that can achieve optimal inter-node communication bandwidth and on the hybrid MPI+OpenMP approach and its programming rules. The communication behavior is compared with the pure MPI programming paradigm and with RDMA and NUMA based programming models.