“Zhores” — Petaflops supercomputer for data-driven modeling, machine learning and artificial intelligence installed in Skolkovo Institute of Science and Technology

2.1 Servers’ Processor Characteristics

The servers have the latest generation of the Intel Xeon processors and Nvidia Volta GPUs. The basic characteristics of each type of the servers are captured in Table 1. We have measured the salient features of these devices.

Intel Xeon 6136 and 6140 “Gold” CPUs of Skylake generation differ by the total number of cores in the package and the working clock frequency (F). Each core features two floating point AVX512 units. This has been tested with a special benchmark to verify that the performance varies with the frequency as expected.

The CPU performance and memory bandwidth of a single core is shown in Figure 2. The benchmark program to test the floating point calculation performance is published elsewhere [3]. It is an unrolled vector loop with vector width 8, precisely tuned for the AVX512 instruction set. In this loop exactly 8 double precision numbers will be computed in parallel in two execution units of each core. With two execution units and the fused multiply-add instruction (FMA) the theoretical Double Precision (DP) performance of a single physical core is 8 × 2 × 2 × F [GHz] and for the maximum of F = 3.5 GHz may reach 112 GFlop/s/core. The performance scales with the frequency to the maximum determined by processor thermal and electrical limits. The total FMA performance on a node when running AVX512 code on all processors in parallel is about 2.0 TFlop/s for C6140 machines (cn nodes, 24 cores) and 2.4 TFlop/s for the C4140 (gn nodes, 36 cores). Summing up all the cn and gn nodes gives the measured maximum CPU performance on the “Zhores” cluster of 150 TFlop/s.

Figure 2

Floating point performance (FMA instructions) on 6136 CPU core and memory bandwidth (STREAM Triad) as a function of clock frequency. Left ordinate shows the FMA performance, the right ordinate represents the memory bandwidth.

The latencies of the processor memory subsystem have been measured with the LMBench program [23] and summarized in Table 2.

The main memory performance is measured with the STREAMS program [22] and shown for the single core as a function of clock frequency in Figure 2. The theoretical performance of the memory bandwidth may be estimated with the Little Law [15] to 14 GB/s per each channel taking into account the memory latency of 27.4 ns given in Table 2. The total memory bandwidth (STREAM Triad) for all cores reached 178.6 GB/s in our measurement using all 6 channels of 2666 MHz DIMMs.

The strong dependency of the FMA performance on the processor clock frequency and the weak dependency of the memory bandwidth on the clock frequency is noted to propose a scheme for the optimization of the power usage for applications with mixed instruction profiles.

2.2 Nvidia V100 GPU

Significant nodes (26) in the “Zhores” cluster are equipped with four Nvidia V100 GPUs each. The GPUs are connected pairwise with NVLink and individually with PCIe gen3 x16 to the CPU host. The principal scheme of the connections is shown in Figure 3. The basic measurements to label the links in the plot have been obtained with Nvidia p2p bandwidth program from the “Samples” directory loaded with GPU drivers. This setup is optimized for parallel computation scaling within the node, while the connections to the cluster network pass from the single PCIe link. The maximum estimated performance of a single V100 GPU is shown in Figure 4. The graphics clock rate was set with the command “nvidia-smi”; same command with different parameters lists the power draw of the device. The computational efficiency measured in Performance per Watt is not evenly distributed as function of frequency, the peak is 67.4 GFlop/s/W (single precision) at 1 GHz and drops to 47.7 GFlop/s/W at 1.5 GHz.

Figure 3

Principal connections between host and graphics subsystem on graphics nodes.

Figure 4

Nvidia V100 GPU floating point performance as a function of graphics clock rate. Electrical power draw corresponding to the set frequency is indicated on the upper axis.

2.3 Mellanox IB EDR network

The high performance cluster network has the Fat Tree topology and is build from six Mellanox SB7890 (unmanaged) and two SB7800 (managed) switches that provide 100 Gbit/s (IB EDR) connections between the nodes. The performance of the interconnect has been measured with the “mpilink” program that times the ping-pong exchange between each node [1]. To make the measurements we have installed Mellanox HPC package drivers and used openMPI version 3.1.2. The results are shown in Figure 5 for serial mode runs and in Figure 6 for parallel mode runs.

Figure 5

Histogram of the ping-pong times/speeds between all nodes using 1 MB packets in serial mode

Figure 6

Histogram of the ping-pong times/speeds between all nodes using 1 MB packets in parallel mode

The serial mode sends packages to each node when previous communication has finished, while in parallel mode all sends and receives are issued at the same time. The parallel mode probes the package contention, while serial mode allows to establish the absolute speed and discover any failing links. The communication in serial mode is centered around the speed of 10.2±0.5GB/s. The parallel mode reveals certain over-subscription of the Fat Tree network — while the computational nodes are balanced the additional traffic from the file services causes delays in the transmission. This problem will be addressed in future upgrades.

2.4 Operating System and cluster management

The “Zhores” cluster is managed by “Luna” [8] provisioning tool which can be installed in a fault tolerant active-passive cluster setup with TrinityX platform. The Luna management system was developed by ClusterVision BV. The system automates the creation of all the services and cluster configuration that make a bunch of servers a unified computational machine.

The cluster management software supports the following essential features:

1. All cluster configuration is kept in the Luna database and all cluster nodes boot from this information which is held in one place. This database is mirrored between the management nodes with the DRBD filesystem and the active management node provides access to data for every node in the cluster with the NFS share, see Figure 7.

2. Node provisioning from OS images is based on the BitTorrent protocol [2] for efficient simultaneous (disk-less or standard) boot; the image management allows to grab an OS image from a running node to a file, clone images for testing or backup purposes; a group of nodes can use the same image for provisioning that fosters unification of cluster configuration. Nodes use PXE protocol to load a service image that implements the booting procedure.

3. All the nodes (or groups of nodes) in the cluster can be switched on/off and reset with the IPMI protocol from the management nodes with a single command.

4. The cluster services setup on the management node in a fault tolerant way include the following: DHCP, DNS, OpenLDAP, Slurm, Zabbix, Docker-repository, etc.

Figure 7

Organization of the “Zhores” cluster management with Luna system.

The management nodes are based on CentOS 7.5 and force same OS on the compute nodes; additional packages, specific drivers and different kernel versions can be included in the images for the cluster nodes. The installation requires each node to have at least two ethernet network interfaces, one dedicated to the management traffic and the other used for administrative access. A single cluster node can be booted within 2.5 minutes (over 1 GbE), and the whole “Zhores” cluster cold start takes 5 minutes to fully operational state.

2.5 The queueing system

Work queues have been organized with the Slurm workload manager to reflect the different application profiles of users of the cluster. Several nodes have been given to dedicated projects (gn26, anlab) and one CPU-only node is setup for debugging work (cn44). The remaining nodes have been combined in queues for the GPU-nodes (gn01– gn25) and for the CPU-nodes (cn01–cn43).

2.6 Linpack run

The Linpack benchmark was performed as a part of the cluster evaluation procedure and to rate the supercomputer for the performance comparison. The results of the run are shown in Table 3 separately for the GPU and all nodes using only CPU computation.

The “Zhores” supercomputer is significant for the Russian computational science community, and currently, it places 7th in the Russian and CIS TOP-50 list [6].

3.1 Algorithms for aggregation and fragmentation equations

In our benchmarks we used parallel implementation of efficient numerical methods for the aggregation and fragmentation equations [13, 18] and also parallel implementation of the solver for advection-driven coagulation process [16]. Its sequential version has already been utilized in a number of applications [17, 19] and can be considered as one of the most efficient algorithms for a class of Smoluchowski-type aggregation kinetic equations. It is worth to stress that parallel algorithm for pure aggregation-fragmentation equations is based mostly on the performance of ClusterFFT operation which is a dominating operation in terms of algorithmic complexity, thus its scalability is extremely limited. Nevertheless for 128 cores we obtain speedup of calculations by more than 85 times, see Table 4.

In the case of the parallel solver, for advection-driven coagulation [21] we obtain almost ideal acceleration with utilization of the algorithm for almost full CPU-based segment. In this case, the algorithm is based on the one-dimensional domain decomposition along the spatial coordinate and has a very good scalability, see Table 5 and Figure 8. The experiments have been performed using Intel® compilers and the Intel® MKL library.

Figure 8

Parallel advection-driven aggregation solver on CPUs, Ballistic kernel, domain size N = 12288.

Alongside with the consideration of the well-known two-particle problem of aggregation, we have measured the performance for a parallel implementation of a more general three-particle (ternary) Smoluchowski-type kinetic aggregation equations [29]. In this case the algorithm is somewhat similar to the one for standard binary aggregation. However the number of the floating point calculations and the size of the allocated memory increases as compared to the binary case, because the dimension of the low rank Tensor Train (TT) decomposition [25] is naturally bigger in ternary case. The most computationally expensive operation in the parallel implementation of the algorithm is also the ClusterFFT. The speedup of the parallel ternary aggregation algorithm applied to the empirically derived ballistic-like kinetic coefficients [20] is shown in Table 6. In full accordance with the structure of ClusterFFT and the problem complexity one needs to increase the parameter N of the used differential equations in order to obtain scalability. Speedups for both implementations of binary and ternary aggregation are shown on Figure 9. The experiments have been performed using Intel compilers and Intel MKL library.

Figure 9

Parallel binary and ternary aggregation solvers on CPU, Ballistic-like kernels, 16 and 10 time-integration steps for N = 2²² and N = 2¹⁹ nonlinear ODEs, respectively. Parameter R denotes the rank of used matrix and tensor decompositions.

3.2 Gromacs

Classical molecular dynamics is an effective method with high predictive ability in a wide range of scientific fields [11, 31]. Using Gromacs 2018.3 software [7, 33] we have performed molecular dynamics simulations in order to test the “Zhores” cluster performance. As a model system we chose 125 million Lennard-Jones spheres with the Van der Waals cut-off radius of 1.2 nm and with the Berendsen thermostat. All tests were conducted with a single precision version of Gromacs.

The results are presented in Figure 10. We measured the performance as a function of the number of nodes; we have used up to 40 CPU nodes and up to 24 GPU nodes. We have used 4 OpenMP threads per MPI process. Each task was performed 5 times with following averaging in order to obtain final performance. Grey and red solid lines show linear acceleration of the program on CPU and GPU nodes, respectively. In case of the CPU-nodes, one can see almost ideal speedup. With a large number of CPU-nodes, the speedup deviates from linear and grows slower.

Figure 10

Performance of the molecular dynamic simulations of 125 million Lennard-Jones spheres using Gromacs 2018.3 as a function of nodes number. Note, that there are only 26 GPU nodes in the cluster.

To test performance on the GPU-nodes, we have performed simulations with 1, 2 and 4 graphics cards per node. The use of all 4 graphics cards demonstrates good scalability, while 2 GPUs per node shows slightly lower speedup. Runs with 1 GPU per node demonstrates worse performance, especially with high number of nodes. To compare the efficiency for different number of GPUs per node, we show the performance for the four configurations (0, 1, 2 and 4 GPUs) using 24 GPU-nodes in Figure 11 as a bar chart. The 4 GPUs per node configuration gives about 2.5 times higher performance than running the program only on the CPU cores. And even 1 GPU per node gives significant performance increase compared to the CPU only run.

Figure 11

Performances for different configurations of 24 GPU-nodes: 0, 1, 2, and 4 GPUs per node