Using hardware counter-based performance model to diagnose scaling issues of HPC applications

Ever-growing supercomputers lead to more and more processing units. The scalability is regarded as one of the most important design objectives of HPC applications [1‐5]. To diagnose an application’s scaling performance, developers have to profile the applications several times on a given number of processes with instrumentation, which may introduce more than 10% overhead for each profiled application run [6, 7] as well as the possible performance deviation. At the same time, the detected performance issues using a specific number of processes may not still be issued when scaling the application. Even conducting laborious analysis on a few selected kernels of the whole application may lead to high risk of missing crucial bottlenecks. Besides, developers may not diagnose applications on real systems because they only have limited opportunities to access the large-scale machine.

Performance modeling is an effective way to understand the scaling performance behaviors and identify the scaling issues earlier. Compared to the traditional diagnose method, performance modeling method is able to predict the scaling performance behaviors by profiling on small-scale parallelisms. Alexandru et al. [1, 2] proposed a performance model to find potential kernels may become bottlenecks when scaling to a large number of processes. However, the kernels they detected do not separate communications and computations. The detected kernels all contain communications. Such results are easily predicted since the time cost of communication usually increases with the growing number of processes. Thus, the computation code sections that do not scale well are still hidden in the complex code.

Analytical performance modeling [8‐11] has ever been widely used to predict application execution time by analyzing the source code and algorithms. It utilizes algorithm details such as iteration times and the number of key variables, to diagnose the scaling runtime of each domain knowledge-oriented kernels. However, such method is highly depended on the algorithm and implementation. Such case-by-case performance modeling approach is very time- and labor-consuming, as well as lacking the portability across applications.

Statistical model [12, 13] tries to overcome the disadvantages by predicting the scaling performance according to a large number of sampled application runs without digging into the source codes. However, it introduces a large number of application runs to train the performance model for achieving satisfying accuracy.

To address the challenges, we propose a resource-based performance modeling method which has the relatively good portability across applications and low diagnose overhead. We diagnose applications’ scaling issue using the performance model-based method with consideration of both resource usages and interactions between applications and the hardware.

To summary, our contributions are:

1. Employing resource-based modeling method to predict large-scale/problem size application runtime using small configuration application profiling efficiently and effectively We propose a resource-based performance modeling technique to predict the scaling runtime and architecture-oriented behaviors of each function. The general idea of our model is to build the performance model from the architecture and system perspective and identify the key and basic performance factors. For compute kernels, these factors are the different types of operations such as executed instructions, loads and stores. We predict these basic operations of each function and combine these predictions into an performance model for the whole application on the target system. For the communication part, the factors are the number and the size of messages. We measure them using the standardized PMPI interface, which is portable across all MPI implementations. And then use LogGP model [14] to predict the communication time. We can build the performance model for different HPC applications with the same modeling methodology and give a comprehensive timing breakdown of the target application as well as the architectural behaviors of each function, without introducing more overhead and potential performance deviation.

2. Employing performance model-based guidance for performance diagnostics Given the fact that users are usually interested in the causes and positions of performance bottlenecks, for example, it is less helpful to tell that a program execution takes square time than pointing to the set of most expensive kernels in the program, and their scaling issues. We use performance modeling technique to predict the application runtime with key performance characters and to discover the performance issue beforehand on three real applications, HOMME [15], CICE [16] and OpenFoam [17]. By using our model, we apply (1) strong scaling diagnostics, (2) week scaling diagnostics, (3) non-scalable compute kernel diagnostics on HOMME and CICE. And we apply (1) strong scaling diagnostics, and (2) load imbalance diagnostics on OpenFoam.

With these efforts, our performance diagnosing method is feasible to provide scaling performance insights and identify the potential performance issues in production-level quality with interpretability and low overhead. We believe our method can be further elaborated to conduct a variety of deeper analyses and possible optimizations for the complex code base.

The primary objective of our method is to identify the potential scaling issues as well as the corresponding code functions. We use a resource-based modeling alongside time approach for the scaling diagnostics. In our work, we split the parallel application into computation and communication two parts and then predict the computation and communication performance separately and the accumulation of these two part is the final runtime prediction.

The computation runtime equals the accumulation of all compute kernels’ runtime. We use hardware counters to profile the computation scaling characteristics to overcome the large overhead of instrumentations [18]. The communication time is the accumulation of point-to-point (p2p) MPI operations and collective MPI operations. We measure MPI operations using the standardized PMPI interface, and then use LogGP model [14] to conduct the communication time prediction. Such method is portable across applications. The framework includes the following four key modules. Figure 1 gives an overview of the key steps necessary to predict runtime and potential performance issues.

1. Performance measurements We generate the hardware counter profiles for computation using the driver of Intel VTune, which records the executions times plus various hardware counters, such as the number of CPU cycles waiting for memory, the number of executed instructions. All metrics are broken down by call path and MPI ranks. Here, we define each element on the call path is a function without including its children. We use precise event-based sampling (PEBS) [19] for hardware counter profiling. The PEBS mechanism is armed by the overflow of the counter with a precise program counter address at granularity of functions. Thus we can obtain the function list after the profiling runs, together with the hardware counter information. However, manual instruction can be added to profiling a lower-level performance, such as loops. We choose functions as kernels because it is clearly to separate communication and computation in parallel applications to enable us to predict them separately. Assume that we use two MPI ranks in one compute node. The hardware counter profiling results of each function are the total amount of the two MPI ranks. We generate communication traces, namely the message size, the message count, the source and destination, by using the standardized PMPI interface. Figure 2 illustrates how we generate the performance measurements on hardware counters and MPI operations.

To summary, the metrics we collected include resource-based metrics and time-based metrics, as Tables 1 and 2 show. Resource-based metrics, such as number of instructions, or the number of messages sent/received, are usually a function of application configuration, and therefore deterministic. We define them as resource-based metrics because they can reflect the resource utilization of a program on the target platform. The resource-based metrics play an important role in our method. Because they can be used to determine the model functions easily, and can be used to predict large-scale performance easily. The model functions are usually a set of pre-defined functions that potentially can be used as a good approximation. Time-based metrics, such as communication time and computation time, are used to determine the coefficients of our model functions. We conduct the performance measurements on several parallelisms. The profiling results of these parallelisms are used to determine the model functions and verify the approximation.

Hardware counter profiling validity can be controlled by its overflow intervals. Our observations show that the default overflow interval, namely 2000 thousands CPU cycles can have 96% confidence intervals on identifying the functions in the program for a 100 ms application run.

3. Resource-based modeling alongside time After performance measurements, a performance model framework is generated according to Table 2 to convolute the communication and computation performance (Eq. 1). In this paper, we assume that applications are implemented following the Bulk Synchronous Parallel (BSP) programming model [20]. Follow the BSP programming model, the total runtime can be calculated by the accumulation of computation runtimes and communication runtimes, which can be found correct in most of today’s real-world HPC applications.

The computation part consists of a number of kernels. The computation time of a kernel can be estimated by two components. The first part is the time taken to execute the computation instructions. The second part is the time associated with fetching the data from storage to compute [21]. There is typically some degree of overlapping between these two components. We introduce a variable named memory blocking factor (\(BF\_mem_i\)), which measures the non-overlapping part for loading data from local memory. The variable i is the index of the detected functions. The accumulation of the p2p and the collective communication times is used to predict the total communication time. A variable named \(BF\_comm\) is measured to evaluate the non-overlapped communication and computation time. Below, we explain how we build the resource-based performance models, and match them with time-based metrics in detail. CPI is regarded as a comprehensive indicator for the performance of kernels, but it alone cannot help us to identify the different performance patterns of kernels since the similar CPIs of kernels can be gained from lower instruction efficiency and better memory traffic, or better instruction efficiency and poorer memory traffic.

The general idea of a resource-based performance model is to account for all key cost factors. For the computation, as Table 1, model terms, such as \(T\_stall_i\), \(T\_L1_i\), \(T\_L2_i\), \(T\_LLC_i\) and \(T\_mainmemory_i\) are derived from different types of hardware counters listed in the third column.

Take \(T\_stall_i\) term as an example, we show how to use hardware counters derived it. In an ideal case, all CPU cycles should be spent to produce useful work. However, there always exists certain part of CPU cycles is spent stalling. At hardware level, the stalling time (\(T\_stall_i\)) represents the waiting time for memory, namely the result of unavailability of processing units or data. Such scenarios often degrade the performance of an application as the number of cores increases [22]. We use the accumulation of two hardware counters, namely \(RESOURCE\_STALLS.LB\) and \(RESOURCE\_STALLS.SB\) to calculate the stall cycles. In this paper, we use five different process configurations for profiling set, and the sixth parallelism is used to verify the regression results. We do the regression for the hardware counters with the best model function in our pre-defined model function set, including one order of polynomial, exponent and logarithm [1, 2]. Because based on massive analysis and experiments, those three kinds of functions and their combinations can cover most of the situation.

Figure 3 is the model results of a kernel called laplace_sphere_wk from HOMME. Figure 3a is the accumulation of \(RESOURCE\_STALLS.LB\) and \(RESOURCE\_STALLS.SB\) (equals to \(T\_stall\)), and Fig. 3b is the number of L1 cache access time. The x-axis is the average compute amount per MPI rank. We do the regression fitting according to the average compute amount because it is easier to predict the performance for another problem size. We will explain it later in the paper. We use one-order power equation, \(a \cdot x ^b +c\), for the \(T\_stall\) and number of L1 access time. The model parameter b is positive for \(T\_stall\) while is negative for the number of L1 access time. It represents that as we assign more MPI ranks in one node, the waiting time for memory (\(T\_stall\)) decreases, and we can fetch more data from L1 cache. Such performance insights can help the users draw a conclusion that the kernel is computation driven in the present cases. According to Table 1, we can model the \(T\_mem\) and \(T\_comp\) as Fig. 3c and d shows. Through the model term \(BF\_mem\), users can obtain the non-overlapped memory time. Therefore, if users want to assign more MPI ranks in the node, they should put their effect on optimizing the memory. Because the total computation time \(T\_comp\) becomes closer to \(BF\_mem \cdot T\_mem\).

For the point-to-point (p2p) communication, we assume a linear relationship because all processes can carry on their operations in parallel. The time cost t of sending a certain number of message m of size s equals to \(t=m\cdot (a\cdot s+b)\). We model the p2p communication time t with total communication size ts as \(a\cdot ts +b\). For the collective communications, the time cost t of a broadcasts a message of size s among all processes P equals to \(t=a\cdot log(P)+b\cdot s +c\). For a sake for simplicity, we do not modeling each individual message sizes, we use an average message size among processes.

To summary, we first model the resource-based metrics separately for each function. We then match the resource-based metric of the functions with time-based metrics using the best one in our pre-defined linear fitting functions. The accumulation time of the compute kernels plus the communication time is the total estimated application runtime. The model inputs are the problem size and number of processes. The outputs are the predicted runtime of each kernel and estimated scaling resource-based metrics.

4. Scaling performance prediction Our model predicts: (1) the execution time \(T\_app\) of a given parallel application on a target scale \(P_t\) by using several applications runs with q processes, where \(q \in {\{2,..P_0\}}\), \(P_0 < P_t\); (2) the execution time \(T\_app\) of a given parallel application on a target scale \(P_k\) with a different problem size \(D_s\). Once the model coefficients are determined, we can use the equations in Table 1 to predict the runtime in Fig. 3.

Our motivation is to identify the kernels from a small problem size D, and build the performance models. We then predict the application runtime for large problem size \(D_s\) without running the application of \(D_s\).

The prediction for another problem size \(D_s\) is conducted according to the average compute amount per MPI rank. Of course, the memory resource contention plays an important role in runtime performance, especially the number of main memory access time. In this paper, we define a threshold \(E=\left\| \frac{N\_LLCmiss\_i_p}{N\_totalmem\_i_p}\right\|\) to evaluate the effectiveness for \(D_s\) runtime prediction from the harm of memory contention. \(N\_LLCmiss\_i_p\) represents the number of last-level cache misses of function i with number of process p, and \(N\_totalmem\_i_p\) is the total number of memory access. Thus by using the profiling results of the five parallelism mentioned above, if \(E \le 1e-4\), it indicts that the performance of last-level cache does not fluctuate much when assigns different compute amount to each MPI rank. Otherwise if \(E > 1e-4\), it indicts that the affect of memory contention has already revealed in the profiling cases. Thus, we have to choose suitable numbers of processes to conduct the profiling runs. We exploit the observation that the big performance fluctuation of last-level cache rarely happens. Besides, it is a common sense in HPC applications that one should choose a suitable number of processes for an application run following certain rule. For example, the number of processes should follow \(6 \cdot n^2\) when we run an atmosphere model with the spectral element dynamical core.

Take HOMME as an example, we predict the runtime of problem size ne256 (grid number: \(256 \cdot 256 \cdot 6\), vertical level: 128) on a target scale 1536 MPI ranks from ne32 (\(32 \cdot 32 \cdot 6\), vertical level: 128). We do not predict the performance per function because our observation shows that the compute amount increment of each function is inconsistent to the \(ratio=\frac{D_s}{D}\), but highly depends on its own inputs. Users with little domain knowledge are hard to estimate the increment without profiling. Therefore, we predict the computation time for the overall computation rather than functions’ time to give a overall runtime for large problem size. The average compute amount of 1536 MPI ranks (problem size ne256) is 16 grids. Thus, we can find how the resource-based metrics behave when we assign 16 grids in one MPI rank by using the models built from ne32. For the communication, we estimate the total message size among processes according to its data decomposition which can be learned from the application’s technical report and its building scripts.

The experiments are carried out on the Intel cluster in National Supercomputing Center in Wuxi of China (NSCC-Wuxi). Each NSCC-Wuxi node contains two Intel Xeon E5-2680v3 processors running at 2.5 GHz with 128 GB memory. The operating system is RedHat 6.6. The MPI version is Intel MPI 15.0, and the network is Mellanox^® FDR InfiniBand. The time of I/O part isn’t taken into consideration in our experiments.

For all the experiments, the profiling overhead of our framework is 3% on average. As we know, the overhead of instrument-based profiling tools, such as Score-P [6], and Scalasca [7], usually exceeds 10%.

Scaling performance prediction of parallel applications has numerous prior work. There are two well-known approaches. One is using a trace-driven simulation to capture detailed performance behavior at a required level, such as MPI-SIM and BigSim [24, 25]. However, it is extremely expensive, not only in terms of time cost of the simulation, but especially in the memory requirement. Zhai et al. [26] extrapolate single-node performance by using representative process to reduce the memory requirements. Wu et al. [27] predict communication performance by extrapolating traces to large-scale application runs. Engelmann et al. [28] develop a simulator permits running an HPC application with millions of threads while observing its performance in a simulated extreme-scale HPC system using architectural models and virtual timing. However, all above works aim to accurately performance prediction but not provide scaling issues for the high-end users beforehand.

Another approach is using analytical performance modeling technique. It is well understood that an analytical performance model has the ability to provide performance insights of a complex parallel program [29, 30]. Hoefler et al. [31] use a simple six-step process to build a performance model for applications with detailed insights. However, it requires tedious code analysis and not being used to predict scalability issues. Calotoiu et al. [1] use performance models to find performance scalability bugs for parallel applications in aspects of communication and floating-point operations. This is probably the most similar work with ours. The main differences are that (1) they aim to report the kernel rankings while they do not separate the computation and communication. However, such results are easily predicted since the time cost of communication usually increases with the growing number of processes. Thus, the computation code sections that do not scale well are still hidden in the complex code while in our work, we separate the communication and computation; and (2) they use communications and floating-point operations as metrics to evaluate the large-scale performance issues, while we provide the possible causes of the potential scaling issues by separating the memory effect from computations. To better understand the fine-grained performance, Bhattacharyya et al. [32] break the whole program into several loop kernels with the assumption that kernels can have simpler performance behaviors. However, this loop-level kernel identification will introduce as many kernels to be instrumented and modeled as there are loops. This can be hundreds even for the NAS parallel benchmarks, and it is not effective to handle the complex loops and functions in real applications. Besides, it lacks the insights for resource consumption of each kernel. Chatzopoulos et al. [22] use the hardware counters to extrapolating the scalability of in-memory applications. There is a consensus that performance modeling technique can be an effective approach for understanding the resource consumption and scalability.

Other approaches focus less on general purpose models but rather on modeling for a specific purpose [33‐35]. Martinasso et al. [36] develop a congestion-aware performance model for PCIe communication to study the impact of PCIe topology. Mondragon et al. use both simulation and modeling technique to profile next-generation interference sources and performance of the HPC benchmarks [12]. Yang et al. [37] performance modeling the applications by running kernels on the target platform and then conduct the prediction cross-platform based on relative performance between the target platforms.

Our work demonstrates that performance modeling technique with hardware counters can be used to help users to understand the potential performance bottlenecks of large-scale runs and large problem size runs. Compared to the laboriously detailed performance models by hand, our performance model is competitive to give an earlier report on the potential performance issues as well as their causes and code positions. Compared to the traditional performance diagnostic method, our method can predict the scaling performance behaviors efficiently and effectively by profiling the application runs on small-scale parallelisms rather than profiling large-scale runs several times. Our model is currently used to conduct the predictions on the same system. The projection across architectures are not considered in the current version of our model. However, the resource-based modeling alongside time makes it easy to build a new model for different architectures. Besides, our resource-based metrics can be used as a preliminary suggestion on how a system is designed to achieve the best performance for a given application.