Using meta-heuristics and machine learning for software optimization of parallel computing systems: a systematic literature review

We have defined the following research questions:

The literature search and selection process are depicted in Fig. 3. Based on the objectives of the study, we have selected an initial set of keywords (see activity 1) that is used to search for articles, such as: parallel computing, machine learning, meta-heuristics and software optimization. To improve the result of the search process, we consider synonyms for the keywords during the search. The search query is executed on digital electronic databases (such as, ACM Digital Library, IEEEXplore, and Google Scholar), conference venues (such as, SC, ISC, ICAC, PPoPP, ICDCS, CGO, ICPP, Euro-Par, and ParCo), and scientific journals (such as, TOCS, JPDC, JOS). The outcome of the search process is a list of potentially relevant scientific publications. Manual selection of these publications by reading the title, abstract, and keywords (activity 4.1) first, then the full paper (activity 4.2) is performed, which results in a filtered list of relevant scientific publications. Furthermore, a recursive procedure of searching for related articles is performed using the corresponding related articles section of each digital library (for example, the ACM Digital Library related papers function powered by IBM Watson, or the Related articles function of Google Scholar).

The initial automatic search on the ACM digital library (see Fig. 3, activity 2.1) returned a list of total 25970 entries (articles). We sorted the entries by relevance, such that the most relevant articles will show up first. As expected, the most relevant articles were found in the first part of the list, and after hundreds of articles, the suggested entries were not relevant to our study. Therefore, we decided to consider only the first 1000 articles. Out of these articles, only 130 were selected for further study based on reading the title and abstract (activity 4.1), and after reading the full article (activity 4.2), 22 were selected as relevant articles. The IEEEXplore returned 40 potentially relevant articles (activity 2.2), 20 of them were selected for further study based on reading the title and abstract (activity 4.1), and 16 were selected as relevant after reading the full paper (activity 4.2). The Google Scholar returned 140 potentially relevant articles (activity 2.3), 31 of them were selected after reading the title and abstract (activity 4.1), and 11 were selected as relevant after reading the full paper (activity 4.2). Searching the conference venues (activity 3.1) and scientific journals (activity 3.2), we selected 28 articles based on reading the title and abstract (activity 4.1), and 8 of them were selected as relevant after reading the full paper (activity 4.2). So, out of more than 1180 articles returned from various sources (activity 2 and 3), 209 were selected manually based on reading the title and abstract (activity 4.1), out of which, after reading the full content (activity 4.2), 57 were selected as relevant to the scope of this paper.

Additionally, the chain sampling technique (also known as snowball sampling, see Fig. 3, activity 5) is used to search for related articles. 39 articles were identified using this technique by reading the title and abstract (activity 4.1), and 8 of them were selected as relevant after reading the full paper (activity 4.2). Chain sampling is a recursive technique that considers existing articles, usually found in the references section of the research publication under study [10]. In total, 65 publications are considered in this review.

The scope of this literature review includes:While other optimization methods (such as, linear programming, dynamic programming, control theory), and other software optimization activities (such as, design-time software optimization) may be of interest, they are left out of scope to keep the systematic review focused.

In accordance with the classification strategy (described in Sect. 3.3) and the defined research questions (described in Sect. 2.1), for each of the selected primary studies we have collected information that we consider important to be recorded in order to perform the literature review.

Table 1 shows an excerpt of the data items (used for quantitative and qualitative analysis) collected for each of the selected studies. Data items 1-3 are used for the quantitative analysis related to RQ1. Data item 4 is used to answer RQ2. Data collected for item 5 is used to answer RQ3, whereas data collected for item 6 is used to answer RQ4. Data item 7 is used to classify the selected scientific publications based on the software life-cycle activities (see Table 3), whereas data item 8 is used for the classification based on the target architecture (see Fig. 6).

A parallel computing system comprises a set of interconnected processing elements and memory modules. Based on the system architecture, generally parallel computers can be categorized into shared and distributed memory. Shared memory parallel computing systems communicate through a global shared memory, whereas in distributed memory systems every processing element has its own local memory and the communication is performed through message passing. While shared memory systems have shown limited scalability, distributed memory systems have demonstrated to be highly scalable. Most of the current parallel computing systems use shared memory within a node, and distributed memory between nodes [6].

According to Top500 [94] in the 90s the commonly used parallel computing systems were symmetric multi-processing (SMP) systems and massive parallel processing (MPP) systems. SMPs are shared memory systems where two or more identical processing units share other system resources (main memory, I/O devices) and are controlled by a single operating system. MPPs are distributed memory systems where a larger number of processing units (or separate computers) are housed in the same place. The disparate processing units share no system resources, they have their own operating system, and communicate through high-speed network. The main computing models within the distributed parallel computing systems include cluster [26, 89], grid [13, 32, 82, 86], and cloud computing [33, 59, 82].

Nowadays, the mainstream platforms for parallel computing, at their node level consist of multi-core and many-core processors. Multi-core processors may have multiple cores (two, four, eight, twelve, sixteen...) and are expected to have even more cores in the future. Many-core systems consist of larger number of cores. The individual cores of the many-core systems are specialized to efficiently perform operations such as, SIMD, SIMT, speculations, and out-of-order execution. These cores are more energy efficient because they usually run at lower frequency.

Systems that comprise multiple identical cores or processors are known as homogeneous systems, whereas heterogeneous systems comprise non-identical cores or processors. As of November 2017, the TOP500 list [94] contains several supercomputers that comprise multiple heterogeneous nodes. For example, a node of Tianhe-2 (2nd most powerful supercomputer) comprises Intel Ivy-Bridge multi-core CPUs and Intel Xeon Phi many-core accelerators; Piz Daint (3rd) consists of Intel Xeon E5 multi-core CPUs and NVIDIA Tesla P100 many-core GPUs [66, 96].

Programming parallel computing systems, especially heterogeneous ones, is significantly more complex than programming sequential processors [78]. Programmers are exposed to various parallel programming languages (often implemented as extensions of general-purpose programming languages such as C and C++), including, OpenMP [72], MPI [42], OpenCL [90], NVIDIA CUDA [70], OpenACC [100] or Intel TBB [97]. Additionally, the programmer is exposed to different architectures with different characteristics (such as the number of CPU/GPU devices, the number of cores, core speed, run-time system, memory and memory levels, cache size). Finding the optimal system configuration that results in the highest performance is challenging. In addition to the programmability challenge, heterogeneous parallel computing systems bring the portability challenge, which means that programs developed for a processor architecture (for instance, Intel Xeon Phi) may not function on another processor architecture (such as, GPU). Manual software porting and performance tuning for various architectures may be prohibitive.

Existing approaches, discussed in this study, propose several solutions that use machine learning or meta-heuristics during compile-time and run-time to alleviate the programmability and performance portability challenges of parallel computing systems.

In computer science selecting the best solution considering different criteria from a set of various available alternatives is a frequent need. Based on what type of values the model variables can take, the optimization problems can be broadly classified in continuous and discrete. Continuous optimization problems are concerned with the case where the model variables can take any value permitted by some given constraints. Continuous optimization problems are easier to solve. Given a point x, using continuous optimization techniques one can infer information about neighboring points of x [39].

In contrast, in discrete optimization (also known as combinatorial optimization) methods the model variables belong to a discrete set (typically subset of integers) of values. Discrete optimization deals with problems where we have to choose an optimal solution from a finite number of possibilities. Discrete optimization problems are usually hard to solve and only enumeration of all possible solutions is guaranteed to give the correct result. However, enumerating across all available solutions in a large search space is prohibitively demanding.

Heuristic-guided approaches are designed to solve optimization problems more quickly by finding approximate solutions when other methods are too slow or fail to find any exact solution. These approaches select near-optimal solutions within a time frame (that is, they trade-off optimality for speed). While heuristics are designed to solve a particular problem (problem-dependent), meta-heuristics can be applied to a broad range of problems. They can be thought as higher-level heuristics that are designed to determine a near-optimal solution to an optimization problem, with limited computation capacity and knowledge about the problem.

In what follows, we first describe the meta-heuristics and list commonly used algorithms, and thereafter, we describe machine learning in the context of software optimization.

Software optimization can happen during different activities of the software life-cycle. We categorize the software optimization activities by the time of their occurrence: Design and Implementation-time, Compile-time, Run-time (Fig. 5).

While software design and implementation activities are performed by the programmer, software activities at compile-time and run-time are completed by tools (such as compilers and run-time systems). Therefore, in this paper we focus on tool-supported software optimization approaches that use approximate techniques (machine learning and meta-heuristics) at compile-time and run-time.

For each of the software optimization life-cycle activities, including Compile-Time (Sect. 4) and Run-Time (Sect. 5), we will describe the context for software optimization goals, discuss the state-of-the-art research, and discuss limitations and future research directions.

In this section we classify the considered scientific publications based on the architecture, software optimization approach, and life cycle activities.

To provide an overview of the current state of the art, we have grouped the scientific publications that use machine learning and meta-heuristics for software optimization of parallel computing systems in the following time periods: 2000–2005, 2006–2011, and 2012–2017. Each of the periods, correspond to the type of the processors that were used the most in the TOP list during that time. For example, even though the first multi-core processor was introduced in 2001 [37], most of the super computers in TOP500 list during years 2000–2005 comprised multiple single-core processors [94]. Further filtering and classification of the considered scientific publications, and visualization of the results in the form of a time-line can be performed using our on-line interactive tool (see Fig. 1).

Architecture Figure 6 shows a classification of the reviewed papers based on the target architecture, including multi-node, single-node, grid, and cloud parallel computing systems. The horizontal axis on the top delineates the common types of processors used during the corresponding time period. For instance, from 2000 to 2005 grids and clusters employed single or multiple sequential processors at node level, whereas during the period from 2006 to 2011 nodes employed multi-core processors. Accelerators combined with multi-core processors can be seen during time period 2012–2017. We may observe that most of the work is focused on optimization of resource utilization at the node level (single-node). Optimization of the resources of multi-node computing systems (including clusters) is addressed by several research studies continuously during the considered periods of time. The optimization of grid computing systems using machine learning and meta-heuristic approaches has received less attention, whereas optimization of cloud computing systems has received attention during the period 2012–2017.

Software optimization approach In Table 2 we classify the selected publications that use intelligent techniques (such as, machine learning and meta-heuristics) for software optimization at compile-time and run-time. We may observe that machine learning is used more often for software optimization during compile-time and run-time compared to meta-heuristics.

Life-cycle activity A classification of the reviewed papers based on the software life-cycle activities (including, code optimization, code generation, scheduling, and adaptation) is depicted in Table 3. We may observe that the scheduling life-cycle activity has received the most attention, especially during 2012–2017 period. The use of machine learning and meta-heuristics for code optimization during compile-time has been addressed by many researchers, especially during the period between 2006 and 2011. Similar trend can be observed for research studies that focus on using intelligent approaches to optimize code generation. Optimization of software through adaptation is addressed during the year of 2006–2011.

Code optimization will not change the program behavior but will optimize the code to reach optimization goals (reducing the execution time, energy consumption, or required resources).

Compiler optimization techniques include loop unrolling, splitting and collapsing, instruction scheduling, software pipelining, auto-vectorization, hyper-block formation, register allocation, and data pre-fetching [88]. Different device-specific code optimization techniques may behave differently in various architectures. Furthermore, choosing more than one optimization technique does not necessarily result in better performance, sometimes combination of different techniques may have negative impact on the final output. Hence, manually writing hard-code heuristics is impractical, and techniques that intelligently select the compiler transformations that result in higher application benefits in a given context are required.

Within the scope of this survey, scientific publications that use machine learning for code optimization at compile time include [1, 17, 34, 35, 57, 69, 87, 95, 99], whereas scientific publications that use meta-heuristics for code optimization include [21, 88, 92, 93]. Table 4 lists the characteristics of the selected primary studies that address code optimization at compile time. Such characteristics include: the algorithm used for optimization, the optimization objectives, the considered features that describe the application being optimized, and type of optimization (on-line or off-line). We may observe that besides the approach proposed by Tiwari and Hollingsworth [92], the rest of them focus on off-line optimization approaches and they are based on historical data (knowledge) that is gathered from previous runs.

As we mentioned earlier, different optimizations can be performed during compilation. We may see that some researchers focus on using intelligent techniques to identify loops that would potentially execute more efficiently when unrolled [69], or selecting the loop unroll factor that yields the best performance [87]. Instruction scheduling [17], partitioning strategy for irregular [57] and streaming [99] applications, determining the list of compiler optimizations that results in the best performance [21, 35, 92] are also addressed by the selected scientific publications. Furthermore, Tournavitis et al. [95] use SVMs to determine whether parallelization of the code would be beneficial, and which scheduling policy to select for the parallelized code.

With regards to the machine learning algorithms used for code optimization, Nearest Neighbor (NN) classifier [1, 57, 87, 99], Support Vector Machine (SVM) [87, 95], and Decision Tree (DT) [69] are the most popular. Other algorithms, such as Ruled Set Induction (RSI) [17], and Predictive Search Distribution (PSD) [34, 35] are also used for code optimization during compilation. Whereas, approaches that are based on search-based algorithms use Genetic Algorithm (GA), Hill Climbing (HC), Greedy Algorithm (GrA), and Parallel Rank Ordering (PRO) for code optimization during compile-time [21, 92, 93].

To achieve the aforementioned objectives, a representative set of program features is extracted through static code analysis, which are considered to be the most informative with regards to the program behavior. The selection of such features is closely related to the optimization goals. For example, to identify loops that benefit from unrolling, Monsifrot et al. [69] use loop characteristics such as, number of memory accesses, arithmetic operations, code statements, control statements, and loop iterations. Such loop characteristics are also used to determine the loop unroll factor [87]. Characteristics related to a specific code block (such as number of instructions, branches, calls, stores) are used when deciding whether applications benefit from instruction scheduling [17]. Determining the partitioning strategy of irregular applications is based on static program features related to basic block, loop characteristics, and the data dependency [57]. Features such as pipeline depth, load/store operations per instruction, number of computations, and computation-communication ratio are used when determining partitioning strategy of streaming applications [99]. Tiwari and Hollingsworth [92] consider architectural specifications such as cache and register capacity, in addition to the application specific parameters, such as tile size in a matrix multiplication algorithm.

The process of transforming code from one representation into another one is called code generation. We call “machine code generation” the code transformation from the high level to low level representation (that is ready for execution), whereas “source code generation” indicates in this paper the source-to-source code transformation.

In the context of parallel computing, a source-to-source compiler is an automatic parallelization compiler that can automatically annotate a sequential code with parallel code annotations (such as, OpenMP pragma directives or MPI code statements). Source-to-source compilers may alleviate the portability issue, by enabling to automatically translate the code into an equivalent representation of the code that is ready to be compiled and executed on target architectures.

In this section, we focus on source code generation techniques that can:During the process of porting applications, programmers are faced with the following problems: (1) demand of device-specific knowledge and API; (2) difficulties to predict whether the application will have performance benefits before it is ported; (3) there exist a large number of programming languages and models that are device (types and manufacturer) specific.

To address such issues, researchers have proposed different solutions. In Table 5, we list the characteristics of these solutions such as, optimization algorithm, optimization objectives, and considered features during optimization.

The optimization objectives are derived from the aforementioned portability challenges. For example, to alleviate the demand for device-specific knowledge, Beach and Avis [7] aim to identify candidate kernels that would likely benefit from parallelization, generate device-specific code from high-level code, and map to the accelerating device that yields the best performance. Similarly, Fonseca and Cabral [31] propose the automatic generation of OpenCL code from Java code. Ansel et al. [5] propose the PetaBricks framework that enables writing multiple versions of algorithms, which are automatically translated into C++ code. The runtime can switch between the available algorithms during program execution. Luk et al. [58] introduce Qilin that enables source-to-source transformation from C++ to TBB and CUDA. It uses machine learning to find the optimal work distribution between the CPU and GPU on a heterogeneous system.

Decision Trees (DT) [7, 31], k-Nearest Neighbor (kNN) [19], Cost Sensitive Decision Table (CSDT), Naive Bayes (NB), Support Vector Machine (SVM), Multi-layer Perceptron (MPL) [31], Linear Regression (LR) [31, 58], and Logistic Regression (LRPR) [77] machine learning algorithms are used during the code-generation.

Beach and Avis [7] considered static loop characteristics to achieve their objectives, whereas Chen and Long [19] use both static and dynamic program features to generate the multi-threaded versions of a selected loop, and then select the most suitable loop version at run-time. Combination of static code features (extracted at compile time), and dynamic features (extracted at run-time) are also used to determine the most suitable processing device for a specific application [31]. To determine the best workload distribution of a parallel application, Luk et al. [58] consider algorithm parameters and hardware configuration parameters. Pekhimenko and Brown [77] consider general and loop-based features to determine the list of program method transformation during code generation that would reduce the compilation time.

In this section, we first discuss the advantages of meta-heuristics and machine learning methods for software optimization at compile-time, followed by a discussion about their limitations. Thereafter, we discuss the future directions.

In Table 6, we list each of the machine learning and meta-heuristic methods used for compile-time software optimization. For each of the used methods, we provide the advantages, such as performance improvement, speedup, and prediction accuracy.

While most of the approaches discussed in this review present significant performance improvement, which is important towards having intelligent compilers that require less engineering effort to provide satisfactory code execution performance, indications that there is still room for improvement can be observed in Stephenson et al. [88] and andWang and O’boyle [99].

Limitations of the compile-time software optimization approaches that use machine learning or meta-heuristics are listed in Table 7, which include: (1) limitation to a specific programming language or model [95], (2) forcing developers to use extra annotations on their code [58], or use not widely known parallel programming languages [5], (3) focusing on single or simpler aspects of optimizations techniques (ex: loop unrolling, unrolling factor, instruction scheduling) [17, 69, 87], whereas more complex compiler optimizations (that are compute-intensive) are not addressed sufficiently.

Furthermore, optimizations based on features derived from static code analysis provide poor global characterization of the dynamic behavior of the applications, whereas using dynamic features requires application profiling, which adds additional execution overhead to the program under study. This additional time can be considered negligible for applications that are executed multiple times after the optimization, however it represents overhead for single-run applications. Approaches that generate many multi-threaded versions of the code [19] might end up with dramatic code increases that make difficult the applicability to embedded parallel computing systems with limited resources. Adaptive compilation techniques [21] add non-negligible compilation overhead.

Future research should address the identified shortcomings in this systematic review by providing intelligent compiler solutions for general-purpose languages (such as, C/C++) and compilers (for instance, GNU Compiler Collection) that are widely used and supported by the community. Many compiler optimization issues are complex and require human resources that are usually not available within a single research group or project.

According to the Cambridge Dictionary,¹ scheduling is “the job or activity of planning the times at which particular tasks will be done or events will happen”. In context of this paper, we use the term scheduling to indicate mapping the tasks onto the processing elements and determining the order of task execution to minimize the overall execution time.

Scheduling may strongly influence the performance of parallel computing systems. Improper scheduling can lead to load imbalance and consequently to sub-optimal performance. Researchers have proposed different approaches that use meta-heuristics or machine learning to find the best scheduling within a reasonable time.

Based on whether the scheduling algorithms can modify the scheduling policy during program execution, generally scheduling algorithms are classified in static and dynamic.

According to the Cambridge Dictionary,² adaptation is “the process of changing to suit different conditions”. In this paper, we use the term adaptation to refer to the property of systems that are capable of evaluating and changing their behavior to achieve specified goals with respect to performance, energy efficiency, or fault tolerance. In dynamic environments, modern parallel computing systems may change their behavior by: (1) changing the number of used processing elements to optimize system resource allocation; (2) changing the algorithm or implementation variant that yields to better results with respect to the specified goals; (3) reducing the quality (accuracy) of the output to meet the performance goals; or (4) changing the clock frequency to reduce energy consumption.

The studied literature in this paper provide examples that adaptation (also referred to as self-adaptation) proved to be an effective approach to deal with the complexity, variability, and dynamism of modern parallel computing systems. Table 10 lists the characteristics (such as, adaptation method and objectives, monitored and tuned parameters) of the scientific publications that use adaptation for software optimization of parallel computing systems.

With regards to the adaptation objectives, Thomas et al. [91] use a custom adaptation loop to adaptively select the most suitable algorithm implementation for a given input data set and system configuration. Hoffmann et al. [46, 47] use an observe-decide-act (ODA) feedback loop to adaptively apply user defined actions to change the program behavior in favor of achieving some user-defined goals, such as energy efficiency and throughput. Adaptation methods are used in the smart-locks library [28], which can change its behavior at run-time to achieve certain goals. Similarly, in [29] adaptation methods are used for optimizing data structure knobs. Adaptive mapping of computations to the processing units is proposed by Luk et al. [58]. The Antarex [84] project aims at providing means for application tuning and adaptation for energy efficient heterogeneous high-performance computing systems, by providing a domain specific language that allows specifying adaptation goals at compile-time.

During the process of adaptation, all of the approaches proposed in the considered scientific publications, have at least three components of an adaptation loop, including monitoring, deciding, and acting. For example, Thomas et al. [91] monitor architecture and environment parameters, then uses a decision tree to analyze such information, and perform the required changes (in this case selecting an algorithm implementation). Similarly, Hoffmann et al. [47] use the so called observe-decide-act (ODA) feedback loop to monitor performance related information (retrieved using the application heartbeats [46]) and use the heart-rate to take some user defined actions, such as adjusting the clock speed, allocating cores, or change the algorithm. Reinforcement learning (RL), an on-line machine learning algorithm, is used to help with the adaptation decisions in both smart-locks [28] and smart data-structures [29], whereas linear regression (LR) is used by Luk et al. [58] for choosing the mapping scheme of computations to processing elements.

In Table 10 we list two types of parameters, the monitored parameters, used to evaluate whether adaptation goals have been met, and tuned parameters, which are basically defined actions that will change the program behavior until the desired goals are achieved. For monitoring, architecture and environment variables (such as, available memory, cache size, number of processors), and performance characteristics are considered by Thomas et al. [91]. Performance related information retrieved from the heartbeats monitor are used as monitoring parameters in the following scientific articles [28, 29, 47]. Luk et al. [58] rely on the execution time of parts of the program, whereas the Antarex framework uses contextual information, requirements, and resource availability for monitoring the program behavior. As tuning parameters, the following are considered, selecting the algorithm implementation [47, 91], adjusting the clock speed, core allocation, select algorithm [47], change lock scheduling policy [28], adjust the scancount [29], change mapping scheme [58], and altering resource allocation and task mapping [84].

In this section, we first discuss the advantages of meta-heuristics and machine learning methods for software optimization at run-time, followed by a discussion about their limitations. Thereafter, we discuss the future directions.

In Table 11, we list each of the machine learning and meta-heuristic methods used for run-time software optimization. For each of the used methods, we provide the advantages, including performance improvement, speedup, or prediction accuracy.

At run-time, many execution environment parameters influence the performance behavior of the program. Exploring this huge parameter space is time consuming and impractical for programs with long execution times and large demand for system resources. Different computing capabilities and energy efficiency of processing elements of heterogeneous parallel computing systems make the scheduling a difficult challenge. Table 12 lists the limitations of the run-time software optimization approaches for parallel computing systems considered in this paper.

We may observe that some of the existing scheduling techniques often assume that the program is executed on a dedicated system and all system resources are available for use. The approach proposed by Grewe et al. [41] propose a co-scheduling technique, which considers that the resources are shared with other applications. However, the adaptation occurs only when the application is executed, but not during program execution. We believe that better results could be achieved if they consider to adapt to changes while the application is being executed. Another issue is that commonly used scheduling techniques ignore slow processing elements due to their low performance capabilities. Mapping computations always to processing units that offer higher performance capability is not optimal, because slower processing elements may never get work to perform. Furthermore, most of the reviewed approaches target specific features of the code only (for example, loops), or are limited to specific programming models and applications (data-bound or compute-bound). Many static scheduling approaches require retraining of the prediction model for each new architecture, limiting their general use because training requires a significant amount of data that is not always available. Approaches that reduce the amount of training data require implementation of multiple machine learning algorithms (for instance, [71]). Approaches that use a single execution [21, 56] by trying various system configurations during the program execution are promising, however the introduced overhead is not negligible. Self-adaptation techniques require the developer to add additional information into the code so that the software would be able to monitor the system and take decisions. Even though such code is not difficult to add for the application programmer, the software development becomes more complex while talking decisions based on these results. Furthermore, such approaches introduce overhead at runtime, because they need to run for a certain amount of time until enough data is collected for the framework to be able to take the most optimal decisions.

Future research should aim at reducing the scheduling and adaption overhead for dynamic approaches. Run-time optimization techniques for heterogeneous systems should be developed that utilize all available computing resources to achieve the optimization goals. There is a need for robust run-time optimization frameworks that are useful for a large spectrum of programs and system architectures. Furthermore, techniques that reduce the amount of data generated from system monitoring are needed in particular for extreme-scale systems.