A systematic comparison of metamodeling techniques for simulation optimization in Decision Support Systems

doi:10.1016/j.asoc.2009.11.034

Applied Soft Computing

Volume 10, Issue 4, September 2010, Pages 1257-1273

https://doi.org/10.1016/j.asoc.2009.11.034 Get rights and content

Abstract

Simulation is a widely applied tool to study and evaluate complex systems. Due to the stochastic and complex nature of real world systems, simulation models for these systems are often difficult to build and time consuming to run. Metamodels are mathematical approximations of simulation models, and have been frequently used to reduce the computational burden associated with running such simulation models. In this paper, we propose to incorporate metamodels into Decision Support Systems to improve its efficiency and enable larger and more complex models to be effectively analyzed with Decision Support Systems. To evaluate the different metamodel types, a systematic comparison is first conducted to analyze the strengths and weaknesses of five popular metamodeling techniques (Artificial Neural Network, Radial Basis Function, Support Vector Regression, Kriging, and Multivariate Adaptive Regression Splines) for stochastic simulation problems. The results show that Support Vector Regression achieves the best performance in terms of accuracy and robustness. We further propose a general optimization framework GA-META, which integrates metamodels into the Genetic Algorithm, to improve the efficiency and reliability of the decision making process. This approach is illustrated with a job shop design problem. The results indicate that GA-Support Vector Regression achieves the best solution among the metamodels.

Introduction

Due to the growing complexities and uncertainties in decision making situations, the model-driven Decision Support Systems (DSS) has become increasingly important to decision makers [1]. The model-driven DSS can assist decision makers in applying quantitative models to support the decision-making process. The simulation model is an important type of quantitative model used in model-driven DSS. A simulation model can imitate the behavior of an actual or anticipated human or physical system. It can capture much more detail about a specific system than algebraic models. It can capture underlying mechanism and dynamics of a system, which enables decision makers to effectively manage daily operations and make long term plans. It provides also a test-bed to assess changes in operations and managerial policies [2].

In practice however, in addition to assessing changes in policies, decision makers are often tasked with finding the best (optimal) decision/policy [3]. This requires combining simulation models with optimization techniques to find the best values of decision variables (inputs) which yield the optimal outputs of the system. As shown in Fig. 1, a typical simulation driven DSS is an integrated system of three components: system controller, models (simulation model and optimizer) and user interface [2]. The system controller handles all system processes including receiving information from users, returning results back to users, executing the simulation, and optimizing the simulation model. The simulation model approximates the behaviors of the real system. The optimizer determines the best set of decision variables for the simulation model, and the user interface manages the interactions between the users and the DSS.

Due to the stochastic and complex nature of most real world systems, simulation models of these systems are themselves difficult to build and time consuming to execute. In many cases, decision makers cannot afford to explore a large area of the decision variable space or to conduct a lengthy search for the best set of decision variables. One feasible alternative is to build a metamodel, which is a ‘model of the model’ [4], and use the metamodel to search for the optimal decision variable set. Since metamodels can run much faster than the underlying (computationally costly) simulation models, it is usually used as the objective function approximator of the simulation optimization problem [5], [6].

To our current knowledge, metamodeling has not been formally integrated within DSS. In Fig. 2, we illustrate with a systems diagram how to incorporate the simulation metamodel into a model-driven DSS. In this DSS, the system controller first determines whether the computational resources available allow for direct optimization of the simulation model. If it is feasible, the system will enter the loop outlined by the solid arrows to directly optimize the simulation model. On the other hand, if the computational resources are insufficient for direct optimization, the system will enter the loop outlined by the dashed arrows to utilize metamodels for optimization. Within this loop, a smaller number of simulation runs is first conducted, and a metamodel is constructed to approximate the simulation model. The optimization is then carried out on the metamodel, and finally, the optimal solutions are returned back to the simulation model for verification and sensitivity analysis. This can be an iterative process, slowly building up the accuracy of the metamodel. With the use of metamodels, this procedure is expected to be much faster than the direct search on the simulation model itself.

Various modeling forms have been introduced for metamodeling, such as Multivariate Adaptive Regression Splines (MARS), Kriging (KG), Radial Basis Function (RBF), Artificial Neural Networks (ANN), and Support Vector Regression (SVR). An important issue in metamodeling is the selection of an appropriate metamodel type under a given condition. In this paper, a systematic comparison of various types of metamodels is first conducted to analyze their strengths and weaknesses. The indicators of comparisons include quantitative measures such as accuracy and robustness, and qualitative measures such as interpretability and availability. A test bed of 16 stochastic simulation problems representing different stochastic behaviors and problem types is used for the comparison. Latin Hypercube designs are applied to generate the training points to construct the metamodels.

Following, we propose an intelligent simulation optimization framework for DSS, which integrates metamodels in an evolutionary optimization scheme. We call this GA-META, where GA is the Genetic Algorithm—an efficient and robust evolutionary optimization algorithm, and META is the metamodel which can take the form of MARS, KG, ANN, RBF, SVR or any other type of metamodel. This approach is demonstrated on a DSS for a job shop design problem.

The rest of this paper is organized as follows: Section 2 presents a short introduction to simulation optimization and metamodeling. Section 3 provides an overview of the five metamodeling techniques in our study. In Section 4, the metamodeling performances are validated and compared. In Section 5, the GA-META framework for simulation optimization DSS is proposed and demonstrated on a job shop design problem. Finally, Section 6 presents the conclusions of our study and a discussion on possible avenues for future work.

Section snippets

Simulation optimization and metamodeling

Simulation optimization is often embedded into DSS to assist in the decision making process. The mathematical form of the stochastic simulation optimization problem (maximization) is given as follows: $\max Y = f (X) + ε_{s} (X)$ subject to $g_{i} (X) \leq 0, i = 1,..., p$ where X = (x₁, x₂, …, x_k) denotes the input variables of the simulation model which is also regarded as the decision variables of the DSS (k is the total number of input variables), Y denotes the outputs of the simulation model, f(X) is the underlying

Multivariate adaptive regression splines (MARS)

Multivariate adaptive regression splines [11] is a nonparametric regression procedure that makes no assumption about the underlying relationship between the input variables and the output variables. Instead, MARS approximates this relationship by a divide-and-conquer strategy which involves partitioning the training data into separate regions and generating regression equations to fit each region. The mathematical form of MARS can be written as: $\hat{y} = \hat{f} (X) = a_{0} + \sum_{m = 1}^{M} a_{m} B_{m} (X),$ where a₀ is a constant, M

Test on metamodeling performances

Four well known deterministic mathematical simulation problems are selected as the basis for constructing the test bed of problems for comparison. They are: (i) 4 dimensional welded beam design [33], (ii) 3 dimensional helical compression spring [34], (iii) 2 dimensional sinusoidal function [35], and (iv) 8 dimensional asymmetric function [36]. These functions are known to be complex with varying degrees of non-linearity and dimensionality.

Since error type (homogenous/heterogeneous) and error

Metamodels in Decision Support System optimization

In this section, the metamodeling techniques are integrated into the DSS. The job shop simulation problem is used as an illustrative example of the proposed methodology.

Conclusions and future works

In our study, the metamodeling techniques are embedded into the Decision Support System to assist the optimization procedure. Our method is particularly useful when the simulation model in the DSS is expensive to run. A systematic study is first conducted to analyze the strengths and weaknesses of five metamodeling techniques (ANN, RBF, SVR, Kriging, and MARS) for stochastic simulation problems.

The indicators of the metamodel performances include quantitative measures like prediction accuracy,

References (56)

D.J. Power et al.
Model-driven decision support systems: Concepts and research directions
Decision Support Systems
(2007)
J.P. Shim et al.
Past, present, and future of decision support technology
Decision Support Systems
(2002)
X.T. Wan et al.
Simulation-based optimization with surrogate models—application to supply chain management
Computers and Chemical Engineering
(2005)
J.P.C. Kleijnen et al.
Robustness of Kriging when interpolating in random simulation with heterogeneous variances: Some experiments
European Journal of Operational Research
(2005)
G. Meghabghab et al.
Stochastic simulations of Web search engines: RBF versus second-order regression models
Information Sciences
(2004)
D.J. Fonseca et al.
Artificial neural networks for job shop simulation
Advanced Engineering Informatics
(2002)
D.J. Fonseca et al.
Simulation metamodeling through artificial neural networks
Engineering Applications of Artificial Intelligence
(2003)
J.P.C. Kleijnen
Kriging metamodeling in simulation: a review
European Journal of Operational Research
(2009)
G. Meghabghab
Iterative radial basis functions neural networks as metamodels of stochastic simulations of the quality of search engines in the World Wide Web
Information Processing & Management
(2001)
M.F. Hussain et al.
Metamodeling: radial basis functions, versus polynomials
European Journal of Operational Research
(2002)

A.C. Keys et al.

A sequential-design metamodeling strategy for simulation optimization

Computers and Operations Research

(2004)

R.A. Kilmer et al.

Computing confidence intervals for stochastic simulation using neural network metamodels

Computers and Industrial Engineering

(1999)

Y.F. Li et al.

A study of project selection and feature weighting for analogy based software cost estimation

Journal of Systems and Software

(2009)

A.C. Lorena et al.

Evolutionary tuning of SVM parameter values in multiclass problems

Neurocomputing

(2008)

C. Gold et al.

Bayesian approach to feature selection and parameter tuning for support vector machine classifiers

Neural Networks

(2005)

F.M. Alam et al.

A comparison of experimental designs in the development of a neural network simulation metamodel

Simulation Modelling Practice and Theory

(2004)

A.J. Smola et al.

The connection between regularization operators and Support Vector kernels

Neural Networks

(1998)

A.G. Greenwood et al.

Simulation optimization decision support system for ship panel shop operations

J.P.C. Kleijnen

Statistical Tools for Simulation Practitioners

(1987)

H. Nakayama et al.

Simulation-based optimization using computational intelligence

Optimization and Engineering

(2002)

G.G. Wang et al.

Review of metamodeling techniques in support of engineering design optimization

Transactions of the ASME, Journal of Mechanical Design

(2007)

T.W. Simpson et al.

Comparison of response surface and kriging models for multidisciplinary design optimization

R. Jin et al.

Comparative studies of metamodeling techniques under multiple modelling criteria

Structural and Multidisciplinary Optimization

(2001)

H. Fanga et al.

A comparative study of metamodeling methods for multiobjective crashworthiness optimization

Computers and Structures

(2005)

J.H. Friedman

Multivariate adaptive regression splines

Annals Statistics

(1991)

W.C.M. van Beers et al.

Kriging for interpolation in random simulation

Journal of the Operational Research Society

(2003)

J.P.C. Kleijnen et al.

Application-driven sequential designs for simulation experiments: Kriging metamodeling

Journal of the Operational Research Society

(2004)

M. Shin et al.

Gaussian radial basis functions for simulation metamodeling

Proceedings of the 2002 Winter Simulation Conference

(2002)

Cited by (163)

An uncertainty-based objective function for hyperparameter optimization in Gaussian processes applied to expensive black-box problems
2024, Applied Soft Computing
As of today, Gaussian processes (GPs) have been widely and successfully used in the context of design optimization based on expensive-to-evaluate functions. This supervised learning method enables a generation of accurate nonlinear surrogate models based on relatively small datasets. Nonetheless, their most valuable asset is to provide uncertainty in predictions. Despite their excellent stochastic properties, Gaussian processes are unfortunately not immune to threats such as the generation of distorted predictions, especially when the amount of data available is very limited. This shortcoming is caused by a poor choice of the GP hyperparameters and represents a serious threat to the efficiency and effectiveness of the whole surrogate-based optimization. In this paper we present the Hybrid Loss (HL), a novel uncertainty-aware objective function for the hyperparameter tuning of Gaussian processes. This method is intended to exploit information coming from the predictive variance to remedy the typical shortcomings of the log marginal likelihood, i.e. the objective function commonly used to optimize GP hyperparameters. By pairing this methodology with a well-known adaptive sampling strategy, we investigate the performance on a wide range of benchmark functions and a real engineering problem. The observed evidence clearly shows how uncertainty can be successfully exploited to make a wiser choice of the hyperparameters. This translates into more accurate predictions, surrogate models less prone to overfitting, and above all, greatly improved convergence rates.
Metamodel-assisted meta-heuristic design optimization of reinforced concrete frame structures considering soil-structure interaction
2023, Engineering Structures
It is well known that conventional heuristic optimization is the most common approach to deal with structural optimization problems. However, metamodel-assisted optimization has become a valuable strategy for decreasing computational consumption. This paper applies conventional heuristic and Kriging-based meta-heuristic optimization to minimize the CO₂ emissions of spatial reinforced concrete frame structures, considering an aspect usually ignored during modeling, such as the soil-structure interaction (SSI). Due to the particularities of the formulated problem, there are better strategies than simple Kriging-based optimization to solve it. Thus, a meta-heuristic strategy is proposed using a Kriging-based two-phase methodology and a local search algorithm. Three different models of structures are used in the study. Results show that including the SSI leads to different design results than those obtained using classical supports. The foundations, usually ignored in this type of research, also prove significant within the structural assembly. Additionally, using an appropriate coefficient of penalization, the meta-heuristic approach can find (on average) results up to 98.24% accuracy for cohesive soils and 98.10% for frictional ones compared with the results of the heuristic optimization, achieving computational savings of about 90%. Therefore, considering aspects such as the SSI, the proposed metamodeling strategy allows for dealing with high-complexity structural optimization problems.
Energy, economic and comfort optimization of building retrofits considering climate change: A simulation-based NSGA-III approach
2023, Energy and Buildings
Improving building energy performance is central to promote sustainability and mitigate climate change, but it has proved to be a challenging process requiring investigation of a large domain of energy retrofit measures and contrasting objectives. Besides, the unique characteristic of Iran, namely heavily subsidized energy prices and tiered utility rates, further complicate the problem. This research introduces a novel simulation-based multi-objective optimization approach based on a modification of NSGA-III algorithm, denoted as prNSGA-III, powered by parallel computing design and result-saving archive that offers an increased computational potential. The method is developed by integrating EnergyPlus as energy simulation engine with optimization algorithm coded in MATLAB programming language. A large domain of active, passive, water conservation, and renewable retrofit measures are explored to maximize environmental performance while minimizing thermal discomfort and life cycle cost. The method is applied to a residential building in Tehran, Iran, subjected to different scenarios with reference to future climate change and energy price variation. The results show that implementing suggested energy retrofit measures can yield substantial improvement in environmental performance and curb CO-eq emission by up to 73%, and also reduce thermal discomfort by up to 46%. However, energy-efficient solutions can only be achieved at the expense of increased life cycle cost. Moreover, the result demonstrates that climate change can affect optimal solutions, particularly heating and cooling systems, due to changes in external thermal loads.
Computational fluid dynamics modelling approaches of gas explosion in the chemical process industry: A review
2023, Process Safety and Environmental Protection
Previous studies have revealed that major accidents in Chemical Process Industry (CPI) are most commonly due to explosions. Thus, analytical studies of explosion-related risk assessments are performed to predict the consequences of potential explosions. As physical experiments on explosions are very expensive, modelling and simulation techniques using theoretical models are becoming increasingly popular, allowing researchers to replicate the potential explosion scenarios. In this regard, computational fluid dynamics (CFD) models are more than appropriate. Although CFD simulations are widely applied, they have several weaknesses such as high computational costs as well as potential simulation inaccuracies due to inaccurate modelling steps. The weaknesses can be overcome with appropriate techniques such as model simplification, defining the appropriate method, grid design and boundary conditions. Many studies have reported different aspects and perspectives of explosion modelling and simulation techniques, but few evaluate the techniques across every different type of explosion. This subject is critical, as modelling steps and techniques directly affect the accuracy of simulation results. Hence, a review of the assumptions and simulation techniques that are used to reduce the computational costs associated with gas explosion modelling for each of the different explosion types is presented.
Integrating Multiple Perspectives in Manufacturing Planning and Control: The Daydreaming Engine Approach
2023, Procedia CIRP
In a world that requires swift responses to a volatile environment, companies need to integrate multiple perspectives seamlessly into their planning processes. While there has been tremendous progress in model-based manufacturing planning and control, both in terms of methods and computing capabilities, the knowledge in many companies remains fractured within their functional departments. This fracture can, for example, lead to production network planning being oblivious to information regarding potential sales development or mismatches in order allocation and logistics planning. These issues may be attributed to the specificity of different heuristics, simulation, optimisation and metamodels and missing avenues to transmit information beyond a singular planned configuration. To overcome this issue, this contribution presents the concept of a daydreaming engine, permits and encourages interaction between different models of manufacturing systems. In a daydreaming engine, the models can request information from each other on-demand, external changes can trigger automatic replanning, and users can utilise daydreaming models to create and explore even unlikely scenarios automatically. Using the daydreaming framework, users can determine possible reactions to scenarios and prepare factors necessary for a quick reaction. The components and modes of interaction within this engine are discussed in detail, and the first steps towards an industrial application are presented. This paper contributes to intelligently connecting physical and virtual production environments using various quantitative models and expertise from several domains.
Data-driven optimization of building-integrated ducted openings for wind energy harvesting: Sensitivity analysis of metamodels
2022, Energy
Metamodels are developed and used for aerodynamic optimization of a ducted opening integrated into a high-rise building to maximize the amplification factor within the duct. The duct consists of a nozzle, a throat, and a diffuser. 211 high-resolution 3D RANS CFD simulations are performed to generate training and testing datasets. The space-filling design and Genetic algorithm are used for data sampling and optimization, respectively. The performance of five commonly-used metamodels is systematically investigated: Response Surface Methodology (RSM), Kriging (KG), Neural Network (NN), Support Vector Regression (SVR), and Genetic Aggregation Response Surface (GARS). The investigation is based on (i) detailed in-sample and out-of-sample evaluations of the metamodels, (ii) annual available power in the wind ( $P_{available}$ ), and (iii) annual energy production (AEP) for a 3-bladed horizontal-axis wind turbine (HAWT) installed in the mid-throat for the optimum designs obtained by the metamodels. The results show that converging-diverging ducted openings can magnify the experienced wind speed by the turbine and enhance the available wind power. In addition, the use of different metamodels can lead to a variation of up to 153% in the estimated $P_{available}$ . For a small dataset, crude yet still acceptable accuracy can be achieved for Genetic Aggregation Response Surface and Kriging at a very low computational time.

View all citing articles on Scopus

View full text

A systematic comparison of metamodeling techniques for simulation optimization in Decision Support Systems

Abstract

Introduction

Section snippets

Simulation optimization and metamodeling

Multivariate adaptive regression splines (MARS)

Test on metamodeling performances

Metamodels in Decision Support System optimization

Conclusions and future works

Decision Support Systems

Decision Support Systems

Computers and Chemical Engineering

European Journal of Operational Research

Information Sciences

Advanced Engineering Informatics

Engineering Applications of Artificial Intelligence

European Journal of Operational Research

Information Processing & Management

European Journal of Operational Research

Computers and Operations Research

Computers and Industrial Engineering

Journal of Systems and Software

Neurocomputing

Neural Networks

Simulation Modelling Practice and Theory

Neural Networks

Simulation optimization decision support system for ship panel shop operations

Statistical Tools for Simulation Practitioners

Simulation-based optimization using computational intelligence

Optimization and Engineering

Review of metamodeling techniques in support of engineering design optimization

Transactions of the ASME, Journal of Mechanical Design

Comparison of response surface and kriging models for multidisciplinary design optimization

Comparative studies of metamodeling techniques under multiple modelling criteria

Structural and Multidisciplinary Optimization

A comparative study of metamodeling methods for multiobjective crashworthiness optimization

Computers and Structures

Multivariate adaptive regression splines

Annals Statistics

Kriging for interpolation in random simulation

Journal of the Operational Research Society

Application-driven sequential designs for simulation experiments: Kriging metamodeling

Journal of the Operational Research Society

Gaussian radial basis functions for simulation metamodeling

Proceedings of the 2002 Winter Simulation Conference