Design and Analysis of Simulation Experiments

This chapter discusses the basics of low-order polynomial regression metamodels and their designs. This chapter is organized as follows. Section 2.1 discusses black-box versus white-box approaches in the design of simulation experiments (DASE). Section 2.2 covers the basics of linear regression analysis. Section 2.3 focuses on first-order polynomial regression. Section 2.4 presents designs for estimating such first-order polynomials; namely, so-called resolution-III (R-III) designs. Section 2.5 augments the first-order polynomial with interactions (cross-products). Section 2.6 discusses resolution-IV (R-IV) designs, which give unbiased estimators of the first-order effects—even if there are two-factor interactions. Section 2.7 presents resolution-V (R-V) designs, which also enable the estimation of all the individual two-factor interactions. Section 2.8 extends the first-order polynomials to second-order polynomials. Section 2.9 presents designs for second-degree polynomials, focussing on central composite designs (CCDs). Section 2.10 briefly examines “optimal” designs and other designs. Section 2.11 summarizes the major conclusions of this chapter. The chapter ends with appendixes, solutions for the exercises, and references.

This chapter is organized as follows. Section 3.1 summarizes the classic assumptions of regression analysis, which were given in the preceding chapter. Section 3.2 discusses multiple simulation outputs (responses, performance measures), which are usual in simulation practice. Section 3.3 addresses possible nonnormality of either the simulation output itself or the regression residuals (fitting errors), including tests of normality, normalizing transformations of the simulation output, and jackknifing and bootstrapping of nonnormal output. Section 3.4 covers variance heterogeneity of the simulation output, which is usual in random simulation. Section 3.5 discusses cross-correlated simulation outputs created through common random numbers (CRN); the use of CRN is popular in random simulation. Section 3.6 discusses the validation of estimated regression models, including the coefficient of determination R ² and the adjusted coefficient \(R_{\mbox{ adj}}^{2}\), and cross-validation; this section also discusses how classic low-order polynomial metamodels (detailed in the preceding chapter) may be improved in practice. Section 3.7 summarizes the major conclusions of this chapter. The chapter ends with solutions for the exercises, and a long list with references for further study.

This chapter is organized as follows. Section 4.1 introduces “screening” defined as searching for the really important inputs in experiments with simulation models that have “very many” inputs (say, hundreds of inputs); this section also gives an overview of several screening methods. Section 4.2 explains a screening method called sequential bifurcation (SB); for simplicity, this section assumes deterministic simulation and first-order polynomial metamodels. Section 4.3 explains SB for deterministic simulations and second-order polynomial metamodels that satisfy the “heredity” assumption; this assumption states that if a specific input has no first-order effect, then this input has no second-order effects either. Section 4.4 explains SB for random simulations with a fixed number of replications per input combination. Section 4.5 explains SB for random simulations with a variable number of replications determined through Wald’s sequential probability ratio test (SPRT). Section 4.6 discusses multiresponse sequential bifurcation (MSB), which extends SB to problems with multiple types of simulation responses (multivariate output). Section 4.7 discusses validation of the SB and MSB assumptions. Section 4.8 summarizes the major conclusions of this chapter. The chapter ends with solutions for the exercises, and a list with references for further study.

This chapter is organized as follows. Section 5.1 introduces Kriging, which is also called Gaussian process (GP) or spatial correlation modeling. Section 5.2 details so-called ordinary Kriging (OK), including the basic Kriging assumptions and formulas assuming deterministic simulation. Section 5.3 discusses parametric bootstrapping and conditional simulation for estimating the variance of the OK predictor. Section 5.4 discusses universal Kriging (UK) in deterministic simulation. Section 5.5 surveys designs for selecting the input combinations that gives input/output data to which Kriging metamodels can be fitted; this section focuses on Latin hypercube sampling (LHS) and customized sequential designs. Section 5.6 presents stochastic Kriging (SK) for random simulations. Section 5.7 discusses bootstrapping with acceptance/rejection for obtaining Kriging predictors that are monotonic functions of their inputs. Section 5.8 discusses sensitivity analysis of Kriging models through functional analysis of variance (FANOVA) using Sobol’s indexes. Section 5.9 discusses risk analysis (RA) or uncertainty analysis (UA). Section 5.10 discusses several remaining issues. Section 5.11 summarizes the major conclusions of this chapter, and suggests topics for future research. The chapter ends with Solutions of exercises, and a long list of references.

This chapter is organized as follows. Section 6.1 introduces the optimization of real systems that are modeled through either deterministic or random simulation; this optimization we call simulation optimization or briefly optimization. There are many methods for this optimization, but we focus on methods that use specific metamodels of the underlying simulation models; these metamodels were detailed in the preceding chapters, and use either linear regression or Kriging. Section 6.2 discusses the use of linear regression metamodels for optimization. Section 6.2.1 summarizes basic response surface methodology (RSM), which uses linear regression; RSM was developed for experiments with real systems. Section 6.2.2 adapts this RSM to the needs of random simulation. Section 6.2.3 presents the adapted steepest descent (ASD) search direction. Section 6.2.4 summarizes generalized RSM (GRSM) for simulation with multiple responses. Section 6.2.5 summarizes a procedure for testing whether an estimated optimum is truly optimal—using the Karush-Kuhn-Tucker (KKT) conditions. Section 6.3 discusses the use of Kriging metamodels for optimization. Section 6.3.1 presents efficient global optimization (EGO), which uses Kriging. Section 6.3.2 presents Kriging and integer mathematical programming (KrIMP) for the solution of problems with constrained outputs. Section 6.4 discusses robust optimization (RO), which accounts for uncertainties in some inputs. Section 6.4.1 discusses RO using RSM, Sect. 6.4.2 discusses RO using Kriging, and Sect. 6.4.3 summarizes the Ben-Tal et al. approach to RO. Section 6.5 summarizes the major conclusions of this chapter, and suggests topics for future research. The chapter ends with Solutions of exercises, and a long list of references.