Skip to main content
SpringerLink
Log in

Survey of modeling and optimization strategies to solve high-dimensional design problems with computationally-expensive black-box functions

  • Research Paper
  • Published:
Structural and Multidisciplinary Optimization Aims and scope Submit manuscript

Abstract

The integration of optimization methodologies with computational analyses/simulations has a profound impact on the product design. Such integration, however, faces multiple challenges. The most eminent challenges arise from high-dimensionality of problems, computationally-expensive analysis/simulation, and unknown function properties (i.e., black-box functions). The merger of these three challenges severely aggravates the difficulty and becomes a major hurdle for design optimization. This paper provides a survey on related modeling and optimization strategies that may help to solve High-dimensional, Expensive (computationally), Black-box (HEB) problems. The survey screens out 207 references including multiple historical reviews on relevant subjects from more than 1,000 papers in a variety of disciplines. This survey has been performed in three areas: strategies tackling high-dimensionality of problems, model approximation techniques, and direct optimization strategies for computationally-expensive black-box functions and promising ideas behind non-gradient optimization algorithms. Major contributions in each area are discussed and presented in an organized manner. The survey exposes that direct modeling and optimization strategies to address HEB problems are scarce and sporadic, partially due to the difficulty of the problem itself. Moreover, it is revealed that current modeling research tends to focus on sampling and modeling techniques themselves and neglect studying and taking the advantages of characteristics of the underlying expensive functions. Based on the survey results, two promising approaches are identified to solve HEB problems. Directions for future research are also discussed.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  • Ahn K-H, Chung WK (2002) Optimization with joint space reduction and extension induced by kinematic limits for redundant manipulators. In: Proceedings of the 2002 IEEE international conference on robotics & automation, Washington DC, 11–15 May

  • Alexandrov N, Alter SJ, Atkins HL, Bey KS, Bibb KL, Biedron RT (2002) Opportunities for breakthroughs in large-scale computational simulation and design: NASA/TM-2002-211747

  • Altus SS, Kroo IM, Gage PJ (1996) A genetic algorithm for scheduling and decomposition of multidisciplinary design problems. ASME J Mech Des 118:486–489

    Google Scholar 

  • An J, Owen A (2001) Quasi-regression. J Complex 17(4):588–607

    MATH  MathSciNet  Google Scholar 

  • Andrews DWK, Whang Y-J (1990) Additive interactive regression models: circumvention of the curse of dimensionality. Econ Theory 6:466–479

    MathSciNet  Google Scholar 

  • Apley DW, Liu J, Chen W (2006) Understanding the effects of model uncertainty in robust design with computer experiments. ASME J Mech Des 128:945–958

    Google Scholar 

  • Arora JS, Elwakeil OA, Chahande AI (1995) Global optimization methods for engineering applications: a review. Struct Optim 9:137–159

    Google Scholar 

  • Audet C, Dennis JEJ (2004) A pattern search filter method for nonlinear programming without derivatives. SIAM J Optim 14(4):980–1010

    MATH  MathSciNet  Google Scholar 

  • Bakr MH, Bandler JW, Biernacki RM, Chen SHS, Madsen K (1998) A trust region aggressive space mapping algorithm for EM Optimization. IEEE Trans Microwave Theor Tech 46(12):2412–2425

    Google Scholar 

  • Bakr MH, Bandler JW, Georgieva N (1999a) An aggressive approach to parameter extraction. IEEE Trans Microwave Theor Tech 47(12):2428–2439

    Google Scholar 

  • Bakr MH, Bandler JW, Georgieva N, Madsen K (1999b) A hybrid aggressive space-mapping algorithm for EM optimization. IEEE Trans Microwave Theor Tech 47(12):2440–2449

    Google Scholar 

  • Bakr MH, Bandler JW, Madsen K, ErnestoRayas-Sanchez J, Sondergaard J (2000a) Space-mapping optimization of microwave circuits exploiting surrogate models. IEEE Trans Microwave Theor Tech 48(12):2297–2306

    Google Scholar 

  • Bakr MH, Bandler JW, Madsen K, Sondergaard J (2000b) Review of the space mapping approach to engineering optimization and modeling. J Optim Eng 1:241–276

    MATH  MathSciNet  Google Scholar 

  • Bandler JW, Biernacki RM, Chen SH, Grobelny PA, Hemmers RH (1994) Space mapping technique for electromagnetic optimization. IEEE Trans Microwave Theor Tech 42(12):2536–2544

    Google Scholar 

  • Bandler JW, Bienacki RM, Chen SH, Hemmers RH, Madsen K (1995a) Electromagnetic optimization exploiting aggressive space mapping. IEEE Trans Microwave Theor Tech 43(12):2874–2882

    Google Scholar 

  • Bandler JW, Biernacki RM, Chen SH, Hemmers RH, Madsen K (1995b) Aggressive space mapping for electromagnetic design. In: IEEE MTT-S int. microwave symp. dig., Orlando, FL, 16–20 May

  • Bandler JW, Cheng QS, Dakroury SA, Mohamed AS, Bakr MH, Madsen K (2004) Space mapping: the state of the art. IEEE Trans Microwave Theor Tech 52(1):337–361

    Google Scholar 

  • Banerjee I, Ierapetritou MG (2002) Design optimization under parameter uncertainty for general black-box models. Ind Eng Chem Res 41:6687–6697

    Google Scholar 

  • Barry D (1986) Nonparametric Bayesian regression. Ann Stat 14(3):934–953

    MATH  MathSciNet  Google Scholar 

  • Bartholomew-Biggs MC, Parkhurst SC, Wilson SP (2003) Global optimization—stochastic or deterministic? Stochastic algorithms: foundations and applications, vol 2827/2003. Springer, Berlin, pp 125–137

    Google Scholar 

  • Bates RA, Buck RJ, Riccomagno E, Wynn HP (1996) Experimental design and observation for large systems. J R Stat Soc B 58(1):77–94

    MATH  MathSciNet  Google Scholar 

  • Björkman M, Holmström K (1999) Global optimization using the DIRECT algorithm in Matlab. Adv Model Optim 1(2):17–37

    MATH  Google Scholar 

  • Booker AJ, Dennis JEJ, Frank PD, Serafini DB, Torczon V, Trosset MW (1999) A rigorous framework for optimization of expensive functions by surrogates. Struct Optim 17(1):1–13

    Google Scholar 

  • Bose RC, Bush KA (1952) Orthogonal arrays of strength two and three. Ann Math Stat 23(4):508–524

    MATH  MathSciNet  Google Scholar 

  • Box GEP (1957) Evolutionary operation: a method for increasing industrial productivity. Appl Stat 6:81–101

    Google Scholar 

  • Brand M (2003) Continuous nonlinear dimensionality reduction by kernel eigenmaps. http://www.merl.com/papers/docs/TR2003-21.pdf. Accessed 8 August 2008

  • Browning TR (2001) Applying the design structure matrix to system decomposition and integration problems: a review and new directions. IEEE Trans Eng Manage 48(3):292–306

    Google Scholar 

  • Byrd RH, Schnabel RB, Shults GA (1987) A trust region algorithm for nonlinearly constrained optimization. SIAM J Numer Anal 24(5):1152–1170

    MATH  MathSciNet  Google Scholar 

  • Celis MR, Dennis JEJ, Tapia RA (1984) A trust region strategy for nonlinear equality constrained optimization. In: Boggs PT, Byrd RH, Schnable RB (eds) Numerical optimization. Society for Industrial and Applied Mathematics, Philadelphia, pp 71–82

    Google Scholar 

  • Chaloner K, Verdinelli I (1995) Bayesian experimental design: a review. Stat Sci 10(3):273–304

    MATH  MathSciNet  Google Scholar 

  • Chan TF, Golub GH, LeVeque RJ (1983) Algorithms for computing the sample variance: analysis and recommendations. The American Statistician 37(3):242–247

    MATH  MathSciNet  Google Scholar 

  • Chan TF, Cong J, Kong T, Shinnerl JR (2000) Multilevel optimization for large-scale circuit placement. In: Proceedings of the 2000 IEEE/ACM international conference on computer-aided design, San Jose, California, 5–9 November

  • Chen Z (1991) Interaction spline models and their convergence rates. Ann Stat 19(4):1855–1868

    MATH  Google Scholar 

  • Chen Z (1993) Fitting multivariate regression functions by interaction spline models. J R Stat Soc 55(2):473–491

    MATH  Google Scholar 

  • Chen L, Li S (2005) Analysis of decomposability and complexity for design problems in the context of decomposition. ASME J Mech Des 127:545–557

    Google Scholar 

  • Chen D-Z, Liu C-P (1999) A hierarchical decomposition scheme for the topological synthesis of articulated gear mechanisms. ASME J Mech Des 121:256–263

    Google Scholar 

  • Chen W, Allen JK, Mavris DN, Mistree R (1996) A concept exploration method for determining robust top-level specifications. Eng Optim 26(2):137–158

    Google Scholar 

  • Chen VCP, Ruppert D, Shoemaker CA (1999) Applying experimental design and regression splines to high-dimensional continuous state stochastic dynamic programming. Oper Res 47(1):38–53

    MATH  MathSciNet  Google Scholar 

  • Chen VCP, Tsui K-L, Barton RR, Allen JK (2003) A review of design and modeling in computer experiments. Handb Stat 22:231–261

    MathSciNet  Google Scholar 

  • Chen L, Ding Z, Li S (2005a) A formal two-phase method for decomposition of complex design problems. ASME J Mech Des 127:184–195

    Google Scholar 

  • Chen L, Ding Z, Li S (2005b) Tree-based dependency analysis in decomposition and re-decomposition of complex design problems. ASME J Mech Des 127:12–23

    Google Scholar 

  • Chen VCP, Tsui K-L, Barton RR, Meckesheimer M (2006) A review on design, modeling and applications of computer experiments. IIE Trans 38:273–291

    Google Scholar 

  • Collobert R, Bengio S (2001) SVMTorch: support vector machines for large-scale regression problems. J Mach Learn Res 1:143–160

    MathSciNet  Google Scholar 

  • Crary SB (2002) Design of computer experiments for metamodel generation. Analog Integr Circuits Signal Process 32:7–16

    Google Scholar 

  • Currin C, Mitchell T, Morris M, Ylvisaker D (1988) A Bayesian approach to the design and analysis of computer experiments. Technical report 6498, Oak Ridge National Laboratory

  • Currin C, Mitchell T, Morris M, Ylvisaker D (1991) Bayesian prediction of deterministic functions, with applications to the design and analysis of computer experiments. J Am Stat Assoc 86(416):953–963

    MathSciNet  Google Scholar 

  • Denison DGT (1997) Simulation based Bayesian nonparametric regression methods. Ph.D. thesis, Imperial College, London University, London

    Google Scholar 

  • Denison DGT (1998) Nonparametric Bayesian regression methods. In: Proceedings of the section on Bayesian statistical science. American Statistics Association. http://www.ma.ic.ac.uk/statistics/links/ralinks/dgtd.link/jsmpaper.ps. Accessed 6 Nov 2008

  • Ding C, He X, Zha H, Simon HD (2002) Adaptive dimension reduction for clustering high dimensional data. In: The 2002 IEEE international conference on data mining (ICDM’02), Maebashi City, Japan, 9–12 December. IEEE, pp 147–154

  • Dorigo M, Maniezzo V, Colorni A (1996) The ant system: optimization by a colony of cooperating agents. IEEE Trans Sys Man Cyber B 26:29–41

    Google Scholar 

  • Dunteman GH (1989) Principal components analysis. Sage, London

    Google Scholar 

  • Eldred MS, Hart WE, Schimel BD, Waanders BGVB (2000) Multilevel parallelism for optimization on MP computers: theory and experiment. In: Proceedings of the 8th AIAA/USAF/NASA/ISSMO symposium on multidisciplinary analysis and optimization, Long Beach, CA, September, AIAA-2000-4818

  • Eldred MS, Giunta AA, Waanders BGB (2004) Multilevel parallel optimization using massively parallel structural dynamics. Struct Multidisc Optim 27(1–2):97–109

    Google Scholar 

  • Fadel GM, Cimtalay S (1993) Automatic evaluation of move-limits in structural optimization. Struct Optim 6:233–237

    Google Scholar 

  • Fadel GM, Riley MF, Barthelemy JM (1990) Two points exponential approximation method for structural optimization. Struct Multidisc Optim 2:117–124

    Google Scholar 

  • Fang H, Horstemeyer MF (2006) Global response approximation with radial basis functions. J Eng Optim 38(4):407–424

    MathSciNet  Google Scholar 

  • Ford I, Titterington DM, Kitsos CP (1989) Recent advances in nonlinear experimental design. Technometrics 31(1):49–60

    MATH  MathSciNet  Google Scholar 

  • Friedman JH (1991) Multivariate adaptive regressive splines. Ann Stat 19(1):1–67

    MATH  Google Scholar 

  • Friedman JH, Silverman BW (1989) Flexible parsimonious smoothing and additive modeling. Technometrics 31(1):3–21

    MATH  MathSciNet  Google Scholar 

  • Friedman JH, Stuetzle W (1981) Projection pursuit regression. J Am Stat Assoc 76(372):817–823

    MathSciNet  Google Scholar 

  • Goldberg DE (1989) Genetic algorithms in search, optimization and machine learning. Addison-Wesley, Boston

    MATH  Google Scholar 

  • Griensven AV et al (2006) A global sensitivity analysis tool for the parameters of multi-variable catchment models. J Hydrol 324:10–23

    Google Scholar 

  • Grignon P, Fadel GM (1994) Fuzzy move limit evaluation in structural optimization. In: The 5th AIAA/NASA/USAF/ISSMO fifth symposium on multidisciplinary analysis and optimization, Panama City, FL, 7–9 September, AIAA-94-4281

  • Gu L (2001) A comparison of polynomial based regression models in vehicle safety analysis. In: Proceedings of 2001 ASME design engineering technical conferences—design automation conference, Pittsburgh, PA, 9–12 September

  • Haftka RT (1991) Combining global and local approximations. AIAA J 29(9):1523–1525

    Google Scholar 

  • Haftka RT, Scott EP, Cruz JR (1998) Optimization and experiments: a survey. Appl Mech Rev 51(7):435–448

    Google Scholar 

  • Hamby DM (1994) A review of techniques for parameter sensitivity analysis of environmental models. Environ Monit Assess 32:135–154

    Google Scholar 

  • Harada T et al (2006) Screening parameters of pulmonary and cardiovascular integrated model with sensitivity analysis. In: Proceedings of the 28th IEEE EMBS annual international conference, New York City, USA, 30 Aug–3 Sept 2006

  • Hedayat AS, Sloane NJA, Stufken J (1999) Orthogonal arrays: theory and applications. Springer, New York

    MATH  Google Scholar 

  • Hill WJ, Hunter WG (1966) A review of response surface methodology: a literature survey. Technometrics 8(4):571–590

    MathSciNet  Google Scholar 

  • Holland JH (1975) Adaptation in natural and artificial systems. University of Michigan Press, Ann Arbor

    Google Scholar 

  • Hooker G (2004) Discovering additive structure in black box functions. In: Proceedings of the tenth ACM SIGKDD international conference on knowledge discovery and data mining, Seattle, WA, USA, 22–25 August

  • Iman RL, Conover WJ (1980) Small sensitivity analysis techniques for computer models with an application to risk assessment. Commun. Stat, Theory and Methods A 9(17):1749–1842

    MathSciNet  Google Scholar 

  • Jiang T, Owen AB (2002) Quasi-regression for visualization and interpretation of black box functions. Stanford University, Stanford

    Google Scholar 

  • Jiang T, Owen AB (2003) Quasi-regression with shrinkage. Math Comput Simul 62(3-6):231–241

    MATH  MathSciNet  Google Scholar 

  • Jin R, Chen W, Simpson TW (2001) Comparative studies of metamodeling techniques under multiple modeling criteria. Struct Multidisc Optim 23(1):1–13

    Google Scholar 

  • Jin R, Chen W, Sudjianto A (2002) On sequential sampling for global metamodeling in engineering design. In: The ASME 2002 design engineering technical conferences and computer and information in engineering conference, Montreal, Canada, 29 September–2 October

  • Jin R, Chen W, Sudjianto A (2004) Analytical metamodel-based global sensitivity analysis and uncertainty propagation for robust design. In: SAE 2004 world congress, Detroit, MI, USA, 8–11 March, SAE 2004-01-0429

  • Jin R, Chen W, Sudjianto A (2005) An efficient algorithm for constructing optimal design of computer experiments. J Stat Plan Inference 134(1):268–287

    MATH  MathSciNet  Google Scholar 

  • John RCS, Draper NR (1975) D-Optimality for regression designs: a review. Technometrics 17(1):15–23

    MATH  MathSciNet  Google Scholar 

  • Johnson ME, Moore LM, Ylvisaker D (1990) Minimax and maximin distance designs. J Stat Plan Inference 26(2):131–148

    MathSciNet  Google Scholar 

  • Jones DR, Perttunen CD, Stuckman BE (1993) Lipschitzian optimization without the Lipschitz constant. J Optim Theory Appl 79(1):157–181

    MATH  MathSciNet  Google Scholar 

  • Jones DR, Schonlau M, Welch WJ (1998) Efficient global optimization of expensive black-box functions. J Glob Optim 13:455–492

    MATH  MathSciNet  Google Scholar 

  • Joseph VR, Hung Y, Sudjianto A (2006) Blind kriging: a new method for developing metamodels. http://www2.isye.gatech.edu/statistics/papers/. Accessed 8 August 2008

  • Kaski S (1998) Dimensionality reduction by random mapping: fast similarity computation for clustering. In: The neural networks proceedings, 1998. IEEE world congress on computational intelligence, Anchorage, AK, USA, 4–9 May

  • Kaufman M, Balabanov V, Burgee SL, Giunta AA, Grossman B, Haftka RT et al (1996) Variable-complexity response surface approximations for wing structural weight in HSCT design. Comput Mech 18:112–126

    MATH  Google Scholar 

  • Kaya H, Kaplan M, Saygin H (2004) A recursive algorithm for finding HDMR terms for sensitivity analysis. Comput Phys Commun 158:106–112

    Google Scholar 

  • Kennedy J, Eberhart RC (1995) Particle swarm optimization. In: Proceedings of IEEE international conference on neural networks, Perth, WA, Australia, 27 Nov 1995, pp 1942–1948

  • Kennedy MC, O’Hagan A (2001) Bayesian calibration of computer models. J R Stat Soc B 63(3):425–464

    MATH  MathSciNet  Google Scholar 

  • Kim HM, Michelena NF, Papalambros PY, Jiang T (2003) Target cascading in optimal system design. ASME J Mech Des 125:474–480

    Google Scholar 

  • Kirkpatrick S et al (1983) Optimization by simulated annealing. Science 220:671–680

    MathSciNet  Google Scholar 

  • Koch PN, Allen JK, Mistree F, Mavris DN (1997) The problem of size in robust design. In: ASME advances in design automation

  • Koch PN, Simpson TW, Allen JK, Mistree F (1999) Statistical approximations for multidisciplinary design optimization: the problem of size. J Aircr 36(1):275–286

    Google Scholar 

  • Koch PN, Mavris D, Mistree F (2000) Partitioned, multilevel response surfaces for modeling complex systems. AIAA J 38(5):875–881

    Google Scholar 

  • Kodiyalam S, Sobieszczanski-Sobieski J (2000) Bilevel integrated system synthesis with response surfaces. AIAA J 38(8):1479–1485

    Google Scholar 

  • Kokkolaras M, Mourelatos ZP, Papalambros PY (2006) Design optimization of hierarchically decomposed multilevel systems under uncertainty. ASME J Mech Des 128:503–508

    Google Scholar 

  • Krishnamachari RS, Papalambros PY (1997a) Hierarchical decomposition synthesis in optimal systems design. ASME J Mech Des 119:448–457

    Google Scholar 

  • Krishnamachari RS, Papalambros PY (1997b) Optimal hierarchical decomposition synthesis using integer programming. ASME J Mech Des 119:440–447

    Google Scholar 

  • Kusiak A, Larson N (1995) Decomposition and representation methods in mechanical design. ASME J Mech Des 117(special 50th anniversary design issue):17–24

    Google Scholar 

  • Kusiak A, Szczerbicki E (1992) A formal approach to specifications in conceptual design. ASME J Mech Des 114:659–666

    Google Scholar 

  • Kusiak A, Wang J (1993) Decomposition of the design process. ASME J Mech Des 115:687–693

    Google Scholar 

  • Lambert TJ III, Epelman MA, Smith RL (2005) A fictitious play approach to large-scale optimization. Oper Res 53(3):477–489

    MATH  MathSciNet  Google Scholar 

  • Leary SJ, Bhaskar A, Keane AJ (2001) A constraint mapping approach to the structural optimization of an expensive model using surrogates. J Optim Eng 2:385–398

    MATH  MathSciNet  Google Scholar 

  • Leary SJ, Bhaskar A, Keane AJ (2003) A knowledge-based approach to response surface modeling in multifidelity optimization. J Glob Optim 26:297–319

    MATH  MathSciNet  Google Scholar 

  • Leoni N, Amon CH (2000) Bayesian surrogates for integrating numerical, analytical and experimental data: application to inverse heat transfer in wearable computers. IEEE Trans Compon Packag Technol 23(1):23–32

    Google Scholar 

  • Li S (2009) Matrix-based decomposition algorithms for engineering application: survey and generic framework. Int J Prod Dev 9:78–110

    Google Scholar 

  • Li G, Rosenthal C, Rabitz H (2001a) High dimensional model representations. J Phys Chem A 105(33):7765–7777

    Google Scholar 

  • Li G, Wang S-W, Rosenthal C, Rabitz H (2001b) High dimensional model representations generated from low dimensional data samples. I. mp-Cut-HDMR. J Math Chem 30(1):1–30

    MathSciNet  Google Scholar 

  • Li G, Hu J, Wang S-W, Georgopoulos PG, Schoendorf J, Rabitz H (2006) Random sampling-high dimensional model representation (RS-HDMR) and orthogonality of its different order component functions. J Phys Chem A 110:2474–2485

    Google Scholar 

  • Lu SC-Y, Tcheng DK (1991) Building layered models to support engineering decision making: a machine learning approach. ASME J Mech Des 113:1–9

    Google Scholar 

  • Marin FTS, Gonzalez AP (2003) Global optimization in path synthesis based on design space reduction. Mech Mach Theory 38:579–594

    MATH  MathSciNet  Google Scholar 

  • Martin JD, Simpson TW (2005) Use of kriging models to approximate deterministic computer models. AIAA J 43(4):853–863

    Google Scholar 

  • McKay MD, Bechman RJ, Conover WJ (1979) A comparison of three methods for selecting values of input variables in the analysis of output from a computer code. Technometrics 21(2):239–245

    MATH  MathSciNet  Google Scholar 

  • Meckesheimer M, Booker AJ, Barton RR, Simpson TW (2002) Computationally inexpensive metamodel assessment strategies. AIAA J 40(10):2053–2060

    Google Scholar 

  • Michelena NF, Papalambros PY (1995a) A network reliability approach to optimal decomposition of design problems. ASME J Mech Des 117:433–440

    Google Scholar 

  • Michelena NF, Papalambros PY (1995b) Optimal model-based decomposition of powertrain system design. ASME J Mech Des 117:499–505

    Google Scholar 

  • Michelena NF, Papalambros PY (1997) A hypergraph framework for optimal model-based decomposition of design problems. Comput Optim Appl 8(2):173–196

    MATH  MathSciNet  Google Scholar 

  • Michelena N, Jiang T, Papalambros P (1995) Decomposition of simultaneous analysis and design models. In: Proceedings of the 1st world congress of structural and multidisciplinary optimization, pp 845–850

  • Michelena N, Papalambros P, Park HA, Kulkarni D (1999) Hierarchical overlapping coordination for large-scale optimization by decomposition. AIAA J 37(7):890–896

    Google Scholar 

  • Mitchell TJ, Morris MD (1992) Bayesian design and analysis of computer experiments: two examples. Stat Sinica 2:359–379

    MATH  Google Scholar 

  • Morris MD (1991) Factorial sampling plans for preliminary computational experiments. Technometrics 33(2):161–174

    Google Scholar 

  • Morris MD, Mitchell TJ (1983) Two-level multifactor designs for detecting the presence of interactions. Technometrics 25(4):345–355

    MATH  MathSciNet  Google Scholar 

  • Morris MD, Mitchell TJ (1995) Exploratory designs for computational experiments. J Stat Plan Inference 43:381–402

    MATH  Google Scholar 

  • Morris MD, Mitchell TJ, Ylvisaker D (1993) Bayesian design and analysis of computer experiments: use of derivatives in surface prediction. Technometrics 35(3):243–255

    MATH  MathSciNet  Google Scholar 

  • Myers RH, Montgomery D (1995) Response surface methodology: process and product optimization using designed experiments. Wiley, Toronto

    MATH  Google Scholar 

  • Nain PKS, Deb K (2002) A computationally effective multi-objective search and optimization technique using coarse-to-fine grain modeling (KanGal report no. 2002005). Indian Institute of Technology Kanpur, Kanpur

    Google Scholar 

  • Oakley JE, O’Hagan A (2004) Probabilistic sensitivity analysis of complex models: a Bayesian approach. J R Stat Soc B 66(3):751–769

    MATH  MathSciNet  Google Scholar 

  • Otto J, Paraschivoiu M, Yesilyurt S, Patera AT (1997) Bayesian-validated computer-simulation surrogates for optimization and design: error estimates and applications. Math Comput Simul 44:347–367

    MATH  MathSciNet  Google Scholar 

  • Owen AB (1992a) Orthogonal arrays for computer experiments, integration, and visualization. Stat Sinica 2:439–452

    MATH  MathSciNet  Google Scholar 

  • Owen AB (1992b) A central limit theorem for Latin hypercube sampling. J R Stat Soc 54(2):541–551

    MATH  MathSciNet  Google Scholar 

  • Owen AB (1998) Detecting near linearity in high dimensions. Stanford University, Stanford

    Google Scholar 

  • Owen AB (2000) Assessing linearity in high dimensions. Ann Stat 28(1):1–19

    MATH  MathSciNet  Google Scholar 

  • Papalambros PY (1995) Optimal design of mechanical engineering systems. ASME J Mech Des 117(special 50th anniversary design issue):55–62

    Google Scholar 

  • Papalambros PY, Michelena NF (1997) Model-based partitioning in optimal design of large engineering systems. In: Multidisciplinary design optimization: state-of-the-art. SIAM, pp 209–226

  • Papalambros PY, Michelena NF (2000) Trends and challenges in system design optimization. In: Proceedings of the international workshop on multidisciplinary design optimization, Pretoria, S. Africa, 7–10 August

  • Penha RML, Hines JW (2001) Using principal component analysis modeling to monitor temperature sensors in a nuclear research reactor. In: Proceedings of the maintenance and reliability conference (MARCON 2001), Knoxville, TN, 6–9 May

  • Pérez VM, Apker TB, Renaud JE (2002a) Parallel processing in sequential approximate optimization. In: The 43rd AIAA/ASME/ASCE/AHS/ASC structures, structural dynamics, and materials conference, Denver, Colorado, 22–25 Apr, AIAA-2002-1589

  • Pérez VM, Renaud JE, Watson LT (2002b) Reduced sampling for construction of quadratic response surface approximations using adaptive experimental design. In: The 43rd AIAA/ASME/ASCE/AHS/ASC structures, structural dynamics, and materials conference, Denver, Colorado, 22–25 Apr, AIAA-2002-1587

  • Queipo NV et al (2005) Surrogate-based analysis and optimization. Prog Aerospace Sci 41:1–18

    Google Scholar 

  • Rabitz H, Alis ÖF (1999) General foundations of high-dimensional model representations. J Math Chem 25:197–233

    MATH  MathSciNet  Google Scholar 

  • Rabitz H, Alis ÖF, Shorter J, Shim K (1999) Efficient input–output model representations. Comput Phys Commun 117:11–20

    MATH  Google Scholar 

  • Rao SS, Mulkay EL (2000) Engineering design optimization using interior-point algorithms. AIAA J 38(11):2127–2132

    Google Scholar 

  • Rassokhin DN, Lobanov VS, Agratiotis DK (2000) Nonlinear mapping of massive data sets by fuzzy clustering and neural networks. J Comput Chem 22(4):373–386

    Google Scholar 

  • Ratschek H, Rokne JG (1987) Efficiency of a global optimization algorithm. SIAM J Numer Anal 24(5):1191–1201

    MATH  MathSciNet  Google Scholar 

  • Regis RG, Shoemaker CA (2007a) Parallel radial basis function methods for the global optimization of expensive functions. Eur J Oper Res 182:514–535

    MATH  MathSciNet  Google Scholar 

  • Regis RG, Shoemaker CA (2007b) A stochastic radial basis function method for the global optimization of expensive functions. INFORMS J Comput 19(4):497–509

    MathSciNet  Google Scholar 

  • Renaud JE (1993) Second order based multidisciplinary design optimization algorithm development. Adv Des Autom 65-2:347–357

    Google Scholar 

  • Renaud JE, Gabriele GA (1991) Sequential global approximation in non-hierarchic system decomposition and optimization. Adv Des Autom 32-1:191–200

    Google Scholar 

  • Rodríguez JF, Renaud JE, Watson LT (1998) Trust region augmented Lagrangian methods for sequential response surface approximation and optimization. ASME J Mech Des 120:58–66

    Google Scholar 

  • Sacks J, Schiller SB, Welch WJ (1989a) Designs for computer experiments. Technometrics 31(1):41–47

    MathSciNet  Google Scholar 

  • Sacks J, Welch WJ, Mitchell TJ, Wynn HP (1989b) Design and analysis of computer experiments. Stat Sci 4(4):409–435

    MATH  MathSciNet  Google Scholar 

  • Saha A, Wu C-L, Tang D-S (1993) Approximation, dimension reduction, and nonconvex optimization using linear superpositions of Gaussians. IEEE Trans Comput 42(10):1222–1233

    MathSciNet  Google Scholar 

  • Sammon JW (1969) A nonlinear mapping for data structure analysis. IEEE Trans Comput C-18(5):401–409

    Google Scholar 

  • Schonlau M, Welch WJ (2006) Screening the input variables to a computer model via analysis of variance and visualization. Paper presented at the screening methods for experimentation in industry, drug discovery, and genetics springer, New York

  • Schonlau M, Welch WJ, Jones DR (1998) Global versus local search in constrained optimization of computer models. In: Flournoy N, Rosenberger WF, Wong WK (eds) New development and applications in experimental design. Lecture notes-monograph series, vol 34. Institute of Mathematical Statistics, Hayward, pp 11–25

    Google Scholar 

  • Shan S, Wang GG (2004) Space exploration and global optimization for computationally intensive design problems: a rough set based approach. Struct Multidisc Optim 28(6):427–441

    Google Scholar 

  • Sharif B, Wang GG, EIMekkawy T (2008) Mode pursuing sampling method for discrete variable optimization on expensive black-box functions. ASME J Mech Des 130:021402-1-11

    Google Scholar 

  • Shen HT, Zhou X, Zhou A (2006) An adaptive and dynamic dimensionality reduction method for high-dimensional indexing. The VLDB Journal. http://www.itee.uq.edu.au/~zxf/_papers/VLDBJ06.pdf. Accessed 8 August 2008

  • Shin YS, Grandhi RV (2001) A global structural optimization technique using an interval method. Struct Multidisc Optim 22:351–363

    Google Scholar 

  • Shlens J (2005) A tutorial on principal component analysis. http://www.snl.salk.edu/~shlens/pub/notes/pca.pdf. Accessed 8 August 2008

  • Shorter JA, Ip PC, Rabitz HA (1999) An efficient chemical kinetics solver using high dimensional model representation. J Phys Chem A 103:7192–7198

    Google Scholar 

  • Siah ES, Sasena M, Volakis JL, Papalambros PY (2004) Fast parameter optimization of large-scale electromagnetic objects using DIRECT with Kriging metamodeling. IEEE Trans Microwave Theor Tech 52(1):276–285

    Google Scholar 

  • Simpson TW (2004) Evaluation of a graphical design interface for design space visualization. In: Proceedings of the 45th AIAA/ASME/ASCE/AHS/ASC structures, structural dynamics & materials conference, Palm Springs, California, 19–22 April, AIAA 2004-1683

  • Simpson TW, Mauery TM, Korte JJ, Mistree F (1998) Comparison of response surface and kriging models for multidisciplinary design optimization. In: The 7th AIAA/USAF/NASA/ISSMO symposium on multidisciplinary analysis & optimization, St. Louis, MI, AIAA-98-4755

  • Simpson TW, Lin DKJ, Chen W (2001a) Sampling strategies for computer experiments: design and analysis. Int J Reliab Appl 2(3):209–240

    Google Scholar 

  • Simpson TW, Peplinski J, Koch PN, Allen JK (2001b) Metamodels for computer-based engineering design: survey and recommendations. Eng Comput 17(2):129–150

    MATH  Google Scholar 

  • Simpson TW, Booker AJ, Ghosh D, Giunta AA, Koch PN, Yang RJ (2004) Approximation methods in multidisciplinary analysis and optimization: a panel discussion. Struct Multidisc Optim 27:302–313

    Google Scholar 

  • Sobieszczanski-Sobieski J (1990) Sensitivity analysis and multidisciplinary optimization for aircraft design: recent advances and results. J Aircr 27(12):993–1001

    Google Scholar 

  • Sobieszczanski-Sobieski J, Haftka RT (1997) Multidisciplinary aerospace design optimization: survey of recent developments. Struct Optim 14(1):1–23

    Google Scholar 

  • Sobol IM (1993) Sensitivity estimates for nonlinear mathematical models. Math Model Comput Exper 1(4):407–414

    MATH  MathSciNet  Google Scholar 

  • Somorjai RL, Dolenko B, Demko A, Mandelzweig M, Nikulin AE, Baumgartner R et al (2004) Mapping high-dimensional data onto a relative distance plane—an exact method for visualizing and characterizing high-dimensional patterns. J Biomed Inform 37:366–376

    Google Scholar 

  • Srivastava A, Hacker K, Lewis KE, Simpson TW (2004) A method for using legacy data for metamodel-based design of large-scale systems. Struct Multidisc Optim 28:146–155

    Google Scholar 

  • Steinberg DM, Hunter WG (1984) Experimental design: review and comment. Technometrics 26(2):71–97

    MATH  MathSciNet  Google Scholar 

  • Stone CJ (1985) Additive regression and other nonparametric models. Ann Stat 13(2):689–705

    MATH  Google Scholar 

  • Storn R, Price K (1995) Differential evolution—a simple and efficient adaptive scheme for global optimization over continuous spaces. Technical report TR-95-012, International Computer Science Institute (ICSI), Berkley, CA, March 1995

    Google Scholar 

  • Stump G, Simpson TW, Yukish M, Bennett L (2002) Multidimensional design and visualization and its application to a design by shopping paradigm. In: The 9th AIAA/USAF/NASA/ISSMO symposium on multidisciplinary analysis and optimization, Atlanta, GA, 4–6 September, AIAA 2002-5622

  • Suh NP (2001) Axiomatic design: advances and applications. Oxford University Press, New York

    Google Scholar 

  • Tang B (1993) Orthogonal array-based Latin hypercubes. J Am Stat Assoc 88(424):1392–1397

    MATH  Google Scholar 

  • Taskin G, Saygin H, Demiralp M, Yanalak M (2002) Least squares curve fitting via high dimensional model representation for digital elevation model. In: The international symposium on GIS, Istanbul-Turkey, 23–26 September

  • Tu J, Jones DR (2003) Variable screening in metamodel design by cross-validated moving least squares method. In: The 44th AIAA/ASME/ASCE/AHS structures, structural dynamics, and materials conference, Norfolk, Virginia, 7–10 April

  • Tunga MA, Demiralp M (2005) A factorized high dimensional model representation on the nodes of a finite hyperprismatic regular grid. Appl Math Comput 164:865–883

    MATH  MathSciNet  Google Scholar 

  • Tunga MA, Demiralp M (2006) Hybrid high dimensional model representation (HHDMR) on the partitioned data. J Comput Appl Math 185:107–132

    MATH  MathSciNet  Google Scholar 

  • Vanderplaats GN (1999) Structural design optimization status and direction. J Aircr 36(1):11–20

    Google Scholar 

  • Wagner S (2007) Global sensitivity analysis of predictor models in software engineering. In: Proceedings of third international workshop on predictor models in software engineering (PROMISE’07), Washington, DC, USA. IEEE Computer Society

  • Wagner TC, Papalambros PY (1993) A general framework for decomposition analysis in optimal design. De-Vol. 65-2. Adv Des Autom 2:315–325

    Google Scholar 

  • Wang H, Ersoy OK (2005) Parallel gray code optimization for high dimensional problems. In: Proceedings of the sixth international conference on computational intelligence and multimedia applications, Las Vegas, Nevada, 16–18 August

  • Wang GG, Shan S (2004) Design space reduction for multi-objective optimization and robust design optimization problems. SAE Trans 113:101–110

    Google Scholar 

  • Wang GG, Shan S (2007) Review of metamodeling techniques in support of engineering design optimization. ASME J Mech Des 129:370–389

    Google Scholar 

  • Wang GG, Simpson TW (2004) Fuzzy clustering based hierarchical metamodeling for space reduction and design optimization. J Eng Optim 36(3):313–335

    Google Scholar 

  • Wang GG, Dong Z, Aitchison P (2001) Adaptive response surface method—a global optimization scheme for computation-intensive design problems. J Eng Optim 33(6):707–734

    Google Scholar 

  • Wang S-W, Georgopoulos PG, Li G, Rabits H (2003) Random sampling-high dimensional model representation (RS-HDMR) with nonuniformly distributed variables: application to an integrated multimedia/multipathway exposure and dose model for trichloroethylene. J Phys Chem, A 107:4707–4716

    Google Scholar 

  • Wang L, Shan S, Wang GG (2004) Mode-pursuing sampling method for global optimization on expensive black-box functions. J Eng Optim 36(4):419–438

    Google Scholar 

  • Wang L, Beeson D et al (2006) A comparison of meta-modeling methods using practical industry requirements. In: The 47th AIAA/ASME/ASCE/AHS/ASC structures, structural dynamics, and materials conference, Newport, Rhode Island, USA, 1–4 May 2006

  • Watson GS (1961) A study of the group screening method. Technometrics 3(3):371–388

    MATH  MathSciNet  Google Scholar 

  • Watson PM, Gupta KC (1996) EM-ANN models for microstrip vias and interconnects in dataset circuits. IEEE Trans Microwave Theor Tech 44(12):2495–2503

    Google Scholar 

  • Weise T (2008) Global optimization algorithms theory and application. http://www.it-weise.de/projects/book.pdf. Accessed 7 Nov 2008

  • Welch WJ, Buck RJ, Sacks J, Wynn HP, Mitchell TJ, Morris MD (1992) Screening, predicting, and computer experiments. Technometrics 34(1):15–25

    Google Scholar 

  • Winer EH, Bloebaum CL (2002a) Development of visual design steering as an aid in large-scale multidisciplinary design optimization. Part I: method development. Struct Multidisc Optim 23(6):412–424

    Google Scholar 

  • Winer EH, Bloebaum CL (2002b) Development of visual design steering as an aid in large-scale multidisciplinary design optimiza tion. Part II: method validation. Struct Multidisc Optim 23(6):425–435

    Google Scholar 

  • Wujek BA, Renaud JE (1998a) New adaptive move-limit management strategy for approximate optimization, Part 1. AIAA J 36(10):1911–1921

    Google Scholar 

  • Wujek BA, Renaud JE (1998b) New adaptive move-limit management strategy for approximate optimization, Part 2. AIAA J 36(10):1922–1934

    Google Scholar 

  • Xiong Y, Chen W, Tsui K-L (2008) A new variable fidelity optimization framework based on model fusion and objective-oriented sequential sampling. ASME J Mech Des 130:111401. doi:10.1115/1.2976449

    Google Scholar 

  • Ye KQ (1998) Orthogonal column Latin hypercubes and their application in computer experiments. J Am Stat Assoc 93(444):1430–1439

    MATH  Google Scholar 

  • Ye T, Kalyanaraman S (2003) A unified search framework for large-scale black-box optimization. http://www.ecse.rpi.edu/Homepages/shivkuma /research/papers/unisearch03.pdf. Accessed 8 August 2008

  • Yoshimura M, Izui K (1998) Machine system design optimization strategies based on expansion and contraction of design spaces. In: Proceedings of the 7th AIAA/USAF/NASA/ISSMO symposium on multidisciplinary analysis and optimization, St. Louis, USA, September. AIAA-98-4749

  • Yoshimura M, Izui K (2004) Hierarchical parallel processes of genetic algorithms for design optimization of large-scale products. ASME J Mech Des 126:217–224

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Mech. and Manuf. Eng., University of Manitoba, E2-327 EITC Building, 75A Chancellors Circle, Winnipeg, Manitoba, R3T 5V6, Canada

    Songqing Shan

  2. School of Engineering Science, Simon Fraser University, Surrey, British Columbia, V3T 0A3, Canada

    G. Gary Wang

Authors
  1. Songqing Shan
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. G. Gary Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to G. Gary Wang.

Additional information

An earlier version of this work was published in Proceedings of the 12th AIAA/ISSMO Multidisciplinary Analysis and Optimization Conference, Sept. 10–12, 2008, Victoria, British Columbia, Canada.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Shan, S., Wang, G.G. Survey of modeling and optimization strategies to solve high-dimensional design problems with computationally-expensive black-box functions. Struct Multidisc Optim 41, 219–241 (2010). https://doi.org/10.1007/s00158-009-0420-2

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00158-009-0420-2

Keywords

Search

Navigation