Koichiro Inoue
Abstract
Simulation is an important tool for comparing the performance of several alternative systems. There is therefore significant interest in procedures that efficiently select the best system, where best is defined by the maximum or minimum expected simulation output. In this paper, we examine both two-stage and sequential procedures that represent three structurally different modeling methodologies for allocating simulation replications to identify the best system, and we evaluate them empirically with respect to several measures of effectiveness. Empirical evidence suggests that sequential procedures perform better than their two-stage counterparts, including a heuristic sequential variation on Rinott's procedure. Further, there appears to be significant benefit to using procedures based on a Bayesian, average-case analysis as opposed to the statistically-conservative indifference-zone formulation.
- BANKS, J., CARSON, J. S., AND NELSON, B. L. 1996. Discrete-Event System Simulation. 2nd. ed. Prentice-Hall, Inc., Upper Saddle River, NJ.Google Scholar
- BECHHOFER, R., SANTNER, T., AND GOLDSMAN, D. 1995. Design and Analysis of Experiments for Statistical Selection, Screening, and Multiple Comparisons. John Wiley and Sons, Inc., New York, NY.Google Scholar
- BERGER, J. O. 1988. A Bayesian approach to ranking and selection of related means with alternatives to analysis-of-variance methodology. J. Am. Stat. Assoc. 83, 402, 364-373.Google Scholar
- BERNARDO, J. M. AND SMITH, A. F. M. 1994. Bayesian Theory. John Wiley and Sons Ltd., Chichester, UK.Google Scholar
- BRACKEN, J. AND SCHLEIFER, A. 1964. Tables for normal sampling with unknown variance. Tech. Rep., Graduate School of Business Admin. Harvard Univ., Cambridge, MA.Google Scholar
- CHEN, C. -H. 1996. A lower bound for the correct subset-selection probability and its application to discrete event simulations. IEEE Trans. Automat. Contr. 41, 8, 1227-1231.Google Scholar
- CHEN, C.-H., CHEN, H.-C., AND DAI, L. 1996. A gradient approach for smartly allocating computing budget for discrete event simulation. In Proceedings of the Winter Conference on Simulation (WSC '96, Coronado, CA, Dec. 8-11), J. M. Charnes, D. J. Morrice, D. T. Brunner, and J. J. Swain, Eds. ACM Press, New York, NY, 398-405. Google Scholar
- CHEN, H.-C., DAI, L., CHEN, C.-H., AND Y CESAN, E. 1997. New development of optimal computing budget allocation for discrete event simulation. In Proceedings of the Winter Conference on Simulation (WSC '97, Atlanta, GA, Dec. 7-10), S. Andrad ttir, K. J. Healy, D. H. Withers, and B. L. Nelson, Eds. ACM Press, New York, NY, 334-341. Google Scholar
- CHICK, S. E. AND INOUE, K. 1988. Sequential allocation procedures that reduce risk for multiple comparisons. In Proceedings of the Winter Conference on Simulation (WSC '88, San Diego, CA, Dec.), D. J. Medeiros, E. J. Watson, M. Manivannan, and J. Carson, Eds. ACM Press, New York, NY, 669-676. Google Scholar
- CHICK, S. E. AND INOUE, K. 1999. New two-stage and sequential procedures for selecting the best simulated system. Tech. Rep., Dept. of Industrial Operations Engineering. University of Michigan, Ann Arbor, MI.Google Scholar
- DE GROOT, M. H. 1970. Optimal Statistical Decisions. McGraw-Hill, Inc., New York, NY.Google Scholar
- DUDEWICZ, E. J. AND DALAL, S. R. 1975. Allocation of observations in ranking and selection with unequal variances. Sanhkya B37, 28-78.Google Scholar
- GOLDSMAN, D. AND NELSON, B. L. 1998. Statistical screening, selection, and multiple comparison procedures in computer simulation. In Proceedings of the Winter Conference on Simulation (WSC '98, Washington D.C., Dec. 13-16), D. J. Medeiros, E. F. Watson, J. S. Carson, and M. S. Manivannan, Eds. IEEE Computer Society Press, Los Alamitos, CA, 159-166. Google Scholar
- INOUE, K. 1999. Bayesian decision-theoretic selection procedures to identify the best simulated system. Ph.D. Dissertation. University of Michigan, Ann Arbor, MI.Google Scholar
- INOUE, K. AND CHICK, S. E. 1998. Comparison of Bayesian and frequentist assessments of uncertainty for selecting the best system. In Proceedings of the Winter Conference on Simulation (WSC '98, Washington D.C., Dec. 13-16), D. J. Medeiros, E. F. Watson, J. S. Carson, and M. S. Manivannan, Eds. IEEE Computer Society Press, Los Alamitos, CA, 727-734. Google Scholar
- KOENIG, L. W. AND LAW, A. M. 1985. A procedure for selecting a subset of size m containing the g best of k independent normal populations, with applications to simulation. Commun. Stat.-Simul. Comput. 14, 3, 719-734.Google Scholar
- LAW, A. AND KELTON, W. 1991. Simulation Modeling and Analysis. 2nd. ed. McGraw-Hill, Inc., New York, NY. Google Scholar
- MIESCKE, K. J. AND GUPTA, S 1996. Bayesian look ahead one-stage sampling allocations for selecting the best population. J. Stat. Plan. Inference 54, 229-244.Google Scholar
- NELSON, B. L. AND MATEJCIK, F.J. 1995. Using common random numbers for indifferencezone selection and multiple comparisons in simulation. Manage. Sci. 41, 12, 1935-1945. Google Scholar
- NELSON, B. L., SWANN, J., GOLDSMAN, D., AND SONG, W. 1999. Simple procedures for selecting the best simulated system when the number of alternatives is large. Tech. Rep., Dept. of Industrial Engineering and Management Science. Northwestern University, Evanston, IL.Google Scholar
- RINOTT, Y. 1978. On two-stage selection procedures and related probability-inequalities. Commun. Stat. A7, 799-811.Google Scholar
- WELCH, B. L. 1938. The significance of the difference between two means when the population variances are unequal. Biometrika 25, 350-362.Google Scholar
Index Terms
- An empirical evaluation of several methods to select the best system
Recommendations
New Procedures to Select the Best Simulated System Using Common Random Numbers
Although simulation is widely used to select the best of several alternative system designs, and common random numbers is an important tool for reducing the computation effort of simulation experiments, there are surprisingly few tools available to help ...
Selecting the best stochastic system for large scale problems in DEDS
We consider the problem of selecting the stochastic system with the best expected performance measure, when the number of alternative systems is large. We consider the case of discrete event dynamic systems (DEDS) where the standard clock simulation ...
Chance Constrained Selection of the Best
Selecting the solution with the largest or smallest mean of a primary performance measure from a finite set of solutions while requiring secondary performance measures to satisfy certain constraints is called constrained selection of the best CSB in the ...
Comments