Skip to main content
SpringerLink
Log in

Opening the ‘black box’ of simulations: increased transparency and effective communication through the systematic design of experiments

  • Published:
Computational and Mathematical Organization Theory Aims and scope Submit manuscript

Abstract

Many still view simulation models as a black box. This paper argues that perceptions could change if the systematic design of experiments (DOE) for simulation research was fully realized. DOE can increase (1) the transparency of simulation model behavior and (2) the effectiveness of reporting simulation results. Based on DOE principles, we develop a systematic procedure to guide the analysis of simulation models as well as concrete templates for sharing the results. A simulation model investigating the performance of learning algorithms in an economic mechanism design context illustrates our suggestions. Overall, the proposed systematic procedure for applying DOE principles complements current initiatives for a more standardized simulation research process.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  • Antony J (2003) Design of experiments for engineers and scientists. Butterworth-Heinemann, Amsterdam

    Google Scholar 

  • Arifovic J, Ledyard J (2004) Scaling up learning models in public good games. J Public Econ Theory 6:203–238

    Article  Google Scholar 

  • Arnold M, Ponick E, Schenk-Mathes H (2008) Groves mechanism vs. profit sharing for corporate budgeting: An experimental analysis with preplay communication. Eur Account Rev 17:37–63

    Article  Google Scholar 

  • Axelrod R (1997) Advancing the art of simulation in the social sciences. In: Conte R, Hegselmann R, Terna P (eds) Simulating social phenomena. Springer, Berlin, pp 21–40

    Google Scholar 

  • Barton R (2004) Designing simulation experiments. In: Proceedings of the winter simulation conference, Washington DC

    Google Scholar 

  • Bettonvil B, Kleijnen JPC (1996) Searching for important factors in simulation models with many factors: Sequential bifurcation. Eur J Oper Res 96:180–194

    Article  Google Scholar 

  • Box GEP, Hunter JS, Hunter WG (2005) Statistics for experimenters: design, innovation, and discovery. Wiley-Interscience, Hoboken

    Google Scholar 

  • Camerer C, Ho T (1999) Experience-weighted attraction learning in normal form games. Econometrica 67:827–874

    Article  Google Scholar 

  • Dutta P (1999) Strategies and games: theory and practice. MIT Press, Cambridge

    Google Scholar 

  • Ewert R, Wagenhofer A (2008) Interne Unternehmensrechnung. Springer, Berlin

    Google Scholar 

  • Field A, Hole G (2003) How to design and report experiments. SAGE, London

    Google Scholar 

  • Fisher RA (1971) The design of experiments. Hafner, New York

    Google Scholar 

  • Gilbert N, Troitzsch K (2005) Simulation for the social scientist. Open University Press, Maidenhead

    Google Scholar 

  • Gilbert N (2008) Agent-Based Models. SAGE, London

    Google Scholar 

  • Gode DK, Sunder S (1997) What makes markets allocationally efficient? Q J Econ 112:602

    Article  Google Scholar 

  • Grimm V, Berger U, Bastiansen F, Eliassen S, Ginot V, Giske J, Goss-Custard J, Grand T, Heinz SK, Huse G, Huth A, Jepsen JU, Jørgensen C, Mooij WM, Müller B, Pe’er G, Piou C, Railsback SF, Robbins AM, Robbins MM, Rossmanith E, Rüger N, Strand E, Souissi S, Stillman RA, VabøR, Visser U, DeAngelis DL (2006) A standard protocol for describing individual-based and agent-based models. Ecol Model 198:115–126

    Article  Google Scholar 

  • Grimm V, Berger U, DeAngelis DL, Polhill G, Giske J, Railsback SF (2010) The ODD protocol: a review and first update. Ecol Model 221:2760–2768

    Article  Google Scholar 

  • Groves T (1973) Incentives in teams. Econometrica 41:617–631

    Article  Google Scholar 

  • Groves T, Loeb M (1979) Incentives in a divisionalized firm. Manag Sci 25:221–230

    Article  Google Scholar 

  • Harrison J, Lin Z, Carroll G, Carley K (2007) Simulation modeling in organizational and management research. Acad Manag Rev 32:1229–1245

    Article  Google Scholar 

  • Heine B-O, Meyer M, Strangfeld O (2005) Stylised facts and the contribution of simulation to the economic analysis of budgeting. J Artif Soc Simul 8(4)

  • Hendricks W, Robey K (1936) The sampling distribution of the coefficient of variation. Ann Math Stat 7:129–132

    Article  Google Scholar 

  • Homburg C, Klarmann M (2003) Empirische Controllingforschung – Anmerkungen aus der Perspektive des Marketing. In: Weber J, Hirsch B (eds) Zur Zukunft der Controllingforschung, Wiesbaden, pp 65–88

    Chapter  Google Scholar 

  • Kelton WD, Barton RR (2003) Experimental design for simulation. In: Proceedings of the 2003 winter simulation conference, New Orleans, Louisiana

    Google Scholar 

  • Kleijnen JPC (1998) Experimental design for sensitivity analysis, optimization, and validation of simulation models. In: Banks J (ed) Handbook of simulation: principles, methodology, advances, applications and practice, New York, pp 173–224

    Google Scholar 

  • Kleijnen JPC, Sanchez S, Lucas T, Cioppa T (2005) A user’s guide to the brave new world of designing simulation experiments. INFORMS J Comput 17:263–289

    Article  Google Scholar 

  • Kleijnen JPC (2008) Design and analysis of simulation experiments. Springer, New York

    Google Scholar 

  • Law A (2007) Simulation modeling and analysis. McGraw-Hill, Boston

    Google Scholar 

  • Malerba F, Nelson R, Orsenigo L, Winter S (1999) History-friendly models of industry evolution: the computer industry. Ind Corp Change 8(1):3–40

    Article  Google Scholar 

  • Manuj I, Mentzer JT, Bowers MR (2009) Improving the rigor of discrete-event simulation in logistics and supply chain research. Int J Phys Distrib Logist Manag 39:172–201

    Article  Google Scholar 

  • Mårtensson A (2003) Managing mission-critical IT in the financial industry. Economic Research Institute, Stockholm School of Economics

  • Mårtensson A, Mårtensson P (2007) Extending rigor and relevance: towards credible, contributory and communicable research. In: Proceedings of the 15th European conference on information systems, St Gallen

    Google Scholar 

  • Montgomery DC (2009) Design and analysis of experiments. Wiley, Hoboken

    Google Scholar 

  • Myerson R (1999) Nash equilibrium and the history of economic theory. J Econ Lit 37:1067–1082

    Article  Google Scholar 

  • North MJ, Macal CM (2007) Managing business complexity: discovering strategic solutions with agent-based modeling and simulation. Oxford University Press, Oxford

    Google Scholar 

  • Oh RPT, Sanchez SM, Lucas TW, Wan H, Nissen ME (2009) Efficient experimental design tools for exploring large simulation models. Comput Math Organ Theory 15:237–257

    Article  Google Scholar 

  • Peck SL (2004) Simulation as experiment: a philosophical reassessment for biological modeling. Trends Ecol Evol 19:530–534

    Article  Google Scholar 

  • Polhill JG, Parker D, Brown D, Grimm V (2008) Using the ODD protocol for describing three agent-based social simulation models of land-use change. J Artif Soc Simul 11(2). http://jasss.soc.surrey.ac.uk/11/2/3.html

  • Raghu TS, Sen PK, Rao HR (2003) Relative performance of incentive mechanisms: computational modeling and simulation of delegated investment decisions. Manag Sci 49:160–178

    Article  Google Scholar 

  • Railsback SF, Grimm V (2011) Agent-based and individual-based modeling: a practical introduction. Princeton University Press, Princeton

    Google Scholar 

  • Richiardi M, Leombruni R, Saam NJ, Sonnessa M (2006) A common protocol for agent-based social simulation. J Artif Soc Simul 9(1). http://jasss.soc.surrey.ac.uk/9/1/15.html

  • Reiss J (2011) A plea for (good) simulations: Nudging economics toward an experimental science. Simul Games 42:243–264

    Article  Google Scholar 

  • Sanchez SM (2006) Work smarter, not harder: Guidelines for designing simulation experiments. In: Proceedings of the 2006 winter simulation conference, Monterey

    Google Scholar 

  • Sarin R, Vahid F (1999) Payoff assessments without probabilities: a simple dynamic model of choice. Games Econ Behav 28:294–309

    Article  Google Scholar 

  • Saltelli A, Chan K, Scott EM (2000) Sensitivity analysis. Wiley, New York

    Google Scholar 

  • Saltelli A, Tarantola S, Campolongo F, Ratto M (2004) Sensitivity analysis in practice. Wiley, Chichester

    Google Scholar 

  • Schmolke A, Thorbek P, DeAngelis DL, Grimm V (2010) Ecological modeling supporting environmental decision making: a strategy for the future. Trends Ecol Evol 25:479–486

    Article  Google Scholar 

  • Siebertz K, Bebber Dv, Hochkirchen T (2010) Statistische Versuchsplanung: Design of Experiments (DoE). Springer, Heidelberg

    Book  Google Scholar 

  • Simon H (1973) The organization of complex systems. In: Pattee HH (ed) Hierarchy theory: the challenge of complex systems. Braziller, New York, pp 1–27

    Google Scholar 

  • Taber CS, Timpone RJ (1996) Computational modeling. SAGE, Thousand Oaks

    Google Scholar 

  • Trocine L, Malone LC (2001) An overview of newer, advanced screening methods for the initial phase in an experimental design. In: Proceedings of the 2001 winter simulation conference, Arlington

    Google Scholar 

  • Wu CFJ, Hamada M (2000) Experiments: planning, analysis, and parameter design optimization. Wiley, New York

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Institute of Management Control and Accounting, Hamburg University of Technology, Hamburg, Germany

    Iris Lorscheid, Bernd-Oliver Heine & Matthias Meyer

Authors
  1. Iris Lorscheid
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Bernd-Oliver Heine
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Matthias Meyer
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Iris Lorscheid.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Lorscheid, I., Heine, BO. & Meyer, M. Opening the ‘black box’ of simulations: increased transparency and effective communication through the systematic design of experiments. Comput Math Organ Theory 18, 22–62 (2012). https://doi.org/10.1007/s10588-011-9097-3

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10588-011-9097-3

Keywords

Search

Navigation