nach oben

Erschienen in:

2017 | OriginalPaper | Buchkapitel

9. Verifying and Validating Simulations

verfasst von : Nuno David, Nuno Fachada, Agostinho C. Rosa

Erschienen in: Simulating Social Complexity

Verlag: Springer International Publishing

Einloggen

Aktivieren Sie unsere intelligente Suche, um passende Fachinhalte oder Patente zu finden.

search-config

KI-gestützte Suche

Aus

Abstract

Verification and validation are two important aspects of model building. Verification and validation compare models with observations and descriptions of the problem modelled, which may include other models that have been verified and validated to some level. However, the use of simulation for modelling social complexity is very diverse. Often, verification and validation do not refer to an explicit stage in the simulation development process, but to the modelling process itself, according to good practices and in a way that grants credibility to using the simulation for a specific purpose. One cannot consider verification and validation without considering the purpose of the simulation. This chapter deals with a comprehensive outline of methodological perspectives and practical uses of verification and validation. The problem of evaluating simulations is addressed in four main topics: (1) the meaning of the terms verification and validation in the context of simulating social complexity; (2) types of validation, as well as techniques for validating simulations; (3) model replication and comparison as cornerstones of verification and validation; and (4) the relationship of various validation types and techniques with different modelling strategies.

Sie haben noch keine Lizenz? Dann Informieren Sie sich jetzt über unsere Produkte:

Springer Professional "Wirtschaft+Technik"

Online-Abonnement

Mit Springer Professional "Wirtschaft+Technik" erhalten Sie Zugriff auf:

über 102.000 Bücher
über 537 Zeitschriften

aus folgenden Fachgebieten:

Automobil + Motoren
Bauwesen + Immobilien
Business IT + Informatik
Elektrotechnik + Elektronik
Energie + Nachhaltigkeit
Finance + Banking
Management + Führung
Marketing + Vertrieb
Maschinenbau + Werkstoffe
Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

Jetzt informieren

Springer Professional "Technik"

Online-Abonnement

Mit Springer Professional "Technik" erhalten Sie Zugriff auf:

über 67.000 Bücher
über 390 Zeitschriften

aus folgenden Fachgebieten:

Automobil + Motoren
Bauwesen + Immobilien
Business IT + Informatik
Elektrotechnik + Elektronik
Energie + Nachhaltigkeit
Maschinenbau + Werkstoffe

Jetzt Wissensvorsprung sichern!

Jetzt informieren

Springer Professional "Wirtschaft"

Online-Abonnement

Mit Springer Professional "Wirtschaft" erhalten Sie Zugriff auf:

über 67.000 Bücher
über 340 Zeitschriften

aus folgenden Fachgebieten:

Bauwesen + Immobilien
Business IT + Informatik
Finance + Banking
Management + Führung
Marketing + Vertrieb
Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

Jetzt informieren

Vorheriges Kapitel The Importance of Ontological Structure: Why Validation by ‘Fit-to-Data’ Is Insufficient

Nächstes Kapitel Understanding Simulation Results

Numerical simulation refers to simulation for finding solutions to mathematical models, normally for cases in which mathematics does not provide analytical solutions. Technical simulation stands for simulation with numerical models in computational sciences and engineering.

Verification in the left quadrant of Fig. 9.1 is sometimes known as “internal validation.”

Alberts, S., Keenan, M. K., D’Souza, R. M., & An, G. (2012). Data-parallel techniques for simulating a mega-scale agent-based model of systemic inflammatory response syndrome on graphics processing units. Simulation, 88(8), 895–907. doi:10.1177/0037549711425180, http://journals.sagepub.com/doi/abs/10.1177/0037549711425180 CrossRef

Altman, M., Borgman, C., Crosas, M., & Matone, M. (2015). An introduction to the joint principles for data citation. Bulletin of the Association for Information Science and Technology, 41(3), 43–45. doi:10.1002/bult.2015.1720410313, http://onlinelibrary.wiley.com/doi/10.1002/bult.2015.1720410313/abstract CrossRef

Amblard, F., Bommel, P., & Rouchier, J. (2007). Assessment and validation of multi-agent models. In Agent-based modelling and simulation in the social and human sciences (pp. 93–116). Oxford: Bardwell Press. http://agritrop.cirad.fr/541339/

Amorim, R. C., Castro, J. A., Silva, J. Rd., & Ribeiro, C. (2015). A comparative study of platforms for research data management: Interoperability, metadata capabilities and integration potential. In New contributions in information systems and technologies (pp. 101–111). Cham: Springer. doi:10.1007/978-3-319-16486-1_10, https://link.springer.com/chapter/10.1007/978-3-319-16486-1_10 CrossRef

Arai, R., & Watanabe, S. (2008). A quantitative method for comparing multi-agent-based simulations in feature space. In Multi-agent-based simulation IX (pp.154–166). Berlin/Heidelberg: Springer. doi:10.1007/978-3-642-01991-3_12, https://link.springer.com/chapter/10.1007/978-3-642-01991-3_12

Assante, M., Candela, L., Castelli, D., & Tani, A. (2016). Are scientific data repositories coping with research data publishing? Data Science Journal, 15, 6. doi:10.5334/dsj-2016-006, http://datascience.codata.org/articles/10.5334/dsj-2016-006/ CrossRef

Axelrod, R. (1993). A Model of the Emergence of New Political Actors. Working paper 93-11-068, Santa Fe Institute. https://www.santafe.edu/research/results/working-papers/a-model-of-the-emergence-of-new-political-actors

Axelrod, R. (1997a). Advancing the art of simulation in the social sciences. In D. R. Conte, P. D. R. Hegselmann, & P. D. P. Terna (Eds.), Simulating Social Phenomena. Lecture notes in economics and mathematical systems (Vol. 456, pp. 21–40). Berlin/Heidelberg: Springer. doi:10.1007/978-3-662-03366-1_2, http://link.springer.com/chapter/10.1007/978-3-662-03366-1_2

Axelrod, R. (1997b). The dissemination of culture: A model with local convergence and global polarization. Journal of Conflict Resolution, 41(2), 203–226. doi:10.1177/0022002797041002001, http://dx.doi.org/10.1177/0022002797041002001

Axtell, R., Axelrod, R., Epstein, J. M., & Cohen, M. D. (1996). Aligning simulation models: A case study and results. Computational & Mathematical Organization Theory, 1(2), 123–141. doi:10.1007/BF01299065, https://link.springer.com/article/10.1007/BF01299065 CrossRef

Balci, O., & Sargent, R. G. (1984). Validation of simulation models via simultaneous confidence intervals. American Journal of Mathematical and Management Sciences, 4(3–4), 375–406. doi:10.1080/01966324.1984.10737151, http://dx.doi.org/10.1080/01966324.1984.10737151 CrossRefMATH

Barreteau, O., Bots, P., Daniell, K., Etienne, M., Perez, P., Barnaud, C., et al. (2017). Participatory approaches. doi:https://doi.org/10.1007/978-3-319-66948-9_12.

Barreteau, O., Bots, P., Daniell, K., Etienne, M., Perez, P., Barnaud, C., et al. (2017). Participatory approaches. In B. Edmonds & R. Meyer (Eds.), Simulating social complexity. Understanding complex systems (2nd ed.). Berlin/Heidelberg: Springer. doi:10.1007/978-3-319-66948-9_12

Boero, R., & Squazzoni, F. (2005). Does empirical embeddedness matter? Methodological issues on agent-based models for analytical social science. Journal of Artificial Societies and Social Simulation, 8(4), 6. http://jasss.soc.surrey.ac.uk/8/4/6.html

Calvez, B., & Hutzler, G. (2005). Automatic tuning of agent-based models using genetic algorithms. In Multi-agent-based simulation VI (pp. 41–57). Berlin/Heidelberg: Springer. doi:10.1007/11734680_4, https://link.springer.com/chapter/10.1007/11734680_4

Collier, N., & North, M. (2013). Parallel agent-based simulation with repast for high performance computing. Simulation, 89(10), 1215–1235. doi:10.1177/0037549712462620, http://journals.sagepub.com/doi/abs/10.1177/0037549712462620 CrossRef

David, N. (2009). Validation and verification in social simulation: patterns and clarification of terminology. In Epistemological aspects of computer simulation in the social sciences (pp. 117–129). Berlin/Heidelberg: Springer. doi:10.1007/978-3-642-01109-2_9, https://link.springer.com/chapter/10.1007/978-3-642-01109-2_9 CrossRef

David, N., Marietto, M. B., Sichman, J. S., & Coelho, H. (2004). The structure and logic of interdisciplinary research in agent-based social simulation. Journal of Artificial Societies and Social Simulation, 7(3), 4. http://jasss.soc.surrey.ac.uk/7/3/4.html

David, N., Sichman, J. S., & Coelho, H. (2005). The logic of the method of agent-based simulation in the social sciences: Empirical and intentional adequacy of computer programs. Journal of Artificial Societies and Social Simulation, 8(4), 2. http://jasss.soc.surrey.ac.uk/8/4/2.html

David, N., Caldas, J. C., & Coelho, H. (2010). Epistemological perspectives on simulation III. Journal of Artificial Societies and Social Simulation, 13(1). doi:10.18564/jasss.1591, http://jasss.soc.surrey.ac.uk/13/1/14.html

Dean, J. S., Gumerman, G. J., Epstein, J. M., Axtell, R. L., Swedlund, A. C., Parker, M. T., et al. (2000). Understanding Anasazi culture change through agent-based modeling. In T. A. Kohler & G. J. Gumerman (Eds.), Dynamics in human and primate societies: Agent-based modeling of social and spatial processes. Santa fe institute studies on the sciences of complexity (pp. 179–205). New York/Oxford: Oxford University Press.

Densmore, O. (2016). AgentScript. http://agentscript.org/

Edmonds, B., & Hales, D. (2003). Replication, replication and replication: Some hard lessons from model alignment. Journal of Artificial Societies and Social Simulation, 6(4), 11. http://jasss.soc.surrey.ac.uk/6/4/11.html

Edmonds, B. & Moss, S. (2005). From KISS to KIDS—an ‘anti-simplistic’ modelling approach. In: P. Davidsson, B. Logan, & K. Takadama (Eds.), Multi-agent and multi-agent-based simulation (Vol. 3415, pp. 130–144). Berlin/Heidelberg: Springer. doi:10.1007/978-3-540-32243-6_11. http://link.springer.com/10.1007/978-3-540-32243-6_11 CrossRef

Epstein, J., & Axtell, R. (1996). Growing artificial societies: Social science from the bottom up. Washington, DC: Brookings Institution Press; Cambridge, MA: MIT Press.

Evans, A., Heppenstall, A., & Birkin, M. (2017). Understanding simulation results. doi: https://doi.org/10.1007/978-3-319-66948-9_10.

Fachada, N., Lopes, V. V., Martins, R. C., & Rosa, A. C. (2015). Towards a standard model for research in agent-based modeling and simulation. PeerJ Computer Science, 1, e36. doi:10.7717/peerj-cs.36, https://peerj.com/articles/cs-36 CrossRef

Fachada, N., Rodrigues, J., Lopes, V. V., Martins, R. C., & Rosa, A. C. (2016). micompr: An R package for multivariate independent comparison of observations. The R Journal, 8(2), 405–420. http://journal.r-project.org/archive/2016-2/fachada-rodrigues-lopes-etal.pdf

Fachada, N., Lopes, V. V., Martins, R. C., & Rosa, A. C. (2017a). Parallelization strategies for spatial agent-based models. International Journal of Parallel Programming, 45(3), 449–481.

Fachada, N., Lopes, V. V., Martins, R. C., & Rosa, A. C. (2017b). Model-independent comparison of simulation output. Simulation Modelling Practice and Theory, 72, 131–149. doi:10.1016/j.simpat.2016.12.013, http://www.sciencedirect.com/science/article/pii/S1569190X16302854

Frank, U., & Troitzsch, K. G. (2005). Epistemological perspectives on simulation. Journal of Artificial Societies and Social Simulation, 8(4), 7. http://jasss.soc.surrey.ac.uk/8/4/7.html

Galán, J. M., Izquierdo, L. R., Izquierdo, S. S., Santos, J. I., Olmo, Rd., & López-Paredes, A. (2017). Checking simulations: Detecting and avoiding errors and artefacts. doi: https://doi.org/10.1007/978-3-319-66948-9_9.

Gilbert, N. (2008). Agent-based models. Thousand Oaks, CA: SAGE. google-Books-ID: Z3cp0ZBK9UsC.

Grimm, V., Polhill, G., & Touza, J. (2017). Documenting social simulation models: The ODD protocol as a standard. doi: https://doi.org/10.1007/978-3-319-66948-9_10.

Gross, D., & Strand, R. (2000). Can agent-based models assist decisions on large-scale practical problems? A philosophical analysis. Complexity, 5(6), 26–33. doi:10.1002/1099-0526(200007/08)5:6¡26::AID-CPLX6¿3.0.CO;2-G, http://onlinelibrary.wiley.com/doi/10.1002/1099-0526(200007/08)5:6<26::AID-CPLX6>3.0.CO;2-G/abstract CrossRef

Kratz, J., & Strasser, C. (2014). Data publication consensus and controversies. F1000Research, 3, 94. doi:10.12688/f1000research.3979.3, http://f1000research.com/articles/3-94/v3

Laird, J. E. (2012). The soar cognitive architecture. Cambridge: MIT Press.

Law, A. M. (2015). Simulation modeling and analysis (5th ed.). New York: McGraw Hill Higher Education.

Lee, J. S., Filatova, T., Ligmann-Zielinska, A., Hassani-Mahmooei, B., Stonedahl, F., Lorscheid, I., et al. (2015). The complexities of agent-based modeling output analysis. Journal of Artificial Societies and Social Simulation, 18(4), 4.CrossRef

McKay, M. D., Beckman, R. J., & Conover, W. J. (1979). Comparison of three methods for selecting values of input variables in the analysis of output from a computer code. Technometrics, 21(2), 239–245. doi:10.1080/00401706.1979.10489755, http://dx.doi.org/10.1080/00401706.1979.10489755 MathSciNetMATH

Merlone, U., Sonnessa, M., & Terna, P. (2008). Horizontal and vertical multiple implementations in a model of industrial districts. Journal of Artificial Societies and Social Simulation 11(2), 5. http://jasss.soc.surrey.ac.uk/11/2/5.html

Miller, J. H. (1998). Active nonlinear tests (ANTs) of complex simulation models. Management Science, 44(6), 820–830. doi:10.1287/mnsc.44.6.820, http://pubsonline.informs.org/doi/abs/10.1287/mnsc.44.6.820 CrossRefMATH

Miodownik, D., Cartrite, B., & Bhavnani, R. (2010). Between replication and docking: “adaptive agents, political institutions, and civic traditions” revisited. Journal of Artificial Societies and Social Simulation, 13(3), 1.CrossRef

Müller, B., Bohn, F., Dreßler, G., Groeneveld, J., Klassert, C., Martin, R., et al. (2013), Describing human decisions in agent-based models – ODD + D, an extension of the ODD protocol. Environmental Modelling & Software, 48, 37–48. doi:10.1016/j.envsoft.2013.06.003, http://www.sciencedirect.com/science/article/pii/S1364815213001394 CrossRef

Müller, B., Balbi, S., Buchmann, C. M., de Sousa, L., Dressler, G., Groeneveld, J., et al. (2014). Standardised and transparent model descriptions for agent-based models: Current status and prospects. Environmental Modelling & Software, 55, 156–163. doi:10.1016/j.envsoft.2014.01.029, http://www.sciencedirect.com/science/article/pii/S1364815214000395 CrossRef

Montgomery, D. C. (2012). Design and analysis of experiments (8th ed.). Hoboken: Wiley.

Moss, S., & Edmonds, B. (2005). Sociology and simulation: Statistical and qualitative cross-validation. American Journal of Sociology, 110(4), 1095–1131. doi:10.1086/427320, http://www.journals.uchicago.edu/doi/abs/10.1086/427320 CrossRef

North, M. J., Collier, N. T., Ozik, J., Tatara, E. R., Macal, C. M., Bragen, M., et al. (2013). Complex adaptive systems modeling with Repast Simphony. Complex Adaptive Systems Modeling, 1(1), 3. doi:10.1186/2194-3206-1-3, http://casmodeling.springeropen.com/articles/10.1186/2194-3206-1-3 CrossRef

Peng, R. D. (2011). Reproducible research in computational science. Science, 334(6060), 1226–1227. doi:10.1126/science.1213847, http://science.sciencemag.org/content/334/6060/1226 CrossRef

Pereda, M., Santos, J. I., & Galan, J. M. (2015). A brief introduction to the use of machine learning techniques in the analysis of agent-based models. SSRN Scholarly Paper ID 2689676. Rochester, NY: Social Science Research Network. https://papers.ssrn.com/abstract=2689676

R Core Team. (2017). R: A language and environment for statistical computing. https://www.R-project.org/

Radax, W., & Rengs, B. (2009). Prospects and pitfalls of statistical testing: Insights from replicating the demographic prisoner’s dilemma. Journal of Artificial Societies and Social Simulation, 13(4), 1.CrossRef

Rollins, N. D., Barton, C. M., Bergin, S., Janssen, M. A., & Lee, A. (2014). A Computational Model Library for publishing model documentation and code. Environmental Modelling & Software, 61, 59–64. doi:10.1016/j.envsoft.2014.06.022, http://www.sciencedirect.com/science/article/pii/S1364815214001959 CrossRef

Rouchier, J., Cioffi-Revilla, C., Polhill, J. G., & Takadama, K. (2008). Progress in model-to-model analysis. Journal of Artificial Societies and Social Simulation, 11(2), 8. http://jasss.soc.surrey.ac.uk/11/2/8.html

Sargent, R. G. (2013). Verification and validation of simulation models. Journal of Simulation, 7(1), 12–24. doi:10.1057/jos.2012.20, https://link.springer.com/article/10.1057/jos.2012.20 CrossRef

Schelling, T. C. (1971). Dynamic models of segregation. The Journal of Mathematical Sociology, 1(2), 143–186. doi:10.1080/0022250X.1971.9989794, http://dx.doi.org/10.1080/0022250X.1971.9989794 CrossRefMATH

Squazzoni, F. (Ed.). (2009). Epistemological aspects of computer simulation in the social sciences. Lecture notes in computer science (Vol. 5466). Berlin/Heidelberg: Springer. doi:10.1007/978-3-642-01109-2, http://link.springer.com/10.1007/978-3-642-01109-2

Stonedahl, F., & Wilensky, U. (2010). Finding forms of flocking: Evolutionary search in ABM parameter-spaces. In Multi-agent-based simulation XI (pp. 61–75). Berlin/Heidelberg: Springer. doi:10.1007/978-3-642-18345-4_5, https://link.springer.com/chapter/10.1007/978-3-642-18345-4_5

Sun, Z., Lorscheid, I., Millington, J. D., Lauf, S., Magliocca, N. R., Groeneveld, J., et al. (2016) Simple or complicated agent-based models? A complicated issue. Environmental Modelling & Software, 86, 56–67. doi:10.1016/j.envsoft.2016.09.006, http://www.sciencedirect.com/science/article/pii/S1364815216306041 CrossRef

Takadama, K., Suematsu, Y. L., Sugimoto, N., Nawa, N. E., & Shimohara, K. (2003). Cross-element validation in multiagent-based simulation: Switching learning mechanisms in agents. Journal of Artificial Societies and Social Simulation, 6(4), 6. http://jasss.soc.surrey.ac.uk/6/4/6.html

Thiele, J. C., & Grimm, V. (2015). Replicating and breaking models: Good for you and good for ecology. Oikos, 124(6), 691–696. doi:10.1111/oik.02170, http://onlinelibrary.wiley.com/doi/10.1111/oik.02170/abstract CrossRef

Troitzsch, K. G. (2004). Validating simulation models. In G. Horton (Ed.), Proceedings of 18th European Simulation Multiconference, ESM 2004 (pp. 265–270). Magdeburg: SCS Publishing House.

Wiersma, W. (2015). AgentBase: Agent based modelling in the browser. http://wybowiersma.net/pub/papers/Wiersma,Wybo,AgentBase_agent_based_modelling_in_the_browser.pdf

Wilensky, U. (1999). NetLogo. http://ccl.northwestern.edu/netlogo/

Wilensky, U., & Rand, W. (2007). Making models match: Replicating an agent-based model. Journal of Artificial Societies and Social Simulation, 10(4), 2. http://jasss.soc.surrey.ac.uk/10/4/2.html

Will, O., & Hegselmann, R. (2008). A replication that failed – On the computational model in ‘Michael W. Macy and Yoshimichi Sato: Trust, cooperation and market formation in the U.S. and Japan. Proceedings of the national academy of sciences, May 2002’. Journal of Artificial Societies and Social Simulation, 11(3), 3. http://jasss.soc.surrey.ac.uk/11/3/3.html

Titel: Verifying and Validating Simulations
verfasst von: Nuno David
Nuno Fachada
Agostinho C. Rosa
Verlag: Springer International Publishing
Buch: Simulating Social Complexity
Print ISBN: 978-3-319-66947-2

Electronic ISBN: 978-3-319-66948-9

Copyright-Jahr: 2017
DOI: https://doi.org/10.1007/978-3-319-66948-9_9

Springer Professional

Abstract

Bitte loggen Sie sich ein, um Zugang zu Ihrer Lizenz zu erhalten.

Sie haben noch keine Lizenz? Dann Informieren Sie sich jetzt über unsere Produkte:

Springer Professional "Wirtschaft+Technik"

Springer Professional "Technik"

Springer Professional "Wirtschaft"

Premium Partner