Skip to main content

2017 | OriginalPaper | Buchkapitel

Experimentation that Matters: A Multi-case Study on the Challenges with A/B Testing

verfasst von : Helena Holmström Olsson, Jan Bosch, Aleksander Fabijan

Erschienen in: Software Business

Verlag: Springer International Publishing

Aktivieren Sie unsere intelligente Suche um passende Fachinhalte oder Patente zu finden.

search-config
loading …

Abstract

From having been exclusive for companies in the online domain, feature experiments are becoming increasingly important for software-intensive companies also in other domains. Today, companies run experiments, such as e.g. A/B tests, to optimize product performance and to learn about user behaviors, as well as to guide product development and innovation. However, although experimentation with customers has become an effective mechanism to improve products and increase revenue, companies struggle with how to leverage the results of the experiments they run. In this paper, we study the reasons for this and we identify three key challenges that make feature experimentation a difficult task. Our research reveals the following challenges: (1) the impact of experiments doesn’t scale, (2) business KPIs and team level metrics are not aligned and (3) it is unclear if the available solutions are applicable across domains.

Sie haben noch keine Lizenz? Dann Informieren Sie sich jetzt über unsere Produkte:

Springer Professional "Wirtschaft+Technik"

Online-Abonnement

Mit Springer Professional "Wirtschaft+Technik" erhalten Sie Zugriff auf:

  • über 102.000 Bücher
  • über 537 Zeitschriften

aus folgenden Fachgebieten:

  • Automobil + Motoren
  • Bauwesen + Immobilien
  • Business IT + Informatik
  • Elektrotechnik + Elektronik
  • Energie + Nachhaltigkeit
  • Finance + Banking
  • Management + Führung
  • Marketing + Vertrieb
  • Maschinenbau + Werkstoffe
  • Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

Springer Professional "Technik"

Online-Abonnement

Mit Springer Professional "Technik" erhalten Sie Zugriff auf:

  • über 67.000 Bücher
  • über 390 Zeitschriften

aus folgenden Fachgebieten:

  • Automobil + Motoren
  • Bauwesen + Immobilien
  • Business IT + Informatik
  • Elektrotechnik + Elektronik
  • Energie + Nachhaltigkeit
  • Maschinenbau + Werkstoffe




 

Jetzt Wissensvorsprung sichern!

Springer Professional "Wirtschaft"

Online-Abonnement

Mit Springer Professional "Wirtschaft" erhalten Sie Zugriff auf:

  • über 67.000 Bücher
  • über 340 Zeitschriften

aus folgenden Fachgebieten:

  • Bauwesen + Immobilien
  • Business IT + Informatik
  • Finance + Banking
  • Management + Führung
  • Marketing + Vertrieb
  • Versicherung + Risiko




Jetzt Wissensvorsprung sichern!

Literatur
1.
Zurück zum Zitat Olsson, H.H., Bosch, J.: Towards data-driven product development: a multiple case study on post-deployment data usage in software-intensive embedded systems. In: Fitzgerald, B., Conboy, K., Power, K., Valerdi, R., Morgan, L., Stol, K.-J. (eds.) LESS 2013. LNBIP, vol. 167, pp. 152–164. Springer, Heidelberg (2013). doi:10.1007/978-3-642-44930-7_10 CrossRef Olsson, H.H., Bosch, J.: Towards data-driven product development: a multiple case study on post-deployment data usage in software-intensive embedded systems. In: Fitzgerald, B., Conboy, K., Power, K., Valerdi, R., Morgan, L., Stol, K.-J. (eds.) LESS 2013. LNBIP, vol. 167, pp. 152–164. Springer, Heidelberg (2013). doi:10.​1007/​978-3-642-44930-7_​10 CrossRef
2.
Zurück zum Zitat Olsson, H.H., Bosch, J.: From opinions to data-driven software R&D: a multi-case study on how to close the ‘Open Loop’ problem. In: Proceedings of EUROMICRO, Software Engineering and Advanced Applications (SEAA), 27–29 August, Verona, Italy (2014) Olsson, H.H., Bosch, J.: From opinions to data-driven software R&D: a multi-case study on how to close the ‘Open Loop’ problem. In: Proceedings of EUROMICRO, Software Engineering and Advanced Applications (SEAA), 27–29 August, Verona, Italy (2014)
3.
Zurück zum Zitat Olsson, H.H., Bosch, J.: Towards evidence-based development: learnings from embedded systems, online games and internet of things. To appear in IEEE Software (forthcoming) Olsson, H.H., Bosch, J.: Towards evidence-based development: learnings from embedded systems, online games and internet of things. To appear in IEEE Software (forthcoming)
4.
Zurück zum Zitat Bosch, J., Eklund, U.: Eternal embedded software: towards innovation experiment systems. In: Margaria, T., Steffen, B. (eds.) ISoLA 2012. LNCS, vol. 7609, pp. 19–31. Springer, Heidelberg (2012). doi:10.1007/978-3-642-34026-0_3 CrossRef Bosch, J., Eklund, U.: Eternal embedded software: towards innovation experiment systems. In: Margaria, T., Steffen, B. (eds.) ISoLA 2012. LNCS, vol. 7609, pp. 19–31. Springer, Heidelberg (2012). doi:10.​1007/​978-3-642-34026-0_​3 CrossRef
5.
6.
Zurück zum Zitat Kohavi, R., Longbotham, R.: Online controlled experiments and A/B tests. In: Encyclopedia of Machine Learning and Data Mining, no. Ries 2011, pp. 1–11 (2015) Kohavi, R., Longbotham, R.: Online controlled experiments and A/B tests. In: Encyclopedia of Machine Learning and Data Mining, no. Ries 2011, pp. 1–11 (2015)
7.
Zurück zum Zitat Fagerholm, F., Guinea, A.F., Mäenpää, H., Münch, J.: Building blocks for continuous experimentation. In: Proceedings of the 1st International Workshop on Rapid Continuous Software Engineering (RCoSE), pp. 26–35 (2014) Fagerholm, F., Guinea, A.F., Mäenpää, H., Münch, J.: Building blocks for continuous experimentation. In: Proceedings of the 1st International Workshop on Rapid Continuous Software Engineering (RCoSE), pp. 26–35 (2014)
8.
Zurück zum Zitat Fabijan, A., Dmitriev, P., Olsson, H.H., Bosch J.: The evolution of continuous experimentation in software product development: from data to a data-driven organization at scale. In: Proceedings of the 39th International Conference on Software Engineering (ICSE), 20–28th May, Buenos Aires, Argentina (2017) Fabijan, A., Dmitriev, P., Olsson, H.H., Bosch J.: The evolution of continuous experimentation in software product development: from data to a data-driven organization at scale. In: Proceedings of the 39th International Conference on Software Engineering (ICSE), 20–28th May, Buenos Aires, Argentina (2017)
9.
Zurück zum Zitat Olsson, H.H., Bosch, J.: So Much Data; So Little Value A multi-case study on improving the impact of data-driven development practices. In Proceedings of the Ibero American Conference on Software Engineering (ClbSE), 22nd–23rd May, Buenos Aires, Argentina (2017) Olsson, H.H., Bosch, J.: So Much Data; So Little Value A multi-case study on improving the impact of data-driven development practices. In Proceedings of the Ibero American Conference on Software Engineering (ClbSE), 22nd–23rd May, Buenos Aires, Argentina (2017)
10.
Zurück zum Zitat Dmitriev, P., Frasca, B., Gupta, S., Kohavi, R., Vaz, G. (forthcoming). Pitfalls of Long-Term Online Controlled Experiments. (To appear in IEEE Big Data) Dmitriev, P., Frasca, B., Gupta, S., Kohavi, R., Vaz, G. (forthcoming). Pitfalls of Long-Term Online Controlled Experiments. (To appear in IEEE Big Data)
11.
Zurück zum Zitat Fagerholm, F., et al.: The RIGHT model for continuous experimentation. J. Syst. Softw. 123, 292–305 (2016) Fagerholm, F., et al.: The RIGHT model for continuous experimentation. J. Syst. Softw. 123, 292–305 (2016)
12.
Zurück zum Zitat Bosch, J.: Future trends in software engineering. IEEE Softw. 33(1), 82–88 (2016) Bosch, J.: Future trends in software engineering. IEEE Softw. 33(1), 82–88 (2016)
13.
Zurück zum Zitat Pohl, K.: Requirements Engineering: Fundamentals, Principles, and Techniques. Springer Publishing Company, Incorporated (2010)CrossRef Pohl, K.: Requirements Engineering: Fundamentals, Principles, and Techniques. Springer Publishing Company, Incorporated (2010)CrossRef
14.
Zurück zum Zitat Ries, E.: The Lean Startup: How Today’s Entrepreneurs Use Continuous Innovation to Create Radically Successful Businesses. Crown Business, New York (2011) Ries, E.: The Lean Startup: How Today’s Entrepreneurs Use Continuous Innovation to Create Radically Successful Businesses. Crown Business, New York (2011)
15.
Zurück zum Zitat Fabijan, A., Olsson, H.H., Bosch, J.: Time to say ‘Good Bye’: feature lifecycle. In: 42nd Euromicro Conference on Software Engineering and Advanced Applications (SEAA), Limassol, Cyprus. 31 August–2 September, pp. 9–16 (2016) Fabijan, A., Olsson, H.H., Bosch, J.: Time to say ‘Good Bye’: feature lifecycle. In: 42nd Euromicro Conference on Software Engineering and Advanced Applications (SEAA), Limassol, Cyprus. 31 August–2 September, pp. 9–16 (2016)
16.
Zurück zum Zitat Kim, M., Zimmermann, T., DeLine, R., Begel, A.: The Emerging Role of Data Scientists on Software Development Teams, no. MSR-TR-2015–30, p. 10 2015 Kim, M., Zimmermann, T., DeLine, R., Begel, A.: The Emerging Role of Data Scientists on Software Development Teams, no. MSR-TR-2015–30, p. 10 2015
17.
Zurück zum Zitat Runesson, P., Höst, M.: Guidelines for conducting and reporting case study research in software engineering. Empirical Softw. Eng. 14, 131 (2009) Runesson, P., Höst, M.: Guidelines for conducting and reporting case study research in software engineering. Empirical Softw. Eng. 14, 131 (2009)
Metadaten
Titel
Experimentation that Matters: A Multi-case Study on the Challenges with A/B Testing
verfasst von
Helena Holmström Olsson
Jan Bosch
Aleksander Fabijan
Copyright-Jahr
2017
DOI
https://doi.org/10.1007/978-3-319-69191-6_12