Skip to main content
Top

2019 | OriginalPaper | Chapter

Predicting and Testing Latencies with Deep Learning: An IoT Case Study

Authors : Bernhard K. Aichernig, Franz Pernkopf, Richard Schumi, Andreas Wurm

Published in: Tests and Proofs

Publisher: Springer International Publishing

Activate our intelligent search to find suitable subject content or patents.

search-config
loading …

Abstract

The Internet of things (IoT) is spreading into the everyday life of millions of people. However, the quality of the underlying communication technologies is still questionable. In this work, we are analysing the performance of an implementation of MQTT, which is a major communication protocol of the IoT. We perform model-based test-case generation to generate log data for training a neural network. This neural network is applied to predict latencies depending on different features, like the number of active clients. The predictions are integrated into our initial functional model, and we exploit the resulting timed model for statistical model checking. This allows us to answer questions about the expected performance for various usage scenarios. The benefit of our approach is that it enables a convenient extension of a functional model with timing aspects using deep learning. A comparison to our previous work with linear regression shows that deep learning needs less manual effort in data preprocessing and provides significantly better predictions.

Dont have a licence yet? Then find out more about our products and how to get one now:

Springer Professional "Wirtschaft+Technik"

Online-Abonnement

Mit Springer Professional "Wirtschaft+Technik" erhalten Sie Zugriff auf:

  • über 102.000 Bücher
  • über 537 Zeitschriften

aus folgenden Fachgebieten:

  • Automobil + Motoren
  • Bauwesen + Immobilien
  • Business IT + Informatik
  • Elektrotechnik + Elektronik
  • Energie + Nachhaltigkeit
  • Finance + Banking
  • Management + Führung
  • Marketing + Vertrieb
  • Maschinenbau + Werkstoffe
  • Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

Springer Professional "Technik"

Online-Abonnement

Mit Springer Professional "Technik" erhalten Sie Zugriff auf:

  • über 67.000 Bücher
  • über 390 Zeitschriften

aus folgenden Fachgebieten:

  • Automobil + Motoren
  • Bauwesen + Immobilien
  • Business IT + Informatik
  • Elektrotechnik + Elektronik
  • Energie + Nachhaltigkeit
  • Maschinenbau + Werkstoffe




 

Jetzt Wissensvorsprung sichern!

Springer Professional "Wirtschaft"

Online-Abonnement

Mit Springer Professional "Wirtschaft" erhalten Sie Zugriff auf:

  • über 67.000 Bücher
  • über 340 Zeitschriften

aus folgenden Fachgebieten:

  • Bauwesen + Immobilien
  • Business IT + Informatik
  • Finance + Banking
  • Management + Führung
  • Marketing + Vertrieb
  • Versicherung + Risiko




Jetzt Wissensvorsprung sichern!

Literature
1.
go back to reference Agha, G., Palmskog, K.: A survey of statistical model checking. ACM Trans. Model. Comput. Simul. (TOMACS) 28(1), 6:1–6:39 (2018)MathSciNetCrossRef Agha, G., Palmskog, K.: A survey of statistical model checking. ACM Trans. Model. Comput. Simul. (TOMACS) 28(1), 6:1–6:39 (2018)MathSciNetCrossRef
2.
go back to reference Aichernig, B.K., et al.: Learning and statistical model checking of system response times. Softw. Qual. J. 27, 757–795 (2019)CrossRef Aichernig, B.K., et al.: Learning and statistical model checking of system response times. Softw. Qual. J. 27, 757–795 (2019)CrossRef
4.
go back to reference Aichernig, B.K., Schumi, R.: Property-based testing with FsCheck by deriving properties from business rule models. In: ICSTW, pp. 219–228. IEEE (2016) Aichernig, B.K., Schumi, R.: Property-based testing with FsCheck by deriving properties from business rule models. In: ICSTW, pp. 219–228. IEEE (2016)
5.
go back to reference Aichernig, B.K., Schumi, R.: Property-based testing of web services by deriving properties from business-rule models. Softw. Syst. Model. 18, 889–911 (2019)CrossRef Aichernig, B.K., Schumi, R.: Property-based testing of web services by deriving properties from business-rule models. Softw. Syst. Model. 18, 889–911 (2019)CrossRef
6.
go back to reference Aichernig, B.K., Schumi, R.: Statistical model checking meets property-based testing. In: ICST, pp. 390–400. IEEE (2017) Aichernig, B.K., Schumi, R.: Statistical model checking meets property-based testing. In: ICST, pp. 390–400. IEEE (2017)
9.
go back to reference Arts, T.: On shrinking randomly generated load tests. In: Erlang 2014, pp. 25–31. ACM (2014) Arts, T.: On shrinking randomly generated load tests. In: Erlang 2014, pp. 25–31. ACM (2014)
10.
go back to reference Ballarini, P., Bertrand, N., Horváth, A., Paolieri, M., Vicario, E.: Transient analysis of networks of stochastic timed automata using stochastic state classes. In: Joshi, K., Siegle, M., Stoelinga, M., D’Argenio, P.R. (eds.) QEST 2013. LNCS, vol. 8054, pp. 355–371. Springer, Heidelberg (2013). https://doi.org/10.1007/978-3-642-40196-1_30CrossRef Ballarini, P., Bertrand, N., Horváth, A., Paolieri, M., Vicario, E.: Transient analysis of networks of stochastic timed automata using stochastic state classes. In: Joshi, K., Siegle, M., Stoelinga, M., D’Argenio, P.R. (eds.) QEST 2013. LNCS, vol. 8054, pp. 355–371. Springer, Heidelberg (2013). https://​doi.​org/​10.​1007/​978-3-642-40196-1_​30CrossRef
11.
go back to reference Banks, A., Gupta, R.: MQTT version 3.1.1. OASIS Standard, December 2014 Banks, A., Gupta, R.: MQTT version 3.1.1. OASIS Standard, December 2014
12.
go back to reference Becker, S., Koziolek, H., Reussner, R.H.: The Palladio component model for model-driven performance prediction. J. Syst. Softw. 82(1), 3–22 (2009)CrossRef Becker, S., Koziolek, H., Reussner, R.H.: The Palladio component model for model-driven performance prediction. J. Syst. Softw. 82(1), 3–22 (2009)CrossRef
13.
go back to reference Bishop, C.M.: Pattern Recognition and Machine Learning. Information Science and Statistics. Springer, New York (2006)MATH Bishop, C.M.: Pattern Recognition and Machine Learning. Information Science and Statistics. Springer, New York (2006)MATH
14.
go back to reference Book, M., Gruhn, V., Hülder, M., Köhler, A., Kriegel, A.: Cost and response time simulation for web-based applications on mobile channels. In: QSIC, pp. 83–90. IEEE (2005) Book, M., Gruhn, V., Hülder, M., Köhler, A., Kriegel, A.: Cost and response time simulation for web-based applications on mobile channels. In: QSIC, pp. 83–90. IEEE (2005)
15.
go back to reference Bulychev, P.E., et al.: UPPAAL-SMC: statistical model checking for priced timed automata. In: QAPL. EPTCS, vol. 85, pp. 1–16. Open Publishing Association (2012) Bulychev, P.E., et al.: UPPAAL-SMC: statistical model checking for priced timed automata. In: QAPL. EPTCS, vol. 85, pp. 1–16. Open Publishing Association (2012)
16.
go back to reference Chen, X., Mohapatra, P., Chen, H.: An admission control scheme for predictable server response time for web accesses. In: WWW, pp. 545–554. ACM (2001) Chen, X., Mohapatra, P., Chen, H.: An admission control scheme for predictable server response time for web accesses. In: WWW, pp. 545–554. ACM (2001)
17.
go back to reference Cho, K., et al.: Learning phrase representations using RNN encoder-decoder for statistical machine translation. arXiv preprint arXiv:1406.1078 (2014) Cho, K., et al.: Learning phrase representations using RNN encoder-decoder for statistical machine translation. arXiv preprint arXiv:​1406.​1078 (2014)
18.
go back to reference Claessen, K., Hughes, J.: QuickCheck: a lightweight tool for random testing of Haskell programs. In: ICFP, pp. 268–279. ACM (2000) Claessen, K., Hughes, J.: QuickCheck: a lightweight tool for random testing of Haskell programs. In: ICFP, pp. 268–279. ACM (2000)
19.
go back to reference Collina, M., Corazza, G.E., Vanelli-Coralli, A.: Introducing the QEST broker: scaling the IoT by bridging MQTT and REST. In: PIMRC, pp. 36–41. IEEE (2012) Collina, M., Corazza, G.E., Vanelli-Coralli, A.: Introducing the QEST broker: scaling the IoT by bridging MQTT and REST. In: PIMRC, pp. 36–41. IEEE (2012)
21.
go back to reference Draheim, D., Grundy, J.C., Hosking, J.G., Lutteroth, C., Weber, G.: Realistic load testing of web applications. In: CSMR, pp. 57–70. IEEE (2006) Draheim, D., Grundy, J.C., Hosking, J.G., Lutteroth, C., Weber, G.: Realistic load testing of web applications. In: CSMR, pp. 57–70. IEEE (2006)
22.
go back to reference Glorot, X., Bengio, Y.: Understanding the difficulty of training deep feedforward neural networks. In: AISTATS. JMLR Proceedings, vol. 9, pp. 249–256. JMLR.org (2010) Glorot, X., Bengio, Y.: Understanding the difficulty of training deep feedforward neural networks. In: AISTATS. JMLR Proceedings, vol. 9, pp. 249–256. JMLR.org (2010)
23.
go back to reference Glorot, X., Bordes, A., Bengio, Y.: Deep sparse rectifier neural networks. In: AISTATS. JMLR Proceedings, vol. 15, pp. 315–323. JMLR.org (2011) Glorot, X., Bordes, A., Bengio, Y.: Deep sparse rectifier neural networks. In: AISTATS. JMLR Proceedings, vol. 15, pp. 315–323. JMLR.org (2011)
24.
go back to reference Goodfellow, I., Bengio, Y., Courville, A.: Deep Learning. MIT Press, Cambridge (2016)MATH Goodfellow, I., Bengio, Y., Courville, A.: Deep Learning. MIT Press, Cambridge (2016)MATH
25.
go back to reference Grinchtein, O.: Learning of Timed Systems. Ph.D. thesis, Uppsala University (2008) Grinchtein, O.: Learning of Timed Systems. Ph.D. thesis, Uppsala University (2008)
26.
go back to reference Hawkins, D.M.: The problem of overfitting. J. Chem. Inf. Model. 44(1), 1–12 (2004)MathSciNet Hawkins, D.M.: The problem of overfitting. J. Chem. Inf. Model. 44(1), 1–12 (2004)MathSciNet
27.
go back to reference Hochreiter, S., Schmidhuber, J.: Long short-term memory. Neural Comput. 9(8), 1735–1780 (1997)CrossRef Hochreiter, S., Schmidhuber, J.: Long short-term memory. Neural Comput. 9(8), 1735–1780 (1997)CrossRef
28.
29.
go back to reference Houimli, M., Kahloul, L., Benaoun, S.: Formal specification, verification and evaluation of the MQTT protocol in the Internet of Things. In: ICMIT, pp. 214–221. IEEE, December 2017 Houimli, M., Kahloul, L., Benaoun, S.: Formal specification, verification and evaluation of the MQTT protocol in the Internet of Things. In: ICMIT, pp. 214–221. IEEE, December 2017
31.
go back to reference Kalaji, A.S., Hierons, R.M., Swift, S.: Generating feasible transition paths for testing from an extended finite state machine. In: ICST, pp. 230–239. IEEE (2009) Kalaji, A.S., Hierons, R.M., Swift, S.: Generating feasible transition paths for testing from an extended finite state machine. In: ICST, pp. 230–239. IEEE (2009)
33.
go back to reference LeCun, Y., Bengio, Y., Hinton, G.: Deep learning. Nature 521(7553), 436–444 (2015)CrossRef LeCun, Y., Bengio, Y., Hinton, G.: Deep learning. Nature 521(7553), 436–444 (2015)CrossRef
34.
go back to reference Lee, S., Kim, H., Hong, D., Ju, H.: Correlation analysis of MQTT loss and delay according to QoS level. In: ICOIN, pp. 714–717. IEEE (2013) Lee, S., Kim, H., Hong, D., Ju, H.: Correlation analysis of MQTT loss and delay according to QoS level. In: ICOIN, pp. 714–717. IEEE (2013)
36.
go back to reference Legay, A., Sedwards, S.: On statistical model checking with PLASMA. In: TASE, pp. 139–145. IEEE (2014) Legay, A., Sedwards, S.: On statistical model checking with PLASMA. In: TASE, pp. 139–145. IEEE (2014)
37.
go back to reference Lu, Y., Nolte, T., Bate, I., Cucu-Grosjean, L.: A statistical response-time analysis of real-time embedded systems. In: RTSS, pp. 351–362. IEEE (2012) Lu, Y., Nolte, T., Bate, I., Cucu-Grosjean, L.: A statistical response-time analysis of real-time embedded systems. In: RTSS, pp. 351–362. IEEE (2012)
38.
39.
go back to reference Papadakis, M., Sagonas, K.: A PropEr integration of types and function specifications with property-based testing. In: Erlang 2011, pp. 39–50. ACM (2011) Papadakis, M., Sagonas, K.: A PropEr integration of types and function specifications with property-based testing. In: Erlang 2011, pp. 39–50. ACM (2011)
40.
go back to reference Pearson, K.: Note on regression and inheritance in the case of two parents. Proc. R. Soc. London 58, 240–242 (1895)CrossRef Pearson, K.: Note on regression and inheritance in the case of two parents. Proc. R. Soc. London 58, 240–242 (1895)CrossRef
41.
go back to reference Tyagi, R.S.: A comparative study of performance testing tools. Int. J. Adv. Res. Comput. Sci. Softw. Eng. IJARCSSE 3(5), 1300–1307 (2013) Tyagi, R.S.: A comparative study of performance testing tools. Int. J. Adv. Res. Comput. Sci. Softw. Eng. IJARCSSE 3(5), 1300–1307 (2013)
42.
go back to reference Schmidt, J., Ghorbani, A., Hapfelmeier, A., Kramer, S.: Learning probabilistic real-time automata from multi-attribute event logs. Intell. Data Anal. 17(1), 93–123 (2013)CrossRef Schmidt, J., Ghorbani, A., Hapfelmeier, A., Kramer, S.: Learning probabilistic real-time automata from multi-attribute event logs. Intell. Data Anal. 17(1), 93–123 (2013)CrossRef
43.
go back to reference Schumi, R.: Predicting and testing system response-times with statistical model checking and property-based testing. Ph.D. thesis, Graz University of Technology (2018) Schumi, R.: Predicting and testing system response-times with statistical model checking and property-based testing. Ph.D. thesis, Graz University of Technology (2018)
44.
45.
go back to reference Tappler, M., Aichernig, B.K., Bloem, R.: Model-based testing IoT communication via active automata learning. In: ICST, pp. 276–287. IEEE (2017) Tappler, M., Aichernig, B.K., Bloem, R.: Model-based testing IoT communication via active automata learning. In: ICST, pp. 276–287. IEEE (2017)
46.
go back to reference Thangavel, D., Ma, X., Valera, A.C., Tan, H., Tan, C.K.: Performance evaluation of MQTT and CoAP via a common middleware. In: ISSNIP, pp. 1–6. IEEE (2014) Thangavel, D., Ma, X., Valera, A.C., Tan, H., Tan, C.K.: Performance evaluation of MQTT and CoAP via a common middleware. In: ISSNIP, pp. 1–6. IEEE (2014)
48.
go back to reference Wald, A.: Sequential Analysis. Courier Corporation, North Chelmsford (1973)MATH Wald, A.: Sequential Analysis. Courier Corporation, North Chelmsford (1973)MATH
49.
go back to reference Wurm, A.: Predicting the latency of MQTT brokers using deep learning. Master’s thesis, Graz University of Technology (2018) Wurm, A.: Predicting the latency of MQTT brokers using deep learning. Master’s thesis, Graz University of Technology (2018)
Metadata
Title
Predicting and Testing Latencies with Deep Learning: An IoT Case Study
Authors
Bernhard K. Aichernig
Franz Pernkopf
Richard Schumi
Andreas Wurm
Copyright Year
2019
DOI
https://doi.org/10.1007/978-3-030-31157-5_7

Premium Partner