nach oben

Erschienen in:

2020 | OriginalPaper | Buchkapitel

Parallel Stream Processing with MPI for Video Analytics and Data Visualization

verfasst von : Adriano Vogel, Cassiano Rista, Gabriel Justo, Endrius Ewald, Dalvan Griebler, Gabriele Mencagli, Luiz Gustavo Fernandes

Erschienen in: High Performance Computing Systems

Verlag: Springer International Publishing

Einloggen

Aktivieren Sie unsere intelligente Suche, um passende Fachinhalte oder Patente zu finden.

search-config

KI-gestützte Suche

Aus

Abstract

The amount of data generated is increasing exponentially. However, processing data and producing fast results is a technological challenge. Parallel stream processing can be implemented for handling high frequency and big data flows. The MPI parallel programming model offers low-level and flexible mechanisms for dealing with distributed architectures such as clusters. This paper aims to use it to accelerate video analytics and data visualization applications so that insight can be obtained as soon as the data arrives. Experiments were conducted with a Domain-Specific Language for Geospatial Data Visualization and a Person Recognizer video application. We applied the same stream parallelism strategy and two task distribution strategies. The dynamic task distribution achieved better performance than the static distribution in the HPC cluster. The data visualization achieved lower throughput with respect to the video analytics due to the I/O intensive operations. Also, the MPI programming model shows promising performance outcomes for stream processing applications.

Sie haben noch keine Lizenz? Dann Informieren Sie sich jetzt über unsere Produkte:

Springer Professional "Wirtschaft+Technik"

Online-Abonnement

Mit Springer Professional "Wirtschaft+Technik" erhalten Sie Zugriff auf:

über 102.000 Bücher
über 537 Zeitschriften

aus folgenden Fachgebieten:

Automobil + Motoren
Bauwesen + Immobilien
Business IT + Informatik
Elektrotechnik + Elektronik
Energie + Nachhaltigkeit
Finance + Banking
Management + Führung
Marketing + Vertrieb
Maschinenbau + Werkstoffe
Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

Jetzt informieren

Springer Professional "Technik"

Online-Abonnement

Mit Springer Professional "Technik" erhalten Sie Zugriff auf:

über 67.000 Bücher
über 390 Zeitschriften

aus folgenden Fachgebieten:

Automobil + Motoren
Bauwesen + Immobilien
Business IT + Informatik
Elektrotechnik + Elektronik
Energie + Nachhaltigkeit
Maschinenbau + Werkstoffe

Jetzt Wissensvorsprung sichern!

Jetzt informieren

Springer Professional "Wirtschaft"

Online-Abonnement

Mit Springer Professional "Wirtschaft" erhalten Sie Zugriff auf:

über 67.000 Bücher
über 340 Zeitschriften

aus folgenden Fachgebieten:

Bauwesen + Immobilien
Business IT + Informatik
Finance + Banking
Management + Führung
Marketing + Vertrieb
Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

Jetzt informieren

Vorheriges Kapitel Video7 Extended Architecture: Project Design and Statistical Analysis

Nächstes Kapitel Tangible Assets to Improve Research Quality: A Meta Analysis Case Study

SPar’s home page: https://gmap.pucrs.br/spar.

Aldinucci, M., Danelutto, M., Kilpatrick, P., Torquati, M.: Fastflow: High-level and Efficient Streaming on Multicore, Chap. 13, pp. 261–280. Wiley-Blackwell, Hoboken (2014)

Andrade, H., Gedik, B., Turaga, D.: Fundamentals of Stream Processing: Application Design, Systems, and Analytics. Cambridge University Press, Cambridge (2014)CrossRef

Ayachit, U.: The ParaView Guide: A Parallel Visualization Application. Kitware Inc., New York (2015)

De Matteis, T., Mencagli, G.: Proactive elasticity and energy awareness in data stream processing. J. Syst. Softw. 127(C), 302–319 (2017). https://doi.org/10.1016/j.jss.2016.08.037CrossRef

Ewald, E., Vogel, A., Rista, C., Griebler, D., Manssour, I., Gustavo, L.: Parallel and distributed processing support for a geospatial data visualization DSL. In: Symposium on High Performance Computing Systems (WSCAD), pp. 221–228. IEEE (2018)

FastFlow: FastFlow (FF) Website (2019). http://mc-fastflow.sourceforge.net/. Accessed Feb 2019

Friedman, E., Tzoumas, K.: Introduction to Apache Flink: Stream Processing for Real Time and Beyond, 1st edn. O’Reilly Media Inc., Sebastopol (2016)

Georges, A., Buytaert, D., Eeckhout, L.: Statistically rigorous java performance evaluation. SIGPLAN Not. 42(10), 57–76 (2007). https://doi.org/10.1145/1297105.1297033CrossRef

Griebler, D., Danelutto, M., Torquati, M., Fernandes, L.G.: SPar: a DSL for high-level and productive stream parallelism. Parallel Process. Lett. 27(01), 1740005 (2017). https://doi.org/10.1142/S0129626417400059MathSciNetCrossRef

10.

Griebler, D., Hoffmann, R.B., Danelutto, M., Fernandes, L.G.: Higher-level parallelism abstractions for video applications with SPar. In: Parallel Computing is Everywhere, Proceedings of the International Conference on Parallel Computing, ParCo 2017, pp. 698–707. IOS Press, Bologna (2017). https://doi.org/10.3233/978-1-61499-843-3-698

11.

Griebler, D., Hoffmann, R.B., Danelutto, M., Fernandes, L.G.: Stream Parallelism with ordered data constraints on multi-core systems. J. Supercomput. 75, 1–20 (2018). https://doi.org/10.1007/s11227-018-2482-7CrossRef

12.

Hirzel, M., Soulé, R., Schneider, S., Gedik, B., Grimm, R.: A catalog of stream processing optimizations. ACM Comput. Surv. 46(4), 46:1–46:34 (2014)CrossRef

13.

Jain, A.: Mastering Apache Storm: Real-time Big Data Streaming Using Kafka, Hbase and Redis. Packt Publishing, Birmingham (2017)

14.

Latham, R., Bautista-Gomez, L., Balaji, P.: Portable topology-aware MPI-I/O. In: IEEE International Conference on Parallel and Distributed Systems (ICPADS), pp. 710–719, December 2017. https://doi.org/10.1109/ICPADS.2017.00096

15.

Ledur, C., Griebler, D., Manssour, I., Fernandes, L.G.: A high-level DSL for geospatial visualizations with multi-core parallelism support. In: 41th IEEE Computer Society Signature Conference on Computers, Software and Applications, COMPSAC 2017, pp. 298–304. IEEE, Torino (2017)

16.

Matteis, T.D., Mencagli, G.: Keep calm and react with foresight: strategies for low-latency and energy-efficient elastic data stream processing. In: Proceedings of the ACM Symposium on Principles and Practice of Parallel Programming, pp. 13:1–13:12 (2016)

17.

McCool, M., Robison, A.D., Reinders, J.: Structured Parallel Programming: Patterns for Efficient Computation. Morgan Kaufmann, Burlington (2012)

18.

Mendez, S., Rexachs, D., Luque, E.: Analyzing the parallel I/O severity of MPI applications. In: IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing, pp. 953–962 (2017). https://doi.org/10.1109/CCGRID.2017.45

19.

Moreland, K.: A survey of visualization pipelines. IEEE Trans. Visual Comput. Graph. 19(3), 367–378 (2013)CrossRef

20.

Pereira, R., Azambuja, M., Breitman, K., Endler, M.: An architecture for distributed high performance video processing in the cloud. In: international Conference on Cloud Computing, pp. 482–489. IEEE (2010)

21.

Perrot, A., Bourqui, R., Hanusse, N., Lalanne, F., Auber, D.: Large interactive visualization of density functions on big data infrastructure. In: IEEE Symposium on Large Data Analysis and Visualization (LDAV), pp. 99–106, October 2015. https://doi.org/10.1109/LDAV.2015.7348077

22.

Quinn, M.J.: Parallel Programming in C with MPI and OpenMP. McGraw-Hill, New York (2003)

23.

Reinders, J.: Intel Threading Building Blocks: Outfitting C++ for Multi-core Processor Parallelism. O’Reilly Media, Sebastopol (2007)

24.

Seo, S., Latham, R., Zhang, J., Balaji, P.: Implementation and evaluation of MPI nonblocking collective I/O. In: IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing, pp. 1084–1091, May 2015. https://doi.org/10.1109/CCGrid.2015.81

25.

Steed, C.A., et al.: Big data visual analytics for exploratory earth system simulation analysis. Comput. Geosci. 61, 71–82 (2013). https://doi.org/10.1016/j.cageo.2013.07.025CrossRef

26.

Tan, H., Chen, L.: An approach for fast and parallel video processing on apache Hadoop clusters. In: 2014 IEEE International Conference on Multimedia and Expo (ICME), pp. 1–6, July 2014. https://doi.org/10.1109/ICME.2014.6890135

27.

Theis, T.N., Wong, H.S.P.: The end of Moore’s law: a new beginning for information technology. Comput. Sci. Eng. 19(2), 41 (2017)CrossRef

28.

Thies, W., Karczmarek, M., Amarasinghe, S.: StreamIt: a language for streaming applications. In: Horspool, R.N. (ed.) CC 2002. LNCS, vol. 2304, pp. 179–196. Springer, Heidelberg (2002). https://doi.org/10.1007/3-540-45937-5_14CrossRef

29.

Vogel, A., Griebler, D., De Sensi, D., Danelutto, M., Fernandes, L.G.: Autonomic and latency-aware degree of parallelism management in SPar. In: Mencagli, G., et al. (eds.) Euro-Par 2018. LNCS, vol. 11339, pp. 28–39. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-10549-5_3CrossRef

30.

Wylie, B.N., Baumes, J.: A unified toolkit for information and scientific visualization. In: VDA, p. 72430 (2009)

31.

Zhang, T., Hua, G., Ligmann-Zielinska, A.: Visually-driven parallel solving of multi-objective land-use allocation problems: a case study in Chelan, Washington. Earth Sci. Inf. 8, 809–825 (2015)CrossRef

Titel: Parallel Stream Processing with MPI for Video Analytics and Data Visualization
verfasst von: Adriano Vogel
Cassiano Rista
Gabriel Justo
Endrius Ewald
Dalvan Griebler
Gabriele Mencagli
Luiz Gustavo Fernandes
Verlag: Springer International Publishing
Buch: High Performance Computing Systems
Print ISBN: 978-3-030-41049-0

Electronic ISBN: 978-3-030-41050-6

Copyright-Jahr: 2020
DOI: https://doi.org/10.1007/978-3-030-41050-6_7

Springer Professional

Abstract

Bitte loggen Sie sich ein, um Zugang zu Ihrer Lizenz zu erhalten.

Sie haben noch keine Lizenz? Dann Informieren Sie sich jetzt über unsere Produkte:

Springer Professional "Wirtschaft+Technik"

Springer Professional "Technik"

Springer Professional "Wirtschaft"