Skip to main content
Disciplines
Automotive Business IT + Informatics Construction + Real Estate Electrical Engineering + Electronics Energy + Sustainability Insurance + Risk Finance + Banking Management + Leadership Marketing + Sales Mechanical Engineering + Materials
Events
DE
EN
Books
Journals
Topic Page
Marketing
Start single access now
Access for companies
EXTENDED SEARCH
Log in
Top
Published in:
A Methodology for Tenant Migration in Legacy Shared-Table Multi-tenant Applications Read chapter Read first chapter

2021 | OriginalPaper | Chapter

SpecK: Composition of Stream Processing Applications over Fog Environments

Authors : Davaadorj Battulga, Daniele Miorandi, Cédric Tedeschi

Published in: Distributed Applications and Interoperable Systems

Publisher: Springer International Publishing

Log in

Activate our intelligent search to find suitable subject content or patents.

search-config
loading …

Abstract

Stream Processing (SP), i.e., the processing of data in motion, as soon as it becomes available, is a hot topic in cloud computing. Various SP stacks exist today, with applications ranging from IoT analytics to processing of payment transactions. The backbone of said stacks are Stream Processing Engines (SPEs), software packages offering a high-level programming model and scalable execution of data stream processing pipelines. SPEs have been traditionally developed to work inside a single datacenter, and optimised for speed. With the advent of Fog computing, however, the processing of data streams needs to be carried out over multiple geographically distributed computing sites: Data gets typically pre-processed close to where they are generated, then aggregated at intermediate nodes, and finally globally and persistently stored in the Cloud. SPEs were not designed to address these new scenarios. In this paper, we argue that large scale Fog-based stream processing should rely on the coordinated composition of geographically dispersed SPE instances. We propose an architecture based on the composition of multiple SPE instances and their communication via distributed message brokers. We introduce SpecK, a tool to automate the deployment and adaptation of pipelines over a Fog computing platform. Given a description of the pipeline, SpecK covers all the operations needed to deploy a stream processing computation over the different SPE instances targeted, using their own APIs and establishing the required communication channels to forward data among them. A prototypical implementation of SpecK is presented, and its performance is evaluated over Grid’5000, a large-scale, distributed experimental facility.

Dont have a licence yet? Then find out more about our products and how to get one now:

Springer Professional "Wirtschaft+Technik"

Online-Abonnement

Mit Springer Professional "Wirtschaft+Technik" erhalten Sie Zugriff auf:

  • über 102.000 Bücher
  • über 537 Zeitschriften

aus folgenden Fachgebieten:

  • Automobil + Motoren
  • Bauwesen + Immobilien
  • Business IT + Informatik
  • Elektrotechnik + Elektronik
  • Energie + Nachhaltigkeit
  • Finance + Banking
  • Management + Führung
  • Marketing + Vertrieb
  • Maschinenbau + Werkstoffe
  • Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

inform now

Springer Professional "Technik"

Online-Abonnement

Mit Springer Professional "Technik" erhalten Sie Zugriff auf:

  • über 67.000 Bücher
  • über 390 Zeitschriften

aus folgenden Fachgebieten:

  • Automobil + Motoren
  • Bauwesen + Immobilien
  • Business IT + Informatik
  • Elektrotechnik + Elektronik
  • Energie + Nachhaltigkeit
  • Maschinenbau + Werkstoffe




 

Jetzt Wissensvorsprung sichern!

inform now

Springer Professional "Wirtschaft"

Online-Abonnement

Mit Springer Professional "Wirtschaft" erhalten Sie Zugriff auf:

  • über 67.000 Bücher
  • über 340 Zeitschriften

aus folgenden Fachgebieten:

  • Bauwesen + Immobilien
  • Business IT + Informatik
  • Finance + Banking
  • Management + Führung
  • Marketing + Vertrieb
  • Versicherung + Risiko




Jetzt Wissensvorsprung sichern!

inform now
previous chapter Network Federation for Inter-cloud Operations
next chapter ASPAS: As Secure as Possible Available Systems
Footnotes
Literature
1.
2.
3.
4.
de Assunção, M.D., Veith, A.D.S., Buyya, R.: Distributed data stream processing and edge computing: a survey on resource elasticity and future directions. J. Netw. Comput. Appl. 103, 1–17 (2018)CrossRef
5.
Balouek-Thomert, D., Renart, E.G., Zamani, A.R., Simonet, A., Parashar, M.: Towards a computing continuum: enabling edge-to-cloud integration for data-driven workflows. Int. J. High Perform. Comput. Appl. 33(6), 1159–1174 (2019)CrossRef
6.
Beckman, P., et al.: Harnessing the computing continuum for programming our world, chap. 7, pp. 215–230. Wiley (2020)
7.
Carbone, P., Katsifodimos, A., Ewen, S., Markl, V., Haridi, S., Tzoumas, K.: Apache flink: stream and batch processing in a single engine. Bull. IEEE Comput. Soc. Tech. Commit. Data Eng. 36(4) (2015)
8.
Cardellini, V., Grassi, V., Lo Presti, F., Nardelli, M.: Distributed QoS-aware scheduling in storm. In: Proceedings of the 9th ACM International Conference on Distributed Event-Based Systems, pp. 344–347 (2015)
9.
Castro Fernandez, R., Migliavacca, M., Kalyvianaki, E., Pietzuch, P.: Integrating scale out and fault tolerance in stream processing using operator state management. In: ACM SIGMOD 2013, pp. 725–736 (2013)
10.
Cheng, B., Papageorgiou, A., Bauer, M.: Geelytics: enabling on-demand edge analytics over scoped data sources. In: 2016 IEEE International Congress on Big Data (BigData Congress), pp. 101–108 (2016)
11.
Claudel, B., Huard, G., Richard, O.: TakTuk, adaptive deployment of remote executions. In: Kranzlmüller, D., Bode, A., Hegering, H., Casanova, H., Gerndt, M. (eds.) Proceedings of the 18th ACM International Symposium on High Performance Distributed Computing, HPDC 2009, Garching, Germany, 11–13 June 2009, pp. 91–100. ACM (2009). https://​doi.​org/​10.​1145/​1551609.​1551629
12.
Fu, X., Ghaffar, T., Davis, J.C., Lee, D.: Edgewise: a better stream processing engine for the edge. In: 2019 USENIX Annual Technical Conference (USENIX ATC 19), pp. 929–946. USENIX Association, Renton, July 2019
13.
Gedik, B., Schneider, S., Hirzel, M., Wu, K.L.: Elastic scaling for data stream processing. IEEE Trans. Parallel Distrib. Syst. 25(6), 1447–1463 (2013)CrossRef
14.
Gibert Renart, E., Da Silva Veith, A., Balouek-Thomert, D., De Assunção, M.D., Lefèvre, L., Parashar, M.: Distributed operator placement for IoT data analytics across edge and cloud resources. In: 2019 19th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing (CCGRID), pp. 459–468 (2019)
15.
Gulisano, V., Jiménez-Peris, R., Patião-Martínez, M., Soriente, C., Valduriez, P.: StreamCloud: an elastic and scalable data streaming system. IEEE Trans. Parallel Distrib. Syst. 23(12), 2351–2365 (2012)CrossRef
16.
Lakshmanan, G.T., Li, Y., Strom, R.: Placement strategies for internet-scale data stream systems. IEEE Internet Comput. 12(6), 50–60 (2008)CrossRef
17.
Light, R.A.: Mosquitto: server and client implementation of the MQTT protocol. J. Open Sour. Softw. 2(13), 265 (2017)CrossRef
18.
Liu, F., Tang, G., Li, Y., Cai, Z., Zhang, X., Zhou, T.: A survey on edge computing systems and tools. Proc. IEEE 107(8), 1537–1562 (2019)CrossRef
19.
20.
Milojicic, D.: The edge-to-cloud continuum. Computer 53(11), 16–25 (2020)CrossRef
21.
Nicolae, B., Bresnahan, J., Keahey, K., Antoniu, G.: Going back and forth: efficient multideployment and multisnapshotting on clouds. In: Proceedings of the 20th International Symposium on High Performance Distributed Computing (HPDC), pp. 147–158 (2011)
22.
O’Keeffe, D., Salonidis, T., Pietzuch, P.: Frontier: resilient edge processing for the internet of things. Proc. VLDB Endow. 11(10), 1178–1191 (2018)CrossRef
23.
Peng, B., Hosseini, M., Hong, Z., Farivar, R., Campbell, R.: R-storm: resource-aware scheduling in storm. In: Proceedings of the 16th Annual Middleware Conference, Middleware 2015, pp. 149–161 (2015)
24.
Pisani, F., Brunetta, J.R., Martins Do Rosario, V., Borin, E.: Beyond the fog: bringing cross-platform code execution to constrained IoT devices. In: 2017 29th International Symposium on Computer Architecture and High Performance Computing (SBAC-PAD), pp. 17–24 (2017)
25.
Prosperi, L., Costan, A., Silva, P., Antoniu, G.: Planner: cost-efficient execution plans placement for uniform stream analytics on edge and cloud. In: 2018 IEEE/ACM Workflows in Support of Large-Scale Science (WORKS), pp. 42–51 (2018)
26.
Rosendo, D., Silva, P., Simonin, M., Costan, A., Antoniu, G.: E2Clab: exploring the computing continuum through repeatable, replicable and reproducible edge-to-cloud experiments. In: Cluster 2020 - IEEE International Conference on Cluster Computing, Kobe, Japan, pp. 1–11, September 2020
27.
Silva, P., Costan, A., Antoniu, G.: Investigating edge vs. cloud computing trade-offs for stream processing. In: 2019 IEEE International Conference on Big Data (Big Data), pp. 469–474 (2019)
28.
Silva, P., Costan, A., Antoniu, G.: Towards a methodology for benchmarking edge processing frameworks. In: 2019 IEEE International Parallel and Distributed Processing Symposium Workshops (IPDPSW), pp. 904–907 (2019)
29.
30.
Snyder, B., Bosanac, D., Davies, R.: Introduction to apache ActiveMQ. Active MQ in action, pp. 6–16 (2017)
31.
Thein, K.M.M.: Apache kafka: next generation distributed messaging system. Int. J. Sci. Eng. Technol. Res. 3(47), 9478–9483 (2014)
32.
33.
Zaharia, M., Das, T., Li, H., Hunter, T., Shenker, S., Stoica, I.: Discretized streams: fault-tolerant streaming computation at scale. In: 24th ACM SIGOPS Symposium on Operating Systems Principles (SOSP 2013), Farmington, USA, pp. 423–438, November 2013
Metadata
Title
SpecK: Composition of Stream Processing Applications over Fog Environments
Authors
Davaadorj Battulga
Daniele Miorandi
Cédric Tedeschi
Copyright Year
2021
Book
Distributed Applications and Interoperable Systems

Print ISBN: 978-3-030-78197-2

Electronic ISBN: 978-3-030-78198-9

Copyright Year: 2021

DOI
https://doi.org/10.1007/978-3-030-78198-9_3

Premium Partner

    Image Credits