Skip to main content
main-content

Tipp

Weitere Kapitel dieses Buchs durch Wischen aufrufen

2021 | OriginalPaper | Buchkapitel

SpecK: Composition of Stream Processing Applications over Fog Environments

verfasst von : Davaadorj Battulga, Daniele Miorandi, Cédric Tedeschi

Erschienen in: Distributed Applications and Interoperable Systems

Verlag: Springer International Publishing

share
TEILEN

Abstract

Stream Processing (SP), i.e., the processing of data in motion, as soon as it becomes available, is a hot topic in cloud computing. Various SP stacks exist today, with applications ranging from IoT analytics to processing of payment transactions. The backbone of said stacks are Stream Processing Engines (SPEs), software packages offering a high-level programming model and scalable execution of data stream processing pipelines. SPEs have been traditionally developed to work inside a single datacenter, and optimised for speed. With the advent of Fog computing, however, the processing of data streams needs to be carried out over multiple geographically distributed computing sites: Data gets typically pre-processed close to where they are generated, then aggregated at intermediate nodes, and finally globally and persistently stored in the Cloud. SPEs were not designed to address these new scenarios. In this paper, we argue that large scale Fog-based stream processing should rely on the coordinated composition of geographically dispersed SPE instances. We propose an architecture based on the composition of multiple SPE instances and their communication via distributed message brokers. We introduce SpecK, a tool to automate the deployment and adaptation of pipelines over a Fog computing platform. Given a description of the pipeline, SpecK covers all the operations needed to deploy a stream processing computation over the different SPE instances targeted, using their own APIs and establishing the required communication channels to forward data among them. A prototypical implementation of SpecK is presented, and its performance is evaluated over Grid’5000, a large-scale, distributed experimental facility.
Literatur
4.
Zurück zum Zitat de Assunção, M.D., Veith, A.D.S., Buyya, R.: Distributed data stream processing and edge computing: a survey on resource elasticity and future directions. J. Netw. Comput. Appl. 103, 1–17 (2018) CrossRef de Assunção, M.D., Veith, A.D.S., Buyya, R.: Distributed data stream processing and edge computing: a survey on resource elasticity and future directions. J. Netw. Comput. Appl. 103, 1–17 (2018) CrossRef
5.
Zurück zum Zitat Balouek-Thomert, D., Renart, E.G., Zamani, A.R., Simonet, A., Parashar, M.: Towards a computing continuum: enabling edge-to-cloud integration for data-driven workflows. Int. J. High Perform. Comput. Appl. 33(6), 1159–1174 (2019) CrossRef Balouek-Thomert, D., Renart, E.G., Zamani, A.R., Simonet, A., Parashar, M.: Towards a computing continuum: enabling edge-to-cloud integration for data-driven workflows. Int. J. High Perform. Comput. Appl. 33(6), 1159–1174 (2019) CrossRef
6.
Zurück zum Zitat Beckman, P., et al.: Harnessing the computing continuum for programming our world, chap. 7, pp. 215–230. Wiley (2020) Beckman, P., et al.: Harnessing the computing continuum for programming our world, chap. 7, pp. 215–230. Wiley (2020)
7.
Zurück zum Zitat Carbone, P., Katsifodimos, A., Ewen, S., Markl, V., Haridi, S., Tzoumas, K.: Apache flink: stream and batch processing in a single engine. Bull. IEEE Comput. Soc. Tech. Commit. Data Eng. 36(4) (2015) Carbone, P., Katsifodimos, A., Ewen, S., Markl, V., Haridi, S., Tzoumas, K.: Apache flink: stream and batch processing in a single engine. Bull. IEEE Comput. Soc. Tech. Commit. Data Eng. 36(4) (2015)
8.
Zurück zum Zitat Cardellini, V., Grassi, V., Lo Presti, F., Nardelli, M.: Distributed QoS-aware scheduling in storm. In: Proceedings of the 9th ACM International Conference on Distributed Event-Based Systems, pp. 344–347 (2015) Cardellini, V., Grassi, V., Lo Presti, F., Nardelli, M.: Distributed QoS-aware scheduling in storm. In: Proceedings of the 9th ACM International Conference on Distributed Event-Based Systems, pp. 344–347 (2015)
9.
Zurück zum Zitat Castro Fernandez, R., Migliavacca, M., Kalyvianaki, E., Pietzuch, P.: Integrating scale out and fault tolerance in stream processing using operator state management. In: ACM SIGMOD 2013, pp. 725–736 (2013) Castro Fernandez, R., Migliavacca, M., Kalyvianaki, E., Pietzuch, P.: Integrating scale out and fault tolerance in stream processing using operator state management. In: ACM SIGMOD 2013, pp. 725–736 (2013)
10.
Zurück zum Zitat Cheng, B., Papageorgiou, A., Bauer, M.: Geelytics: enabling on-demand edge analytics over scoped data sources. In: 2016 IEEE International Congress on Big Data (BigData Congress), pp. 101–108 (2016) Cheng, B., Papageorgiou, A., Bauer, M.: Geelytics: enabling on-demand edge analytics over scoped data sources. In: 2016 IEEE International Congress on Big Data (BigData Congress), pp. 101–108 (2016)
11.
Zurück zum Zitat Claudel, B., Huard, G., Richard, O.: TakTuk, adaptive deployment of remote executions. In: Kranzlmüller, D., Bode, A., Hegering, H., Casanova, H., Gerndt, M. (eds.) Proceedings of the 18th ACM International Symposium on High Performance Distributed Computing, HPDC 2009, Garching, Germany, 11–13 June 2009, pp. 91–100. ACM (2009). https://​doi.​org/​10.​1145/​1551609.​1551629 Claudel, B., Huard, G., Richard, O.: TakTuk, adaptive deployment of remote executions. In: Kranzlmüller, D., Bode, A., Hegering, H., Casanova, H., Gerndt, M. (eds.) Proceedings of the 18th ACM International Symposium on High Performance Distributed Computing, HPDC 2009, Garching, Germany, 11–13 June 2009, pp. 91–100. ACM (2009). https://​doi.​org/​10.​1145/​1551609.​1551629
12.
Zurück zum Zitat Fu, X., Ghaffar, T., Davis, J.C., Lee, D.: Edgewise: a better stream processing engine for the edge. In: 2019 USENIX Annual Technical Conference (USENIX ATC 19), pp. 929–946. USENIX Association, Renton, July 2019 Fu, X., Ghaffar, T., Davis, J.C., Lee, D.: Edgewise: a better stream processing engine for the edge. In: 2019 USENIX Annual Technical Conference (USENIX ATC 19), pp. 929–946. USENIX Association, Renton, July 2019
13.
Zurück zum Zitat Gedik, B., Schneider, S., Hirzel, M., Wu, K.L.: Elastic scaling for data stream processing. IEEE Trans. Parallel Distrib. Syst. 25(6), 1447–1463 (2013) CrossRef Gedik, B., Schneider, S., Hirzel, M., Wu, K.L.: Elastic scaling for data stream processing. IEEE Trans. Parallel Distrib. Syst. 25(6), 1447–1463 (2013) CrossRef
14.
Zurück zum Zitat Gibert Renart, E., Da Silva Veith, A., Balouek-Thomert, D., De Assunção, M.D., Lefèvre, L., Parashar, M.: Distributed operator placement for IoT data analytics across edge and cloud resources. In: 2019 19th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing (CCGRID), pp. 459–468 (2019) Gibert Renart, E., Da Silva Veith, A., Balouek-Thomert, D., De Assunção, M.D., Lefèvre, L., Parashar, M.: Distributed operator placement for IoT data analytics across edge and cloud resources. In: 2019 19th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing (CCGRID), pp. 459–468 (2019)
15.
Zurück zum Zitat Gulisano, V., Jiménez-Peris, R., Patião-Martínez, M., Soriente, C., Valduriez, P.: StreamCloud: an elastic and scalable data streaming system. IEEE Trans. Parallel Distrib. Syst. 23(12), 2351–2365 (2012) CrossRef Gulisano, V., Jiménez-Peris, R., Patião-Martínez, M., Soriente, C., Valduriez, P.: StreamCloud: an elastic and scalable data streaming system. IEEE Trans. Parallel Distrib. Syst. 23(12), 2351–2365 (2012) CrossRef
16.
Zurück zum Zitat Lakshmanan, G.T., Li, Y., Strom, R.: Placement strategies for internet-scale data stream systems. IEEE Internet Comput. 12(6), 50–60 (2008) CrossRef Lakshmanan, G.T., Li, Y., Strom, R.: Placement strategies for internet-scale data stream systems. IEEE Internet Comput. 12(6), 50–60 (2008) CrossRef
17.
Zurück zum Zitat Light, R.A.: Mosquitto: server and client implementation of the MQTT protocol. J. Open Sour. Softw. 2(13), 265 (2017) CrossRef Light, R.A.: Mosquitto: server and client implementation of the MQTT protocol. J. Open Sour. Softw. 2(13), 265 (2017) CrossRef
18.
Zurück zum Zitat Liu, F., Tang, G., Li, Y., Cai, Z., Zhang, X., Zhou, T.: A survey on edge computing systems and tools. Proc. IEEE 107(8), 1537–1562 (2019) CrossRef Liu, F., Tang, G., Li, Y., Cai, Z., Zhang, X., Zhou, T.: A survey on edge computing systems and tools. Proc. IEEE 107(8), 1537–1562 (2019) CrossRef
20.
Zurück zum Zitat Milojicic, D.: The edge-to-cloud continuum. Computer 53(11), 16–25 (2020) CrossRef Milojicic, D.: The edge-to-cloud continuum. Computer 53(11), 16–25 (2020) CrossRef
21.
Zurück zum Zitat Nicolae, B., Bresnahan, J., Keahey, K., Antoniu, G.: Going back and forth: efficient multideployment and multisnapshotting on clouds. In: Proceedings of the 20th International Symposium on High Performance Distributed Computing (HPDC), pp. 147–158 (2011) Nicolae, B., Bresnahan, J., Keahey, K., Antoniu, G.: Going back and forth: efficient multideployment and multisnapshotting on clouds. In: Proceedings of the 20th International Symposium on High Performance Distributed Computing (HPDC), pp. 147–158 (2011)
22.
Zurück zum Zitat O’Keeffe, D., Salonidis, T., Pietzuch, P.: Frontier: resilient edge processing for the internet of things. Proc. VLDB Endow. 11(10), 1178–1191 (2018) CrossRef O’Keeffe, D., Salonidis, T., Pietzuch, P.: Frontier: resilient edge processing for the internet of things. Proc. VLDB Endow. 11(10), 1178–1191 (2018) CrossRef
23.
Zurück zum Zitat Peng, B., Hosseini, M., Hong, Z., Farivar, R., Campbell, R.: R-storm: resource-aware scheduling in storm. In: Proceedings of the 16th Annual Middleware Conference, Middleware 2015, pp. 149–161 (2015) Peng, B., Hosseini, M., Hong, Z., Farivar, R., Campbell, R.: R-storm: resource-aware scheduling in storm. In: Proceedings of the 16th Annual Middleware Conference, Middleware 2015, pp. 149–161 (2015)
24.
Zurück zum Zitat Pisani, F., Brunetta, J.R., Martins Do Rosario, V., Borin, E.: Beyond the fog: bringing cross-platform code execution to constrained IoT devices. In: 2017 29th International Symposium on Computer Architecture and High Performance Computing (SBAC-PAD), pp. 17–24 (2017) Pisani, F., Brunetta, J.R., Martins Do Rosario, V., Borin, E.: Beyond the fog: bringing cross-platform code execution to constrained IoT devices. In: 2017 29th International Symposium on Computer Architecture and High Performance Computing (SBAC-PAD), pp. 17–24 (2017)
25.
Zurück zum Zitat Prosperi, L., Costan, A., Silva, P., Antoniu, G.: Planner: cost-efficient execution plans placement for uniform stream analytics on edge and cloud. In: 2018 IEEE/ACM Workflows in Support of Large-Scale Science (WORKS), pp. 42–51 (2018) Prosperi, L., Costan, A., Silva, P., Antoniu, G.: Planner: cost-efficient execution plans placement for uniform stream analytics on edge and cloud. In: 2018 IEEE/ACM Workflows in Support of Large-Scale Science (WORKS), pp. 42–51 (2018)
26.
Zurück zum Zitat Rosendo, D., Silva, P., Simonin, M., Costan, A., Antoniu, G.: E2Clab: exploring the computing continuum through repeatable, replicable and reproducible edge-to-cloud experiments. In: Cluster 2020 - IEEE International Conference on Cluster Computing, Kobe, Japan, pp. 1–11, September 2020 Rosendo, D., Silva, P., Simonin, M., Costan, A., Antoniu, G.: E2Clab: exploring the computing continuum through repeatable, replicable and reproducible edge-to-cloud experiments. In: Cluster 2020 - IEEE International Conference on Cluster Computing, Kobe, Japan, pp. 1–11, September 2020
27.
Zurück zum Zitat Silva, P., Costan, A., Antoniu, G.: Investigating edge vs. cloud computing trade-offs for stream processing. In: 2019 IEEE International Conference on Big Data (Big Data), pp. 469–474 (2019) Silva, P., Costan, A., Antoniu, G.: Investigating edge vs. cloud computing trade-offs for stream processing. In: 2019 IEEE International Conference on Big Data (Big Data), pp. 469–474 (2019)
28.
Zurück zum Zitat Silva, P., Costan, A., Antoniu, G.: Towards a methodology for benchmarking edge processing frameworks. In: 2019 IEEE International Parallel and Distributed Processing Symposium Workshops (IPDPSW), pp. 904–907 (2019) Silva, P., Costan, A., Antoniu, G.: Towards a methodology for benchmarking edge processing frameworks. In: 2019 IEEE International Parallel and Distributed Processing Symposium Workshops (IPDPSW), pp. 904–907 (2019)
30.
Zurück zum Zitat Snyder, B., Bosanac, D., Davies, R.: Introduction to apache ActiveMQ. Active MQ in action, pp. 6–16 (2017) Snyder, B., Bosanac, D., Davies, R.: Introduction to apache ActiveMQ. Active MQ in action, pp. 6–16 (2017)
31.
Zurück zum Zitat Thein, K.M.M.: Apache kafka: next generation distributed messaging system. Int. J. Sci. Eng. Technol. Res. 3(47), 9478–9483 (2014) Thein, K.M.M.: Apache kafka: next generation distributed messaging system. Int. J. Sci. Eng. Technol. Res. 3(47), 9478–9483 (2014)
33.
Zurück zum Zitat Zaharia, M., Das, T., Li, H., Hunter, T., Shenker, S., Stoica, I.: Discretized streams: fault-tolerant streaming computation at scale. In: 24th ACM SIGOPS Symposium on Operating Systems Principles (SOSP 2013), Farmington, USA, pp. 423–438, November 2013 Zaharia, M., Das, T., Li, H., Hunter, T., Shenker, S., Stoica, I.: Discretized streams: fault-tolerant streaming computation at scale. In: 24th ACM SIGOPS Symposium on Operating Systems Principles (SOSP 2013), Farmington, USA, pp. 423–438, November 2013
Metadaten
Titel
SpecK: Composition of Stream Processing Applications over Fog Environments
verfasst von
Davaadorj Battulga
Daniele Miorandi
Cédric Tedeschi
Copyright-Jahr
2021
DOI
https://doi.org/10.1007/978-3-030-78198-9_3

Premium Partner