Skip to main content
Top
Published in: Computing 6/2020

03-06-2019

Cloud resource management using 3Vs of Internet of Big data streams

Authors: Navroop Kaur, Sandeep K. Sood, Prabal Verma

Published in: Computing | Issue 6/2020

Log in

Activate our intelligent search to find suitable subject content or patents.

search-config
loading …

Abstract

Internet of things (IoT) allows various smart devices to get connected to anything, anywhere, and at anytime. The ubiquitous nature of IoT devices generates huge volume of data called Internet of Big data (IoBd). IoBd is generated in continuous streams and at unprecedented speed. The rapid analysis of such IoBd streams is the need of hour. Moreover, the allocation of optimal number of cloud resources for real time analysis of IoBd streams is a challenging task. Most of the current methods use data characteristics provided by the user to allocate cloud nodes. But in case of IoBd streams, data characteristics are usually unknown to the user because of the stochastic nature of IoT devices. This poses difficulty in selecting appropriate cloud resources. This paper proposes an efficient method to tackle this issue. The proposed method first predicts the data characteristics of IoBd stream in terms of volume, velocity and variety (3Vs). Later, these predicted values are expressed in terms of a triplet called Charactrization of Stream (CoSt). On the other hand, self-organizing maps are used to create dynamic clusters of cloud resources. One of the clusters is allocated to IoBd stream based upon its CoSt. Experimental results show that the proposed method effectively boosted the performance of cloud resources and minimized the execution and waiting time of IoBd stream processing.

Dont have a licence yet? Then find out more about our products and how to get one now:

Springer Professional "Wirtschaft"

Online-Abonnement

Mit Springer Professional "Wirtschaft" erhalten Sie Zugriff auf:

  • über 67.000 Bücher
  • über 340 Zeitschriften

aus folgenden Fachgebieten:

  • Bauwesen + Immobilien
  • Business IT + Informatik
  • Finance + Banking
  • Management + Führung
  • Marketing + Vertrieb
  • Versicherung + Risiko




Jetzt Wissensvorsprung sichern!

Springer Professional "Technik"

Online-Abonnement

Mit Springer Professional "Technik" erhalten Sie Zugriff auf:

  • über 67.000 Bücher
  • über 390 Zeitschriften

aus folgenden Fachgebieten:

  • Automobil + Motoren
  • Bauwesen + Immobilien
  • Business IT + Informatik
  • Elektrotechnik + Elektronik
  • Energie + Nachhaltigkeit
  • Maschinenbau + Werkstoffe




 

Jetzt Wissensvorsprung sichern!

Springer Professional "Wirtschaft+Technik"

Online-Abonnement

Mit Springer Professional "Wirtschaft+Technik" erhalten Sie Zugriff auf:

  • über 102.000 Bücher
  • über 537 Zeitschriften

aus folgenden Fachgebieten:

  • Automobil + Motoren
  • Bauwesen + Immobilien
  • Business IT + Informatik
  • Elektrotechnik + Elektronik
  • Energie + Nachhaltigkeit
  • Finance + Banking
  • Management + Führung
  • Marketing + Vertrieb
  • Maschinenbau + Werkstoffe
  • Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

Literature
1.
go back to reference Zheng Z, Wu X, Zhang Y, Lyu MR, Wang J (2013) QoS ranking prediction for cloud services. IEEE Trans Parallel Distrib Syst 24(6):1213–1222CrossRef Zheng Z, Wu X, Zhang Y, Lyu MR, Wang J (2013) QoS ranking prediction for cloud services. IEEE Trans Parallel Distrib Syst 24(6):1213–1222CrossRef
2.
go back to reference Sandhu R, Sood SK (2014) Scheduling of big data applications on distributed cloud based on QoS parameters. Clust Comput 18(2):817–828CrossRef Sandhu R, Sood SK (2014) Scheduling of big data applications on distributed cloud based on QoS parameters. Clust Comput 18(2):817–828CrossRef
5.
6.
go back to reference Chen M, Mao S, Liu Y (2014) Big data: a survey. Mob Netw Appl 19(2):171–209CrossRef Chen M, Mao S, Liu Y (2014) Big data: a survey. Mob Netw Appl 19(2):171–209CrossRef
7.
go back to reference Philip Chen CLL, Zhang CYY (2014) Data-intensive applications, challenges, techniques and technologies: a survey on Big data. Inf Sci NY 275:314–347CrossRef Philip Chen CLL, Zhang CYY (2014) Data-intensive applications, challenges, techniques and technologies: a survey on Big data. Inf Sci NY 275:314–347CrossRef
8.
go back to reference Hashem IAT, Yaqoob I, Badrul Anuar N, Mokhtar S, Gani A, Ullah Khan S (2015) The rise of ‘Big Data’ on cloud computing: review and open research issues. Inf Syst 47:98–115CrossRef Hashem IAT, Yaqoob I, Badrul Anuar N, Mokhtar S, Gani A, Ullah Khan S (2015) The rise of ‘Big Data’ on cloud computing: review and open research issues. Inf Syst 47:98–115CrossRef
9.
go back to reference Rao J, Wei Y, Gong J, Xu CZ (2013) QoS guarantees and service differentiation for dynamic cloud applications. IEEE Trans Netw Serv Manag 10(1):43–55CrossRef Rao J, Wei Y, Gong J, Xu CZ (2013) QoS guarantees and service differentiation for dynamic cloud applications. IEEE Trans Netw Serv Manag 10(1):43–55CrossRef
10.
go back to reference Wang W-J, Chang Y-S, Lo W-T, Lee Y-K (2013) Adaptive scheduling for parallel tasks with QoS satisfaction for hybrid cloud environments. J Supercomput 66(2):783–811CrossRef Wang W-J, Chang Y-S, Lo W-T, Lee Y-K (2013) Adaptive scheduling for parallel tasks with QoS satisfaction for hybrid cloud environments. J Supercomput 66(2):783–811CrossRef
11.
go back to reference Zhu Z, Li S, Chen X (2013) Design QoS-aware multi-path provisioning strategies for efficient cloud-assisted SVC video streaming to heterogeneous clients. IEEE Trans Multimed 15(4):758–768CrossRef Zhu Z, Li S, Chen X (2013) Design QoS-aware multi-path provisioning strategies for efficient cloud-assisted SVC video streaming to heterogeneous clients. IEEE Trans Multimed 15(4):758–768CrossRef
12.
go back to reference Hsu W-H, Lo C-H (2014) QoS/QoE mapping and adjustment model in the cloud-based multimedia infrastructure. IEEE Syst J 8(1):247–255CrossRef Hsu W-H, Lo C-H (2014) QoS/QoE mapping and adjustment model in the cloud-based multimedia infrastructure. IEEE Syst J 8(1):247–255CrossRef
13.
go back to reference Chang JM (2013) QoS-aware data replication for data-intensive applications in cloud computing systems. IEEE Trans Cloud Comput 1(1):101–115CrossRef Chang JM (2013) QoS-aware data replication for data-intensive applications in cloud computing systems. IEEE Trans Cloud Comput 1(1):101–115CrossRef
14.
go back to reference Misra S, Das S, Khatua M, Obaidat MS (2014) QoS-guaranteed bandwidth shifting and redistribution in mobile cloud environment. IEEE Trans Cloud Comput 2(2):181–193CrossRef Misra S, Das S, Khatua M, Obaidat MS (2014) QoS-guaranteed bandwidth shifting and redistribution in mobile cloud environment. IEEE Trans Cloud Comput 2(2):181–193CrossRef
15.
go back to reference Chen KT, Chang YC, Hsu HJ, Chen DY, Huang CY, Hsu CH (2014) On the quality of service of cloud gaming systems. IEEE Trans Multimed 16(2):480–495CrossRef Chen KT, Chang YC, Hsu HJ, Chen DY, Huang CY, Hsu CH (2014) On the quality of service of cloud gaming systems. IEEE Trans Multimed 16(2):480–495CrossRef
16.
go back to reference Sood SK (2016) Function points-based resource prediction in cloud computing. Concurr Comput Pract Exp 28(10):2781–2794CrossRef Sood SK (2016) Function points-based resource prediction in cloud computing. Concurr Comput Pract Exp 28(10):2781–2794CrossRef
17.
go back to reference Sood SK, Sandhu R (2015) Matrix based proactive resource provisioning in mobile cloud environment. Simul Model Pract Theory 50:83–95CrossRef Sood SK, Sandhu R (2015) Matrix based proactive resource provisioning in mobile cloud environment. Simul Model Pract Theory 50:83–95CrossRef
18.
20.
go back to reference Olston C, Chiou G, Chitnis L, Liu F, Han Y, Larsson M, Neumann A, Rao VBN, Sankarasubramanian V, Seth S, Tian C, Zicornell T, Wang X (2011) Nova: continuous pig/hadoop workflows. In: Proceedings of the 2011 ACM SIGMOD international conference on management of data. ACM, pp 1081–1090 Olston C, Chiou G, Chitnis L, Liu F, Han Y, Larsson M, Neumann A, Rao VBN, Sankarasubramanian V, Seth S, Tian C, Zicornell T, Wang X (2011) Nova: continuous pig/hadoop workflows. In: Proceedings of the 2011 ACM SIGMOD international conference on management of data. ACM, pp 1081–1090
26.
go back to reference Olston C, Seth S, Tian C, ZiCornell T, Wang X, Chiou G, Chitnis L, Liu F, Han Y, Larsson M, Neumann A, Rao VBN, Sankarasubramanian V (2011) Nova. In: Proceedings of the international conference on management of data—SIGMOD’11, p 1081 Olston C, Seth S, Tian C, ZiCornell T, Wang X, Chiou G, Chitnis L, Liu F, Han Y, Larsson M, Neumann A, Rao VBN, Sankarasubramanian V (2011) Nova. In: Proceedings of the international conference on management of data—SIGMOD’11, p 1081
27.
go back to reference Bhatotia P, Wieder A, Rodrigues R, Acar Ua, Pasquin R (2011) Incoop: MapReduce for incremental computations. In: Proceedings of the 2nd ACM symposium on cloud computing—SOCC’11, pp 1–14 Bhatotia P, Wieder A, Rodrigues R, Acar Ua, Pasquin R (2011) Incoop: MapReduce for incremental computations. In: Proceedings of the 2nd ACM symposium on cloud computing—SOCC’11, pp 1–14
28.
go back to reference Neumeyer L, Robbins B, Nair A, Kesari A (2010) S4: distributed stream computing platform. In: IEEE international conference on data mining workshops, pp 170–177 Neumeyer L, Robbins B, Nair A, Kesari A (2010) S4: distributed stream computing platform. In: IEEE international conference on data mining workshops, pp 170–177
31.
go back to reference Zhang F, Cao J, Khan SU, Li K, Hwang K (2015) A task-level adaptive MapReduce framework for real-time streaming data in healthcare applications. Futur Gener Comput Syst 43–44:149–160CrossRef Zhang F, Cao J, Khan SU, Li K, Hwang K (2015) A task-level adaptive MapReduce framework for real-time streaming data in healthcare applications. Futur Gener Comput Syst 43–44:149–160CrossRef
32.
go back to reference Zhang Q, Chen Z, Yang LT (2015) A nodes scheduling model based on Markov chain prediction for big streaming data analysis. Int J Commun Syst 28(9):1610–1619CrossRef Zhang Q, Chen Z, Yang LT (2015) A nodes scheduling model based on Markov chain prediction for big streaming data analysis. Int J Commun Syst 28(9):1610–1619CrossRef
33.
go back to reference Jain, A, Chang EY (2004) Adaptive sampling for sensor networks. In: Proceedings of the 1st international workshop on data management for sensor networks in conjunction with VLDB 2004—DMSN’04, p 10 Jain, A, Chang EY (2004) Adaptive sampling for sensor networks. In: Proceedings of the 1st international workshop on data management for sensor networks in conjunction with VLDB 2004—DMSN’04, p 10
35.
go back to reference Ranger C, Raghuraman R, Penmetsa A, Bradski G, Kozyrakis C (2007) Evaluating MapReduce for multi-core and multiprocessor systems. In: Proceedings of the IEEE 13th international symposium on high performance computer architecture, pp 13–24 Ranger C, Raghuraman R, Penmetsa A, Bradski G, Kozyrakis C (2007) Evaluating MapReduce for multi-core and multiprocessor systems. In: Proceedings of the IEEE 13th international symposium on high performance computer architecture, pp 13–24
38.
go back to reference Ekanayake J, Li H, Zhang B, Gunarathne T, Bae S-H, Qiu J, Fox G (2010) Twister. In: Proceedings of the 19th ACM international symposium on high performance distributed computing—HPDC’10, p 810 Ekanayake J, Li H, Zhang B, Gunarathne T, Bae S-H, Qiu J, Fox G (2010) Twister. In: Proceedings of the 19th ACM international symposium on high performance distributed computing—HPDC’10, p 810
39.
go back to reference Dou A, Kalogeraki V, Gunopulos D, Mielikainen T, Tuulos VH (2010) Misco. In: Proceedings of the 3rd international conference on PErvasive technologies related to assistive environments—PETRA’10, p 1 Dou A, Kalogeraki V, Gunopulos D, Mielikainen T, Tuulos VH (2010) Misco. In: Proceedings of the 3rd international conference on PErvasive technologies related to assistive environments—PETRA’10, p 1
40.
go back to reference Li R, Hu H, Li H, Wu Y, Yang J (2015) MapReduce parallel programming model: a state-of-the-art survey. Int J Parallel Progr 44(4):832–866CrossRef Li R, Hu H, Li H, Wu Y, Yang J (2015) MapReduce parallel programming model: a state-of-the-art survey. Int J Parallel Progr 44(4):832–866CrossRef
41.
go back to reference Feng B, Fu M, Ma H, Xia Y, Wang B (2014) Kalman filter with recursive covariance estimation-sequentially estimating process noise covariance. IEEE Trans Ind Electron 61(11):6253–6263CrossRef Feng B, Fu M, Ma H, Xia Y, Wang B (2014) Kalman filter with recursive covariance estimation-sequentially estimating process noise covariance. IEEE Trans Ind Electron 61(11):6253–6263CrossRef
42.
go back to reference Chandrasekhar VR, Bach J, Girod B, Chen DM, Tsai SS, Cheung N-M, Chen H, Takacs G, Reznik Y, Vedantham R, Grzeszczuk R (2011) The Stanford mobile visual search data set. In: Proceedings of the second annual ACM conference on multimedia systems—MMSys’11, p 117 Chandrasekhar VR, Bach J, Girod B, Chen DM, Tsai SS, Cheung N-M, Chen H, Takacs G, Reznik Y, Vedantham R, Grzeszczuk R (2011) The Stanford mobile visual search data set. In: Proceedings of the second annual ACM conference on multimedia systems—MMSys’11, p 117
50.
go back to reference Jiang Y, Huang Z, Tsang DH (2018) Towards max–min fair resource allocation for stream big data analytics in shared clouds. IEEE Trans Big Data 4(1):130–137CrossRef Jiang Y, Huang Z, Tsang DH (2018) Towards max–min fair resource allocation for stream big data analytics in shared clouds. IEEE Trans Big Data 4(1):130–137CrossRef
51.
go back to reference Hassan MM, Song B, Hossain MS, Alamri A (2014) Efficient resource scheduling for big data processing in cloud platform. In: International conference on internet and distributed computing systems, pp 51–63 Hassan MM, Song B, Hossain MS, Alamri A (2014) Efficient resource scheduling for big data processing in cloud platform. In: International conference on internet and distributed computing systems, pp 51–63
52.
go back to reference Kollenstart M, Harmsma E, Langius E, Andrikopoulos V, Lazovik A (2018) Adaptive provisioning of heterogeneous cloud resources for big data processing. Big Data Cogn Comput 2(3):1–18 Kollenstart M, Harmsma E, Langius E, Andrikopoulos V, Lazovik A (2018) Adaptive provisioning of heterogeneous cloud resources for big data processing. Big Data Cogn Comput 2(3):1–18
Metadata
Title
Cloud resource management using 3Vs of Internet of Big data streams
Authors
Navroop Kaur
Sandeep K. Sood
Prabal Verma
Publication date
03-06-2019
Publisher
Springer Vienna
Published in
Computing / Issue 6/2020
Print ISSN: 0010-485X
Electronic ISSN: 1436-5057
DOI
https://doi.org/10.1007/s00607-019-00732-5

Other articles of this Issue 6/2020

Computing 6/2020 Go to the issue

Premium Partner