An energy, performance efficient resource consolidation scheme for heterogeneous cloud datacenters

https://doi.org/10.1016/j.jnca.2019.102497Get rights and content

Highlights

  • A consolidation method is suggested to manage datacenter resources – particularly, when container run inside VMs.

  • We consider various approaches to migrate VMs, containers and VMs plus containers (applications).

  • We model heterogeneity of datacenter resources (hosts), performance of VM, container migrations, as well as applications.

  • The “containerCloudSim” simulator is extended to make containers run inside VMs, perform VMs and containers migrations.

  • Using real workload traces, we suggest application migration more energy, performance efficient than containers and VMs.

Abstract

Datacenters are the principal electricity consumers for cloud computing that provide an IT backbone for today's business and economy. Numerous studies suggest that most of the servers, in the US datacenters, are idle or less-utilised, making it possible to save energy by using resource consolidation techniques. However, consolidation involves migrations of virtual machines, containers and/or applications, depending on the underlying virtualisation method; that can be expensive in terms of energy consumption and performance loss. In this paper, we: (a) propose a consolidation algorithm which favours the most effective migration among VMs, containers and applications; and (b) investigate how migration decisions should be made to save energy without any negative impact on the service performance. We demonstrate through a number of experiments, using the real workload traces for 800 hosts, approximately 1516 VMs, and more than million containers, how different approaches to migration, will impact on datacenter's energy consumption and performance. We suggest, using reasonable assumptions for datacenter set-up, that there is a trade-off involved between migrating containers and virtual machines. It is more performance efficient to migrate virtual machines; however, migrating containers could be more energy efficient than virtual machines. Moreover, migrating containerised applications, that run inside virtual machines, could lead to energy and performance efficient consolidation technique in large-scale datacenters. Our evaluation suggests that migrating applications could be ~5.5% more energy efficient and ~11.9% more performance efficient than VMs migration. Further, energy and performance efficient consolidation is ~14.6% energy and ~7.9% performance efficient than application migration. Finally, we generalise our results using several repeatable experiments over various workloads, resources and datacenter set-ups.

Introduction

Cloud computing and distributed systems are playing a key role in changing the information technology (IT) industry through offering on-demand and elastic provisioning of compute resources, such as processors, storage, networks, and applications. These are essentially supported through the formation of hyper-scale datacenters from where these resources are being offered. Subsequently, these datacenters consume considerable quantities of energy, along with associated cooling costs and, therefore, adding to substantial CO2 and green house gases (GHG) emissions. These datacenters consumed approximately 70 billion kWh of energy in 2014, which is equal to ~1.8% of the total energy consumption in the United States; and are projected to consume ~73 billion kWh by 2020, possibly, due to growth in mobile users, games and other on-line services, and Internet of Things (IoT) devices (Shehabi et al., 2016). National problems of energy supply, rising fuel costs, water complications, and global warming, bring the need for energy aware computation into sharp focus. The closing down of several nuclear power plants in countries like France and Germany, and depletion in coal-fired power plants in the United Kingdom, providing a projected energy safety margin (i.e. capacity to demand ration) of only 0.1% in 2017, carry the very real danger of load shedding and power outages (http://www.telegraph.co.uk). Due to increase in renewables, a slight increase in the UK energy safety margin can be seen in 2018 i.e. uptake from 29% to 36%. If we supposed comparable consumption rates to the United States of 1.8% of whole energy consumption (Shehabi et al., 2016), a 6% improvement in datacenter energy efficiency could double such a safety margin. Furthermore, Shehabi et al. (2016), describes that, possibly due to migration of organisational workloads from private systems (internal) to the public cloud, datacenters energy consumption will possibly remain unchanged till 2020.

These kinds of issues can be solved, partly, using techniques such as resource allocation, scheduling and consolidation i.e. efficient resource management (NRDC, 2014). Resource management techniques are reliant on the background technologies (e.g. virtualisation and containerisation – container-based virtualisation) which are widely used by cloud service providers to offer resources to IaaS (Infrastructure as a Service) customers. Virtualisation brings the concept of a VM (Virtual Machine) while containerisation describes the VM concept as a container; both run over virtualised servers. VMs have been extensively used in public clouds, particularly, the state-of-the-art in IaaS is widely aware with VMs. Cloud service providers such as Google, Microsoft Azure, and Amazon EC2 offer VMs services to their customers and also execute applications (workloads/services) within VMs. Moreover, various PaaS (Platform as a Service) and SaaS (Software as a Service) providers, such as Google App, Gmail, are made on top of IaaS where they run all their applications and workloads within VMs. Containerisation offers an alternative approach to virtualisation in the cloud i.e. containers; which are comparable to lightweight VMs because they share a single operating system (OS) kernel. Furthermore, by mixing containers with “traditional” VMs (run containers inside VMs), maximum utilisation of the resources are assured through consolidation. However, consolidation of VMs, containers and applications involves migrations that could be expensive in terms of extra energy consumption, loss in workload performance and, therefore, users monetary costs. Moreover, this additional cost is widely not assumed in numerous published models with the notable exception of Zakarya and Gillam (2016) – which suggests that it can be more energy and performance, hence, cost efficient either: (a) not to consolidate; (b) migrate VMs only; (c) migrate containers instead of VMs; or (d) migrate an application (code segment) running inside a container instead of migrating a container or a VM (as a whole).

The goal of the research, presented in this paper, is to explore further savings as may be possible through approaches such as efficient scheduling and consolidation. We explore how consolidation of VMs, containers and containerised applications, that run inside VMs, can help to decrease datacenter energy consumption whilst ensuring that performance of the workload is not affected, negatively. An algorithm for energy-performance efficient consolidation is suggested, implemented through modifications and extensions to two popular cloud simulation environments, CloudSim (Calheiros et al., 2011) and containerCloudSim (Piraghaj et al., 2017), and evaluated with respect to two large-scale datasets of real workload information from major cloud providers. This paper offers the following major contributions which, we believe, are noteworthy and have practical merit and potential impact:

  • a resource consolidation approach is presented to manage resources in a heterogeneous containerised datacenter – particularly when containerised applications, those run in containers, run inside VMs;

  • we consider various approaches to consolidate and migrate VMs, containers and VMs plus containers (applications);

  • several performance models are presented to model heterogeneity of datacenter resources (hosts), VMs and containers migrations, as well as applications, using two real datasets from Google and PlanetLab clouds, in a simulated platform;

  • the cloud simulator “containerCloudSim” was extended in order to make containers run inside VMs and to perform migrations of both VMs and containers. Moreover, continuing work would address, later, how a single (centralised) cloud broker would be able to predict energy, performance and cost-efficient consolidation opportunities through migrating either containers, VMs or both is a hybrid cloud platform; and

  • an empirical study to find out if application migration is cost friendly than VM or container migration.

The rest of the paper is organized as follows. An overview of service migration (containers, VMs and applications) and the technologies used to make it possible, is given in Sec. 2. In Sec. 3, we identify the research problem. Various assumptions, models and consolidation algorithms are proposed in Sec. 4. We validate container, VM and application migration using real workload traces from Google cluster (Reiss et al., 2011) and PlanetLab (Piraghaj et al., 2017) in Sec. 5; and show that migrating an application (both VMs and containers) to efficient hosts can reduce the datacenter energy consumption while providing the expected level of workload performance. From practical implementation point of view, an overview of several resource management systems is described in Sec. 6; that could help readers to understand how the proposed techniques would be implemented in a real cloud platform. We offer an overview of the related work in Sec. 7. Finally, Sec. 8 concludes this paper with future research directions.

Section snippets

Background

A cloud service is likely to have an operating system (OS) as well as application dependencies that need to be met by its hosting server. Therefore, applications are usually encapsulated into some pre-configured environments – for example, the famous hypervisor-based virtualisation (VMs), or comparatively the new containerisation technique (containers) which offers easy deployment and distribution (Machen et al., 2018). Virtualisation supports the creation and execution of numerous independent

Problem description

VM migration is largely discussed and implemented in the context of a cloud platform. However, container migration is a relatively new field and is not enough mature, in the existing literature. As containers often have smaller sizes than VMs, it can be beneficial to run container-based applications in emerging cloud platforms such as mobile clouds and MECs (mobile edge clouds) – with limited processing capability and storage. Existing VM migration methods for cloud platforms are live and

Proposed technique

In this section, we provide an overview of several energy consumption and performance models used in our empirical evaluation. We, then propose an energy-performance efficient consolidation technique, and a framework as shown in Fig. 2, that migrates applications running on VMs and containers both. Migration decisions are based on the energy estimation and performance prediction models which are explained in Sec. 4.1, Sec. 4.2 and Sec. 4.3, respectively. For every migratable entity such as VM,

Performance evaluation

The consolidation approach can be assumed as a type of bin-packing problem with respect to various sizes, costs of bins and items. The bins represent various resources (servers) and items denote containers and/or VMs for allocation. Furthermore, bin sizes may refer to resource (e.g. CPU, RAM) capacities of servers and costs represent the power consumption of servers. VM or container consolidation can be supposed as a multi-objective optimisation problem with the objective(s) to decrease the

Application consolidation: from a practical perspective

Despite the large volume of research available on VM consolidation with migrations, there are only few software tools available on-line that support consolidation and are used to design clouds. In the literature, the earliest open-source implementation of server consolidation is Entropy.17 A second framework for VM management in private clouds called Snooze.18 A third open-source implementation of OpenStackNeat,19

Related work

Continual investigation of energy efficient resource management will be necessary to help in addressing the key problems of national energy supply, rising fuel costs, water complications, and global warming, and to evade the need for datacenter outages and electricity load-shedding. Improvements to energy efficiency should decrease energy consumption, and therefore energy bills, and other costs related to energy infrastructures, with associated environmental aids. Moreover, performance of

Conclusions and future work

In this paper, we: (a) proposed a consolidation technique which favours the most effective migration that could be either a VM, a container or a particular application running inside a container; (b) modelled the heterogeneity of cloud applications and resources; and (c) demonstrated how consolidation of heterogeneous applications, containers and VMs affect the performance and energy efficiency of heterogeneous datacenters. Using plausible assumptions, models and various workload datasets, our

Acknowledgement

This work is supported in part by Abdul Wali Khan University Mardan (AWKUM), Pakistan, and in part by Higher Education Commission (HEC), Pakistan. We would like to thank all team members of the iFuture: a leading research group, at AWKUM, for their support and efforts during the experimental and evaluation phase.

Ayaz Ali Khan is currently a PhD student in the Department of Computer Science, Abdul Wali Khan University Mardan, Pakistan. He completed his M.Phil (MS) in computer science from COMSATS Institute of Information Technology (CIIT), Islamabad, Pakistan. His area of research includes energy-aware and performance-efficient scheduling, resource allocation, placement and management, at datacenter level. Moreover, he has enough knowledge of distributed systems, optimisation, game theory and computer

References (67)

  • Bandar Aldawsari et al.

    Trusted energy efficient cloud-based services brokerage platform

    Int. J. Intell. Comput. Res.

    (2015)
  • Ibrahim Alzamil et al.

    Energy prediction for cloud workload patterns

  • Anton Beloglazov et al.

    Optimal online deterministic algorithms and adaptive heuristics for energy and performance efficient dynamic consolidation of virtual machines in cloud data centers

    Concurrency Comput. Pract. Ex.

    (2012)
  • Anton Beloglazov et al.

    Openstack neat: a framework for dynamic and energy-efficient consolidation of virtual machines in openstack clouds

    Concurrency Comput. Pract. Ex.

    (2015)
  • William Lloyd Bircher et al.

    Complete system power estimation using processor performance events

    IEEE Trans. Comput.

    (2012)
  • Rodrigo N. Calheiros et al.

    Cloudsim: a toolkit for modeling and simulation of cloud computing environments and evaluation of resource provisioning algorithms

    Softw. Pract. Exp.

    (2011)
  • Maxime Colmant et al.

    Process-level power estimation in vm-based systems

  • Eli Cortez et al.

    Resource central: understanding and predicting workloads for improved resource management in large cloud platforms

  • Miyuru Dayarathna et al.

    Data center energy consumption modeling: a survey

    IEEE Commun. Surv. Tutor.

    (2015)
  • Vincenzo De Maio et al.

    A workload-aware energy model for virtual machine migration

  • Ranjan Dhar

    Comparative Evaluation of Virtual Environments: Virtual Machines and Containers

    (2016)
  • Diamanti

    2018 Container Adoption Benchmark Survey

    (2018)
  • Diamanti

    Five Reasons You Should Run Containers on Bare Metal, Not Vms

    (2018)
  • Wes Felter et al.

    An updated performance comparison of virtual machines and linux containers

  • Lee Gillam

    Will cloud gain an edge, or, closer, to the edge

  • Lee Gillam et al.

    Exploring edges for connected and autonomous driving

  • C. Jiang et al.

    Energy efficiency comparison of hypervisors

    Sustain. Comput.: Inf. Syst.

    (2017)
  • Aman Kansal et al.

    Virtual machine power metering and provisioning

  • A.A. Khan et al.

    An energy and performance aware consolidation technique for containerized datacenters

    IEEE Trans. Cloud Comput.

    (2019)
  • A.A. Khan et al.

    H2a hybrid heterogeneity aware resource orchestrator for cloud platforms

    IEEE Syst. J.

    (2019)
  • V. Kherbache et al.

    Scheduling live migration of virtual machines

    IEEE Trans. Cloud Comput.

    (2017)
  • Sai Venkat Naresh Kotikalapudi

    Comparing Live Migration between Linux Containers and Kernel Virtual Machine: Investigation Study in Terms of Parameters

    (2017)
  • Cited by (0)

    Ayaz Ali Khan is currently a PhD student in the Department of Computer Science, Abdul Wali Khan University Mardan, Pakistan. He completed his M.Phil (MS) in computer science from COMSATS Institute of Information Technology (CIIT), Islamabad, Pakistan. His area of research includes energy-aware and performance-efficient scheduling, resource allocation, placement and management, at datacenter level. Moreover, he has enough knowledge of distributed systems, optimisation, game theory and computer programming. His work has been published in reputed internal journals.

    Muhammad Zakarya received the PhD degree in computer science from the University of Surrey, Guildford, U.K. He is currently a Lecturer with the Department of Computer Science, Abdul Wali Khan University Mardan, Pakistan. His research interests include cloud computing, mobile edge clouds, performance, energy efficiency, algorithms, and resource management. He has deep understanding of the theoretical computer science and data analysis. Furthermore, he also owns deep understanding of various statistical techniques which are, largely, used in applied research. His research has been appeared in several international journals of repute and conferences.

    Rahim Khan received the PhD degree in computer science from the Ghulam Ishaq Khan Institute (GIKI), Swabi, Pakistan. He is currently an Assistant Professor with the Department of Computer Science, Abdul Wali Khan University Mardan, Pakistan. His research interest includes the Wireless Sensor Networks (WSNs) deployment, Internet of Thing (IoT), routing protocols, outliers' detection, techniques for congestion control, Decision Support System (DSS), vehicular ad-hoc networks, data analysis and similarity measures.

    Izaz Ur Rahman received the PhD degree in computer science from the Department of Electronic and Computer Engineering, Brunel University, UK. He is currently an Assistant Professor with the Department of Computer Science, Abdul Wali Khan University Mardan, Pakistan. His research interest includes power systems, optimisation algorithms, internet of things and artificial intelligence.

    Mukhtaj Khan received the PhD degree in computer science from the Department of Electronic and Computer Engineering, Brunel University, UK. He is currently an Assistant Professor with the Department of Computer Science, Abdul Wali Khan University Mardan, Pakistan. His research interest includes Big data analytics, smart grids, cloud computing and distributed systems. Moreover, he owns deep understanding over the performance modelling of Hadoop systems.

    Atta ur Rehman Khan is an Associate Professor at Faculty of Computing and Information Technology, Sohar University Oman. He has extensive experience in teaching, research, and industry at key positions. He holds a PhD from University of Malaya, Kuala Lumpur, Malaysia. He has completed renowned projects, published research articles in reputed journals/conferences, and edited/co-authored multiple books. Currently, he is an Associate Editor of IEEE Access, Elsevier Journal of Network and Computer Applications (JNCA), Associate Technical Editor of IEEE Communications Magazine, Editor of Springer Journal of Cluster Computing, Oxford Computer Journal, IEEE SDN Newsletter, SpringerOpen Human-centric Computing and Information Sciences, SpringerPlus, and Ad hoc & Sensor Wireless Networks journal. He is a Senior Member of IEEE and Professional Member of ACM.

    View full text