Minimizing Resource Allocation for Cloud-Native Microservices

Cloud computing is widely used in the digital industry as a technology that enables cheap and easy deployment not only for online web services but also for big data processing, machine learning, and storing data for the long term. The cloud is backed by physical data centers worldwide that host virtual machines offered to customers and users. The cloud concept empowers users to replace their hardware with cloud server instances and creates a new economic model where the customer pays only for the usage and not for the entire hardware itself. However, decision-making in cloud environments can be complex due to the diversity in pricing models and service offerings. There are no rules of thumb as each customer could have a specific set of constraints and requirements of their cloud application when it comes to selecting the perfect cloud environment [1].

Microservices have recently gained striking popularity due to being highly maintainable and easy to develop. Microservices-based applications allow the deployment of each microservice (or component) to be in physically separated virtual machines if needed. Microservices, being separately manageable, have a massive advantage in scaling, which is one of the most critical features of the cloud context. Instead of launching multiple instances of a whole application, there is a possibility to only scale in or scale out a specific microservice component [2]. Hence creating microservices with the help of container technologies can result in robust and easy deployment with a small footprint. However, the goal is not only to create a small footprint application, but also to match the Quality of Service (QoS) of the original monolith performance-wise. When it comes to data-intensive applications, data throughput is the most crucial aspect. The components of the application communicate with each other via APIs. Wrongly configuring a microservice component where the data throughput is consequently low can result in a bottleneck in terms of the whole application’s performance. Scaling the bottleneck components until the QoS is met will mitigate the issue. The problem we tackle is how to correctly configure the resource provisioning of each and every microservice component so that the overall application performs as required.

To create a resource-efficient microservice, the most critical settings to tune are the CPU and memory limits that control the resource usage of a microservice instance. These limits can prevent components from using more resources than needed, and with the help of these limits, the app provider or operator makes sure in a private cloud that concurrent applications use the cluster as efficiently as possible, whereas, in a public cloud, limits help to pre-estimate costs. Setting the limits for several components at once raises a complex optimization problem when it comes to microservices. Application components have their tasks while working closely with other components, and if one of the components lags, the whole microservices-based application will suffer. On the other hand, resource provisioning must consider the shared resources: over-provisioning one component can cause starvation at other components. Furthermore, each component may have several configuration parameters which need to be tested in terms of resource usage to find the optimal limit values making the specific component work as efficiently as possible.

In this paper, we tackle this exact challenge: we propose an optimization framework that estimates resource usage vs performance of microservices-based, data-intensive applications with the help of machine learning, then minimizes the allocated resources given the QoS constraints. In our prior work [3] we tackled the optimization of CPU provisioning. The current manuscript has been extended with our approach for memory optimization, which is of paramount importance when it comes to cloud cost estimation.

The rest of the paper is organized as follows. Section 2 presents the related work, and in Sect. 3 we propose the methodology to solve the problem at hand. In Sect. 4 we introduce an illustrative example for microservice-based applications, Cortex, then we train regression models on the performance and resource consumption measurements data and create a linear programming formula for resource optimization, completed with numerical analysis. Section 5 concludes our findings.

Performance modeling of microservices-based applications allows us to determine the capacity distribution among each microservice. This enables planning for applications and the detection of the bottleneck in microservices. [4] proposes to apply statistical models, e.g., Theil-Sen estimator or Support Vector Regression, for this purpose. After analyzing the data acquired by their approach on the example applications, it was identified that the microservices follow a typical performance versus workload relationship pattern, which suggests the performance degrading with the increase of workload up until a certain point when all the virtual resources are used. In all tests, the CPU utilization increased linearly with the number of requests sent to the microservices. The approach was tested on several test applications, including a compute-intensive application, a database accessing an application, and a web accessing application. Similarly to [4] we also want to create a model to detect and avoid bottlenecks; however, we want to provision the resources individually for each microservice using as minimal resources as possible.

Zhang et al. [5] presented Sinan, a machine learning- driven resource manager for microservice-based applications. Sinan presents the challenges of managing complex microservices and leverages a set of scalable and validated models to reduce resource usage while meeting the end-to-end QoS. Sinan trains two models with the traces: a CNN (Convolutional Neural Network) model for short-term performance prediction and a Boosted Trees model to evaluate long-term performance evolution. Combining the two models allows Sinan to be effective in both near-future and distant future resource management. Similarly to [5], our goal is to meet the QoS requirements with an analytical model which can predict the expected average resource usage.

The detection of a bottleneck component is important if we want to increase the performance of a complex application deployed in the cloud. [6] presents an analytical model that can detect bottlenecks and predict the performance of a multi-tier application. The suggested approach consists of two resource provisioning steps: a predictive one for long-term scales and a reactive one for short timescales. Long-term prediction is useful when the load can be predicted, e.g., a typical daily pattern, but only reactive provisioning can handle unexpected high workloads. Using both and combining them can create an effective provisioning scheme. With this approach in a scenario where the workload of a three-tier application has been doubled, the technique showcased in [6] was able to double the application capacity within five minutes while maintaining the QoS targets. Unlike [6], our focus is to predict and create resource limits for the average resource usage of microservices of any cloud-native application.

Our model shows similarities to CostHat [7], an approach to model the deployment costs, including compute and IO costs. CostHat is a graph-based model of the deployment costs and can be used for applications implemented on top of AWS Lambda. Just like our results, [7] demonstrates that between the used features, there are linear dependencies. Similarly to [7] we focus on reducing costs if the model is used in a public cloud environment. However, our model calculates CPU and memory usage as output, which is extremely useful for private clouds.

Leitner et al. [7] presented a cost model to calculate and optimize the deployment cost of microservices. Their model takes into account several characteristics, e.g., the connection between services, the processing capacity and expected load. Results show that the proposed algorithm can calculate the cost in real-time, however the cost model does not consider the adaptation of the application: it is designed only for static application analysis. Ma et al. in [8] introduced a task scheduling cost optimization framework for cloud IoT applications, where they consider cost and deadline parameters. Their algorithm achieves state-of-the-art performance in simulations, on the other hand the main focus of the paper is on scheduling tasks to a given infrastructure, and does not consider scaling of the application. Authors of [9] proposed a reinforcement learning (RL)-based solution for horizontal and vertical resource provisioning. They use a cost function which considers the cost of scaling, running the application and violating SLA, and the agents choose their actions to minimize this cost. With enough training data and time, RL agents are able to find the optimal resource provisioning action in a given state. However, the state on which the agents are trained contains only the current resource and system information, therefore the agent can hardly understand the long-term effect of its actions. Authors of [10‐12] developed cost models and used them for cost-aware resource provisioning in edge-cloud environment [10] and for optimizing network functions in a telco environment [11, 12]. The model in [10] also considers the load prediction for the next time interval, however the applicability of the models for long-term traffic or resource allocation are not discussed. Indeed, workload prediction is being widely explored to solve issues such as resource under-utilization, load balancing and power consumption, using time series analysis regression and neural networks based models. The time series analysis based models are unable to capture the dynamics in the workload behavior whereas neural network based models offer better accuracy on the cost of high training time. [13] presents a workload prediction model based on extreme learning machines whose learning time is very low and forecasts the workload more accurately. The performance of the model is evaluated over two real world cloud server workloads i.e. CPU and Memory demand traces of Google cluster and compared with predictive models based on state-of-art techniques. It is observed that the proposed model outperforms the state-of-art techniques by reducing the mean prediction error up to 100% and 99% on CPU and memory request traces respectively. Comparing with the aforementioned works, our method achieves both short-term and long-term resource optimization considering the cost of running and scaling the cloud native application according to the load.

Our goal is to minimize the CPU limit provisioned for each component in a microservice-based application for a static load. To this end, we formulate mathematical programming problems with user-defined constraints. Then the result of the optimization yields the expected CPU usage based on which the resource limits of each microservice can be appropriately set. In order to formulate the optimization model, first, we need to find the most important features of the operation of the microservices, as well as their coefficients. Our approach is to build a regression model from a measurement dataset that incorporates the inference between configuration settings, application performance and resource footprint. Figure 1 shows the steps of our proposed approach.

The measurement dataset contains the monitored microservices’ resource usage. Assuming a data-intensive application at hand, there are typically two paths that must be monitored: the writing path (data ingestion) and the reading path (data retrieval). The cloud-native application is typically constituted by various microservices which are closely coupled together. These microservices (or components) can be customized with configuration parameters and settings. While measuring the effects of these settings and scaling in and out the components, a data collector pipeline stores the microservices’ resource usage and performance indicators. The important features that are later used in the data mining step can be, for example, the replica number of each component, the number of incoming service requests, or the number of served queries per second. After measuring several scenarios with various settings and analyzing the key differences between each setup, we propose to train a regression model.

We create a separate regression model for every main component of the application. Feature selection is important for keeping the model simple. Most of the potential features show linear dependencies with the resource usage, as pointed out in [4] and [7], therefore in this paper, we make a case for linear relationships so that we can later formulate the optimization with linear programming. By keeping only the highly correlating features, the linear programming solvers can find the solution quickly even if some variables are integers, turning the problem into non-deterministic polynomial-time (NP) complexity [14]. After training the regression model on the measurement dataset, given the coefficients of the regression, we can continue with linear programming. Note that in case the features exhibit a non-linear effect on the resource usage or on the application performance, then polynomial regression might be used, but then the respective optimization problem will not be a linear programming instance.

The decision variables are the important features from the regression model. Using these variables, one must create the algebraic expression which describes the target function; in our case, we strive to minimize the CPU resource usage. As the regression model is trained for precisely that dependent variable, the target function is produced by the regression formula: features as variables, their coefficients and the intercept. Based on the measurement data, one can create constraints on the features to be taken into account in the optimization problem. Typical constraints include but are not limited to high availability requirements on microservice instances, maximal memory consumption, application performance in terms of data throughput or rate of requests served. In general, it is beneficial to minimize the search space as much as possible, resulting in faster optimization. Non-negativity constraints can be applied for most of the features.

With the optimization model formulated, any solver can optimize the problem with the given user inputs and suggest the best possible setup for all the features. In the next section, we illustrate how this approach can save resources for the operation of a cloud-native microservices-based application.

Cortex [15] is a horizontally scalable, multi-tenant, long term storage for Prometheus [16] written in Go. It is a Cloud Native Computing Foundation (CNCF) project [17]. Cortex can receive metrics from multiple Prometheus servers and serve aggregated queries across all data in a single place which is useful when the cloud application operator wants to get an overview of all the metrics and data collected by these Prometheus servers. Our long-term storage choice was MinIO [18]. Figure 2 illustrates our setup with Cortex components.

Cortex consists of multiple horizontally scalable microservices from which we introduce the most important ones in terms of performance and resource footprint in this section. Figure 2 contains most of Cortex’s components, including the external applications that vary on the use case. Writing path is the microservice’s data ingestion process where the collected and gathered data gets stored in the long-term storage following pre-processing. On the other hand, the reading path is the process when the data gets retrieved from long-term storage. Some components are vital components for both paths. When it comes to the writing path, the key components are the Distributor and Ingester. The Ingesters have a similarly important role regarding the reading path. In addition to the Ingester and Distributor, we introduce the Querier, Query frontend (optional in deployment), Ingester, and the Store-gateway.

The Distributor microservice is responsible for handling the incoming samples of Prometheus. It is the first step in the writing path. When the Distributor receives samples from the Prometheus servers, each piece is checked and validated for correctness and to make sure that it is within the preconfigured tenant limits and falling back to preconfigured ones in case limits have not been overridden for the specific tenant. The valid samples are then split into batches and sent to multiple Ingesters in parallel by the Distributor.

The Ingester microservice is responsible for writing incoming time series to a long-term storage backend (in our case, into MinIO) on the writing path and returning samples stored in-memory for queries on the reading path.

The Querier microservice handles queries using the PromQL query language. The Queriers fetch samples both from the Ingesters and long-term storage solution: the Ingesters hold the in-memory series which have not yet been written to the long-term storage.

The Query frontend is an optional microservice implementing Querier’s API endpoints and used to accelerate the reading path. If the Query frontend is deployed, incoming query requests should be directed to a Query frontend rather than straight to the Queriers. The Querier microservice will still be required within the cluster for executing the queries. The Frontend does not query the necessary data. The Query frontend internally performs query adjustments and holds queries in an internal queue similarly to a database. The Store-gateway is the Cortex microservice responsible for query series from blocks; each block is composed of chunk files containing the timestamp-value pairs for multiple series and an index, which indexes the metric names and the labels.

In this paper we argued that resource provisioning is essential and, in fact, possible to perform scientifically when using cloud-native microservices. We proposed an approach that we demonstrated on a data ingestion and storage cloud-native application, Cortex. After measuring the performance vs resource footprint trade-offs with several configuration settings and creating a regression model for resource usage, we showed the most critical features and their quantitative effects on CPU utilization, which proved to be linear. Then we introduced an integer linear programming formulation that can be solved to minimize the expected CPU usage for given user inputs. With some illustrative examples, we showed that a prudent resource reservation might halve the costs paid for cloud resources while meeting the QoS constraints set for the application.