A big data placement method using NSGA-III in meteorological cloud platform

In this paper, we focus on a big data-oriented MCP that provides flexible resources for storing meteorological data. Suppose that there are N meteorological datasets in the current MCP that need to be placed on the physical machines, denoted as D={d₁,d₂,…,d_N}, and M tasks have arrived at the MCP to be performed, denoted as T={t₁,t₂,…,t_M}. A task may require multiple meteorological datasets to perform, and a meteorological dataset may also be accessed by multiple tasks. Since the number of pods is p in the MCP, thus there are p³/4 physical machines provided to place the datasets and the tasks. Assuming W is the total number of physical machines, W=p³/4, then the physical machines in the MCP can be represented by PM={pm₁,pm₂,…,pm_W}. Hence, set X={x₁,x₂,…,x_N} be the collection of the placement strategy for each meteorological dataset, and each placement strategy x_n∈PM(1≤n≤N). Similarly, set Y={y₁,y₂,…,y_M} be the collection of the placement location of each task set, and each placement location y_m∈PM(1≤m≤M). Therefore, this paper will solve the placement strategy X of meteorological dataset according to the known task placement strategy Y. Meteorological data refers to information elements or numerical analysis results collected or processed by all possible means of observation, detection, and telemetry, which come from the Earth’s atmosphere and its adjacent layers and are related to the law of atmospheric state change.

According to the types of data in meteorological information system, meteorological data can be divided into two categories which are the meteorological data and industrial social data [6]. Meteorological data, including atmospheric data and surface data, are the main data resources for studying and forecasting meteorological information, and their data volume is relatively large, while industrial social data mainly include geographic information, national economy and other data, which provide assistance for the analysis of meteorological information. Because the industrial social data play a restrictive and auxiliary role in meteorological analysis, the industrial social data should be placed on the physical machine where the task set is requested, which means x_n=y_m. Using binary flag ξ_n to represent if d_n is the industrial social data, thus

In MCP, task sets and datasets are managed by virtual machines placed on physical machines. The resources required by datasets and task sets and the capacity of physical machines can be expressed by the number of virtual machine instances. Suppose c_w be the capacity of the wth physical machine pm_w.

Resource utilization is an important metric to manage the utilization of MCP resources. According to the placement strategy X of meteorological datasets, the resource utilization of each physical machine can be obtained by detecting the resource utilization of virtual machine instances. Suppose u_w(X) be the resource utilization of the wth physical machine under the placement strategy X, which can be calculated by

where θ_n represents the resource demand of meteorological dataset d_n, and I_n,w(X) is the binary flag to judge whether d_n is placed on pm_w, thus

Then, the overall resource utilization of MCP refers to the usage status of all occupied physical machines. Suppose λ(X) be the number of employed physical machines, which can be calculated by

where F_w(X) is the binary flag to judge whether pm_w is employed, thus

Then, the average resource utilization of the MCP can be obtained by using the following formula

In the MCP, the task set needs to be implemented by the meteorological dataset deployed on the physical machine. Hence, when considering the big data distribution strategy of meteorological cloud platform, it is necessary to consider the data access time of task set accessing meteorological dataset.

Suppose l_m,n is the binary flag to judge whether t_m needs to access d_n, thus

Suppose g_m,n is the access frequency of t_m access to d_n in a task execution cycle R, then the total access frequency K in a period of execution cycle can be calculated by the following formula

In the fat-tree topology, the number of exchanges occurring in data transmission is closely related to the distribution of datasets and task sets. Suppose ES(pm) be the edge switch connected to the physical machines and Pod(pm) represents the pod of this edge switch. Then, the number of exchanges in data transmission can be divided into four cases, respectively: (1) t_m and d_n are placed on the same physical machine, which means x_n=y_m. (2) t_m and d_n are placed on the different physical machines but connected to the same edge switch, which means x_n≠y_m,ES(x_n)=ES(y_m). (3) The physical machines of t_m and d_n belongs to different edge switches but the same pod, which means ES(x_n)≠ES(y_m),Pod(x_n)=Pod(y_m). (4) The physical machines of t_m and d_n belong to different pods, which means Pod(x_n)≠Pod(y_m). Based on the analysis above, the access time t_m,n(X) of t_m to access d_n once can be calculated by

where ρ_n represents the data volume of d_n and b_PE represents the bandwidth between physical machines and edge switches; similarly, b_EA represents the bandwidth between edge switches and aggregation switches and b_AC represents the bandwidth between aggregation switches and the core switches. Then, the average data access time T(X) of the MCP can be calculated using the following formula

In this paper, we focus on the energy consumption of physical machines, virtual machines, and switches in the process of data layout. First of all, all physical machines consume a certain amount of energy to run. We call this energy consumption the basic energy consumption of physical machines, which can be calculated as

where α_w is the basic energy rate of pm_w.

Virtual machine also consumes a certain amount of energy. Its energy consumption can be divided into two situations. If the virtual machine instance is occupied by datasets, it will consume more energy. We call this energy consumption as active virtual machine energy consumption, which can be calculated according to the following formula

where η is the active energy rate of the virtual machines. And the virtual machine instances that are not occupied by meteorological datasets will consume less energy, which we call it the idle virtual machine consumption, denoted as IE, can be calculated by

Therefore, the virtual machine consumption in the MCP can be expressed as

In addition, the energy consumption of switches due to data transmission needs to be considered. In the fat-tree topology model, the total number of switches is 5/4p², in addition, the number of switches passed by t_m to access d_n, denoted as δ_m,n(X), can be calculated by

Hence, the energy consumption of the switches SE can be calculated by

where β is the basic energy rate of the switches and γ is the working energy rate of the switches.

Therefore, the overall energy consumption of meteorological cloud platform can be calculated by

The focus of our research is to solve the problem of big data placement in the environment of MCP. On the basis of known task set placement strategy, we optimize the placement strategy of meteorological dataset in order to improve resource utilization, reduce data access time and energy consumption.

After the above analysis, the multi-objective problem in this paper can be expressed as

A data placement strategy in MCP is proposed in this subsection. The data placement problem is defined as a multi-objective optimization problem. NSGA-III is an accurate and robust method for addressing the optimization problem with multiple objectives. Here, NSGA-III is adopted to solve the multi-objective optimization problem presented in [18].

Firstly, we encode for the physical machines and then give the fitness function and constraints for the multi-objective optimization problem. In addition, we generate new chromosomes using the crossover and mutation operations. At last, we pick out the chromosomes of the next generation, using usual domination principle and reference-point-based selection. The overview of the multi-objective data placement is elaborated.

We select the chromosomes for the next generation in the selection phase with higher fitness values. We encode for the physical machines and each chromosome represents all the data placement strategies of the meteorological datasets. As is discussed above, the number of fitness functions for each chromosome is 3, i.e., the resource utilization, the data access time, and the energy consumption. After crossover and mutation operations are conducted, S chromosomes are generated and the size of population becomes 2S. We aim at select S chromosomes for the next generation.

Firstly, the three objectives are evaluated according to [6], [10], and [17]. Then, usual domination principle is conducted to sort the 2S solutions using the three fitness values to generate some non-dominated fronts. Suppose that there are n non-dominated fronts, and the datasets in the first front have the most optimal fitness values.

We select one chromosome at random from the first front to the nth fronts every time until S chromosomes have been picked out. Assume the last chosen solution is in the fth front and the number of the selected chromosomes in the fth front is s. If the number of solutions in the fth non-dominated front is s, that is all the individuals in the fth front are selected, then the selection phase finishes.

However, if part of the chromosomes in the fth front are chosen, then further selection is required to determine the chromosomes in the fth front going into the next chromosomes.

In further selection, the three fitness functions of each individual in the generation are normalized. The maximization of the resource utilization, the minimization of the data access time, and the energy consumption for the population are represented as U^∗(X),T^∗(X), and E^∗(X) respectively. We search U^∗(X),T^∗(X), and E^∗(X) in the population firstly. Then the fitness functions are updated as

U_m,T_m, and E_m represent the extreme values of the average resource utilization, the data access time, and the energy consumption respectively which are calculated by

where W_U,W_T, and W_E are the weight vectors to the three functions.

Assume each fitness function as an axis and the intercepts of each axis can be determined according to the three extreme values, denoted as ρ_U,ρ_T, and ρ_E respectively. Then, the fitness functions are normalized by

After normalization, the fitness values of the resource utilization, the data access time, and the energy consumption are between zero and one. The normalized solutions are scattered in the hyperplane composed by the three fitness axes. Each solution of the individual in the population is corresponding to a triple.

Divide each axis into g subsections, and then, the number of the reference points, denoted as ζ, is calculated by

ζ is approximately equal to S to guarantee each individual associate with a reference point. Then, we associate the normalized solutions with the reference points.

We sort the solutions in the fth non-dominated front, using the number of the associated reference points. Assume the solutions in the fth front are sorted into d levels and the reference points in the first level represents the maximum reference points. Each time selects one solution randomly from the first level until the n individuals have been selected.

The proposed strategy aims at achieving a trade-off in optimizing the average resource utilization, the data access time of the tasks, and the energy consumption of MCP. In each population, there are N chromosomes and each chromosome x_n represents a hybrid data placement of N meteorological datasets. In addition, dynamic schedule of datasets is considered and, to select relatively better schedule solution of each placement strategy, simple additive weighting (SAW) and multiple criteria decision making (MCDM) are employed.

The resource utilization is a positive criterion, i.e., the higher the resource utilization is, the better the solution becomes. Oppositely, the data access time and the energy consumption are the negative criterion. We normalize the resource utilization in the nth placement strategy as

where U^max and U^min represent the maximum and minimum fitness for resource utilization in the nth placement strategy. Similarly, the time consumption of nth placement strategy is normalized as

where T^max and T^min represent the maximum and minimum fitness for data access time in the nth placement strategy. And the energy consumption of nth placement strategy is normalized as

where E^max and E^min represent the maximum and minimum fitness for energy consumption in the nth placement strategy. In addition, to calculate the utility value of each solution, the weight of each objective function requires determination.

In this paper, we do an overall consideration of three objectives for each strategy. The utility value of the nth placement strategy is calculated by

where UV(x_n) represents the utility value of the nth placement strategy and κ_U,κ_T, and κ_E represent the weight of the resource utilization, the data access time, and the energy consumption respectively. Therefore, for each chromosome in the population, we have calculated the utility value of the same schedule. The formalized optimization problem given in (18) can be represent by

The solution selection strategy is elaborated in Algorithm 1. We select the maximum and minimum value of the three objections (Lines 1-6). Then the utility values of the three objections and the solutions are evaluated (Lines 7-12). The maximum utility value of solution is selected (Line 13). Finally, the solution with maximum utility value is output.

In this paper, we aim to optimize the resource utilization, reduce the data access time, and decrease the energy consumption, which is determined as a multi-objective optimization problem. The NSGA-III is adopted to achieve the global optimal data placement strategy. First, we encode for the physical machines and each gene represents the data placement strategy for a meteorological dataset. Then, the fitness functions and constraints of the multi-objective optimization problem are elaborated. Crossover and mutation operations are conducted to generate new chromosomes and usual domination principle and reference-point-based selection in NSGA-III are adopted to select the individuals going into the next generation.

The overview of the proposed method is elaborated in Algorithm 2. We input the tasks, the physical machines, and the placement strategy of the tasks. The algorithm starts with the first iteration, and for each iteration, we conduct the crossover and mutation operation to generate new solutions (Line 3). Then, we evaluate the fitness functions of each chromosomes and obtain the placement routing (Lines 5). We conduct the selection, including the non-dominant sorting, the primary selection, and the further selection (Lines 7-13). When the algorithm reaches the maximum iteration, we evaluate the solutions by SAW and MCDM (Line 17). Finally, the optimal data placement methods are output.

In our simulation, six datasets with different scales of the tasks are applied for our experiments, and the number of datasets is set to 200, 400, 600, 800, 1000, and 1200 and the industrial social datasets is set to 50, 100, 150, 200, 250, and 300 in each scale of the meteorological datasets. The specified parameter settings in this experiment are illustrated in Table 1 [25].

To conduct the comparison analysis, we employ some other basic data placement method besides our DPMM. The comparative methods are briefly expounded as follows:

• Benchmark: In this method, the meteorological datasets are placed in the order of the physical machines. When the former physical machine is full, the datasets left are placed on the next physical machine. This process is repeated until all datasets have been placed.

• First fit decreasing in MCP (FFD-M): The meteorological datasets are sorted in descending order according to the dataset requests first. Then, the sorted datasets are placed on the physical machines. If the left resources of current physical machine are insufficient for the resource requirement of a dataset, the dataset is placed on the physical machine with enough resource. This process is repeated until all datasets have been placed.

• Best fit decreasing in MCP (BFD-M): The meteorological datasets and the physical machines are both sorted in descending order according to the dataset request and the space of physical machines first. Then, the sorted datasets are placed on the sorted physical machines. If the left resources of the physical machine are insufficient for the resource requirement of the dataset, the dataset is placed on the physical machine with enough resource next to this physical machine in optimal principle. This process is repeated until all datasets have been placed.

The data placement methods are implemented on a personal computer with Intel Core i7-4720HQ 3.60 GHz processors and 4 GB RAM.

The proposed DPMM is intended to achieve a trade-off with optimizing the resource utilization while reducing the data access time and the energy consumption. We conducted 50 experiments in the case of convergence for each dataset scale, and multiple sets of results are obtained.

Figure 5 shows the comparison of the utility value of the solutions generated by DPMM at different dataset scales. It is illustrated that when the dataset scale is 200, 400, 600, 800, 1000, and 1200, solutions generated by DPMM are 3, 2, 3, 4, 3, and 3 respectively. For the solutions generated by DPMM, we attempt to obtain the most balanced data placement strategy by judging the utility value given in [28]. After statistics and analysis, the solution with the maximum utility value is considered as the most balanced strategy. For instance, in Fig. 5a, the final selected strategy is solution 3 because it achieves the highest utility value.

In this subsection, the comparisons of Benchmark, FFD-M, BFD-M, and DPMM with the same experimental context are analyzed in detail. The resource utilization, the data access time, and the energy consumption are the main metrics for evaluating the performance of the data placement method. The corresponding results are shown in Figs. 6, 7, and 8.

(1) Comparison of resource utilization: After placing all the meteorological datasets on the physical machines via the data placement methods, the occupation of the VM instances is achieved. Figure 6 shows the comparison of the resource utilization in MCP of Benchmark, FFD-M, BFD-M, and DPMM at different dataset scales. The resource utilization is calculated according to the number of employed physical machines and the employed VM instances in each physical machine. Fewer employed physical machines with more occupied VM instances contribute to a higher resource utilization. It is intuitive from Fig. 6 that our proposed method DPMM achieves higher and stable resource utilization. That is, DPMM reduces the number of unemployed VM instances and wastes less resources than other data placement method.

(2) Comparison of data access time: Fig. 7 shows the comparison of the data access time of the meteorological tasks of Benchmark, FFD-M, BFD-M, and DPMM at different dataset scales. In this paper, we try to realize a balance between the resource utilization, the data access time and the energy consumption; however, in order to improve the resource utilization and reduce the energy consumption, the meteorological datasets may place on the physical machine which may be far from the tasks. It is illustrated from Fig. 7 that the data access time of our proposed method DPMM is a little more than the other data placement method, which means our proposed method may sacrifice some data access time to optimize the resource utilization and the energy consumption.

(3) Comparison of energy consumption: As the energy consumption is the sign of the cost the MCP needs to give out, it becomes the most important metrics that the manager of the MCP concerned. Figure 8 shows the comparison of energy consumption in MCP of Benchmark, FFD-M, BFD-M, and DPMM at different dataset scales. When the number of the datasets is small, the difference of the energy consumption for those four data placement methods are not obvious, but with the increase of the dataset scales, the energy consumption of DPMM is better than the other data placement method, which means that our proposed method DPMM may well be applied in the big data environment like MCP.