Scalable design and algorithm for science DMZ by considering the nature of research traffic

A high degree of complexity in big-science research requires close collaborations between researchers [1], where the performance of the data-transfer network strongly affects research progress. The scientific research data flows show the unique nature which is far different from that of commercial flows [2]. In the scientific research network, a limited number of researchers transfer a huge amounts of research data. The majority of research flows are terabyte- and petabyte-scale experiments and observation data, which are sensitive to networking performance statistics, such as throughput, packet loss, and delays. However, in the commercial general-purpose network, kilobyte- and megabyte-scale data of diverse applications from public users comprise the major flow. To serve such commercial flows efficiently, generality is regarded as one of the most important virtues in the general-purpose network. Complex deployments of diverse networking devices in the general-purpose network introduce this generality at the cost of functional overhead and redundancy, which can cause performance degradation. An unprecedented increase in traffic in the big-science research communities and aforementioned differences between commercial and research flows raise a fundamental question: Can the current general-purpose networking technology be optimized for the nature of big-science research data?

The Energy Sciences Network (ESnet) addresses this question by developing the science demilitarization zone (DMZ) [3, 4]. The Science DMZ overcomes physical limitations in the general-purpose network by providing a friction-free network path for scientific research data transfer. The use of the Science DMZ is limited to authorized researchers. To address the nature of research flow, the Science DMZ is physically separated from the general-purpose network and commercial applications are prohibited in the Science DMZ. Its authorized users and physical isolation enable the Science DMZ to form a trustworthy closed network, thus allowing firewalls to be avoided, as they are major obstacles to achieving a high throughput in the network [2]. The access control list (ACL) [5] provides security in the Science DMZ. To do so, the IP address of the authorized researchers should be registered with the ACL in advance. Similarly, to avoid any performance degradation of the networking equipment, it is recommended that the amount of networking equipment is minimized through the end-to-end path in the Science DMZ [2]. The Science DMZ deploys data-transfer nodes (DTNs) as end devices, which directly affect the performance of the end-to-end data transfer. The role of a DTN is restricted to research data transfer only, so all the computing resources concentrate on data transfer [4]. The transport control protocol (TCP) is the most widely used protocol on the Internet, where the upper bound of a throughput is expressed as MSS/RTT \(\sqrt L\), where MSS, RTT, and L are the maximum segment size, round trip time, and packet loss, respectively [6]. The friction-free path in the Science DMZ effectively minimizes L and RTT. To maximize the upper bound of TCP throughput, a large MSS is preferred in the Science DMZ, such as Ethernet jumbo frame [7]. Burst flows require a large size of buffer in networking equipment, which can cause a buffer-bloat problem [8]. However, since the well-optimized Science DMZ guarantees throughput close to its physical capacity [2], we regard the buffer-bloat problem in the Science DMZ negligible.

Korea Institute of Science and Technology Information has been operating KREONET since 1988. KREONET is a unique networking infrastructure specialized for practical scientific research data transfer in South Korea [20]. We have monitored the research traffic of 25 laboratories over KREONET from July 25 to September 25, 2019. The laboratories are representative research groups, who engage in collaborative research for significant big-science problems. Figure 2 depicts the inbound and outbound research traffic of six randomly selected laboratories located in different cities in South Korea. The research traffic of three astronomy laboratories is illustrated in Fig. 2a–c. Figure 2d, e, f shows climate, high energy, and supercomputing research laboratories, respectively. The amount of traffic in Fig. 2 is calculated as the cumulative amount of traffic (bits) over a 5-min interval, divided by 300 s and plotted in 5-min intervals. Note that the traffic data in Fig. 2 are from randomly selected research laboratories and thus do not represent the nature of traffic in any specific research field.

As shown in Fig. 2a, Astronomy_a shows two huge amounts of inbound traffic at July 28 and August 10, respectively. One can also find a huge amount of outbound traffic from Astronomy_b in Fig. 2b. Because the traffic amount and date coincide between the first inbound of Astronomy_a and outbound of Astronomy_b, we can interpret that there was a data transfer in astronomy community, from Astronomy_b to Astronomy_a at July 28. Both inbound and outbound traffic show a periodicity sawtooth-like pattern with some burstiness in Astronomy_c. As shown in Fig. 2d, outbound traffic from the climate laboratory is relatively stable, where a small amount of traffic continuously flows out from the laboratory. However, high variations in inbound traffic are observed with an order of magnitude difference between peak and average. Burstiness is observed in both the inbound and outbound traffic of the high-energy research group in Fig. 2e. The inbound traffic of a supercomputing research laboratory is relatively small, whereas its outbound traffic fluctuates highly, as illustrated in Fig. 2f. The research traffic over KREONET widely shows burstiness. Specifically, burst transactions rarely present time correlations between different research groups.

Initiated by the monumental works in [21], the burstiness of network traffic is observed [22‐24] and analyzed [25‐28]. The burstiness of the network traffic requires careful considerations in network dimensioning. The burstiness of network traffic over a wide range of time scale exhibits heavy-tail distribution which decreases with a hyperbolical function, namely slower than exponential function [28]. Accordingly, the tail in the heavy-tail distribution is heavier than that in the exponential distribution. Because a cumulative density function (CDF) of an exponential distribution is expressed as 1 − e^−λx with a random variable x and mean value λ, taking a logarithm of the complementary CDF (CCDF) of the exponential distribution produces a linear graph with a slope of − λ. In a log–log plot of CCDF, a heavy-tail distribution forms a linear graph [28]. Figure 3 describes the log-linear and log–log plots of the CCDF for incoming traffic of Astronomy_a illustrated in Fig. 2a. A log-linear plot of the CCDF of Astronomy_a incoming traffic shows slow decrease in 98% of traffic and sudden sharp decrease in the top 2% of traffic. Similarly, a combination of two linear graphs fits the log–log plot of CCDF with a slow linear slope (− 0.23) for 98% of traffic and a fast-linear slope (− 27.64) for top 2% incoming traffic of Astronomy_a. “Appendix” summarizes the log-linear and log–log plots of the research traffic for all six research laboratories shown in Fig. 2.

Table 1 summarizes the analysis results of the research traffic shown in Fig. 2. We analyze the average value, maximum value, ratio of top-20% traffic to total traffic, ratio of the time for 80% of the traffic to the total time, and approximation model. The Pareto distribution is a typical example of a heavy-tail distribution, which holds an 80:20 rule; 80% of the total traffic belongs to 20% of the total time [29]. Degrees of bias of both the incoming and outgoing traffic of Astronomy_a exceed the 80:20 rule: the top 20% of time holds 99% and 96% of total traffic in the incoming and outgoing traffic, respectively. The 80:20 rule fits the incoming traffic of the high-energy laboratory well, whereas it fails for some research traffic, such as that of the climate laboratory. We fit each research traffic with diverse distribution models, namely Exponential, Pareto (shape parameter, scale parameter), Lognormal (mean, variance), and Weibull (shape parameter, scale parameter). Table 1 summarizes the best approximation model for each type of research traffic based on the goodness-of-fit of the Kolmogorov–Smirnov test [30]. From the best approximation model, for example, we can interpret that the outgoing traffic of Astronomy_a shows heavy-tail distribution, as the Weibull distribution exhibits heavy-tail property when its shape parameter is lower than 1 [31]. The fittings are determined by EasyFit Software [32].

The Science DMZ consists of a combination of network equipment, such as the DTN, switch, monitoring tool, security appliances, and interconnection of legacy networks [4]. The DTN is an essential component in the Science DMZ for high performance of end-to-end data transfer. Regarding DTN deployment scenario, one simple design of Science DMZ is that using dedicated DTNs, where a DTN is allocated per user, as shown in Fig. 4a. This paper uses the terms “lab” and “user” interchangeably. To introduce higher scalability, we propose a design of the Science DMZ with shared DTN, as shown in Fig. 4b, by tailoring the nature of research traffic. For simplicity, Fig. 4 omits the monitoring tool, security appliances, and interconnection of the legacy network, as they do not differ much in each scenario. There are various end hosts of the Science DMZ, including research laboratory, local host, and supercomputer [4].

Due to the uncertainty of network traffic, a dimensioning solution derived from the past traffic cannot guarantee the optimality for the future traffic. Although we allow an infeasible assumption that future traffic information is known in advance, finding the minimum-CAPEX solution of a dimensioning problem in the Science DMZ with shared DTN is impractical. The number of permutations of |U| objects to |D| indistinguishable partitions is analyzed in [33]. Because |D| in our dimensioning problem can vary from 1 to |U|, the search space to minimize (2) span ofwhere N_(k=K) denotes the number of partitions whose cardinality k is equal to K.

The complexity of the network control algorithm is another important measure of its scalability in the network. To find a practical dimensioning solution, Fig. 5 suggests an iterative greedy heuristic algorithm. The algorithm decouples the DTN number decision problem from the DTN–user mapping problem and solves one by one with a greedy manner. These procedures repeat iteratively until an acceptable CAPEX solution is found. The final solution is determined by comparing CAPEXs between iterations.

This section analyzes the CAPEX for DTNs and the computation time of algorithms to evaluate the scalability of the proposed Science DMZ design, as well as the dimensioning algorithm. As a benchmark solution, exhaustive search-based minimum-CAPEX algorithm for the Science DMZ with shared DTN is compared. Because the genetic algorithm is widely used for a set-partitioning problem, we also evaluate the performance of the crossover-based genetic algorithm [36]. In the genetic algorithm, an initial individual is randomly generated as a matrix form (|D| × |U|) with binary elements. We assume that the ith DTN is assigned for the jth user if the element of the matrix at (i, j) is 1. Uniform crossover inside each column in the individual generates offspring where a probability of uniform crossover operator is fixed as 0.5. The fitness function is defined as 1/CAPEX. The algorithm terminates when the number of generations reaches 1000. Because the randomly generated initial solution affects the final solution in the genetic algorithm, the performance is averaged over 1000 runs. In this section, we use the monitored traffic in Sect. 2 as the input of the algorithms.

Figure 6 compares C_dedicated and C_shared as a function of the number of users for 2 months of research traffic with α = 2. The dimensioning problem in the Science DMZ with shared DTN is solved by minimum-CAPEX, suggested, and genetic-based algorithms. Among the 25 laboratories in Sect. 2, a set of users is selected by descending order of their peak traffic workloads. Because of this user selection rule, an increase in |U| results in rapid increases in C_dedicated and C_shared when |U| is small. Because the shared DTN can take advantage of traffic multiplexing between users, the increase in C_shared is negligibly small when |U| is large. However, the dedicated DTN requires |D| = |U|, and thus, C_dedicated constantly increases with respect to the increase in |U|. The proposed algorithm requires at most 8% C_shared overhead from that of the minimum-CAPEX solution. The minimum-CAPEX solution becomes intractable when |U| is larger than 18. The genetic and suggested algorithms show similar C_shared in the entire range of |U|. The suggested algorithm requires at most 2% overhead in C_shared from that of the genetic algorithm when |U| is 10. For the genetic algorithm, the number of DTNs is determined by the DTN number decision algorithm in Sect. 4.1.

Figure 7 compares computation time of each algorithms for the Science DMZ with shared DTN when |U| = 15. By decoupling problems as well as adopting iteration and greedy approaches, the suggested algorithm consumes 0.02 s, whereas the genetic-based heuristic and exhaustive search-based minimum-CAPEX algorithms take 7 and 1200 s, respectively. The suggested algorithm reduces the computation time by 2 and 5 orders of magnitude from those of genetic and the minimum algorithm, respectively, at the cost of negligible CAPEX overhead. The simulation result manifests a high scalability of the shared DTN with suggested dimensioning algorithm both in network design and control complexity. All simulations are performed in MATLAB on a laptop with a 1.99-GHz quad-core CPU and 8-GB memory.

Table 2 summarizes the calculation results of |D| of the suggested and minimum-CAPEX algorithms. Determining α is an important problem which highly affects both network performance and CAPEX. A small value of |D| is preferred as long as α lies in a small value. As shown in Fig. 1, a price of DTN scales exponentially with respect to α, and thus, the suggested algorithm tries to increase |D| when α becomes large. To evaluate the performance of dimensioning algorithms with respect to the period of the input traffic, we consider three periods of input traffic, which are subsets of the 2 months of monitored traffic in Sect. 2. The period is defined as a dimensioning period. As shown in Table 2, the algorithms find slightly different values for |D| under some conditions. We find that the different |D| is the the main reason behind C_shared differences between the minimum solution and heuristic solutions (suggested and genetic) in Fig. 6. We leave this for future work. Because the amount of traffic increases with large |U|, the suggested algorithm finds equal or larger |D| for larger |U|.

As shown in Fig. 8, C_dedicated and C_shared exponentially increase as a function of α because of an exponential fitting function between price and performance of a computing machine in Fig. 1. In the low-α regime, a DTN-user mapping result is critical in neither C_dedicated nor C_shared, because a small value is determined for |D|. Therefore, C_shared of the suggested algorithm is almost equal to that of the minimum-CAPEX algorithm in the low-α regime. An increase in α results in a large value of |D|; thus, C_dedicated and C_shared strongly depend on the DTN–user mapping result in the high-α regime. Therefore, the difference in C_shared between the suggested and minimum-CAPEX algorithms appears in the high-α regime. In all conditions, the differences in C_shared between the suggested and minimum-CAPEX algorithms lie between 15% (α = 2 in Fig. 8c) and 1% (α = 1 in Fig. 8b). The genetic-based algorithm shows similar (at least 98%) C_shared of those of the proposed algorithm, while requiring a 2-order-of-magnitude longer computation time.

Because traffic in the |U| = 15 case is a subset of that of |U| = 25 case, C_dedicated in Fig. 8a, b, c are smaller than those in Fig. 8d, e, f, respectively. The suggested algorithm finds an equal or larger |D| for the case of |U| = 25 than that for |U| = 15. The large number of |D| introduces a high degree of freedom for distributing users into DTNs. The combination of a high degree of freedom and effective DTN-user mapping algorithm can also lower C_shared. Therefore, a shared DTN design with the suggested algorithm effectively suppresses the degree of increment of C_shared with respect to the increment of |U|, whereas the dedicated DTN fails. The increases in C_dedicated and C_shared from |U| = 15–25 reach up to 45% (α = 1 in Fig. 8a, d and 3% (α = 3 in Fig. 8b, e) in dedicated DTN and shared DTN with suggested algorithm, respectively. In Fig. 6, different periods of dimensioning traffic rarely affect trends of CAPEX difference between each design and algorithm.

Owing to the uncertainty of network traffic, it is impossible to forecast the exact amount of future traffic. To effectively provision future traffic, the network dimensioning requires a careful consideration of α. We define a DTN capacity overflow for the case when the amount of traffic workload to a DTN exceeds the capacity of the DTN. To evaluate the DTN capacity overflow, we consider a provisioning scenario in which a dimensioning solution is calculated by the dimensioning period traffic. Then, the remaining traffic out of the 2 months of monitored traffic is used as the provisioning traffic to evaluate the DTN capacity overflow. For example, Fig. 9a shows the number of DTN capacity overflows of the provisioning traffic from August 11 to September 25, under the given dimensioning solution determined by dimensioning traffic from 25 July to 10 August.

As shown in Fig. 9, a higher α dramatically decreases the DTN capacity overflow counts at the cost of CAPEX overhead. Compared with the shared DTN, the dedicated DTN significantly suffers from the overflow counts in the same α condition. Similarly, the required value of α for a zero overflow in the shared DTN is much lower than that of the dedicated DTN. This observation is due to the low time-correlation of burst traffic between users who share a shared DTN. In Fig. 9a, the required values of α for zero overflow are calculated as 1.5 and 2.5 for a shared DTN with the suggested algorithm and dedicated DTN, which correspond to 17,123 and 80,437 USD in Fig. 8a, respectively. Therefore, to satisfy an overflow-free requirement during provisioning, the shared DTN with suggested algorithm reduces 79% CAPEX from that of the distributed DTN. We observe that a few randomly generated initial populations in the genetic algorithm exaggerate the average value of DTN capacity overflows, as shown in Fig. 9a, d.

Science DMZ is an optimized networking technology for research data transfer, where a DTN is an essential component to take full advantage of the plentiful network bandwidth provided by the Science DMZ infrastructure. The design of the Science DMZ regarding DTN deployment is an important dimensioning problem in the scalability of scientific research networks, as the DTN is a costly networking component and requires considerable efforts for monitoring and management. This paper proposes a new design of Science DMZ with shared DTN as a promising solution for highly scalable research data-transfer networking. In-depth explorations of the 2-month-long real research traffic of 25 laboratories over KREONET provided interesting observations in the research traffic, including burstiness, heavy-tail distribution, fitness of the 80:20 rule, and low time correlations of burst transactions between different research communities. Based on extensive studies of research traffic, this paper proposes an iterative greedy heuristic algorithm for the shared DTN design, which aims to minimize the peak traffic on the shared DTNs. The proposed algorithm achieves a short computation time by decoupling problems and adopting iterative and greedy approaches. The simulation studies on practical research traffic manifest the high scalability of the shared DTN using the suggested algorithm, which achieves 2- and 5-order-of-magnitude computation time reductions at costs of acceptable CAPEX overheads from those of genetic-based and minimum-CAPEX solutions, respectively. Specifically, the shared DTN with suggested algorithm saves at most 79% of CAPEX from that of dedicated DTN design under the strict provisioning criteria. This paper leaves studies on the nature of research traffic with respect to time scale, including the self-similarity and scale-free properties, for future studies.