Combinatorial Testing in Cloud Computing

This chapter gives an introduction to software testing and cloud testing. It describes the basic concepts of combinatorial testing in detail, including Latin squares, orthogonal arrays, and covering arrays. The popular combinatorial testing algorithms, such as AETG and IPO, are discussed.

This chapter gives an introduction to combinatorial testing. In particular, it describes the applications and challenges of combinatorial testing in cloud environment and briefly introduces solutions to address challenges. It also discusses and compares existing faulty location analysis solutions of combinatorial testing.

This chapter discusses combinatorial testing in multi-tenancy Software-as-a-Service (SaaS) system. SaaS often uses multi-tenancy architecture (MTA) where tenant developers compose their applications online using the components stored in the SaaS database. Tenant applications need to be tested, and combinatorial testing can be used. While numerous combinatorial testing techniques are available, most of them produce static sequence of test configurations and their goal is often to provide sufficient coverage such as 2-way interaction coverage. But the goal of SaaS testing is to identify those compositions that are faulty for tenant applications. In this chapter, it proposes an adaptive test configuration generation algorithm called adaptive reasoning (AR) that can rapidly identify those faulty combinations so that those faulty combinations cannot be selected by tenant developers for composition. Whenever a new component is submitted to the SaaS database, the AR algorithm can be applied so that any faulty interactions with new components can be identified to continue to support future tenant applications.

This chapter proposes a new algebraic system, test algebra (TA) for identifying faults in combinatorial testing for Software-as-a-Service (SaaS) applications. In the context of cloud computing, SaaS is a new software delivery model, in which mission-critical applications are composed, deployed, and executed on cloud platforms. Testing SaaS applications is challenging because new applications need to be tested once they are composed, and prior to their deployment. A composition of components providing services yields a configuration providing a SaaS application. While individual components in the configuration may have been thoroughly tested, faults still arise due to interactions among the components composed, making the configuration faulty. When there are k components, combinatorial testing algorithms can be used to identify faulty interactions for t or fewer components, for some threshold \(2 \le t \le k\) on the size of interactions considered. In general, these methods do not identify specific faults, but rather indicate the presence or absence of some fault. To identify specific faults, an adaptive testing regime repeatedly constructs and tests configurations in order to determine, for each interaction of interest, whether it is faulty or not. In order to perform such testing in a loosely coupled distributed environment such as the cloud, it is imperative that testing results can be combined from many different servers. The TA defines rules to permit results to be combined and to identify the faulty interactions. Using the TA, configurations can be tested concurrently on different servers and in any order. The results, using the TA, remain the same.

Software-as-a-Service (SaaS) application plays an important role in daily life and needs to have high reliability and availability before publishing. Testing SaaS applications become important, as the large number of testing prior to their deployment. TA identifies faults in combinatorial testing for SaaS applications using existing test results and eliminates those related faults. Although TA eliminates a large number of configurations from considerations, it is still difficult to finish testing enormous combinations of services in a reasonable time. To improve TA analysis, this chapter proposes a concurrent TA analysis. It allocates workloads into different clusters of computers and performs TA analysis from 2-way to 6-way configurations. Different database designs are used to store the test results of various configurations. Faulty and operational table search algorithms are proposed to retrieve existing test results. One 25-component experiment is simulated using the proposed solutions. The same experiment is also simulated on multiple processors for concurrent TA analysis.

Testing SaaS applications are challenging, because a large number of configurations need to be tested. Faulty configurations should be identified and corrected before the delivery of SaaS applications. TA proposes an effective way to reuse existing test results to identify test results of candidate configurations, and it also defines rules to permit results to be combined, and to identify the faulty interactions. Using the TA, configurations can be tested concurrently on different servers and in any order. This chapter proposes one MapReduce design of TA concurrent execution in a cloud environment. The optimization of TA analysis is discussed. The proposed solutions are simulated using Hadoop in a cloud environment.

A new integrated testing framework is proposed to use adaptive reasoning algorithm with automated test cases generation (ARP) and test algebra (TA) for increasing SaaS testing efficiency in faulty combination identification and elimination. The ARP algorithm has been evaluated by both simulation and real experimentation using a MTA SaaS sample running on GAE (Google App Engine). Both the simulation and experiment show that the ARP algorithm can identify those faulty combinations rapidly and TA can eliminate a large number of faults from candidate test set with a small number of seeded faults.

Testing-as-a-Service (TaaS) in a cloud environment can leverage the computation power provided by the cloud. Specifically, testing can be scaled to large and dynamic workloads, executed in a distributed environment with hundreds of thousands of processors, and these processors may support concurrent and distributed test execution and analysis. TaaS may be implemented as SaaS and used to test SaaS applications. This chapter proposes a TaaS design for SaaS combinatorial testing. TA and AR algorithm are used in the TaaS design.

Testing-as-a-Service (TaaS) is a software testing service in a cloud that can leverage the computation power provided by the cloud. Specifically, a TaaS can be scaled to large and dynamic workloads, executed in a distributed environment with hundreds of thousands of processors, and these processors may support concurrent and distributed test execution and analysis. This chapter proposes a TaaS system based on AR and TA for combinatorial testing (CT). AR performs testing and identifies faulty interactions, and TA eliminates related configurations from testing and there can be carried out concurrently. By combining these two, it is possible to perform large CT that was not possible before. Specifically, experiments with \(2^{50}\) components with \(2.83*10^{87}\) 6-way interactions with about \(2^{1.1*10^{15}}\) configurations were performed, and this may be the largest CT experimentation as of 2014. 98.6% of configurations have been eliminated out of total number of configurations.