Testing Software and Systems

Models play an essential role in the design process of cyber-physical systems. They form the basis for simulation and analysis and help in identifying design problems as early as possible. However, the construction of models that comprise physical and digital behavior is challenging. Therefore, there is considerable interest in learning such hybrid behavior by means of machine learning which requires sufficient and representative training data covering the behavior of the physical system adequately. In this work, we exploit a combination of automata learning and model-based testing to generate sufficient training data fully automatically.

Experimental results on a platooning scenario show that recurrent neural networks learned with this data achieved significantly better results compared to models learned from randomly generated data. In particular, the classification error for crash detection is reduced by a factor of five and a similar F1-score is obtained with up to three orders of magnitude fewer training samples.

Modern-day demands for services often require an availability on a 24/7 basis as well as online accessibility around the globe. For this sake, personalized software systems, called chatbots, are applied. Chatbots offer services, goods or information in natural language. These programs respond to the user in real-time and offer an intuitive and simple interface to interact with. Advantages like these makes them increasingly popular. Chatbots can even act as substitutes for humans for specific purposes. Since the chatbot market is growing, chatbots might outperform and replace classical web applications in the future. For this reason, ensuring correct functionality of chatbots is of high and increasing importance. However, since different implementations and user behavior result in unpredictable results, the chatbot’s output is difficult to predict and classify as well. In fact, testing of chatbots represents a challenge because of the unavailability of a test oracle. In this paper, we introduce a metamorphic testing approach for chatbots. In general, metamorphic testing can be applied to situations where no expected values are available. In addition, we discuss how to obtain test cases for chatbots, i.e. sequences of interactions with a chatbot, in an according manner. We demonstrate our approach using a hotel booking system and discuss first experimental results.

Although both positive and negative testing are important for assuring quality of programs, generating a variety of test inputs for such testing purposes is difficult for machine learning software. This paper studies why it is difficult, and then proposes a new method of generating datasets that are test inputs to machine learning programs. The proposed idea is demonstrated with a case study of classifying hand-written numbers.

In this paper we propose a new approach to conformance testing based on Abstract State Machine (ASM) model refinement. It consists in generating test sequences from ASM models and checking the conformance between code and models in multiple iterations. This process is applied at different models, starting from the more abstract model to the one that is very close to the code. The process consists of the following steps: (1) model the system as an Abstract State Machine, (2) generate test sequences based on the ASM model, (3) compute the code coverage using generated tests, (4) if the coverage is low refine the Abstract State Machine and return to step 2. We have applied the proposed approach to Antidote, an open-source implementation of IEEE 11073-20601 Personal Health Device (PHD) protocol which allows personal healthcare devices to exchange data with other devices such as small computers and smartphones.

Homing, synchronizing and distinguishing sequences (HSs, SSs, and DSs) are used in FSM (Finite State Machine) based testing for state identification and can significantly reduce the size of a returned test suite with guaranteed fault coverage. However, such preset sequences not always exist for nondeterministic FSMs and are rather long when existing. Adaptive HSs, SSs and DSs are known to exist more often and be much shorter that makes them attractive for deriving test suites and adaptive checking sequences. As nowadays, a number of specifications are represented by nondeterministic FSMs, the deeper study of such sequences, their derivation strategies, and related complexity estimations/reductions is in great demand. In this paper, we evaluate the complexity of deriving adaptive HSs and SSs for noninitialized FSMs, the complexity of deriving DSs for noninitialized merging-free FSMs.

The problem of generating tests detecting all logical and timing faults which can occur in real-time systems is challenging; this is because the number of (timing) faults is potentially too big or infinite. As a result, it might be time consuming to generate an important number of adequate tests. The traditional model based testing approach considers a fault domain as the universe of all machines with a given number of states and input-output alphabet while mutation based approaches define a list of mutants to kill with a test suite. In this paper, we combine the two approaches by developing a mutation testing technique for real-time systems represented with deterministic timed finite state machines with timed guards and timeouts (TFSM-TG). In this approach, fault domains consisting of fault-seeded versions of the specification (mutants) are represented with non-deterministic TFSM-TG. The test generation avoids the one-by-one enumeration of the mutants and is based on constraint solving. We present the results of an empirical proof-of-concept implementation of the proposed approach.

Automated testing techniques can effectively explore mobile applications in order to find faults that manifest as program crashes. A number of different techniques for automatically testing apps have been proposed and empirically compared, but previous studies focused on comparing different tools, rather than techniques. Although these studies have shown search-based approaches to be effective, it remains unclear whether superior performance of one tool compared to another is due to fundamental advantages of the underlying search technique, or due to certain engineering choices made during the implementation of the tools. In order to provide a better understanding of app testing as a search problem, we empirically study different search algorithms within the same app testing framework. Experiments on a selection of 10 non-trivial apps reveal that the costs of fitness evaluations are inhibitive, and prevent the choice of algorithm from having a major effect.

In this paper, we compare the performance of a genetic algorithm for test parameter optimization with simulated annealing and random testing. Simulated annealing and genetic algorithm both represent search-based testing strategies. In the context of autonomous and automated driving, we apply these methods to iteratively optimize test parameters, to aim at obtaining critical scenarios that form the basis for virtual verification and validation of Advanced Driver Assistant System (ADAS). We consider a test scenario to be critical if the underlying parameter set causes a malfunction of the system equipped with the ADAS function (i.e., near-crash or crash of the vehicle). To assess the criticality of each test scenario we rely on time-to-collision (TTC), which is a well-known and often used time-based safety indicator for recognizing rear-end conflicts. For evaluating the performance of each testing strategy, we set up a simulation framework, where we automatically run simulations for each approach until a predefined minimal TTC threshold is reached or a maximal number of iterations has passed. The genetic algorithm-based approach showed the best performance by generating critical scenarios with the lowest number of required test executions, compared to random testing and simulated annealing.

Bounded exhaustive testing (BET) is an elementary technique in automated unit testing. It consists in testing a function with all input data up to a given size bound. We implement BET to check logical and program properties, before attempting to prove them formally with the deductive verification tool Why3. We also present a library of enumeration programs for BET, certified by formal proofs of their properties with Why3. In order to make BET more efficient, we study and compare several strategies to optimize these programs.

In this paper it is demonstrated that the capabilities of state-of-the-art proof assistant tools are sufficient to present mechanised and, at the same time, human-readable proofs establishing completeness properties of test methods and the correctness of associated test generation algorithms. To this end, the well-known Isabelle/HOL proof assistant is used to mechanically verify a complete test theory elaborated by the second author for checking the reduction conformance relation between a possibly nondeterministic finite state machine (FSM) serving as reference model and an implementation whose behaviour can also be represented by an FSM. The formalisation also helps to clarify an ambiguity in the original test generation algorithm which was specified in natural language and could be misinterpreted in a way leading to insufficient fault coverage.

The paper is devoted to the verification of Software Defined Networking (SDN) components and their compositions. We focus on the interaction between three basic entities, an application, a controller, and a switch. When the application submits a request to the controller, containing a set of rules to configure, these rules are expected to be ‘pushed’ and correctly applied by the switch of interest. However, this is not always the case, and one of the reasons is the presence of races or concurrency issues in SDN components and related interfaces. We propose a model checking based approach for deriving test sequences that can identify SDN races. The test generation strategy is based on model checking, and related formal verification is performed with the use of extended automata specifying the behavior of the components of interest; Linear Temporal Logic (LTL) formulas are utilized to express the properties to check. We generalize the races of interest and propose an approach for deriving the corresponding LTL formulas that are later used for verifiation. The Spin model checker is used for that purpose and thus, Promela specifications for interacting components are also provided; those are: the ONOS REST API, the ONOS controller and an OpenFlow Switch. An experimental evaluation with the aforementioned components showcases the existence of race conditions in their compositions.

Performance testing is a critical task to ensure an acceptable user experience with software systems, especially when there are high numbers of concurrent users. Selecting an appropriate test workload is a challenging and time-consuming process that relies heavily on the testers’ expertise. Not only are workloads application-dependent, but also it is usually unclear how large a workload must be to expose any performance issues that exist in an application. Previous research has proposed to dynamically adapt the test workloads in real-time based on the application behavior. By reducing the need for the trial-and-error test cycles required when using static workloads, dynamic workload adaptation can reduce the effort and expertise needed to carry out performance testing. However, such approaches usually require testers to properly configure several parameters in order to be effective in identifying workload-dependent performance bugs, which may hinder their usability among practitioners. To address this issue, this paper examines the different criteria needed to conduct performance testing efficiently using dynamic workload adaptation. We present the results of comprehensively evaluating one such approach, providing insights into how to tune it properly in order to obtain better outcomes based on different scenarios. We also study the effects of varying its configuration and how this can affect the results obtained.

Performance testing is a critical task to assure optimal experience for users, especially when there are high loads of concurrent users. JMeter is one of the most widely used tools for load and stress testing. With JMeter, it is possible to test the performance of static and dynamic resources on the web. This paper presents DYNAMOJM, a novel tool built on top of JMeter that enables testers to create a dynamic workload for performance testing. This tool implements the DYNAMO approach, which has proven useful to find performance issues more efficiently than static testing techniques.

Web services allow the communication of heterogeneous systems and are particularly suitable for building cloud applications. Furthermore, such applications must verify some static properties, but also tolerate attacks at runtime to ensure service continuity. To achieve this, in this paper we propose an attack tolerance framework that includes the risks of attacks. After describing the foundation of this framework, we propose expressing cloud applications as choreographies of services that take into account their distributed nature. Then, we extended the framework to introduce choreography verification by incorporating monitoring (passive tests) and reaction mechanisms. These techniques are validated through relevant experiments. As a result, our framework ensures the required attack tolerance of such cloud applications.

In a software development process, the integration and verification of the different parts of the application under development often require a lot of effort. Component Based Software Engineering (CBSE) approaches help cut software integration costs by enabling the automatic generation of data types, method signatures and middleware configuration from a model of the application structure. Model Based Testing (MBT) techniques help cut software verification costs by enabling the automatic generation of test oracles from a model of the expected application behaviour. Models for CBSE and MBT are usually separate. This may result in discrepancies between them, especially when the application architecture is updated, which always happens.

In this paper, we describe how to rely on a single CBSE model to produce both code generation and oracles for some tests, thus ensuring consistency between them. Our work is based on existing OMG standards, mainly UCM and UML.

This paper presents the main results of the experiments conducted using the MMT-IoT security analysis solution run on a IoT Fed4Fire+ platform (Virtual Wall - w.iLab proposed by IMEC, Belgium). MMT is a monitoring framework developed by Montimage, and MMT-IoT is the tool that allows monitoring and analysing the security and performance of IoT networks. The results obtained concern two principal advancements. First, the adaptations made to deploy MMT-IoT on the IoT platform in order to run the tool on the platform’s IoT devices. Second, the deployment of the software allowed us to run preliminary tests on the selected platform for performing initial validation and scalability tests on this real environment. To this end, Montimage defined and implemented three test scenarios related to security and scalability with 1 or more clients. These results will be used to prepare a new experimentation phase involving also another Fed4Fire+ platform (LOG-a-TEC proposed by IJS, Slovenia).

When deadlines and resources of software projects become scarce, testing is usually in the first row to have its activities aborted or reduced. If defects cannot be found, products quality can be affected. In a software development process, aborted or reduced activities that can bring short-term benefits, but can be harmful to the project in a long run, are considered Technical Debt (TD). When TDs impact testing activities, they are called Test Debt. There are several studies dealing with Test Debt, however, current solutions often deal with specific types of tests (e.g., exploratory and automated tests) and do not address the whole software testing process. Aiming to fill these gaps, this paper proposes a Test Debt Catalog with subtypes of Test Debts and technical debt management activities. This catalog was built based on semi-structured interviews conducted with practitioners who perform testing activities in five projects from industry. With our catalog, we intend to help the management of test debts during the execution of software testing processes.