Testing of Communicating Systems

An argument is made that predictive metrics provide a very powerful means for organizations to assess characteristics of their software systems and allow them to make critical decisions based on the value computed. Five different predictors are discussed aimed at different stages of the software lifecycle ranging from a metric that is based on an architecture review which is done at the earliest stages of development, before even low-level design has begun, to one designed to predict the risk of releasing a system in its current form. Other predictors discussed include the identification of characteristics of files that are likely to be particularly fault-prone, a metric to help a tester charged with regression testing to determine whether or not a particular selective regression testing algorithm is likely to be cost effective to run on a given software system and test suite, and a metric to help determine whether a system is likely to be able to handle a significantly increased workload while maintaining acceptable performance levels.

This paper presents a framework based on the 3GPP test model for a Virtual Test Environment which is the main tool used by wireless-engineers for development and for testing the complete UMTS terminal software. The main requirements to the system are ease of deployment on all engineering workstations, support the design of test scenarios at a high abstraction level and, when running the test scenarios, the system shall perform much faster than in a real run in order to efficiently exercise and debug the relevant functionalities. Furthermore, the Virtual Test Environment is used to run complete test suites within a couple of hours in order to guarantee the quality of the software when it is delivered to the customer.

Due to immense competition in the market, mobile equipment vendors and service providers are faced with the challenge of delivering solutions early, way ahead of their competitors. Time-to-market pressures necessitate a carefully worked out test strategy for verifying and validating the correctness of mobile communication solutions. This paper outlines the various challenges that are faced in the protocol testing of mobile communication products. An evaluation of formal languages such as SDL and TTCN in the design, development and testing phases of such products is also included in the paper. The strategy employed in the testing of a 3G 1xEV-DV Base Station stack has been covered as a case study.

Routing protocols are typical distributed systems characterized by dynamic, concurrent and distributed behaviors. As the primary function of routing protocols, routing information processing constitutes the main content of routing protocol conformance testing. However, there isn’t a clear model based on which convincing test architecture and test notation can be designed. Also, some important features of routing protocols have not been considered sufficiently in existing test practice. On the other hand, generalized distributed system models, test architectures and test notations usually have limitations when applied directly to specific protocol testing. So it is necessary to study specific cases in order to find a pragmatic and efficient approach. In this paper, an MP-FSM (Finite State Machine with Multiple Ports) model is proposed to describe routing information processing of IP routing protocols. Based on this model and other test requirement particularities, a test architecture called PADTACC (Parallel And Distributed Test Architecture with Centralized-Control) is presented and a test notation called RIPTS (Routing Information Processing Test Script) is defined. We implement the whole approach into a software tester — IRIT (IP Routing Information Tester).

This paper describes the TTCN-3 Data Presentation Format (DPF). DPF is an intuitive graphical notation for representing TTCN-3 Core Language (CL) [1] types and values. The major advantage of using DPF compared to free-text editing is that a DPF implementation ensures a consistent type and template structure by excluding references to unexisting entities while significantly reducing typing work. The result is a shorter test suite development time. DPF covers all excluded parts of Graphical Presentation Format (GFT) [3]. We believe that DPF and GFT together could be the basis for building the ultimate graphical representation of TTCN-3.

UML models focus primarily on the definition of system structure and behaviour, but provide only limited means for describing test objectives and test procedures. However, with the approach towards system engineering with automated code generation, the need for solid conformance testing has increased. In June 2001, an OMG Request For Proposal (RFP) on an UML2.0 Testing Profile (UTP) has been initiated. This RFP solicits proposals for a UML2.0 profile, which enables the specification of tests for structural and behavioural aspects of computational UML models, and which is capable to interoperate with existing test technologies for black box testing. This paper discusses different approaches for testing with UML and discusses the ongoing work of the Testing Profile. Special emphasize is laid on the mapping of UML2.0 testing concepts to the standardized Testing and Test Control Notation (TTCN-3).

Distributed test setups for efficient load, performance, scalability, interworking, and end-to-end tests are gaining importance for the assessment of distributed communicating systems. The Testing and Test Control Notation TTCN-3 provides concepts for component-based distributed test systems in dynamic test configurations, where test components may reside on various network nodes to be near the interfaces of the tested system. The realization of executable TTCN-3 tests on concrete test platforms involves TTCN-3 compilation/interpretation and adaptations to the test platform. The TTCN-3 Control Interfaces TCI define entities, interfaces, types and operations needed to flexibly manage and distribute TTCN-3 based test systems. It complements and completes the TTCN-3 Runtime Interface TRI. This paper discusses the underlying concepts of TCI and demonstrates its use for the realization of a distributed test for the Session Initiation Protocol SIP.

The textual Testing and Test Control Notation (TTCN-3) is frequently used in combination with Message Sequence Chart (MSC) and the MSC-based Graphical Presentation Format for TTCN-3 (GFT). Both, MSC and GFT allow an automatic generation of TTCN-3 test case descriptions.

TimedTTCN-3 is an extension of TTCN-3 for testing real-time properties and has been submitted for standardization. For a complete integration of TimedTTCN-3 into the TTCN-3-based testing process, the usage of TimedTTCN-3 in combination with MSC and GFT needs to be established.

This paper presents our approach for graphical real-time test specification based on MSC and TimedGFT, which is our real-time extension of GFT. We explain how MSC can be used for the description of real-time test purposes and define TimedGFT. Our approach includes the automatic generation of TimedTTCN-3 test cases based on MSC test purposes and TimedGFT diagrams.

Euclid of Alexandria, the most prominent mathematician of antiquity, is also the leading mathematics teacher of all times due to his treatise on mathematics, ‘The Elements’. When once asked whether there wasn’t an easier way to study geometry than ‘The Elements’, Euclid was said to respond: ‘There is no royal road to geometry’.

Interoperability is what most standards are about. It is the only thing that these standards are about. So is there a royal road to interoperability? Unfortunately, interoperability does not do any better than geometry — there is no royal road to interoperability. Interoperability is extremely difficult to achieve.

Two ways of validating a standard and an implementation are ‘conformance testing’ and ‘interoperability testing’. In conformance testing, manufacturers run their implementation against a set of ‘golden’ test suites; in ‘interoperability testing’, they run their implementations against each other. Frequently, ‘interoperability testing’ is done at ‘bake-offs’, i.e. open events where developers get together to debug a standard and their implementations.

Views on which method is best are at times strongly divergent. Theoretically, if a standard is tight and has only a limited number of options, implementations that pass conformance tests will automatically be interoperable. In practice, however, a ‘pass’ in conformance testing does not necessarily guarantee that two implementations are interoperable. Likewise, if two implementations are interoperable, they may not necessarily conform to the standard. In reality, things probably lie somewhere in the middle, and both conformance testing and interoperability testing pave the road towards interoperability.

The paper studies testing based on input/output transition systems, also known as input/output automata. It is assumed that a tester can never prevent an implementation under test (IUT) from producing outputs, while the IUT does not block inputs from the tester, either. Thus, input from the tester and output from the IUT may occur simultaneously and should be queued in finite buffers between the tester and the IUT. A framework for so-called queued-quiescence testing is developed, based on the idea that the tester should consist of two test processes, one applying inputs via a queue to an IUT and the other reading outputs from a queue until it detects no more outputs of the IUT, i.e., the tester detects quiescence of the IUT. The testing framework is then extended with so-called queued-suspension testing by considering a tester that has several pairs of input and output processes. Test derivation procedures are elaborated with a fault model in mind.

The objective of testing is to determine whether an implementation under test conforms to its specification. In distributed test architectures involving multiple testers, this objective can be complicated by the fact that testers may encounter problems relating to controllability and observability during the application of tests. The controllability problem manifests itself when a tester is required to send the current input and because it did not send the previous input nor did it receive the previous output it cannot determine when to send the input. The observability problem manifests itself when a tester is expecting an output in response to either the previous input or the current input and because it is not the sender of the current input, it cannot determine when to start and stop waiting for the output. Based on a distinguishing sequence, a checking sequence construction method is proposed to yield a sequence that is free from controllability and observability problems.

This paper introduces a new conformance relation between a specification and an implementation of a distributed system. It is based on a local view which allows to avoid or reduce the state explosion problem. The conformance relation is defined via Petri nets and shows not only equivalence between transitions but also equivalence between local states. This equivalence depends on the structural properties of the Petri net and is independent of any specific initial marking. We compare our notion of conformance to classical ones and give model checking and test case generation algorithms for it.

We present a framework for property testing where a partially ordered execution trace of a distributed system is modeled by a collection of communicating automata. We prove that the model exactly characterizes the causality relation between the events in the observed trace. We present the implementation of this approach in SDL, where ObjectGEODE is used to verify properties, and illustrate the approach with an industrial case study.

In this paper, we propose a method for the derivation of an adaptive diagnostic test suite when the system specification and implementation are given in the form of an extended finite state machine. The method enables us to decide if it is possible to identify the faulty transitions in the system when faults have been detected in a system implementation. If this is possible, the method also returns test cases for locating the faulty transitions. An example is used to demonstrate the steps of the method.

Real-time systems are those systems whose behaviors are time dependent. Reliability is one of the characteristics of such systems and testing is one of the techniques that can be used to ensure reliable real-time systems. This paper presents a method for testing real-time systems specified by Timed Input Output Automata (TIOA). Our method is based on the concept of test purposes. The use of test purposes helps reduce the number of test cases generated since an exhaustive testing of a TIOA causes the well-known state explosion problem. The approach we present in this paper consists of three main steps. First, a synchronous product of the specification and test purpose is computed. Then, a sub-automaton (called Grid Automata) representing a subset of the state space of this product is derived. Finally, test cases are generated from the resulting grid automata. The test cases generated by our method are executable and can easily be represented in TTCN (Tabular Tree Combined Notation).

This paper presents a fault model for interoperability testing of communication protocols that are modeled by communicating finite state machines, and proposes a technique that extends an initial interoperability test suite, which is given by another existing method, to be a test suite that can detect “almost all” interaction faults based on the fault model. We start with an interoperability test suite derived by a known method and develop a technique for the fault coverage analysis and a technique for the extension of the test suite in order to achieve high fault coverage. We illustrate the proposed techniques with TCP protocol. The fault coverage analysis concludes that the test suite has 100% fault coverage with respect to the proposed fault domain and does not need to be extended. It is shown that our method is applicable to practical protocols and can be used to make interoperability test suites have high fault-detecting capability.

In IP world, interoperability testing is heavily used to check the correctness of different implementations. Internet protocols have growing importance in communicating systems. In our paper we show an automatic interoperability test approach and also present its application on an IP-based protocol, ROHC. The primary goal of our work was to define an interoperability testing framework in TTCN-3 that can be used in general. After giving an overview on the ROHC protocol, we also describe the way we have conformance tested it. Then we present MAIT (Model for Automated Interoperability Test) and give detailed explanation on its components and their roles. At the end, we compare the advantages and disadvantages of conventional conformance testing and our interoperability testing model.

In this paper we briefly describe the main goals and organization of TestNet, a proposal for the creation of a Network of Excellence in the scope of the 6th Framework Programme of the European Community. TestNet: Integration of Testing Methodologies represents the joint effort of the different European testing communities to create a common framework to improve all the aspects of the testing process.

In the prevailing competitive environment, companies are facing tremendous market pressures to launch defect-free products in a timely manner. This challenge is compounded when a product runs into sustenance phase because complete regression runs need to be performed for each change/enhancement made in every build/release of the product-under-test. This paper discusses an architectural framework approach to address the above challenge, thereby aiming to make regression testing a simple, repeatable and automated exercise. The framework design encapsulates hierarchical test case management, multi-user support, product version maintenance, and automated test execution and result analysis to facilitate easy testing. The open architecture of the framework allows it to augment capabilities of some other testing tools by providing adapters to them, thus, eliminating the rigidity of use of a particular tool. The paper also draws a comparison of the architectural approach with other existing frameworks and presents a cost-benefit analysis of the suggested approach.

The test definition language TTCN-3 is currently under standardization by ETSI/ITU-T. Its intended field of application is testing and performance measurement of communication hard- and software. TTCN-3 does include mechanisms for specifying remote hardware access and for distributed execution of testing code components.

But for running compiled TTCN-3 code distributed onto distinct nodes of different vendors an architecture is needed which offers standardized means for dynamic, program controlled deployment, configuration and status inquiry of active and passive resources.

TUB-TCI is a proposal for such an architecture, characterized by (1) totally generic definition of component classes, (2) coexistence of standardized (XML based) and specialized (high-speed) communication channels, (3) a model-based, strictly formal definition given in Z and (4) a simple, minimized but powerful execution model.

Especially because of the integration of non-standard, high speed data channels TUB-TCI seems applicable for dynamical routing of real-time signals in general, beyond the field of test execution.

We present a novel methodology to perform passive testing. The usual approach consists in recording the trace produced by the implementation under test and trying to find a fault by comparing this trace with the specification. We propose a more active approach to passive testing where the minimum set of (critical) properties required to a correct implementation may be explicitly indicated. In short, an invariant expresses that each time that the implementation under test performs a given sequence of input/output actions, then it must show a behavior reflected in the invariant. By using an adaptation of the classical pattern matching algorithms on strings, we obtain that the complexity of checking whether an invariant is fulfilled by the observed trace is in \( \mathcal{O}\left( {nm} \right) \) , where n and m are the lengths of the trace and the invariant, respectively. If the length of the invariant is much smaller than the length of the trace then this complexity is almost linear with respect to the length of the trace. Actually, this is usually the case for most practical examples. In addition to our methodology, we present the case study that was the driving force for the development of our theory: The Wireless Application Protocol (WAP). We present a test architecture for WAP as well as the experimental results obtained from the application of our passive testing with invariants approach.