Measuring progress in Premo order-verification

Since verification of computational simulation codes requires significant resources, the ability to measure progress in verification is critical to assess whether resources are being applied appropriately. Additionally, potential users need to know what fraction of the software has been order-verified. In this study, the procedures and progress measures presented by Knupp et al. (Measuring progress order-verification within software development projects. Engineering with Computers, appears in this issue, 2007) are demonstrated on the Premo software, which simulates compressible aerodynamics through and around general geometries. Premo was selected for this demonstration because extensive order-verification tests have been previously performed, yet no systematic effort has been made to assess test-suite completeness or progress. This effort was performed to identify the practical issues encountered when attempting to apply the ideas by Knupp (Measuring progress order-verification within software development projects. Engineering with Computers, appears in this issue, 2007) to existing production-quality software. In this work, a non-specific order-verification exercise is considered, as opposed to an application-specific order-verification exercise, since past and present Premo order-verification efforts have been motivated by the need to verify all of the code, rather than portions relevant for specific applications. Constructing an order-verification test suite that verifies the order of accuracy of all the code capabilities is a major step in measuring progress. A practical approach to test-suite construction is described that helps create a complete test suite through a combination of coarse-grain code coverage, input-keyword inspection, discretization-algorithm documentation, and expert knowledge. Some of the difficulties and issues encountered during the construction of the test suite are described, along with recommendations on how to deal with them. Once the test suite is constructed, the progress measures proposed by Knupp (Measuring progress order-verification within software development projects. Engineering with Computers, appears in this issue, 2007) can be evaluated and used to reconstruct the history of progress in Premo verification over the past several years. Gaps in Premo verification are identified and indicate future directions for making progress. Additionally, a measure of Premo verification fitness is computed for selected applications commonly simulated in the aerospace industry. It is hoped that this demonstration will provide a practical example for other software-development groups in measuring their own verification progress.