Anzeige
Erschienen in:
01.02.2012
A permutation test based on regression error characteristic curves for software cost estimation models
Erschienen in: Empirical Software Engineering | Ausgabe 1-2/2012
EinloggenAktivieren Sie unsere intelligente Suche, um passende Fachinhalte oder Patente zu finden.
Wählen Sie Textabschnitte aus um mit Künstlicher Intelligenz passenden Patente zu finden. powered by
Markieren Sie Textabschnitte, um KI-gestützt weitere passende Inhalte zu finden. powered by
Abstract
Background
Regression Error Characteristic (REC) curves provide a visualization tool, able to characterize graphically the prediction power of alternative predictive models. Due to the benefits of using such a visualization description of the whole distribution of error, REC analysis was recently introduced in software cost estimation to aid the decision of choosing the most appropriate cost estimation model during the management of a forthcoming project.
Aims
Although significant information can be retrieved from a readable graph, REC curves are not able to assess whether the divergences between the alternative error functions can constitute evidence for a statistically significant difference.
Method
In this paper, we propose a graphical procedure that utilizes (a) the process of repetitive permutations and (b) and the maximum vertical deviation between two comparative Regression Error Characteristic curves in order to conduct a hypothesis test for assessing the statistical significance of error functions.
Results
In our case studies, the data used come from software projects and the models compared are cost prediction models. The results clearly showed that the proposed statistical test is necessary in order to assess the significance of the superiority of a prediction model, since it provides an objective criterion for the distances between the REC curves. Moreover, the procedure can be easily applied to any dataset where the objective is the prediction of a response variable of interest and the comparison of alternative prediction techniques in order to select the best strategy.
Conclusions
The proposed hypothesis test, accompanying an informative graphical tool, is more easily interpretable than the conventional parametric and non-parametric statistical procedures. Moreover, it is free from normality assumptions of the error distributions when the samples are small-sized and highly skewed. Finally, the proposed graphical test can be applied to the comparisons of any alternative prediction methods and models and also to any other validation procedure.