Development of a Web Based Framework to Objectively Compare and Evaluate Software Solutions

We created a web based framework to objectively evaluate different software solutions using different approaches to the same problem. Solutions can either be single algorithms, software modules or entire programs. The evaluation strongly relies on the human component. In this paper we describe the conceptual structure as well as the evaluation of the first field test of the developed framework by utilizing the implementation of two distinct georouting algorithms as well as their validation with test users. No modification to the tested software needs to be done, as the state of each software component is saved and synchronized by the framework. The aim of the investigation of our implementation is to gather usage data, which then allows us to derive proposals for improvements upon the area of usability and user experience and to identify properties that increases the productivity of users as well as finding out about bottlenecks. The framework is validated by assessing the collected metrics and the user feedback produced by the test, as distinctions in usage, user experience and usability between the two test algorithms were identified.