Lessons from SmartCampus: External Experimenting with User-Centric Internet-of-Things Testbed

Creating Internet-of-Things (IoT) solutions that can be deployed at scale requires adequate experimentation environments. In the area of experimentation, two trends can be observed. First, there is a shift from lab-based, controlled experiments to experimenting “in the wild”: researchers tend to augment the users’ natural environments and observe how people integrate a new solution into their everyday lives. Second, when a substantial investment in setting up an experimentation infrastructure has been made, it makes sense to open it to a wide community of researchers; the concept of Experimentation-as-a-Service (EaaS) is emerging along these lines. SmartCampus, an IoT testbed developed at the University of Surrey, fits the both trends very well. It involves real users in a natural setting, as IoT devices are deployed in the users’ offices. Further, several user-centric experiments conducted in the SmartCampus were driven by external researchers, i.e., people who do not belong to the team that developed the testbed. In this paper we report on lessons learned from such IoT experiments. After a brief overview of SmartCampus and the experiments themselves, we offer a simple experiment stakeholder model, which identifies key actors and interfaces between them. We then focus on issues related to the external experimenters who take advantage of the experimentation “service.” That focus is motivated by our realization that EaaS, while attractive in principle, gives rise to a number of non-trivial challenges.