1 Introduction
2 Background and objectives
2.1 Internet of Things devices and IoT processes
-
Sensors measure physical properties and produce discrete or continuous streams of events. We also view abstract event sources producing data as (virtual) sensors. In IoT, data from sensors are often consumed by clients on a publish-subscribe basis using standard Web-based protocols (e.g. MQTT2).
-
Actuators receive commands and manipulate the physical or virtual world by executing these commands. In IoT, the functionality of actuators is often accessed remotely in a service-oriented manner using Web services based on the REST paradigm and HTTP protocol [18].
2.2 IoT process modelling on “Classical” GUIs
2.2.1 Business process-oriented modelling
2.2.2 Flow-based modelling
2.3 Process modelling in mixed reality
2.4 Objectives
3 Related work
3.1 IoT process modelling and execution on “Classical” GUIs
3.2 Mixed reality in process-based IoT applications
4 HoloFlows: mixed reality-based IoT process modelling
4.1 HoloFlows IoT process modelling language
IoT Device
is a composition of one or multiple IoT Components
. IoT components have a name, an identifier and an icon representing the device in the user interface. Sensors
and Actuators
are the atomic units of composite IoT devices. Sensors have a state and an associated unit of measure. Actuators are able to execute one or more Commands
that require input and output parameters. An actuator can produce output data (e.g. have a state) and therefore may also act as one or more sensors.IoT Process
also follows a minimal approach [90]. An IoT process can be composed of Gateways
and has to have at least one Connection
. A connection has one source and one target, which can both be either an IoT device or a gateway. A gateway is of a specific type (i. e., OR
, AND
). A connection may have an associated activation Condition
. This condition refers to the state of a sensor (left side), a comparison operator, and a specified threshold (right side) defining the activation of the connection. Following the ECA (Event-Condition-Action) paradigm [91], a connection can also have an actuator command specified as action to be executed as result of its activation. Depending on the nature of source and target and existence of a condition, we distinguish between different types of connections (i. e., direct sensor–actuator connection, conditional sensor–actuator connection, actuator–actuator connection, connection with a gateway). With this simplified meta-model, processes of varying complexity based on single connections—possibly with an activation condition—between two instances of IoT devices or a device and a gateway can be developed. The expressiveness should thereby be sufficient for most modelling tasks performed by end-users in IoT environments [14].4.2 HoloFlows mixed reality user interface
4.2.1 Design principles
4.2.2 IoT process visualization and interactions
OR
and AND
gateways. A gateway can serve as source or target of a connection. The user selects and places the respective OR
and AND
cube holograms freely within the holographic scene. Connections from and to a sensor or an actuator can be created by either first selecting the gateway and then the sensor/actuator or by first selecting the sensor/actuator and then the gateway. This way processes with multiple sensor-related conditions joined via an AND
or OR
gateway and parallel actuator calls split via an AND
gateway can be defined.4.3 IoT process examples in mixed reality
-
Figure 8 shows a live view from the HoloFlows app with two simple conditional sensor–actuator connections being modelled: one defining that if the light levels are below 150 Lux, then the coffee maker should start, and one defining that if the temperature is above 26°C, then the power level of the fan should be switched to 100 %. Other sensors integrated into the HoloFlows app comprise an NFC tag reader (e.g. to create an authentication process to open the door when a certain NFC tag is present) and a wearable hand movement sensor (e.g. to switch on the light via a simple swipe gesture).
-
Figure 9 shows an exemplary direct sensor–actuator connection between a potentiometer as sensor (values from 0 to 100 %) to the power level of a dimmer switch controlling a fan (or lamp) (values from 0 to 100 %).
-
Figure 10 shows the live view of an IoT process with a conditional sensor–actuator connection and an
AND
split defining that if the light levels are below 100 Lux, then the power levels of Lamp 1 should be increased to 50 %AND
of Lamp 2 to 79 % in parallel. -
Figure 11 shows an IoT process with two conditional sensor–actuator connections and an
OR
join defining that the door should be opened if the humidity exceeds 65.8 %OR
the temperature is above 28.4°C.