A semantic blocks model for human activity prediction in smart environments using time-windowed contextual data

At The University of Manchester, we have been working on computational methods of human behaviour modelling and prediction. In this paper, we present work on the next step in that research, which goes beyond simple sequence prediction, and presents a proof-of-concept method for refining prediction results by creating an extensible model of contextual data to enable more accurate inference systems for smart environments. The SBM builds upon previous research which indicates that contextual data is key to improving the accuracy of predictive models in this field.

The state of the art in computational methods of human behaviour prediction falls into three categories: machine learning (ML), statistical forecasting and trend analysis, and Probabilistic Graphical Models (PGM). ML predictive models use sequence prediction with Recurrent Neural Networks (RNN), and more specialised subsets of RNNs such Long Short-Term Memory (LSTM) and Gated Recurrent Unit (GRU) [3, 23]. These all produce good results for simple data with accuracies \(> 90\%\). Statistical methods such as AutoRegressive Integrated Moving Average (ARIMA), Seasonal ARIMA (SARIMA), and Facebook’s Prophet library tend to look at macro level behaviours for higher level trends in human behaviour [22]. Lastly, PGMs which are essentially graph data structures with probability functions on the graph edges representing state transitions from one vertex, or state, or another. These take many forms depending on the probability method and nuances of modelling behaviour. Dynamic Bayesian Network (DBN) is good for representing a chain of causal events [17]. If we only want the current state observable in determining the next state, then a Markov Model (MM) can be used. Hidden Markov models (HMM) allow for representation of hidden states in influencing state transition [15, 21]. There are many variations of PGMs in the literature that represent the different facets of behaviour. Many of these have accuracy results of \(> 90\%\) for simple behaviours such as moving from one location. Human activities, however, are complex stochastic processes, which overlap, stop, start, and can be difficult to predict. Adding contextual data to a model can provide an insight into this complexity and disambiguate the inference process [1, 9]. PGMs are adept at extension and augmentation with data due to the flexibility of the graph data structure. A significant research contribution in the field of Ambient Intelligence (AmI) and smart environments is the CRAFFT algorithm [17]. A PGM in the form of a Bayesian Network that uses current activity and features to predict next features, by augmenting with contextual data. Context aware ontologies and semantic representation frameworks for intelligent and smart environments exist in the literature and could be leveraged in combination with the ML, PGM, and statistical approaches to provide improved contextual data support to a predictive system [2, 6, 11, 18].

The CRAFFT algorithm is the closest benchmark to the semantic blocks model and achieves an average accuracy of 66.91% over three of the CASAS smart home datasets, as shown in Table 1.

CRAFFT differs from many of the inference models by augmenting a well known algorithm with added contextual data. The SBM model presented here further extends this novelty by applying data clustering via time windows to add another dimension to the inference model.

The shift from reactive to predictive smart environment systems is a rapidly growing area of research, particularly with Ambient Assistive Living (AAL) use cases, such as determining behavioural trends or anomalous activity [12], but also in supporting activities of daily living (ADL) such as using electronic devices, cooking, and dressing [16]. The research findings presented here build on previous human activity prediction research questions conducted by this research team at the University of Manchester in the Interaction Analysis and Modelling Laboratory (https://​archive.​is/​cQZ98) and Advanced Interfaces Group Laboratory (https://​archive.​is/​igPa), which is focussed on developing these computational capabilities in the context of realising the potential of Ambient Intelligence [7, 19] by improving smart environment technology.

There are many approaches to Human Activity Prediction (HAP) including Bayesian Networks [17], Markov Model variants [26], and Deep Learning such as RNNs and LSTMs [23]. All of which are capable of producing results with varying degrees of accuracy depending on the complexity of the activity and number of occupants in the observed space. The motivation for our work here is that a broad range of different methods have been applied to this prediction problem, and what is now needed is a refinement of the inference capability to make this technology more accurate and robust. To enable more accurate predictive models, we can make better use of secondary data outside of the event or activity sequences. This contextual data support the inference process by enabling differentiation between similar sequences of data. The SBM proposes one method which can be used towards this research goal.

We present a novel proof-of-concept model which improves prediction accuracy of human activity in a smart home, demonstrated using the CASAS smart home datasets [4], and benchmarked against an established method in this area; see Table 1. The model calculates the probability intersection of the next activity label, and sets of contextual labels, based on previously observed data from the same environment. We demonstrate improved prediction accuracy against a leading peer in this area—the Bayesian Network CRAFFT model—and also demonstrate a novel contribution to research by building upon CRAFFT and adding time window data clustering to improve the inference model.

The experimentation code and results data that support the findings of this study are available in the Mendeley Data repository: a semantic blocks model for contextually enhanced human activity prediction, with the identifier DOI 10.17632/jm28kgt7tm.1 available online and open access¹.

Human activity prediction in smart environments can be improved by enriching the data with contextual labels and a time window, to enable differentiation between similar events. We hypothesise that adding more contextual data will improve the prediction accuracy of the next activity label.

In this paper, we propose the semantic blocks model (SBM), a deep contextual data probability model, which will improve inference accuracy by taking a list of next activity labels, which could be the causal probability output of a predictive model, e.g. a Dynamic Bayesian Network (DBN) and then calculating the intersection of the set of current contextual labels, the current set of time (current time window), and the set of next activity labels. The “deep” descriptor refers to the layers of contextual data, as opposed to a “shallow” model which only uses the event sequence data. We posit that this method is extensible by adding more sets of contextual data which would improve the accuracy further:

The set of sequential time values indicating the current time window, simply the start and end for the time window: \(T = \{t_1,t_2\}\)The set of potential activities where each element is an activity that could occur next. This could be a shortlist output from a sequential prediction model, e.g. a set of causal activities from a DBN. Or a less optimal list of all activity labels:The set of contextual labels present in the current time window, as obtained from the smart environment data:The set of person metadata for the person being observed, as obtained from the smart environment data:Any other sets of data, so that the system is extensible:Previously observed data, the historic dataset of the smart environment:

We then select the event data from the specified time window, and calculate the intersection value of the other sets of contextual data within that time window, to determine the likelihood of the next activity label occurring after the current activity label.In the equation above, where N is the set of next activities (e.g. a shortlist of known activity labels output by a DBN), and C is the set of current contextual labels, and T is the current time window (start and end time), it is possible to take the set N of next activity labels and return a set of probabilities of the likelihood that the activity label will be next, by calculating the probability intersection of contextual labels, next activity, and person metadata. A large overlap or high intersection between the sets outputs a high probability, and inversely lower overlap produces a lower probability. In plain English, if these groups of data are not usually seen together at this time then the activity label has a low probability of being the next activity—but if all the contextual labels are seen at this time and location with this person, then it has a high probability that activity label is going to occur next. The activity \(n \in N\) with the highest probability is returned as the inferred next activity.

This model can be extended by adding more sets of different kinds of data to the probability intersection. Lastly, if the next activity labels N do not intersect with the set of time values (the current time window) T, then the probability must be zero, \(Pr(N \cap T) = 0\), as those labels have never been observed in the time window previously.

We will test this hypothesis using the original CASAS datasets from the CRAFFT model experiments, by comparing the accuracy of next activity inferences to the actual next activity label in the dataset.

For our experiment, we used the CASAS smart home dataset [4] made publicly available and open access by the Centre of Advanced Studies in Adaptive Systems at Washington State University (https://​archive.​md/​EQRrK). This is the same dataset used by the CRAFFT research and we are able to use the data from three apartments to compare our results to the CRAFFT benchmark. The CASAS dataset uses several smart home apartments fitted with sensors and single occupants who continuously generate data. We selected three single occupancy apartments (HH101, HH102, and HH103) to simplify our inference model, as multiple occupancy prediction is out of scope of this research. In Fig. 1, we can see the floorplan of one of the apartments with sensor locations throughout the rooms. These are the data points we use for our event states and constitute the activity labels in our dataset, e.g. Leave_Home, Enter_Home, Watch_TV, Cook.

Each dataset subset is split into a 90/10 train test split, where the ‘training’ is done on-the-fly by using the train dataset as the historic observable data for the smart home, and the test dataset as the ‘live’ data to assess accuracy of next activity prediction.

Table 2 shows a sample of the CASAS dataset for the HH101 apartment shown in Fig. 1, which is floor plan of the apartment, with the event timestamp, the sensor code—which can also be seen on the floor plan, and the event code for that sensor, e.g. a door opening or closing, or a device being switched on. These data are the input for both the CRAFFT study, and our SBM proof-of-concept. The contextual data extracted and used as input variables in the SBM are: Current activity, Preceding event, Time window, Day of the week, Weekend or weekday, Location in the apartment.

To test our hypothesis, we used an experimental prototyping methodology to develop a proof-of-concept. To enable this we created a test framework, as shown in Fig. 2, in which we loaded the CASAS smart home datasets in a 90/10 train test split into separate tables. Using Python we coded the SBM to read the events sequentially from the test table, calculate the intersections of the data groups based on the train table, then output the event with the highest value as the predicted next activity. This value was logged in a results table in the database for comparison with the test table for accuracy at the end.

The SBM is designed to take any list of possible next activities. We envisage this as a secondary model to existing predictive models, such as a DBN—that would shortlist potential next activities. However, the SBM can take any number of next activities as input and will calculate the probability of occurrence for each. As there was no requirement for performance in this initial experiment for the proof-of-concept (i.e. this isn’t yet a production system) we can simply pass all known activities for each prediction step and calculate the probabilities and select the highest as the predicted next activity.

The time complexity of our model is O(n) linear time dependent on the size of the input, e.g. the length of the list of next activities. Combining the SBM with another model, e.g. a DBN, would optimise the performance by shortlisting or reducing the list of next activities.

The probability intersection for our sets of contextual data and activities implements the equation in Sect. 2.1 in Python. For each activity observed in the smart home sequence, a time window is looked up, as shown in Table 3. The data from the train table (the historic dataset) is selected within this time window (semantic block). Any contextual data present with the current activity is passed to the model. The SBM then calculates for each known activity (or a shortlist if used in conjunction with another predictive model) how often each potential next activity occurs after the current one, how often it occurs in that time window, and how often it is present with the contextual labels. All these bits of intersecting data together, as illustrated in Fig. 3, produce a probability score. The highest score of all processed activities is the predicted value.

The CRAFFT algorithm uses the current state (activity) along with the contextual data: activity location, activity time, activity day, and preceding activity from the CASAS dataset to augment its prediction calculation. Similarly we use as our contextual data: location, time, day, and also preceding activity. We chose to base our model on CRAFFT for its interesting use of contextual data. We determined that it can be used to create a model that extends the concept of CRAFFT; improving its performance but also designing it to be coupled with DBN or DL methods to give even better predictive output than the standalone SBM itself.

To determine where to demarcate the time windows we generated heatmaps of each dataset to elucidate any patterns in the occupant’s behaviour. In Fig. 4, we include all the activity events in the dataset, and we can clearly see bands of activity and inactivity. For example, the dark band between 0100 and 0600 would indicate sleeping. In the morning, there is the maximum amount of activity, which then reduces during the day. If we single out one activity, such as ‘bathe’ we can see in Fig. 5 how this activity always occurs within the same time window between 0800 and 1200. Ideally we would dynamically generate time windows for each individual occupant perhaps using a clustering algorithm such as K-means or K-Nearest Neighbour. However, this is out of scope for our proof-of-concept, so we manually selected time windows based on the average demarcation of the activity heatmaps, which are shown in Table 3.

To summarise, the SBM is a three stage process: