Methods from the field of Human Activity Recognition (HAR) can be employed to implement a REACH subsystem for activity monitoring. Such algorithmic methods use annotated training data, collected from on-body and ambient sensor technology, to construct mathematical models of activity patterns based on supervised machine learning methods (Figure 1). The trained models, eventually, are able to scan through newly incoming sensor signals in order to detect if and when certain target activities were performed by persons who are monitored by these sensors. Thus, aggregated statistical activity profiles can be constructed based on the detected activities throughout continuous sensor measurements with the help of Human Activity Recognition models.

Data annotation

Human activities can be described on many different semantic layers, ranging from basic actions like hand gestures to complex composite activities (Figure 1). Moreover, locomotion patterns of the entire body, as well as actions of the left and right hand have to be considered separately. For training supervised activity recognition models, sensor time series need to be labelled with the exact start and stop time points of activities throughout all these different layers.

The labelling of sensor time series data has to be performed manually with the help of video recordings of the performed activities (Figure 2). Because this has to be done for all activity types and layers, the creation of labeled training data is very time-consuming.

Figure 1: Supervised HAR for different complexity levels of activities

Figure 2: Video-based manual annotation process for sensor time series

Data acquisition protocol

In order to acquire sufficient amounts of training data for HAR models, controlled experiments are necessary, in which test subjects are instructed to follow certain activity sequences in a repeated fashion. For Touchpoint 2, Schön Klinik, TUM and Fraunhofer IAIS jointly designed such data acquisition experiments. For that purpose, scripted activity protocols were designed that aimed to be realistic with respect to the daily routines of real patients. Such scripts need to outline the experimental setup, as well as a high-level scenario and very detailed action sequences (Figure 3).

Figure 3: Essential components of a data acquisition protocol