Human-centred design of robotic systems and exoskeletons using digital human models within the research project SOPHIA

Human factors and ergonomics have long been applied to the development process of robotic systems (Parsons and Kearsley 1982; Sanders and McCormick 1998) but became more and more important for several industrial fields when the need for a flexible and adaptable production was met by a closer cooperation between humans and robots (Charalambous et al. 2015; Gualtieri et al. 2021). A recent literature review by Gualtieri et al. (2021) shows that, although safety aspects in the field of collaborative robotics in the industry have been studied more extensively until now, there is a steep increase of literature focussing on physical and cognitive ergonomics in the last years. Inherent to this line of research is a human-centred approach when designing human-robot interaction (HRI) systems. As part of the ISO 9241-210 (ISO 2019), there are clear recommendations and requirements to ensure a human-centred design when developing software and hardware parts of these interactive systems. With this, researchers, designers and manufacturers aim at ensuring the worker’s physical and psychological wellbeing when working with technological systems such as robots and exoskeletons (Lindblom et al. 2020). Another crucial factor for the success of HRI in collaborative workspaces is the acceptance of the technologies (Davis 1993).

Prior to the implementation of new technologies, the usage of digital human modelling software is a valuable addition to the user-centred and ergonomic analysis of workplaces. It enables the graphical modelling of human work activities as well as its time and ergonomic evaluation. Simulation results are based on methods in the field of Industrial Engineering, such as EAWS (Ergonomics Assessment Worksheet) and MTM (Methods-Time-Measurement) for estimating ergonomic risks and production time. Various key performance indicators (KPIs) for performance assessment of the work system can be derived and automatically calculated by the software. With this, it is possible to plan, simulate and visualize manual production tasks. Various valuable parameters for production, such as cycle time, desired task distribution, synchronization/waiting times and walk/drive paths can be compared before and after the implementation of a new technology.

When looking at the monitoring of physical aspects of work processes within a, for example, manufacturing setting, ergonomic and subjective data can, to some extent, be measured via questionnaires, Likert scales and interview protocols which are then analysed by statistics. Moreover, there are solutions for ergonomic monitoring that rely on the observation of whole-body postures or simple measurements like kinematics. However, these methods are often time-consuming and provide results only weeks after the initial subject’s assessment. In addition, they mostly rely on subjective data that was obtained after a complete shift, which impairs exact understanding of the relationship between certain technological changes and the human state. Looking at more extensive biomechanical analyses, similar problems occur which are aggravated by the fact that these kinds of analyses are bound to laboratory settings and cannot be transferred to real industrial workplaces due to lengthy data acquisition and complex measurement techniques (Seth et al. 2011). This implies that with current methods it is hardly possible to infer the user’s current physiological state or to adapt the robotic system or exoskeleton in real time. More precisely, the main limiting factor in monitoring ergonomics is the inability of measuring individual muscle forces that occur during a task and predicting the likelihood of the onset of future discomfort or injury over time (Winter 2009). Therefore, research aims at developing accurate assessment tools of worker’s muscle function that require short preparation time and can provide instant feedback so they can be viably transferred to factory settings (Ranavolo et al. 2018). In order to assist a worker optimally throughout a prolonged working task, muscular strength and stiffness need to be known variables so that the system can infer which muscle groups will be or have been affected by fatigue. Moreover, it is important to understand how a specific external intervention (e.g., the assistance of an exoskeleton or collaborative robot) might change the kinematics of the worker and could therefore contribute to muscle fatigue in other parts of the body than implied by simple kinematic measurements.

The EU funded SOPHIA (Socio-Physical Interaction Skills for Cooperative Human-Robot Systems in Agile Production) project tries to bridge this gap and proposes the development of a new data-driven method that is combined with numerical modelling in order to enable dynamic monitoring of human body movement with minimal set-up time and real-time assessment of muscle function and injury likelihood (SOPHIA 2022). The project’s goal is to use advanced non-invasive, low-cost sensing technologies for measuring the human body kinematics as well as bio-signals such as EMG (and/or external interaction forces) to create a personalized digital human model. First, this would have the advantage of a more accurate identification of internal body states in real-time since the musculoskeletal models would be fed by actual movement data, rather than by pre-defined mathematical rules. Secondly, this kind of personalized models would prevent the gender bias of commonly used masculine models, as it would be possible to adapt the model to the operator physical characteristics. Lastly, the proposed human models could play a significant role in decreasing the number of heavy manual lifting operations, and therewith absenteeism through musculoskeletal disorders, significantly. That is, the close feedback loop of actual muscle fatigue or sub-optimal ergonomic postures would make it possible to prompt the workers to optimize their resting periods throughout their shifts and assist them optimally despite individual differences or agile production lines.

This paper provides an example on how digital human modelling can be used for planning and simulating complete work processes and what parameters can be inferred by using this modelling approach. The described method is used for evaluating physical ergonomics and productivity of work processes at an early design stage that includes human-robot-collaboration. This will be closely linked to the ongoing activities and results from the SOPHIA project. Moreover, it will be shown how personalized human models are developed in the SOPHIA project to optimise the worker’s ergonomics when performing tasks with a robotic system or exoskeleton. For the former, the shown approach proposes how to develop highly adaptable robots that are able to reduce stress on joints and muscles by including real-time ergonomic parameters of the human into their movement strategy. The latter method will include steps to develop easy to use, cable-less and model-driven wearable technologies that are able to communicate ergonomic states to the human as well as adapting to them for assistance in manual handling tasks. Lastly, there will be a short description of what still needs to be done to ensure reliable, robust and personalized modelling in an industrial setting.

The term “Digital human modelling” (DHM) is not restricted to a specific type of model, software or outcome variables. There are multiple tools for simulating human interaction with the product or workplace and for designing ergonomic work environments. Some of them, for example Siemens Jack, DELMIA Human or EMMA are commercially available. So is the standalone software used in the SOPHIA project: ema Work Designer (EMA). In the following, this tool will be explained in more detail as an example of the current possibilities of such systems. EMA can be used to create a digital model of the complete workspace that can help to ensure safety in process and workplace design (cp Fig. 1): the workplace is virtually simulated and the software allows the evaluation of ergonomic and economic criteria (Bulliger-Hoffmann and Mühlstedt 2016). It was provided by a project consortium member and was used for the evaluation of physical ergonomics and productivity within an industrial use-case as it allows to simulate manual and semiautomatic work processes based on a digital human model in a 3D factory environment.

To this end, EMA includes a library of predefined tasks that is used to create a parameterized job description. The term “task” represents a human activity as a course of movement with specific start and end conditions satisfying certain constraints, for example: Pick the screw out of the {“box X”}, grab the {“screwdriver Y”} and screw {“screwing point Z”}. The software considers the required safety technology, such as sensors and safety zones, by analysing the movement areas and dimensions of the robot (Bauer et al. 2019). The most important result of an EMA simulation is the statement about the general feasibility and plausibility of the planned work process. In addition, the human model’ movements are analysed with established ergonomic methods. Ergonomic parameters are calculated based on the simulated motion data and the work environment and then assessed using standardized ergonomics assessment methods such as EAWS (Schaub et al. 2012), NIOSH (Waters et al. 1993) and OCRA (Occhipinti 1998). A second option is to fill the digital planning environment with data using motion-capturing methods. Motion data from a lab or field study, captured by various motion capturing systems, can be imported into the software and evaluated (Ivaldi et al. 2017). This means that real movement data can be used in virtual work planning.

In the field of human-robot collaboration (HRC) digital planning can help to ensure safety in process and workplace design. The workstation can be designed considering the required safety technology, such as sensors and safety zones, by analysing the movement areas and dimensions of the robot. For such applications, an extended robot library was included in EMA. It contains specific technical parameters (speed, degrees of freedom, etc.) for many robots available. One of the main branches using this software system is the German automotive industry (Fritzsche et al. 2019). In addition to them, several other industry branches like aerospace, machinery and supplier have found EMA to be a useful digital planning tool for their manufacturing processes.

The involved industrial partner for the presented use case of the SOPHIA project was a company that produces high-quality gears and mechanical precision parts and acts as a supplier for industrial branches (http://​www.​hankamp.​nl/​). Iterative analyses of the work instructions and videos of the current work process as well as CAD-data of machines, parts and logistics were used for a first virtual evaluation in EMA. The solution for a specific manual handling task, provided by the partners of the SOPHIA project, was a cooperative robot that would allow workers to operate multiple machines more quickly and with a lighter physical load. Within this simulation a digital model was therefore not only created for the worker but also for the robot in a 3D factory environment. At this point, task distribution between human worker and robot needs to be discussed and assigned within the software. With this, it is possible to better understand and communicate the requirements of this industrial use case to workers and other stakeholders in the company as well as to experts from the technology supplier before the robotic system is actually implemented (cp. Fig. 2).

The performance assessment, including ergonomic parameters, (cp. Table 1) were created semi-automatically through the digital simulation based on the input of load weights and occurring forces. Using various human model percentiles and the simulation of male and female workers based on the anthropometric data of DIN 33402‑2, EMA can account for individual differences.

Another possibility to make the ergonomic assessment even more precise, is importing kinematic measurements into the software. The respective data has to be analysed and stored as personalized ability profiles of employees by including characteristics from EAWS (postures, action forces, loads) (Spitzhirn and Kaiser 2020). With this, the software provides the possibility for accurate digital human modelling within industrial workplaces that entails individual ergonomic information about a specific worker. Although importing existing measurements is a valuable addition to virtual ergonomic evaluations, the SOPHIA project has a bigger goal in mind: real-time and automatic measurement and evaluation of ergonomic data. With this, the projects’ aim is to influences the human-robot interaction positively by adapting the robotic system or exoskeleton to the current state of the human.

Measuring the human body, and more specifically human kinematics, is not an easy task and many issues need to be faced since the human body is characterized by an extremely complex architecture. Existing DHM software, as described above, is a valuable tool for ergonomic risk assessment. This holds for desk jobs just as much as for the industrial sector where workers might need to lift heavy objects and might be assisted by robots, exoskeletons or other technologies more often in the near future. However, a highly versatile tool for biomechanical risk assessment becomes important when manufacturing processes change due to an agile production line. Moreover, the development of cooperative human-robot systems that aim to be adaptable to the human inputs would profit from real-time modelling of the human state. One goal of the SOPHIA project was therefore to develop a digital human model that can be used to classify the biomechanical risk from individual muscle groups during working tasks. Not only would this model contribute to an effective and easy way to ergonomic risk assessment, it would also allow to facilitate the import of personalized and precise data into modelling software such as EMA.

Current approaches in DHM systems provide a valuable starting point for the computational analysis of motor function and kinematics but synergies between muscles, or even between motor units, are highly variable across tasks, training, and fatigue levels (Buchanan and Lloyd 1995; De Serres and Milner 1991; Tax et al. 1990). Static optimization on the other hand relies on the so called “inverse-dynamics” approach, which requires joint moments to serve as the reference target for optimizing muscle forces. To estimate joint moments, force plates are needed to measure the ground reaction forces, which is not suitable for a factory setting or outdoor applications. Another huge disadvantage of these methods is their dissociation from real-time muscle activation and therefore their limited ability to describe the actual state of a worker. Common methods resolve this problem by modelling muscle reflex rules and by optimizing parameters like muscle-tendon, metabolic cost of transport (Markowitz and Herr 2016; Song and Geyer 2015) or the squared muscle activation sum a priori (Pandy and Andriacchi 2010). However, this a priori approach limits the design of effective technologies personalized to an individual worker since different objective functions will lead to different muscle force estimations and it is still unknown what the correct objective function for each movement is. In this context, the a priori optimization criteria chosen to evaluate the human ergonomic factors have a substantial impact on determining the final assessment.

One alternative solution for this problem was brought forward by partners of the SOPHIA project who contributed to the development of a whole-body dynamic online model to monitor and classify the joint overloading in real-time to reduce stressful body postures (cp. Fig. 3). For this, the common method on joint moment estimation was extended by allowing the subject to be in a dynamic state rather than static one and including the presence of external objects/tools that are manipulated. That is, the model enabled the tracking of task dynamics effects on human joints (Peternel et al. 2017). Therewith, it facilitates the online adaptation of the model parameters to the workers’ physical capabilities. This model was then able to estimate the joint moments online without the need of measuring it with a force plate. Such an approach can be applied in cases when either a collaborative robot can estimate the parameters of the unknown object/tool (e.g., measurement by its own force/torque sensor) or when objects/tools are known and the corresponding information are stored in a dedicated database (e.g., the tool can be detected by a vision system). As a result, the overloading joint moments can be obtained for subjects moving freely in the workspace. One valuable progress with this approach is that it renders the usually time-consuming and off-line procedure like the identification of human inertial parameter obsolete.

With these models, a robot is able to use a whole-body dynamic model of the human to optimise its position within a co-manipulation working task (Kim et al. 2021). The effects of an external load on the joints of the human body, overloading joint moments, were hereby minimized through detecting the mass properties of the human tool with cameras. The main advantage of this approach is that the robot can help to reduce the work-related strain and increase the productivity of the human co-worker.

Besides the accuracy of person-specific models for risk assessment, another important aspect for DHM systems is its real-time operation. For this, accurate digital copies of a person’s musculoskeletal systems are needed to understand biomechanical variables like muscle force, tendon strain, joint compressive load, and joint stiffness that are not applied in current kinematic models (Durandau et al. 2017). Knowledge about these parameters is crucial to foresee and prevent the onset and progression of musculoskeletal injuries in occupational settings. Moreover, real-time modelling makes it possible that an assistive technology, for example an exoskeleton, help when needed. This dynamic modelling approach, with an emphasis on the musculoskeletal system, is a core objective technology within the SOPHIA project and goes beyond the state-of-the-art measurements of joint angles, foot-ground pressure, and electromyograms, which only provide indirectly related information to injury likelihood (Lloyd 2021).

One solution is the combination of numerical modelling with data-driven methods. For this, physiologically correct computational models of the human musculoskeletal system were developed that are driven by EMG-activation patterns (Durandau et al. 2017), as well as by low-dimensional sets of activation primitives (Sartori et al. 2013), rather than pre-defined mathematical rules. This enables the accurate estimation of internal body states that are important indicators of human ergonomics and tightly depend on multi-muscle co-activation including joint compressive loads (Fernandez et al. 2014; Gerus et al. 2013) and joint stiffness (Sartori et al. 2015). These EMG-driven modelling methods are employed for understanding human-machine physical interaction in individuals wearing assistive technologies such as exoskeletons (Sartori et al. 2016). Recently these methods were translated to operate in real-time to establish novel model-based human-machine interfacing (HMIs) schemes (Durandau et al. 2016). The use of a motion capture system to compute joint position by a fully wearables system was compensated by inertial measurement units (IMU) suit that compute in real-time joint position. Here, the actual muscle EMGs are measured experimentally and use these to drive forward dynamic simulation of muscles. This resulted in real-time EMG driven model-based estimation of musculoskeletal forces. Only requiring EMG and kinematic recordings to explore the internal musculoskeletal variables significantly decreases the set-up time for industrial application (Durandau et al. 2020). However, in a factory setting, a fully wearable system is indispensable as cables could create unsafe environments. Therefore, muscle synergies and spinal reflexes could replace the EMG sensors in newer biomechanical models, so that it can better fit the factory and outdoor scenarios. In one experiment, the real-time implementation of muscle synergies was tested for the prediction of joint moment. Input of the synergies model, gait segmentation, was computed using the ground reaction forces of the treadmill in real-time. Therewith, the need for measuring EMGs was removed (cp. Fig. 4), resulting in real-time EMG-less model-based estimation of musculoskeletal forces with results for joint moment close to that of inverse dynamics that is considered the current golden standard. Besides, considering machine learning (ML), especially reinforcement learning, has shown strong potential to solve complex problems, ML neural controllers were explored with a new established fast simulation framework (MyoSuite, see also Durandau et al. 2022).

Another important aspect that enables the monitoring of physical endurance and therewith preventing injury risk in HRC is the development of a tool to assess online muscle fatigue (Wang et al. 2022) and human impedance models (Ajoudani et al. 2017; Fang et al. 2018; Dugan and Frontera 2000). Fatigue progression influences the maximum forces a muscle can generate and also the maximum velocity of contraction (Devrome and MacIntosh 2018). Therefore, the human musculoskeletal models must be updated with a fatigue progression model that also includes a recovery model. Such models give us an idea of how to vary the assistance provided to the user especially when they are performing an intensive task over an extended period of time.

It has to be considered that not every kinematically correct posture is also free from mechanical tensions and loads on musculoskeletal tissues, in particular if the worker uses compensatory strategies. Furthermore, new findings suggest that a model combining control- and error-based learning is the best choice to describe human control actions.

Many known approaches for an ergonomic risk assessment are valuable but far from being computationally fast and at the same time accurate enough for assessing the worker’s bodily state in real time at industrial workplaces. DHM software such as EMA enables user-centred ergonomic risk assessment in a simulated 3D environment and is most valuable for incorporating human factor principles at an early stage of work system design. However, these software applications are not made to assess the bodily state of a specific human being. It is used as a planning tool that can be easily adapted to several redesigns and customization. Therefore, importing personalized human data can be helpful for further insight into the requirements of specific workstations but is not necessary for a first design phase. Nevertheless, this information is crucial for an accurate analysis of the workers’ state and to prevent bias or miscalculation due to commonly used masculine model. Therefore, the development of computationally fast (i.e., real-time) models becomes very important after a design is chosen, because a detailed digital human model can be used to verify the results and further tune the design components. Some of this data could also be displayed (as an addition to the traditional ergonomic risk assessment by standards like EAWS and NIOSH) in the EMA results section for a deeper evaluation of the work situation and possible measures for improvement. Not only could they help to increase the value of modelling software like EMA but their estimated musculoskeletal forces (e.g. muscle and joint forces) could also be used on-the-fly for model-based robotic control paradigms. Within the SOPHIA project, a step to create these detailed digital human models is taken. At this time, the accurate estimation of muscle and joint forces in the intact moving human is not completely possible. One limitation lies in the use of EMG sensors which is not feasible for most of the industrial workplaces. However, the developed technologies provide new opportunities to predict injury risk in reconstructed working task within a lab environment and thereby develop bio-protective technologies. The EMG-less digital human models are not yet ready to predict correct muscle forces in a larger repertoire of movements but the developed technologies within the SOPHIA project can serve as a starting point for a more generalized model that can be applied in a variety of industrial settings. This would enable researchers and manufacturer to include a human-centred approach more easily into the design of human-robot-interaction system and provide new possibilities for biomechanical risk assessments and optimal assistance of humans during their working tasks.