Catering to Real-Time Requirements of Cloud-Connected Mobile Manipulators

Robots are inherently flexible machines and are thus essential parts of smart production systems. Here we can expect carefully engineered processes and workspaces, even in somehow flexible or adaptable production settings. This includes readily available models suitable for detailed planning of robot actions. Flexibility could be categorized as follows: (1) the ability to deal with local uncertainties like variations in part positions or wear of tools. (2) Overcoming larger process variations like random placement of parts (e.g. bin picking), unexpected obstacles in shared workspaces, or new part variations. (3) Supporting implementation of new processes. We consider category 3 to be relevant during design time, when commissioning, or even during ramp-up or training phases. Category 1 and 2 are run-time properties and thus relevant functions of a suitable control system. While the category 1 can be achieved by simple sense-act scheme using locally installed sensors, category 2 is better served using sense-plan-act as an approach. Even though planning is a broad topic, it involves data or computationally intensive tasks like managing or updating models [12], as well as running planning algorithms. When considering mobile robots, planning tasks are obvious candidates to be offloaded to remote servers. However, sensing and motion generation can also be computationally intensive. Offloading computation from the mobile robot would save space, weight and battery power. The ANNIE robot¹ we use in our experiments requires about 250W for the onboard computers. Assuming a lithium ion battery with a specific energy of 140 Wh/kg is used to power the system over an 8 h operating cycle, we see that ANNIE needs 14 kg of battery weight just for onboard computations. Designing a distributed system, especially one that incorporates heterogeneous hard- and software while still being supportive in addressing key challenges in domain specific application development, is an extensive undertaking requiring considerations in many different directions. This paper is limited in scope to an initial motivation for using a mixed real-time system in conjunction with standard and advanced communication and IT-concepts, a classification of typical manipulation tasks dependent on elasticity concerning quality of service, and an analysis of an exemplary use-case.

This paper is structures as follows: After the discussion of the related work, in particular concerning progress in wireless networking as well as related software, we motivate timeliness from the point of view of a dynamic robot application. In Sect. 4 we present a scheme for fostering a distributed soft real-time system in conjunction with a fault tolerance scheme in order to implement a control task. This is followed by an in-depth analysis of a concrete application scenario in Sect. 5.

Our work is the result of an ongoing interdisciplinary project with experts from both the robotics domain as well as from communication and cloud systems. One aspect is the integration of technologies. Thus, it is necessary to briefly summarize the state of the art in both communication- and IT-systems as well as in the robotics and automation software domain for a broader perspective.

Robots implement interaction with the physical world and are by themselves physical objects. This implies real-time constraints on their control systems. On the other hand, the advent of ROS [9] has proven the usefulness of software systems with weak or no real-time support in this area. When taking a look at robot applications we can see different tasks or sub-tasks, some of which are sensitive to timing in some way, others are not. Timing-sensitive tasks can be found in applications where the process to be carried out is timing sensitive. An example is glue application. Here, deviations from a carefully engineered trajectory will cause imperfect results. The application process is dynamic and a temporary loss of control results in undesired behavior. Other processes are not inherently timing sensitive. For example, bin picking: The objects in the bin do not move on their own, the robot system can be completely implemented with best-effort timings. In this case, the robot would move incrementally whenever a new waypoint is generated. The system would be quite slow and may not be feasible for commercial application. There is however a limit on how fast a best-effort robot can become. The transition to swift and fluid movement of the robot again introduces a dynamic process where temporary loss of control again results in undesired behavior. Robotic control architectures can be classified in hierarchical, behavioral, hybrid approaches. All of which are organised in higher and lower level subtasks. The low-level part can only act upon locally available information. This part can easily made real-time compliant, in contrast to high-level tasks. Furthermore, reactions are either very limited or become autonomous, which can be undesirable because of the limited access to relevant knowledge on the lower level. In the remainder of this paper we discuss a solution whereby we continue controlling the robot in case of a timing fault of the computer or the communication system. This is done for the most part deliberatively based on pre-planned motion at a higher level. We also incorporate data for fast reaction from a soft-real-time sensor based on sensor prediction.

Our basic architectures (see. Fig. 1) consist of a high-level, time-aware runtime, a low-level runtime, and additional time-aware data-sources. The low-level runtime implements a time-triggered control loop and is local to the robot. It is hard real-time and has access to local information from select sensors. It furthermore subscribes to data sources with a similar time scope from the event driven high-level system, if necessary, to make decisions. It is only concerned with actions relevant to the current situation. The high-level runtime implements the overarching robot program. It is time-aware in the sense that it knows timing requirements of both the low-level runtime and the application task either explicitly from the user provided program or implicitly derived from models. It continuously communicates with the low-level runtime during task execution and can be consolidated onto a remote server. This is most preferably an edge server with bounded communication latency, but implementation on a core server with soft real-time communication link to the robot is conceptually also possible. Time aware data sources are endpoints in event-driven processing chains which provide useful information. The emission of such events can be aperiodic, but because the source is aware of the timing requirements of the subscriber, information is processed to best fit the subscribers needs. This is being achieved by interpolation or prediction onto the required points in time.

In order to have the benefits of global knowledge gained from various data processing services including object detection, recognition and tracking, as well as demanding algorithms like collision-free motion planning and optimization, we want these tasks to be performed on off-board. As a layered control architecture, we need a scheme for mediation between higher-level (slow, soft real-time, event-based) and lower-level (fast, hard real-time, cyclic) layers.

While it is possible to trigger a sequence of otherwise autonomous behaviors implemented at the lower level, this approach does not fully benefit from detailed knowledge and models available only at a higher level. Thus, we implemented a hybrid approach. The concept is based on a large number of pre-computed movement variations to form an overall deliberative action. Tightly spaced transition points enable the change between trajectories in order to be able to still react swiftly to external events. This included real-time guarantees. We propose a spacing of 5–20 ms depending on the application as a suitable amount of transition points. This is also dependent on the capabilities of external data sources needed to make decisions. A time-aware data source, such as a camera system with object tracking capability, is expected to provide aligned data, if necessary based on predictions as input to the low-level runtime (see Fig. 2) as well as for the high-level runtime for planning.

The various movement variations can be pre-planned using parallelization on powerful off-board servers with a larger granularity of about ten transition points. We will present values for a concrete example in Sect. 5.2. The number of variations grows rapidly over time so it is important to limit the number of transition points as well as the number of alternative planning goals to a number necessary for the application. This requires a better understanding of the application (sub-) task that we want to implement, but avoids computationally expensive overprovisioning.

The movement is encoded in the form of a graph structure where transition points are a group of nodes labeled with the same timestamp, t (see Fig. 3). Each node represents an intermediate pose, i.e. an intermediate goal configuration. The edges represent trajectory segments to reach the next set of nodes at the next transition point. The maximum number of trajectory segments between transitions is \(n_{t-1} \cdot n_{t} \cdot n_{t+1}\) with n as number of nodes. Not all transitions are possible in all cases, because they exceed the dynamic limits of the robot. Other nodes can lead to paths where no alternatives are available or where alternatives have a different spacing of transition points. Autonomous behaviors are still necessary for implementing fastest transitions to desired actions, e.g. reactions to collisions. Special care must be taken for transitioning back from autonomous to pre-planned actions. The simplest approach to this problem is to do it when stopped. Another is to use motion generation at the low-level runtime for a brief period until a node in the deliberatively planned graph structure is reached. This also requires the use of relative timestamps for nodes because execution time of the transition is be variable. Actions group steps or branches of the program with distinct purpose (see Fig. 3). They are similar to traditional behaviors in the sense that activation and execution is autonomous based on locally available information to make decisions. However, the comparison can also be misleading since, in our case, we emphasize deliberatively planned movement variations using global models. The decision process within the local real-time controller is realized through a user defined script provided at run-time initialization. This transition program is executed with every cycle of the control loop and is implemented with the lightweight embedded language, Lua version 5.x [5], which we adapted for real-time use. In every cycle the transition program checks which transitions are available for the current action and on the current cycle. It then evaluates their necessity based on locally available information or on the lack of information due to communication interruption. Transitions to autonomous behavior may occur on any cycle. Transitions to pre-computed alternatives only occur on specific transition points: points in time for which nodes are available in the graph structure. At these points transitions between pre-planned trajectories are available.

In this section, we discuss a scenario as a use-case for our system architecture. It consists of an assembly line with multiple work stations in sequence. The work stations are designed for human employees to operate. We want to alternatively operate one workstation using an anthropomorphic mobile manipulator. This implies not only a collaborative setting, i.e. co-existence [2] with human co-workers at neighboring workplaces, but also the requirement of completing tasks at the same speed as a human workers.

Real-time requirements in robotics depend on the specific tasks of an application. Some tasks (Sect. 3) require hard real-time control and a fixed update rate; others are more flexible but still sensitive to timing faults at relevant execution speed. We have presented a scheme of realizing such tasks using a distributed runtime which deliberatively plans many alternatives of concrete behavior to relax time constraints on higher-level planning. As a basis for choosing between alternatives in a timely manner we used aligned predictions from asynchronous event driven sensor data processing which also reduces real-time constraints on sensing. Overall our approach represents a tradeoff between computational resources and communication bandwidth on the one hand and latency as well as jitter in processing and communication on the other hand. In future work we plan on using the presented scheme in the implementation of other sub-tasks of the application. An interesting question is the possibility of performance improvements in the discussed sub-task for tool insertion by automatically adjusting the planning horizon based on the observed dynamics of the process. Further automatic adjustments include the timing properties of the off-board, time-aware data sources, in particular the communication channel. In this work we had to identify these manually. However, we expect this to be an automatic feature, especially in next generation cellular networks where even short-term predictions are possible.