Supporting AI-powered real-time cyber-physical systems on heterogeneous platforms via hypervisor technology

This work presents a concrete software architecture for supporting AI-enabled CPS with mixed-criticality components. The proposed architecture targets heterogeneous computing platforms that couple asymmetric multicores with programmable logic (FPGA). It leverages hypervisor technology with strong isolation, hardware acceleration of AI algorithms implemented in programmable logic, and monitoring strategies to take over the control of the system whenever non-critical software components fail, are attacked, or produce results that are deemed unsafe. The architecture is then specialized for the case of autonomous flying drones, showing how it can be used to build a safe and secure tracking application.

The rest of the paper is organized as follows: Sect. 2 discusses some relevant related work; Sect. 3 presents the general architectural approach; Sect. 4 describes how the proposed architecture has been instantiated to a specific use case consisting of a visual tracking application performed by a drone; Sect. 5 reports some experimental results; and, finally, Sect. 6 concludes the paper and presents some future work.

Type-1 hypervisors are considered a preferable choice for the proposed architecture thanks to their small code base and better hardware control, which can guarantee high levels of security, safety, and time predictability. Furthermore, Type-1 hypervisors are more capable of guaranteeing predictable virtualization-related latency (e.g., in response to interrupts), inter-domain communication, and scheduling of virtual machines.

Several Type-1 hypervisors are available and some relevant research work has been done to integrate them for executing specific CPS applications, which are reviewed next. Klein et al. (2018) proposed a solution based on seL4 to separate trusted and untrusted software in a UAV platform, focusing on the issue of guaranteeing security and isolation in the system. Almeida and Prochazka (2009) presented a solution based on PikeOS (SYSGO) to provide safe and secure partitioning for integrated modular avionics (IMA) in spacecraft applications. Craveiro et al. (2009) proposed to use the partitioning features of ARINC 653 in Space Real-Time Operating System (AIR) for providing isolation in the development of aerospace systems, but only for IA-32 and Sparc architectures. Pérez and Gutiérrez (2016); Pérez et al. (2016) implemented a real-time publish-subscribe communication mechanism in the Xstratum (Crespo et al. 2009) hypervisor integrating ARINC-653 with the Data Distribution Service (DDS). Biondi et al. (2021) proposed a hypervisor-based architecture for safety-critical embedded systems providing both time/memory isolation, security, real-time communication channels, as well as I/O virtualization to allow different virtual machines to share the peripheral devices. Farrukh and West (2022) proposed a hypervisor-based solution characterized by low overheads in accessing resources. Their approach requires strict time guarantees for both domains, forcing the execution of Linux on a single core with SCHED_DEADLINE (Lelli et al. 2016), which is a viable solution in terms of real-time constraints, but can introduce several limitations in the implementation of complex AI-based solutions.

A peculiar feature of AI-powered cyber-physical systems is their massive computational workload for executing AI algorithms such as deep neural networks. The most demanding functions of these algorithms need to be accelerated on specific hardware, such as general-purpose graphics processing units (GPUs) or field-programmable gate arrays (FPGA), to satisfy real-time requirements.

Modern GPU-based heterogeneous platforms benefit from powerful and mature software support to accelerate AI algorithms, which allows the user to significantly contain the effort for achieving efficient implementations of tasks like object detection, image segmentation, and tracking. The acceleration frameworks for GPU-based platforms also allow a developer to seamlessly use, with no or just a few modifications, the AI models available in frameworks such as Tensorflow, PyTorch, and Caffe, even with the native parameters with floating-point precision.

On the other hand, when compared to FPGA-based platforms, GPU-based platforms are very demanding in terms of power consumption and struggle in providing a high degree of time predictability for hardware acceleration. Their power consumption can be one order of magnitude larger than the one required by FPGA-based platforms (Sciangula et al. 2022; Qasaimeh et al. 2019). Furthermore, as observed by Cavicchioli et al. (2017), GPU acceleration introduces highly variable delays that cannot easily be bounded a priori, also due to the contention occurring on shared memory in the case of memory-intensive GPU tasks. As such, GPU-based are not the ideal solution for battery-operated CPS such as drones.

Besides being characterized by less energy consumption, FPGAs provide a very predictable execution behavior with respect to GPUs for hardware acceleration.

Two main approaches are used to accelerate deep neural networks by means of FPGA technology: The main disadvantage of FPGAs is that they require a larger programming effort than GPUs, especially when developing a network-specific accelerator. Another restriction is the limited amount of FPGA resources that is available in several embedded platforms, which calls for the usage of dynamic partial reconfiguration (Biondi et al. 2016; Seyoum et al. 2021) of the FPGA at the cost of additional delays when serving acceleration requests. Furthermore, as observed by some authors (Vaishnav et al. 2018; Happe et al. 2015; Rupnow et al. 2009), even without dynamic reconfiguration, sharing an FPGA among tasks managed by a preemptive scheduling policy is not trivial, due to the significant amount of time required to save the state of the device.

Fortunately, especially in the case of softcore accelerators such as the DPU, compilation and optimization frameworks are available to drastically simplify the deployment of accelerated neural networks. These frameworks employ pruning and quantization (Zhou et al. 2017) of the network parameters to achieve an efficient execution on the FPGA. The accuracy drop of these optimization processes was found not to be significant in several application scenarios (Gholami et al. 2022; Liang et al. 2021). The optimization algorithms dealing with the conversion from floating point to integer values are indeed now efficient enough to guarantee consistency in the transformation of the models from one platform to another (GPU to FPGA) (Ding et al. 2019). This makes FPGA-based MPSoCs the ideal reference platform for resource-constrained embedded systems.

The works reviewed in Sect. 2.1 share two main limitations: they are all tied to a specific application and do not address the support of AI algorithms with hardware acceleration. Furthermore, none of them investigated in details the usage of hypervisor technology to implement fail-safe/fail-operational control algorithms. Furthermore, despite the benefits of FPGA acceleration highlighted in Sect. 2.2, none of the works discussed in Sect. 2.1 considered this relevant technology.

This work advances the state of the art by presenting a general architecture for autonomous CPS that leverages FPGA-based embedded platforms. The performance and capabilities of the presented architecture are then evaluated for a visual tracking application implemented by an autonomous drone, putting particular emphasis on the role of the architecture in the development of fail-safe/fail-operational control algorithms.

The FPGA is used to synthesize a number of devices that are assigned to the virtual machines by the hypervisor. In particular, each device is exclusively assigned in a pass-through way to one domain only, while the hypervisor is responsible for providing strong isolation. In particular, the Xilinx DPU device is accessible by the Linux domain, while all the other devices are assigned to the critical domain. All custom devices synthesized on the programmable logic are described in the following list: Efficient implementations of the drivers for the above peripherals, except the DPU, were performed from scratch to offload the CPU as much as possible, as well as minimize execution time variability and the number of memory accesses.

The two domains exchange data by means of two non-blocking communication channels based on shared-memory regions provided by CLARE, where the Linux domain acts as a producer and the critical one as a consumer. The channels are accessed by means of a middleware (available for both Linux and FreeRTOS) that does not require the intervention of the hypervisor at each access and ensures wait-free synchronization in the presence of concurrent accesses. The first channel is used to exchange setpoints for the drone controller, whereas the second one is used to transmit heartbeat packets for health monitoring.

The Linux domain is responsible for visual tracking and navigation. It includes four tasks, namely Camera, Detector, Tracker, and HB generator. Details on these tasks are reported in Table 2.

The Camera task periodically captures a new frame from the camera and puts it in a queue of frames ready to be processed. The Detector task performs object detection by accelerating the inference of a YOLOv3 (Redmon and Farhadi 2018) deep neural network on the Xilinx DPU. The YOLOv3 model was trained on the cityscape dataset (Cordts et al. 2015), using a Darknet-53 backbone, modified (with respect to the standard implementation) to process an extra output from the pyramidal feature extraction stage, to improve its performance.

To be executed on the DPU, the neural model was quantized (transforming its weights from 32-bit floating point values to 8-bit integer values) and pruned (removing the parameters with less contribution) to reduce the memory footprint and the amount of operations to be executed by the accelerator. The model was then compiled to generate DPU-specific instructions to speed up the inference. This process was performed by means of Xilinx’s Vitis AI.

Besides the inference, a YOLOv3 model requires a pre-processing and a post-processing stage. In particular, the pre-processing stage takes a frame from the Ready-frame queue, rescaling and normalizing it to the YOLOv3 input size (512\(\times \)256\(\times \)3). The post-processing stage derives all bounding-box coordinates, executes the Non-Maximum Suppression algorithm, and inserts the result into a queue to be processed by the Tracker task. The purpose of the Object Tracking function is to predict (by a Kalman filter) the position of a tracked object to keep identifying it even when it is missed by the object detector. The coordinates of the tracked target are then used to compute the setpoint to be sent to the critical domain.

The minimum, average, and maximum observed execution times of the functions involved in the object detection pipeline are reported in Table 3.

The execution times reported in Table 3 have been measured by a performance counter for a time interval of 20 min, running each task in isolation to prevent interference from higher priority tasks. As clear from the results, the pre-processing and post-processing functions represent the major bottleneck in achieving real-time performance. To overcome this problem, all three functions performing object detection were implemented as a concurrent multi-thread pipeline, as illustrated in Fig. 3.

The number of parallel threads has been set equal to the number of cores assigned to the Linux domain (three, in our setup), with the benefit of increasing the throughput from 16 to 27 frames per second (FPS). Note that the tracking function cannot be parallelized, because the position of the tracked objects depends on the computations related to the previous frame.

The HB generator task is responsible for the health monitoring activities within the Linux domain. It periodically produces two heartbeat timestamps, reporting the health of the Linux system and the application. The system-level timestamp is updated each time this task is executed, while the application-level timestamp is updated every time a new setpoint is generated by the Tracker task.

The critical domain is composed of four tasks, whose periods and priorities are reported in Table 4, while execution times are reported in Table 5.

Observed execution times of Table 5 were obtained using the same technique adopted for Table 3.

The HB checker task periodically verifies the heartbeat values sent by the HB generator to detect possible faults related to the Linux domain, notifying the Safety module task when a fault is detected. The reaction time to a fault is given by the period of this task multiplied by a user-defined parameter that indicates the number of tolerable task executions without heartbeat updates.

The Backup controller task is responsible for handling an alternative setpoint generation when the fail-safe mode is activated. As shown in Table 5, the execution time of this task is significantly shorter than the others because, in the current implementation, it simply keeps the quadcopter hovering in the position recorded when the fault is detected. In general, however, this task could be used to implement more complex actions, such as controlling the drone to perform a safe landing.

The Safety module task is responsible for producing the input to the flight controller. If no fault is detected by the HB checker task, it reads the latest setpoint provided by the Linux application, otherwise, it switches to the Backup controller as an alternative source of setpoints. The Safety module task also performs high-level health monitoring functions related to the behavior of the application, verifying that the provided set points are not jeopardizing designated safety constraints. The current implementation reads LiDAR data and conducts an additional control loop to keep the drone at a minimum user-defined distance from possible obstacles on the path to the next waypoint.

The Flight Control task can work in two different modes, manual and AI-driven, selected by a switch on the radio transmitter. In manual mode, the setpoint is taken from the radio transmitter, while in AI-driven mode the setpoint is taken from the Safety module task. In both cases, the Flight Control task reads the IMU data and stabilizes the quadcopter using two hierarchical PID control loops. The inner loop controls the angular rates, while the outer loop controls the quadcopter attitude. Finally, the outputs generated by the Flight controller are sent to the Motor mixer for actuation.

The vehicle used for the implementation is an F450 class quadcopter with four 1045 propellers controlled in the X standard configuration. Figure 4 shows the block diagram of the Xilinx ZCU-104 Ultrascale+ MPSoC configuration used for our application.

The camera is a Logitech HD Pro C920, which natively provides frames with a resolution of 320\(\times \)240, which is the closest one to the input size of the adopted neural network. The IMU is an MPU-9250, a 9-DOF inertial device with gyroscopes, accelerometers, and magnetometers that communicates with the system using a 3.3 V \(I^{2}C\) device (set in Fast mode to work with 400 kHz clock) implemented inside the FPGA programmable logic. Additionally, this device has a configurable filter that can be used to reduce noise and improve communication reliability.

The radio receiver also takes advantage of FPGA acceleration. It uses the custom AXI peripheral we designed to handle the PPM external interrupts (EXTI line) to offload the CPU from the interrupt service routine and reduce interferences and jitter on the control task.

LiDARs communicate through a dedicated UART implemented in FPGA with a baud rate of 115,200 bit/s. Although it is possible to generate PWM signals using the hardware timers provided by the processing system, we built our custom PWM peripheral to be completely independent from the CPU, ensuring exclusive access by the RTOS domain using a dedicated driver.

The Electronic Speed Controllers (ESCs) used in the quadcopter are the BL-Heli, which accept PWM pulses from 50Hz (legacy PWM) to 12KHz (OneShot-42 protocol), so they can perfectly handle the 250Hz PWM signal of the custom HLS peripheral. The Racestar 980 KV brushless motors are paired with the 1045 propellers and can provide a thrust of 3.2 kg.

The Xilinx DPU is the most complex and resource-demanding FPGA device required by the application because it uses a very large portion of LUTs in the programmable logic region and almost all the marginal resources of the board, like Digital Signal Processors (DSP), Block RAM (BRAM), and Ultra RAM (URAM), as reported in Table 6.

Note that the design occupies less than half of the available resources. This choice is motivated by the fact that, in the next future, the complete application will be moved to the Kria K26 SOM, with an ad-hoc carrier board to further reduce weight and consumption, but its SoC contains half of the resources available on the XCZU7EV-2FFVC1156 powering the ZCU104 board.

This specific setup led to the power specifications reported in Fig. 5. How it can be seen from the figure, the major source of power consumption resulted to be the dynamic one in PL due to the DPU runtime and the intense switching activity it involves.

The battery selected as a power source for the system is a 3300 mAh 4 S LiPo, which is sufficient to guarantee the maximum power consumption specified in Fig. 5. In fact, it can always provide a sufficient voltage/current to the D36V50F12 voltage regulator to maintain it in its optimal operating spot and a stable power rail of 12 V with a maximum load current of 4.5 A. The maximum payload of the quadcopter resulted be 1.5 kg in this specific configuration. Additionally, the external devices connected to the platform present the following power specifications. The Racestar BR2212 motor datasheet reports that, applying a Voltage of 11.1 V, each motor paired with 1045 propellers (the ones we used) absorbs a maximum current of 10.6 A, for a total maximum current of 10.6 \(\times \) 4 = 42.4 A. The Fs-iA6 radio receiver, supplied with a voltage between 4 and 6.5 V, has a maximum power consumption no higher than 520 mAh. The power consumption of the MPU-9250 is quite low, since it must be supplied with a voltage of 3.3 V and reaches 3.7 mA when all three sensors (gyroscope, accelerometer, and magnetometer) are powered on. Each directional LiDAR works with an input voltage between 3.7 V and 5.2 V, absorbs an average current less or equal to 70 mA, a peak current of 150 mA, and a total power of less than 0.35 W. In this setup, we used two of these sensors.

Figure 6 shows the frame rate distribution of the object detector task under the configuration without hypervisor (red bars) and with hypervisor (blue bars). The part of the plot in purple color represents the overlapping portion of the two histograms. The smoother envelopes of the two histograms are also shown with the corresponding colors. As clear from the plot, the two histograms almost overlap, meaning that the hypervisor introduces negligible overhead and does not degrade the overall system performance significantly. On the other hand, the hypervisor allows isolating the two execution domains, preventing malicious attacks carried out on the Linux domain to propagate and affect the critical functions running in the RTOS domain.

In another test, the object detection performance resulting from a 3-core configuration with hypervisor has been compared with the one achievable on the full ZCU-104 without hypervisor, that is, assigning all four cores to Linux. The results are illustrated in Fig. 7, which shows that, by allocating one extra core to Linux, the average frame rate of the object detection task increases from 27.5 FPS to 29.6 FPS.

Note that, in the 4-core configuration, the object detection pipeline uses four parallel threads to match the number of physical cores available on the platform. As expected, this leads to a performance increase, but the observed improvement is not significant, since the processing pipeline is constrained by the acquisition rate of the camera (30 FPS), which limits the benefit of the increased HW and SW parallelism.

Viewed from another angle, the results reported in Fig. 7 show that the two-domain architecture enabled by the hypervisor does not significantly degrade the performance, with respect to a full platform configuration, but certainly provides other relevant advantages in terms of time predictability and security for the critical components of the system.

This section reports on two experiments aimed at showing the time it takes for the generated data to be propagated from one domain to the other. Since this operation involves a data exchange between two very different operating systems, the communication latency depends on different factors. In particular, from the moment in which the data is written by Linux into the shared memory, three factors come into play: (i) the period of the consumer task running in the FreeRTOS domain, (ii) the time at which this task is scheduled by FreeRTOS, and (iii) the interference experienced by the consumer from the other tasks, which depends on the assigned priorities and the task execution times.

Figure 8 illustrates a possible interleaving of the producer and consumer tasks, where the delay is significant. In the figure, the message is sent by the producer at time \(t_1\), it is delivered to the other domain at time \(t_a\), and finally consumed at time \(t_2\). As it can be seen, the overall end-to-end delay (\(t_2 - t_1\)) is given by the sum of the channel latency (\(L_c\)), the activation interval (\(A_c\)), the interference of the high-priority tasks (\(I_{hp}\)), and the computation time of the consumer task (\(C_c\)). Since the channel latency is always below one microsecond and the execution times of FreeRTOS tasks are in the order of a few hundred microseconds, the major contribution to the end-to-end communication delay is due to the period of the consumer task, which is in the order of milliseconds.

The best-case situation for the end-to-end communication delay is illustrated in Fig. 9, where the consumer task is executed just after the message is delivered to the FreeRTOS domain. In this case, the end-to-end delay is in the order of a few hundred microseconds.

The end-to-end delay measurements performed in this experiment confirm the observations reported above. Figure 10 shows the distribution of the end-to-end delay for the setpoint communication, measured over one hour of continuous execution from the time the setpoint is generated in Linux to the time it is read in FreeRTOS by the Safety module task, which has a period of 10 ms and the lowest priority.

As expected, the maximum observed delay resulted to be 9.5 ms (close to the period of 10 ms assigned to the Safety module task), whereas the minimum observed delay resulted to be 890 \(\mu \)s, due to the sum of its own execution time and the suffered interference from some higher priority task.

Figure 11 shows the distribution of the end-to-end delay from the time the heartbeat is generated in Linux to the time it is received in FreeRTOS.

In this case, the maximum observed delay resulted to be of 3.74 ms (close to the period of 4 ms of the HB checker task), whereas the minimum observed delay resulted to be of about 303 \(\mu \)s, shorter than the other, because the HB checker task has the highest priority and cannot suffer interference from the other tasks.

A final experiment was carried out to measure the latency of the fail-safe procedure triggered by a system fault. For this specific test, the drone is programmed to track a target by controlling only the yaw angle. Hence, Fig. 12 reports the variation of the yaw angle during a tracking operation, when the backup controller is invoked to keep the drone in a safe state after a system fault is injected in Linux.

In this experiment, the system heartbeat validation threshold was set to 3, meaning that the HB checker task can tolerate 3 readings of heartbeat data in the FreeRTOS domain without detecting an update. Note that, since the period of the HB checker is set to 4 ms and the threshold is 3, the expected delay to detect a fault is between 8 and 12 ms. In Fig. 12, the measured delay is represented by the purple arrow between the two vertical red dashed lines, denoting the transient interval between the normal functioning and the fail-safe mode.

In this test, the fault has been injected in Linux after 6.235 s from the beginning of the plot and the backup controller took place at \(t = 6.245\) seconds, that is, after about 10 ms. The orange horizontal line highlights the yaw angle recorded when the fail-safe mode was enabled. In this implementation, the backup controller has been programmed to keep the yaw angle to that reference value.

The plot shows that, when the backup controller was activated, the yaw angle was \(36.3^{\circ }\) and the flight controller acted to keep the yaw angle at this reference value, while simultaneously keeping both pitch and roll angles at \(0^{\circ }\) (hovering). Notice that a little overshoot occurs in most cases, since the yaw PID controller is the one with less authority over the physical system. In fact, yaw actuation is generated by differences in motor torques, while pitch and roll are generated by changing thrusts, so the faster the control (larger gains) the higher the overshoot to be compensated. This can be observed in the figure after the second red line, when the angle continues increasing for a few milliseconds, after which it converges to the reference angle recorded at the fail-safe activation. This behavior is caused by the system dynamics rather than by a delay in the execution of the flight control task.

Note that, without the fail-safe mechanism, the drone would no longer receive a setpoint and therefore would continue to apply the last valid setpoint received, thus keeping rotating around itself. In a more complex use case involving multiple degrees of freedom, this situation would inevitably lead to dangerous consequences.