9. Intelligence Inference on IoT Devices

With the rapid advancement of artificial intelligence (AI), the proliferation of deep neural networks (DNNs) has ushered in a transformative era, revolutionizing modern lifestyles and enhancing production efficiency. However, the substantial computational and data requirements generated by Internet of Things (IoT) devices present a significant bottleneck, rendering traditional cloud-based computing models inadequate for real-time processing tasks. In response to these challenges, developers have increasingly turned to cloud offloading as a solution, despite the high infrastructure costs and heavy reliance on network conditions associated with this approach. Meanwhile, the emergence of SoCs has enabled on-device execution, particularly on high-tier platforms capable of effectively handling SOTA DNNs. This chapter offers a comprehensive review of intelligent inference approaches, with a specific emphasis on reducing inference time and minimizing transmitted bandwidth between IoT devices and the cloud. The review encompasses various aspects, including the background of inference, hardware architectures supporting inference, a diverse range of intelligent applications, inference libraries tailored for IoT devices, and different types of inference techniques for applications. Additionally, this work addresses the current challenges in intelligent inference, discusses future development trends, and provides future research directions.