Lembda: Optimizing LLM Inference on Embedded Platforms via CPU/FPGA Co-processing

The rapid advancement of Large Language Models (LLMs) has provided new opportunities for edge applications. Embedded FPGA platforms are well-suited for deploying LLMs at the edge due to their hardware programmability and high energy efficiency. However, the limited hardware resources of embedded systems pose significant challenges to comprehensive acceleration. As not all operators can be offloaded to programmable logic (PL), the remaining operations must execute on the processing system (PS), creating potential performance bottlenecks and degrading inference efficiency. In this paper, we introduce Lembda, a collaborative optimization framework that harnesses the computational capability of both PL and PS for efficient embedded LLM inference. On the PL side, we employ W4A8 quantization and implement high-throughput GEMM/GEMV kernels. On the PS side, we optimize high-precision operations by exploiting model sparsity in attention layers and approximate nonlinear functions via lightweight polynomial fitting. Moreover, we carefully orchestrate PL/PS operations to exploit operation-level parallelism and further enhance performance. Evaluations on the AMD Kria KV260 platform demonstrates that Lembda delivers \(187.9195\ \mathrm {tok/s}\) for prefilling and \(9.7857\ \mathrm {tok/s}\) for decoding on the Qwen2.5-0.5B-Instruct model, achieving \(65.9\times \)/\(3.8\times \) speedup compared to the baseline method with negligible accuracy loss.