VIRSA: Vectorized In-Register RSA Computation with Memory Disclosure Resistance

Memory disclosure attacks give adversaries access to sensitive data in memory, posing a serious threat to the security of cryptographic systems. For example, the plain private key in RAM is exposed to the attacker during RSA operation. In this paper, we propose a register-based RSA system with high efficiency, called VIRSA, so that CRT-enabled 2048-bit RSA is entirely carried out on CPU registers. The private key and the intermediate results during the calculation process are all stored in registers, and will not appear in memory, which effectively prevents memory disclosure attacks. The input RSA parameters are encrypted by an AES key. The AES key is stored in the privileged debug registers. For performance, we use AVX-512F instruction set to accelerate the RSA calculation. We adopt vector instructions to implement 1024-bit Montgomery multiplication and make use of redundant representation to solve the carry propagation problem. Experiments on Intel Xeon Silver 4208 CPU shows that VIRSA achieves a performance factor of 0.8 compared to the OpenSSL RSA implementation, which outperforms existing approaches such as PRIME. Furthermore, we make use of the windowing method to improve the RSA performance. The precomputed table is encrypted by the AES key to ensure security. The performance of VIRSA using the fixed windowing method slightly exceeds OpenSSL, achieving a performance factor of 1.02.