Inference engines are specialized software systems designed to execute pre-trained Machine Learning models. ONNX Runtime (ORT) emerges as a leading open-source inference engine for the Open Neural Network Exchange (ONNX) format, allowing models to be deployed regardless of the framework in which they were trained.
While ORT provides a flexible architecture for deploying models across diverse hardware, it currently lacks architecture-specific optimizations for RISC-V. Consequently, computationally intensive tasks such as the convolution operation—which accounts for the majority of inference time in Convolutional Neural Networks (CNNs)—suffer from hardware underutilization by relying on standard scalar instructions. In this paper, we address this gap by proposing an optimized convolution implementation leveraging the RISC-V Vector Extension (RVV) and integrating it as a custom Execution Provider in ORT. We evaluate our solution on a Banana Pi BPI-F3 board across six standard reference CNN models. Experimental results show that our RVV-accelerated implementation achieves speedups of up to 3x compared to the official scalar ORT release, significantly improving CNN inference performance on RISC-V platforms.