Due to the rapid growth of mold technology, medical equipment, or devices, especially with complex curved SKD11 steel material, the surface quality of workpieces requires a smoother finish during operation. Thus, this article facilitates the development of a non-resonant vibration-assisted polishing (NVAP) system, also known as fishing technology, for SKD11 steel curved surfaces to enhance surface quality, thereby establishing a significant theoretical foundation for vibration-assisted machining. This study aims to examine the impact of material removal depth according to Hertz contact material removal rate theory and the Preston equation. The X and Y two-way vibration apparatus enables the workpiece to oscillate in a spiral or circular trajectory due to variations in amplitude, frequency, and phase shift. Additionally, the impact of complex surface toolpath strategies, combined with workpiece vibration trajectories, on the material removal rate is analyzed and evaluated. This facilitates the anticipation of ideal solutions for selecting the necessary vibration trajectory, contact force, and tool path. Additionally, the experimental validation of the material removal rate and surface polishing of the workpieces is presented. Finally, the real NVAP system is built, and experimental investigations are carried out for six cases (cases 1–6). After polishing, the surface roughness and microtopography of the many workpiece samples are tested and compared with traditional polishing technology. Case #1, the experimental results showed that when the amplitude was 10 μm and the frequency was 100 Hz, the surface roughness Ra of 0.29 μm is achieved, and it improved by 29.27% compared to conventional polishing. Case #2, when the frequency is increased to 300 Hz, the surface roughness Ra value of 0.23 μm is enhanced by 43.9%. Case #3, when the amplitude increased to 20 μm and the frequency is 100 Hz, the improvement ratio is 34.14%, and the achieved roughness Ra of 0.27 μm. Case 5 demonstrates superior surface quality, achieving a Ra of 0.22 μm, which is attributable to the reduced tool feed while maintaining the same input amplitude and frequency. This study demonstrated that the material removal mechanism of NVAP technology for SKD11 steel enhances surface quality. Theoretical and experimental investigations have produced the benefits of precision machining technology.