ViBe algorithm is a powerful moving object detection algorithm. It has many advantages, such as simple method, easy implementation and high computational efficiency, but there are also many shortcomings, such as ghost problem, susceptibility to noise and illumination changes and inadaptability to dynamic scenes. Aiming at the above shortcomings, an adaptive ViBe algorithm based on dispersion coefficients and spatial consistency factor is proposed. Firstly, the mode of multi-frame images is used to replace single image to realize initialization, which reduces the interference of ghost to background model; Secondly, dispersion coefficient is used to establish adaptive dynamic threshold to improve the adaptability of the algorithm to dynamic background; Finally, the spatial consistency factor with spatial information is used to establish adaptive update factor, which reduces the error rate and enhances the robustness of the algorithm. The experimental results show that our improved ViBe algorithm can effectively eliminate ghosts, better adapt to noise, illumination and dynamic background, have more complete detection results and higher detection accuracy than the traditional and others’ improved ViBe algorithms and Gaussian mixture model.