Iteratively Modeling Based Cleansing Interactively Samples of Big Data

Taking advantage of big data means analyzing it and building prediction model on it. However, the data obtained in reality often contains dirty data due to various factors. One method of using big data is to clean the whole data at first, and then train predictive model on cleaned data, but existing cleaning approaches often need lots of completely clean data as guide to fix errors, that is impractical to obtain many clean data. Another method is to train predictive model on raw data directly, which causes the model is not accurate. Therefore, we explore the iterative updating model process and propose an updating algorithm combining data cleaning and conjugate gradient. In this paper, we incrementally update initial model trained on raw data towards the optimum by cleaning samples instead of whole data at each iteration. And the updating direction is established according to gradient of data. After multiple iterations, we can obtain the optimal model that still works well without cleaning data when new data comes in. We also present cluster descent sampling algorithm to accelerate model convergence. Our evaluation on real datasets shows that the approach significantly improves model accuracy compared with training model directly on raw data.