Incremental Input Variable Selection by Block Addition and Block Deletion

In selecting input variables by block addition and block deletion (BABD), multiple input variables are added and then deleted, keeping the cross-validation error below that using all the input variables. The major problem of this method is that selection time becomes large as the number of input variables increases. To alleviate this problem, in this paper, we propose incremental block addition and block deletion of input variables. In this method, for an initial subset of input variables we select input variables by BABD. Then in the incremental step, we add some input variables that are not added before to the current selected input variables and iterate BABD. To guarantee that the cross-validation error decreases monotonically by incremental BABD, we undo incremental BABD if the obtained cross-validation error rate is worse than that at the previous incremental step. We evaluate incremental BABD using some benchmark data sets and show that by incremental BABD, input variable selection is speeded up with the approximation error comparable to that by batch BABD.