Knowledge Reverse Distillation Based Confidence Calibration for Deep Neural Networks

Deep neural networks, as a key technical breakthrough in machine learning field, have been widely used in various practical scenarios. However, the existing deep neural networks often generate the predictions with high confidence risks, which are prone to mislead practitioners and limit the deploying of deep neural networks in high-risk decision-making fields. In order to solve this issue, this paper proposes a confidence calibration method for deep neural networks by designing one novel knowledge reverse distillation strategy. Traditional knowledge distillation strategy takes the accuracy as the knowledge, and transfers it from the teacher network (usually one complex deep network) to the student network (usually one simple network). Different from this traditional distillation strategy, the proposed knowledge reverse distillation strategy regards the confidence as the knowledge, and constructs one reverse knowledge transfer pathway by applying the confidence knowledge in the simple network to calibrate the complex deep network. Experimental results on three benchmark image datasets show that the knowledge reverse distillation strategy can effectively improve the calibration capability of complex networks so that the complex deep neural network captures the high confidence along with the high prediction accuracy.