Skip to main content
SpringerLink
Log in

Deep reinforcement learning for imbalanced classification

  • Published:
Applied Intelligence Aims and scope Submit manuscript

Abstract

Data in real-world application often exhibit skewed class distribution which poses an intense challenge for machine learning. Conventional classification algorithms are not effective in case of imbalanced data distribution, and may fail when the data distribution is highly imbalanced. To address this issue, we propose a general imbalanced classification model based on deep reinforcement learning, in which we formulate the classification problem as a sequential decision-making process and solve it by a deep Q-learning network. In our model, the agent performs a classification action on one sample in each time step, and the environment evaluates the classification action and returns a reward to the agent. The reward from the minority class sample is larger, so the agent is more sensitive to the minority class. The agent finally finds an optimal classification policy in imbalanced data under the guidance of the specific reward function and beneficial simulated environment. Experiments have shown that our proposed model outperforms other imbalanced classification algorithms, and identifies more minority samples with better classification performance.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4

Similar content being viewed by others

Notes

  1. Retrievethedatasetsfromhttps://keras.io/datasets

References

  1. Japkowicz N, Stephen S (2002) The class imbalance problem: a systematic study. Intell Data Anal 6 (5):429–449

    Article  Google Scholar 

  2. Weiss GM (2004) Mining with rarity: a unifying framework. ACM Sigkdd Explorations Newsletter 6(1):7–19

    Article  Google Scholar 

  3. He H, Garcia EA (2008) Learning from imbalanced data. IEEE Trans Knowl Data Eng 9:1263–1284

    Google Scholar 

  4. Haixiang G, Yijing L, Shang J, Mingyun G, Yuanyue H, Bing G (2017) Learning from class-imbalanced data: Review of methods and applications. Expert Syst Appl 73:220–239

    Article  Google Scholar 

  5. Mnih V, Kavukcuoglu K, Silver D, Graves A, Antonoglou I, Wierstra D, Riedmiller M (2013) Playing atari with deep reinforcement learning. arXiv:1312.5602

  6. Gu S, Holly E, Lillicrap T, Levine S (2017) Deep reinforcement learning for robotic manipulation with asynchronous off-policy updates. In: 2017 IEEE International Conference on Robotics and Automation (ICRA), IEEE, pp 3389–3396

  7. Zhao X, Zhang L, Ding Z, Yin D, Zhao Y, Tang J (2017) Deep reinforcement learning for list-wise recommendations. arXiv:1801.00209

  8. Feng J, Huang M, Zhao L, Yang Y, Zhu X (2018) Reinforcement learning for relation classification from noisy data. In: Proceedings of AAAI

  9. Martinez C, Perrin G, Ramasso E, Rombaut M (2018) A deep reinforcement learning approach for early classification of time series. In: EUSIPCO, 2018

  10. Drummond C, Holte RC, et al. (2003) C4. 5, class imbalance, and cost sensitivity: why under-sampling beats over-sampling. In: Workshop on Learning from Imbalanced datasets II, vol 11, Citeseer, pp 1–8

  11. Han H, Wang W-Y, Mao B-H (2005) Borderline-smote: a new over-sampling method in imbalanced data sets learning. In: International Conference on Intelligent Computing, Springer, pp 878–887

  12. Mani I (2003) I Zhang, knn approach to unbalanced data distributions: a case study involving information extraction. In: Proceedings of Workshop on Learning from Imbalanced Datasets, vol 126

  13. Batista GE, Prati RC, Monard MC (2004) A study of the behavior of several methods for balancing machine learning training data. ACM SIGKDD Explorations Newsletter 6(1):20–29

    Article  Google Scholar 

  14. Akkasi A, Varoğlu E, Dimililer N (2017) Balanced undersampling: a novel sentence-based undersampling method to improve recognition of named entities in chemical and biomedical text. Appl Intell, pp 1–14

  15. Gupta D, Richhariya B (2018) Entropy based fuzzy least squares twin support vector machine for class imbalance learning. Appl Intell 48(11):4212–4231

    Article  Google Scholar 

  16. Wu G, Chang EY (2005) Kba: Kernel boundary alignment considering imbalanced data distribution. IEEE Trans Knowl Data Eng 17(6):786–795

    Article  Google Scholar 

  17. Tang Y, Zhang Y-Q, Chawla NV, Krasser S (2009) Svms modeling for highly imbalanced classification. IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics) 39(1):281–288

    Article  Google Scholar 

  18. Su C, Cao J (2018) Improving lazy decision tree for imbalanced classification by using skew-insensitive criteria. Applied Intelligence

  19. Zadrozny B, Elkan C (2001) Learning and making decisions when costs and probabilities are both unknown. In: Proceedings of the seventh ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, ACM, pp 204–213

  20. Zadrozny B, Langford J, Abe N (2003) Cost-sensitive learning by cost-proportionate example weighting. In: ICDM, 2003, Third IEEE International Conference on Data Mining, 2003, IEEE, pp 435–442

  21. Zhou Z-H, Liu X-Y (2006) Training cost-sensitive neural networks with methods addressing the class imbalance problem. IEEE Trans Knowl Data Eng 18(1):63–77

    Article  MathSciNet  Google Scholar 

  22. Krawczyk B, Woźniak M (2015) Cost-sensitive neural network with roc-based moving threshold for imbalanced classification. In: International Conference on Intelligent Data Engineering and Automated Learning, Springer, pp 45–52

  23. Chen J, Tsai C-A, Moon H, Ahn H, Young J, Chen C-H (2006) Decision threshold adjustment in class prediction. SAR QSAR Environ Res 17(3):337–352

    Article  Google Scholar 

  24. Yu H, Sun C, Yang X, Yang W, Shen J, Qi Y (2016) Odoc-elm: Optimal decision outputs compensation-based extreme learning machine for classifying imbalanced data. Knowl-Based Syst 92:55–70

    Article  Google Scholar 

  25. Ting KM (2000) A comparative study of cost-sensitive boosting algorithms. In: Proceedings of the 17th International Conference on Machine Learning Citeseer

  26. Janisch J, Pevnỳ T, Lisỳ V (2017) Classification with costly features using deep reinforcement learning. arXiv:1711.07364

  27. Wang S, Liu W, Wu J, Cao L, Meng Q, Kennedy PJ (2016) Training deep neural networks on imbalanced data sets, in Neural Networks (IJCNN). In: 2016 International Joint Conference on. IEEE, pp 4368–4374

  28. Huang C, Li Y, Change Loy C, Tang X (2016) Learning deep representation for imbalanced classification. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp 5375–5384

  29. Yan Y, Chen M, Shyu M-L, Chen S-C (2015) Deep learning for imbalanced multimedia data classification. In: Multimedia (ISM). In: 2015 IEEE International Symposium on. IEEE, pp 483– 488

  30. Khan SH, Hayat M, Bennamoun M, Sohel FA, Togneri R (2018) Cost-sensitive learning of deep feature representations from imbalanced data. IEEE Trans Neural Netw Learn Syst 29(8):3573–3587

    Article  Google Scholar 

  31. Dong Q, Gong S, Zhu X (2018) Imbalanced deep learning by minority class incremental rectification. IEEE Transactions on Pattern Analysis and Machine Intelligence

  32. Wiering MA, van Hasselt H, Pietersma A-D, Schomaker L (2011) Reinforcement learning algorithms for solving classification problems. In: 2011 IEEE Symposium on Adaptive Dynamic Programming And Reinforcement Learning (ADPRL), IEEE, pp 91–96

  33. Zhang T, Huang M, Zhao L (2018) Learning structured representation for text classification via reinforcement learning. AAAI

  34. Liu D, Jiang T (2018) Deep reinforcement learning for surgical gesture segmentation and classification. arXiv:1806.08089

  35. Zhao D, Chen Y, Lv L (2017) Deep reinforcement learning with visual attention for vehicle classification. IEEE Trans Cogn Develop Syst 9(4):356–367

    Article  Google Scholar 

  36. Abdi L, Hashemi S (2014) An ensemble pruning approach based on reinforcement learning in presence of multi-class imbalanced data. In: Proceedings of the Third International Conference on Soft Computing for Problem Solving, Springer, pp 589–600

  37. Dixit AK, Sherrerd JJ, et al. (1990) Optimization in economic theory. Oxford University Press on Demand

  38. Lin T-Y, Goyal P, Girshick R, He K, Dollár P (2017) Focal loss for dense object detection. In: Proceedings of the IEEE International Conference on Computer Vision, pp 2980–2988

  39. Gu Q, Zhu L, Cai Z (2009) Evaluation measures of the classification performance of imbalanced data sets. In: International Symposium on Intelligence Computation and Applications, Springer, pp 461–471

  40. Bengio Y (2012) Practical recommendations for gradient-based training of deep architectures. In: Neural Networks: Tricks of the Trade, Springer, pp 437–478

  41. Demšar J (2006) Statistical comparisons of classifiers over multiple data sets. J Mach Learn Res 7(Jan):1–30

    MathSciNet  MATH  Google Scholar 

  42. Benavoli A, Corani G, Mangili F (2016) Should we really use post-hoc tests based on mean-ranks? J Mach Learn Res 17(1):152–161

    MathSciNet  MATH  Google Scholar 

  43. Wilcoxon F (1992) Individual comparisons by ranking methods. In: Breakthroughs in Statistics, Springer, pp 196–202

  44. Holm S (1979) A simple sequentially rejective multiple test procedure. Scandinavian journal of statistics, pp 65–70

Download references

Author information

Authors and Affiliations

  1. School of Computer Science and Engineering, South China University of Technology, Guangzhou, 510006, China

    Enlu Lin, Qiong Chen & Xiaoming Qi

Authors
  1. Enlu Lin
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Qiong Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Xiaoming Qi
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Qiong Chen.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Lin, E., Chen, Q. & Qi, X. Deep reinforcement learning for imbalanced classification. Appl Intell 50, 2488–2502 (2020). https://doi.org/10.1007/s10489-020-01637-z

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10489-020-01637-z

Keywords

Search

Navigation