Skip to main content
SpringerLink
Log in

A repetitive feature selection method based on improved ReliefF for missing data

  • Published:
Applied Intelligence Aims and scope Submit manuscript

Abstract

ReliefF is a representative and efficient algorithm amongst many feature selection methods, however, in the face of missing data, ReliefF and its variants might be invalid. To address this problem, a novel feature selection method, namely repetitive feature selection based on improved ReliefF, is proposed to obtain the optimal feature subset and make an accurate imputation delicately for missing data. The main idea is three-fold: 1) the data distribution determined by the distance of class center is introduced into the feature weights to construct a proper objective function, which greatly helps select significant and highly relevant features while removing redundant/noise ones; 2) the improved ReliefF is applied both before and after imputation to make full use of known data, and a non-negativity matrix factorization (NMF) model is established to make a sound imputation for missing data; and 3) during the NMF model learning, the mini-batch gradient descent (MBGD) technique is employed to accelerate the convergence and avoid trapping in local optima. Experiments on seven public data sets are utilized to show the effectiveness of the proposed feature selection method.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2

Similar content being viewed by others

References

  1. Breiman L (2001) Random forests. Mach Learn 45(1):5–32

    Article  MATH  Google Scholar 

  2. Cortes C, Vapnik V (1995) Support-vector networks. Mach Learn 20(3):273–297

    Article  MATH  Google Scholar 

  3. Chebel-Morello B, Malinowski S, Senoussi H (2016) Feature selection for fault detection systems: Application to the Tennessee Eastman process. Appl Intell 44:111–122

    Article  Google Scholar 

  4. Cai H, Ruan P, Ng M, Akutsu T (2014) Feature weight estimation for gene selection: A local hyperlinear learning approach. BMC Bioinform, vol 15

  5. Cekik R, Uysal AK (2020) A novel filter feature selection method using rough set for short text data. Expert Syst Appl 160:113691

    Article  Google Scholar 

  6. Doquire G, Verleysen M (2012) Feature selection with missing data using mutual information estimators. Neurocomputing 90:3–11

    Article  Google Scholar 

  7. Guyon IM, Elisseeff A (2003) An introduction to variable and feature selection. J Mach Learn Res 3:1157–1182

    MATH  Google Scholar 

  8. Heikki M (2002) Local and global methods in data mining: Basic techniques and open problems. Springer, Berlin, pp 57–68

    MATH  Google Scholar 

  9. Hong JH, Cho SB (2006) Efficient huge-scale feature selection with speciated genetic algorithm. Pattern Recognit Lett 27(2):143–150

    Article  Google Scholar 

  10. Hunt R, Neshatian K, Zhang M (2012) A genetic programming approach to hyper-heuristic feature selection. In: Asia-Pacific conference on simulated evolution and learning, pp 320–330

  11. Haq AU, Zeb A, Lei Z, Zhang D (2021) Forecasting daily stock trend using multi-filter feature selection and deep learning. Expert Syst Appl 168(3):114444

    Article  Google Scholar 

  12. Huang Z, Yang C, Zhou X, Huang T (2018) A hybrid feature selection method based on binary state transition algorithm and ReliefF. IEEE J Biomed Health Inform 23(5):1888–1898

    Article  Google Scholar 

  13. Lichman M (2016) UCI machine learning repository, [Online]. Available: http://archive.ics.uci.edu/ml

  14. Kaiser J (2014) Dealing with missing values in data. J Syst Integr 5(1):42–51

    Article  Google Scholar 

  15. Kira K, Rendell LA (1992) The feature selection problem: traditional methods and a new algorithm. Aaai 2:129–134

    Google Scholar 

  16. Kononenko I (1994) Estimating attributes: Analysis and extensions of RELIEF. In: European conference on machine learning on machine learning. Springer, Berlin

  17. Lall S, Sinha D, Ghosh A, Sengupta D, Bandyopadhyay S (2020) Stable feature selection using copula based mutual information. Pattern Recognit 112(1):107697

    Google Scholar 

  18. Liu SG, Zhang J, Xiang Y, Zhou WL (2017) Fuzzy-based information decomposition for incomplete and imbalanced data learning. IEEE Trans Fuzzy Syst 25(6):1476–1490

    Article  Google Scholar 

  19. Luo X, Zhou M, Xia Y, et al. (2014) An efficient non-negative matrix-factorization-based approach to collaborative filtering for recommender systems. IEEE Trans Ind Inform 10(2):1273–1284

    Article  Google Scholar 

  20. Marill T, Green D (1963) On the effectiveness of receptors in recognition systems. IEEE Trans Inf Theory 9(1):11–17

    Article  Google Scholar 

  21. Mu Y, Liu W, Liu X, Fan W (2017) Stochastic gradient made stable: a manifold propagation approach for large-scale optimization. IEEE Trans Knowl Data Eng 29(2):458–471

    Article  Google Scholar 

  22. Pudil P, Novovičová J, Kittler J (1994) Floating search methods in feature selection. Pattern Recognit Lett 15(11):1119–1125

    Article  Google Scholar 

  23. Peng H, Long F, Ding C (2005) Feature selection based on mutual information criteria of max-dependency, max-relevance, and min-redundancy. IEEE Trans Pattern Anal Mach Intell 27(8):1226–1238

    Article  Google Scholar 

  24. Peterson LE (2009) K-nearest neighbor. Scholarpedia 4(2):1883

    Article  Google Scholar 

  25. Ratsch G (2001) Soft margins for AdaBoost. Mach Learn 42(3):287–320

    Article  MATH  Google Scholar 

  26. Robnik-Šikonja M, Kononenko I (2003) Theoretical and empirical analysis of ReliefF and RRelieff. Mach Learn 53(1-2):23–69

    Article  MATH  Google Scholar 

  27. Shang W, Huang H, Zhu H, Lin Y, Qu Y, Wang Z (2007) A novel feature selection algorithm for text categorization. Expert Syst Appl 33(1):1–5

    Article  Google Scholar 

  28. Song Y, Si W, Dai F, Yang G (2020) Weighted relief with thresholds of feature selection for imbalanced data classification. Concurr Comput 32(14):e5691

    Article  Google Scholar 

  29. Solorio-Fernández S, Martínez-Trinidad J F, Carrasco-Ochoa JA (2020) A supervised filter feature selection method for mixed data based on spectral feature selection and information-theory redundancy analysis. Pattern Recognit Lett 138:321–328

    Article  Google Scholar 

  30. Sun Y (2007) Iterative RELIEF for feature weighting: Algorithms, theories, and applications. IEEE Trans Pattern Anal Mach Intell 29(6):1035–1051

    Article  Google Scholar 

  31. Song Y, Li M, Luo X, Yang G, Wang C (2019) Improved symmetric and nonnegative matrix factorization models for undirected, sparse and large-scaled networks: a triple factorization-based approach. IEEE Trans Ind Inform 16(5):3006–3017

    Article  Google Scholar 

  32. Tang J, Alelyani S, Liu H (2014) Feature selection for classification: A review. Documentación Administrativa, pp 37– 64

  33. Tang B, Zhang L (2020) Local preserving logistic I-Relief for semi-supervised feature selection. Neurocomputing 399(1):48–64

    Article  Google Scholar 

  34. Tran CT, Zhang M, Andreae P, Bing X, Giovanni S, Paolo B (2016) A wrapper feature selection approach to classification with missing data. In: European conference on the applications of evolutionary computation, vol 9597. Springer, Cham, pp 658–700

  35. Thevenaz P, Unser M (2000) Optimization of mutual information for multiresolution image registration. IEEE Trans Image Process 9(12):2081–1099

    MATH  Google Scholar 

  36. Whitney AW (1971) A direct method of nonparametric measurement selection. IEEE Trans Comput 20(9):1100–1103

    Article  MATH  Google Scholar 

  37. Wei M et al (2019) Bas-relief modeling from normal layers. IEEE Trans Vis Comput Graph 25 (4):1651–1665

    Article  Google Scholar 

  38. Xue B, Zhang M, Browne WN, Yao X (2015) A survey on evolutionary computation approaches to feature selection. IEEE Trans Evol Comput 20(4):606–626

    Article  Google Scholar 

  39. Yu L, Liu H (2004) Efficient feature selection via analysis of relevance and redundancy. J Mach Learn Res 5:1205–1224

    MathSciNet  MATH  Google Scholar 

  40. Zahin SA, Ahmed CF, Alam T (2018) An effective method for classification with missing values. Appl Intell 48:3209–3230

    Article  Google Scholar 

  41. Zhang XX, Li TS (2012) Multivariate regression analytical method based on heuristic constructed variable under condition of incomplete data. J Comput Appl 32(8):2202–2274

    Google Scholar 

Download references

Acknowledgments

This work was supported in part by the National Natural Science Foundation of China under Grants 62073223, the Natural Science Foundation of Shanghai under Grant 22ZR1443400, and the Open Project of Key Laboratory of Aerospace Flight Dynamics and National Defense Science and Technology under Grants 6142210200304.

Author information

Authors and Affiliations

  1. School of Business, University of Shanghai for Science and Technology, Shanghai, 200093, China

    Haiyan Fan

  2. School of Science, University of Shanghai for Science and Technology, Shanghai, 200093, China

    Luyu Xue

  3. Department of Control Science and Engineering, University of Shanghai for Science and Technology, Shanghai, 200093, China

    Yan Song & Ming Li

Authors
  1. Haiyan Fan
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Luyu Xue
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Yan Song
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Ming Li
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Yan Song.

Ethics declarations

Conflict of Interests

We declare that no conflict of interest exists in the submission of this manuscript, and the manuscript is approved by all authors for publication.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Fan, H., Xue, L., Song, Y. et al. A repetitive feature selection method based on improved ReliefF for missing data. Appl Intell 52, 16265–16280 (2022). https://doi.org/10.1007/s10489-022-03327-4

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10489-022-03327-4

Keywords

Search

Navigation