An improved method to construct basic probability assignment based on the confusion matrix for classification problem

doi:10.1016/j.ins.2016.01.033

Information Sciences

Volumes 340–341, 1 May 2016, Pages 250-261

https://doi.org/10.1016/j.ins.2016.01.033 Get rights and content

Abstract

The determination of basic probability assignment (BPA) is a crucial issue in the application of Dempster–Shafer evidence theory. Classification is a process of determining the class label that a sample belongs to. In classification problem, the construction of BPA based on the confusion matrix has been studied. However, the existing methods do not make full use of the available information provided by the confusion matrix. In this paper, an improved method to construct the BPA is proposed based on the confusion matrix. The proposed method takes into account both the precision rate and the recall rate of each class. An illustrative case regarding the prediction of transmembrane protein topology is given to demonstrate the effectiveness of the proposed method.

Introduction

Dempster–Shafer evidence theory [9], [45], also called Dempster–Shafer theory, has been widely applied in many fields, such as information fusion [50], classification [5], [37], [38], and others [4], [10], [11], [12], [13], [16], [20], [25], [42], [49], [57], [68]. Tabassian et al. [51], [52] used Dempster–Shafer theory to handle data with imperfect labels in ensemble learning, and addressed the situation that the class memberships of the training data are subject to ambiguity. Deng [19] proposed a generalized evidence theory (GET) to address conflict management in an open world environment. Thanks to its flexibility, Dempster–Shafer theory has been combined with other theories like fuzzy set theory [8], [31], [69] and genetic algorithm [22], and many useful tools have been developed to handle various types of uncertainty, which further extends the application of the Dempster–Shafer theory. For instance, in [32], Kang et al. proposed an uncertain-graph structure, called evidential cognitive map (ECM), to represent causal reasoning by combining the cognitive maps and Dempster–Shafer theory. Recently, in the fields of evolutionary game theory [6], [7], [15], [58], [59], [60], [61], [62], [63], [64] and game theory [53], Dempster–Shafer evidence theory has also attracted some interests [17], [18], [35].

The determination of basic probability assignment (BPA) is one of the most important problems in evidential systems. The construction of BPA based on the confusion matrix is a practical and effective method [1], [2], [24], [40], [43], [54], [65]. In a previous related study, Xu et al. [65] presented an elegant method for the construction of BPA based on recognition rate, substitution rate, and rejection rate of the confusion matrix. However, Xu et al.’s method does not consider the difference of the classifier’s recognition ability for different classes. To overcome the shortcoming, Parikh et al. [40] proposed a modified method, which is more effective and has been successfully used in condition monitoring. The improvement proposed by Parhikh et al. is on the basis of the prior knowledge provided by the confusion matrix. Specifically, that method utilized the precision rate of each actual class according to the confusion matrix. However, the prior knowledge contained in the confusion matrix is not only the precision rate, but also the recall rate of each class which is another important aspect to reflect the classifier’s recognition ability for each class.

Based on this idea, an improved BPA construction method is proposed based on the confusion matrix in this paper. Section 2 introduces some basic concepts and related previous work. Section 3 presents the proposed method. Section 4 gives an illustrative case to demonstrate the effectiveness of the proposed method. Section 5 concludes the paper.

Section snippets

Basic concepts

The Dempster–Shafer evidence theory [9], [45], first proposed by Dempster and further developed by Shafer, is widely used to handle uncertain information. In this theory, basic probability assignment (BPA) is used to represent the uncertain information, and Dempster’s rule of combination is used to combine multiple BPAs.

In Dempster–Shafer theory, a problem domain denoted by a finite nonempty set Ω of mutually exclusive and exhaustive hypotheses is called the frame of discernment. Let $2^{Ω}$ denote

Proposed method

Based on the abundant prior knowledge contained in the confusion matrix, an improved method to construct the BPA is proposed in this section. Generally, a confusion matrix contains the information about actual and predicted classifications given by a classification system. In order to evaluate the performance of such systems, the data in the matrix is usually used. Based on the data coming from the confusion matrix, some indices, for instances accuracy, sensitivity (also called recall),

Case study

In this section, the prediction of transmembrane protein topology is used to illustrate the our proposed method, partial results are from our previous study [14]. The topology of transmembrane proteins, i.e. the number and position of the transmembrane helixes and the in/out location of the N and C terminals of the protein sequence, is an important issue in the study of transmembrane proteins [21], [36]. For a protein sequence, if both the transmembrane helixes and location of the N and C

Conclusion

As frequently stressed in previous studies, the determination of BPA is one of the most key problem in the application of Dempster–Shafer evidence theory. And in an evidential multiple classifiers system, the construction of BPA has a great influence on the classification performance. In this paper, an improved BPA construction method is proposed based on the confusion matrix for classification problem. The proposed method makes full use of the available information contained in the confusion

Acknowledgments

The authors thank the anonymous reviewers for their valuable comments and suggestions to improve this paper. The work is partially supported by National High Technology Research and Development Program of China (863 Program) (Grant No. 2013AA013801), National Natural Science Foundation of China (Grant Nos. 61174022, 61573290), the open funding project of State Key Laboratory of Virtual Reality Technology and Systems, Beihang University (Grant No. BUAA-VR-14KF-02), Fundamental Research Funds for

References (69)

M.J. Beynon et al.
Evidence-based modelling of strategic fit: an introduction to RCaRBS
Eur. J. Oper. Res.
(2010)
Y. Bi et al.
The combination of multiple classifiers using an evidential reasoning approach
Artif. Intell.
(2008)
S. Boccaletti et al.
The structure and dynamics of multilayer networks
Phys. Rep.
(2014)
A. Burkov et al.
Computing equilibria in discounted dynamic games
Appl. Math. Comput.
(2015)
C.-C. Chou et al.
An evaluation of airline service quality using the fuzzy weighted SERVQUAL method
Appl. Soft Comput.
(2011)
X. Deng et al.
Parameter estimation based on interval-valued belief structures
Eur. J. Oper. Res.
(2015)
X. Deng et al.
Environmental impact assessment based on D numbers
Expert Syst. Appl.
(2014)
X. Deng et al.
Supplier selection using AHP methodology extended by D numbers
Expert Syst. Appl.
(2014)
X. Deng et al.
D-CFPR: D numbers extended consistent fuzzy preference relations
Knowl. Based Syst.
(2015)
X. Deng et al.
A belief-based evolutionarily stable strategy
J. Theor. Biol.
(2014)

X. Deng et al.

An evidential game theory framework in multi-criteria decision making process

Appl. Math. Comput.

(2014)

Y. Deng et al.

Scoring hidden Markov models to discriminate β-barrel membrane proteins

Comput. Biol. Chem.

(2004)

L. Diaz-Mas et al.

Shape from silhouette using Dempster-Shafer theory

Patt. Recognit.

(2010)

X.F. Fan et al.

Fault diagnosis of machines based on D-S evidence theory. Part 2: application of the improved D-S evidence theory in gearbox fault diagnosis

Pattern Recog. Lett.

(2006)

M.C. Florea et al.

Robust combination rules for evidence theory

Inf. Fusion

(2009)

B. Kang et al.

Evidential cognitive maps

Knowl.-Based Syst.

(2012)

E. Lefèvre et al.

How to preserve the conflict as an alarm in the combination of belief functions?

Decis. Support Syst.

(2013)

Y. Li et al.

Comprehensive consideration of strategy updating promotes cooperation in the prisoner’s dilemma game

Physica A: Stat. Mech. Appl.

(2014)

Q. Liu et al.

A HMM-based method to predict the transmembrane regions of β-barrel membrane proteins

Comput. Biol. Chem.

(2003)

Z. Liu et al.

A new belief-based K-nearest neighbor classification method

Pattern Recog.

(2013)

C.K. Murphy

Combining belief functions when evidence conflicts

Decis. Supp. Syst.

(2000)

C.R. Parikh et al.

Application of Dempster-Shafer theory in condition monitoring applications: a case study

Pattern Recog. Lett.

(2001)

B.S. Reddy et al.

Concept-based evidential reasoning for multimodal fusion in human computer interaction

Appl. Soft Comput.

(2010)

G. Rogova

Combining the results of several neural network classifiers

Neural Netwo.

(1994)

X. Su et al.

Dependence assessment in human reliability analysis using evidence theory and AHP

Risk Anal.

(2015)

M. Tabassian et al.

Combining complementary information sources in the Dempster-Shafer framework for solving classification problems with imperfect labels

Knowl. Based Syst.

(2012)

Z. Tao et al.

Group decision making with fuzzy linguistic preference relations via cooperative games method

Comput. Indust. Eng.

(2015)

C. Thiel et al.

Using Dempster-Shafer theory in MCF systems to reject samples

Lect. Notes Comput. Sci.

(2005)

Z. Wang et al.

Universal scaling for the dilemma strength in evolutionary games

Phys. Life Rev.

(2015)

Z. Wang et al.

Insight into the so-called spatial reciprocity

Phys. Rev. E

(2013)

Z. Wang et al.

Evolutionary games on multilayer networks: a colloquium

Eur. Phys. J. B

(2015)

R.R. Yager

Combining various types of belief structures

Inf. Sci.

(2015)

J.-B. Yang et al.

Belief rule-based methodology for mapping consumer preferences and setting product targets

Expert Syst. Appl.

(2012)

X. Zhang et al.

IFSJSP: a novel methodology for the job-shop scheduling problem based on intuitionistic fuzzy sets

Int. J. Prod. Res.

(2013)

Cited by (375)

Reconstruction of lithofacies using a supervised Self-Organizing Map: Application in pseudo-wells based on a synthetic geologic cross-section
2024, Artificial Intelligence in Geosciences
Recently, machine learning (ML) has been considered a powerful technological element of different society areas. To transform the computer into a decision maker, several sophisticated methods and algorithms are constantly created and analyzed. In geophysics, both supervised and unsupervised ML methods have dramatically contributed to the development of seismic and well-log data interpretation. In well-logging, ML algorithms are well-suited for lithologic reconstruction problems, once there is no analytical expressions for computing well-log data produced by a particular rock unit. Additionally, supervised ML methods are strongly dependent on a accurate-labeled training data-set, which is not a simple task to achieve, due to data absences or corruption. Once an adequate supervision is performed, the classification outputs tend to be more accurate than unsupervised methods. This work presents a supervised version of a Self-Organizing Map, named as SSOM, to solve a lithologic reconstruction problem from well-log data. Firstly, we go for a more controlled problem and simulate well-log data directly from an interpreted geologic cross-section. We then define two specific training data-sets composed by density (RHOB), sonic (DT), spontaneous potential (SP) and gamma-ray (GR) logs, all simulated through a Gaussian distribution function per lithology. Once the training data-set is created, we simulate a particular pseudo-well, referred to as classification well, for defining controlled tests. First one comprises a training data-set with no labeled log data of the simulated fault zone. In the second test, we intentionally improve the training data-set with the fault. To bespeak the obtained results for each test, we analyze confusion matrices, logplots, accuracy and precision. Apart from very thin layer misclassifications, the SSOM provides reasonable lithologic reconstructions, especially when the improved training data-set is considered for supervision. The set of numerical experiments shows that our SSOM is extremely well-suited for a supervised lithologic reconstruction, especially to recover lithotypes that are weakly-sampled in the training log-data. On the other hand, some misclassifications are also observed when the cortex could not group the slightly different lithologies.
A novel uncertainty modeling method in complex evidence theory for decision making
2024, Engineering Applications of Artificial Intelligence
The Dempster-Shafer (D-S) evidence theory, renowned for its efficacy in processing uncertain information, has garnered substantial attention in recent decades. A generalized D-S evidence theory called complex evidence theory (CET), is tailored to model and handle uncertainty on the complex plane. Within this context, generating complex basic belief assignments (CBBA) in CET stands as a critical challenge to improve the decision level for CET-based information fusion. This paper proposes an innovative method for generating CBBA to model uncertainty on the complex plane. Initially, a complex attribute membership function is introduced to construct the model. Subsequently, the CBBA is generated by inputting data into this model. Based on this CBBA generation method, a new decision-making algorithm is presented. To validate the effectiveness of the proposed CBBA generation method and decision-making algorithm, they are applied in a classification application. Comparative analysis demonstrates the superior performance of the proposed algorithm in CBBA generation compared to other methods.
An effective targeted label adversarial attack on graph neural networks by strategically allocating the attack budget
2024, Knowledge-Based Systems
Although graph neural networks (GNNs) have been extensively employed in various research fields, they are susceptible to adversarial attacks, as revealed in previous studies. Targeted label attacks, in which the victim node is anticipated to fall into a desired category, are more consistent with practical circumstances than untargeted label attacks. However, there has been relatively little research on them, and existing attackers require access to the target model and do not consider the budget allocation; the budget refers to the maximum number of perturbations permitted. Furthermore, the performance of untargeted label attacks is weakened when they are converted into targeted label attacks. From the perspective of budget allocation, we devise an effective targeted label attack (ETLA). The prior knowledge includes training labels, node features, and the graph structure. The main concept is based on the strategical allocation of a given budget from the search space and optimisation objective. For the search space, a stratified node selection strategy and an adaptive edge choice strategy are developed to capture insertable and removable candidates. The former exploits unlabelled node information through massive pseudo-labels, whereas the latter capitalises on the similarity of node features. Regarding the optimisation objective, a temperature factor is introduced into the attack loss to better direct the flipping of the candidates. After a finite number of perturbations, the final disturbed graph is fed into various target models to evaluate attack performance. Generous trials indicate that our ETLA has excellent attack capability compared with other attack techniques with the same budget.
Landslide susceptibility mapping based on the reliability of landslide and non-landslide sample
2024, Expert Systems with Applications
Spatial data sampling can improve the performance in geo-spatial prediction. However, measuring the reliability of polygon-based data in sampling process is still a challenge. In this study, a reliability-based sampling (RBS) method was proposed to deal with this question and it was applied in landslide susceptibility mapping. First, the prototype of landslide was extracted from landslide polygon data, then, the reliability of landslide samples and non-landslide samples is measured using the similarity in environmental factor between the candidate samples and the prototype. The mutual exclusion reliability threshold setting method is used to collect the landslide samples and non-landslide samples with reliability over certain threshold. A case study demonstrates that the RBS method is better than existing representative method (i.e. Landslide entity) in terms of Accuracy and AUC with different sample sizes. In summary, The RBS is an efficient method to improve the spatial pattern of samples can also be applied to in other geo-spatial predictions.
YOLOv5 based object detection in reel package X-ray images of semiconductor component
2024, Heliyon
The industrial manufacturing landscape is currently shifting toward the incorporation of technologies based on artificial intelligence (AI). This transition includes an evolution toward smart factory infrastructure, with a specific focus on AI-driven strategies in production and quality control. Specifically, AI-empowered computer vision has emerged as a potent tool that offers a departure from extant rule-based systems and provides enhanced operational efficiency at manufacturing sites. As the manufacturing sector embraces this new paradigm, the impetus to integrate AI-integrated manufacturing is evident. Within this framework, one salient application is AI deep learning–facilitated small-object detection, which is poised to have extensive implications for diverse industrial applications. This study describes an optimized iteration of the YOLOv5 model, which is known for its efficacious single-stage object-detection abilities underpinned by PyTorch. Our proposed “improved model” incorporates an additional layer to the model's canonical three-layer architecture, augmenting accuracy and computational expediency. Empirical evaluations using semiconductor X-ray imagery reveal the model's superior performance metrics. Given the intricate specifications of surface-mount technologies, which are characterized by a plethora of micro-scale components, our model makes a seminal contribution to real-time, in-line production assessments. Quantitative analyses show that our improved model attained a mean average precision of 0.622, surpassing YOLOv5's 0.349, and a marked accuracy enhancement of 0.865, which is a significant improvement on YOLOv5's 0.552. These findings bolster the model's robustness and potential applicability, particularly in discerning objects at reel granularities during real-time inferencing.
Development of hybrid models based on deep learning and optimized machine learning algorithms for brain tumor Multi-Classification
2024, Expert Systems with Applications
Accurate classification of magnetic resonance imaging (MRI) images of brain tumors is crucial for early diagnosis and effective treatment in clinical studies. In these studies, many models supported by artificial intelligence (AI) have been proposed as assistant systems for experts. In particular, state-of-the-art deep learning (DL) models that have proven themselves in different fields have been effectively used in the classification of brain MRI images. However, the low accuracy of multiple classification of these images still leads researchers to conduct different studies in this field. Especially there is a need to develop models that achieve high accuracy on original images, and it is believed that this need can be met not only by DL models but also by classical machine learning (ML) algorithms. However, it is critical to choose the hyperparameters correctly for the hybrid use of ML algorithms with DL models. This study proposes a powerful new hybrid method to perform multiple classifications of brain tumors with high accuracy. This method also uses a novel convolutional neural network (CNN) model for feature extraction, and ML algorithms are used for feature classification. In addition, nine state-of-the-art CNN models are used for CNN performance comparison. The Bayesian optimization algorithm is used to obtain the optimal hyperparameter values of ML algorithms. The results obtained from the experimental studies show that the proposed hybrid model achieved 97.15% mean classification accuracy and 97% recall, precision, and F1-score values. Other hybrid models, including DarkNet19-SVM, DarkNet53-SVM, DenseNet201-SVM, EfficientNetB0-SVM, InceptionV3-SVM, NasNetMobile-SVM, ResNet50-SVM, ResNet101-SVM, and Xception-SVM, achieved mean classification accuracies of 95.01%, 95.58%, 96.87%, 97.01%, 95.3%, 95.01%, 96.3%, 95.87%, and 96.23%, respectively. Additionally, the proposed hybrid model exhibited remarkable time efficiency, accomplishing the classification process in a mere 67 min. Conversely, the model that exhibited the lowest time efficiency was the InceptionV3, with a processing time of 370 min. In terms of computational complexity, the EfficientNetB0 model is the most efficient. Despite the higher computational complexity of the proposed CNN model compared to some other models, it achieves the second-best classification accuracy. These results show that the proposed method performs better than previous studies on the same dataset. Especially in the classification problem, the optimized ML algorithms were superior to CNN classifiers. Finally, except for one, the proposed CNN model achieved better classification accuracies than the state-of-the-art CNN models.

View all citing articles on Scopus

View full text

An improved method to construct basic probability assignment based on the confusion matrix for classification problem

Abstract

Introduction

Section snippets

Basic concepts

Proposed method

Case study

Conclusion

Acknowledgments

Eur. J. Oper. Res.

Artif. Intell.

Phys. Rep.

Appl. Math. Comput.

Appl. Soft Comput.

Eur. J. Oper. Res.

Expert Syst. Appl.

Expert Syst. Appl.

Knowl. Based Syst.

J. Theor. Biol.

Appl. Math. Comput.

Comput. Biol. Chem.

Patt. Recognit.

Pattern Recog. Lett.

Inf. Fusion

Knowl.-Based Syst.

Decis. Support Syst.

Physica A: Stat. Mech. Appl.

Comput. Biol. Chem.

Pattern Recog.

Decis. Supp. Syst.

Pattern Recog. Lett.

Appl. Soft Comput.

Neural Netwo.

Risk Anal.

Knowl. Based Syst.

Comput. Indust. Eng.

Lect. Notes Comput. Sci.

Phys. Life Rev.

Phys. Rev. E

Eur. Phys. J. B

Inf. Sci.

Expert Syst. Appl.

Int. J. Prod. Res.