nach oben

Data Mining and Knowledge Discovery

Erschienen in:

13.04.2016

Evidence-based uncertainty sampling for active learning

verfasst von: Manali Sharma, Mustafa Bilgic

Erschienen in: Data Mining and Knowledge Discovery | Ausgabe 1/2017

Einloggen

Aktivieren Sie unsere intelligente Suche, um passende Fachinhalte oder Patente zu finden.

search-config

KI-gestützte Suche

Aus

Abstract

Active learning methods select informative instances to effectively learn a suitable classifier. Uncertainty sampling, a frequently utilized active learning strategy, selects instances about which the model is uncertain but it does not consider the reasons for why the model is uncertain. In this article, we present an evidence-based framework that can uncover the reasons for why a model is uncertain on a given instance. Using the evidence-based framework, we discuss two reasons for uncertainty of a model: a model can be uncertain about an instance because it has strong, but conflicting evidence for both classes or it can be uncertain because it does not have enough evidence for either class. Our empirical evaluations on several real-world datasets show that distinguishing between these two types of uncertainties has a drastic impact on the learning efficiency. We further provide empirical and analytical justifications as to why distinguishing between the two uncertainties matters.

Vorheriger Artikel Outlying property detection with numerical attributes

Nächster Artikel Efficient histogram dictionary learning for text/image modeling and classification

Sie haben noch keine Lizenz? Dann Informieren Sie sich jetzt über unsere Produkte:

Springer Professional "Wirtschaft+Technik"

Online-Abonnement

Mit Springer Professional "Wirtschaft+Technik" erhalten Sie Zugriff auf:

über 102.000 Bücher
über 537 Zeitschriften

aus folgenden Fachgebieten:

Automobil + Motoren
Bauwesen + Immobilien
Business IT + Informatik
Elektrotechnik + Elektronik
Energie + Nachhaltigkeit
Finance + Banking
Management + Führung
Marketing + Vertrieb
Maschinenbau + Werkstoffe
Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

Jetzt informieren

Springer Professional "Technik"

Online-Abonnement

Mit Springer Professional "Technik" erhalten Sie Zugriff auf:

über 67.000 Bücher
über 390 Zeitschriften

aus folgenden Fachgebieten:

Automobil + Motoren
Bauwesen + Immobilien
Business IT + Informatik
Elektrotechnik + Elektronik
Energie + Nachhaltigkeit
Maschinenbau + Werkstoffe

Jetzt Wissensvorsprung sichern!

Jetzt informieren

Springer Professional "Wirtschaft"

Online-Abonnement

Mit Springer Professional "Wirtschaft" erhalten Sie Zugriff auf:

über 67.000 Bücher
über 340 Zeitschriften

aus folgenden Fachgebieten:

Bauwesen + Immobilien
Business IT + Informatik
Finance + Banking
Management + Führung
Marketing + Vertrieb
Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

Jetzt informieren

1,507 citations on Google Scholar on April 4th, 2016.

In practice, however, \(E_{+1}(x^{(i)})\) and \(E_{-1}(x^{(i)})\) might not be exactly equal to each other for all uncertain instances, and hence the ranking of uncertain instances based on evidence according to Eqs. 9, 10, 11, and 12 may be different.

This figure does not correspond to a real-time simulation of active learning with users. When the user-provided labels are used, the underlying active learning strategy, whether it be UNC-CE or UNC-IE, would potentially take a different path per user based on their labels. Then, each user would potentially differ on the documents they label, and therefore meaningful comparisons of time and accuracy across users would not be possible.

Abe N, Mamitsuka H (1998) Query learning strategies using boosting and bagging. In: Proceedings of the fifteenth international conference on machine learning, pp 1–9

Bilgic M, Mihalkova L, Getoor L (2010) Active learning for networked data. In: Proceedings of the 27th international conference on machine learning, pp 79–86

Chao C, Cakmak M, Thomaz AL (2010) Transparent active learning for robots. In: 5th ACM/IEEE international conference on Human–Robot interaction (HRI), IEEE, pp 317–324

Cohn DA (1997) Minimizing statistical bias with queries. In: Advances in neural information processing systems, pp 417–423

Cohn DA, Ghahramani Z, Jordan MI (1996) Active learning with statistical models. J Artif Intell Res 4:129–145MATH

Dagan I, Engelson SP (1995) Committee-based sampling for training probabilistic classifiers. In: Proceedings of the twelfth international conference on machine learning, pp 150–157

Donmez P, Carbonell JG, Bennett PN (2007) Dual strategy active learning. In: Machine learning: ECML 2007. Springer, pp 116–127

Frank A, Asuncion A (2010) UCI machine learning repository. http://archive.ics.uci.edu/ml

Frey PW, Slate DJ (1991) Letter recognition using holland-style adaptive classifiers. Mach Learn 6(2):161–182

Gu Q, Zhang T, Han J, Ding CH (2012) Selective labeling via error bound minimization. In: Advances in neural information processing systems, pp 323–331

Gu Q, Zhang T, Han J (2014) Batch-mode active learning via error bound minimization. In: Proceedings of the Thirtieth conference annual conference on uncertainty in artificial intelligence (UAI-14). AUAI Press, Corvallis, Oregon, pp 300–309

Guyon I et al (2011) Datasets of the active learning challenge. J Mach Learn Res

Hoi SC, Jin R, Lyu MR (2006a) Large-scale text categorization by batch mode active learning. In: Proceedings of the 15th international conference on World Wide Web, ACM, pp 633–642

Hoi SC, Jin R, Zhu J, Lyu MR (2006b) Batch mode active learning and its application to medical image classification. In: Proceedings of the 23rd international conference on machine learning, ACM, pp 417–424

Lewis DD, Gale WA (1994) A sequential algorithm for training text classifiers. In: Proceedings of the 17th annual international ACM SIGIR conference on research and development in information retrieval. Springer-Verlag New York, Inc., pp 3–12

Maas AL, Daly RE, Pham PT, Huang D, Ng AY, Potts C (2011) Learning word vectors for sentiment analysis. In: Proceedings of the 49th annual meeting of the association for computational linguistics: human language technologies, vol 1. Association for Computational Linguistics, pp 142–150

MacKay DJ (1992) Information-based objective functions for active data selection. Neural Comput 4(4):590–604CrossRef

McCallum A, Nigam K et al (1998) A comparison of event models for naive bayes text classification. In: AAAI-98 workshop on learning for text categorization, Citeseer, vol 752, pp 41–48

Melville P, Mooney RJ (2004) Diverse ensembles for active learning. In: Proceedings of the twenty-first international conference on machine learning, pp 74

Mitchell TM (1982) Generalization as search. Artif Intell 18(2):203–226MathSciNetCrossRef

Nguyen HT, Smeulders A (2004) Active learning using pre-clustering. In: Proceedings of the twenty-first international conference on machine learning, ACM, p 79

Pace RK, Barry R (1997) Sparse spatial autoregressions. Stat Probab Lett 33(3):291–297CrossRefMATH

Roy N, McCallum A (2001) Toward optimal active learning through sampling estimation of error reduction. In: Proceedings of the eighteenth international conference on machine learning. Morgan Kaufmann Publishers Inc., ICML ’01, pp 441–448

Sculley D (2007) Online active learning methods for fast label-efficient spam filtering. In: Fourth conference on email and anti-spam (CEAS)

Segal R, Markowitz T, Arnold W (2006) Fast uncertainty sampling for labeling large e-mail corpora. In: Third conference on email and anti-spam (CEAS)

Senge R, Bösner S, Dembczyński K, Haasenritter J, Hirsch O, Donner-Banzhoff N, Hüllermeier E (2014) Reliable classification: Learning classifiers that distinguish aleatoric and epistemic uncertainty. Inf Sci 255:16–29MathSciNetCrossRefMATH

Settles B (2012) Active learning. Synth Lect Artif Intell Mach Learn 6(1):1–114MathSciNetCrossRefMATH

Settles B, Craven M (2008) An analysis of active learning strategies for sequence labeling tasks. In: Proceedings of the conference on empirical methods in natural language processing. Association for Computational Linguistics, pp 1070–1079

Seung HS, Opper M, Sompolinsky H (1992) Query by committee. In: Proceedings of the fifth annual workshop on computational learning theory, ACM, pp 287–294

Sharma M, Bilgic M (2013) Most-surely vs. least-surely uncertain. In: IEEE 13th international conference on data mining (ICDM), pp 667–676

Sindhwani V, Melville P, Lawrence RD (2009) Uncertainty sampling and transductive experimental design for active dual supervision. In: Proceedings of the 26th annual international conference on machine learning, ACM, pp 953–960

Steuer RE (1989) Multiple criteria optimization: theory, computations, and application. Krieger Pub Co

Thompson CA, Califf ME, Mooney RJ (1999) Active learning for natural language parsing and information extraction. In: Proceedings of the sixteenth international conference on machine learning, pp 406–414

Tong S, Chang E (2001) Support vector machine active learning for image retrieval. In: Proceedings of the ninth ACM international conference on multimedia, ACM, pp 107–118

Xu Z, Yu K, Tresp V, Xu X, Wang J (2003) Representative sampling for text classification using support vector machines. In: Advances in information retrieval. Lecture notes in computer science, vol 2633, pp 393–407

Yu K, Bi J, Tresp V (2006) Active learning via transductive experimental design. In: Proceedings of the 23rd international conference on machine learning, ACM, pp 1081–1088

Zhang C, Chen T (2002) An active learning framework for content-based information retrieval. IEEE Trans Multimedia 4(2):260–268CrossRef

Zhu J, Wang H, Yao T, Tsou BK (2008) Active learning with sampling by uncertainty and density for word sense disambiguation and text classification. In: Proceedings of the 22nd international conference on computational linguistics, vol 1, pp 1137–1144

Titel: Evidence-based uncertainty sampling for active learning
verfasst von: Manali Sharma
Mustafa Bilgic
Publikationsdatum: 13.04.2016
Verlag: Springer US
Erschienen in: Data Mining and Knowledge Discovery / Ausgabe 1/2017
Print ISSN: 1384-5810
Elektronische ISSN: 1573-756X
DOI: https://doi.org/10.1007/s10618-016-0460-3

Springer Professional

Abstract

Bitte loggen Sie sich ein, um Zugang zu Ihrer Lizenz zu erhalten.

Sie haben noch keine Lizenz? Dann Informieren Sie sich jetzt über unsere Produkte:

Springer Professional "Wirtschaft+Technik"

Springer Professional "Technik"

Springer Professional "Wirtschaft"

Weitere Artikel der Ausgabe 1/2017

Reliable early classification of time series based on discriminating the classes over time

Outlying property detection with numerical attributes

Generalizing DTW to the multi-dimensional case requires an adaptive approach

TBM, a transformation based method for microaggregation of large volume mixed data

Hierarchical evolving Dirichlet processes for modeling nonlinear evolutionary traces in temporal data

SimUSF: an efficient and effective similarity measure that is invariant to violations of the interval scale assumption