nach oben

Discover Computing

Erschienen in:

21.09.2019 | Axiomatic Thinking for Information Retrieval

Evaluation measures for quantification: an axiomatic approach

verfasst von: Fabrizio Sebastiani

Erschienen in: Discover Computing | Ausgabe 3/2020

Einloggen

Aktivieren Sie unsere intelligente Suche, um passende Fachinhalte oder Patente zu finden.

search-config

KI-gestützte Suche

Aus

Abstract

Quantification is the task of estimating, given a set \(\sigma \) of unlabelled items and a set of classes \({\mathcal {C}}=\{c_{1}, \ldots , c_{|{\mathcal {C}}|}\}\), the prevalence (or “relative frequency”) in \(\sigma \) of each class \(c_{i}\in {\mathcal {C}}\). While quantification may in principle be solved by classifying each item in \(\sigma \) and counting how many such items have been labelled with \(c_{i}\), it has long been shown that this “classify and count” method yields suboptimal quantification accuracy. As a result, quantification is no longer considered a mere byproduct of classification, and has evolved as a task of its own. While the scientific community has devoted a lot of attention to devising more accurate quantification methods, it has not devoted much to discussing what properties an evaluation measure for quantification (EMQ) should enjoy, and which EMQs should be adopted as a result. This paper lays down a number of interesting properties that an EMQ may or may not enjoy, discusses if (and when) each of these properties is desirable, surveys the EMQs that have been used so far, and discusses whether they enjoy or not the above properties. As a result of this investigation, some of the EMQs that have been used in the literature turn out to be severely unfit, while others emerge as closer to what the quantification community actually needs. However, a significant result is that no existing EMQ satisfies all the properties identified as desirable, thus indicating that more research is needed in order to identify (or synthesize) a truly adequate EMQ.

Vorheriger Artikel On the foundations of similarity in information access

Nächster Artikel How do interval scales help us with better understanding IR evaluation measures?

Sie haben noch keine Lizenz? Dann Informieren Sie sich jetzt über unsere Produkte:

Springer Professional "Wirtschaft+Technik"

Online-Abonnement

Mit Springer Professional "Wirtschaft+Technik" erhalten Sie Zugriff auf:

über 102.000 Bücher
über 537 Zeitschriften

aus folgenden Fachgebieten:

Automobil + Motoren
Bauwesen + Immobilien
Business IT + Informatik
Elektrotechnik + Elektronik
Energie + Nachhaltigkeit
Finance + Banking
Management + Führung
Marketing + Vertrieb
Maschinenbau + Werkstoffe
Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

Jetzt informieren

Springer Professional "Technik"

Online-Abonnement

Mit Springer Professional "Technik" erhalten Sie Zugriff auf:

über 67.000 Bücher
über 390 Zeitschriften

aus folgenden Fachgebieten:

Automobil + Motoren
Bauwesen + Immobilien
Business IT + Informatik
Elektrotechnik + Elektronik
Energie + Nachhaltigkeit
Maschinenbau + Werkstoffe

Jetzt Wissensvorsprung sichern!

Jetzt informieren

Springer Professional "Wirtschaft"

Online-Abonnement

Mit Springer Professional "Wirtschaft" erhalten Sie Zugriff auf:

über 67.000 Bücher
über 340 Zeitschriften

aus folgenden Fachgebieten:

Bauwesen + Immobilien
Business IT + Informatik
Finance + Banking
Management + Führung
Marketing + Vertrieb
Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

Jetzt informieren

Nur mit Berechtigung zugänglich

Consistently with most mathematical literature, we use the caret symbol (\(^{\hat{\,}}\)) to indicate estimation.

In order to keep things simple we avoid overspecifying the notation, thus leaving some aspects of it implicit; e.g., in order to indicate a true distribution p of the unlabelled items in a sample \(\sigma \) across a codeframe \({\mathcal {C}}\) we will simply write p instead of the more cumbersome \(p_{\sigma }^{{\mathcal {C}}}\), thus letting \(\sigma \) and \({\mathcal {C}}\) be inferred from context.

In this paper we do not discuss the evaluation of ordinal quantification (OQ), defined as SLQ with a codeframe \({\mathcal {C}}=\{c_{1}, \ldots , c_{|{\mathcal {C}}|}\}\) on which a total order \(c_{1} \prec \cdots \prec c_{|{\mathcal {C}}|}\) is defined. Aside from reasons of space, the reasons for disregarding OQ is that there has been very little work on it [the only papers we know being (Da San Martino et al. 2016a, b; Esuli 2016)], and that only one measure for OQ (the Earth Mover’s Distance—see Esuli and Sebastiani 2010) has been proposed and used so far. For the same reasons we do not discuss regression quantification (RQ), the task that stands to metric regression as single-label quantification stands to single-label classification. RQ has been studied even less than OQ, the only work appeared on this theme so far being, to the best of our knowledge (Bella et al. 2014), which as an evaluation measure has proposed the Cramér-von-Mises u-statistic (see Bella et al. 2014 for details).

Note that two distributions p(c) and \({\hat{p}}(c)\) over \({\mathcal {C}}\) are essentially two nonnegative-valued, length-normalized vectors of dimensionality \(|{\mathcal {C}}|\). The literature on EMQs thus obviously intersects the literature on functions for computing the similarity of two vectors.

A divergence is often indicated by the notation \(D(p||{\hat{p}})\)); we will prefer the more neutral notation \(D(p,{\hat{p}})\). Note also that a divergence can take as arguments any two distributions p and q defined on the same space of events, i.e., p and q need not be a true distribution and a predicted distribution. However, since we will consider divergences only as measures of fit between a true distribution and a predicted distribution, we will use the more specific notation \(D(p,{\hat{p}})\) rather than the more general D(p, q).

By the “range” of an EMQ here we actually mean its image (i.e., the set of values that the EMQ actually takes for its admissible input values), and not just its codomain.

One might argue that underestimating the prevalence of a class \(c_{1}\) always implies overestimating the prevalence of another class \(c_{2}\). However, there are cases in which \(c_{1}\) and \(c_{2}\) are not equally important. For instance, if \({\mathcal {C}}=\{c_{1},c_{2}\}\), with \(c_{1}\) the class of patients that suffer from a certain rare disease (say, one such that \(p(c_{1})=.0001\)) and \(c_{2}\) the class of patients who do not, the class whose prevalence we really want to quantify is \(c_{1}\), the prevalence of \(c_{2}\) being derivative. So, what we really care about is that underestimating \(p(c_{1})\) and overestimating \(p(c_{1})\) are equally penalized. The formulation of IMP, which involves underestimation and overestimation in a perfectly symmetric way, is strong enough that IMP is not satisfied (as we will see in Sect. 4) by a number of important EMQs.

In this case we enter the realm of cost-sensitive quantification, which is outside the scope of this paper; see (Forman 2008, §4 and §5) and (González et al. 2017, §10) for more on the relationships between quantification and cost.

The symbol \({\pm }\) stands for “plus or minus” while \({\mp }\) stands for “minus or plus”; when symbol \({\pm }\) evaluates to \(+\), symbol \({\mp }\) evaluates to −, and vice versa.

This is the basis of the “Strict Monotonicity” property discussed in Sebastiani (2015) for the evaluation of classification systems.

In Eq. 14 and in the rest of this paper the \(\log \) operator denotes the natural logarithm.

Since the standard logistic function \(\frac{e^{x}}{e^{x}+1}\) ranges (for the domain \([0,+\,\infty )\) we are interested in) on [\(\frac{1}{2}\),1], a multiplication by 2 is applied in order for it to range on [1,2], and 1 is subtracted in order for it to range on [0,1], as desired.

Esuli and Sebastiani (2014) mistakenly defined \({{\,\mathrm{NKLD}\,}}(p,{\hat{p}})\) as \(\frac{e^{{{\,\mathrm{KLD}\,}}(p,{\hat{p}})}-1}{e^{{{\,\mathrm{KLD}\,}}(p,{\hat{p}})}}\); this was later corrected into the formulation of Eq. 16 [which is equivalent to \(\frac{e^{{{\,\mathrm{KLD}\,}}(p,{\hat{p}})}-1}{e^{{{\,\mathrm{KLD}\,}}(p,{\hat{p}})}+1}\)] by Gao and Sebastiani (2016).

This is true only at a first approximation, though. In more precise terms, the maximum value that \({{\,\mathrm{NKLD}\,}}\) can have is strictly smaller than 1 because the maximum value that \({{\,\mathrm{KLD}\,}}\) can have is finite (see Eq. 15) and, as discussed at the end of Sect. 4.7, dependent on p, on the cardinality of \({\mathcal {C}}\), and even on the value of \(\epsilon \); as a result, the maximum value that \({{\,\mathrm{NKLD}\,}}\) can have is also dependent on these three variables (although it is always very close to 1—see the example in “Appendix 2.1” section).

It has to be remarked that, in some cases, differences of the latter type may be moderate, especially when |C| is high. For instance, when \(|C|=2\) the value of \(z_{AE}\) ranges on [0.5,1.0], but when \(|C|=10\) it ranges on [0.18, 0.20].

A similar situation occurs when evaluating multi-label classification via “microaveraged \(F_{1}\)”, a measure in which the classes with higher prevalence weigh more on the final result.

It is this author’s experience that even measures such as \(F_{1}\) can be considered by customers “esoteric”.

As an example, assume a (very realistic) scenario in which \(|\sigma |=1000\), \({\mathcal {C}}=\{c_{1},c_{2}\}\), \(p(c_{1})=0.01\), and in which three different quantifiers \({\hat{p}}'\), \({\hat{p}}''\), \({\hat{p}}'''\) are such that \({\hat{p}}'(c_{1})=0.0101\), \({\hat{p}}''(c_{1})=0.0110\), \({\hat{p}}'''(c_{1})=0.0200\). In this scenario \({{\,\mathrm{KLD}\,}}\) ranges on [0, 7.46], \({{\,\mathrm{KLD}\,}}(p,{\hat{p}}')=4.78\)e–07, \({{\,\mathrm{KLD}\,}}(p,{\hat{p}}'')=4.53\)e–05, \({{\,\mathrm{KLD}\,}}(p,{\hat{p}}''')=3.02\)e–03, i.e., the difference between \({{\,\mathrm{KLD}\,}}(p,{\hat{p}}')\) and \({{\,\mathrm{KLD}\,}}(p,{\hat{p}}'')\) (and the one between \({{\,\mathrm{KLD}\,}}(p,{\hat{p}}'')\) and \({{\,\mathrm{KLD}\,}}(p,{\hat{p}}''')\)) is 2 orders of magnitude, while the difference between \({{\,\mathrm{KLD}\,}}(p,{\hat{p}}')\) and \({{\,\mathrm{KLD}\,}}(p,{\hat{p}}''')\) is no less than 4 orders of magnitude. The increase in error (as computed by \({{\,\mathrm{KLD}\,}}\)) deriving from using \({\hat{p}}'''\) instead of \({\hat{p}}'\) is + 632,599%.

We assume \(|D|=1{,}000{,}000\). This assumption has no relevance on the qualitative conclusions we draw here, and only affects the magnitude of the values in the table (since the value of |D| affects the value of \(\epsilon \), and thus of \({{\,\mathrm{RAE}\,}}\), \({{\,\mathrm{NRAE}\,}}\), \({{\,\mathrm{DR}\,}}\), \({{\,\mathrm{KLD}\,}}\), \({{\,\mathrm{NKLD}\,}}\), \({{\,\mathrm{PD}\,}}\)—see Sect. 4.3) and following.

For the EMQs that require smoothed probabilities to be used, these definitions obviously need to be replaced by \(a\equiv p_{s}(c_{1})\) and \(x\equiv {\hat{p}}_{s}(c_{1})\).

Alaíz-Rodríguez, R., Guerrero-Curieses, A., & Cid-Sueiro, J. (2011). Class and subclass probability re-estimation to adapt a classifier in the presence of concept drift. Neurocomputing, 74(16), 2614–2623. https://doi.org/10.1016/j.neucom.2011.03.019.CrossRef

Ali, S. M., & Silvey, S. D. (1966). A general class of coefficients of divergence of one distribution from another. Journal of the Royal Statistical Society, Series B, 28(1), 131–142. https://doi.org/10.1111/j.2517-6161.1966.tb00626.x.MathSciNetCrossRefMATH

Amigó, E., Gonzalo, J., & Verdejo, F. (2011). A comparison of evaluation metrics for document filtering. In Proceedings of the 2nd international conference of the cross-language evaluation forum (CLEF 2011). Amsterdam, NL (pp. 38–49). https://doi.org/10.1007/978-3-642-23708-9_6.

Baccianella, S., Esuli, A., & Sebastiani, F. (2013). Variable-constraint classification and quantification of radiology reports under the ACR index. Expert Systems and Applications, 40(9), 3441–3449. https://doi.org/10.1016/j.eswa.2012.12.052.CrossRef

Barranquero, J., Díez, J., & del Coz, J. J. (2015). Quantification-oriented learning based on reliable classifiers. Pattern Recognition, 48(2), 591–604. https://doi.org/10.1016/j.patcog.2014.07.032.CrossRefMATH

Barranquero, J., González, P., Díez, J., & del Coz, J. J. (2013). On the study of nearest neighbor algorithms for prevalence estimation in binary problems. Pattern Recognition, 46(2), 472–482. https://doi.org/10.1016/j.patcog.2012.07.022.CrossRefMATH

Beijbom, O., Hoffman, J., Yao, E., Darrell, T., Rodriguez-Ramirez, A., Gonzalez-Rivero, M., et al. (2015). Quantification in-the-wild: Data-sets and baselines. In Presented at the NIPS 015 workshop on transfer and multi-task learning, CA: Montreal. CoRR. arXiv:1510.04811.

Bella, A., Ferri, C., Hernández-Orallo, J., & Ramírez-Quintana, M. J. (2010). Quantification via probability estimators. In Proceedings of the 11th IEEE international conference on data mining (ICDM 2010). Sydney, AU (pp. 737–742). https://doi.org/10.1109/icdm.2010.75.

Bella, A., Ferri, C., Hernández-Orallo, J., & Ramírez-Quintana, M. J. (2014). Aggregative quantification for regression. Data Mining and Knowledge Discovery, 28(2), 475–518. https://doi.org/10.1007/s10618-013-0308-z.MathSciNetCrossRefMATH

Busin, L., & Mizzaro, S. (2013). Axiometrics: An axiomatic approach to information retrieval effectiveness metrics. In Proceedings of the 4th international conference on the theory of information retrieval (ICTIR 2013). Copenhagen, DK (p. 8). https://doi.org/10.1145/2499178.2499182.

Card, D., & Smith, N. A. (2018). The importance of calibration for estimating proportions from annotations. In Proceedings of the 2018 conference of the North American chapter of the association for computational linguistics (HLT-NAACL 2018). New Orleans, US (pp. 1636–1646). https://doi.org/10.18653/v1/n18-1148.

Ceron, A., Curini, L., & Iacus, S. M. (2016). iSA: A fast, scalable and accurate algorithm for sentiment analysis of social media content. Information Sciences, 367(368), 105–124. https://doi.org/10.1016/j.ins.2016.05.052.CrossRef

Csiszár, I., & Shields, P. C. (2004). Information theory and statistics: A tutorial. Foundations and Trends in Communications and Information Theory, 1(4), 417–528. https://doi.org/10.1561/0100000004.CrossRefMATH

Da San Martino, G., Gao, W., & Sebastiani, F. (2016a). Ordinal text quantification. In Proceedings of the 39th ACM conference on research and development in information retrieval (SIGIR 2016). Pisa, IT (pp. 937–940). https://doi.org/10.1145/2911451.2914749.

Da San Martino, G., Gao, W., & Sebastiani, F. (2016b). QCRI at SemEval-2016 task 4: Probabilistic methods for binary and ordinal quantification. In Proceedings of the 10th international workshop on semantic evaluation (SemEval 2016). San Diego, US (pp. 58–63). https://doi.org/10.18653/v1/s16-1006.

dos Reis, D. M., Maletzke, A., Cherman, E., & Batista, G. E. (2018a). One-class quantification. In Proceedings of the European conference on machine learning and principles and practice of knowledge discovery in databases (ECML-PKDD 2018). Dublin, IE.

dos Reis, D. M., Maletzke, A. G., Silva, D. F., & Batista, G. E. (2018b). Classifying and counting with recurrent contexts. In Proceedings of the 24th ACM international conference on knowledge discovery and data mining (KDD 2018). London, UK (pp. 1983–1992). https://doi.org/10.1145/3219819.3220059.

du Plessis, M. C., Niu, G., & Sugiyama, M. (2017). Class-prior estimation for learning from positive and unlabeled data. Machine Learning, 106(4), 463–492. https://doi.org/10.1007/s10994-016-5604-6.MathSciNetCrossRefMATH

du Plessis, M. C., & Sugiyama, M. (2012). Semi-supervised learning of class balance under class-prior change by distribution matching. In Proceedings of the 29th international conference on machine learning (ICML 2012). Edinburgh, UK.

du Plessis, M. C., & Sugiyama, M. (2014). Class prior estimation from positive and unlabeled data. IEICE Transactions, 97–D(5), 1358–1362. https://doi.org/10.1587/transinf.e97.d.1358.CrossRef

Esuli, A. (2016). ISTI-CNR at SemEval-2016 task 4: Quantification on an ordinal scale. In Proceedings of the 10th international workshop on semantic evaluation (SemEval 2016). San Diego, US. https://doi.org/10.18653/v1/s16-1011.

Esuli, A., Moreo, A., & Sebastiani, F. (2018). A recurrent neural network for sentiment quantification. In Proceedings of the 27th ACM international conference on information and knowledge management (CIKM 2018). Torino, IT (pp. 1775–1778). https://doi.org/10.1145/3269206.3269287.

Esuli, A., & Sebastiani, F. (2010). Sentiment quantification. IEEE Intelligent Systems, 25(4), 72–75.CrossRef

Esuli, A., & Sebastiani, F. (2014). Explicit loss minimization in quantification applications (preliminary draft). In Proceedings of the 8th international workshop on information filtering and retrieval (DART 2014). Pisa, IT (pp. 1–11).

Esuli, A., & Sebastiani, F. (2015). Optimizing text quantifiers for multivariate loss functions. ACM Transactions on Knowledge Discovery and Data, 9(4), 27. https://doi.org/10.1145/2700406.CrossRef

Ferrante, M., Ferro, N., & Maistro, M. (2015). Towards a formal framework for utility-oriented measurements of retrieval effectiveness. In Proceedings of the 5th ACM international conference on the theory of information retrieval (ICTIR 2015). Northampton, US (pp. 21–30). https://doi.org/10.1145/2808194.2809452.

Ferrante, M., Ferro, N., & Pontarollo, S. (2018). A general theory of IR evaluation measures. IEEE Transactions on Knowledge and Data Engineering,. https://doi.org/10.1109/TKDE.2018.2840708.CrossRef

Forman, G. (2005). Counting positives accurately despite inaccurate classification. In Proceedings of the 16th European conference on machine learning (ECML 2005). Porto, PT (pp. 564–575). https://doi.org/10.1007/11564096_55.

Forman, G. (2006). Quantifying trends accurately despite classifier error and class imbalance. In Proceedings of the 12th ACM SIGKDD international conference on knowledge discovery and data mining (KDD 2006). Philadelphia, US (pp. 157–166). https://doi.org/10.1145/1150402.1150423.

Forman, G. (2008). Quantifying counts and costs via classification. Data Mining and Knowledge Discovery, 17(2), 164–206. https://doi.org/10.1007/s10618-008-0097-y.MathSciNetCrossRef

Gao, W., & Sebastiani, F. (2015). Tweet sentiment: From classification to quantification. In Proceedings of the 7th international conference on advances in social network analysis and mining (ASONAM 2015). Paris, FR (pp. 97–104). https://doi.org/10.1145/2808797.2809327.

Gao, W., & Sebastiani, F. (2016). From classification to quantification in tweet sentiment analysis. Social Network Analysis and Mining, 6(19), 1–22. https://doi.org/10.1007/s13278-016-0327-z.CrossRef

González, P., Álvarez, E., Díez, J., López-Urrutia, Á., & del Coz, J. J. (2017). Validation methods for plankton image classification systems. Limnology and Oceanography: Methods, 15, 221–237. https://doi.org/10.1002/lom3.10151.CrossRef

González, P., Castaño, A., Chawla, N. V., & del Coz, J. J. (2017). A review on quantification learning. ACM Computing Surveys, 50(5), 74:1–74:40. https://doi.org/10.1145/3117807.CrossRef

González, P., Díez, J., Chawla, N., & del Coz, J. J. (2017). Why is quantification an interesting learning problem? Progress in Artificial Intelligence, 6(1), 53–58. https://doi.org/10.1007/s13748-016-0103-3.CrossRef

González-Castro, V., Alaiz-Rodríguez, R., & Alegre, E. (2013). Class distribution estimation based on the Hellinger distance. Information Sciences, 218, 146–164. https://doi.org/10.1016/j.ins.2012.05.028.CrossRef

González-Castro, V., Alaiz-Rodríguez, R., Fernández-Robles, L., Guzmán-Martínez, R., & Alegre, E. (2010). Estimating class proportions in boar semen analysis using the Hellinger distance. In Proceedings of the 23rd international conference on industrial engineering and other applications of applied intelligent systems (IEA/AIE 2010). Cordoba, ES (pp. 284–293). https://doi.org/10.1007/978-3-642-13022-9_29.

Hopkins, D. J., & King, G. (2010). A method of automated nonparametric content analysis for social science. American Journal of Political Science, 54(1), 229–247. https://doi.org/10.1111/j.1540-5907.2009.00428.x.CrossRef

Kar, P., Li, S., Narasimhan, H., Chawla, S., & Sebastiani, F. (2016). Online optimization methods for the quantification problem. In Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining (KDD 2016). San Francisco, US (pp. 1625–1634). https://doi.org/10.1145/2939672.2939832.

Keith, K. A., & O’Connor, B. (2018). Uncertainty-aware generative models for inferring document class prevalence. In Proceedings of the conference on empirical methods in natural language processing (EMNLP 2018). Brussels, BE.

King, G., & Ying, L. (2008). Verbal autopsy methods with multiple causes of death. Statistical Science, 23(1), 78–91. https://doi.org/10.1214/07-sts247.MathSciNetCrossRefMATH

Levin, R., & Roitman, H. (2017). Enhanced probabilistic classify and count methods for multi-label text quantification. In Proceedings of the 7th ACM international conference on the theory of information retrieval (ICTIR 2017). Amsterdam, NL (pp. 229–232). https://doi.org/10.1145/3121050.3121083.

Liese, F., & Vajda, I. (2006). On divergences and informations in statistics and information theory. IEEE Transactions on Information Theory, 52(10), 4394–4412. https://doi.org/10.1109/tit.2006.881731.MathSciNetCrossRefMATH

Lin, J. (1991). Divergence measures based on the Shannon entropy. IEEE Transactions on Information Theory, 37(1), 145–151. https://doi.org/10.1109/18.61115.MathSciNetCrossRefMATH

MacKay, D. J. (2003). Information theory, inference and learning algorithms. Cambridge: Cambridge University Press.MATH

Maletzke, A. G., dos Reis, D. M., & Batista, G. E. (2017). Quantification in data streams: Initial results. In Proceedings of the 2017 Brazilian conference on intelligent systems (BRACIS 2017). Uberlândia, BZ (pp. 43–48). https://doi.org/10.1109/BRACIS.2017.74.

Maletzke, A. G., Moreira dos Reis, D., & Batista, G. E. (2018). Combining instance selection and self-training to improve data stream quantification. Journal of the Brazilian Computer Society, 24(12), 43–48. https://doi.org/10.1186/s13173-018-0076-0.CrossRef

Milli, L., Monreale, A., Rossetti, G., Giannotti, F., Pedreschi, D., & Sebastiani, F. (2013). Quantification trees. In Proceedings of the 13th IEEE international conference on data mining (ICDM 2013). Dallas, US (pp. 528–536). https://doi.org/10.1109/icdm.2013.122.

Milli, L., Monreale, A., Rossetti, G., Pedreschi, D., Giannotti, F., & Sebastiani, F. (2015). Quantification in social networks. In Proceedings of the 2nd IEEE international conference on data science and advanced analytics (DSAA 2015). Paris, FR. https://doi.org/10.1109/dsaa.2015.7344845.

Moffat, A. (2013). Seven numeric properties of effectiveness metrics. In Proceedings of the 9th conference of the Asia information retrieval societies (AIRS 2013). Singapore, SN (pp. 1–12). https://doi.org/10.1007/978-3-642-45068-6_1.

Nakov, P., Farra, N., & Rosenthal, S. (2017). SemEval-2017 task 4: Sentiment analysis in Twitter. In Proceedings of the 11th international workshop on semantic evaluation (SemEval 2017). Vancouver, CA. https://doi.org/10.18653/v1/s17-2088.

Nakov, P., Ritter, A., Rosenthal, S., Sebastiani, F., & Stoyanov, V. (2016). SemEval-2016 task 4: Sentiment analysis in Twitter. In Proceedings of the 10th international workshop on semantic evaluation (SemEval 2016). San Diego, US (pp. 1–18). https://doi.org/10.18653/v1/s16-1001.

Pérez-Gállego, P., Castaño, A., Quevedo, J. R., & del Coz, J. J. (2019). Dynamic ensemble selection for quantification tasks. Information Fusion, 45, 1–15. https://doi.org/10.1016/j.inffus.2018.01.001.CrossRef

Pérez-Gállego, P., Quevedo, J. R., & del Coz, J. J. (2017). Using ensembles for problems with characterizable changes in data distribution: A case study on quantification. Information Fusion, 34, 87–100. https://doi.org/10.1016/j.inffus.2016.07.001.CrossRef

Saerens, M., Latinne, P., & Decaestecker, C. (2002). Adjusting the outputs of a classifier to new a priori probabilities: A simple procedure. Neural Computation, 14(1), 21–41. https://doi.org/10.1162/089976602753284446.CrossRefMATH

Sanya, A., Kumar, P., Kar, P., Chawla, S., & Sebastiani, F. (2018). Optimizing non-decomposable measures with deep networks. Machine Learning, 107(8–10), 1597–1620. https://doi.org/10.1007/s10994-018-5736-y.MathSciNetCrossRefMATH

Sebastiani, F. (2015). An axiomatically derived measure for the evaluation of classification algorithms. In Proceedings of the 5th ACM international conference on the theory of information retrieval (ICTIR 2015). Northampton, US (pp. 11–20). https://doi.org/10.1145/2808194.2809449.

Tang, L., Gao, H., & Liu, H. (2010). Network quantification despite biased labels. In Proceedings of the 8th workshop on mining and learning with graphs (MLG 2010). Washington, US (pp. 147–154). https://doi.org/10.1145/1830252.1830271.

Tasche, D. (2017). Fisher consistency for prior probability shift. Journal of Machine Learning Research, 18, 95:1–95:32.MathSciNetMATH

Vaz, A. F., Izbicki, R., & Stern, R. B. (2018). Quantification under prior probability shift: The ratio estimator and its extensions. arXiv preprint arXiv:1807.03929.

Zhang, Z., & Zhou, J. (2010). Transfer estimation of evolving class priors in data stream classification. Pattern Recognition, 43(9), 3151–3161. https://doi.org/10.1016/j.patcog.2010.03.021.CrossRefMATH

Titel: Evaluation measures for quantification: an axiomatic approach
verfasst von: Fabrizio Sebastiani
Publikationsdatum: 21.09.2019
Verlag: Springer Netherlands
Erschienen in: Discover Computing / Ausgabe 3/2020
Print ISSN: 2948-2984
Elektronische ISSN: 2948-2992
DOI: https://doi.org/10.1007/s10791-019-09363-y

Springer Professional

Abstract

Bitte loggen Sie sich ein, um Zugang zu Ihrer Lizenz zu erhalten.

Sie haben noch keine Lizenz? Dann Informieren Sie sich jetzt über unsere Produkte:

Springer Professional "Wirtschaft+Technik"

Springer Professional "Technik"

Springer Professional "Wirtschaft"

Weitere Artikel der Ausgabe 3/2020

Axiomatic thinking for information retrieval: introduction to special issue

On the nature of information access evaluation metrics: a unifying framework

On the foundations of similarity in information access

An axiomatic approach to corpus-based cross-language information retrieval

How do interval scales help us with better understanding IR evaluation measures?

Premium Partner