Skip to main content
Top

2017 | OriginalPaper | Chapter

Targeted Feedback Collection Applied to Multi-Criteria Source Selection

Authors : Julio César Cortés Ríos, Norman W. Paton, Alvaro A. A. Fernandes, Edward Abel, John A. Keane

Published in: Advances in Databases and Information Systems

Publisher: Springer International Publishing

Activate our intelligent search to find suitable subject content or patents.

search-config
loading …

Abstract

A multi-criteria source selection (MCSS) scenario identifies, from a set of candidate data sources, the subset that best meets a user’s needs. These needs are expressed using several criteria, which are used to evaluate the candidate data sources. A MCSS problem can be solved using multi-dimensional optimisation techniques that trade-off the different objectives. Sometimes we may have uncertain knowledge regarding how well the candidate data sources meet the criteria. In order to overcome this uncertainty, we may rely on end users or crowds to annotate the data items produced by the sources in relation to the selection criteria. In this paper, we introduce an approach called Targeted Feedback Collection (TFC), which aims to identify those data items on which feedback should be collected, thereby providing evidence on how the sources satisfy the required criteria. TFC targets feedback by considering the confidence intervals around the estimated criteria values. The TFC strategy has been evaluated, with promising results, against other approaches to feedback collection, including active learning, using real-world data sets.

Dont have a licence yet? Then find out more about our products and how to get one now:

Springer Professional "Wirtschaft+Technik"

Online-Abonnement

Mit Springer Professional "Wirtschaft+Technik" erhalten Sie Zugriff auf:

  • über 102.000 Bücher
  • über 537 Zeitschriften

aus folgenden Fachgebieten:

  • Automobil + Motoren
  • Bauwesen + Immobilien
  • Business IT + Informatik
  • Elektrotechnik + Elektronik
  • Energie + Nachhaltigkeit
  • Finance + Banking
  • Management + Führung
  • Marketing + Vertrieb
  • Maschinenbau + Werkstoffe
  • Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

Springer Professional "Technik"

Online-Abonnement

Mit Springer Professional "Technik" erhalten Sie Zugriff auf:

  • über 67.000 Bücher
  • über 390 Zeitschriften

aus folgenden Fachgebieten:

  • Automobil + Motoren
  • Bauwesen + Immobilien
  • Business IT + Informatik
  • Elektrotechnik + Elektronik
  • Energie + Nachhaltigkeit
  • Maschinenbau + Werkstoffe




 

Jetzt Wissensvorsprung sichern!

Springer Professional "Wirtschaft"

Online-Abonnement

Mit Springer Professional "Wirtschaft" erhalten Sie Zugriff auf:

  • über 67.000 Bücher
  • über 340 Zeitschriften

aus folgenden Fachgebieten:

  • Bauwesen + Immobilien
  • Business IT + Informatik
  • Finance + Banking
  • Management + Führung
  • Marketing + Vertrieb
  • Versicherung + Risiko




Jetzt Wissensvorsprung sichern!

Literature
1.
go back to reference Belhajjame, K., Paton, N.W., Embury, S.M., Fernandes, A.A.A., Hedeler, C.: Incrementally improving dataspaces based on user feedback. Inf. Syst. 38(5), 656–687 (2013)CrossRef Belhajjame, K., Paton, N.W., Embury, S.M., Fernandes, A.A.A., Hedeler, C.: Incrementally improving dataspaces based on user feedback. Inf. Syst. 38(5), 656–687 (2013)CrossRef
2.
go back to reference Bozzon, A., Brambilla, M., Ceri, S.: Answering search queries with crowdsearcher. In: WWW 2012, Lyon, France, pp. 1009–1018, 16–20 April 2012 Bozzon, A., Brambilla, M., Ceri, S.: Answering search queries with crowdsearcher. In: WWW 2012, Lyon, France, pp. 1009–1018, 16–20 April 2012
3.
go back to reference Bulmer, M.G.: Principles of Statistics. Dover Publications, New York (1979)MATH Bulmer, M.G.: Principles of Statistics. Dover Publications, New York (1979)MATH
4.
go back to reference Crescenzi, V., Merialdo, P., Qiu, D.: Crowdsourcing large scale wrapper inference. Distrib. Parallel Databases 33(1), 95–122 (2015)CrossRef Crescenzi, V., Merialdo, P., Qiu, D.: Crowdsourcing large scale wrapper inference. Distrib. Parallel Databases 33(1), 95–122 (2015)CrossRef
5.
go back to reference Dong, X.L., Saha, B., Srivastava, D.: Less is more: selecting sources wisely for integration. PVLDB 6(2), 37–48 (2012) Dong, X.L., Saha, B., Srivastava, D.: Less is more: selecting sources wisely for integration. PVLDB 6(2), 37–48 (2012)
6.
go back to reference Foley, D.H.: Considerations of sample and feature size. IEEE Trans. Inf. Theor. 18(5), 618–626 (1972)CrossRefMATH Foley, D.H.: Considerations of sample and feature size. IEEE Trans. Inf. Theor. 18(5), 618–626 (1972)CrossRefMATH
7.
go back to reference Franklin, M., Kossmann, D., Kraska, T., Ramesh, S., Xin, R.: Crowddb: answering queries with crowdsourcing. In: ACM SIGMOD, pp. 61–72 (2011) Franklin, M., Kossmann, D., Kraska, T., Ramesh, S., Xin, R.: Crowddb: answering queries with crowdsourcing. In: ACM SIGMOD, pp. 61–72 (2011)
8.
go back to reference Halevy, A., Korn, F., Noy, N.F., Olston, C., Polyzotis, N., Roy, S., Whang, S.E.: Goods: organizing google’s datasets. In: ACM SIGMOD, pp. 795–806 (2016) Halevy, A., Korn, F., Noy, N.F., Olston, C., Polyzotis, N., Roy, S., Whang, S.E.: Goods: organizing google’s datasets. In: ACM SIGMOD, pp. 795–806 (2016)
9.
go back to reference Hung, N.Q.V., Thang, D.C., Weidlich, M., Aberer, K.: Minimizing efforts in validating crowd answers. In: SIGMOD, Australia, pp. 999–1014 (2015) Hung, N.Q.V., Thang, D.C., Weidlich, M., Aberer, K.: Minimizing efforts in validating crowd answers. In: SIGMOD, Australia, pp. 999–1014 (2015)
10.
go back to reference Knezevic, A.: Overlapping confidence intervals and statistical significance. StatNews, Cornell University, Cornell Statistical Consulting Unit 73 (2008) Knezevic, A.: Overlapping confidence intervals and statistical significance. StatNews, Cornell University, Cornell Statistical Consulting Unit 73 (2008)
11.
go back to reference Lewis, D.D., Gale, W.A.: A sequential algorithm for training text classifiers. In: ACM-SIGIR, pp. 3–12 (1994) Lewis, D.D., Gale, W.A.: A sequential algorithm for training text classifiers. In: ACM-SIGIR, pp. 3–12 (1994)
12.
go back to reference Lewis, J.R., Sauro, J.: When 100% really isn’t 100%: improving the accuracy of small-sample estimates of completion rates. JUS 3(1), 136–150 (2006) Lewis, J.R., Sauro, J.: When 100% really isn’t 100%: improving the accuracy of small-sample estimates of completion rates. JUS 3(1), 136–150 (2006)
13.
go back to reference Liu, X., Lu, M., Ooi, B.C., Shen, Y., Wu, S., Zhang, M.: CDAS: a crowdsourcing data analytics system. PVLDB 5(10), 1040–1051 (2012) Liu, X., Lu, M., Ooi, B.C., Shen, Y., Wu, S., Zhang, M.: CDAS: a crowdsourcing data analytics system. PVLDB 5(10), 1040–1051 (2012)
14.
go back to reference Mozafari, B., Sarkar, P., Franklin, M.J., Jordan, M.I., Madden, S.: Scaling up crowd-sourcing to very large datasets: a case for active learning. PVLDB 8(2), 125–136 (2014) Mozafari, B., Sarkar, P., Franklin, M.J., Jordan, M.I., Madden, S.: Scaling up crowd-sourcing to very large datasets: a case for active learning. PVLDB 8(2), 125–136 (2014)
15.
go back to reference Pipino, L.L., Lee, Y.W., Wang, R.Y.: Data quality assessment. Commun. ACM 45(4), 211–218 (2002). Supporting community and building social capital, USACrossRef Pipino, L.L., Lee, Y.W., Wang, R.Y.: Data quality assessment. Commun. ACM 45(4), 211–218 (2002). Supporting community and building social capital, USACrossRef
17.
go back to reference Rekatsinas, T., Dong, X.L., Getoor, L., Srivastava, D.: Finding quality in quantity: the challenge of discovering valuable sources for integration. In: CIDR (2015) Rekatsinas, T., Dong, X.L., Getoor, L., Srivastava, D.: Finding quality in quantity: the challenge of discovering valuable sources for integration. In: CIDR (2015)
18.
go back to reference Rekatsinas, T., Dong, X.L., Srivastava, D.: Characterizing and selecting fresh data sources. In: SIGMOD, pp. 919–930 (2014) Rekatsinas, T., Dong, X.L., Srivastava, D.: Characterizing and selecting fresh data sources. In: SIGMOD, pp. 919–930 (2014)
19.
go back to reference Ríos, J.C.C., Paton, N.W., Fernandes, A.A.A., Belhajjame, K.: Efficient feedback collection for pay-as-you-go source selection. In: SSDBM, pp. 1:1–1:12 (2016) Ríos, J.C.C., Paton, N.W., Fernandes, A.A.A., Belhajjame, K.: Efficient feedback collection for pay-as-you-go source selection. In: SSDBM, pp. 1:1–1:12 (2016)
21.
go back to reference Ting, S.C., Cho, D.I.: An integrated approach for supplier selection and purchasing decisions. Supply Chain Manag. Int. J. 13(2), 116–127 (2008)CrossRef Ting, S.C., Cho, D.I.: An integrated approach for supplier selection and purchasing decisions. Supply Chain Manag. Int. J. 13(2), 116–127 (2008)CrossRef
Metadata
Title
Targeted Feedback Collection Applied to Multi-Criteria Source Selection
Authors
Julio César Cortés Ríos
Norman W. Paton
Alvaro A. A. Fernandes
Edward Abel
John A. Keane
Copyright Year
2017
DOI
https://doi.org/10.1007/978-3-319-66917-5_10

Premium Partner