Skip to main content
Disciplines
Automotive Business IT + Informatics Construction + Real Estate Electrical Engineering + Electronics Energy + Sustainability Insurance + Risk Finance + Banking Management + Leadership Marketing + Sales Mechanical Engineering + Materials
Events
DE
EN
Books
Journals
Topic Page
Marketing
Start single access now
Access for companies
EXTENDED SEARCH
Log in
Top
Published in:
Defining and Applying a Language for Discovery Read chapter Read first chapter

2014 | OriginalPaper | Chapter

Detector Performance Prediction Using Set Annotations

Authors : Robin Aly, Martha Larson

Published in: Adaptive Multimedia Retrieval: Semantics, Context, and Adaptation

Publisher: Springer International Publishing

Log in

Activate our intelligent search to find suitable subject content or patents.

search-config
loading …

Abstract

Content-based videos search engines often use the output of concept detectors to answer queries. The improvement of detectors requires computational power and human labor. It is therefore important to predict detector performance economically and improve detectors adaptively. Detector performance prediction, however, has not received much research attention so far. In this paper, we propose a prediction approach that uses human annotators. The annotators estimate the number of images in a grid in which a concept is present, a task that can be performed efficiently. Using these estimations, we define a model for the posterior probability of a concept being present given its confidence score. We then use the model to predict the average precision of a detector. We evaluate our approach using a TRECVid collection of Internet archive videos, comparing it to an approach that labels individual images. Our approach requires fewer resources while achieving good prediction quality.

Dont have a licence yet? Then find out more about our products and how to get one now:

Springer Professional "Wirtschaft+Technik"

Online-Abonnement

Mit Springer Professional "Wirtschaft+Technik" erhalten Sie Zugriff auf:

  • über 102.000 Bücher
  • über 537 Zeitschriften

aus folgenden Fachgebieten:

  • Automobil + Motoren
  • Bauwesen + Immobilien
  • Business IT + Informatik
  • Elektrotechnik + Elektronik
  • Energie + Nachhaltigkeit
  • Finance + Banking
  • Management + Führung
  • Marketing + Vertrieb
  • Maschinenbau + Werkstoffe
  • Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

inform now

Springer Professional "Technik"

Online-Abonnement

Mit Springer Professional "Technik" erhalten Sie Zugriff auf:

  • über 67.000 Bücher
  • über 390 Zeitschriften

aus folgenden Fachgebieten:

  • Automobil + Motoren
  • Bauwesen + Immobilien
  • Business IT + Informatik
  • Elektrotechnik + Elektronik
  • Energie + Nachhaltigkeit
  • Maschinenbau + Werkstoffe




 

Jetzt Wissensvorsprung sichern!

inform now

Springer Professional "Wirtschaft"

Online-Abonnement

Mit Springer Professional "Wirtschaft" erhalten Sie Zugriff auf:

  • über 67.000 Bücher
  • über 340 Zeitschriften

aus folgenden Fachgebieten:

  • Bauwesen + Immobilien
  • Business IT + Informatik
  • Finance + Banking
  • Management + Führung
  • Marketing + Vertrieb
  • Versicherung + Risiko




Jetzt Wissensvorsprung sichern!

inform now
previous chapter Adaptive Temporal Modeling of Audio Features in the Context of Music Structure Segmentation
next chapter Personas – The Missing Link Between User Simulations and User-Centered Design?
Appendix
Available only for authorised users
Footnotes
1
Performance evaluation and performance prediction can be performed similar but differ in their aim: performance evaluation aims at comparing detectors and performance prediction aims at deriving actions (e.g. change of detector technique).
 
2
Detector confidence scores indicates the belief of a detector that an image contains a concept.
 
3
We measured the pure annotation time, excluding the time to load the images.
 
Literature
1.
2.
Aly, R., Hiemstra, D., de Jong, F., Apers, P.: Simulating the future of concept-based video retrieval under improved detector performance. Multimed. Tools Appl. 60(1), 203–231 (2012)CrossRef
3.
Aly, R., Hiemstra, D., de Vries, A.P.: Reusing annotation labor for concept selection. In: CIVR ’09: Proceedings of the International Conference on Content-Based Image and Video Retrieval. ACM, New York (2009)
4.
Goldstein, E.: The perception of multiple images. Educ. Technol. Res. Dev. 23, 34–68 (1975)
5.
Hajimirza, S., Proulx, M., Izquierdo, E.: Reading users’ minds from their eyes: a method for implicit image annotation. IEEE Trans. Multimed. 14(3), 805–815 (2012)CrossRef
6.
Hauff, C., Azzopardi, L., Hiemstra, D., de Jong, F.: Query performance prediction: evaluation contrasted with effectiveness. In: Gurrin, C., He, Y., Kazai, G., Kruschwitz, U., Little, S., Roelleke, T., Rüger, S., van Rijsbergen, K. (eds.) ECIR 2010. LNCS, vol. 5993, pp. 204–216. Springer, Heidelberg (2010)CrossRef
7.
8.
Jonassen, D.: Implications of multi-image for concept acquisition. Educ. Technol. Res. Dev. 27(4), 291–302 (1979)
9.
Over, P., Awad, G., Fiscus, J., Antonishek, B., Michel, M., Smeaton, A., Kraaij, W., Quénot, G.: TRECVID 2011 – An overview of the goals, tasks, data, evaluation mechanisms, and metrics. In: TREC 2011 Video Retrieval Evaluation Online Proceedings (TRECVid 2010). National Institute of Standards and Technology, Gaithersburg (2011)
10.
Smeaton, A.F., Over, P., Kraaij, W.: Evaluation campaigns and TRECVid. In: MIR ’06: Proceedings of the 8th ACM International Workshop on Multimedia Information Retrieval, pp. 321–330. ACM, New York (2006)
11.
Snoek, C.G.M., Worring, M.: Are concept detector lexicons effective for video search? In: 2007 IEEE International Conference on Multimedia and Expo, pp. 1966–1969 (2007)
12.
Snoek, C.G.M., Worring, M.: Concept-based video retrieval. Found. Trends Inf. Retr. 4(2), 215–322 (2009)
13.
Yang, J., Hauptmann, A.G.: (un)Reliability of video concept detection. In: CIVR ’08: Proceedings of the 2008 International Conference on Content-based Image and Video Retrieval, pp. 85–94. ACM, New York (2008)
14.
Yilmaz, E., Kanoulas, E., Aslam, J.: A simple and efficient sampling method for estimating AP and NDCG. In: SIGIR’08: Proceedings of the 31st Annual International ACM SIGIR Conference on Research And Development in Information Retrieval, pp. 603–610. ACM, New York (2008)
15.
Yilmaz, E., Aslam, J.A.: Inferred AP: estimating average precision with incomplete judgments. In: Fifteenth ACM International Conference on Information and Knowledge Management (CIKM), pp. 102–111. ACM, New York, November 2006
Metadata
Title
Detector Performance Prediction Using Set Annotations
Authors
Robin Aly
Martha Larson
Copyright Year
2014
Book
Adaptive Multimedia Retrieval: Semantics, Context, and Adaptation

Print ISBN: 978-3-319-12092-8

Electronic ISBN: 978-3-319-12093-5

Copyright Year: 2014

DOI
https://doi.org/10.1007/978-3-319-12093-5_16