Skip to main content
Top

2021 | OriginalPaper | Chapter

A System for Interactive Multimedia Retrieval Evaluations

Authors : Luca Rossetto, Ralph Gasser, Loris Sauter, Abraham Bernstein, Heiko Schuldt

Published in: MultiMedia Modeling

Publisher: Springer International Publishing

Activate our intelligent search to find suitable subject content or patents.

search-config
loading …

Abstract

The evaluation of the performance of interactive multimedia retrieval systems is a methodologically non-trivial endeavour and requires specialized infrastructure. Current evaluation campaigns have so far relied on a local setting, where all retrieval systems needed to be evaluated at the same physical location at the same time. This constraint does not only complicate the organization and coordination but also limits the number of systems which can reasonably be evaluated within a set time frame. Travel restrictions might further limit the possibility for such evaluations. To address these problems, evaluations need to be conducted in a (geographically) distributed setting, which was so far not possible due to the lack of supporting infrastructure. In this paper, we present the Distributed Retrieval Evaluation Server (DRES), an open-source evaluation system to facilitate evaluation campaigns for interactive multimedia retrieval systems in both traditional on-site as well as fully distributed settings which has already proven effective in a competitive evaluation.

To get access to this content you need the following product:

Springer Professional "Wirtschaft+Technik"

Online-Abonnement

Mit Springer Professional "Wirtschaft+Technik" erhalten Sie Zugriff auf:

  • über 102.000 Bücher
  • über 537 Zeitschriften

aus folgenden Fachgebieten:

  • Automobil + Motoren
  • Bauwesen + Immobilien
  • Business IT + Informatik
  • Elektrotechnik + Elektronik
  • Energie + Nachhaltigkeit
  • Finance + Banking
  • Management + Führung
  • Marketing + Vertrieb
  • Maschinenbau + Werkstoffe
  • Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

Springer Professional "Technik"

Online-Abonnement

Mit Springer Professional "Technik" erhalten Sie Zugriff auf:

  • über 67.000 Bücher
  • über 390 Zeitschriften

aus folgenden Fachgebieten:

  • Automobil + Motoren
  • Bauwesen + Immobilien
  • Business IT + Informatik
  • Elektrotechnik + Elektronik
  • Energie + Nachhaltigkeit
  • Maschinenbau + Werkstoffe




 

Jetzt Wissensvorsprung sichern!

Springer Professional "Wirtschaft"

Online-Abonnement

Mit Springer Professional "Wirtschaft" erhalten Sie Zugriff auf:

  • über 67.000 Bücher
  • über 340 Zeitschriften

aus folgenden Fachgebieten:

  • Bauwesen + Immobilien
  • Business IT + Informatik
  • Finance + Banking
  • Management + Führung
  • Marketing + Vertrieb
  • Versicherung + Risiko




Jetzt Wissensvorsprung sichern!

Literature
1.
go back to reference Awad, G., et al.: Trecvid 2019: an evaluation campaign to benchmark video activity detection, video captioning and matching, and video search & retrieval. In: Proceedings of TRECVID 2019. NIST, USA (2019) Awad, G., et al.: Trecvid 2019: an evaluation campaign to benchmark video activity detection, video captioning and matching, and video search & retrieval. In: Proceedings of TRECVID 2019. NIST, USA (2019)
3.
go back to reference Gasser, R., Rossetto, L., Heller, S., Schuldt, H.: Cottontail DB: an open source database system for multimedia retrieval and analysis. In: Proceedings of the 28th ACM International Conference on Multimedia (MM 2020). ACM, Seattle, October 2020 Gasser, R., Rossetto, L., Heller, S., Schuldt, H.: Cottontail DB: an open source database system for multimedia retrieval and analysis. In: Proceedings of the 28th ACM International Conference on Multimedia (MM 2020). ACM, Seattle, October 2020
4.
go back to reference Giangreco, I., Schuldt, H.: Adam pro: database support for big multimedia retrieval. Datenbank-Spektrum 16(1), 17–26 (2016)CrossRef Giangreco, I., Schuldt, H.: Adam pro: database support for big multimedia retrieval. Datenbank-Spektrum 16(1), 17–26 (2016)CrossRef
5.
go back to reference Gurrin, C., et al.: Introduction to the third annual lifelog search challenge (LSC’20). In: Proceedings of the 2020 International Conference on Multimedia Retrieval, pp. 584–585 (2020) Gurrin, C., et al.: Introduction to the third annual lifelog search challenge (LSC’20). In: Proceedings of the 2020 International Conference on Multimedia Retrieval, pp. 584–585 (2020)
6.
go back to reference Gurrin, C., et al.: [Invited papers] comparing approaches to interactive lifelog search at the lifelog search challenge (LSC 2018). ITE Trans. Media Technol. Appl. 7(2), 46–59 (2019)CrossRef Gurrin, C., et al.: [Invited papers] comparing approaches to interactive lifelog search at the lifelog search challenge (LSC 2018). ITE Trans. Media Technol. Appl. 7(2), 46–59 (2019)CrossRef
7.
go back to reference Lokoč, J., et al.: Interactive search or sequential browsing? A detailed analysis of the video browser showdown 2018. ACM Trans. Multimed. Comput. Commun. Appl. 15(1), 1–18 (2019) Lokoč, J., et al.: Interactive search or sequential browsing? A detailed analysis of the video browser showdown 2018. ACM Trans. Multimed. Comput. Commun. Appl. 15(1), 1–18 (2019)
8.
go back to reference Pouyanfar, S., Yang, Y., Chen, S.C., Shyu, M.L., Iyengar, S.S.: Multimedia big data analytics: a survey. ACM Comput. Surv. 51(1) (2018) Pouyanfar, S., Yang, Y., Chen, S.C., Shyu, M.L., Iyengar, S.S.: Multimedia big data analytics: a survey. ACM Comput. Surv. 51(1) (2018)
11.
go back to reference Schoeffmann, K.: A user-centric media retrieval competition: the video browser showdown 2012–2014. IEEE Multimed. 21(4), 8–13 (2014)CrossRef Schoeffmann, K.: A user-centric media retrieval competition: the video browser showdown 2012–2014. IEEE Multimed. 21(4), 8–13 (2014)CrossRef
12.
go back to reference Schoeffmann, K.: Video browser showdown 2012–2019: a review. In: 2019 International Conference on Content-Based Multimedia Indexing (CBMI), pp. 1–4. IEEE (2019) Schoeffmann, K.: Video browser showdown 2012–2019: a review. In: 2019 International Conference on Content-Based Multimedia Indexing (CBMI), pp. 1–4. IEEE (2019)
13.
go back to reference Smeaton, A.F., Over, P., Taban, R.: The TREC-2001 video track report. In: TREC (2001) Smeaton, A.F., Over, P., Taban, R.: The TREC-2001 video track report. In: TREC (2001)
14.
go back to reference Snoek, C.G., Worring, M., de Rooij, O., van de Sande, K.E., Yan, R., Hauptmann, A.G.: VideOlympics: real-time evaluation of multimedia retrieval systems. IEEE Multimed. 15(1), 86–91 (2008)CrossRef Snoek, C.G., Worring, M., de Rooij, O., van de Sande, K.E., Yan, R., Hauptmann, A.G.: VideOlympics: real-time evaluation of multimedia retrieval systems. IEEE Multimed. 15(1), 86–91 (2008)CrossRef
Metadata
Title
A System for Interactive Multimedia Retrieval Evaluations
Authors
Luca Rossetto
Ralph Gasser
Loris Sauter
Abraham Bernstein
Heiko Schuldt
Copyright Year
2021
DOI
https://doi.org/10.1007/978-3-030-67835-7_33

Premium Partner