Skip to main content

2019 | OriginalPaper | Buchkapitel

7. Online Multimodal Co-indexing and Retrieval of Social Media Data

verfasst von : Lei Meng, Ah-Hwee Tan, Donald C. Wunsch II

Erschienen in: Adaptive Resonance Theory in Social Media Data Clustering

Verlag: Springer International Publishing

Aktivieren Sie unsere intelligente Suche um passende Fachinhalte oder Patente zu finden.

search-config
loading …

Abstract

Effective indexing of social media data is key to searching for information on the social Web. However, the characteristics of social media data make it a challenging task. The large-scale and streaming nature is the first challenge, which requires the indexing algorithm to be able to efficiently update the indexing structure when receiving data streams. The second challenge is utilizing the rich meta-information of social media data for a better evaluation of the similarity between data objects and for a more semantically meaningful indexing of the data, which may allow the users to search for them using the different types of queries they like. Existing approaches based on either matrix operations or hashing usually cannot perform an online update of the indexing base to encode upcoming data streams, and they have difficulty handling noisy data. This chapter presents a study on using the Online Multimodal Co-indexing Adaptive Resonance Theory (OMC-ART) for an effective and efficient indexing and retrieval of social media data. More specifically, two types of social media data are considered: (1) the weakly supervised image data, which is associated with captions, tags and descriptions given by the users; and (2) the e-commerce product data, which includes product images, titles, descriptions and user comments. These scenarios make this study related to multimodal web image indexing and retrieval. Compared with existing studies, OMC-ART has several distinct characteristics. First, OMC-ART is able to perform online learning of sequential data. Second, instead of a plain indexing structure, OMC-ART builds a two-layer one, in which the first layer co-indexes the images by the key visual and textual features based on the generalized distributions of the clusters they belong to; while in the second layer, the data objects are co-indexed by their own feature distributions. Third, OMC-ART enables flexible multimodal searching by using either visual features, keywords, or a combination of both. Fourth, OMC-ART employs a ranking algorithm that does not need to go through the whole indexing system when only a limited number of images need to be retrieved. Experiments on two publicly accessible image datasets and a real-world e-commerce dataset demonstrate the efficiency and effectiveness of OMC-ART. The content of this chapter is summarized and extended from [13] (https://​doi.​org/​10.​1145/​2671188.​2749362), and the Python codes of OMC-ART with examples on building an e-commerce product search engine are available at https://​github.​com/​Lei-Meng/​OMC-ART-Build-a-toy-online-search-engine-.

Sie haben noch keine Lizenz? Dann Informieren Sie sich jetzt über unsere Produkte:

Springer Professional "Wirtschaft+Technik"

Online-Abonnement

Mit Springer Professional "Wirtschaft+Technik" erhalten Sie Zugriff auf:

  • über 102.000 Bücher
  • über 537 Zeitschriften

aus folgenden Fachgebieten:

  • Automobil + Motoren
  • Bauwesen + Immobilien
  • Business IT + Informatik
  • Elektrotechnik + Elektronik
  • Energie + Nachhaltigkeit
  • Finance + Banking
  • Management + Führung
  • Marketing + Vertrieb
  • Maschinenbau + Werkstoffe
  • Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

Springer Professional "Technik"

Online-Abonnement

Mit Springer Professional "Technik" erhalten Sie Zugriff auf:

  • über 67.000 Bücher
  • über 390 Zeitschriften

aus folgenden Fachgebieten:

  • Automobil + Motoren
  • Bauwesen + Immobilien
  • Business IT + Informatik
  • Elektrotechnik + Elektronik
  • Energie + Nachhaltigkeit
  • Maschinenbau + Werkstoffe




 

Jetzt Wissensvorsprung sichern!

Springer Professional "Wirtschaft"

Online-Abonnement

Mit Springer Professional "Wirtschaft" erhalten Sie Zugriff auf:

  • über 67.000 Bücher
  • über 340 Zeitschriften

aus folgenden Fachgebieten:

  • Bauwesen + Immobilien
  • Business IT + Informatik
  • Finance + Banking
  • Management + Führung
  • Marketing + Vertrieb
  • Versicherung + Risiko




Jetzt Wissensvorsprung sichern!

Literatur
1.
Zurück zum Zitat Caicedo JC, Moreno JG, Niño EA, González FA (2010) Combining visual features and text data for medical image retrieval using latent semantic kernels. In: Proceedings of the international conference on multimedia information retrieval, pp 359–366 Caicedo JC, Moreno JG, Niño EA, González FA (2010) Combining visual features and text data for medical image retrieval using latent semantic kernels. In: Proceedings of the international conference on multimedia information retrieval, pp 359–366
2.
Zurück zum Zitat Caicedo JC, BenAbdallah J, González FA, Nasraoui O (2012) Multimodal representation, indexing, automated annotation and retrieval of image collections via non-negative matrix factorization. Neurocomputing 76(1):50–60CrossRef Caicedo JC, BenAbdallah J, González FA, Nasraoui O (2012) Multimodal representation, indexing, automated annotation and retrieval of image collections via non-negative matrix factorization. Neurocomputing 76(1):50–60CrossRef
3.
Zurück zum Zitat Chandrika P, Jawahar CV (2010) Multi modal semantic indexing for image retrieval. In: Proceedings of the international conference on image and video retrieval, pp 342–349 Chandrika P, Jawahar CV (2010) Multi modal semantic indexing for image retrieval. In: Proceedings of the international conference on image and video retrieval, pp 342–349
4.
Zurück zum Zitat Chua T, Tang J, Hong R, Li H, Luo Z, Zheng Y (2009) NUS-WIDE: a real-world web image database from National University of Singapore. In: CIVR, pp 1–9 Chua T, Tang J, Hong R, Li H, Luo Z, Zheng Y (2009) NUS-WIDE: a real-world web image database from National University of Singapore. In: CIVR, pp 1–9
5.
Zurück zum Zitat Duygulu P, Barnard K, de Freitas JF, Forsyth DA (2002) Object recognition as machine translation: learning a lexicon for a fixed image vocabulary. In: ECCV, pp 97–112 Duygulu P, Barnard K, de Freitas JF, Forsyth DA (2002) Object recognition as machine translation: learning a lexicon for a fixed image vocabulary. In: ECCV, pp 97–112
6.
Zurück zum Zitat Escalante HJ, Montes M, Sucar E (2012) Multimodal indexing based on semantic cohesion for image retrieval. Inf Retr 15(1):1–32CrossRef Escalante HJ, Montes M, Sucar E (2012) Multimodal indexing based on semantic cohesion for image retrieval. Inf Retr 15(1):1–32CrossRef
7.
Zurück zum Zitat Gong Y, Wang L, Hodosh M, Hockenmaier J, Lazebnik S (2014) Improving image-sentence embeddings using large weakly annotated photo collections. In: Proceedings of the European conference on computer vision (ECCV), pp 529–545 Gong Y, Wang L, Hodosh M, Hockenmaier J, Lazebnik S (2014) Improving image-sentence embeddings using large weakly annotated photo collections. In: Proceedings of the European conference on computer vision (ECCV), pp 529–545
8.
Zurück zum Zitat Gonzalez F, Caicedo J (2010) NMF-based multimodal image indexing for querying by visual example. In: Proceedings of the international conference on image and video retrieval, pp 366–373 Gonzalez F, Caicedo J (2010) NMF-based multimodal image indexing for querying by visual example. In: Proceedings of the international conference on image and video retrieval, pp 366–373
9.
Zurück zum Zitat Li M, Xue XB, Zhou ZH (2009) Exploiting multi-modal interactions: a unified framework. In: IJCAI, pp 1120–1125 Li M, Xue XB, Zhou ZH (2009) Exploiting multi-modal interactions: a unified framework. In: IJCAI, pp 1120–1125
10.
Zurück zum Zitat Lienhart R, Romberg S, Hörster E (2009) Multilayer pLSA for multimodal image retrieval. In: Proceedings of the ACM international conference on image and video retrieval Lienhart R, Romberg S, Hörster E (2009) Multilayer pLSA for multimodal image retrieval. In: Proceedings of the ACM international conference on image and video retrieval
11.
Zurück zum Zitat Mei T, Rui Y, Li S, Tian Q (2014) Multimedia search reranking: a literature survey. ACM Comput Surv (CSUR) 46(3):38CrossRef Mei T, Rui Y, Li S, Tian Q (2014) Multimedia search reranking: a literature survey. ACM Comput Surv (CSUR) 46(3):38CrossRef
12.
Zurück zum Zitat Meng L, Tan AH, Xu D (2014) Semi-supervised heterogeneous fusion for multimedia data co-clustering. IEEE Trans Knowl Data Eng 26(9):2293–2306CrossRef Meng L, Tan AH, Xu D (2014) Semi-supervised heterogeneous fusion for multimedia data co-clustering. IEEE Trans Knowl Data Eng 26(9):2293–2306CrossRef
13.
Zurück zum Zitat Meng L, Tan AH, Leung C, Nie L, Chua TS, Miao C (2015) Online multimodal co-indexing and retrieval of weakly labeled web image collections. In: Proceedings of the 5th ACM on international conference on multimedia retrieval. ACM, pp 219–226. https://doi.org/10.1145/2671188.2749362 Meng L, Tan AH, Leung C, Nie L, Chua TS, Miao C (2015) Online multimodal co-indexing and retrieval of weakly labeled web image collections. In: Proceedings of the 5th ACM on international conference on multimedia retrieval. ACM, pp 219–226. https://​doi.​org/​10.​1145/​2671188.​2749362
14.
Zurück zum Zitat Mu Y, Shen J, Yan S (2010) Weakly-supervised hashing in kernel space. In: Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR), pp 3344–3351 Mu Y, Shen J, Yan S (2010) Weakly-supervised hashing in kernel space. In: Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR), pp 3344–3351
15.
Zurück zum Zitat Nie L, Wang M, Zha ZJ, Li G, Chua TS (2011) Multimedia answering: enriching text QA with media information. In: SIGIR, pp 695–704 Nie L, Wang M, Zha ZJ, Li G, Chua TS (2011) Multimedia answering: enriching text QA with media information. In: SIGIR, pp 695–704
16.
Zurück zum Zitat Nie L, Wang M, Gao Y, Zha ZJ, Chua TS (2013) Beyond text QA: multimedia answer generation by harvesting web information. IEEE Trans Multimed 15(2):426–441CrossRef Nie L, Wang M, Gao Y, Zha ZJ, Chua TS (2013) Beyond text QA: multimedia answer generation by harvesting web information. IEEE Trans Multimed 15(2):426–441CrossRef
17.
Zurück zum Zitat Smeulders AW, Worring M, Santini S, Gupta A, Jain R (2000) Content-based image retrieval at the end of the early years. IEEE Trans Pattern Anal Mach Intell 22(12):1349–1380CrossRef Smeulders AW, Worring M, Santini S, Gupta A, Jain R (2000) Content-based image retrieval at the end of the early years. IEEE Trans Pattern Anal Mach Intell 22(12):1349–1380CrossRef
18.
Zurück zum Zitat Su JH, Wang BW, Hsu TY, Chou CL, Tseng VS (2010) Multi-modal image retrieval by integrating web image annotation, concept matching and fuzzy ranking techniques. Int J Fuzzy Syst 12(2):136–149 Su JH, Wang BW, Hsu TY, Chou CL, Tseng VS (2010) Multi-modal image retrieval by integrating web image annotation, concept matching and fuzzy ranking techniques. Int J Fuzzy Syst 12(2):136–149
19.
Zurück zum Zitat Yu FX, Ji R, Tsai MH, Ye G, Chang SF (2012) Weak attributes for large-scale image retrieval. In: Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR), pp 2949–2956 Yu FX, Ji R, Tsai MH, Ye G, Chang SF (2012) Weak attributes for large-scale image retrieval. In: Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR), pp 2949–2956
20.
Zurück zum Zitat Zhang S, Yang M, Wang X, Lin Y, Tian Q (2013) Semantic-aware co-indexing for image retrieval. In: Proceedings of the IEEE international conference on computer vision (ICCV), pp 1673–1680 Zhang S, Yang M, Wang X, Lin Y, Tian Q (2013) Semantic-aware co-indexing for image retrieval. In: Proceedings of the IEEE international conference on computer vision (ICCV), pp 1673–1680
Metadaten
Titel
Online Multimodal Co-indexing and Retrieval of Social Media Data
verfasst von
Lei Meng
Ah-Hwee Tan
Donald C. Wunsch II
Copyright-Jahr
2019
DOI
https://doi.org/10.1007/978-3-030-02985-2_7