Skip to main content
Disciplines
Automotive Business IT + Informatics Construction + Real Estate Electrical Engineering + Electronics Energy + Sustainability Insurance + Risk Finance + Banking Management + Leadership Marketing + Sales Mechanical Engineering + Materials
Events
DE
EN
Books
Journals
Topic Page
Marketing
Start single access now
Access for companies
EXTENDED SEARCH
Log in
Top
Published in:
Improving BERT-based Query-by-Document Retrieval with Multi-task Optimization Read chapter Read first chapter

2022 | OriginalPaper | Chapter

Enhanced Sentence Meta-Embeddings for Textual Understanding

Authors : Sourav Dutta, Haytham Assem

Published in: Advances in Information Retrieval

Publisher: Springer International Publishing

Log in

Activate our intelligent search to find suitable subject content or patents.

search-config
loading …

Abstract

Sentence embeddings provide vector representations for sentences and short texts, enabling the capture of contextual and semantic meaning for different applications. However, the diversity of sentence embedding techniques poses a challenge, in terms of choosing the model best suited for the downstream task. As such, meta-embeddings study different techniques for combining embeddings from multiple sources. In this paper, we propose CINCE, a principled meta-embedding framework for aggregating various semantic information, captured by different embeddings techniques, via multiple component analysis strategies. Experiments on SentEval benchmark exhibit improved performance for semantic understanding and text classification, compared to existing approaches.

Dont have a licence yet? Then find out more about our products and how to get one now:

Springer Professional "Wirtschaft+Technik"

Online-Abonnement

Mit Springer Professional "Wirtschaft+Technik" erhalten Sie Zugriff auf:

  • über 102.000 Bücher
  • über 537 Zeitschriften

aus folgenden Fachgebieten:

  • Automobil + Motoren
  • Bauwesen + Immobilien
  • Business IT + Informatik
  • Elektrotechnik + Elektronik
  • Energie + Nachhaltigkeit
  • Finance + Banking
  • Management + Führung
  • Marketing + Vertrieb
  • Maschinenbau + Werkstoffe
  • Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

inform now

Springer Professional "Technik"

Online-Abonnement

Mit Springer Professional "Technik" erhalten Sie Zugriff auf:

  • über 67.000 Bücher
  • über 390 Zeitschriften

aus folgenden Fachgebieten:

  • Automobil + Motoren
  • Bauwesen + Immobilien
  • Business IT + Informatik
  • Elektrotechnik + Elektronik
  • Energie + Nachhaltigkeit
  • Maschinenbau + Werkstoffe




 

Jetzt Wissensvorsprung sichern!

inform now

Springer Professional "Wirtschaft"

Online-Abonnement

Mit Springer Professional "Wirtschaft" erhalten Sie Zugriff auf:

  • über 67.000 Bücher
  • über 340 Zeitschriften

aus folgenden Fachgebieten:

  • Bauwesen + Immobilien
  • Business IT + Informatik
  • Finance + Banking
  • Management + Führung
  • Marketing + Vertrieb
  • Versicherung + Risiko




Jetzt Wissensvorsprung sichern!

inform now
previous chapter Temporal Event Reasoning Using Multi-source Auxiliary Learning Objectives
next chapter Match Your Words! A Study of Lexical Matching in Neural Information Retrieval
Literature
1.
Andrew, G., Arora, R., Bilmes, J., Livescu, K.: Deep canonical correlation analysis. In: International Conference on Machine Learning (ICML), pp. 1247–1255 (2013)
2.
Artetxe, M., Schwenk, H.: Massively multilingual sentence embeddings for zero-shot cross-lingual transfer and beyond. Trans. Assoc. Comput. Linguist. 7, 597–610 (2019)CrossRef
3.
Assem, H., Dutta, S., Burgin, E.: DTAFA: decoupled training architecture for efficient FAQ retrieval. In: Annual Meeting of the Special Interest Group on Discourse and Dialogue (SIGDIAL), pp. 423–430 (2021)
4.
Bao, C., Bollegala, D.: Learning word meta-embeddings by autoencoding. In: Conference on Computational Linguistics (COLING), pp. 1650–1661 (2018)
5.
Bojanowski, P., Grave, E., Joulin, A., Mikolov, T.: Enriching word vectors with subword information. Trans. Assoc. Comput. Linguist. 5, 135–146 (2017)CrossRef
6.
7.
Chelba, C., et al.: One billion word benchmark for measuring progress in statistical language modeling. In: Conference of the International Speech Communication Association (INTERSPEECH), pp. 2635–2639 (2014)
8.
Chen, W., Sheng, M., Mao, J., Sheng, W.: Investigating word meta-embeddings by disentangling common and individual information. IEEE Access 8, 11692–11699 (2020)CrossRef
9.
Coates, J.N., Bollegala, D.: Frustratingly easy meta-embedding - computing meta-embeddings by averaging source word embeddings. In: North American Chapter of the Association for Computational Linguistics (NAACL-HLT), pp. 194–198 (2018)
10.
Comon, P.: Independent component analysis - a new concept? Signal Process. 36, 287–314 (1994)CrossRefMATH
11.
Conneau, A., Kiela, D.: SentEval: an evaluation toolkit for universal sentence representations. In: International Conference on Language Resources and Evaluation (LREC), pp. 1699–1704 (2018)
12.
Conneau, A., Kiela, D., Schwenk, H., Barrault, L., Borders, A.: Supervised learning of universal sentence representations from natural language inference data. In: Empirical Methods in Natural Language Processing (EMNLP), pp. 670–680 (2017)
13.
Conneau, A., Kruszewski, G., Lample, G., Barrault, L., Baroni, M.: What you can cram into a single \$&!#* vector: probing sentence embeddings for linguistic properties. In: Annual Meeting of the Association for Computational Linguistics (ACL), pp. 2126–2136 (2018)
14.
Devlin, J., Chang, M., Lee, K., Toutanova, K.: BERT: pre-training of deep bidirectional transformers for language understanding. In: North American Chapter of the Association for Computational Linguistics (NAACL-HLT), pp. 4171–4186 (2019)
15.
Dutta, S.: “Alignment is All You Need”: analyzing cross-lingual text similarity for domain-specific applications. In: The Web Conference - Workshop on Cross-lingual Event-centric Open Analytics, pp. 85–92 (2021)
16.
Goswami, K., Dutta, S., Assem, H., Fransen, T., McCrae, J.: Cross-lingual sentence embedding using multi-task learning. In: Empirical Methods in Natural Language Processing (EMNLP) (2021)
17.
Hazem, A., Hernandez, N.: Meta-embedding sentence representation for textual similarity. In: Recent Advances in Natural Language Processing (RANLP), pp. 465–473 (2019)
18.
Hirota, W., Suhara, Y., Golshan, B., Tan, W.C.: EMU: enhancing multilingual sentence embeddings with semantic specialization. In: AAAI Conference on Artificial Intelligence (AAAI), pp. 7935–7943 (2020)
19.
Hotelling, H.: Analysis of a complex of statistical variables into principal components. J. Educ. Psychol. 24(417–441), 498–520 (1933)CrossRefMATH
20.
Hotelling, H.: Relations between two sets of variates. Biometrika 28(3–4), 321–377 (1936)CrossRefMATH
21.
Hyvärinen, A., Oja, E.: Independent component analysis: algorithms and applications. Neural Netw. 13(4–5), 411–430 (2000)CrossRef
22.
Jutten, C., Hérault, J.: Blind separation of sources, Part I: an adaptive algorithm based on neuromimetic architecture. Signal Process. 24, 1–10 (1991)CrossRefMATH
23.
24.
Kiela, D., Wang, C., Cho, K.: Dynamic meta-embeddings for improved sentence representations. In: Empirical Methods in Natural Language Processing (EMNLP), pp. 1466–1477 (2018)
25.
Knapp, T.R.: Canonical correlation analysis: a general parametric significance-testing system. Psychol. Bull. 85(2), 410–416 (1978)MathSciNetCrossRef
26.
Kornblith, S., Norouzi, M., Lee, H., Hinton, G.: Similarity of neural network representations revisited. In: International Conference on Machine Learning (ICML), pp. 3519–3529 (2019)
27.
Lample, G., Conneau, A.: Cross-lingual language model pretraining. In: Conference on Neural Information Processing Systems (NIPS), pp. 1–11 (2019)
28.
29.
Li, B., Zhou, H., He, J., Wang, M., Yang, Y., Li, L.: On the sentence embeddings from pre-trained language models. In: Empirical Methods in Natural Language Processing (EMNLP), pp. 9119–9130 (2020)
30.
Lin, Z., et al.: A structured self-attentive sentence embedding. In: International Conference on Learning Representations (ICLR), pp. 1–15 (2017)
31.
Mika, S., Schölkopf, B., Smola, A., Müller, K.R., Scholz, M., Rätsch, G.: Kernel PCA and de-noising in feature spaces. In: Conference on Neural Information Processing Systems (NIPS), pp. 536–542 (1998)
32.
Mikolov, T., Sutskever, I., Chen, K., Corrado, G.S., Dean, J.: Distributed representations of words and phrases and their compositionality. In: Conference on Neural Information Processing Systems (NIPS), pp. 3111–3119 (2013)
33.
O’Neill, J., Bollegala, D.: Meta-embedding as auxiliary task regularization. In: European Conference on Artificial Intelligence (ECAI), pp. 2124–2131 (2020)
34.
Patel, R.N., Burgin, E., Assem, H., Dutta, S.: Efficient multi-lingual sentence classification framework with sentence meta encoders. In: IEEE International Conference on Big Data (2021)
35.
Pearson, K.: On lines and planes of closest fit to systems of points in space. Phil. Mag. 2(11), 559–572 (1901)CrossRefMATH
36.
Pennington, J., Socher, R., Manning, C.D.: GloVe: global vectors for word representation. In: Empirical Methods in Natural Language Processing (EMNLP), pp. 1532–1543 (2014)
37.
Poerner, N., Waltinger, U., Schütze, H.: Sentence meta-embeddings for unsupervised semantic textual similarity. In: Annual Meeting of the Association for Computational Linguistics (ACL), pp. 7027–7034 (2020)
38.
Reimers, N., Gurevych, I.: Sentence-BERT: sentence embeddings using Siamese BERT-networks. In: Empirical Methods in Natural Language Processing (EMNLP), pp. 3982–3992 (2019)
39.
Schölkopf, B.: Nonlinear component analysis as a kernel eigenvalue problem. Neural Comput. 10(5), 1299–1319 (1998)CrossRef
40.
Senel, L.K., Utlu, I., Yucesoy, V., Koc, A., Cukur, T.: Semantic structure and interpretability of word embeddings. IEEE/ACM Trans. Audio Speech Lang. Process. 26(10), 1769–1779 (2018)CrossRef
41.
42.
Sørensen, M., Kanatsoulis, C.I., Sidiropoulos, N.D.: Generalized canonical correlation analysis: a subspace intersection approach. arXiv:​2003.​11205 (2020)
43.
Tsatsishvilil, V., Cong, F., Toiviainen, P., Ristaniemil, T.: Combining PCA and multiset CCA for dimension reduction when group ICA is applied to decompose naturalistic fMRI data. In: International Joint Conference on Neural Networks (IJCNN), pp. 1–6 (2015)
44.
Wieting, J., Gimpel, K.: ParaNMT-50M: pushing the limits of paraphrastic sentence embeddings with millions of machine translations. In: Annual Meeting of the Association for Computational Linguistics (ACL), pp. 451–462 (2018)
45.
Wieting, J., Kiela, D.: No Training required: exploring random encoders for sentence classification. In: International Conference on Learning Representations (ICLR), pp. 1–16 (2019)
46.
Yin, W., Schütze, H.: Learning word meta-embeddings. In: Annual Meeting of the Association for Computational Linguistics (ACL), pp. 1351–1360 (2016)
Metadata
Title
Enhanced Sentence Meta-Embeddings for Textual Understanding
Authors
Sourav Dutta
Haytham Assem
Copyright Year
2022
Book
Advances in Information Retrieval

Print ISBN: 978-3-030-99738-0

Electronic ISBN: 978-3-030-99739-7

Copyright Year: 2022

DOI
https://doi.org/10.1007/978-3-030-99739-7_13