Top

Published in:

2023 | OriginalPaper | Chapter

Visconde: Multi-document QA with GPT-3 and Neural Reranking

Authors : Jayr Pereira, Robson Fidalgo, Roberto Lotufo, Rodrigo Nogueira

Published in: Advances in Information Retrieval

Publisher: Springer Nature Switzerland

Activate our intelligent search to find suitable subject content or patents.

search-config

AI-assisted search

Off

Abstract

This paper proposes a question-answering system that can answer questions whose supporting evidence is spread over multiple (potentially long) documents. The system, called Visconde, uses a three-step pipeline to perform the task: decompose, retrieve, and aggregate. The first step decomposes the question into simpler questions using a few-shot large language model (LLM). Then, a state-of-the-art search engine is used to retrieve candidate passages from a large collection for each decomposed question. In the final step, we use the LLM in a few-shot setting to aggregate the contents of the passages into the final answer. The system is evaluated on three datasets: IIRC, Qasper, and StrategyQA. Results suggest that current retrievers are the main bottleneck and that readers are already performing at the human level as long as relevant passages are provided. The system is also shown to be more effective when the model is induced to give explanations before answering a question. Code is available at https://github.com/neuralmind-ai/visconde.

Dont have a licence yet? Then find out more about our products and how to get one now:

Springer Professional "Wirtschaft+Technik"

Online-Abonnement

Mit Springer Professional "Wirtschaft+Technik" erhalten Sie Zugriff auf:

über 102.000 Bücher
über 537 Zeitschriften

aus folgenden Fachgebieten:

Automobil + Motoren
Bauwesen + Immobilien
Business IT + Informatik
Elektrotechnik + Elektronik
Energie + Nachhaltigkeit
Finance + Banking
Management + Führung
Marketing + Vertrieb
Maschinenbau + Werkstoffe
Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

inform now

Springer Professional "Technik"

Online-Abonnement

Mit Springer Professional "Technik" erhalten Sie Zugriff auf:

über 67.000 Bücher
über 390 Zeitschriften

aus folgenden Fachgebieten:

Automobil + Motoren
Bauwesen + Immobilien
Business IT + Informatik
Elektrotechnik + Elektronik
Energie + Nachhaltigkeit
Maschinenbau + Werkstoffe

Jetzt Wissensvorsprung sichern!

inform now

Springer Professional "Wirtschaft"

Online-Abonnement

Mit Springer Professional "Wirtschaft" erhalten Sie Zugriff auf:

über 67.000 Bücher
über 340 Zeitschriften

aus folgenden Fachgebieten:

Bauwesen + Immobilien
Business IT + Informatik
Finance + Banking
Management + Führung
Marketing + Vertrieb
Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

inform now

previous chapter Where a Little Change Makes a Big Difference: A Preliminary Exploration of Children’s Queries

next chapter Towards Detecting Interesting Ideas Expressed in Text

The name is a homage to Visconde de Sabugosa a fictional character invented by Monteiro Lobato that is a corn cob doll whose wisdom comes from reading books.

We used the 3 billion parameters version, whose checkpoint is available at https://huggingface.co/castorini/monot5-3b-msmarco-10k.

We used this model as our sentence encoder: sentence-transformers/msmarco-bert-base-dot-v5.

https://leaderboard.allenai.org/strategyqa/submissions/public. Accessed on July 20, 2022.

Boerschinger, B., et al.: Boosting search engines with interactive agents. Transactions on Machine Learning Research (2022). https://openreview.net/pdf?id=0ZbPmmB61g

Brown, T., et al.: Language models are few-shot learners. In: Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M., Lin, H. (eds.) Advances in Neural Information Processing Systems, vol. 33, pp. 1877–1901. Curran Associates, Inc. (2020). https://proceedings.neurips.cc/paper/2020/file/1457c0d6bfcb4967418bfb8ac142f64a-Paper.pdf

Creswell, A., Shanahan, M.: Faithful reasoning using large language models. arXiv preprint arXiv:2208.14271 (2022)

Das, R., Dhuliawala, S., Zaheer, M., McCallum, A.: Multi-step retriever-reader interaction for scalable open-domain question answering (2019). https://doi.org/10.48550/ARXIV.1905.05733, https://arxiv.org/abs/1905.05733

Dasigi, P., Lo, K., Beltagy, I., Cohan, A., Smith, N.A., Gardner, M.: A dataset of information-seeking questions and answers anchored in research papers. In: Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 4599–4610. Association for Computational Linguistics, June 2021. https://doi.org/10.18653/v1/2021.naacl-main.365, https://aclanthology.org/2021.naacl-main.365

Feldman, Y., El-Yaniv, R.: Multi-hop paragraph retrieval for open-domain question answering. In: Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. pp. 2296–2309. Association for Computational Linguistics, Florence, Italy, July 2019. https://doi.org/10.18653/v1/P19-1222, https://aclanthology.org/P19-1222

Ferguson, J., Gardner, M., Hajishirzi, H., Khot, T., Dasigi, P.: IIRC: a dataset of incomplete information reading comprehension questions. in: proceedings of the 2020 conference on empirical methods in Natural Language Processing (EMNLP), pp. 1137–1147. Association for Computational Linguistics, November 2020. https://doi.org/10.18653/v1/2020.emnlp-main.86, https://aclanthology.org/2020.emnlp-main.86

Ferguson, J., Hajishirzi, H., Dasigi, P., Khot, T.: Retrieval data augmentation informed by downstream question answering performance. In: Proceedings of the Fifth Fact Extraction and VERification Workshop (FEVER), pp. 1–5. Association for Computational Linguistics, Dublin, Ireland, May 2022. https://doi.org/10.18653/v1/2022.fever-1.1, https://aclanthology.org/2022.fever-1.1

Geva, M., Khashabi, D., Segal, E., Khot, T., Roth, D., Berant, J.: Did Aristotle use a laptop? A question answering benchmark with implicit reasoning strategies. Trans. Assoc. Comput. Linguist. 9, 346–361 (2021). https://doi.org/10.1162/tacl_a_00370, https://doi.org/10.1162/tacl_a_00370

10.

Guo, M., et al.: Longt5: efficient text-to-text transformer for long sequences (2021). https://doi.org/10.48550/ARXIV.2112.07916, https://arxiv.org/abs/2112.07916

11.

Huebscher, M.C., Buck, C., Ciaramita, M., Rothe, S.: Zero-shot retrieval with search agents and hybrid environments (2022). https://doi.org/10.48550/ARXIV.2209.15469, https://arxiv.org/abs/2209.15469

12.

Izacard, G., Grave, E.: Leveraging passage retrieval with generative models for open domain question answering. In: Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pp. 874–880. Association for Computational Linguistics, April 2021. https://doi.org/10.18653/v1/2021.eacl-main.74, https://aclanthology.org/2021.eacl-main.74

13.

Karpukhin, V., et al.: Dense passage retrieval for open-domain question answering. In: Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 6769–6781. Association for Computational Linguistics, November 2020. https://doi.org/10.18653/v1/2020.emnlp-main.550, https://aclanthology.org/2020.emnlp-main.550

14.

Khashabi, D., et al.: Unifiedqa: crossing format boundaries with a single QA system (2020)

15.

Khashabi, D., Kordi, Y., Hajishirzi, H.: Unifiedqa-v2: stronger generalization via broader cross-format training. arXiv preprint arXiv:2202.12359 (2022)

16.

Kojima, T., Gu, S.S., Reid, M., Matsuo, Y., Iwasawa, Y.: Large language models are zero-shot reasoners (2022). https://doi.org/10.48550/ARXIV.2205.11916, https://arxiv.org/abs/2205.11916

17.

Lazaridou, A., Gribovskaya, E., Stokowiec, W., Grigorev, N.: Internet-augmented language models through few-shot prompting for open-domain question answering. arXiv preprint arXiv:2203.05115 (2022)

18.

Lewis, P., et al.: Retrieval-augmented generation for knowledge-intensive NLP tasks. In: Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M., Lin, H. (eds.) Advances in Neural Information Processing Systems, vol. 33, pp. 9459–9474. Curran Associates, Inc. (2020). https://proceedings.neurips.cc/paper/2020/file/6b493230205f780e1bc26945df7481e5-Paper.pdf

19.

Lin, J., Ma, X., Lin, S.C., Yang, J.H., Pradeep, R., Nogueira, R.: Pyserini: a python toolkit for reproducible information retrieval research with sparse and dense representations. In: Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 2356–2362. SIGIR 2021, Association for Computing Machinery, New York, NY, USA (2021). https://doi.org/10.1145/3404835.3463238

20.

Liu, J., Shen, D., Zhang, Y., Dolan, B., Carin, L., Chen, W.: What makes good in-context examples for gpt-\(3\)? (2021). https://doi.org/10.48550/ARXIV.2101.06804, https://arxiv.org/abs/2101.06804

21.

Nakano, R., et al.: Webgpt: browser-assisted question-answering with human feedback (2021). https://doi.org/10.48550/ARXIV.2112.09332, https://arxiv.org/abs/2112.09332

22.

Nogueira, R., Jiang, Z., Pradeep, R., Lin, J.: Document ranking with a pretrained sequence-to-sequence model. In: Findings of the Association for Computational Linguistics: EMNLP 2020, pp. 708–718. Association for Computational Linguistics, November 2020. https://doi.org/10.18653/v1/2020.findings-emnlp.63, https://aclanthology.org/2020.findings-emnlp.63

23.

Perez, E., Lewis, P., Yih, W.T., Cho, K., Kiela, D.: Unsupervised question decomposition for question answering. In: Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 8864–8880 (2020)

24.

Press, O., Zhang, M., Min, S., Schmidt, L., Smith, N.A., Lewis, M.: Measuring and narrowing the compositionality gap in language models. arXiv preprint arXiv:2210.03350 (2022)

25.

Raffel, C., et al.: Exploring the limits of transfer learning with a unified text-to-text transformer. J. Mach. Learn. Res. 21(140), 1–67 (2020)MathSciNetMATH

26.

Reimers, N., Gurevych, I.: Sentence-bert: sentence embeddings using siamese bert-networks (2019). https://doi.org/10.48550/ARXIV.1908.10084, https://arxiv.org/abs/1908.10084

27.

Robertson, S.E., Walker, S., Jones, S., Hancock-Beaulieu, M., Gatford, M.: Okapi at trec-3. In: TREC (1994)

28.

Sachan, D.S., Lewis, M., Yogatama, D., Zettlemoyer, L., Pineau, J., Zaheer, M.: Questions are all you need to train a dense passage retriever. arXiv preprint arXiv:2206.10658 (2022)

29.

Tay, Y., et al.: Unifying language learning paradigms (2022). https://doi.org/10.48550/ARXIV.2205.05131, https://arxiv.org/abs/2205.05131

30.

Trivedi, H., Balasubramanian, N., Khot, T., Sabharwal, A.: Teaching broad reasoning skills via decomposition-guided contexts (2022). https://doi.org/10.48550/ARXIV.2205.12496, https://arxiv.org/abs/2205.12496

31.

Wang, X., et al.: Self-consistency improves chain of thought reasoning in language models (2022). https://doi.org/10.48550/ARXIV.2203.11171, https://arxiv.org/abs/2203.11171

32.

Wei, J., et al.: Chain of thought prompting elicits reasoning in large language models (2022). https://doi.org/10.48550/ARXIV.2201.11903, https://arxiv.org/abs/2201.11903

33.

Xiong, W., Gupta, A., Toshniwal, S., Mehdad, Y., Yih, W.T.: Adapting pretrained text-to-text models for long text sequences (2022). https://doi.org/10.48550/ARXIV.2209.10052, https://arxiv.org/abs/2209.10052

34.

Xiong, W., et al.: Answering complex open-domain questions with multi-hop dense retrieval (2020). https://doi.org/10.48550/ARXIV.2009.12756, https://arxiv.org/abs/2009.12756

35.

Xu, W., Napoles, C., Pavlick, E., Chen, Q., Callison-Burch, C.: Optimizing statistical machine translation for text simplification. Trans. Assoc. Comput. Linguist. 4, 401–415 (2016). https://doi.org/10.1162/tacl_a_00107, https://doi.org/10.1162/tacl_a_00107

36.

Yang, Z., et al.: Hotpotqa: a dataset for diverse, explainable multi-hop question answering. In: Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 2369–2380 (2018)

37.

Zhu, F., Lei, W., Wang, C., Zheng, J., Poria, S., Chua, T.S.: Retrieving and reading: a comprehensive survey on open-domain question answering (2021). https://doi.org/10.48550/ARXIV.2101.00774, https://arxiv.org/abs/2101.00774

Title: Visconde: Multi-document QA with GPT-3 and Neural Reranking
Authors: Jayr Pereira
Robson Fidalgo
Roberto Lotufo
Rodrigo Nogueira
Publisher: Springer Nature Switzerland
Book: Advances in Information Retrieval
Print ISBN: 978-3-031-28237-9

Electronic ISBN: 978-3-031-28238-6

Copyright Year: 2023
DOI: https://doi.org/10.1007/978-3-031-28238-6_44

Springer Professional

Abstract

Please log in to get access to your license.

Dont have a licence yet? Then find out more about our products and how to get one now:

Springer Professional "Wirtschaft+Technik"

Springer Professional "Technik"

Springer Professional "Wirtschaft"