Top

Published in:

2024 | OriginalPaper | Chapter

A Second Look on BASS – Boosting Abstractive Summarization with Unified Semantic Graphs

A Replication Study

Authors : Osman Alperen Koraş, Jörg Schlötterer, Christin Seifert

Published in: Advances in Information Retrieval

Publisher: Springer Nature Switzerland

Activate our intelligent search to find suitable subject content or patents.

search-config

AI-assisted search

Off

Abstract

We present a detailed replication study of the BASS framework, an abstractive summarization system based on the notion of Unified Semantic Graphs. Our investigation includes challenges in replicating key components and an ablation study to systematically isolate error sources rooted in replicating novel components. Our findings reveal discrepancies in performance compared to the original work. We highlight the significance of paying careful attention even to reasonably omitted details for replicating advanced frameworks like BASS, and emphasize key practices for writing replicable papers.

Dont have a licence yet? Then find out more about our products and how to get one now:

Springer Professional "Wirtschaft+Technik"

Online-Abonnement

Mit Springer Professional "Wirtschaft+Technik" erhalten Sie Zugriff auf:

über 102.000 Bücher
über 537 Zeitschriften

aus folgenden Fachgebieten:

Automobil + Motoren
Bauwesen + Immobilien
Business IT + Informatik
Elektrotechnik + Elektronik
Energie + Nachhaltigkeit
Finance + Banking
Management + Führung
Marketing + Vertrieb
Maschinenbau + Werkstoffe
Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

inform now

Springer Professional "Technik"

Online-Abonnement

Mit Springer Professional "Technik" erhalten Sie Zugriff auf:

über 67.000 Bücher
über 390 Zeitschriften

aus folgenden Fachgebieten:

Automobil + Motoren
Bauwesen + Immobilien
Business IT + Informatik
Elektrotechnik + Elektronik
Energie + Nachhaltigkeit
Maschinenbau + Werkstoffe

Jetzt Wissensvorsprung sichern!

inform now

Springer Professional "Wirtschaft"

Online-Abonnement

Mit Springer Professional "Wirtschaft" erhalten Sie Zugriff auf:

über 67.000 Bücher
über 340 Zeitschriften

aus folgenden Fachgebieten:

Bauwesen + Immobilien
Business IT + Informatik
Finance + Banking
Management + Führung
Marketing + Vertrieb
Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

inform now

previous chapter Weighted AUReC: Handling Skew in Shard Map Quality Estimation for Selective Search

next chapter Performance Comparison of Session-Based Recommendation Algorithms Based on GNNs

Available only for authorised users

https://github.com/osmalpkoras/bass-replication.

https://static-content.springer.com/image/chp%3A10.1007%2F978-3-031-56066-8_11/MediaObjects/562556_1_En_11_Figbh_HTML.gif

https://static-content.springer.com/image/chp%3A10.1007%2F978-3-031-56066-8_11/MediaObjects/562556_1_En_11_Figbi_HTML.gif

https://static-content.springer.com/image/chp%3A10.1007%2F978-3-031-56066-8_11/MediaObjects/562556_1_En_11_Figbj_HTML.gif

We excluded less than 0.015% of the documents, having no effect on the final score.

While the BASS paper reports sentence-level R-L scores, they systematically match better with our summary-level R-L\(_{sum}\) scores, which may indicate that prior results are actually R-L\(_{sum}\) scores. Hence we place the scores reported by BASS between columns and compare them with our R-L\(_{sum}\) scores.

We did not train further, because we already doubled the computational budget used for the original paper.

The loss curve is decreasing by 0.003 points every 10,000 steps in the range of 300,000 to 450,000 steps, and by 0.002 points every 10,000 steps in the range of 450,000 to 600,000 steps at average.

Belz, A., Agarwal, S., Shimorina, A., Reiter, E.: A systematic review of reproducibility research in natural language processing. In: Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pp. 381–393. Association for Computational Linguistics, Online (2021). https://doi.org/10.18653/v1/2021.eacl-main.29

Belz, A., Thomson, C., Reiter, E., Mille, S.: Non-repeatable experiments and non-reproducible results: the reproducibility crisis in human evaluation in NLP. In: Findings of the Association for Computational Linguistics: ACL 2023, Toronto, Canada, pp. 3676–3687. Association for Computational Linguistics (2023). https://doi.org/10.18653/v1/2023.findings-acl.226. https://aclanthology.org/2023.findings-acl.226

Brody, S., Alon, U., Yahav, E.: How attentive are graph attention networks? In: International Conference on Learning Representations (2022). https://openreview.net/forum?id=F72ximsx7C1

Devlin, J., Chang, M.W., Lee, K., Toutanova, K.: BERT: pre-training of deep bidirectional transformers for language understanding. In: Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Minneapolis, Minnesota (Volume 1: Long and Short Papers), pp. 4171–4186. Association for Computational Linguistics (2019). https://doi.org/10.18653/v1/N19-1423

Dong, L., Xu, S., Xu, B.: Speech-transformer: a no-recurrence sequence-to-sequence model for speech recognition. In: 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 5884–5888 (2018). https://doi.org/10.1109/ICASSP.2018.8462506

Dosovitskiy, A., et al.: An image is worth \(16 \times 16\) words: transformers for image recognition at scale. arXiv abs/2010.11929 (2020)

Dou, Z.Y., Liu, P., Hayashi, H., Jiang, Z., Neubig, G.: GSum: a general framework for guided neural abstractive summarization. In: Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 4830–4842. Association for Computational Linguistics, Online (2021). https://doi.org/10.18653/v1/2021.naacl-main.384

Dozat, T., Manning, C.D.: Simpler but more accurate semantic dependency parsing. In: Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, Melbourne, Australia (Volume 2: Short Papers), pp. 484–490. Association for Computational Linguistics (2018). https://doi.org/10.18653/v1/P18-2077

Dror, R., Baumer, G., Shlomov, S., Reichart, R.: The hitchhiker’s guide to testing statistical significance in natural language processing. In: Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, Melbourne, Australia (Volume 1: Long Papers), pp. 1383–1392. Association for Computational Linguistics (2018). https://doi.org/10.18653/v1/P18-1128

10.

El-Kassas, W.S., Salama, C.R., Rafea, A.A., Mohamed, H.K.: Automatic text summarization: a comprehensive survey. Expert Syst. Appl. 165, 113679 (2021). https://doi.org/10.1016/j.eswa.2020.113679CrossRef

11.

Fan, A., Gardent, C., Braud, C., Bordes, A.: Using local knowledge graph construction to scale Seq2Seq models to multi-document inputs. In: Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), Hong Kong, China, pp. 4186–4196. Association for Computational Linguistics (2019). https://doi.org/10.18653/v1/D19-1428

12.

Gibney, E.: Could machine learning fuel a reproducibility crisis in science? Nature 608, 250–251 (2022). https://api.semanticscholar.org/CorpusID:251102207

13.

Google LLC: Rouge-score. https://pypi.org/project/rouge-score

14.

Gundersen, O.E., Coakley, K., Kirkpatrick, C.R.: Sources of irreproducibility in machine learning: a review. arXiv abs/2204.07610 (2022). https://api.semanticscholar.org/CorpusID:248227686

15.

Hu, J., et al.: Word graph guided summarization for radiology findings. In: Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pp. 4980–4990. Association for Computational Linguistics, Online (2021). https://doi.org/10.18653/v1/2021.findings-acl.441

16.

Huang, L., Wu, L., Wang, L.: Knowledge graph-augmented abstractive summarization with semantic-driven cloze reward. In: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 5094–5107. Association for Computational Linguistics, Online (2020). https://doi.org/10.18653/v1/2020.acl-main.457

17.

Jin, H., Wang, T., Wan, X.: Semsum: semantic dependency guided neural abstractive summarization. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 34, no. 05, pp. 8026–8033 (2020). https://doi.org/10.1609/aaai.v34i05.6312. https://ojs.aaai.org/index.php/AAAI/article/view/6312

18.

Klicpera, J., Bojchevski, A., Günnemann, S.: Predict then propagate: graph neural networks meet personalized pagerank. In: International Conference on Learning Representations (2018)

19.

Lewis, M., et al.: BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. In: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 7871–7880. Association for Computational Linguistics, Online (2020). https://doi.org/10.18653/v1/2020.acl-main.703

20.

Li, H., Peng, Q., Mou, X., Wang, Y., Zeng, Z., Bashir, M.F.: Abstractive financial news summarization via transformer-bilstm encoder and graph attention-based decoder. IEEE/ACM Trans. Audio Speech Lang. Process. 31, 3190–3205 (2023). https://doi.org/10.1109/TASLP.2023.3304473CrossRef

21.

Liu, Y., Lapata, M.: Text summarization with pretrained encoders. In: Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, Hong Kong, China (EMNLP-IJCNLP), pp. 3730–3740. Association for Computational Linguistics (2019). https://doi.org/10.18653/v1/D19-1387

22.

Manning, C.D., Surdeanu, M., Bauer, J., Finkel, J., Bethard, S.J., McClosky, D.: The Stanford CoreNLP natural language processing toolkit. In: Association for Computational Linguistics (ACL) System Demonstrations, pp. 55–60 (2014). https://www.aclweb.org/anthology/P/P14/P14-5010

23.

Paulus, R., Xiong, C., Socher, R.: A deep reinforced model for abstractive summarization (2017)

24.

Qi, P., Huang, Z., Sun, Y., Luo, H.: A knowledge graph-based abstractive model integrating semantic and structural information for summarizing Chinese meetings. In: 2022 IEEE 25th International Conference on Computer Supported Cooperative Work in Design (CSCWD), pp. 746–751 (2022). https://doi.org/10.1109/CSCWD54268.2022.9776298

25.

Raffel, C., et al.: Exploring the limits of transfer learning with a unified text-to-text transformer. J. Mach. Learn. Res. 21(1) (2020)

26.

Sharma, E., Li, C., Wang, L.: BIGPATENT: a large-scale dataset for abstractive and coherent summarization. In: Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, Florence, Italy, pp. 2204–2213. Association for Computational Linguistics (2019). https://doi.org/10.18653/v1/P19-1212

27.

Vaswani, A., et al.: Attention is all you need. In: Proceedings of the 31st International Conference on Neural Information Processing Systems, NIPS 2017, pp. 6000–6010. Curran Associates Inc., Red Hook (2017)

28.

Velickovic, P., Cucurull, G., Casanova, A., Romero, A., Lio’, P., Bengio, Y.: Graph attention networks. arXiv abs/1710.10903 (2017)

29.

Wolf, T., et al.: Transformers: state-of-the-art natural language processing. In: Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pp. 38–45. Association for Computational Linguistics, Online (2020). https://doi.org/10.18653/v1/2020.emnlp-demos.6

30.

Wu, W., et al.: BASS: boosting abstractive summarization with unified semantic graph. In: Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pp. 6052–6067. Association for Computational Linguistics, Online (2021). https://doi.org/10.18653/v1/2021.acl-long.472

31.

Wu, Y., et al.: Google’s neural machine translation system: bridging the gap between human and machine translation. CoRR abs/1609.08144 (2016). https://arxiv.org/abs/1609.08144

32.

Xu, J., Gan, Z., Cheng, Y., Liu, J.: Discourse-aware neural extractive text summarization. In: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 5021–5031. Association for Computational Linguistics, Online (2020). https://doi.org/10.18653/v1/2020.acl-main.451

33.

Ying, C., et al.: Do transformers really perform bad for graph representation? In: Neural Information Processing Systems (2021)

34.

Zhang, J., Zhao, Y., Saleh, M., Liu, P.J.: Pegasus: pre-training with extracted gap-sentences for abstractive summarization. In: Proceedings of the 37th International Conference on Machine Learning, ICML 2020. JMLR.org (2020)

35.

Zhang, T., Kishore, V., Wu, F., Weinberger, K.Q., Artzi, Y.: BERTScore: evaluating text generation with BERT. In: International Conference on Learning Representations (2020). https://openreview.net/forum?id=SkeHuCVFDr

36.

Zhu, C., et al.: Enhancing factual consistency of abstractive summarization. In: Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 718–733. Association for Computational Linguistics, Online (2021). https://doi.org/10.18653/v1/2021.naacl-main.58

37.

Zhuang, L., Wayne, L., Ya, S., Jun, Z.: A robustly optimized BERT pre-training approach with post-training. In: Proceedings of the 20th Chinese National Conference on Computational Linguistics, Huhhot, China, pp. 1218–1227. Chinese Information Processing Society of China (2021)

Title: A Second Look on BASS – Boosting Abstractive Summarization with Unified Semantic Graphs
Authors: Osman Alperen Koraş
Jörg Schlötterer
Christin Seifert
Publisher: Springer Nature Switzerland
Book: Advances in Information Retrieval
Print ISBN: 978-3-031-56065-1

Electronic ISBN: 978-3-031-56066-8

Copyright Year: 2024
DOI: https://doi.org/10.1007/978-3-031-56066-8_11

Springer Professional

Abstract

Please log in to get access to your license.

Dont have a licence yet? Then find out more about our products and how to get one now:

Springer Professional "Wirtschaft+Technik"

Springer Professional "Technik"

Springer Professional "Wirtschaft"

Premium Partner