Top

Published in:

01-12-2023 | Case Report

Song lyrics recommendation for social media captions using image captioning, image emotion, and caption-lyric matching via universal sentence embedding

Authors: Navapol Sanguansub, Pakawut Kamolrungwarakul, Supisara Poopair, Kantima Techaphonprasit, Thitirat Siriborvornratanakul

Published in: Social Network Analysis and Mining | Issue 1/2023

Activate our intelligent search to find suitable subject content or patents.

search-config

AI-assisted search

Off

Abstract

The article introduces a cutting-edge system that recommends song lyrics for social media captions by analyzing images. It leverages image captioning, emotion recognition, and advanced NLP techniques to match lyrics with images effectively. The system is designed to enhance the emotional impact and meaning of social media posts by suggesting appropriate song sections as captions. The proposed method combines deep learning-based image captioning, object detection, and sentiment analysis to provide users with relevant lyric options. The article also discusses related works, implementation details, and future research directions, highlighting the innovative approach of the proposed system.

AI Generated

This summary of the content was generated with the help of AI.

previous article Enhancing decision-making support by mining social media data with social network analysis

Dont have a licence yet? Then find out more about our products and how to get one now:

Springer Professional "Wirtschaft+Technik"

Online-Abonnement

Mit Springer Professional "Wirtschaft+Technik" erhalten Sie Zugriff auf:

über 102.000 Bücher
über 537 Zeitschriften

aus folgenden Fachgebieten:

Automobil + Motoren
Bauwesen + Immobilien
Business IT + Informatik
Elektrotechnik + Elektronik
Energie + Nachhaltigkeit
Finance + Banking
Management + Führung
Marketing + Vertrieb
Maschinenbau + Werkstoffe
Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

inform now

Springer Professional "Wirtschaft"

Online-Abonnement

Mit Springer Professional "Wirtschaft" erhalten Sie Zugriff auf:

über 67.000 Bücher
über 340 Zeitschriften

aus folgenden Fachgebieten:

Bauwesen + Immobilien
Business IT + Informatik
Finance + Banking
Management + Führung
Marketing + Vertrieb
Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

inform now

Springer Professional "Technik"

Online-Abonnement

Mit Springer Professional "Technik" erhalten Sie Zugriff auf:

über 67.000 Bücher
über 390 Zeitschriften

aus folgenden Fachgebieten:

Automobil + Motoren
Bauwesen + Immobilien
Business IT + Informatik
Elektrotechnik + Elektronik
Energie + Nachhaltigkeit
Maschinenbau + Werkstoffe

Jetzt Wissensvorsprung sichern!

inform now

Baijal A, Agarwal V, Hyun D (2021) Analyzing Images for music recommendation. In:IEEE international conference on consumer electronics (ICCE), Las Vegas, NV, USA, 10–12 January 2021. https://doi.org/10.1109/ICCE50685.2021.9427619

Chen M, Zhang L, Allebach JP (2015) Learning deep features for image emotion classification. In: IEEE international conference on image processing (ICIP), Quebec City, QC, Canada, 27–30 September 2015. https://doi.org/10.1109/ICIP.2015.7351656

Girdhar R, El-Nouby A, Liu Z, Singh M, Alwala KV, Joulin A, Misra I (2023) ImageBind: one embedding space to bind them all. In:IEEE/CVF conference on computer vision and pattern recognition (CVPR), pp 15180–15190, Vancouver, Canada, 18–22 June 2023

Gupta V, Jeevaraj S, Kumar S (2021) Songs recommendation using Context-based Semantic Similarity between Lyrics. In: IEEE India Council International Subsections Conference (INDISCON), Nagpur, India, 27–29 August 2021. https://doi.org/10.1109/INDISCON53343.2021.9582158

He K, Zhang X, Ren S, Sun J (2016) Identity mappings in deep residual networks. In: European conference on computer vision (ECCV), pp 630–645, Amsterdam, Netherlands, 11–14 October 2016. https://doi.org/10.1007/978-3-319-46493-0_38

Hossain MDZ, Sohel F, Shiratuddin MF, Laga H (2019a) A comprehensive survey of deep learning for image captioning. ACM Comput Surv 51(6):1–36. https://doi.org/10.1145/3295748CrossRef

Hossain R, Sarker RKR, Mimo M, Morouf AA, Pandey B (2019b) Recommendation approach of english songs title based on latent dirichlet allocation applied on lyrics. In: IEEE international conference on electrical, computer and communication technologies (ICECCT), Coimbatore, India, 20–22 February 2019b. https://doi.org/10.1109/ICECCT.2019.8869198

Jaknamon T, Marukatat S (2022) ThaiTC: thai transformer-based image captioning. In: IEEE international joint symposium on artificial intelligence and natural language processing (iSAI-NLP), Chiang Mai, Thailand, 5–7 November 2022. https://doi.org/10.1109/iSAI-NLP56921.2022.9960246

Jocher G (2022) v6.1 - TensorRT, TensorFlow Edge TPU and OpenVINO Export and Inference. GitHub, 22-Feb-2022. Available:https://github.com/ultralytics/yolov5/releases/tag/v6.1. Accessed 10 Dec 2022

Krupa KS, Ambara G, Rai K, Choudhury S (2020) Emotion aware smart music recommender system using two level CNN. In: IEEE international conference on smart systems and inventive technology (ICSSIT), pp 1322–1327, Tirunelveli, India, 20–22 August 2020. https://doi.org/10.1109/ICSSIT48917.2020.9214164

Li X, Hu D, Lu X (2017) Image2song: song retrieval via bridging image content and lyric words. In:IEEE international conference on computer vision (ICCV), pp 5649–5658, Venice, Italy, 22–29 October 2017. https://doi.org/10.1109/ICCV.2017.602

Marouf AA, Hossian R (2019) Lyricist identification using stylometric features utilizing BanglaMusicStylo dataset. In: IEEE international conference on bangla speech and language processing (ICBSLP), Sylhet, Bangladesh, 27–28 September 2019. https://doi.org/10.1109/ICBSLP47725.2019.201534

Mookdarsanit P, Mookdarsanit L (2020) Thai-IC: thai image captioning based on CNN-RNN architecture. Int J Appl Comput Technol Inf Syst10(1)

OpenAI (2023) GPT-4 technical report. ArXiv, 27-Mar-2023. Available: https://arxiv.org/abs/2303.08774. Accessed 9 Jun 2023

Panda R, Zhang J, Li H, Lee J-Y, Lu X, Roy-Chowdhury AK (2018) Contemplating Visual emotions: understanding and overcoming dataset bias. In:European conference on computer vision (ECCV), pp 594–612, Munich, Germany, 8–14 September 2018. https://doi.org/10.1007/978-3-030-01216-8_36

Pennington J, Socher R, Manning C (2014) Glove: global vectors for word representation. In: Proceedings of the conference on empirical methods in natural language processing (EMNLP),pp 1532–1543, Doha, Qatar, 25–29 October 2014. http://dx.doi.org/https://doi.org/10.3115/v1/D14-1162

Revathy VR, Pillai AS, Daneshfar F (2023) LyEmoBERT: classification of lyrics’ emotion and recommendation using a pre-trained model. Procedia Comput Sci 218:1196–1208. https://doi.org/10.1016/j.procs.2023.01.098CrossRef

Simonyan K, Zisserman A (2015) Very deep convolutional networks for large-scale image recognition.In: International conference on learning representations (ICLR), San Diego, CA, USA, 7–9 May 2015

Szegedy C, Vanhoucke V, Ioffe S, Shlens J, Wojna Z (2016) Rethinking the inception architecture for Computer Vision. In: IEEE conference on computer vision and pattern recognition (CVPR), Las Vegas, NV, USA, 27–30 June 2016. https://doi.org/10.1109/CVPR.2016.308

Wang T, Deng X (2022) User characteristics, social media use, and fatigue during the coronavirus pandemic: a stressor-strain-outcome framework. Comput Hum Behav Rep 7:1196–1208. https://doi.org/10.1016/j.chbr.2022.100218CrossRef

Yang Z, Zhang Y-J, ur Rehman S, Huang Y (2017) Image captioning with object detection and localization.arXiv [cs.CV], 2017. https://arxiv.org/abs/1706.02430

Yang Y, Cer D, Ahmad A, Guo M, Law J, Constant N, Abrego GH, Yuan S, Tar C, Sung Y, Strope B, Kurzweil R (2020) Multilingual universal sentence encoder for semantic retrieval. In: Proceedings of the annual meeting of the association for computational linguistics: system demonstrations, pp 87–94, ACL, 2020. http://dx.doi.org/https://doi.org/10.18653/v1/2020.acl-demos.12

Yan J, Xie Y, Luan X, Guo Y, Gong Q, Feng S (2022) Caption TLSTMs: combining transformer with LSTMs for image captioning. Int J Multimedia Inf Retrieval 11:111–121. https://doi.org/10.1007/s13735-022-00228-7CrossRef

Title: Song lyrics recommendation for social media captions using image captioning, image emotion, and caption-lyric matching via universal sentence embedding
Authors: Navapol Sanguansub
Pakawut Kamolrungwarakul
Supisara Poopair
Kantima Techaphonprasit
Thitirat Siriborvornratanakul
Publication date: 01-12-2023
Publisher: Springer Vienna
Published in: Social Network Analysis and Mining / Issue 1/2023
Print ISSN: 1869-5450
Electronic ISSN: 1869-5469
DOI: https://doi.org/10.1007/s13278-023-01097-6

Deep learning for Arabic healthcare: MedicalBot

Open Access
Original Article

BERT-deep CNN: state of the art for sentiment analysis of COVID-19 tweets

Original Article

A comparison of text preprocessing techniques for hate and offensive speech detection in Twitter

Original Article

Performance analysis of annotation detection techniques for cyber-bullying messages using word-embedded deep neural networks

Original Article

Spectral approach to localize information spread in a network using Rao’s metaheuristic variants

Original Article

Locating community smells in software development processes using higher-order network centralities

Open Access
Original Article

Springer Professional

Song lyrics recommendation for social media captions using image captioning, image emotion, and caption-lyric matching via universal sentence embedding

Abstract

Abstract

Dont have a licence yet? Then find out more about our products and how to get one now:

Springer Professional "Wirtschaft+Technik"

Springer Professional "Wirtschaft"

Springer Professional "Technik"

Other articles of this Issue 1/2023

Deep learning for Arabic healthcare: MedicalBot

BERT-deep CNN: state of the art for sentiment analysis of COVID-19 tweets

A comparison of text preprocessing techniques for hate and offensive speech detection in Twitter

Performance analysis of annotation detection techniques for cyber-bullying messages using word-embedded deep neural networks

Spectral approach to localize information spread in a network using Rao’s metaheuristic variants

Locating community smells in software development processes using higher-order network centralities

Premium Partner

Springer Professional

Abstract

Please log in to get access to your license.

Dont have a licence yet? Then find out more about our products and how to get one now:

Springer Professional "Wirtschaft+Technik"

Springer Professional "Wirtschaft"

Springer Professional "Technik"

Deep learning for Arabic healthcare: MedicalBot

BERT-deep CNN: state of the art for sentiment analysis of COVID-19 tweets

A comparison of text preprocessing techniques for hate and offensive speech detection in Twitter

Performance analysis of annotation detection techniques for cyber-bullying messages using word-embedded deep neural networks

Spectral approach to localize information spread in a network using Rao’s metaheuristic variants

Locating community smells in software development processes using higher-order network centralities