Skip to main content
SpringerLink
Log in

A deep multi-modal neural network for informative Twitter content classification during emergencies

  • S.I. : Design and Management of Humanitarian Supply Chains
  • Published:
Annals of Operations Research Aims and scope Submit manuscript

Abstract

People start posting tweets containing texts, images, and videos as soon as a disaster hits an area. The analysis of these disaster-related tweet texts, images, and videos can help humanitarian response organizations in better decision-making and prioritizing their tasks. Finding the informative contents which can help in decision making out of the massive volume of Twitter content is a difficult task and require a system to filter out the informative contents. In this paper, we present a multi-modal approach to identify disaster-related informative content from the Twitter streams using text and images together. Our approach is based on long-short-term-memory and VGG-16 networks that show significant improvement in the performance, as evident from the validation result on seven different disaster-related datasets. The range of F1-score varied from 0.74 to 0.93 when tweet texts and images used together, whereas, in the case of only tweet text, it varies from 0.61 to 0.92. From this result, it is evident that the proposed multi-modal system is performing significantly well in identifying disaster-related informative social media contents.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11

Similar content being viewed by others

Notes

  1. http://time.com/4921961/hurricane-harvey-twitter-facebook-social-media/.

  2. It is freely available at https://nlp.stanford.edu/projects/glove/.

References

  • Aipe, A., Ekbal, A., Mukuntha, N., & Kurohashi, S. (2018). Linguistic feature assisted deep learning approach towards multi-label classification of crisis related tweets. In Proceedings of the 15th ISCRAM Conference (pp. 705–717).

  • Akter, S., & Wamba, S. F. (2017). Big data and disaster management: A systematic review and agenda for future research. Annals of Operations Research. https://doi.org/10.1007/s10479-017-2584-2.

    Article  Google Scholar 

  • Alalwan, A. A., Rana, N. P., Dwivedi, Y. K., & Algharabat, R. (2017). Social media in marketing: A review and analysis of the existing literature. Telematics and Informatics, 34(7), 1177–1190.

    Article  Google Scholar 

  • Alam, F., Imran, M., & Ofli, F. (2017). Image4act: Online social media image processing for disaster response. In Proceedings of the 2017 IEEE/ACM international conference on advances in social networks analysis and mining 2017 (pp. 601–604). ACM.

  • Alam, F., Ofli, F., & Imran, M. (2018). Crisismmd: Multimodal Twitter datasets from natural disasters. In Proceedings of the 12th international AAAI conference on web and social media (ICWSM).

  • Ashktorab, Z., Brown, C., Nandi, M., & Culotta, A. (2014). Tweedr: Mining Twitter to inform disaster response. In Proceedings of the 11th ISCRAM conference (pp. 354–358).

  • Atefeh, F., & Khreich, W. (2015). A survey of techniques for event detection in Twitter. Computational Intelligence, 31, 132–164.

    Article  Google Scholar 

  • Cameron, M. A., Power, R., Robinson, B., & Yin, J. (2012). Emergency situation awareness from Twitter for crisis management. In Proceedings of the 21st international conference on world wide web (pp. 695–698). ACM.

  • Caragea, C., Mcneese, N., Jaiswal, A., Traylor, G., Woo Kim, H., Mitra, P., et al. (2011). Classifying text messages for the Haiti earthquake. In Proceedings of the 8th international conference on information systems for crisis response and management (ISCRAM2011).

  • Caragea, C., Silvescu, A., & Tapia, A. H. (2016). Identifying informative messages in disaster events using convolutional neural networks. In International conference on information systems for crisis response and management (pp. 137–147).

  • Chaudhuri, N., & Bose, I. (2019). Application of image analytics for disaster response in smart cities. In Proceedings of the 52nd Hawaii international conference on system sciences. https://doi.org/10.24251/hicss.2019.367

  • Daly, S., & Thom, J. A. (2016). Mining and classifying image posts on social media to analyse fires. In Proceedings of the 13th ISCRAM conference (pp. 1–14).

  • Dubey, R. (2019). Developing an integration framework for crowdsourcing and internet of things with applications for disaster response. In Social entrepreneurship: Concepts, methodologies, tools, and applications (pp. 274–283). IGI Global.

  • Dubey, R., Ali, S. S., Aital, P., Venkatesh, V., et al. (2014). Mechanics of humanitarian supply chain agility and resilience and its empirical validation. International Journal of Services and Operations Management, 17, 367–384.

    Article  Google Scholar 

  • Dubey, R., Altay, N., & Blome, C. (2017). Swift trust and commitment: The missing links for humanitarian supply chain coordination? Annals of Operations Research. https://doi.org/10.1007/s10479-017-2676-z.

    Article  Google Scholar 

  • Dubey, R., Gunasekaran, A., Childe, S. J., Roubaud, D., Wamba, S. F., Giannakis, M., et al. (2019). Big data analytics and organizational culture as complements to swift trust and collaborative performance in the humanitarian supply chain. International Journal of Production Economics, 210, 120–136.

    Article  Google Scholar 

  • Dwivedi, Y. K., Kapoor, K. K., & Chen, H. (2015). Social media marketing and advertising. The Marketing Review, 15(3), 289–309.

    Article  Google Scholar 

  • Dwivedi, Y. K., Shareef, M. A., Mukerji, B., Rana, N. P., & Kapoor, K. K. (2018). Involvement in emergency supply chain for disaster management: A cognitive dissonance perspective. International Journal of Production Research, 56, 6758–6773.

    Article  Google Scholar 

  • Elbanna, A., Bunker, D., Levine, L., & Sleigh, A. (2019). Emergency management in the changing world of social media: Framing the research agenda with the stakeholders through engaged scholarship. International Journal of Information Management, 47, 112–120.

    Article  Google Scholar 

  • Graf, D., Retschitzegger, W., Schwinger, W., Pröll, B., & Kapsammer, E. (2018). Cross-domain informativeness classification for disaster situations. In Proceedings of the 10th international conference on management of digital ecosystems (pp. 183–190). ACM.

  • Guha-Sapir, D., Vos, F., Below, R., & Ponserre, S. (2012). Annual disaster statistical review 2011: The number sandtrends. Technical Report Centre for Research on the Epidemiology of Disasters (CRED).

  • Gupta, A., Lamba, H., Kumaraguru, P., & Joshi, A. (2013, May). Faking sandy: Characterizing and identifying fake images on Twitter during hurricane sandy. In Proceedings of the 22nd international conference on world wide web (pp. 729–736). ACM.

  • Hochreiter, S., & Schmidhuber, J. (1997). Long short-term memory. Neural Computation, 9, 1735–1780.

    Article  Google Scholar 

  • Huang, Q., & Xiao, Y. (2015). Geographic situational awareness: Mining tweets for disaster preparedness, emergency response, impact, and recovery. ISPRS International Journal of Geo-Information, 4, 1549–1568.

    Article  Google Scholar 

  • Imran, M., Castillo, C., Diaz, F., & Vieweg, S. (2015). Processing social media messages in mass emergency: A survey. ACM Computing Surveys (CSUR), 47, 67.

    Article  Google Scholar 

  • Imran, M., Castillo, C., Lucas, J., Meier, P., & Vieweg, S. (2014). Aidr: Artificial intelligence for disaster response. In Proceedings of the 23rd international conference on world wide web (pp. 159–162). ACM.

  • Imran, M., Elbassuoni, S., Castillo, C., Diaz, F., & Meier, P. (2013a). Extracting information nuggets from disaster-related messages in social media. In Proceedings of the 10th ISCRAM conference (pp. 791–801).

  • Imran, M., Elbassuoni, S., Castillo, C., Diaz, F., & Meier, P. (2013b). Practical extraction of disaster-relevant information from social media. In Proceedings of the 22nd international conference on world wide web (pp. 1021–1024). ACM.

  • Jabbour, C. J. C., Sobreiro, V. A., de Sousa Jabbour, A. B. L., de Souza Campos, L. M., Mariano, E. B., & Renwick, D. W. S. (2017). An analysis of the literature on humanitarian logistics and supply chain management: Paving the way for future studies. Annals of Operations Research. https://doi.org/10.1007/s10479-017-2536-x.

    Article  Google Scholar 

  • Jamali, M., Nejat, A., Ghosh, S., Jin, F., & Cao, G. (2019). Social media data and post-disaster recovery. International Journal of Information Management, 44, 25–37.

    Article  Google Scholar 

  • Jin, S., Jeong, S., Kim, J., & Kim, K. (2015). A logistics model for the transport of disaster victims with various injuries and survival probabilities. Annals of Operations Research, 230, 17–33.

    Article  Google Scholar 

  • John, L., Gurumurthy, A., Soni, G., & Jain, V. (2018). Modelling the inter-relationship between factors affecting coordination in a humanitarian supply chain: A case of Chennai flood relief. Annals of Operations Research. https://doi.org/10.1007/s10479-018-2963-3.

    Article  Google Scholar 

  • Kapoor, K. K., Tamilmani, K., Rana, N. P., Patil, P., Dwivedi, Y. K., & Nerur, S. (2018). Advances in social media research: Past, present and future. Information Systems Frontiers, 20(3), 531–558.

    Article  Google Scholar 

  • Kim, J., Bae, J., & Hastak, M. (2018). Emergency information diffusion on online social media during storm Cindy in US. International Journal of Information Management, 40, 153–165.

    Article  Google Scholar 

  • Kim, J., & Hastak, M. (2018). Social network analysis: Characteristics of online social networks after a disaster. International Journal of Information Management, 38(1), 86–96.

    Article  Google Scholar 

  • Kingma, D. P., & Ba, J. (2014). Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980.

  • Kumar, A., & Singh, J. P. (2019). Location reference identification from tweets during emergencies: A deep learning approach. International Journal of Disaster Risk Reduction, 33, 365–375.

    Article  Google Scholar 

  • Kumar, A., Singh, J. P., & Rana, N. P. (2017). Authenticity of geo-location and place name in tweets. In Proceedings of the 23rd Americas conference on information systems (AMCIS) (pp. 1–9)

  • Kumar, A., & Singh, J. P., Saumya, S. (2019). A comparative analysis of machine learning techniques for disaster related tweet classification. In IEEE region 10 humanitarian technology conference, (pp. 222–227).

  • Lagerstrom, R., Arzhaeva, Y., Szul, P., Obst, O., Power, R., Robinson, B., et al. (2016). Image classification to support emergency situation awareness. Frontiers in Robotics and AI, 3, 54.

    Article  Google Scholar 

  • Li, H., Guevara, N., Herndon, N., Caragea, D., Neppalli, K., Caragea, C., et al. (2015). Twitter mining for disaster response: A domain adaptation approach. In Proceedings of the 12th ISCRAM conference.

  • Mouzannar, H., Rizk, Y., & Awad, M. (2018). Damage identification in social media posts using multimodal deep learning. In ISCRAM.

  • Nair, V., & Hinton, G. E. (2010). Rectified linear units improve restricted Boltzmann machines. In Proceedings of the 27th international conference on machine learning (ICML-10) (pp. 807–814).

  • Nguyen, D. T., Al Mannai, K. A., Joty, S., Sajjad, H., Imran, M., & Mitra, P. (2017a). Robust classification of crisis-related data on social networks using convolutional neural networks. In 11th international conference on web and social media, ICWSM 2017 (pp. 632–635). AAAI Press.

  • Nguyen, D. T., Alam, F., Ofli, F., & Imran, M. (2017b). Automatic image filtering on social networks using deep learning and perceptual hashing during crises. arXiv preprint arXiv:1704.02602.

  • Nguyen, D. T., Joty, S., Imran, M., Sajjad, H., & Mitra, P. (2016). Applications of online deep learning for crisis response using social media information. arXiv preprint arXiv:1610.01030.

  • Nguyen, D. T., Ofli, F., Imran, M., & Mitra, P. (2017c). Damage assessment from social media imagery data during disasters. In Proceedings of the 2017 IEEE/ACM international conference on advances in social networks analysis and mining 2017 (pp. 569–576). ACM.

  • Olteanu, A., Castillo, C., Diaz, F., & Vieweg, S. (2014). Crisislex: A lexicon for collecting and filtering microblogged communications in crises. In Eighth international AAAI conference on weblogs and social media.

  • Papadopoulos, T., Gunasekaran, A., Dubey, R., Altay, N., Childe, S. J., & Fosso-Wamba, S. (2017). The role of big data in explaining disaster resilience in supply chains for sustainability. Journal of Cleaner Production, 142, 1108–1118.

    Article  Google Scholar 

  • Paul, J. A., & Hariharan, G. (2012). Location-allocation planning of stockpiles for effective disaster mitigation. Annals of Operations Research, 196, 469–490.

    Article  Google Scholar 

  • Pennington, J., Socher, R., & Manning, C. (2014). Glove: Global vectors for word representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP) (pp. 1532–1543).

  • Ragini, J. R., Anand, P. R., & Bhaskar, V. (2018). Big data analytics for disaster response and recovery through sentiment analysis. International Journal of Information Management, 42, 13–24.

    Article  Google Scholar 

  • Rizk, Y., Jomaa, H. S., Awad, M., & Castillo, C. (2019). A computationally efficient multi-modal classification approach of disaster-related Twitter images. In Proceedings of the 34th ACM/SIGAPP symposium on applied computing (pp. 2050–2059). ACM.

  • Rudra, K., Banerjee, S., Ganguly, N., Goyal, P., Imran, M., & Mitra, P. (2016). Summarizing situational tweets in crisis scenario. In Proceedings of the 27th ACM conference on hypertext and social media (pp. 137–147). ACM.

  • Sakaki, T., Okazaki, M., & Matsuo, Y. (2013). Tweet analysis for real-time event detection and earthquake reporting system development. IEEE Transactions on Knowledge and Data Engineering, 25, 919–931.

    Article  Google Scholar 

  • Shareef, M. A., Dwivedi, Y. K., Mahmud, R., Wright, A., Rahman, M. M., Kizgin, H., et al. (2018). Disaster management in Bangladesh: Developing an effective emergency supply chain network. Annals of Operations Research. https://doi.org/10.1007/s10479-018-3081-y.

    Article  Google Scholar 

  • Simonyan, K., & Zisserman, A. (2014). Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556.

  • Singh, J. P., Dwivedi, Y. K., Rana, N. P., Kumar, A., & Kapoor, K. K. (2019). Event classification and location prediction from tweets during disasters. Annals of Operations Research, 283(1–2), 737–757. https://doi.org/10.1007/s10479-017-2522-3.

    Article  Google Scholar 

  • Sinha, A., Kumar, P., Rana, N. P., Islam, R., & Dwivedi, Y. K. (2019). Impact of internet of things (IoT) in disaster management: A task-technology fit perspective. Annals of Operations Research, 283, 759–794. https://doi.org/10.1007/s10479-017-2658-1.

    Article  Google Scholar 

  • Son, J., Lee, H. K., Jin, S., & Lee, J. (2019). Content features of tweets for effective communication during disasters: A media synchronicity theory perspective. International Journal of Information Management, 45, 56–68.

    Article  Google Scholar 

  • Sundermeyer, M., Schlüter, R., & Ney, H. (2012). LSTM neural networks for language modeling. In Thirteenth annual conference of the international speech communication association.

  • Verma, S., Vieweg, S., Corvey, W. J., Palen, L., Martin, J. H., Palmer, M., et al. (2011). Natural language processing to the rescue? Extracting” situational awareness” tweets during mass emergency. In Fifth international AAAI conference on weblogs and social media.

  • Yu, M., Huang, Q., Qin, H., Scheele, C., & Yang, C. (2019). Deep learning for real-time social media text classification for situation awareness using hurricanes sandy, Harvey, and Irma as case studies. International Journal of Digital Earth. https://doi.org/10.1080/17538947.2019.1574316.

    Article  Google Scholar 

  • Zhang, C., Fan, C., Yao, W., Hu, X., & Mostafavi, A. (2019). Social media for intelligent public information and warning in disasters: An interdisciplinary review. International Journal of Information Management, 49, 190–207.

    Article  Google Scholar 

  • Zheng, X., Han, J., & Sun, A. (2018). A survey of location prediction on Twitter. IEEE Transactions on Knowledge and Data Engineering, 30, 1652–1671.

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Computer Science and Engineering, National Institute of Technology Patna, Patna, India

    Abhinav Kumar & Jyoti Prakash Singh

  2. School of Management, Emerging Markets Research Centre (EMaRC), Swansea University Bay Campus, Fabian Way, Swansea, SA1 8EN, UK

    Yogesh K. Dwivedi

  3. School of Management, University of Bradford, Richmond Rd, Bradford, BD7 1DP, UK

    Nripendra P. Rana

Authors
  1. Abhinav Kumar
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Jyoti Prakash Singh
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Yogesh K. Dwivedi
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Nripendra P. Rana
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Yogesh K. Dwivedi.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Kumar, A., Singh, J.P., Dwivedi, Y.K. et al. A deep multi-modal neural network for informative Twitter content classification during emergencies. Ann Oper Res 319, 791–822 (2022). https://doi.org/10.1007/s10479-020-03514-x

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10479-020-03514-x

Keywords

Search

Navigation