Michele Corazza
Skip Abstract Section
Abstract
The increasing popularity of social media platforms such as Twitter and Facebook has led to a rise in the presence of hate and aggressive speech on these platforms. Despite the number of approaches recently proposed in the Natural Language Processing research area for detecting these forms of abusive language, the issue of identifying hate speech at scale is still an unsolved problem. In this article, we propose a robust neural architecture that is shown to perform in a satisfactory way across different languages; namely, English, Italian, and German. We address an extensive analysis of the obtained experimental results over the three languages to gain a better understanding of the contribution of the different components employed in the system, both from the architecture point of view (i.e., Long Short Term Memory, Gated Recurrent Unit, and bidirectional Long Short Term Memory) and from the feature selection point of view (i.e., ngrams, social network–specific features, emotion lexica, emojis, word embeddings). To address such in-depth analysis, we use three freely available datasets for hate speech detection on social media in English, Italian, and German.
- Sweta Agrawal and Amit Awekar. 2018. Deep learning for detecting cyberbullying across multiple social media platforms. In Proceedings of the 40th European Conference on IR Research—Advances in Information Retrieval (ECIR’18) (Lecture Notes in Computer Science), Gabriella Pasi, Benjamin Piwowarski, Leif Azzopardi, and Allan Hanbury (Eds.), Vol. 10772. Springer, 141--153. DOI:https://doi.org/10.1007/978-3-319-76941-7_11Google Scholar
Cross Ref
- Luis Enrique Argota Vega, Jorge Carlos Reyes-Magaña, Helena Gómez-Adorno, and Gemma Bel-Enguix. 2019. Detecting hate speech in Twitter using multiple features in a combinatorial framework. In Proceedings of the 13th International Workshop on Semantic Evaluation. Association for Computational Linguistics, 447--452. DOI:https://doi.org/10.18653/v1/S19-2079Google Scholar
- Pinar Arslan, Michele Corazza, Elena Cabrio, and Serena Villata. 2019. Overwhelmed by negative emotions? Maybe you are being cyber-bullied! In Proceedings of the 34th ACM/SIGAPP Symposium on Applied Computing (SAC’19).Google Scholar
- Xiaoyu Bai, Flavio Merenda, Claudia Zaghi, Tommaso Caselli, and Malvina Nissim. 2018. Hate speech detection in Italian social media. In Proceedings of the 6th Evaluation Campaign of Natural Language Processing and Speech Tools for Italian. Final Workshop (EVALITA’18) co-located with the 5th Italian Conference on Computational Linguistics (CLiC-it’18).Google Scholar
- Xiaoyu Bai, Flavio Merenda, Claudia Zaghi, Tommaso Caselli, and Malvina Nissim. 2018. Detecting offensive speech in German social media. In Proceedings of the GermEval 2018 and 14th Conference on Natural Language Processing (KONVENS’18).Google Scholar
- Francesco Barbieri, Francesco Ronzano, and Horacio Saggion. 2016. What does this emoji mean? A vector space skip-gram model for Twitter emojis. In Proceedings of the Language Resources and Evaluation Conference (LREC’16).Google Scholar
- Valerio Basile, Cristina Bosco, Elisabetta Fersini, Debora Nozza, Viviana Patti, Francisco Manuel Rangel Pardo, Paolo Rosso, and Manuela Sanguinetti. 2019. Multilingual detection of hate speech against immigrants and women in Twitter. In Proceedings of the 13th International Workshop on Semantic Evaluation. Association for Computational Linguistics. 54--63. DOI:https://doi.org/10.18653/v1/S19-2007Google Scholar
- Valerio Basile and Malvina Nissim. 2013. Sentiment analysis on Italian tweets. In Proceedings of the 4th Workshop on Computational Approaches to Subjectivity, Sentiment and Social Media Analysis. 100--107.Google Scholar
- Elisa Bassignana, Valerio Basile, and Viviana Patti. 2018. Hurtlex: A multilingual lexicon of words to hurt. In Proceedings of the 5th Italian Conference on Computational Linguistics (CLiC-it’18), Vol. 2253. CEUR-WS, 1--6.Google ScholarCross Ref
- Christos Baziotis, Nikos Pelekis, and Christos Doulkeridis. 2017. Deep LSTM with attention for message-level and topic-based sentiment analysis. In Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval’17). Association for Computational Linguistics, 747--754.Google ScholarCross Ref
- Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. 2017. Enriching word vectors with subword information. Trans. Assoc. Computat. Ling. 5 (2017), 135--146.Google ScholarCross Ref
- Cristina Bosco, Felice Dell’Orletta, Fabio Poletto, Manuela Sanguinetti, and Maurizio Tesconi. 2018. Overview of the EVALITA 2018 hate speech detection task. In Proceedings of the 6th Evaluation Campaign of Natural Language Processing and Speech Tools for Italian. Final Workshop (EVALITA’18) co-located with the 5th Italian Conference on Computational Linguistics (CLiC-it’18).Google Scholar
- Miguel Ángel Álvarez Carmona, Estefanía Guzmán-Falcón, Manuel Montes-y-Gómez, Hugo Jair Escalante, Luis Villaseñor Pineda, Verónica Reyes-Meza, and Antonio Rico Sulayes. 2018. Authorship and aggressiveness analysis in Mexican Spanish tweets. In Proceedings of the 3rd Workshop on Evaluation of Human Language Technologies for Iberian Languages (IberEval’18) co-located with the 34th Conference of the Spanish Society for Natural Language Processing (SEPLN’18). 74--96.Google Scholar
- Kyunghyun Cho, Bart van Merrienboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014. Learning phrase representations using RNN encoder–decoder for statistical machine translation. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP’14). Association for Computational Linguistics, 1724--1734.Google ScholarCross Ref
- François Chollet et al. 2015. Keras. Retrieved from https://github.com/fchollet/keras.Google Scholar
- Mark Cieliebak, Jan Milan Deriu, Dominic Egger, and Fatih Uzdilli. 2017. A Twitter corpus and benchmark resources for German sentiment analysis. In Proceedings of the 5th International Workshop on Natural Language Processing for Social Media. Association for Computational Linguistics, 45--51.Google ScholarCross Ref
- Andrea Cimino, Lorenzo De Mattei, and Felice Dell’Orletta. 2018. Multi-task learning in deep neural networks at EVALITA 2018. In Proceedings of the 6th Evaluation Campaign of Natural Language Processing and Speech Tools for Italian. Final Workshop (EVALITA’18) co-located with the 5th Italian Conference on Computational Linguistics (CLiC-it’18).Google ScholarCross Ref
- Michele Corazza, Stefano Menini, Pinar Arslan, Rachele Sprugnoli, Elena Cabrio, Sara Tonelli, and Serena Villata. 2018. Comparing different supervised approaches to hate speech detection. In Proceedings of the 6th Evaluation Campaign of Natural Language Processing and Speech Tools for Italian. Final Workshop (EVALITA’18) co-located with the 5th Italian Conference on Computational Linguistics (CLiC-it’18).Google ScholarCross Ref
- Michele Corazza, Stefano Menini, Pinar Arslan, Rachele Sprugnoli, Elena Cabrio, Sara Tonelli, and Serena Villata. 2018. Identifying offensive tweets using recurrent neural networks. In Proceedings of the GermEval Workshop.Google Scholar
- Michele Corazza, Stefano Menini, Elena Cabrio, Sara Tonelli, and Serena Villata. 2019. Cross-platform evaluation for Italian hate speech detection. In Proceedings of the 6th Italian Conference on Computational Linguistics. Retrieved from http://ceur-ws.org/Vol-2481/paper22.pdf.Google Scholar
- Michele Corazza, Stefano Menini, Elena Cabrio, Sara Tonelli, and Serena Villata. 2019. InriaFBK drawing attention to offensive language at Germeval2019. In Proceedings of the 15th Conference on Natural Language Processing (KONVENS’19). Retrieved from https://corpora.linguistik.uni-erlangen.de/data/konvens/proceedings/papers/germeval/Germeval_Task_2_2019_paper_1.INRIA.pdf.Google Scholar
- Thomas Davidson, Dana Warmsley, Michael W. Macy, and Ingmar Weber. 2017. Automated hate speech detection and the problem of offensive language. In Proceedings of the 11th International Conference on Web and Social Media (ICWSM’17). 512--515.Google Scholar
- Liangjie Hong, Brian D. Davison, April Kontostathis, Lynne Edwards, Dawei Yin, and Zhenzhen Xue. 2009. Detection of harassment on Web 2.0. In Proceedings of the Content Analysis in the Web Conference. 1--7.Google Scholar
- Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT’19), Volume 1 (Long and Short Papers). 4171--4186. Retrieved from https://aclweb.org/anthology/papers/N/N19/N19-1423/.Google Scholar
- Polina Stadnikova, Dietrich Klakow, Dominik Stammbach, and Azin Zahraei. 2018. Offensive language detection with neural networks for Germeval Task 2018. In Proceedings of the GermEval 2018 and 14th Conference on Natural Language Processing (KONVENS’18).Google Scholar
- Elisabetta Fersini, Paolo Rosso, and Maria Anzovino. 2018. Overview of the task on automatic misogyny identification at IberEval 2018. In Proceedings of the Workshop on Evaluation of Human Language Technologies for Iberian Languages (IberEval@SEPLN’18) (CEUR Workshop Proceedings), Vol. 2150. CEUR-WS.org, 214--228.Google Scholar
- Darja Fišer, Ruihong Huang, Vinodkumar Prabhakaran, Rob Voigt, Zeerak Waseem, and Jacqueline Wernimont. 2018. In Proceedings of the 2nd Workshop on Abusive Language Online (ALW2’18). Association for Computational Linguistics.Google Scholar
- Paula Fortuna, Ilaria Bonavita, and Sérgio Nunes. 2018. Merging datasets for hate speech classification in Italian. In Proceedings of the 6th Evaluation Campaign of Natural Language Processing and Speech Tools for Italian. Final Workshop (EVALITA’18) co-located with the 5th Italian Conference on Computational Linguistics (CLiC-it’18).Google ScholarCross Ref
- Antigoni-Maria Founta, Despoina Chatzakou, Nicolas Kourtellis, Jeremy Blackburn, Athena Vakali, and Ilias Leontiadis. 2018. A unified deep learning architecture for abuse detection. CoRR abs/1802.00385 (2018). arxiv:1802.00385.Google Scholar
- Antigoni-Maria Founta, Constantinos Djouvas, Despoina Chatzakou, Ilias Leontiadis, Jeremy Blackburn, Gianluca Stringhini, Athena Vakali, Michael Sirivianos, and Nicolas Kourtellis. 2018. Large scale crowdsourcing and characterization of Twitter abusive behavior. In Proceedings of the 12th International Conference on Web and Social Media (ICWSM’18). 491--500.Google Scholar
- Björn Gambäck and Utpal Kumar Sikdar. 2017. Using convolutional neural networks to classify hate-speech. In Proceedings of the 1st Workshop on Abusive Language Online. Association for Computational Linguistics, 85--90. Retrieved from: http://aclweb.org/anthology/W17-3013.Google ScholarCross Ref
- Mario Graff, Sabino Miranda-Jiménez, Eric Sadit Tellez, Daniela Moctezuma, Vladimir Salgado, José Ortiz-Bejar, and Claudia N. Sánchez. 2018. Author profiling and aggressiveness analysis in Twitter using μTC and EvoMSA. In Proceedings of the 3rd Workshop on Evaluation of Human Language Technologies for Iberian Languages (IberEval’18) co-located with the 34th Conference of the Spanish Society for Natural Language Processing (SEPLN’18). 128--133.Google Scholar
- Edouard Grave, Piotr Bojanowski, Prakhar Gupta, Armand Joulin, and Tomas Mikolov. 2018. Learning word vectors for 157 languages. In Proceedings of the International Conference on Language Resources and Evaluation (LREC’18).Google Scholar
- Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural Comput. 9, 8 (Nov. 1997), 1735--1780.Google ScholarDigital Library
- Matthew Honnibal and Ines Montani. 2017. spaCy 2: Natural language understanding with Bloom embeddings, convolutional neural networks and incremental parsing. To Appear (2017).Google Scholar
- Tianran Hu, Han Guo, Hao Sun, Thuy-vy Thi Nguyen, and Jiebo Luo. 2017. Spice up your chat: The intentions and sentiment effects of using emojis. In Proceedings of the 11th International Conference on Web and Social Media (ICWSM’17). 102--111.Google Scholar
- Vijayasaradhi Indurthi, Bakhtiyar Syed, Manish Shrivastava, Nikhil Chakravartula, Manish Gupta, and Vasudeva Varma. 2019. Using sentence embeddings to identify hate speech against immigrants and women in Twitter. In Proceedings of the 13th International Workshop on Semantic Evaluation. Association for Computational Linguistics, 70--74. DOI:https://doi.org/10.18653/v1/S19-2009Google Scholar
- Rohan Kshirsagar, Tyrus Cukuvac, Kathy McKeown, and Susan McGregor. 2018. Predictive embeddings for hate speech detection on Twitter. In Proceedings of the 2nd Workshop on Abusive Language Online (ALW2’18). Association for Computational Linguistics, 26--32.Google ScholarCross Ref
- Gretel Liz De la Peña Sarracén, Reynaldo Gil Pons, Carlos Enrique Muñiz-Cuza, and Paolo Rosso. 2018. Hate speech detection using attention-based LSTM. In Proceedings of the 6th Evaluation Campaign of Natural Language Processing and Speech Tools for Italian. Final Workshop (EVALITA’18) co-located with the 5th Italian Conference on Computational Linguistics (CLiC-it’18).Google Scholar
- Younghun Lee, Seunghyun Yoon, and Kyomin Jung. 2018. Comparative studies of detecting abusive language on Twitter. CoRR abs/1808.10245 (2018). arxiv:1808.10245Google Scholar
- Ping Liu, Wen Li, and Liang Zou. 2019. Transfer learning for offensive language detection using bidirectional transformers. In Proceedings of the 13th International Workshop on Semantic Evaluation. Association for Computational Linguistics, 87--91. DOI:https://doi.org/10.18653/v1/S19-2011Google Scholar
- Tomas Mikolov, Edouard Grave, Piotr Bojanowski, Christian Puhrsch, and Armand Joulin. 2018. Advances in pre-training distributed word representations. In Proceedings of the International Conference on Language Resources and Evaluation (LREC’18).Google Scholar
- Pushkar Mishra, Helen Yannakoudakis, and Ekaterina Shutova. 2018. Neural character-based composition models for abuse detection. In Proceedings of the 2nd Workshop on Abusive Language Online (ALW’18). Association for Computational Linguistics, 1--10. DOI:https://doi.org/10.18653/v1/W18-5101Google ScholarCross Ref
- Saif M. Mohammad and Peter D. Turney. 2010. Emotions evoked by common words and phrases: Using Mechanical Turk to create an emotion lexicon. In Proceedings of the NAACL HLT 2010 Workshop on Computational Approaches to Analysis and Generation of Emotion in Text. Association for Computational Linguistics, 26--34.Google ScholarDigital Library
- Saif M. Mohammad and Peter D. Turney. 2013. Crowdsourcing a word–emotion association lexicon. Computat. Intell. 29, 3 (2013), 436--465.Google ScholarCross Ref
- Roberto Navigli and Simone Paolo Ponzetto. 2012. BabelNet: The automatic construction, evaluation, and application of a wide-coverage multilingual semantic network. Artif. Intell. 193 (2012), 217--250.Google ScholarDigital Library
- Chikashi Nobata, Joel R. Tetreault, Achint Thomas, Yashar Mehdad, and Yi Chang. 2016. Abusive language detection in online user content. In Proceedings of the 25th International Conference on World Wide Web (WWW’16). 145–153.Google ScholarDigital Library
- Joaquin Padilla Montani and Peter Schüller. 2018. German abusive tweet detection. In Proceedings of the GermEval 2018 and 14th Conference on Natural Language Processing (KONVENS’18).Google Scholar
- F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. 2011. Scikit-learn: Machine learning in Python. J. Mach. Learn. Res. 12 (2011), 2825--2830.Google ScholarDigital Library
- Juan Manuel Pérez and Franco M. Luque. 2019. Robust embeddings for tweet classification. In Proceedings of the 13th International Workshop on Semantic Evaluation. Association for Computational Linguistics, 64--69. DOI:https://doi.org/10.18653/v1/S19-2008Google Scholar
- Emanuele Pianta, Luisa Bentivogli, and Christian Girardi. 2002. MultiWordNet: Developing an aligned multilingual database. In Proceedings of the 1st International Conference on Global WordNet.Google Scholar
- Marco Polignano and Pierpaolo Basile. 2018. HanSEL: Italian hate speech detection through ensemble learning and deep neural networks. In Proceedings of the 6th Evaluation Campaign of Natural Language Processing and Speech Tools for Italian. Final Workshop (EVALITA’18) co-located with the 5th Italian Conference on Computational Linguistics (CLiC-it’18).Google ScholarCross Ref
- Valentino Santucci, Stefania Spina, Alfredo Milani, Giulio Biondi, and Gabriele Di Bari. 2018. Detecting hate speech for Italian language in social media. In Proceedings of the 6th Evaluation Campaign of Natural Language Processing and Speech Tools for Italian. Final Workshop (EVALITA’18) co-located with the 5th Italian Conference on Computational Linguistics (CLiC-it’18).Google ScholarCross Ref
- Tatjana Scheffler, Erik Haegert, Santichai Pornavalaia, and Mino Lee Sasse. 2018. Feature explorations for hate speech classification. In Proceedings of the GermEval 2018 and 14th Conference on Natural Language Processing (KONVENS’18).Google Scholar
- Mike Schuster, Kuldip K. Paliwal, and A. General. 1997. Bidirectional recurrent neural networks. IEEE Trans. Sig. Proc. 45, 11 (1997), 2673--2681.Google ScholarDigital Library
- Abhishek Singh, Eduardo Blanco, and Wei Jin. 2019. Incorporating emoji descriptions improves tweet classification. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics (NAACL’19).Google ScholarCross Ref
- Samuel L. Smith, David H. P. Turban, Steven Hamblin, and Nils Y. Hammerla. 2017. Offline bilingual word vectors, orthogonal transformations, and the inverted softmax. In Proceedings of the 5th International Conference on Learning Representations (ICLR’17).Google Scholar
- Samuel L. Smith, David H. P. Turban, Steven Hamblin, and Nils Y. Hammerla. 2017. Offline bilingual word vectors, orthogonal transformations and the inverted softmax. CoRR abs/1702.03859 (2017). arxiv:1702.03859Google Scholar
- Jacopo Staiano and Marco Guerini. 2014. Depeche mood: A lexicon for emotion analysis from crowd annotated news. In Proceedings of the 52nd Meeting of the Association for Computational Linguistics (Volume 2: Short Papers). 427--433.Google ScholarCross Ref
- Dirk von Grunigen, Ralf Grubenmann, Fernando Benites, Pius Von Daniken, and Mark Cieliebak. 2018. Classification of offensive content in tweets using convolutional neural networks and gated recurrent units. In Proceedings of the GermEval 2018 and 14th Conference on Natural Language Processing (KONVENS’18).Google Scholar
- Zeerak Waseem, Wendy Hui Kyong Chung, Dirk Hovy, and Joel Tetreault. 2017. In Proceedings of the 1st Workshop on Abusive Language Online. Association for Computational Linguistics.Google Scholar
- Zeerak Waseem and Dirk Hovy. 2016. Hateful symbols or hateful people? Predictive features for hate speech detection on Twitter. In Proceedings of the Student Research Workshop in Conjunction with NAACL HLT 2018.Google ScholarCross Ref
- Gregor Wiedeman, Eugen Ruppert, Raghav Jindal, and Chris Biemann. 2018. Transfer learning from LDA to BiLSTM-CNN for offensive language detection in Twitter. In Proceedings of the GermEval 2018 and 14th Conference on Natural Language Processing (KONVENS’18).Google Scholar
- Michael Wiegand, Anastasija Amann, Tatiana Anikina, Aikaterini Azoidou, Anastasia Borisenkov, Kirstin Kolmorgen, Insa Kroger, and Christine Schafer. 2018. Examining different types of classifiers and features. In Proceedings of the GermEval 2018 and 14th Conference on Natural Language Processing (KONVENS’18).Google Scholar
- Michael Wiegand, Josef Ruppenhofer, and Thomas Kleinbauer. 2019. Detection of abusive language: The problem of biased datasets. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers). Association for Computational Linguistics, 602--608. DOI:https://doi.org/10.18653/v1/N19-1060Google Scholar
- Michael Wiegand, Melanie Siegel, and Josef Ruppenhofer. 2018. Overview of the GermEval 2018 shared task on the identification of offensive language. In Proceedings of the GermEval 2018 and 14th Conference on Natural Language Processing (KONVENS’18).Google Scholar
- Ellery Wulczyn, Nithum Thain, and Lucas Dixon. 2017. Ex Machina: Personal attacks seen at scale. In Proceedings of the 26th International Conference on World Wide Web (WWW’17). 1391--1399.Google ScholarDigital Library
- Marcos Zampieri, Shervin Malmasi, Preslav Nakov, Sara Rosenthal, Noura Farra, and Ritesh Kumar. 2019. Identifying and categorizing offensive language in social media (OffensEval). In Proceedings of the 13th International Workshop on Semantic Evaluation. Association for Computational Linguistics, 75--86. DOI:https://doi.org/10.18653/v1/S19-2010Google Scholar
- Robinson D. Zhang, Z. and J. Tepper. 2018. Detecting hate speech on Twitter using a convolution-GRU based deep neural network. In Proceedings of the Semantic Web Conference Proceedings (ESWC’18). Springer Verlag, 745--760.Google Scholar
Index Terms
- A Multilingual Evaluation for Online Hate Speech Detection
Recommendations
Improving hate speech detection using Cross-Lingual Learning
AbstractThe growth of social media worldwide has brought social benefits and challenges. One problem we highlight is the proliferation of hate speech on social media. We propose a novel method for detecting hate speech in texts using Cross-Lingual ...
Highlights- The development of a new methodology for hate speech detection.
- Portuguese hate speech detection using Cross-Lingual Learning.
- Up to 20% performance improvement over other models using the OffComBr-2 corpus.
HateCircle and Unsupervised Hate Speech Detection Incorporating Emotion and Contextual Semantics
The explosive growth of social media has fueled an extensive increase in online freedom of speech. The worldwide platform of human voice creates possibilities to assail other users without facing any consequences, and flout social etiquettes, resulting in ...
Mono vs Multilingual BERT for Hate Speech Detection and Text Classification: A Case Study in Marathi
Artificial Neural Networks in Pattern RecognitionAbstractTransformers are the most eminent architectures used for a vast range of Natural Language Processing tasks. These models are pre-trained over a large text corpus and are meant to serve state-of-the-art results over tasks like text classification. ...
Comments