1 Introduction
-
A newly manually annotated dataset of 2014 Twitter posts combined with our previously annotated dataset from (Kwarteng et al. 2021) of 2519 Twitter posts capturing public responses of misogynoir online (both supportive and non-supportive messages)
-
A dataset of 300 Twitter posts multiple-coded as Misogynoir, Allyship, and Tone policing, Racial gaslighting, White centring, Defensiveness and General sampled from the dataset.
-
An evaluation of current hate speech detection approaches on our misogynoir dataset.
-
An analysis of the challenges and opportunities for understanding misogynoir online.
2 Related work
2.1 Models of misogynoir
2.1.1 Tokenism
2.1.2 White centring
2.1.3 Tone policing
2.1.4 Racial gaslighting
2.1.5 Defensiveness
2.1.6 Unacknowledged privilege
2.2 Challenges of detecting hateful and abusive speech
2.3 Intersectional hate detection
3 Definitions of misogynoir terms and expressions
Categories | Definition | Examples | Terms |
---|---|---|---|
Tone policing (TP) | Language criticising the form of someone’s argument, rather than the content | “Not constructive”, “complaining about”, “whining about” | 45 |
White centring (WC) | Language that seeks to re-contextualise the targets’ challenges inside of white culture and values | “Why does everything have to be about..., “why didn’t she do...”, | 26 |
Racial gaslighting (RG) | Language that seeks to downplay or dismiss the role of race in the targets’ experience | “Reverse racism”, “the only race is the human race”, “colourblind” | 93 |
Defensiveness (D) | Language that talks about calling out bad behaviour as an attack of some sort or an assassination of character | “Cancel culture”, “block the conversation”, “friends who are Black” | 39 |
General (G) | Language that more generally refers to racism, sexism or more general support/non-support | “Sexism”, “Yaaas”, “Thank you!” | 51 |
Categories | Example tweet |
---|---|
Tone policing (TP) | “I think what you are doing can be called womansplaining your rude and arrogant way of speaking” |
White centring (WC) | “I find it extremely hard to believe Pinterest will send a PI after you. If there are 2 people vying for one promotion, ANY company will ‘pit’ employees against one other (regardless of their friendship status/ race). Stop blaming your incompetence on race.” |
Racial gaslighting (RG) | “From what I can gather, the point is to push the “white people are bad” narrative.” |
Defensiveness (D) | “So you are saying you’ve read the email that got her terminated and it was not a firing offense? Or are you just blindly defending another female out of an emotional requirement to defend a perceived social injustice? And you hold a PhD? Fascinating.” |
General (G) | “wow!” |
Categories | Definition | Examples |
---|---|---|
Sharing experiences (E) | Users sharing their own experiences of misogynoir as an act of solidarity or allyship | “@company @company. Are some of the most racist companies I worked with. At that time i even had a recruiter say “yeah we know it’s a problem but it’s a big account for us” |
Showing thanks and gratitude (T) | Users expressing their gratitude toward those sharing their experiences of misogynoir | “Thank you for this”, “I’m sorry about this @user and thanks for sharing.” |
Generic(GR) | More general messages of support | “I am so sorry @user. This is unbelievable. I am speechless.” |
4 Analysis approach
4.1 Dataset
4.2 Data annotation
Labels | Tweets Annotated | Filtered | Remained |
---|---|---|---|
Allyship | 3862 | 886 | 2976 |
Misogynoir | 183 | 21 | 162 |
4.3 Hate speech detection tools
5 Experimental settings
6 Results
6.1 Balanced dataset
HateSonar | Google’s Perspective API | |||||
---|---|---|---|---|---|---|
Precision | Recall | F1-Score | Precision | Recall | F1-Score | |
Allyship | 0.53 | 0.91 | 0.67 | 0.55 | 0.83 | 0.66 |
Misogynoir | 0.68 | 0.19 | 0.29 | 0.66 | 0.33 | 0.44 |
Accuracy | 0.55 | 0.58 |
6.2 Imbalanced dataset
HateSonar | Google’s Perspective API | |||||
---|---|---|---|---|---|---|
Precision | Recall | F1-Score | Precision | Recall | F1-Score | |
Allyship | 0.95 | 0.92 | 0.94 | 0.96 | 0.84 | 0.89 |
Misogynoir | 0.11 | 0.19 | 0.14 | 0.10 | 0.33 | 0.15 |
6.3 Analysis of the models of misogynoir and allyship types based on the HateSonar and Perspective API
Types | No. of tweets | HateSonar | Google’s perspective API |
---|---|---|---|
Defensiveness | 9 | 1 (11%) | 3 (33%) |
General | 62 | 10 (16%) | 11 (18%) |
Racial gaslighting | 53 | 9 (17%) | 27 (57%) |
Tone policing | 22 | 7 (32%) | 9 (41%) |
White centring | 16 | 3 (19%) | 3 (19%) |
Types | No. of tweets | HateSonar | Google’s Perspective API |
---|---|---|---|
Experience (E) | 112 | 16 (14%) | 32 (29%) |
Generic (GR) | 2003 | 175 (9%) | 366 (18%) |
Thanks (T) | 861 | 50 (6%) | 82 (10%) |
6.4 Analysis of misogynoir and allyship based on the HateSonar and Perspective API
6.5 Analysis of Google’s Perspective API attributes
Attributes | Precision | Recall | F1-Score | Accuracy |
---|---|---|---|---|
Severe toxicity | 0.67 | 0.12 | 0.20 | 0.53 |
Identity attack | 0.66 | 0.31 | 0.42 | 0.57 |
Insult | 0.70 | 0.29 | 0.41 | 0.58 |
Profanity | 0.62 | 0.10 | 0.17 | 0.52 |
Threat | 0.74 | 0.11 | 0.20 | 0.54 |
Attributes | Precision | Recall | F1-Score | Accuracy |
---|---|---|---|---|
Severe toxicity | 0.11 | 0.12 | 0.11 | 0.90 |
Identity attack | 0.09 | 0.32 | 0.14 | 0.79 |
Insult | 0.11 | 0.29 | 0.16 | 0.84 |
Profanity | 0.08 | 0.10 | 0.09 | 0.89 |
Threat | 0.01 | 0.13 | 0.11 | 0.90 |