1 Introduction
1.1 Query completion suggestions in SEs
Type of misspelling | Examples |
---|---|
Leet converted words | @$$hole, 81tch (referring to bitch) |
Repeated characters | Fuckkk, fuccckkkk, niggger |
Short forms | F U, pls , stfu |
Similar sounding words | Bich, fack off, suk my dik |
Vowels removed words | Btch, fckd |
1.2 User conversations in messengers
-
you are great _/\(\backslash \)_
-
Thank you :) :D
-
Awesome job (y)
-
We introduce the research problem of automatic identification of inappropriate content in text.
-
We propose a novel deep learning-based approach called "Convolutional, Bi-Directional LSTM (C-BiLSTM)" for identifying inappropriate query suggestions in web search.
-
We evaluate the various techniques proposed so far on query suggestions, including standard deep learning techniques (such as CNN, LSTM and BLSTM), on a real-world datasets and compare their effectiveness.
-
We also evaluate performance of sequential models like LSTM and BLSTM to identify inappropriate conversations.
2 Related work
3 Inappropriate text detection on web search queries
3.1 C-BiLSTM for inappropriate query detection
3.1.1 Input query embedding and padding
3.1.2 Learning feature representations using convolution layer
Parameter | CNN | LSTM | BLSTM | C-BiLSTM |
---|---|---|---|---|
Batch size | 1000 | 1000 | 1000 | 1000 |
Max len. | 24 | 24 | 24 | 24 |
WordVecDim. | 300 | 300 | 300 | 300 |
CNN depth | 4 | NA | NA | 3 |
Filter size | 2 \(\times \) 20 | NA | NA | 3 \(\times \) 25 |
Pooling | Max-pooling | NA | NA | NA |
Nonlinearity | ReLU | NA | NA | ReLU |
LSTM cells | NA | 40 | 40 | 32 |
Optimizer | Adagrad | Adagrad | Adagrad | Adagrad |
Learning rate | 0.01 | 0.05 | 0.05 | 0.05 |
Epsilon | 1e−08 | 1e−08 | 1e−08 | 1e−08 |
Category | # Samples | Sample queries |
---|---|---|
ExtremeViolence/SelfHarm /IllegalActivity
| 1619 |
Woman beheaded video how many pills does it take to kill yourself growing marijuana indoors for beginners
|
Race/Religion/Sexual Orientation/Gender
| 2241 |
New zealanders hate americans anti islam shirts gays are destroying this country butch clothing for women
|
OtherOffensive/Celebrity
| 1124 |
Jokes about short people louie gohmert stupid quotes
|
Clean
| 74,057 |
20 adjectives that describe chocolate
|
what is the order of the planets
| ||
Total
|
79,041
|
3.1.3 Capturing sequential patterns with Bi-directional LSTM
3.2 Model training
3.3 Experimental setup
3.3.1 Dataset details
Label | Training | Validation | Test | Total |
---|---|---|---|---|
Inappropriate | 4594 | 212 | 178 | 4984 |
Clean | 65,447 | 4788 | 3822 | 74,057 |
Total
| 70,041 | 5000 | 4000 | 79,041 |
Sample Patterns
|
Sample Matching Queries
|
Sample Words in Category
|
---|---|---|
Proper Noun
AND
NegativePersonalityWord
|
hillary clinton is terrible marie harf is an idiot
|
NegativePersonalityWord:
terrible, idiot, moron, miser..
|
Proper Noun AND
‘not’/‘no’ AND
PositivePersonalityWord
|
ellen degeneres is not funny
|
PositivePersonalityWord:
calm, affectionate, charming..
|
SelfHarmPrefix
AND
SelfHarmSuffix
|
how can i commit suicide methods to kill myself
|
SelfHarmPrefix:
how can I, how should I, ways of...
SelfHarmSuffix:
hang myself, shoot myself, commit suicide...
|
Ethnicity/Religion
AND
CommunityDislikeWord
|
americans hate black people muslims murdered christians
|
Ethnicity/Religion:
americans, jews, muslims..
CommunityDislikeWord:
hate, disrespect, kill...
|
CoreOffensiveWord
|
slut shaming quotes the bitch is back
|
CoreOffensiveWord:
fuck, asshole, bitch, slut..
|
3.4 Baseline approaches
Model | Precision | Recall | F1 score |
---|---|---|---|
PKF | 0.625 | 0.2142 | 0.3190 |
BDT | 0.7926 | 0.2784 | 0.4120 |
BDT-DSSM |
0.9474
| 0.3051 | 0.4615 |
SVM | 0.8322 | 0.3593 | 0.5019 |
SVM-DSSM | 0.9241 | 0.4101 | 0.5680 |
CNN | 0.7148 |
0.8952
| 0.7949 |
LSTM | 0.8862 | 0.7047 | 0.7850 |
BLSTM | 0.8018 | 0.8285 | 0.8149 |
C-BiLSTM | 0.9246 | 0.8251 |
0.8720
|
3.5 Results and discussion
3.5.1 Qualitative analysis
Query | True Label | C-BLSTM Label | Judgment Explanation | Comments |
---|---|---|---|---|
nigerplease.com
| Inappropriate | Inappropriate | The word niger is an Inappropriate word | Since nigerplease.com is a single word, PKF, SVM, BDT models fail. Other deep learnt models also misclassify this query. C-BLSTM alone correctly classifies this one. |
shake and bake meth instructions
| Inappropriate | Inappropriate | “meth” is a drug which is illegal in USA and some other parts of world | "meth" is a short form for Methamphetamine and hence PKF, SVM, BDT models fail. Other deep learnt models also misclassify this query. C-BLSTM alone correctly classifies this one. |
a**monkey
| Inappropriate | Inappropriate | It refers to the Inappropriate word—“assmonkey” | C-BLSTM perfectly classifies it. PKF, SVM, BDT fail because it includes “**”. Other deep learnt models also fail in this case. |
hore in the bible
| Inappropriate | Inappropriate | It is a spell mistake of the word "whore" and is Inappropriate to christians | PKF, SVM, BDT fail because of not catching the spell mistake. Other deep learnt models also fail except for C-BLSTM. |
marvin gaye if i should die tonight download
| Clean | Clean | Not Inappropriate since it is a song download | PKF misclassifies it as “Inappropriate” due to the presence of “die tonight” pattern. Remaining models classify correctly. |
asshat in sign language
| Inappropriate | Clean | An Inappropriate term in sign language | BDT perfectly classifies it. Remaining all models misclassify it. |
why do asians speak the ching chong
| Inappropriate | Clean | Ching chong is a pejorative term for chinese language | PKF classifies it correctly since “ching chong” is included in list of core Inappropriate words. Remaining classifiers fail. |
3.5.2 Query autocompletion filtering task
4 Inappropriate text detection for conversations
4.1 LSTM and BLSTM for inappropriate conversation detection
4.1.1 Input conversation embedding
Parameter | LSTM | BLSTM |
---|---|---|
Batch size | 1000 | 1000 |
Embedding dim. | 150 | 150 |
LSTM cells | 50 | 50 |
Optimizer | SGD | SGD |
Learning rate | 0.01 | 0.05 |
Epsilon | 1e−08 | 1e−08 |
Dataset | Offensive | Clean | Total |
---|---|---|---|
Training data | 27,311 | 220,663 | 247,974 |
Validation set | 2000 | 8000 | 10,000 |
Zo test set | 1805 | 2146 | 3951 |
Xbox test set | 2303 | 7697 | 10,000 |
4.1.2 LSTM
4.1.3 Bi-directional LSTM (BLSTM)
4.2 Model training
Category | Examples |
---|---|
Offensive | dude i hope you burn in hell |
your awful drink bleach and delete smite | |
go fuck yourself fuckin faggot | |
hindu are shit sweaty cunts | |
f off u american prick go eat some tacobell | |
i bet you guys have aids | |
Clean | you’re a great guy. you know that |
u being a good boy | |
im in ur game join my party | |
wat should i buy |
Model | Precision | Recall | F1 score |
---|---|---|---|
PKF | 0.865 | 0.427 | 0.572 |
BDT |
0.964
| 0.451 | 0.614 |
LSTM | 0.93 | 0.833 | 0.879 |
BLSTM | 0.921 |
0.867
|
0.893
|
Model
|
Precision
|
Recall
|
F1 score
|
---|---|---|---|
PKF | 0.789 | 0.413 | 0.542 |
BDT |
0.935
| 0.544 | 0.687 |
LSTM | 0.857 | 0.731 |
0.789
|
BLSTM | 0.828 |
0.749
| 0.786 |