Abstract
Online harassment is a pervasive and pernicious problem. Techniques like natural language processing and machine learning are promising approaches for identifying abusive language, but they fail to address structural power imbalances perpetuated by automated labeling and classification. Similarly, platform policies and reporting tools are designed for a seemingly homogenous user base and do not account for individual experiences and systems of social oppression. This paper describes the design and evaluation of HeartMob, a platform built by and for people who are disproportionately affected by the most severe forms of online harassment. We conducted interviews with 18 HeartMob users, both targets and supporters, about their harassment experiences and their use of the site. We examine systems of classification enacted by technical systems, platform policies, and users to demonstrate how 1) labeling serves to validate (or invalidate) harassment experiences; 2) labeling motivates bystanders to provide support; and 3) labeling content as harassment is critical for surfacing community norms around appropriate user behavior. We discuss these results through the lens of Bowker and Star's classification theories and describe implications for labeling and classifying online abuse. Finally, informed by intersectional feminist theory, we argue that fully addressing online harassment requires the ongoing integration of vulnerable users' needs into the design and moderation of online platforms.
- Sara Ahmed. 2017. Living a Feminist Life. Duke University Press.Google Scholar
- Julia Angwin. 2017. Facebook's Secret Censorship Rules Protect White Men from Hate Speech But Not Black Children. ProPublica.Google Scholar
- Zahra Ashktorab and Jessica Vitak. 2016. Designing Cyberbullying Mitigation and Prevention Solutions Through Participatory Design With Teenagers. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (CHI '16). ACM, New York, NY, USA, 3895--3905. Google ScholarDigital Library
- Shaowen Bardzell and Jeffrey Bardzell. 2011. Towards a feminist HCI methodology: social science, feminism, and HCI. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '11). ACM, New York, NY, USA, 675--684. Google ScholarDigital Library
- Howard Becker. 1963. Outsiders. Glencoe. The Free Press 9: 1982.Google Scholar
- Lindsay Blackwell, Jean Hardy, Tawfiq Ammari, Tiffany Veinot, Cliff Lampe, and Sarita Schoenebeck. 2016. LGBT Parents and Social Media: Advocacy, Privacy, and Disclosure During Shifting Social Movements. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (CHI '16). ACM, New York, NY, USA, 610--622. Google ScholarDigital Library
- Geoffrey C. Bowker and Susan Leigh Star. 2000. Sorting things out: Classification and its consequences. MIT press. Google ScholarDigital Library
- Amy Bruckman, Catalina Danis, Cliff Lampe, Janet Sternberg, and Chris Waldron. 2006. Managing deviant behavior in online communities. In CHI '06 Extended Abstracts on Human Factors in Computing Systems (CHI EA '06). ACM, New York, NY, USA, 21--24. Google ScholarDigital Library
- Eshwar Chandrasekharan, Mattia Samory, Anirudh Srinivasan, and Eric Gilbert. 2017. The Bag of Communities: Identifying Abusive Behavior Online with Preexisting Internet Data. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems (CHI '17). ACM, New York, NY, USA, 3175--3187. Google ScholarDigital Library
- Justin Cheng, Michael Bernstein, Cristian Danescu-Niculescu-Mizil, and Jure Leskovec. 2017. Anyone Can Become a Troll: Causes of Trolling Behavior in Online Discussions. In Proceedings of the 2017 ACM Conference on Computer Supported Cooperative Work and Social Computing (CSCW '17). ACM, New York, NY, USA, 1217--1230. Google ScholarDigital Library
- Robert B. Cialdini. 2007. Descriptive social norms as underappreciated sources of social control. Psychometrika 72, 2: 263.Google ScholarCross Ref
- Danielle Keats Citron and Mary Anne Franks. 2014. Criminalizing Revenge Porn. Social Science Research Network, Rochester, NY.Google Scholar
- Rep Katherine Clark. 2015. Online Violence Against Trans Women Perpetuates Dangerous Cycle. Huffington Post.Google Scholar
- Keith Collins. 2017. Tech is overwhelmingly white and male, and white men are just fine with that. Quartz.Google Scholar
- Patricia Hill Collins. 1993. Black feminist thought in the matrix of domination. Social theory: The multicultural and classic readings: 615--625.Google Scholar
- Kimberle Crenshaw. 1991. Mapping the margins: Intersectionality, identity politics, and violence against women of color. Stanford law review: 1241--1299.Google Scholar
- Julian Dibbell. 1993. A Rape in Cyberspace. Village Voice XXXVIII, 51.Google Scholar
- Jill P. Dimond, Michaelanne Dye, Daphne Larose, and Amy S. Bruckman. 2013. Hollaback!: the role of storytelling online in a social movement organization. In Proceedings of the 2013 conference on Computer supported cooperative work (CSCW '13). ACM, New York, NY, USA, 477--490. Google ScholarDigital Library
- Judith Donath. 1999. Identity and Deception in the Virtual Community. Communities in cyberspace. Psychology Press.Google Scholar
- Maeve Duggan. 2014. Online Harassment. Pew Research Center.Google Scholar
- Maeve Duggan, 2017. Online Harassment 2017. Pew Research Center.Google Scholar
- Noah J. Goldstein, Robert B. Cialdini, and Vladas Griskevicius. 2008. A room with a viewpoint: Using social norms to motivate environmental conservation in hotels. Journal of consumer Research 35, 3: 472--482.Google ScholarCross Ref
- Mark Griffiths. 2002. Occupational health issues concerning Internet use in the workplace. Work & Stress 16, 4: 283--286.Google ScholarCross Ref
- Sandra Harding. 1992. Rethinking standpoint epistemology: What is "strong objectivity"? The Centennial Review 36, 3: 437--470.Google Scholar
- Bell Hooks. 2000. Feminism is for everybody: Passionate politics. Pluto Press.Google Scholar
- Hossein Hosseini, Sreeram Kannan, Baosen Zhang, and Radha Poovendran. 2017. Deceiving Google's Perspective API Built for Detecting Toxic Comments. arXiv:1702.08138 {cs}.Google Scholar
- Sara Kiesler, Robert E. Kraut, Paul Resnick, and Aniket Kittur. 2012. Regulating behavior in online communities. Building Successful Online Communities: Evidence-Based Social Design. MIT Press, Cambridge, MA.Google Scholar
- Cliff Lampe. 2014. Gamification and social media. The Gameful world: Approaches, issues, applications. MIT Press, Cambridge, MA.Google Scholar
- Cliff Lampe and Paul Resnick. 2004. Slash (dot) and burn: distributed moderation in a large online conversation space. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '04). ACM, New York, NY, USA, 543--550. Google ScholarDigital Library
- Amanda Lenhart, Michelle Ybarra, Kathryn Zickuhr, and Myeshia Prive-Feeney. 2016. Online Harassment, Digital Abuse, and Cyberstalking in America. Data & Society Institute.Google Scholar
- Eden Litt and Eszter Hargittai. 2016. The imagined audience on social network sites. Social Media + Society 2, 1: 2056305116633482.Google Scholar
- Alice E. Marwick. 2012. The public domain: Social surveillance in everyday life. Surveillance & Society 9, 4: 378.Google ScholarCross Ref
- Karl Marx and Friedrich Engels. 1967. The communist manifesto (1848). Trans. Samuel Moore. London: Penguin.Google Scholar
- Leysia Palen and Paul Dourish. 2003. Unpacking privacy for a networked world. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '03). ACM, New York, NY, USA, 129--136. Google ScholarDigital Library
- Michael L. Pittaro. 2007. Cyber stalking: An analysis of online harassment and intimidation. International Journal of Cyber Criminology 1, 2: 180--197.Google Scholar
- Lee Rainie, Janna Anderson, and Jonathan Albright. 2017. The Future of Free Speech, Trolls, Anonymity and Fake News Online. Pew Research Center.Google Scholar
- Sarita Schoenebeck, Nicole B. Ellison, Lindsay Blackwell, Joseph B. Bayer, and Emily B. Falk. 2016. Playful Backstalking and Serious Impression Management: How Young Adults Reflect on Their Past Identities on Facebook. In Proceedings of the 19th ACM Conference on Computer-Supported Cooperative Work & Social Computing (CSCW '16). ACM, New York, NY, USA, 1475--1487. Google ScholarDigital Library
- Michael D. Shear. 2016. Trump as Cyberbully in Chief? Twitter Attack on Union Boss Draws Fire. The New York Times.Google Scholar
- Peter K. Smith, Jess Mahdavi, Manuel Carvalho, Sonja Fisher, Shanette Russell, and Neil Tippett. 2008. Cyberbullying: its nature and impact in secondary school pupils. Journal of Child Psychology and Psychiatry 49, 4: 376--385.Google ScholarCross Ref
- Lee Sproull, Sara Kiesler, and Sara B. Kiesler. 1992. Connections: New Ways of Working in the Networked Organization. MIT Press. Google ScholarDigital Library
- Susan Leigh Star and James R. Griesemer. 1989. Institutional ecology, 'translations' and boundary objects: Amateurs and professionals in Berkeley's Museum of Vertebrate Zoology, 1907--39. Social studies of science 19, 3: 387--420.Google Scholar
- Thomas, D. R. (2006). A general inductive approach for analyzing qualitative evaluation data. American journal of evaluation, 27(2), 237--246.Google Scholar
- U. S. Department of Labor. 2015. Current Population Survey: Detailed occupation by sex and race. Bureau of Labor Statistics.Google Scholar
- Jessica Vitak, Stacy Blasiola, Eden Litt, and Sameer Patil. 2015. Balancing Audience and Privacy Tensions on Social Network Sites: Strategies of Highly Engaged Users. International Journal of Communication 9: 20.Google Scholar
- Joseph B. Walther and Malcolm R. Parks. 2002. Cues filtered out, cues filtered in. Handbook of interpersonal communication 3: 529--563.Google Scholar
- Christina Warren. 2017. Twitter's New Abuse Filter Works Great, If Your Name Is Mike Pence. Gizmodo.Google Scholar
- Jeffrey Weeks. 1999. Discourse, desire and sexual deviance: some problems in a history of homosexuality. Culture, society and sexuality. A reader: 119--42.Google Scholar
- Amanda M. Williams and Lilly Irani. 2010. There's methodology in the madness: toward critical HCI ethnography. In CHI '10 Extended Abstracts on Human Factors in Computing Systems (CHI EA '10). ACM, New York, NY, USA, 2725--2734. Google ScholarDigital Library
- Rhiannon Williams. 2014. Facebook's 71 gender options come to UK users. The Telegraph.Google Scholar
- Ellery Wulczyn, Nithum Thain, and Lucas Dixon. 2016. Ex Machina: Personal Attacks Seen at Scale. In Proceedings of the 26th International Conference on World Wide Web (WWW '17). International World Wide Web Conferences Steering Committee, Republic and Canton of Geneva, Switzerland, 1391--1399. Google ScholarDigital Library
- Dawei Yin, Zhenzhen Xue, Liangjie Hong, Brian D. Davison, April Kontostathis, and Lynne Edwards. 2009. Detection of harassment on web 2.0. Proceedings of the Content Analysis in the WEB 2: 1--7.Google Scholar
Index Terms
- Classification and Its Consequences for Online Harassment: Design Insights from HeartMob
Recommendations
Evaluating Automated System Interventions Against Email Harassment
CHI EA '21: Extended Abstracts of the 2021 CHI Conference on Human Factors in Computing SystemsEmail and other forms of electronic communication are becoming increasingly more essential to our everyday lives. However, with this growth comes the paralleled increased risk of email harassment, exacerbated by the current lack of platform support for ...
Silencing Women? Gender and Online Harassment
While gendered online harassment has received increased attention in academic and public debates, there is a lack of empirical studies examining gender differences in experiences with online harassment. Relying on two independent large-scale population ...
Empathy, self-control, and online harassment: A partial test of Agnew's social concern theory
AbstractIn conceptualizing Social Concern Theory, Robert Agnew argued that social concern can serve as a protective factor against crime and influence the relationship between traditional correlates of crime (e.g., low self-control) and ...
Highlights- We conducted a partial test of Agnew's (2014) social concern theory in explaining online harassment perpetration.
Comments