Top

Published in:

01-04-2024

Diversity in issue assignment: humans vs bots

Authors: Aniruddhan Murali, Gaurav Sahu, Kishanthan Thangarajah, Brian Zimmerman, Gema Rodríguez-Pérez, Meiyappan Nagappan

Published in: Empirical Software Engineering | Issue 2/2024

Activate our intelligent search to find suitable subject content or patents.

search-config

AI-assisted search

Off

Abstract

Issue assignment process is a common practice in open source projects for managing incoming and existing issues. While traditionally performed by humans, the adoption of software bots for automating this process has become prevalent in recent years. The objective of this paper is to examine the diversity in issue assignments between bots and humans in open source projects, with the aim of understanding how open source communities can foster diversity and inclusivity. To achieve this, we conducted a quantitative analysis on three major open source projects hosted on GitHub, focusing on the most likely racial and ethnic diversity of both human and bot assignors during the issue assignment process. We analyze how issues are assigned by humans and bots, as well as the distribution of issue types among White and Non-White open source collaborators. Additionally, we explore how the diversity in issue assignments evolves over time for human and bot assignors. Our results reveal that both human and bot assignors majorly assign issues to developers of the same most likely race and ethnicity. Notably, we find bots assign more issues to perceived White developers than Non-White developers. In conclusion, our findings suggest that bots display higher levels of bias than humans in most cases, although humans also demonstrate significant bias in certain instances. Thus, open source communities must actively address these potential biases in their GitHub issue assignment process to promote diversity and inclusivity.

next article HyperPUT: generating synthetic faulty programs to challenge bug-finding tools

Dont have a licence yet? Then find out more about our products and how to get one now:

Springer Professional "Wirtschaft"

Online-Abonnement

Mit Springer Professional "Wirtschaft" erhalten Sie Zugriff auf:

über 67.000 Bücher
über 340 Zeitschriften

aus folgenden Fachgebieten:

Bauwesen + Immobilien
Business IT + Informatik
Finance + Banking
Management + Führung
Marketing + Vertrieb
Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

inform now

Springer Professional "Technik"

Online-Abonnement

Mit Springer Professional "Technik" erhalten Sie Zugriff auf:

über 67.000 Bücher
über 390 Zeitschriften

aus folgenden Fachgebieten:

Automobil + Motoren
Bauwesen + Immobilien
Business IT + Informatik
Elektrotechnik + Elektronik
Energie + Nachhaltigkeit
Maschinenbau + Werkstoffe

Jetzt Wissensvorsprung sichern!

inform now

Springer Professional "Wirtschaft+Technik"

Online-Abonnement

Mit Springer Professional "Wirtschaft+Technik" erhalten Sie Zugriff auf:

über 102.000 Bücher
über 537 Zeitschriften

aus folgenden Fachgebieten:

Automobil + Motoren
Bauwesen + Immobilien
Business IT + Informatik
Elektrotechnik + Elektronik
Energie + Nachhaltigkeit
Finance + Banking
Management + Führung
Marketing + Vertrieb
Maschinenbau + Werkstoffe
Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

inform now

We acknowledge that code review and issue assignment serve different purposes, but in both cases, there is an assignee and an assignor.

https://github.com/Demfier/diversity-in-issue-assignment-humans-vs-bots

https://www.deepcode.ai/

http://oss.x-lab.info/github-insight-report-2020-en.pdf

https://www.census.gov/topics/population/race/about.html

NameSor outputs a fifth category called “Other” in addition to the four categories.

Full probability distribution for this example: \(p_1=48\%, p_2=2\%, p_3=25\%, p_4=25\%\)

More details about vscode-triage-bot: https://github.com/microsoft/vscode-github-triage-actions

https://www.census.gov/topics/population/race/about.html

Adams B, Khomh F (2020) The diversity crisis of software engineering for artificial intelligence. IEEE Softw 37(5):104–108CrossRef

Ali M, Kulik CT, Metz I (2011) The gender diversity-performance relationship in services and manufacturing organizations. Int J Hum Resour Manag 22(07):1464–1485CrossRef

Balachandran V (2013) Reducing human effort and improving quality in peer code reviews using automatic static analysis and reviewer recommendation. In: 2013 35th international conference on software engineering (ICSE). IEEE, pp 931–940

Bellamy RKE, Dey K, Hind M, Hoffman SC, Houde S, Kannan K, Lohia P, Martino J, Mehta S, Mojsilovic A et al (2018) Ai fairness 360: an extensible toolkit for detecting, understanding, and mitigating unwanted algorithmic bias. arXiv:1810.01943

Blau PM (1977) Inequality and heterogeneity: a primitive theory of social structure, vol 7. Free Press, New York

Bosu A, Sultana KZ (2019) Diversity and inclusion in open source software (oss) projects: where do we stand? In: 2019 ACM/IEEE international symposium on empirical software engineering and measurement (ESEM). IEEE, pp 1–11

Brokmeier P (2017) Project level effects of gender on contribution evaluation on github. PeerJ PrePrints 5:e2989v1

Brown C, Parnin C (2019) Sorry to bother you: designing bots for effective recommendations. In: 2019 IEEE/ACM 1st international workshop on bots in software engineering (BotSE). IEEE, pp 54–58

Buolamwini J, Gebru T (2018) Gender shades: intersectional accuracy disparities in commercial gender classification. In: Conference on fairness, accountability and transparency. PMLR, pp 77–91

Caldarelli G, De Nicola R, Del Vigna F, Petrocchi M, Saracco F (2020) The role of bot squads in the political propaganda on twitter. Commun Phys 3(1):1–15CrossRef

Carleton AD, Harper E, Menzies T, Xie T, Eldh S, Lyu MR (2020) The ai effect: working at the intersection of ai and se. IEEE Softw 37(4):26–35CrossRef

Chakraborty J, Majumder S, Menzies T (2021) Bias in machine learning software: why? How? What to do? arXiv:2105.12195

Chakraborty J, Majumder S, Yu Z, Menzies T (2020) Fairway: a way to build fair ml software. In: Proceedings of the 28th ACM joint meeting on European software engineering conference and symposium on the foundations of software engineering, pp 654–665

Chen J, Ren Y, Riedl J (2010) The effects of diversity on group productivity and member withdrawal in online volunteer groups. In: Proceedings of the SIGCHI conference on human factors in computing systems, pp 821–830

Chouldechova A, Roth A (2018) The frontiers of fairness in machine learning. arXiv:1810.08810

Corbett-Davies S, Goel S (2018) The measure and mismeasure of fairness: a critical review of fair machine learning. arXiv:1808.00023

Dabbish L, Stuart C, Tsay J, Herbsleb J (2012) Social coding in github: transparency and collaboration in an open software repository. In: Proceedings of the ACM 2012 conference on computer supported cooperative work, pp 1277–1286

Dastin J (2023) Amazon scraps secret AI recruiting tool that showed bias against women. In: Ethics of data and analytics. Auerbach Publications, pp 296–299

Devlin J, Chang M-W, Lee K, Toutanova K (2019) BERT: pre-training of deep bidirectional transformers for language understanding. In: Proceedings of the 2019 conference of the north American chapter of the association for computational linguistics: human language technologies, vol 1 (Long and Short Papers). Association for Computational Linguistics, Minneapolis, pp 4171–4186

Erlenhov L, de Oliveira Neto FG, Scandariato R, Leitner P (2019) Current and future bots in software development. In: 2019 IEEE/ACM 1st international workshop on bots in software engineering (BotSE). IEEE, pp 7–11

Feng Z, Chatterjee A, Sarma A, Ahmed I (2022) Implicit mentoring: the unacknowledged developer efforts in open source. arXiv:2202.11300

Github rest api documentation. https://docs.github.com/en/rest. Accessed 19 Aug 2022

Harrison DA, Klein KJ (2007) What’s the difference? Diversity constructs as separation, variety, or disparity in organizations. Acad Manag Rev 32(4):1199–1228CrossRef

Hern A. Twitter apologises for ’racist’ image-cropping algorithm

Hill K (2020) Wrongfully accused by an algorithm. In: Ethics of data and analytics. Auerbach Publications, pp 138–142

Huang Y, Leach K, Sharafi Z, McKay N, Santander T, Weimer W (2020) Biases and differences in code review using medical imaging and eye-tracking: genders, humans, and machines. In: Proceedings of the 28th ACM joint meeting on European software engineering conference and symposium on the foundations of software engineering, pp 456–468

Jackson SE, Joshi A (2004) Diversity in social context: a multi-attribute, multilevel analysis of team diversity and sales performance. J Org Behav Int J Indust Occup Org Psychol Behav 25(6):675–702

Jiang J, He J-H, Chen X-Y (2015) Coredevrec: automatic core member recommendation for contribution evaluation. J Comput Sci Technol 30:998–1016CrossRef

Johnson B, Bartola J, Angell R, Keith K, Witty S, Giguere SJ, Brun Y (2020) Fairkit, fairkit, on the wall, who’s the fairest of them all? Supporting data scientists in training fair models. arXiv:2012.09951

Klare BF, Burge MJ, Klontz JC, Vorder Bruegge RW, Jain AK (2012) Face recognition performance: role of demographic information. IEEE Trans Info Forensic Sec 7(6):1789–1801CrossRef

Kubernetes ci bot. https://github.com/k8s-ci-robot. Accessed 19 Aug 2022

Lebeuf C, Storey M-A, Zagalsky A (2017) Software bots. IEEE Softw 35(1):18–23CrossRef

Lebeuf C, Zagalsky A, Foucault M, Storey M-A (2019) Defining and classifying software bots: a faceted taxonomy. In: 2019 IEEE/ACM 1st international workshop on bots in software engineering (BotSE). IEEE, pp 1–6

Ledford H (2019) Millions of black people affected by racial bias in health-care algorithms. Nature 574(7780):608–610CrossRef

Lin T-Y, Goyal P, Girshick R, He K, Dollár P (2017) Focal loss for dense object detection. In: Proceedings of the IEEE international conference on computer vision, pp 2980–2988

Martin N (2018) Are AI hiring programs eliminating bias or making it worse? Forbes. https://www.forbes.com/sites/nicolemartin1/2018/12/13/are-ai-hiring-programs-eliminating-bias-or-making-it-worse/. Accessed 13 Dec 2018

Monperrus M, Urli S, Durieux T, Martinez M, Baudry B, Seinturier L (2019) Repairnator patches programs automatically. Ubiquity 2019(July):1–12CrossRef

Murphy-Hill E, Dicker J, Horvath A, Hodges MM, Egelman CD, Weingart LR, Jaspan C, Green C, Chen N (2023) Systemic gender inequities in who reviews code. Proc ACM Human-Comput Inter 7(CSCW1):1–59CrossRef

Murphy-Hill E, Jaspan C, Egelman C, Cheng L (2022) The pushback effects of race, ethnicity, gender, and age in code review. Commun ACM 65(3):52–57CrossRef

Nadri R, Rodriguez-Perez G, Nagappan M (2021) On the relationship between the developer’s perceptible race and ethnicity and the evaluation of contributions in OSS. IEEE Trans Softw Eng

Nadri R, Rodriguez-Perez G, Nagappan M (2021) Insights into nonmerged pull requests in github: is there evidence of bias based on perceptible race? IEEE Softw 38(2):51–57CrossRef

Namsor tool. https://www.namsor.com/. Accessed 19 Aug 2022

Obermeyer Z, Powers B, Vogeli C, Mullainathan S (2019) Dissecting racial bias in an algorithm used to manage the health of populations. Science 366(6464):447–453CrossRef

Peng Z, Yoo J, Xia M, Kim S, Ma X (2018) Exploring how software developers work with mention bot in Github. In: Proceedings of the sixth international symposium of Chinese CHI, pp 152–155

Rodríguez-Pérez G, Nadri R, Nagappan M (2021) Perceived diversity in software engineering: a systematic literature review. Empir Softw Eng 26(5):1–38CrossRef

Sebo P (2021) Performance of gender detection tools: a comparative study of name-to-gender inference services. J Med Library Assoc JMLA 109(3):414

Steinmacher I, Pinto G, Wiese IS, Gerosa MA (2018) Almost there: a study on quasi-contributors in open source software projects. In: Proceedings of the 40th international conference on software engineering, pp 256–266

Storey M-A, Zagalsky A (2016) Disrupting developer productivity one bot at a time. In: Proceedings of the 2016 24th ACM SIGSOFT international symposium on foundations of software engineering, pp 928–931

Teachman JD (1980) Analysis of population diversity: measures of qualitative variation. Sociol Methods Res 8(3):341–362CrossRef

Tensorflowbutler. https://github.com/tensorflowbutler. Accessed 19 Aug 2022

Terrell J, Kofink A, Middleton J, Rainear C, Murphy-Hill ER, Parnin C (2016) Gender bias in open source: pull request acceptance of women versus men. PeerJ Prepr 4:e1733

Vasilescu B, Posnett D, Ray B, van den Brand MGJ, Serebrenik A, Devanbu P, Filkov V (2015) Gender and tenure diversity in github teams. In: Proceedings of the 33rd annual ACM conference on human factors in computing systems. ACM, New York, pp 3789–3798

Vaswani A, Shazeer N, Parmar N, Uszkoreit J, Jones L, Gomez AN, Kaiser Ł, Polosukhin I (2017) Attention is all you need. Adv Neural Info Process Syst 30

Vscode triage bot. https://github.com/vscode-triage-bot. Accessed 19 Aug 2022

Web scraper - the #1 web scraping tool. https://webscraper.io/. Accessed 19 Aug 2022

Wessel M, De Souza BM, Steinmacher I, Wiese IS, Polato I, Chaves AP, Gerosa MA (2018) The power of bots: characterizing and understanding bots in OSS projects. Proc ACM Human-Comput Inter 2(CSCW):1–19

Wessel M, Serebrenik A, Wiese I, Steinmacher I, Gerosa MA (2020) Effects of adopting code review bots on pull requests to OSS projects. In: 2020 IEEE international conference on software maintenance and evolution (ICSME). IEEE, pp 1–11

Wessel M, Serebrenik A, Wiese I, Steinmacher I, Gerosa MA (2020) What to expect from code review bots on Github? A survey with OSS maintainers. In: Proceedings of the 34th Brazilian symposium on software engineering, pp 457–462

Wessel M, Steinmacher I, Wiese I, Gerosa MA (2019) Should i stale or should i close? An analysis of a bot that closes abandoned issues and pull requests. In: 2019 IEEE/ACM 1st international workshop on bots in software engineering (BotSE). IEEE, pp 38–42

Wessel M, Wiese I, Steinmacher I, Gerosa MA (2021) Don’t disturb me: challenges of interacting with software bots on open source software projects. Proc ACM Human-Comput Inter 5(CSCW2):1–21CrossRef

Wohlin C, Runeson P, Höst M, Ohlsson MC, Regnell B, Wesslén A (2012) Experimentation in software engineering. Springer Science & Business Media, BerlinCrossRef

Wolf MJ, Miller K, Grodzinsky FS (2017) Why we should have seen that coming: comments on microsoft’s tay" experiment," and wider implications. Acm Sigcas Comput Soc 47(3):54–64CrossRef

Wyrich M, Bogner J (2019) Towards an autonomous bot for automatic source code refactoring. In: 2019 IEEE/ACM 1st international workshop on bots in software engineering (BotSE). IEEE, pp 24–28

Zhang BH, Lemoine B, Mitchell M (2018) Mitigating unwanted biases with adversarial learning. In: Proceedings of the 2018 AAAI/ACM conference on AI, ethics, and society, pp 335–340

Zhang JM, Harman M (2021) “ignorance and prejudice” in software fairness. In: 2021 IEEE/ACM 43rd international conference on software engineering (ICSE). IEEE, pp 1436–1447

Title: Diversity in issue assignment: humans vs bots
Authors: Aniruddhan Murali
Gaurav Sahu
Kishanthan Thangarajah
Brian Zimmerman
Gema Rodríguez-Pérez
Meiyappan Nagappan
Publication date: 01-04-2024
Publisher: Springer US
Published in: Empirical Software Engineering / Issue 2/2024
Print ISSN: 1382-3256
Electronic ISSN: 1573-7616
DOI: https://doi.org/10.1007/s10664-023-10424-6

Springer Professional

Abstract

Please log in to get access to your license.

Dont have a licence yet? Then find out more about our products and how to get one now:

Springer Professional "Wirtschaft"

Springer Professional "Technik"

Springer Professional "Wirtschaft+Technik"

Other articles of this Issue 2/2024

Investigating the readability of test code

Studying the impact of risk assessment analytics on risk awareness and code review performance

Quantifying and characterizing clones of self-admitted technical debt in build systems

Evolution of automated weakness detection in Ethereum bytecode: a comprehensive study

Evaluating the impact of flaky simulators on testing autonomous driving systems

When less is more: on the value of “co-training” for semi-supervised software defect predictors

Premium Partner