Top

Published in:

2021 | OriginalPaper | Chapter

22. The Ethics of Web Crawling and Web Scraping in Cybercrime Research: Navigating Issues of Consent, Privacy, and Other Potential Harms Associated with Automated Data Collection

Authors : Russell Brewer, Bryce Westlake, Tahlia Hart, Omar Arauza

Published in: Researching Cybercrimes

Publisher: Springer International Publishing

Activate our intelligent search to find suitable subject content or patents.

search-config

AI-assisted search

Off

Abstract

The abundance of data easily accessible on the internet has made it an appealing domain for conducting social science research. Because of the volume of data available, automated collection software, such as web crawlers and web scrapers, are increasingly being deployed. Ethical guidelines that have developed over the decades, however, are not always directly applicable to online research. As a result, there can be ambiguity as to when and how informed consent needs to be obtained, how privacy can be protected, and what potential harms websites, their owners, and even researchers could experience, and how to address them. In this chapter, we argue that automated data collection practices need not be unethical, provided special care is taken by researchers to acknowledge and explicitly address the complexities surrounding informed consent, privacy, and a myriad of other risks and potential harms (to subjects, websites, and researchers).

Dont have a licence yet? Then find out more about our products and how to get one now:

Springer Professional "Wirtschaft+Technik"

Online-Abonnement

Mit Springer Professional "Wirtschaft+Technik" erhalten Sie Zugriff auf:

über 102.000 Bücher
über 537 Zeitschriften

aus folgenden Fachgebieten:

Automobil + Motoren
Bauwesen + Immobilien
Business IT + Informatik
Elektrotechnik + Elektronik
Energie + Nachhaltigkeit
Finance + Banking
Management + Führung
Marketing + Vertrieb
Maschinenbau + Werkstoffe
Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

inform now

Springer Professional "Technik"

Online-Abonnement

Mit Springer Professional "Technik" erhalten Sie Zugriff auf:

über 67.000 Bücher
über 390 Zeitschriften

aus folgenden Fachgebieten:

Automobil + Motoren
Bauwesen + Immobilien
Business IT + Informatik
Elektrotechnik + Elektronik
Energie + Nachhaltigkeit
Maschinenbau + Werkstoffe

Jetzt Wissensvorsprung sichern!

inform now

Springer Professional "Wirtschaft"

Online-Abonnement

Mit Springer Professional "Wirtschaft" erhalten Sie Zugriff auf:

über 67.000 Bücher
über 340 Zeitschriften

aus folgenden Fachgebieten:

Bauwesen + Immobilien
Business IT + Informatik
Finance + Banking
Management + Führung
Marketing + Vertrieb
Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

inform now

previous chapter Researching Crime and Deviance in Southeast Asia: Challenges and Ethics When Using Online Data

next chapter Does the Institution Have a Plan for That? Researcher Safety and the Ethics of Institutional Responsibility

Abbass, Z., Ali, Z., Ali, M., Akbar, B., & Saleem, A. (2020). A framework to predict social crime through Twitter tweets by using machine learning. 2020 IEEE 14th International Conference on Semantic Computing (ICSC), 363–368.

Alim, S. (2013). Automated data extraction from online social network profiles: Unique ethical challenges for researchers. International Journal of Virtual Communities and Social Networking (IJVCSN),5(4), 24–42.CrossRef

Alim, S. (2014). An initial exploration of ethical research practices regarding automated data extraction from online social media user profiles. First Monday, 19(7).

Askitas, N., & Zimmermann, K. F. (2015). The Internet as a data source for advancement in social sciences. International Journal of Manpower,36(1), 2–12.CrossRef

Barrera, V., Malm, A., Décary-Hétu, D., & Munksgaard, R. (2019). Size and scope of the tobacco trade on the darkweb. Global Crime,20(1), 26–44.CrossRef

Bent-Goodley, T. B. (2007). Teaching social work students to resolve ethical dilemmas in domestic violence. Journal of Teaching in Social Work,27(1–2), 73–88.CrossRef

Bouchard, M., Joffres, K., & Frank, R. (2014). Preliminary analytical considerations in designing a terrorism and extremism online network extractor. In V. Mago & V. Dabbaghian (Eds.), Computational models of complex systems (pp. 171–184). Springer.

Bouwman, H., de Reuver, M., Heerschap, N., & Verkasalo, H. (2013). Opportunities and problems with automated data collection via smartphones. Mobile Media & Communication,1(1), 63–68.CrossRef

Bulakh, V., & Gupta, M. (2015). Characterizing credit card black markets on the web. Proceedings of the 24th International Conference on World Wide Web, 1435–1440.

Broadhurst, R., Ball, M., & Jiang, C. (2020). Availability of COVID-19 related products on Tor darknet markets. Statistical Bulletin, no. 24. Canberra: Australian Institute of Criminology.

Broadhurst, R., Ball, M., & Trivedi, H. (2020). Fentanyl availability on darknet markets. Trends & issues in crime and criminal justice, no. 590. Canberra: Australian Institute of Criminology.

Broadhurst, R., Lord, D., Maxim, D., Woodford-Smith, H., Johnston, C., Chung, H.W., et al. (2018). Malware trends on Darknet crypto-markets: Research review. ANU Cybercrime Observatory. Canberra.

Calvey, D. (2013). Covert ethnography in criminology: A submerged yet creative tradition. Current Issues in Criminal Justice,25(1), 541–550.CrossRef

Capriello, A., & Rossi, P. (2013). Spidering scripts for opinion monitoring. In H. Rahman & I. Ramos (Eds.), Ethical data mining applications for socio-economic development. IGI Global.

Chang, L. Y. C. (2018). Internet vigilantism co-production of security and compliance in the digital age. In Brewer R. (Ed.), criminal justice and regulation revisited: Essays in honour of Peter Grabosky. Routledge.

Chiauzzi, E., & Wicks, P. (2019). Digital trespass: Ethical and terms-of-use violations by researchers accessing data from an online patient community. Journal of Medical Internet Research, 21(2).

Christin, N. (2013). Traveling the Silk Road: A measurement analysis of a large anonymous online marketplace. Proceedings of the 22nd International Conference on World Wide Web. International World Wide Web Conferences Steering Committee, 213–224.

Crosignani, M., Macchiavelli, M., & Silva, A. F. (2020). Pirates without borders: The propagation of cyberattacks through firms’ supply chains. SSRN Electronic Journal.

Cunliffe, J., Martin, J., Décary-Hétu, D., & Aldridge, J. (2017). An island apart? Risks and prices in the Australian cryptomarket drug trade. The International Journal of Drug Policy,50, 64–73.CrossRef

Décary-Hétu, D., & Aldridge, J. (2015). Sifting through the net: Monitoring of online offenders by researchers. European Review of Organised Crime,2(2), 122–141.

Décary-Hétu, D., & Dupont, B. (2013). Reputation in a dark network of online criminals. Global Crime,14(2–3), 175–196.CrossRef

Décary-Hétu, D., & Quessy-Doré, O. (2017). Are repeat buyers in cryptomarkets loyal customers? Repeat business between dyads of cryptomarket vendors and users. American Behavioral Scientist,61(11), 1341–1357.CrossRef

Décary-Hétu, D., Dupont, B., & Fortin, F. (2014). Policing the hackers by hacking them: Studying online deviants in irc chat rooms. In A. J. Masys (Ed.), Networks and network analysis for defence and security. Springer.

Demant, J., Munksgaard, R., & Houborg, E. (2018). Personal use, social supply or redistribution? Cryptomarket demand on Silk Road 2 and Agora. Trends in Organized Crime,21(1), 42–61.CrossRef

Demant, J., Munksgaard, R., Décary-Hétu, D., & Aldridge, J. (2018). Going local on a global platform: A critical analysis of the transformative potential of cryptomarkets for organized illicit drug crime. International Criminal Justice Review,28(3), 255–274.CrossRef

Fournier, R., Cholez, T., Latapy, M., Chrisment, I., Magnien, C., Festor, O., & Daniloff, I. (2014). Comparing pedophile activity in different P2P systems. Social Sciences,3(3), 314–325.CrossRef

Frank, R., & Mikhaylov, A. (2020). Beyond the ‘Silk Road’: Assessing illicit drug marketplaces on the public web. In M. A. Tayebi., U. Glässer, & D. B. Skillicorn (Eds.), Open source intelligence and cyber crime. Springer.

Freelon, D. (2018). Computational research in the post-API Age. Political Communication,35(4), 665–668.CrossRef

Fussell, S. (2019). You no longer own your face. The Atlantic. Available at: https://www.theatlantic.com/technology/archive/2019/06/universities-record-students-campuses-research/592537/.

García-Retuerta, D., Bartolomé, Á., Chamoso, P., & Corchado, J. M. (2019). Counter-terrorism video analysis using hash-based algorithms. Algorithms, 12(5).

Gata, W., & Bayhaqy, A. (2020). Analysis sentiment about islamophobia when Christchurch attack on social media. Telkomnika,18(4), 1819–1827.CrossRef

Giles, C., Sun, Y., & Councill, I. (2010). Measuring the web crawler ethics. Proceedings of the 19th International Conference on World Wide Web, 1101–1102.

Gök, A., Waterworth, A., & Shapira, P. (2015). Use of web mining in studying innovation. Scientometrics,102(1), 653–671.CrossRef

Gold, Z., & Latonero, M. (2018). Robots welcome? Ethical and legal consideration for web crawling and scraping. Washington Journal for Law, Technology & Arts,13(3), 275–312.

Gorro, K. D., Sabellano, M. J. G., Maderazo, C. V., Ceniza, A. M., & Gorro, K. (2017). Exploring Facebook for sharing crime experiences using selenium and support vector machine. Proceedings of the 2017 International Conference on Information Technology, 218–222.

Gueta, K., Eytan, S., & Yakimov, P. (2020). Between healing and revictimization: The experience of public self-disclosure of sexual assault and its perceived effect on recovery. Psychology of Violence,10(6), 626–637.CrossRef

Hand, D. J. (2018). Aspects of data ethics in a changing world: Where are we now? Big Data,6(3), 176–190.CrossRef

Hansen, A. L. S., Li, A., Joly, D., Mekaru, S., & Brownstein, J. S. (2012). Digital surveillance: A novel approach to monitoring the illegal wildlife trade. PLoS ONE,7(12), e51156.CrossRef

Hardy, W., Krawczyk, M., & Tyrowicz, J. (2014). Internet piracy and book sales: A field experiment. Faculty of Economic Sciences, University of Warsaw Working Papers, 23(140), 1–22.

Harlow, L. L., & Oswald, F. L. (2016). Big data in psychology: Introduction to the special issue. Psychological Methods,21(4), 447–457.CrossRef

Hayes, D. R., Cappa, F., & Cardon, J. (2018). A framework for more effective dark web marketplace investigations. Information (basel),9(8), 186–204.CrossRef

Holt T. J., Smirnova, O., Strumsky, D., & Kilger, M. (2014). Advancing research on hackers through social network data. In C. D. Marcum & G. E. Higgins (Eds.), Social networking as a criminal enterprise. Taylor Francis.

Hooley, T., Marriott, J., & Wellens, J. (2012). What is online research? Using the Internet for social science research. Bloomsbury Academic.

Israel, M. (2004). Strictly confidential? Integrity and the disclosure of criminological and socio-legal research. British Journal of Criminology,44(5), 715–740.CrossRef

Jacques, S., Garstka, K., Hviid, M., & Street, J. (2018). An empirical study of the use of automated anti-piracy systems and their consequences for cultural diversity. SCRIPT-Ed,15(2), 277–312.CrossRef

Joffres, K., Bouchard, M., Frank, R., & Westlake, B. G. (2011). Strategies to disrupt online child pornography networks. 2011 European Intelligence and Security Informatics Conference, 163–170. IEEE.

Keyvanpour, M. R., Javideh, M., & Ebrahimi, M. R. (2011). Detecting and investigating crime by means of data mining: A general crime matching framework. Procedia Computer Science,3, 872–880.CrossRef

Kusz, J., & Bouchard, M. (2020). Nymphet or lolita? A gender analysis of online child pornography websites. Deviant Behavior,41(6), 805–813.CrossRef

Landers, R., Brusso, R., Cavanaugh, K., & Collmus, A. (2016). A primer on theory-driven web scraping: Automatic extraction of big data from the Internet for use in psychological research. Psychological Methods,21(4), 475–492.CrossRef

Latapy, M., Magnien, C., & Fournier, R. (2013). Quantifying paedophile activity in a large P2P system. Information Processing & Management,49(1), 248–263.CrossRef

Lyu, Y., Xie, J., & Xie, B. (2020). The attitudes of Chinese online users towards movie piracy: A content analysis. In A. Sundqvist, G. Berget, J. Nolin, & K. Skjerdingstad (Eds.), Sustainable digital communities (pp. 169–185). Springer.

Macdonald, M., Frank, R., Mei, J., & Monk, B. (2015). Identifying digital threats in a hacker web forum. Proceedings of the 2015 IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining 2015, 926–933.

Magdy, W., Elkhatib, Y., Tyson, G., Joglekar, S., Sastry, N. (2017). Fake it till you make it: Fishing for catfishes. Proceedings of the 2017 IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining 2017, 497–504.

Martin, J., & Christin, N. (2016). Ethics in cryptomarket research. International Journal of Drug Policy,35, 84–91.CrossRef

Martin, J., Cunliffe, J., Décary-Hétu, D., & Aldridge, J. (2018a). Effect of restricting the legal supply of prescription opioids on buying through online illicit marketplaces: Interrupted time series analysis. British Medical Journal,361, 1–7.

Martin, J., Cunliffe, J. D., Décary-Hétu, D., & Aldridge, J. (2018b). The international darknet drugs trade-a regional analysis of cryptomarkets. Australasian Policing,10(3), 25–29.

McAlister, R. (2015). Webscraping as an investigation tool to identify potential human trafficking operations in Romania. Proceedings of the ACM Web Science Conference, 1–2.

McAuliffe, D. (2005). I’m still standing: Impacts and consequences of ethical dilemmas for social workers in direct practice. Journal of Social Work Values and Ethics,2(1), 1–10.

McCann, M. (2016). The smartphones study: An analysis of disciplinary differences in research ethics committee responses to phone app-based automated data collection. European Journal of Public Health, 26(suppl. 1).

Mei, J., & Frank, R. (2015). Sentiment crawling: Extremist content collection through a sentiment analysis guided web-crawler. IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining (ASONAM),2015, 1024–1027.

Menczer, F. (2011). Web crawling. In B. Liu (Ed.), Web data mining: Exploring hyperlinks, contents, and usage data, 311 Data-Centric Systems and Applications (pp. 311–362). Springer.

Millstein, K. (2000). Confidentiality in direct social-work practice: Inevitable challenges and ethical dilemmas. Families in Society,81(3), 270–282.CrossRef

Moeller, K., Munksgaard, R., & Demant, J. (2020). Illicit drug prices and quantity discounts: A comparison between a cryptomarket, social media, and police data. The International Journal of Drug Policy (online first).

Olston, C., & Najork, M. (2010). Web crawling. Foundations and Trends in Information Retrieval,4(3), 175–246.CrossRef

O’Reilly, S. (2007). Nominative fair use and Internet aggregators: Copyright and trademark challenges posed by bots, web crawlers and screen-scraping technologies. Loyola Consumer Law Review,19(3), 273–288.

Ozalp, S., Williams, M. L., Burnap, P., Liu, H., & Mostafa, M. (2020). Antisemitism on Twitter: Collective efficacy and the role of community organisations in challenging online hate speech. Social Media + Society, 6(2), 1–20.

Pannu, M., Kay, I., & Harris, D. (2018). Using dark web crawler to uncover suspicious and malicious websites. International Conference on Applied Human Factors and Ergonomics (pp. 108–115). Springer.

Paquet-Clouston, M., Décary-Hétu, D., & Morselli, C. (2018). Assessing market competition and vendors’ size and scope on AlphaBay. International Journal of Drug Policy,54, 87–98.CrossRef

Pastrana, S., Thomas, D. R., Hutchings, A., & Clayton, R. (2018). Crimebb: Enabling cybercrime research on underground forums at scale. Proceedings of the 2018 World Wide Web Conference, 1845–1854.

Pitman, B., Ralph, A. M., Camacho, J., & Monk-Turner, E. (2019). Social media users’ interpretations of the Sandra Bland arrest video. Race and Justice,9(4), 479–497.CrossRef

Scrivens, R., Davies, G., & Frank, R. (2017). Searching for signs of extremism on the web: An introduction to Sentiment-based Identification of Radical Authors. Behavioral Sciences of Terrorism and Political Aggression,10(1), 39–59.CrossRef

Scrivens, R., Gaudette, T., Davies, G., & Frank, R. (2019). Searching for extremist content online using the dark crawler and sentiment analysis. In M. Deflem & D. M. D Silva (Eds.), Methods of criminology and criminal justice research (Sociology of Crime, Law and Deviance). Emerald Publishing Limited.

Shavitt, Y., & Zilberman, N. (2013). On the presence of child sex abuse in BitTorrent networks. IEEE Internet Computing,17(3), 60–66.CrossRef

Sidhu, N., & Srinivasraghavan, J. (2016). Ethics and medical practice: Why psychiatry is unique. Indian Journal of Psychiatry,58(6), 199–202.CrossRef

Silva, E., Till, A., & Adshead, G. (2017). Ethical dilemmas in psychiatry: When teams disagree. Bjpsych Advances,23(4), 231–239.CrossRef

Solberg, L. B. (2010). Data mining on Facebook: A free space for researchers or an IRB nightmare? University of Illinois Journal of Law, Technology & Policy,2, 311–343.

Soska, K., & Christin, N. (2015). Measuring the longitudinal evolution of the online anonymous marketplace ecosystem. In USENIX Security Symposium (USENIX Security), 33–48.

Sun, Y., Councill, I. G., & Giles, C. L. (2010). The ethicality of web crawlers. 2010 IEEE/WIC/ACM International Conference on Web Intelligence and Intelligent Agent Technology, 1, 668–675.

Subramaniyaswamy, V., Logesh, R., Abejith, M., Umasankar, S., & Umamakeswari, A. (2017). Sentiment analysis of tweets for estimating criticality and security of events. Journal of Organizational and End User Computing,29(4), 51–71.CrossRef

Tavani, H. T. (1999). Informational privacy, data mining, and the Internet. Ethics and Information Technology,1(2), 137–145.CrossRef

Thelwall, M., & Stuart, D. (2006). Web crawling ethics revisited: Cost, privacy, and denial of service. Journal of the American Society for Information Science and Technology,57(13), 1771–1779.CrossRef

Thomas, D. M., & Mathur, S. (2019). Data analysis by web scraping using python. 2019 3rd International conference on Electronics, Communication and Aerospace Technology (ICECA), 450–454.

Tsatsou, P. (2014). Research and the Internet: Fast-growing Internet research. In P. Tsatsou (Ed.), Internet studies: Past, present and future directions. Ashgate Publishing Ltd.

Turk, K., Pastrana, S., & Collier, B. (2020). A tight scrape: Methodological approaches to cybercrime research data collection in adversarial environments. Workshop on Actors in Cybercrime Operations, 428–437.

Westlake, B. G., & Bouchard, M. (2016a). Criminal careers in cyberspace: Examining website failure within child exploitation networks. Justice Quarterly,33(7), 1154–1181.CrossRef

Westlake, B. G., & Bouchard, M. (2016b). Liking and hyperlinking: Examining reciprocity and diversity in online child exploitation network communities. Social Science Research,59, 23–36.CrossRef

Westlake, B. G., Bouchard, M., & Frank, R. (2011). Finding the key players in online child exploitation networks. Policy and Internet,3(2), 1–32.CrossRef

Westlake, B. G., Bouchard, M., & Frank, R. (2012). Comparing methods for detecting child exploitation content online. European Intelligence and Security Informatics Conference, 156–163.

Westlake, B. G., Bouchard, M., & Frank, R. (2017). Assessing the validity of automated webcrawlers as data collection tools to investigate online child sexual exploitation. Sexual Abuse,29(7), 685–708.CrossRef

Westlake, B. G., & Frank, R. (2016). Seeing the forest through the trees: Identifying key players in online child sexual exploitation distribution networks. In T. Holt (Ed.), Cybercrime through an interdisciplinary lens. New York: Routledge.

Wilson, R. E., Gosling, S. D., & Graham, L. T. (2012). A review of Facebook research in the social sciences. Perspectives on Psychological Science,7(3), 203–220.CrossRef

Xin, Y., & Cai, T. (2018). Child trafficking in China: Evidence from sentencing documents. International Journal of Population Studies,4(2), 1–10.

Xu, Q., Cai, M., & Mackey, T. K. (2020). The illegal wildlife digital market: An analysis of Chinese wildlife marketing and sale on Facebook. Environmental Conservation,47(3), 206–212.CrossRef

Title: The Ethics of Web Crawling and Web Scraping in Cybercrime Research: Navigating Issues of Consent, Privacy, and Other Potential Harms Associated with Automated Data Collection
Authors: Russell Brewer
Bryce Westlake
Tahlia Hart
Omar Arauza
Publisher: Springer International Publishing
Book: Researching Cybercrimes
Print ISBN: 978-3-030-74836-4

Electronic ISBN: 978-3-030-74837-1

Copyright Year: 2021
DOI: https://doi.org/10.1007/978-3-030-74837-1_22

Springer Professional

Abstract

Please log in to get access to your license.

Dont have a licence yet? Then find out more about our products and how to get one now:

Springer Professional "Wirtschaft+Technik"

Springer Professional "Technik"

Springer Professional "Wirtschaft"

Premium Partner