An Introduction to Web Mining

Table of Contents

1. Frontmatter

2. Context, Relevance, and the Basics

   1. Frontmatter

   2. Chapter 1. Introduction

      Ulrich Matter

      Abstract

      The diffusion of the Internet has led to a stark increase in the availability of digital data describing all kind of everyday human activities (Edelman, J Econ Perspect 26(2):189–206, 2012; Einav and Levin, Science 346(6210):1243089–1–1243089–6, 2014; Matter and Stutzer, PLOS ONE 10(7):1–21, 2015). The dawn of such web-based big data offers various opportunities for empirical research in economics and the social sciences in general. While web (data) mining has for many years rather been a discipline within computer science with a focus on web application development (such as recommender systems and search engines), the recent rise in well-documented open-source tools to automatically collect data from the Web makes this endeavor more accessible for researchers and data analytics practitioners without a background in web technologies.

   3. Chapter 2. The Internet as a Data Source

      Ulrich Matter

      Abstract

      In this chapter, we will start with some ideas of why the Internet serves as a powerful data source for researchers and business/data science practitioners. You will see that the underlying technological layer of the Internet, as well as the social/human layer, reflected in the Web’s content.

3. Web Technologies and Automated Data Extraction

   1. Frontmatter

   2. Chapter 3. Web 1.0 Technologies: The Static Web

      Ulrich Matter

      Abstract

      In this chapter, we will explore some of the foundational aspects of the static web—how servers and clients communicate, the basics of HTTP requests, and the role that HTML and CSS play in structuring and presenting web content. This introduction clarifies why these concepts matter for web mining and how they set the stage for more complex, dynamic setups covered in later chapters.

   3. Chapter 4. Web Scraping: Data Extraction from Websites

      Ulrich Matter

      Abstract

      This chapter focuses on extracting data from (mostly Web 1.0) webpages, building on the basics of static web technologies introduced in the previous chapter. It begins by presenting the basic toolbox used for web mining throughout this book and then proceeds with two detailed tutorials on the first steps of data extraction.

   4. Chapter 5. Web 2.0 Technologies: The Programmable/Dynamic Web

      Ulrich Matter

      Abstract

      The web technologies discussed in the previous chapters have been around (at least in a very similar form) right since the beginning of the World Wide Web. While they are still the basis of most websites in one way or the other, new technologies are constantly extending and changing how data is presented and exchanged over the Web. Similar to the older, basic web technologies like HTTP and HTML, we do not have to master all these new technologies in every detail in order to productively engage with the automated extraction of data. In fact, such an endeavor would require a whole curriculum on diverse computer languages, including Python, JavaScript, and SQL.

   5. Chapter 6. Extracting Data from the Programmable Web

      Ulrich Matter

      Abstract

      In the previously introduced new web application model powering dynamic/interactive websites, the data integrated in the webpage on the client side is usually exchanged between server and client in standardized formats such as XML and JSON. The concept of web APIs is based on the same idea and specifically aimed at facilitating the integration of web data in various applications/websites over the Internet. Web APIs thus serve as data hubs providing data to various web applications which might further process the data and finally display it in a graphical user interface (e.g., a webpage). Large parts of the explicit exchange of data over the Internet through such applications are thus happening “programmatically” and not “manually” by users explicitly requesting data by typing a URL into the browser bar.

   6. Chapter 7. Data Extraction from Dynamic Websites

      Ulrich Matter

      Abstract

      The programmable web offers new opportunities for web developers to integrate and share data across different applications over the Web. In recent chapters, we have learned about some of the key technological aspects of this programmable web and dynamic websites.

4. Advanced Topics in Web Mining

   1. Frontmatter

   2. Chapter 8. Web Mining Programs

      Ulrich Matter

      Abstract

      Web data mining, in its most basic form, is the creation of programs that automatically download webpages. The majority of the simpler programs and scripts we have implemented thus far can be referred to as “web scrapers” or simply “scrapers,” referring to the task of automatically extracting (scraping) specific data from a webpage. Scrapers are typically designed to extract data from a specific website or a clearly defined set of websites (for example, scrape the headlines from all Swiss newspaper homepages).

   3. Chapter 9. Crawler Implementation

      Ulrich Matter

      Abstract

      The simple breadth-first crawler implemented in the previous chapter can be extended and refined in various ways. Depending on the data extraction tasks foreseen for the crawler, a lot of the refinement might go into the “data part” which can be extended with all kind of scraping and data manipulation techniques (extract only specific components of a webpage, store all the text contained in the crawled webpage after removing all HTML tags, etc.). Beyond the implementation issues discussed in the context of web scraping, there are some additional crawler-specific implementation issues to be considered.

   4. Chapter 10. Appearance and Authentication

      Ulrich Matter

      Abstract

      In simple terms, a big part of what makes the scraping/crawling of modern websites difficult is the need to implement the crawling/scraping procedures in such a way that the scraping instance(s) appears and acts as if they were human. Without addressing this issue so directly, the previous Chaps. 7 to 8 provided insights into several techniques that work toward “human-like” appearance and actions (such as using an actual browser to interact with the website for scraping). However, there are still many websites that can detect and block automated traffic, even when using a browser. These websites may use techniques such as CAPTCHAs or blocking certain user agents or IP addresses. While these measures protect websites from too extensive automated traffic, they can also hinder the more polite web mining efforts in the context of legitimate data collection for academic research purposes. In addition, and rather related to human-like actions than human-like appearance, some websites require users to log in to access certain content, which can be a challenge for web scrapers. Of course, the idea here is that you are in a situation where you have legitimate access to the website and thus have the right credentials to log in. However, if you want to automate such a process, you need to be able to handle the login process in your web scraping code.

   5. Chapter 11. Scaling Web Mining in the Cloud

      Ulrich Matter

      Abstract

      Web mining projects often start small—scraping a few pages, processing manageable amounts of data. However, as projects grow, you might need to scale up your data collection and processing capabilities. Cloud computing offers flexible solutions for scaling the kind of web mining pipelines we have discussed in previous chapters. This can include simple storage solutions to complex distributed systems.

   6. Chapter 12. AI Tools for Web Mining: Overview and Outlook

      Ulrich Matter

      Abstract

      In the previous four chapters, we have dealt with many potential challenges for practical modern web mining. In doing so, we have looked into several tools (like browser automation, fingerprint managers, and proxy servers) that help us address these challenges, often in the form of additional software layers that we can integrate into our R workflows. Without the ambition of complete coverage of all relevant tools, the aim of this final chapter of this part of the book is to provide an overview of additional tools that might be of interest for the web mining inclined. And, without a doubt, AI tools are the one major modern web mining theme we have (intentionally) ignored so far.

5. Ethical, Legal, and Scientific Rigor

   1. Frontmatter

   2. Chapter 13. Ethics and Legal Considerations

      Ulrich Matter

      Abstract

      This chapter provides some context on ethical and legal issues, and how these considerations fit into the broader practice of responsible web mining. In the following sections, we will examine how to consider server resource limitations, the business interests of the websites scraped, and user privacy. By understanding these themes, readers can better anticipate where potential conflicts may arise and learn how to minimize harm while conducting their web mining projects. We will also discuss how these guidelines connect to real-world legal precedents and ethical considerations. Before diving into it, please observe the following disclaimer.

   3. Chapter 14. Web Mining and Scientific Rigor

      Ulrich Matter

      Abstract

      In this final chapter, we discuss how reproducible and replicable research is facilitated by automated data collection, as well as the pitfalls—like selection bias and representativity issues—that one must consider when using scraped web data. Finally, we outline best practices for organizing code and data for transparency and integrity, before concluding with an overview of sampling concerns and external validity.

6. Backmatter