Big Data Factories

The chapter argues that accessibility and flexibility are the two principles and practices that can bring big data projects the closest to a data factory ideal. The chapter elaborates on the necessity of these two principles, offering a reasoned explanation for their value in context. Using two big data social scientific research projects as a springboard for conversation the chapter highlights both the advantages and the practical limits within which accessibility and flexibility operate. The authors avoid both utopian and dystopian tropes about big data approaches. In addition, they offer a critical feminist discussion of big data collaboration. Of particular interest are also the manner in which specific characteristics of big data projects, especially volume and velocity, may affect multidisciplinary collaborations.

While online behavior creates an enormous amount of digital data that can be the basis for social science research, to date, the science has been conducted piecemeal, one Internet address at a time, often without social or scholarly impact beyond the site’s own stakeholders. Scientists lack the tools, methods, and practices to combine, compare, contrast, and communicate about online behavior across Internet addresses or over time. In response, we are building the infrastructure for computational social scientists, social scientists, and citizens to make corresponding advances in our understanding of online human interactions. In this chapter, we present our effort to (1) specify the Open Community Data Exchange (OCDX) metadata standard to describe datasets, (2) introduce concepts from the data curation lifecycle to social computing research, and (3) describe candidate infrastructure for creating, editing, viewing, sharing, and analyzing manifests.

The explosion of data available from online systems such as social media is creating a wealth of trace data, that is, data that record evidence of human activity. The volume of data available offers great potential to advance social and behavioural science research. However, the data are of a very different kind than more conventional social and behavioural science data, posing challenges to use. This paper adopts a data framework from Earth observation science and applies it to trace data to identify possible issues in analysing trace data. Application of the framework also reveals issues for sharing and reusing data.

This chapter describes ten drivers of the adoption of open source software that enables e-research in data factories for open innovations. More specifically, the chapter discusses the emerging phenomena of big data and e-research, along with their various defining characteristics. Then the chapter makes a case for the importance of understanding the adoption of open source software for processing and harnessing big data. In other words, big data which remain in the raw form will continue to be big data with hidden insights uncovered without the adoption of appropriate software. Open source software applications, along with the larger concept of cyberinfrastructure, play a critical role in our ability to optimize the full potential of big data. The chapter also includes critical questions community stakeholders should keep in mind when promoting the diffusion and dissemination of good software applications that will support data factories for open innovations.

A growing literature looks at the processes that drive the most successful collaborative communities via wiki sites. One key issue that has emerged in the last few decades is the nature of the social order that informs them. Yet, researchers struggle with a fragmentary approach to aligning and comparing the social processes that generate social order. Furthermore, social order is itself under dispute, with some claiming that online collaborative communities represent a completely new form of organization (Benkler, The wealth of networks: How social production transforms markets and freedom. New Haven: Yale University Press, 2006) and others arguing that the traditional concept of hierarchical order itself is obsolete and needs to be abandoned (Brafman and Beckstrom, The starfish and the spider: The unstoppable power of leaderless organizations. New York: Penguin, 2006). In this chapter, we propose a theoretical strategy complete with some measurements that may make the problem of social order more tractable and comparisons across wiki and, more generally, online collaborative spaces and datasets more comparable. We strive to go beyond surface-level characteristics, mining key qualities that explain the success of wikis and other types of online communities. Our approach and the analytic framework we have developed can be used as a way to think about and experiment with not only the concept of order in the abstract but also with new tools, such as our Visible Effort. In what follows, we will detail our approach, complete with implementation recommendations for the measures proposed to capture social order.

In 2004 a collaborative research team based at Syracuse University and Elon University began collecting and sharing data in order to understand how free/libre open source software (FLOSS) is made. Embodying some of the same FLOSS ethos, this team created a public-facing repository for their own data and analyses and encouraged other researchers to use it and contribute to it. This chapter tells the story of how the FLOSSmole project began, where the data comes from and what we have learned from it, and how the project has grown and changed over the years. In addition to capturing snapshots of the current state of the FLOSS landscape, FLOSSmole also serves as a mirror to the larger FLOSS ecosystem, since changes in FLOSSmole’s mission and goals over the years necessarily reflect some of the cultural and technological changes taking place in FLOSS itself. As such, FLOSSmole will continue to face many challenges in the future, including the continual need to provide broader access and more sophisticated and relevant data and analyses and to do all this in a way that is sustainable and community driven.

The authors explore the intellectual and pedagogical implications of big data visualizations. Representing data visually implies simplifying and essentializing information. However, the selective nature of information visualization can lend itself to lies, manipulations, and misleading information. To avoid these pitfalls, data analysts should focus and embrace specific principles and practices that aim to represent complete, contextualized, comparable, and scalable information in a way that reveals rather than isolates the viewer and the problem at hand from the problem space it reflects.

Nearly every published discussion of data science education begins with a reflection on an acute shortage in labor markets of professional data scientists with the skills necessary to extract business value from burgeoning datasets created by online communities like Facebook, Twitter, and LinkedIn. This model of data science—professional data scientists mining online communities for the benefit of their employers—is only one possible vision for the future of the field. What if everybody learned the basic tools of data science? What if the users of online communities—instead of being ignored completely or relegated to the passive roles of data producers to be shaped and nudged—collected and analyzed data about themselves? What if, instead, they used data to understand themselves and communicate with each other? What if data science was treated not as a highly specialized set of skills but as a basic literacy in an increasingly data-driven world?