2. Text Analysis Pipelines

The understanding of natural language is one of the primary abilities that provide the basis for human intelligence. Since the invention of computers, people have thought about how to operationalize this ability in software applications (Jurafsky and Martin 2009). The rise of the internet in the 1990s then made explicit the practical need for automatically processing natural language in order to access relevant information. Search engines, as a solution, have revolutionalized the way we can find such information ad-hoc in large amounts of text (Manning et al. 2008). Until today, however, search engines excel in finding relevant texts rather than in understanding what information is relevant in the texts. Chapter 1 has proposed text mining as a means to achieve progress towards the latter, thereby making information search more intelligent. At the heart of every text mining application lies the analysis of text, mostly realized in the form of text analysis pipelines. In this chapter, we present the basics required to follow the approaches of this book to improve such pipelines for enabling text mining ad-hoc on large amounts of text as well as the state of the art in this respect.

Text mining combines techniques from information retrieval, natural language processing, and data mining. In Sect. 2.1, we first provide a focused overview of those techniques referred to in this book. Then, we define the text analysis processes and pipelines that we consider in our proposed approaches (Sect. 2.2). We evaluate the different approaches based on texts and pipelines from a number of case studies introduced in Sect. 2.3. Finally, Sect. 2.4 surveys and discusses related existing work in the broad context of ad-hoc large-scale text mining.