Information Systems Management in the Big Data Era

What do we mean by the term “Big Data”? Far from being self-explanatory, it often seems that big data can mean different things to different people or companies, dependent upon their perspectives. In this chapter we examine what big data is and how it can be defined, before investigating the convergence of technological advances that are driving big data. Given the increasing media coverage surrounding big data and analytics, we then explore some of the stories behind the hype. It may be argued big data reached the peak of the “hype cycle” towards the end of 2013, carried by overly inflated expectations – followed by a growing sense of disillusionment in 2014 as some early trials ended in highly publicised failure and media interest waned. However, in this chapter we suggest that despite the hype cycle, big data will, ultimately, result in major changes to both social and business life – though this change may takes decades to complete. Finally we introduce a framework which allows all aspects of big data to be represented and explored holistically: the 9S-framework.

Search any bookstore, online or high street, and you will find a surplus of books on business strategy. Some are ‘how to’ recipe-style books, others are written by various captains of industry urging you to do it “my way”. This Chapter gives an introductory view of strategy but focuses primarily on the significance of big data for the traditional strategy development process and its potential to inspire new, information-driven strategies. Big data is defined by three attributes; Volume, Velocity and Variety, all of which have consequences for strategic analysis and strategic direction. The promise of detailed data granularity, extensive transparency and rapid information transfer makes innovative business models increasingly viable and offers unique dimensions of competitive advantage. However, integrating big data into cohesive corporate strategy requires a strong technical environment and accomplishing this, while maintaining strategic flexibility, presents a number of dilemmas. Accordingly, this chapter also examines the perennial challenge of aligning IT/IS strategy with business strategy and discusses some of the tools and techniques used to achieve this.

Organisational structure, much like the 9Ss framework we propose in Chap. 1, comprises both ‘hard’ and ‘soft’ elements. Like an iceberg, the ‘hard’ elements of structure; the hierarchical chart, the job descriptions and spans of control, are easy to see in that they are explicit and, often, in the public domain. On the contrary, the ‘soft’ elements, those that make up the organisational culture, are often submerged beneath company routine, rituals, symbols and stories. Often it is these hidden elements of organisational structure that determine how a company ‘works’ and how it responds to change. Management can be thought of as a blend of five interrelated functions; commanding, controlling, coordinating, organising and planning and of these five functions, four are associated with organisational structure. But how is structure created? How does it evolve? And what are its implications for big data? In this chapter we examine the role of company founders on establishing and developing both formal and informal structure. We investigate in what way the internal organisational environment and the wider business environment shape how a company responds to change. Finally, by reflecting on some of failed enterprise resource planning (ERP) system implementations of the last 25 years, we examine what lessons can be learned for the roll-out of big data projects.

Style is about how things are done. In this Chapter we examine the process of management (that is, how management is ‘done’) and investigate the implications of big data for existing management styles. Today’s managers view big data in one of three ways; some have anticipated the impact of big data and have positioned themselves (and their companies) for the opportunities; some recognise the threats that big data brings but have buried their heads in the sand and, finally, some see neither opportunity or threat and believe that the role of the manager in an era of big data will be unchanged. In this chapter we argue that big data disrupts the role of management in four ways; it requires that all staff, including managers, become data-literate, it questions existing notions of experience and expertise, it disrupts how decisions are made (an important lever of management control) and it defines the role and responsibilities of the data Scientist. Big data also affects the balance between ‘management’ and ‘leadership’ within an organisation. In this chapter we examine the difference between management and leadership, how the business environment shapes the balance between the two and why, arguably, leadership is more important than management for big data.

In this chapter we examine how the demands of big data are changing the roles of staff and the skills required to fulfil those roles. We start by examining “the sexiest job of the 21st century”, that of the Data Scientist. What, precisely, is a Data Scientists? What do they do? And where do they fit within the organisation? At first sight, the skills needed for Big Data are both broad and deep and individuals with this highly desirable skill set are already in huge demand. But do these ‘Data Scientists’ actually ‘do’ science? Indeed, does the company really want these employees to be ‘scientists’? Perhaps, more importantly, do individuals with this huge set of skills actually exist? Taking a practical view of the big data skill shortage we argue that a more realistic approach may be to build data science teams. Consequently, we examine how teams are created, how they develop and how they are managed. Lastly, we argue that for big data to grow rapidly companies need to recruit and develop data science teams as part of a wider big data strategy.

This chapter is devoted to ‘statistical thinking’ rather than to the formulae, equations and mathematics of statistics per se. How do we, as individual human beings, interpret data, process information and define knowledge? Why are our instincts, gut-feelings and hunches often irrational, illogical and sometimes lead to poor decision-making? In this chapter we investigate how our internal cognitive and perceptual filters assign data into meaningful categories, mould expectations and condition the characteristics that we adopt towards the world i.e. our knowledge. We examine why statistical thinking is hard work, often counter-intuitive and requires a clear understanding of randomness, probability and correlation. We also explore our two, quite different, but inextricably linked cognitive systems; the first, a fast, emotional, automatic system, and the second, a slow, rational, lazy system. The first often responds to situations in an irrational or illogical way due to the bias-laden heuristics used to achieve lightening-speed reactions. Statistical thinking requires an acute awareness of these biases. In this chapter we investigate these biases as barriers to statistical thinking and examine how our emotional brain, in seeking to make sense of a complex world, often misinterprets information and jumps to incorrect conclusions. That this chapter forms the hub of the 9S-framework should come as no surprise, after all statistical thinking is at the heart of Big Data.

In the previous chapter we examine the strategy-making process in great detail but say little of how that strategy might be implemented. In this chapter we discuss some of the challenges of implementing strategy, with a particular focus on IS/IT strategy. We investigate the importance of the requirements-gathering process and examine some of the methodologies used to bring additional rigour to this process. We also consider the significance of change management and the role users play in actively promoting or resisting the implementation of a new system. In this chapter we consider some of the main reasons for project failure and what lessons can be learned from earlier project failures. We discuss why effective project management is imperative if the system roll-out is to be a success and what the terms ‘success’ and ‘failure’ mean in the context of information systems. For many executives success may be defined as built to specification and delivered on time and on budget. However, the measurement of system benefits is often an after-thought. We close the chapter with a discussion of benefits realisation and the impact of big data and the cloud on the concept of success.

In this chapter we explore different methods of storing data. While we touch on the ubiquitous Relational Database Management System (RDBMS), we also explore more exotic ‘big data’ storage systems such as Hadoop, Cassandra and MongoDb. We investigate how different systems support the use of data in different ways and contrast online transactional processing systems, decision support systems and column-based databases. In this chapter we also discuss the practical aspects of design and maintenance such as scalability, performance, availability and data migration. We also contrast system costing; including the open source versus proprietary software dilemma that faces many companies. Cloud computing in particular has transformed the cost of system ownership and increased the viability of open-source solutions, even for many wealthy companies. Consequently, we investigate both cloud computing and open-source tools such as Hadoop and NoSQL

In this chapter we look specifically at the ‘data’ aspect of big data. Regardless of the volume, velocity, variety and veracity of the data (the big data four “V”s), it must be accessed, processed and stored–all of which offer their own unique challenges in a big data environment. Big data often draws upon a wide variety of different sources, usually in different systems and structures but, as we discuss in this chapter, there are some immutable characteristics of data regardless of how it is stored. To this end we explore a number of data classification alternatives. But what happens when the situation is reversed? When an organisation provides data to many individuals or sub-contractors and asks them to process the data? In this chapter we discuss the relatively new concept of Crowdsourcing and ask, “What is actually happening when organisations, in effect, outsource to many?” We also investigate the thorny issue of data quality, arguably data’s most important attribute, and the one that, even today, absorbs the time of most data scientists. Lastly, this chapter examines the question of data ownership. Who owns your medical records or your bank statements? If you buy something online should the vendor have the right to sell your details to a third party? What about the ‘right-to-be-forgotten’?

The subject of IS Security covers a broad spectrum of issues that cannot be covered comprehensively in a single chapter. However, no book on big data would be complete without recognition of the issues and some of the solutions. In practice, IS security is about balancing the business risks posed by a security failure and the cost of making a system secure. This chapter begins with risk assessment; what constitutes a security threat and what is the scale of those threats? We explore different aspects of hacking such as denial-of-service attacks, viruses, worms, Trojan Horses and the use of spyware. We also examine some of the counter-measures that are available to protect systems and data. In this chapter we also highlight the weakest link in the security ‘chain’; people… Uncontrolled access to server rooms, unattended workstations left logged-on, lap-tops stolen from cars or left on trains and confidential files transmitted over coffee-shop wireless networks etc. Finally, recent developments such as wiki-leaks and the revelations of Edward Snowden have brought the issues of data privacy, ethics and governance to the public’s attention. In this chapter we examine these issues and, more generally, the role of data protection and data protection legislation.

We believe that the best way to understand new IS/IT concepts is to explore them, to immerse yourself in them and to experience for yourself what they can bring to your organisation. Thus, the aim of this chapter is to provide some hands-on experience of the big data applications discussed in the book and a to get a ‘feel’ for the Hadoop environment. We use the Sandbox to play safely and introduce tools, such as MapReduce, Hive and Pig, before finishing off with an example of visualization using Tableau. The chapter begins by discussing how to use Hadoop for Big Data and examines what other tools are required to make Hadoop useful and, in turn, how those tools are used. As an alternative to Hadoop, we explore NoSQL alternatives and how to use them. We also examine where SQL fits in the Big Data landscape.

In this chapter we discuss the unpredictability of future developments in the IT arena and emphasise the importance of keeping abreast of the latest trends and tools to avoid being left behind by the competition. However, we note that it is imperative that any review is suitably critical. In this chapter we explain why we expect Hadoop and MapReduce to be with us for the long-term and how the shortfall in appropriately skilled Data Scientists will impact on organisation’s abilities to make the most of Big Data. We go on to explore why this shortfall will drive the move toward more friendly interfaces for the current Big Data tools. Of course, the pressure to keep up with technology is just one influence upon the decision-making practice of any organisation. Managers of Information Systems will have new tools, but also will have complex new architectures to manage. This chapter examines how architectures are changing and the implications for IS Managers.