How can organizations cope with accelerating change in more complex worlds? The growth of capabilities produces expanded scales of operation, extensive interdependencies, new vulnerabilities, and puzzling failures. The result is the strategic agility gap where organizations are slow and stale in recognizing changing risks and fall behind the pace of change. The chapter addresses what factors drive the gap and what adaptive capabilities allow organizations to flourish in the gap. The result is a new paradigm for continuous adaptability illustrated in web-powered enterprises.
1 Introduction
Organizations face the challenge of how to adapt to the increasing pace of change in more complex worlds. The growth of capability brings rapid changes to society as new opportunities arise, complexities grow, and new threats emerge. The impact of deploying new technological capabilities has led to expanded scales of operation, dramatic new capabilities, extensive and hidden interdependencies, intensified pressures, new vulnerabilities, and puzzling failures.
Can organizations keep pace with the trajectory of change? Experience across industries indicates organizations are slow and stale in adapting to new threats, as well as to seize new opportunities. Surprising failures and service outages are regular occurrences in the news. One example is the threat of ransomware which offsets the value brought by new levels of computerized connectivity. This threat arose quickly with attacks on hospitals in 2016/2017 (CedarSinai/Medstar in US; Wannacry attack in the UK). Computer connectivity provided value that led to increased reliance, but also provided means for others to hijack the capability for their purposes. As capability grows to improve performance on some criteria, interdependencies become more extensive and produce surprising anomalies as the systems also become more brittle.
Advertisement
The strategic agility gap is the difference between the rate at which an organization adapts to change and the rise of new unexpected challenges at a larger industry/society scale. It is a mismatch in velocities of change and velocities of adaptation (Fig. 1). Can organizations learn how to offset changing risks before failures occur? Can organizations build capabilities to be poised to adapt to keep pace with and stay ahead of the trajectory of growing complexity? The chapter addresses what drives the strategic agility gap and what adaptive capabilities organizations need to flourish in the gap.
Fig. 1
The strategic agility gap
×
2 Organizations in the Gap—Synchronizing Activities to Keep Pace with Cascading Events?
The gap arises as the pace of change accelerates reshaping the risks and opportunities organizations face. Organizations need the ability to adapt as challenges change. But experience shows organizations generally are slow and stale to respond to challenge events [13]. Consider high frequency computerized financial trading. Some people recognized that a relative speed advantage in trading was a special resource that could be leveraged. They invested tens of millions of dollars for slivers of relative speed, while others were oblivious to the changes underway. The speed advantage made possible through computerization led to wholly new forms of trading, such as ‘dark pools’, adapted to make large profits. The financial advantage of speed arose from other effects of the shift to computerized trading: expanding volumes and the multiplication of stock exchanges available. New scales of operation and speed emerged. Everything became software dependent: regulatory changes, external competitive changes, internal changes to better compete, all were changes in software. The growth trajectory resulted in new relationships, new scale of operations, new interdependencies, new tempos of operation, accompanied by new risks which were difficult to see ahead.
This trajectory of growth means disturbances/challenges can grow and cascade faster than responses can be decided on and deployed to effect. To overcome this risk requires enhancing the ability to anticipate and build a readiness-to-respond in advance of challenge events. To fail to anticipate means a new response has to be generated during the challenge event—greatly increasing the risk of failing to keep up with the tempo. This aspect of the demand trajectory in Fig. 1 means organizations need to coordinate and synchronize activities over changing tempos, otherwise decisions will end up slow and stale. This occurred dramatically in a case of “runaway” automation in financial trading.1
2.1 Knight Capital Collapse 2012
As one part of the organization deployed new software in order to take advantage of changes in the industry, the rollout did not go as expected, producing anomalous behavior. The team tried to rollback to the previous software configuration as is standard practice for reliability. But the rollback produced more anomalous behavior. The roles responsible for the digital infrastructure struggled to understand what produced the anomalies and why normal attempts to recover had failed. Meanwhile, automated trading continued.
Advertisement
It took time before the team decided to involve upper management—to say the IT team did not understand the problem, were unable to block the cascade of effects, and the only action available was to stop trading. As upper management became informed and authorized a trading stop, it was too late—automated trading had gone on so long the company was, for all practical purposes, bankrupt from an untenable market position.
The case illustrates risks for organizations in the strategic agility gap. First, small problems can interact and cascade quickly and surprisingly given the tangle of dependencies across layers inside and outside the organization. Second, as effects cascade and uncertainties grow, multiple roles struggle to understand anomalies, diagnose underlying drivers, identify compensatory actions. Third, difficulties arise getting authorization from appropriate roles to make non-routine, risky, and resource costly actions, while uncertainty remains. Fourth, all of the above take effort, time, and require coordination across roles. Meanwhile, time pressures grow as situations deteriorate. Fifth, when critical replanning decisions require serial communication vertically through the levels of the organization, responses are unable to keep pace with events. The case illustrates the need to synchronize activities across roles and layers of the organization as tempo varies.
2.2 Coping with Hurricane Sandy 2012
Other cases highlight how to be poised to adapt. Deary examined how a large transportation firm learned to reconfigure coordination across roles and layers when events with unpredictable risky demands occurred. He observed how the organization used these techniques during hurricane Sandy [7]. To adapt effectively, the organization:
re-prioritized over multiple conflicting goals,
sacrificed cost control processes in the face of safety risks,
valued timely responsive decisions and actions,
coordinated horizontally across functions to reduce the risk of missing critical information or side effects when replanning under time pressure,
controlled the cost of coordination to avoid overloading already busy people and communication channels,
pushed initiative and authority down to the lowest unit of action in the situation to increase the readiness to respond when unanticipated challenges arose.
Upper management developed mechanisms for this shift prior to particular challenge events. As hurricane Sandy approached New York, temporary teams were created quickly to provide timely updates (weather impact analysis teams). In temporary local command centers key personnel from different functions worked together to keep track of the evolving situation and re-plan. The horizontal and vertical coordination possible through these centers worked to balance the efficiency-thoroughness tradeoff in a new way for a situation that presented surprising challenges and demanded high responsiveness [10]. The firm used mechanisms to expand/speed coordination across roles in order to match the tempo of events, even though these mechanisms sacrificed economics and standard processes. These mechanisms existed because this firm’s business model, environment, clientele, and external events regularly required adaptation as surprises were a normal experience.
2.3 Contrasting the Cases
The cases reveal how to be poised to adapt. Simply working to plan is not sufficient to handle exceptions, anomalies, and surprises, regardless of the contingencies built in the standard practices [14]. Anticipation and initiative are necessary in order for systems to adapt given the potential for difficulties to cascade [9]. When a unit confronts situations that challenge plans, delays are inevitable if the unit must first inform others and then wait for new instructions before initiating a response. In this reactive mode for revising plans in progress, performance is guaranteed to be slow and stale with limited ability to keep pace with change, as in the Knight Capital case. In contrast, the organization facing hurricane Sandy shifted to value responsiveness, push initiative down to units of action, and invoke mechanisms for timely coordination across roles as events unfolded.
In both cases multiple tempos of operation went on in parallel—which is basic for adaptive systems in complex worlds. When the connections across the mixed tempos were serial, responses lagged events. Facing hurricane Sandy, the other organization changed how it functioned to coordinate activities across the mix of tempos—which changed unexpectedly. From facing surprises in the past, the varying roles/levels had opportunities to exercise their coordinative ‘muscles,’ even though this specific event presented unique difficulties. In the strategic agility gap, the challenge for organizations is to develop new forms of coordination across functional, spatial, and temporal scales—otherwise organizations will be slow, stale and fragmented as they inevitably confront surprising challenges.
3 Systems Are Messy
The cases described to illustrate the strategic agility gap, highlight how systems are messy, fundamentally [1, 14]. All systems are developed and operate given finite resources and live in a changing environment [5]. As a result, plans, procedures, automation, all agents and roles are inherently limited and unable to completely cover the complexity of activities, events, demands, and change. All systems operate under pressures and in degraded modes. People and operations adapt to meet the inevitable challenges, pressures, trade-offs, resource scarcity, and surprises. To summarize the point vividly, Cook and Woods [6] use a coinage from the American soldier in WWII: SNAFU is the natural state of systems—where SNAFU, stands for Situation Normal All F_ _ _ ed Up. With SNAFU normal, SNAFU catching is essential—resilient performance depends on the ability to adapt outside of standard plans as these inevitably break down. SNAFU catching, however technologically facilitated, is a fundamentally human capability essential for organizational viability [15, 16]. Some people in some roles provide the essential adaptive capacity for SNAFU catching, though this may be local, underground, and invisible to distant perspectives [12].
The synthesis presented here begins with the recognition that all organizations are adaptive systems, consist of a network of adaptive systems, and exist in a web of adaptive systems—i.e., the resilience engineering paradigm. All human adaptive systems make trade-offs to cope with finite resource and all live in a changing world. The pace of change is accelerated by past successes, as growth stimulates more adaptation by more players in a more interconnected system. The scale and pace of change grow so that synchronizing over more roles at multiple tempos gets harder.
The strategic agility gap captures the dynamic whereby growing capabilities—which must produce markers of success on some indicators—also grow interdependencies and scales of operation that invoke complexity penalties (Fig. 1’s mismatched trajectories). The capability growth will produce new forms of conflict, congestion, cascade and surprise so that operating in the strategic agility gap is unavoidable.
SNAFU catching is essential for the viability of adaptive systems in complex worlds. But organizations rationalize this core finding away on grounds of rarity, prevention, compliance. The first claim is: SNAFUs occur rarely given the organization’s design thus investing in SNAFU catching is a narrow issue of low priority. The second claim is: there is a record of improvement that reduces the likelihood/severity/difficulty of SNAFUs. Third, when SNAFUs occur, poor response is due to people who fail to work to the rules for their role within the organization’s design.
These rationalizations are wrong empirically, technically, theoretically. As organizations focus on making systems work faster, better, and cheaper, they develop new plans embodied in procedures, automation, policies, and forcing functions. These plans are seen as effective since they represent improvements relative to how the system worked previously. When surprising results occur, the organization interprets the surprises as deviations—erratic people were unable to work to plan, to work to their role within the plan, and to work to the rules prescribed for their role. The countermeasures become more stringent pressures to work-to-plan, work-to-role and work-to-rule [8]. The compliance pressure undermines the adaptive capacities needed for SNAFU catching (initiative), creates double binds that drive adaptations to make the system work ‘underground,’ and generates role retreat that undermines coordinated activities.
In every risky world, improvements continue, yet we also continue to experience major failures that puzzle organizations, industries, and stakeholders. SNAFU recurs visibly—in June 2018 IT failures stopped online financial trading (TSB in the UK and Canadian Stock exchanges). Befuddlement arises from a background of continued improvement on some indicators, coupled with surprising sudden performance collapses. This combination is the signature of adaptive systems in complex environments. The scale complexity that arises from changes to increase optimality comes at the cost of increased brittleness leading to systems
which are robust to perturbations they were designed to handle, yet fragile to unexpected perturbations and design flaws [4, p. 2529].
As scale and interdependencies increase, a system’s performance on average increases, but there is also an increase in the proportion of large collapses/failures.
Given the pursuit of optimality increases brittleness, why don’t more failures occur?—SNAFU catching. Adapting to handle the regular occurrence of SNAFUs makes the work of SNAFU catching almost invisible [15]. The fluency law states:
well adapted activity occurs with a facility that belies the difficulty of the demands resolved and the dilemmas balanced [16].
Systems that continue to adapt to changing environments, stakeholders, demands, contexts, and constraints are poised to adapt through enabling SNAFU catching [6].
Ironically, what drives the strategic agility gap is past success. Success from new capabilities produces growth. Improvements drive a pattern in adaptive cycles: effective leaders take advantage of improvements to drive systems to do more, do it faster, and in more complicated ways. Growth, and the capabilities that power it, creates opportunities for others to hijack new capabilities as they pursue their goals. Success drives increasing scale complexity which leads to the emergence of new forms of SNAFU and SNAFU catching, as systems become messy again. This is seen the rise of high frequency trading in financial markets, in ransomware, and the influence of internet bots in elections, and more. In episodes of technology change, new forms of conflict, congestion and cascade arise as apparent benefits are hijacked.
4 Continuous Adaptability
If organizations today must live in the strategic agility gap, given the growth driven by technology, how can they flourish despite complexity penalties?
Answers to this question have emerged from research on resilient performance of human adaptive systems. For organizations to flourish in the gap they need to build and sustain the ability to continuously adapt. Today this paradigm exists in web engineering and operations because it was necessary to keep pace with the accelerating consequences of change as new kinds of services arose from internet fueled capabilities [3]. Web-based companies live or die by the ability to scale their infrastructure to accommodate increasing demand as their services provide value. Planning for such growth requires organizations to be fluent at change and poised to adapt. Because these organizations recognize that they operate at some velocity, they know they will experience anomalies that threaten those services. Because web-based services provide growing value, the value moves from optional to standard to critical and on to existential [5].
4.1 Lessons from Web Operations
Web engineering and operations serve as a natural laboratory for studying responses to the strategic agility gap. Outages and near outages are common even at the best-in-class providers. Past success fuels the pace of change. Systems work at increasing scale in a constantly changing environment of opportunity and risk. Web engineering and operations is important also because all organizations are or are becoming digital service organizations. For example, recently multiple airlines have suffered major economic losses when IT service outages led to the collapse of the airlines ability to manage flights. Results from this natural laboratory help reveal fundamental constraints on how human adaptive systems function and how organizations can flourish in the strategic agility gap.
Organizational systems succeed despite the basic limits of plans in a complex, interdependent and changing environment because responsible people adapt to make the system work despite its design—SNAFU catching. The ingredients are:
anticipation—seeing developing signs of trouble ahead to begin to adapt before the evidence is definitive (waiting till evidence is definitive almost guarantees being slow and stale);
contingent synchronization—adjusting how different roles at different levels coordinate their activities to keep pace with tempo of events;
readiness to respond—developing deployable and mobilizable response capabilities in advance of surprises;
proactive learning—learning about brittleness and sources of resilient performance before major collapses or accidents occur by studying how surprises are caught and resolved.
4.2 Four Capabilities for Continuous Adaptation
Results on resilient performance in web operations reveals specific capabilities for effective organizations living in the gap. Initiative is essential for adaptation to conflicting pressures, constant risk of overload, and inevitable surprises [16]. Organizations need to guide the expression of initiative to ensure synchronization across roles tailored to changing situations. This requires pushing initiative down to units of action [9]. Initiative can run too wide when undirected leading to fragmentation, working at cross-purposes, and mis-synchronization across roles. However, initiative is reduced or eliminated by pressure to work-to-rule/work-to-plan, especially by threats of sanctions should adaptations prove ineffective or erroneous in hindsight. Emphasis on work-to-rule/work-to-plan compliance limits adaptive capacity when events occur that do not meet assumptions in the plan, impasses block progress, or when opportunities arise.
Resilience engineering is then left with the task of specifying what system architecture balances the expression of initiative as the potential for surprise waxes and wanes. The pressures generated by other interdependent units either energizes or reduces initiative and therefore the capacity to adapt. These pressures also change how initiative is synchronized across roles and levels. The pressures constrain and direct how the expression of initiative prioritizes some goals and sacrifices other goals when conflicts across goals intensify.
Effective organizations living in the gap build reciprocity across roles and levels [11]. Reciprocity in collaborative work is commitment to mutual assistance. With reciprocity, one unit donates from their limited resources now to help another in their role, so both achieve benefits for overarching goals, and trusts that when the roles are reversed, the other unit will come to its aid.
Each unit operates under limited resources in terms of energy, workload, time, attention for carrying out each role. Diverting some these resources to assist creates opportunity costs and workload management costs for the donating unit. Units can ignore other interdependent roles and focus their resources on meeting just the performance standards set for their role alone. Pressures for compliance undermine the willingness to reach across roles and coordinate when anomalies and surprises occur. This increases brittleness and undermines coordinated activity. Reciprocity overcomes this tendency to act selfishly and narrowly. Interdependent units in a network should show a willingness to invest energy to accommodate other units, specifically when the other units’ performance is at risk.
Third, a key lesson from studies of resilience is that tangible experiences of surprise are powerful drivers for learning how to guide adaptability. Tangible experience with surprises helps organizations see SNAFU concretely and to see how people adapt as difficulties and challenges grow over time. Episodes of surprise provide the opportunity to see when and how people re-prioritize across multiple goals when operating in the midst of uncertainties, changing tempos and pressures.
Fourth, proactive learning from well-handled surprises contributes to re-calibration and model updating [15]. This starts with careful study of sets of incidents that reveal SNAFU catching [2]. What is an ‘interesting’ incident changes. Organizations usually reserve limited resources to study events that threatened or resulted in significant economic loss or harm to people. But this is inherently reactive and many factors narrow the learning possible. To be proactive in learning about resilience shifts the focus: study how systems work well usually despite difficulties, limited resources, trade-offs, and surprises—SNAFU catching. In addition, effective learning requires organizations to develop lightweight mechanisms to foster the spread of learning about SNAFU catching across roles and levels.
Strategic agility gap arises as organizations’ trajectory of improvement cannot match the emergence of new challenges, risks, and opportunities as complexity penalties grow (Fig. 1). To flourish in the gap requires organizations to build and sustain capabilities for SNAFU catching.
Open Access This chapter is licensed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made.
The images or other third party material in this chapter are included in the chapter's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.
