Leadership and the choice of order: Complexity and hierarchical perspectives near the edge of chaos
Introduction
Just north of Australia deep in the rugged highlands of New Guinea's interior was one of the very few societies whose people had been able to live self-sustainably. For millennia this society evolved without the benefits and curses of modern civilization simply because the terrain ruggedness confined European explorers to the coast and lowland rivers. For over 400 years it was assumed that the interior of the island was covered with forests and uninhabited (Diamond, 2005, pp. 279–286). It was not.
Diamond (2005) describes a society that appears to be far from our collective vision of the modern corporation. Highlander society evolved from bottom-up dynamics to achieve ecological sustainability without the “assistance” of outside civilization. It achieved a preferred niche that yielded a sustainability until the civilizers imposed changes for a new order. The changes almost destroyed this viable society. Ultimately, the newly imposed order was considered more desirable by the imposers, yet the tale of the highlanders raises some fundamental questions. What should leaders do if existing, emergent order is not considered desirable? How can senior executives lead a modern corporation without destroying the emergent processes most likely to yield the developments that are key to the firm's future viability?
The isolated sustainability of the highlanders was an extraordinary feat, especially given that for most of this time the inhabitants could neither read nor write. Critical problems were mastered, the Achilles heels of many failed civilizations. These people conquered deforestation, erosion, the lack of water, declining soil fertility, and ultimately the resulting need for population control. Diamond (2005) tells us the highlanders mastered these challenges and, hence, survived, without importing economically important trade items from outside their area. Over time the highlanders became so proficient that their techniques serve as textbook examples for modern-day agriculture.
There is solid historical evidence attesting to all the above, but little historical evidence of how it occurred. Diamond tells us, however, that current New Guinea inhabitants have among the strongest “bottom-up orientations” in the world, attributable in large measure to their insistence on frequent meetings with a strong push for consensus. For our purposes, we can set the stage for the rest of this article by briefly describing highlander activities and then interpreting this description using various complexity science concepts implied by the title of this article.
We start with the fact that when the highlands were originally settled, there was an existing, dynamic equilibrium among habitable ecological factors: there were forests, rich soil and various other components that invited settlement. The people who were to become the highlanders had to adapt to this environment and the environment (e.g., food sources) had to adapt to their presence. Of course not all was perfect—the settlement would be in mountains several thousand feet high with frost at the upper levels even though New Guinea is located close to the equator. Over time, the highlanders found that the rugged mountainsides with sharp ridges and narrow valleys created many problems with which they had to deal. One of these was deforestation that resulted from the building of fires, houses, and related activities. To adapt, the highlanders learned which plants yielded the most food and could replace those originally growing. They also learned to provide appropriate watering systems and fertilization, and ultimately, they learned how to provide population control strategies (which, among other things, included ejecting the elderly and infirm from their society).
The inhabitants lived in small villages (frequently, but not always, at war), worked through trial and error processes and evolved their bottom-up consensus approach. Ultimately, over a very long period of time, they were able to develop ways to reforest, modify the plants and provide for soil fertility, along with myriad other activities to provide sustainability. In terms of leadership they depended on a “big man” (emergent leader by the force of his personality) in each village who lived and worked beside them.
New Guinea's highlander society may be seen as a complex adaptive system (CAS). That is, it is an identifiable collection of interacting elements characterized by dynamic and non-linear interaction (non-proportional interaction, where small changes in one element can have large results and vice versa, (see, Cilliers, 1998). Interactions occurred primarily among neighbors within the groups (the groups may be seen as what Kauffman, 1995, labels, patches, defined loosely as interdependent clusters) and sometimes between neighboring groups. These patches are similar to the divisions in a large organization. As groups of highlanders (patches) interacted with one another in their attempts to improve their lot, the process forced all groups to adapt to each other as all were adapting to changes in their collective ecological setting.
These interactions altered the prospects of fitness. This dynamic can be represented as a fitness landscape—a notion drawn from biology and discussed by Kauffman, 1993, Kauffman, 1995. In simplest terms, these landscapes represent different choices in terms of consequences. The range of all possible consequences may be depicted as a fitness landscape representing strategic choices a population could make. For instance, higher fitness strategies may be depicted as a higher peak on the landscape. Across all consequences the fitness landscape may show variations in ruggedness or in the number of peaks and in the variation between the fitness levels across the peaks. It is important to note that the fitness landscape for a single complex adaptive system, here the highlanders, is altered by attempts of other complex adaptive systems to improve their lot. As Kauffman noted, “... all organizations evolve and co-evolve on rugged, deforming, fitness landscapes” (Kauffman, 1995, p 246).
As discussed shortly, fitness may be represented by a single criterion of interest or more realistically by a mix of variables related to articulations of desired conditions (cf. Caldart and Ricart, 2004, Osborn et al., 2002). In the highlander example, Diamond chose to emphasize sustainability as the most important aspect of fitness. This notion of a fitness landscape has also been applied as a mapping of the actions of sets of individuals and their performance (cf. Levinthal and Warglain, 1999, McKelvey, 1999, for a much more detailed discussion). For the highlanders this latter view may be expressed in terms of the landscapes involved in individual issues (e.g., reforestation vs. population control) as well as deforming landscapes charting sustainability with conflicting internal and external constraints (other predators seeking greater fitness and adjusting to changes in the ecology made by the highlanders).
Also, for the highlanders we can imagine their “order” as relating to those interactions involved in sustainability. When the society's members entered the highlands, they evolved to be in a state near the “edge of chaos”: a delicately poised, transition zone between stability and chaotic systems. If the behavior of a system is too ordered, there is not enough variability or novelty; if, in contrast, the behavior of a system is too disordered, there is too much noise. For successful adaptation a system should be neither too methodical nor too carefree in adaptive behaviors (cf. Osborn et al., 2002). Thus, near the edge of chaos behaviors should be far from equilibrium (for in complexity science equilibrium means death; cf. Dooley, 2004). We interpret this to mean that the highlanders have been near the desirable edge of chaos state for many years, even as their environment has changed as a function of nature, encroaching civilization and increasing top-down governmental hierarchical forces. Put another way, this society was not stable—neither was it in chaos.
Organizations may be expected to seek higher fitness, yet as Kauffman (1995) notes “…organizations, when complex, all face conflicting constraints and seek excellence as, ‘the best compromises we can attain’” (p. 246). Our analysis, then, as noted, will be primarily restricted to complex adaptive systems (large economic organizations). And we will investigate various aspects of what might constitute the best compromises we can obtain.
It is worth briefly considering hierarchy and the previously mentioned “big man” leadership in highlander society as well as the hierarchy and leadership imposed by government officials. The social hierarchy of highlander society was comparatively flat with no established bureaucracy. Groups of members met frequently under the leadership of a “big man” to define problems, discuss issues and seek solutions to common problems. Of course, the discovery of these assumed poor isolated backward villages prompted government officials to replace impromptu governance gatherings with a full-blown governmentally based bureaucracy for governance. The charge government officials gave themselves was to provide appropriate “guidance” to spur development and progress for the assumed underprivileged, uneducated, unsophisticated villagers. The civilizers were to bring highlanders the benefits of modern society.
As told by Diamond, these officials almost killed the highlander society by imposing bureaucratic goals, plans, solutions and processes—they disrupted the fitness peak that the highlanders had developed over centuries. For instance, officials imposed a new, modern and well-tested irrigation plan based on extensive theoretical and practical experience in other lands. The new plan almost destroyed the delicate soils on the harsh terrain of the highlander's mountain fields. There also were official protests to well-established highlander practices not only on irrigation but also reforestation, not to mention population control practices inconsistent with established religions and “human rights.” Clearly the vision of the bureaucrats was different than the shared wisdom of the highlanders.
For the highlanders the earlier mentioned “big men” were emergent leaders. In meetings they emphasized connecting members to each other, facilitating dialog and discussion to reach consensus and encouraging new solutions to persistent problems. This profile is in stark contrast to the profile of officials brought in to help the highlanders. Officials brought in additional recognized experts each with his/her own specialized goal, backed by a plan to implement best practices to improve the lot of villagers. Each imposed a “better (but often unneeded and more bureaucratic) solution” on these reluctant villagers.
In a sense, our example of the highlander society illustrates the complexity theory tenet of order for free (Holland, 1995, Kauffman, 1993). Complex adaptive systems will naturally self-organize to seek greater fitness in an uncertain world. Clearly the highlanders did this in isolation for thousands of years. Yet, for the home country bureaucratic experts, this order did not fit their perspective of how the highlanders should be operating and the highlander's emphasis simply on sustainability. In complexity terms order was for free; yet a desired order was not free (Osborn et al., 2002).
Though sustainability may be highly desirable and a requisite for long-term societal survival, we do not live in only one complex adaptive system. Instead we live in a whole series of complex adaptive organizations—bureaucracies with specialized goals based on specialized contributions to society (Parsons, 1960). While the highlanders relied on emergent leaders, modern corporations use multiple layers of formally appointed leaders. Also, while no doubt emergent leadership is still important in the organizations comprising modern developed societies (e.g., Seers, Keller, & Wilkerson, 2003), there is little question about the dominance of formally appointed leaders in organizations. However, as shown below, this dominance does not mean these formally appointed leaders need to make the same mistakes as the civilizers.
Modern organizations are intricately intertwined with individual and social demands, constraints, and choices (Stewart, 1982a) where those at the pinnacle are often expected to have a powerful influence on the entire organization (e.g., Boal and Hooijberg, 2001, Hunt, 1991, Jaques, 1989). From our complexity perspective, then, a primary task of leadership in organizations is to establish a dynamic system where bottom-up structuration emerges (cf. Marion & Uhl-Bien, 2001) and moves the system and its components to a more desirable level of fitness.
With our emphasis on appointed leaders, we need to outline our view of leadership. As used in this article, leadership is built upon Katz and Kahn's (1978) idea that formal leadership is the incremental influence of position holders exercised via direct and indirect means to maintain and/or alter the existing dynamics in and of a system. It is having an impact beyond other formally designated aspects of the system. This leadership conception allows formally designated managers to exert much or little leadership, depending on system requirements and constraints and the extent to which one is able to move beyond these.2
While what those position holders at the top of organizations desire, expect and seek may be framed with the best of intentions and in the most positive terms, we suggest they may well engage in practices as deleterious as were the irrigation experiments of the bureaucrats wanting to help highlander society. We will challenge the normal definition of desired order promulgated by and for senior managerial leaders using complexity theory. Yet, we still recognize that to operate near the edge of chaos poses a dilemma for both leaders and leadership researchers. As shown shortly, it is not a matter of adjusting the ingredients to some known formula for success. It calls for a deeper understanding of both the context for leadership and leadership itself—an understanding we do not now have but argue we should seek.
The treatment above leads us to an analysis highlighting choice rather than determinism, multiplicity rather than singular vision, dynamic instrumentalism rather than dramatic transformation, dialog and discussion rather than leadership magnetism, and open self-interest rather than hidden self-centeredness. In many respects, then, this article echoes the themes of the sustainable highlander society. Specifically we discuss the interplay among organizations, leadership and the choices of various kinds of order. We contend that for the analysis of a desired order, the notion of organizational hierarchy, as emphasized especially from strategic leadership, should be included in our analysis and compared and contrasted with order for free arguments of complexity science researchers. The discussion of such analysis ultimately leads us to focus on different kinds of order and on possible paths forward that begin the integration of CAS and hierarchical approaches.
To conduct the kind of analysis just mentioned, we start by considering, in more detail, what is meant by order for free and how complex adaptive systems are able to achieve such a state. Such discussion moves us into an examination of fitness and fitness landscapes and potential conflict among various kinds of fitness or order criteria. Then we move to the differing ways that order can be achieved. This discussion brings us to treatment of non-linear dynamics and traditional hierarchical emphases that deal with leadership at various organizational levels based on hierarchical frameworks from the traditional organizational literature. Such frameworks involve recognition of organizational context and an emphasis on various hierarchical managerial leadership approaches with special emphasis on the stratified systems or requisite organization work of Elliott Jaques and closely related organizational hierarchical treatments. There, as one moves up the organizational hierarchy, one finds increasing task complexity by domain and echelon. We discuss hierarchy in some detail and it is compared and contrasted with the order for free notion of CAS.
With the above as a backdrop, we then discuss a reconciliation of the CAS and traditional hierarchical perspectives. For this reconciliation, we compare and contrast in some detail a number of key issues helping to define each of the two perspectives. These issues start with the emphasis on decomposition or reductionism in each, with CAS arguing for its necessity to a relatively limited degree to provide appropriate focus. A second emphasis is on the roles of CAS and hierarchical position holders. This emphasis is followed by the importance and interpretation of the concept of organizational attachment, one form of which was originally set forth by Etzioni, 1961, Etzioni, 1975. The discussion in this section is finalized with the hierarchical role of direct and indirect managerial leadership in organizations.
We conclude the overall analysis with a summary reiterating an emphasis on five main points from our opening vignette concerning the highlanders and a final argument for the application of order from leadership researchers. The first of these points is that the consideration of CAS and hierarchy suggest the virtual impossibility of an integrated theory of leadership effectiveness in the foreseeable future and the necessity for researchers to be explicit about the desired order implicit in their work. Second, we show that a movement toward partial reconciliation calls for partial organizational decomposition (reductionism), and an examination of externalities and consideration of system commitment. Third, we argue that organizational decomposition needs to preserve the ability to detect dynamic interaction, the analysis of direct and indirect managerial leadership, and emphasis on studies where complex adaptive features are most likely to be important. Fourth, we contend that high singular interdependence on powerful external entities is a potential problem for CAS's because of its overemphasis on top-down processes. This contrasts with hierarchical emphases where such interdependence is desirable even though difficult to obtain. Finally, affiliation is discussed as an important basis for attachment/compliance.
We conclude that it is possible but unlikely in large-scale economic organizations, to have order for free in CAS's operating in turbulent environments. Rather, what is needed is bottom-up structuration combined with top-down hierarchy. At the same time, the above discussion is reinforced by an ongoing treatment of possible paths forward to strengthen the integration of these various elements.
We further conclude by arguing for the partial integration of complexity views with leadership research in order to discuss a series of alternative desired orders. These orders would in one way or another combine order for free notions with hierarchy to emphasize such desirability. They also would provide leadership researchers a voice in focusing on different orders.
Section snippets
How can there be order for free and how do systems achieve it?
With an understanding of where we are heading, we now ask the reader to consider a world where organizations confront pervasive dynamism, non-linearity, and non-predictability. Once again we use the highlanders as an example even though strictly speaking they are not a large economic organization. However, once the outside governmental forces moved to institute changes that would influence a very large portion of the highlander's existence we consider them as a medium to large sized political
Fitness, fitness landscapes and the potential conflict among preferred criteria
We begin our discussion with a brief exploration into the criteria used in complexity based views of leadership and more conventional leadership analyses. While, as we have argued, the notion of fitness and a fitness landscape is central to the notion of “order for free” in complexity theory, the criteria to determine leadership effectiveness in many studies are almost conceptual afterthoughts. Although Hunt (1991 pp. 91–96), discusses several aspects of organizational effectiveness, it is
Non-linear dynamics
How is order achieved? With a complexity perspective order is achieved by elements interacting to produce non-linear dynamics that yield bounded instability. There is both determinism and human agency. For example, each time an agent (individual or unit) interacts with another, the agent is free to follow, ignore or slightly modify the arrangement. There is bounded choice and actions and interactions have consequences in the form of feedback loops. Different actors within the system resonate
Jaques' framework
We start this treatment with Jaques' (1989) hierarchical framework and extend and refine it within the edge of chaos perspective to help in illustrating our interpretation of its linkage with the order for free concept. Jaques' framework, like the one earlier mentioned by Katz and Kahn (1978), uses open systems notions. Also, both conceptualize different and increasingly complex requirements as one moves higher in the organization. Both are based on the pervasive Weberian notion (Weber, 1947)
Toward a reconciliation of the order for free and hierarchical perspectives
As this quick review of the leadership domain research suggests, leadership researchers face a number of important and fundamental challenges in understanding how, when and why individuals and teams in upper and middle echelon organizational positions can or cannot facilitate or direct the type of emergence expected to yield greater fitness. From the extensions of Jaques by Jacobs and McGee (2001), it seems clear that as researchers move up the organizational hierarchy there is an increase in
Summary and conclusion
In this article, we critically examined the view held by many complexity scientists that as complex adaptive systems operating in turbulent environments, organizations tend to exhibit order for free behavior. That is, like the highlanders, they will naturally self-organize to seek greater fitness. We compared and contrasted this perspective with the traditional organization and leadership approaches emphasizing hierarchy in one form or another. Here we found a very heavy emphasis on a desired
Acknowledgements
We acknowledge the assistance of A-Team Members: Adam Bailey, John Davis and Donna Hunt for help with this manuscript.
References (74)
- et al.
Authentic leadership development: Getting to the root of positive forms of leadership
The Leadership Quarterly
(2005) - et al.
Strategic leadership research: Moving on
The Leadership Quarterly
(2001) - et al.
Leadership déjà vu all over again
The Leadership Quarterly
(2000) - et al.
Leadership in complex organizations
The Leadership Quarterly
(2001) - et al.
Leading creative people: Orchestrating expertise and relationships
The Leadership Quarterly
(2002) - et al.
Toward a contextual theory of leadership
The Leadership Quarterly
(2002) - et al.
The origins of vision: Effects of reflection, models, and analysis
The Leadership Quarterly
(2005) Complexity theory and organization science
Organization Science
(1999)Design for a brain
(1952)- et al.
An organizational life cycle approach to leadership
Functions of the executive
Managing with style: The effect of managers on firm policies
The Quarterly Journal of Economics
Bass and Stogdill's handbook of leadership: Theory, research and managerial applications
Order is free: On the ontological status of organizations
The management of innovation
Corporate strategy revisited: A view from complexity theory
European Management Review
Organizations and complexity: Searching for the edge of chaos
Computational and Mathematical Organization Theory
Complexity and postmodernism: Understanding complex systems
Complexity models of organizational change
The coming of the new organization
Harvard Business Review
Metaphors of leadership: An overview
A comparative analysis of complex organizations
A comparative analysis of complex organizations
Strategic leadership: Top executives and their effects on organizations
The tipping point
Threshold models of collective behavior
American Journal of Sociology
Organizations: structures, processes, and outcomes
Strategic management
Hidden order
Leadership effectiveness: Past perspectives and future directions for research
Leadership: A new synthesis
Leadership in complex systems
Competitive advantage: Conceptual imperatives for executives
Requisite organization
Cited by (81)
Collective and network approaches to leadership: Special issue introduction
2016, Leadership QuarterlyCitation Excerpt :This research introduces the leadership field to collective intelligence and lays the groundwork for future exploration into collective decision making as a collective leadership process. Two articles in the special issue specifically draw on complexity theory (e.g. Lichtenstein & Plowman, 2009, Marion & Uhl-Bien, 2001, Osborn & Hunt, 2007) as their guiding theoretical framework. Complexity theory offers insight into collective leadership by emphasizing interactions between parts of a social system (actors) to understand emergent properties of the system (collective behavior).
When Normal is Not Normal: A Theory of the Non-Linear and Discontinuous Process of Desired Change and its Managerial Implications
2023, Journal of Applied Behavioral ScienceComplex adaptive systems
2022, Handbook of Theories for Purchasing, Supply Chain and Management ResearchDisaster and Emergency Pharmacy A Guide to Preparation and Management
2022, Disaster and Emergency Pharmacy: A Guide to Preparation and ManagementComplex Systems in the Social and Behavioral Sciences: Theory, Method and Application
2021, Complex Systems in the Social and Behavioral Sciences: Theory, Method and Application
- 1
Tel.: +1 806 742 3175x795 4582.