The science of organizational design: fit between structure and coordination

The design of an organization has a significant impact on the performance of the organization (Doty et al. 1993). Thus, it is important to know how a particular organization should be designed. Van de Ven et al. 2013 state: “Much has been learned, and even more needs to be learned, about designing organizations and institutions.” Further, they urge scholars to return to the frontier of organization studies by addressing a new agenda in designing organizations with promising new research methods. Levitt (2012) suggests that future research on organization design extends the frontiers of organizational micro-contingency theory. Gulati et al. (2012), on the other hand, suggest a focus on meta-organization design where organizational design is looked upon in an inter-organizational and community context.

Do we have scientifically based knowledge that can help us design efficient and effective organizations for the future, and is there a science of organizational design? At first, it may seem that science and design are opposites and that the two are not compatible. Science is the intellectual and practical activity encompassing the systematic study of the structure and behavior of the physical and natural world through observation and experiment, while design is a plan or drawing produced to show the look and function or workings of a building, garment, organization, or other object before it is made.

In this paper, we will discuss and present a science of organizational design, including a discussion of models and theories of designing organizations. We argue that experimentation of what is and what might be is the link between science and design for a science of organizational design (Burton and Obel 2011). We may address the issue of designing organizations by invoking new research methods and new ways of imagining possibilities. However, we should also employ our current knowledge to design new organizations for new conditions. The use of experimentation based on our current knowledge is the way to move forward. This is the only way we can generalize existing knowledge to help design organizations for the future.

The paper will also present a methodology perspective of the science of organization design. We will discuss what we have learned from the science of organization design. Building upon Simon’s book The Sciences of the Artificial (Simon 1996) and related notions, we develop a science of organizational design where the basic organizational design question is how to create a fit between structure and coordination. Structure is to break a big purpose or problem into smaller problems and units. The result is a set of tasks that have to be performed. The coordination is managing these smaller problems, units, and tasks into a whole so that they fit together to achieve an overall purpose. Finally, we discuss questions, which the science of organization design should address in future research.

By science, we mean knowledge and understanding about the world through observation of what is, and experimentation of “what is” and “what might be.” Design focuses on imagination and creation of what might be to achieve a purpose, moving toward “what should be” (Burton and Obel 2011: 467).

Organizational theory provides the theoretical underpinnings for organizational design. Organization theory describes and explains for our understanding how the world works; in complement, organization design builds on this to understand how the world could possibly work. Organization theory is a positive science to explain and understand the structure, behavior, and effectiveness of an organization—what is; organizational design is a normative science to recommend what might be designs for increased effectiveness and efficiency.

According to Simon (1996), an organization is an artifact that must be created in concept before it is created in reality. Romme (2003: 558), building upon Simon, argues that the “idea of a design involves inquiry into systems that do not yet exist—either complete new systems or new states of existing systems.” Organizational design is thus prescribing how an organization should be structured in order to function effectively and efficiently. Organization design is a systematic approach to aligning structures, processes, leadership, culture, people, practices, and metrics to enable organizations to achieve their mission and strategy. The basic premise is that there is no one best way of organizing and that different organizations are not equally effective or efficient (Galbraith 1973). This introduces the concept of contingency thinking, where the organization should be designed to fit the particular circumstance, which may be new and not experienced before.

How can we create information or knowledge about something that does not yet exist? We need knowledge-based experimentation and observation. Observation is an “as is” experiment or natural experiment where the researcher describes the “what is” situation and variations among variables that are manipulated by others or nature. Experimentation involves manipulation of variables to understand the effects.

We need to experiment to generalize from one study to another. We want to understand cause and effect in the science of organizational design through experimental manipulation of factors. There are a number of methodologies: simulation, laboratory studies, field studies, ethnographies, and large data analyses, among others (Scandura and Williams 2000). All utilize experimentation and observation to investigate and to understand the world of what is, as well as to explore the world of what might be.

Simulation modeling provides a powerful methodology for advancing theory and research on complex behaviors and systems (Harrison et al. 2007). The modeling requires current knowledge, and the simulations allow experimentation with models of new designs with full control of the setup (Burton and Obel 1995). Laboratory experiments introduce human behavior and require less formal modeling. Much of the power, beauty, and pleasure of models comes from inventing and elaborating models for exploring their implications in new domains (Lave and March 1975). This is the basis for the experimentation of what might be in a way that the results can be generalized. Many experiments are done in the real world (Puranam and Håkonsson 2015; Burton et al. 2017), but they may not be easily generalized. A triangulation approach using observation, simulation, and laboratory experiments will drive the knowledge and science of organization design forward (Burton and Obel 2011).

The challenge for the science of organization design is to create predictive models of future organizational designs. Prototyping new organizational designs could happen either through simulations or in the lab—with new proposed organizational arrangements being tested for unanticipated consequences before being implemented (Puranam 2012). Levitt (2012) states that “organizational chemistry” (goal conflict, institutional differences, etc.) and “organizational biology” (individual learning, organizational learning, evolution and regeneration of networks of organizations) will eventually yield robust and accurate enough agent-based modeling, analysis, and validation so that simulation of these phenomena will become useful to managers. This is supported by Puranam (2012) who states that we need models “which goes well beyond providing general advice to prototyping new organizational designs.”

In an analysis of novelty in forms of organizing, Puranam et al. (2014) argue that a new form is one that embodies new solutions to the basic problems of organizing—the division of labor and the integration of effort—in contrast to the solutions used by existing organizations.

An organization is a social unit of people that is structured and managed to meet a need or to pursue collective goals. All organizations have a management structure that determines relationships between the different activities and the members and subdivides and assigns roles, responsibilities, and authority to carry out different tasks. The activities must then be coordinated (Burton and Obel 2004) to obtain the collective goals. Structure and coordination are thus the fundamental choices in organizational design.

Structure includes the assignment of tasks to individuals or subunits, the apportionment of resources to these units, the designation of customers and markets to units, and generally the breakdown of the larger problem for smaller units. Coordination is bringing the units together through communications, IT, leadership, culture, incentives, routines and procedures, and generally what we call management (Van de Ven et al. 1976; Burton et al. 2015).

The structure and coordination choices are not independent. Once a structure is chosen, the coordination choices are limited in order to achieve a good fit. Further, coordination requires much more information processing than finding the structure, and designing the coordination has a different time perspective than designing the structure. The structure issue is a decision problem or analytical problem, while coordination is a management issue. Coordination is done in real time, as it must be done when activities are under way. But, the coordination mechanism is a design problem.

Within the framework of structure and coordination, organizational design boils down to who does what when, or how to allocate tasks, resources, customers, etc. to each of the small problems and how to coordinate these small units and tasks. This gives the framework within which experimentation should be done. However, experimentation also requires a theoretical framework with which the experimentation should be done to allow the required ability to generalize. One such framework is the information-processing paradigm.

Information processing is work in modern organizations: “Who talks to whom about what, who makes which decisions based upon what information” (Marschak and Radner 1972). Simon (1967: 1) is more succinct: Organizational design “is to investigate the information flows that are essential for accomplishing the organization’s objectives, then examine what these information patterns imply for organization structure.” A basic theory of organization design is balancing the information-processing capacity of the organization with the information-processing demand (Galbraith 1973, 1974). Underlying the theory is the assumption that “the greater the task uncertainty, the greater information-processing demands by decision makers” (Galbraith 1973, 1974). Further, the more interdependency between the sub-tasks, the more information-processing capacity is needed. Uncertainty and interdependency create the need for information processing in an organization.

Uncertainty has been defined as an incomplete description of the world (Arrow 1974), unpredictability, or perhaps more precisely as Knightian uncertainty where the probability distribution is not well defined. Further, uncertainty has included complexity or the number of variables in the environmental space (Ashby 1956; Burton and Obel 2004). Both the organization’s environment and its tasks can be uncertain. The management of environmental and organizational uncertainty requires the coordination of the organization’s tasks (Galbraith 1973).

Interdependency can be defined as the correlation among the variables in the environmental space or task space. Simon (1996) examines interdependencies as the degree of divisibility or decomposability using a matrix representation of the connections. The more connected or dense the matrix, the more interdependent the tasks; and the sparser the matrix entries, the less connected and the more divisible the tasks. The tasks’ interdependence may arise out the problem to be solved. It may however also be due to the particular task design, e.g., due to availability of different types of individuals to solve the task. Further, the decomposability of the matrix may be due to the basic organizational setup. Burton and Obel (1984, chapter 2) show how different matrices with different degrees of decomposability arise out of different organizational structures. The interdependency determines whom in the organization talks to whom about what and when.

To balance the information-processing demand and capacity, Galbraith (1973, 1974) offers two different organizational design strategies: reduce the need for information by creating semi-independent units (structure), or increase the information capacity with greater communications, either hierarchical or lateral (coordination). The information-processing perspective has to be seen in the particular context, e.g. digitalization, which changes both the information-processing demand and the ability to create capacity (Haußmann et al. 2012).

Tushman and Nadler (1978) and Burton and Obel (2004) argue that the concepts of uncertainty and information processing can be used to integrate the diverse organization design and structure literatures. They suggest a contingency approach based on the information-processing paradigm to design a feasible set of structural alternatives from which the organization can choose (Tushman and Nadler 1978; Burton and Obel 1995, 2004). Further, the information-processing paradigm is a general theory and rather robust to changes in circumstances, and it will allow us to say something about what might be designs from knowledge about what is. The information-processing paradigm also provides basis on which generalizable experimentation and observation can be done. Information-processing thinking can capture many theoretical issues, such as bounded rationality (Van Zandt 1999), learning (Shiffrin and Schneider 1977), and cognition (Klahr and Kotovsky 2013).

Simon (1947: 293) writes:

The division of labor is quite as important in organizing decision making as in organizing production. From the information-processing point of view, division of labor means factoring the total system of decisions that need to be made into relatively independent subsystems, each one of which can be designed with only minimal concern for its interactions with the others. The division is necessary because the processors that are available to organizations, whether humans or computers, are very limited in their processing capacity in comparison with the magnitude of the decision problems that organizations face.

How should the big task be structured or partitioned, and what resources should be allocated to the particular task? For example, deciding between a functional and divisional structure is choosing the basis for breaking up the big task. Then you must choose how many departments or divisions you would like to have. For the divisional structure, you can choose to allocate private customers to one division and corporate customers to another, or you can base your divisions on types of products. Concurrent with the structure, you have made a choice of how to coordinate. Coordination mechanisms require information, communications, cooperation, decisions, rules, routines, trust, incentives, and leadership, among others.

Thompson (1967) analyzes the organization in terms of uncertainty and technology or work flows of pooled, sequential, and reciprocal relations. Miles and Snow (1978) analyze the design problem in terms of structure and process and develop a typology of four organizational prototypes based on a particular choice of strategy. Lawrence and Lorsch (1967) use the concept’s differentiation and integration, which are similar to structure and coordination, and develop a contingency theory based on the particular type of environment. These approaches are single-contingency theories.

Using the information-processing concept, Burton and Obel (2004) developed the multi-contingency theory of organizational design and further developed these concepts in Burton et al. (2015). This view says that an organization’s design should be chosen based on the particular context and further that the description of the context should be multi-dimensional, including both structural and human components. Structural components of organizational design include goals, strategy, and structure and tasks. Human components include leadership, work processes, and people. Coordination includes control systems, decision systems, information systems, and incentive mechanisms.

In the multi-contingency theory, the relationships between structural, human, and coordination components are represented as a series of interconnected design rules. Design rules are “what should be” relationships (Burton et al. 2002). They incorporate both feasibility of “what might be” and desirability for the organization. The development of design rules has originally been related to simple design rules focusing on one or a limited set of contingencies, such as Lawrence and Lorsch (1967) on the relationship between environment and organizational design, or Woodward (1965) on technology and organizational design. These design rules were based on observation of what is. Later, these simple design rules have been combined into a set of more complex design rules (Burton and Obel 2004).

Design rules can be based upon “what is,” using the logic that what has been successful in the past in somewhat similar conditions is likely to work for the future—even applied in circumstances going beyond what has been observed. Design rules can also be developed based on the theory of balancing the information-processing capacity with information-processing demand. Further experimentation using simulations, laboratory studies, and empirical research are the basis for design rules (Burton and Obel 2013).

Experimentation and simulation can be the basis for theory development (Davis et al. 2007). An experiment is a test, trial, or tentative procedure—an act or operation for the purpose of discovering something unknown or of testing a principle or supposition. Experiments provide insight into cause and effect by demonstrating what outcome occurs when a particular factor is manipulated. Empirical studies evaluate what is, while simulation and experimentation can help in finding what might be and what should be.

Experimentation requires a theoretical basis and an experimental setup, and models can be of the organization system or of the agents in the system. Models of the system include system dynamic models (Lomi et al. 1997; Klaas 2004) as well as mathematical programming models (Burton and Obel 1984). Agent-based modeling is a relational, bottom-up understanding of organizations as ongoing processes arising out of individual and group decisions (Rivkin and Siggelkow 2003). Further, it can be based on interaction between “agents.” Levitt et al. (1999) developed a model that extends and operationalizes Galbraith’s (1973) information-processing view of organizations. The model allows for simulation of the micro-level information processing, communication, and coordination behavior of participants/agents in a project organization.

Most simulations of organization design are computer-based, some are laboratory experiments with humans as the agents (Håkonsson et al. 2016), while others are mixed, and some agents being computer agents and others being individuals (Burton and Obel 1988). Models of social and economic organizations based on the interaction between agents are becoming more common (Davis et al. 2007; Harrison et al. 2007).

This stream of research has developed an integrated multi-contingency theory utilizing theory development, laboratory experiments, simulation, and empirical analysis together for deeper understanding of contingency theory for organization design—a scientific approach to organization design. The work has thus contributed to the theory of organizational design as well as the extension of triangulation as a research design. Further, a forward-looking perspective of organizational design has been argued (Burton and Obel 2011, 2013).

The M-form, or multidivisional form, is widely utilized in industry. It is structured by product, customer, or country. Early on, Chandler (1962), in his study of large American corporations, described how the M-form worked and how it was efficient as adopted by General Motors and DuPont. Both corporations were able to coordinate production and customer needs more efficiently and effectively. Later, Williamson (1975, p. 150) presented an economic-reasoned argument in support and formalized the M-form hypothesis:

The organization and operation of the large enterprise along the lines of the M-form favors goal pursuit and least-cost behavior more nearly associated with the neoclassical profit maximization hypothesis than does the U-form (or functional form) organizational alternative.

The M-form is structured around demands in the market, where the U-form is structured around production, sales, finance, human resources, and perhaps other specializations.

In the 80s, Burton and Obel did a series of simulation studies to investigate contingency theory concepts related to the M-form hypothesis with the purpose of developing a more elaborate set of design rules. Using parsimonious models, they conducted an experimental design to test performance consequences of various coordination mechanisms, including the budget versus pricing planning systems and different organizational information systems (Burton and Obel 1980a). Then they investigated the M-form hypothesis (vs. U-form) for different levels of decomposable technologies (Burton and Obel 1980b, 1984) using the same experimental design. The results of the experiments confirmed the M-form hypothesis: The M-form is generally more efficient than the U-form, but with some conditions. Comparing the M-form and the U-form, the M-form does better for both low and high interdependency tasks, but the coordination mechanism is an important factor. The M-form allocation works very well for less interdependency and an iterative price coordination mechanism, while the U-form works well for a centralized coordination mechanism. Steer and Cable (1978) found that the U-form can be more efficient when centralized control is utilized, which is more prevalent in small firms. Size is not the main issue, but control and coordination are. For an organizational design, the choice of the structure and the coordination mechanism are not independent. The M-form design hypothesis is a contingent statement depending upon the task interdependency and the coordination mechanism, not a universal one. The experiment included two that might be alternatives. The M-form did not perform well for a less decomposable task, and the U-form did not perform well for a highly decomposable structure. We seldom observe these designs as natural experiments in the real world, and for good reason as indicated by the experiment. This experiment included both an examination of the M-form hypothesis, but also investigation of what might be design alternatives.

Further, in the M-form, a non-optimal allocation of resources to a division does not yield large opportunity losses if these resources are used well within each division, i.e., the intra-divisional coordination is done well. In the U-form, if the coordination is not done well, e.g., when the sales department and the production department do not use the same quantities, the opportunity losses are quite high. A transfer pricing mechanism can yield large losses if the price is not optimal. Stated differently as implications for organizational design, good coordination is more important than good structure or resource allocation.

In a later laboratory experiment, Burton and Obel (1988) included opportunism, which Williamson defines as self-interest seeking with guile (1975: 26). In the experiment, opportunism arises when an M-form division or U-form function manager can request non-optimal quantities from the headquarters that enhance her own divisional or functional performance at the expense of other divisions or functions and the overall firm performance.

For the U-form design alternative, the experiment found incentive schemes based on functional profit to be inefficient, while the U-form worked better for an incentive scheme based on corporate profit. The M-form alternatives were, however, both slightly better than the best U-form alternative, given the particular situation. The discussions about opportunistic behavior and different kinds of hierarchies are an ongoing debate (Foss and Weber 2016).

Under opportunism, the M-form performed much better than the U-form for corporate profits (Burton and Obel 1988: 111). The opportunity losses in the U-form were more severe than in the M-form—confirming the M-form design hypothesis. Again, coordination losses were more serious than loss due to structure, which confirms that good coordination is more important than good structure.

Putting the various experiments together, the M-form design hypothesis is supported without opportunism, and it is strengthened when opportunism is present. First, the M-form is confirmed parsimoniously using minimal information-processing mechanisms. Second, when the manipulation of information is possible, the M-form hypothesis is strengthened. Resource allocation is less important than coordination under decentralization with opportunism and without opportunism. Experimentation allowed a finer grained set of design rules than the rules only developed out of economic theory and empirical observations.

Design rules are “if-then” heuristics, which guide thoughtful guidance for what might be. Burton and Obel (2013) developed design rules based upon the M-form hypotheses and the experiments above:

If the task is nearly decomposable, then the divisional or M-form is superior to the unitary or functional form (p. 226),

The degree of decomposability is primary contingency for this rule. A corollary design rule is then:

If the task in not decomposable, then coordination is the main issue and can be realized with a unitary or functional form with a centralized coordination.

A design rule, which emerges directly from the opportunism lab experiment is:

If the organization has a unitary or functional form, then a unit profit scheme should not be used.

In the experiment, this incentive scheme is very likely to yield cheating with very high opportunity losses. What are the risk implications, if coordination is not achieved as desired? The M-form suffers less opportunity loss than the unitary form. That is, less risk is incurred with the M-form than the unitary form.

If the designer is risk averse, then the M-form organization is preferred.

There are many other design rules to state and develop.

A formal rule-based multi-contingency theory of organization (Burton and Obel 2004) was developed based on empirical observations, the information-processing theory, and simulation and laboratory experiments. The multi-contingency theory model was tested on an empirical basis using the survey methodology, and it was shown that misfits in the model led to significant loss in performance. Not only the main effects but also the interaction effects were shown to be important (Burton et al. 2002). Further studies then investigated specific interaction effects between leadership, climate, and strategy (Burton et al. 2004; Håkonsson et al. 2008, 2012). Not only are the interactions important, but there are asymmetric effects such that the misfit effects in dynamic situations are much more important than in non-dynamic situations (Håkonsson et al. 2012). Examining business-unit adaptation through new-product introductions in the global mobile device industry, Gaba and Joseph (2013) find that business-unit responses to poor performance lead to greater new product introductions. Taken together, these results were further expanded into a diamond model of contingency fit (Burton et al. 2011; Burton et al. 2015).

Van de Ven et al. (2013) state that there is much more to learn about designing organizations and institutions. We have argued that experimentation is at the heart of the organizational design challenge. Experimentation permits us to investigate both what is and what might be—the latter being the fundamental for design. Design requires the specification of both the assignment of tasks to units and individuals and the coordination of these tasks through communications, IT, leadership, culture, incentives, and management. Coordination is necessary, as the tasks are interdependent and uncertain, and the organization exists in an uncertain environment. At a fundamental level, this is the organizational design problem. However, each organization must be designed to meet specific situations and goals.

Simulation has been used to investigate many organizational design issues in what-might-be situations. Rivkin and Siggelkow (2003) investigated the conventional wisdom that firm-wide incentives and capable subordinates make top-level oversight less valuable. They further identified circumstances in which vertical hierarchies can lead to inferior long-term performance.

Their results can be stated as design rules, and one of which is:

If the competitive landscape shifts, then decentralize temporarily (Burton and Obel 2013, p. 237).

Levinthal et al. (2017) model and simulate governance issues in multi-authority, single authority, and autonomous organizations. Lee et al. (2015) introduce computational designs and evaluations of alternative organizational structures for disaster responses to resolve the disconnections between resource demands and supplies.

Next, we outline a few challenges where we do not have well-established design rules. These challenges can be addressed through experimentation of what might be.

Collaborative communities do not have a strong hierarchy, but they do have agents or individuals who interact and follow protocols on a “commons” to achieve individual goals. Fjeldstad et al. (2012) develop an architecture of collaborative communities, but we do not have detailed design rules. Since collaborative communities are rare in the business world, we have more to learn about how to design them and make them work. We suggest that the collaborative community is a very important new organizational form for which what might be simulations hold great promise.

Platform and digital confederations of firms which are neither market nor hierarchy but use extensive contracts are becoming more common. Platform firms can be very small but have enormous reach beyond their own employees and resources. They are a nexus of formal and informal contracts—going beyond the make-buy problem. At the same time, platform-based firms can be very large, such as Amazon with some 500,000 employees. These firms are going forth as natural experiments exploring what might be and making what might have been yesterday into what is today.

How do we design temporary organizations that start up quickly and disband quickly? These are related to big construction projects as building a bridge, making a movie, or organizing the Rolling Stones World Tour. Some are informal, without structure and coordination norms but with well-defined goals of success. How are the fundamentals of task assignment and coordination realized? Does opportunism enter, and if so, how?

Entrepreneurial ventures and start-ups are not well understood from an organizational design point of view. Are they different from traditional firms in allocation and coordination? Can they thrive without a hierarchy (Burton et al. 2017)? What is fundamental; what is new?

There is a call for agile organization today. What does this mean—quick response or adjustment to variation in the environment or technology? These are old questions, but they take on new dimensions in the digital world of today. They may require organizational designs that are quite different from traditional ones, which could and did response to variations in a slower world. Is time a critical difference here?

Design and experimentation are moving ahead as natural experiments that explore a portion of the “what might be” space. We are observing closely for insights and understanding. We need to go further to expand the design space and investigate the underlying mechanisms of structure and coordination through experimentation utilizing lab studies, simulations, field studies, and ethnographies, among other approaches. Romme (2003: 558) stated that the “idea of a design involves inquiry into systems that do not yet exist—either complete new systems or new states of existing systems.” Experimentation is at the heart of the science of organizational design.

Without a science of organizational design, we cannot generalize and use our accumulated knowledge to be able to design effective and efficient organizations that serve their purposes well.