To this point we have argued that problems differ in complexity and in optimal forms of solution search. Complex problems require solution search that involves extensive knowledge transfer, which is discouraged by knowledge-exchange hazards. Efficiency thus dictates the selection of governance forms, which support the appropriate level of knowledge transfer. At the same time, efficiency dictates the maintenance of high- powered incentives that encourage actors to actively seek knowledge through solution search. As we will see, there is a fundamental trade-off between governance forms that powerfully motivate search effort and those that more effectively support heuristic development.
In the discussion below, we explore three prototypical alternative governance forms as mechanisms for governing solution search: markets, authority-based hierarchies, and consensus-based hierarchies.5 Following Williamson (1991), we contend that these three governance forms differ fundamentally in their use of three organizational features: (1) decision rights over the path of solution search, (2) communication channels to support knowledge transfer, and (3) incentives to motivate search. In the discussion below, we argue that these organizational features are configured in unique and complementary ways to achieve three organizational forms or prototypes. While scholars have touted the virtues of each form as a solution to the governance of knowledge exchange, these treatments are generally not comparative and provide no basis for discriminating among them.6 Thus, we develop here an alignment between governance forms and problem types.
We begin with a discussion of markets, focusing particularly on Hayek’s discussion of its virtues as a device for efficiently governing knowledge exchange. Markets, however, have a very low capacity for remedying knowledge-exchange hazards and thus fail as problems become more complex. We argue that hierarchy fundamentally comes in two forms: authority based and consensus based. Each possesses a unique resolution to knowledge exchange hazards. These two governance forms correspond roughly to the distinct and fundamentally competing representations of hierarchy within the knowledge-based theories (see Demsetz 1988, Conner and Prahalad 1996 versus Arrow 1973, Kogut and Zander 1996). Both more efficiently cope with knowledge-formation hazards than markets and in the process support heuristic search more effectively than markets. Nonetheless, these two forms of hierarchy also differ significantly in their approach to supporting heuristic search, with one promoting knowledge transfer and the other essentially economizing on it. In our discussion below, we match these governance solutions and their support for directional or heuristic search to the complexity of problems.
1. Markets
Our conceptualization of the knowledge-based governance choice is very consistent with Hayek’s (1945) view that the fundamental economic problem is determining the “best way of utilizing knowledge initially dispersed among all people” (p. 520). From this perspective, the manager must decide the best way to organize the creation of knowledge when the existing knowledge is widely dispersed among actors. In addressing this problem of governing knowledge, Hayek espoused the “marvel” of the market, arguing that markets provide “inducements which…make the individuals do the desirable things without anyone having to tell them what to do.” Markets, thereby, “dispense with the need of conscious control” (Hayek 1945, p. 527). In our context of problem solving, market governance determines the path of search by decentralizing control to those in possession of valuable, specialized knowledge. In markets, prices provide high-powered incentives that motivate actors to search for solutions that both exploit and enhance their specialized knowledge and that can be encapsulated into saleable products. For a given problem, the path of search is merely an aggregation of the individual search decisions made by a set of actors autonomously choosing paths of search that they anticipate will lead to improved and marketable solutions. Thus, markets achieve unique combinations of knowledge using fully decentralized decision making.
Markets are ideally suited for governing directional search. As previously discussed, effective directional search involves individual actors altering design features based on knowledge they possess and then observing the resulting change in solution performance. Markets also provide high- powered incentives that motivate actors to pursue trials that expand their knowledge or the sale of products and services based on that knowledge. At the same time, however, markets provide weak support for knowledge sharing. They provide little protection against knowledge appropriation and no clear disincentives against strategic manipulation of search heuristics. Further, knowledge sharing requires the formation of a common language by which to communicate knowledge. Markets provide weak incentives to invest in the formation of such language.
However, when problems are decomposable and direc-tional search is desired, knowledge sharing is largely unnecessary. Indeed, it is precisely on knowledge transfer that markets are designed to economize. As Hayek (1945) contends regarding markets: “The most significant fact about this system is the economy of knowledge which it operates, or how little the individual participants need to know [about other actors] in order to be able to take the right action” (p. 527). Thus, the efficiency of markets in governing decomposable problems is that they avoid knowledge-exchange hazards by severely restricting knowledge exchange. Instead, markets offer powerful incentives for these individual actors to make optimal use of the knowledge. Each possesses incentives to engage in local search and thus to focus on trials that enhance their own accumulation of specialized knowledge.
Markets, however, provide a rather cumbersome and costly mechanism (i.e., the courts and classical contracting law7) for developing the heuristics necessary for searching complex solution landscapes. Independent actors governed by the market could in theory contractually agree to particular patterns of search, but disputes over performance in pursuing these search patterns would potentially require the intervention of courts and contract law. When solutions to complex problems are sought and a search heuristic is required to enhance the probability of discovering a valuable solution, conflicts among independent actors in developing this heuristic may be intense because of the knowledge hazards discussed above. Thus, the capacity for markets, even when supported by contracts, to manage heuristic search development is quickly exhausted. However, when markets are matched to decomposable problems where actors possessing independent knowledge sets can make independent design choices, commonly shared heuristics are unnecessary and such disagreements are unlikely to arise.
As governance devices, markets quickly fail as problems become more complex and demand heuristic search. Indeed, markets exacerbate the knowledge- exchange hazards that contaminate efforts to perform heuristic search. Markets’ high-powered incentives discourage knowledge sharing and instead promote knowledge hoarding. Further, these incentives encourage individual actors to strategically shape heuristics and subsequent patterns of trials in ways that benefit them individually. Moreover, these incentives actively discourage investments in the formation of common language required for knowledge sharing and heuristic search.
Given the set of organizational features that commonly define markets, we posit markets are poorly suited for governing the process of solution search for a complex, nondecomposable problem. The necessary knowledge sharing is undermined by the absence of both appropriate incentives and shared language. Efforts to manage heuristic search through markets or contracts are limited by the high costs of resolving through legal means the disputes that would emerge as individual actors strategically appropriate knowledge and seek to opportunistically alter the trajectory of search in ways that benefit them individually. Thus, markets are quite efficient in guiding directional search for solutions to decomposable problems but quickly fail as problems become more complex and demand heuristic search. They fail because knowledge-exchange hazards are poorly managed through contracts, and the high- powered incentives of markets simply promote these hazards.
2. Authority-Based Hierarchy
Many argue that the capacity to use centralized authority provides the distinguishing advantage of hierarchy over markets (e.g., Arrow 1974, pp. 68-70; Milgrom and Roberts 1990, p. 72; Simon 1951, p. 294; Williamson 1985). The efficiency properties of authority have been particularly highlighted in the context of problem solving and knowledge formation. As Arrow (1974) argues, “Authority, the centralization of decision making, serves to economize on the transmission and handling of knowledge” (p. 69).8 Demsetz (1988) concurs, noting the potential inefficiency of a market in which individual actors must first be convinced of the appropriateness of each action they take. As he contends, the virtue of hierarchy is that “[direction substitutes for education (that is, for the transfer of the knowledge itself).” Thus, in solving identified problems, the advantage of authority in hierarchy is the capacity for one actor to identify the precise order of trials, thereby circumventing the need to contractually manage the order of trials or to spontaneously achieve some type of consensus through extensive knowledge sharing.
Because Hayek argued similarly for the efficiency of markets in economizing on knowledge transfer, how do we reconcile the advocated knowledge-based efficiency of both autonomy in markets and authority in hierarchy? The answer, of course, centers on the problem for which efficient governance is sought. Markets are efficient when knowledge transmission is directed at solving decomposable problems where there is little interaction among design choices. When problems are decomposable, knowledge is embedded in products and services and knowledge transmission is largely limited to what can be contained in prices and bundled into products and services. However, bundling knowledge sets within a single firm and exercising authority to direct search becomes efficient when problems become complex and efficient search demands shared heuristics to sequentially guide design choices. In this circumstance, authority economizes on the extensive and costly knowledge sharing and education that would need to occur were the governance of solution search organized through a market interface. As described below, authority-based hierarchy more efficiently resolves the knowledge-based exchange hazards that arise in markets when problems are complex and demand knowledge sharing.
Authority-based hierarchy is composed of a set of complementary features distinct from those that support markets. It is well suited to supporting some degree of heuristic search, though at the expense of dampening incentives for directional search. The key feature of authority-based hierarchy is a central figure who invests in understanding critical knowledge interactions and then composes suitable heuristics to guide search. For instance, the central figure defines and structures subproblems by imposing constraints or design rules on each subproblem, which greatly narrows the area of the solution landscape to be searched.9 Other features of the governance form also support this form of search. Within the boundaries of hierarchy, the courts exercise forbearance. They refuse to hear disputes internal to the firm, including disputes about the proper path of problem search. It is precisely this forbearance that grants authority within the bounds of hierarchy. Because of this forbearance, individual actors have limited capacity and motivation to strategically manipulate the path of search.
Other features of authority-based hierarchy also facilitate centrally directed heuristic search. The high- powered incentives that motivate and reward directional search and knowledge accumulation in markets are limited within hierarchies. Although workers can exit an employer and take with them some portion of the knowledge they accumulate, the incentive structure within the firm is quite different. Employees grant managers authority in exchange for a wage. This act severs the direct linkage between the knowledge they accumulate and the wage they receive. This departure from market incentives also encourages investments in communication channels and codes that lower the cost of a central authority assimilating, accumulating, and applying knowledge. However, authority- based hierarchy does not promote horizontal communication channels that would support broad knowledge sharing among peers. It is precisely such knowledge transmission and broad education for which the authority and direction of this governance form provide a substitute. Authority-based hierarchy is thus superior to markets in supporting heuristic search but inferior in supporting directional search due to incentive damage. It dampens incentives to strategically manipulate the path of search and provides a central authority that economizes on costly knowledge transmission.
The use of authority within hierarchy to guide search is ideally suited to a range of problems that are moderate in their complexity. Authority has significant limits that become apparent as problems become either too complex and nondecomposable or too simple and decomposable. An authority’s heuristic is only valuable when it more efficiently guides search than actors searching independently, each basing their own search on their own specialized knowledge. However, managers have a propensity to meddle in subordinates’ decisions (Williamson 1985), in part due to overconfidence in their own judgment (Bazerman 1994). As problems become more complex and nondecomposable, the cognitive limits of managers to develop useful search heuristics, combined with a manager’s propensity to meddle, contaminates the efficiency of search. As Weber and other organizational theorists remind us, ‘Authority does not imply expertise” (see Hammond and Miller 1985, p. 2). When authority is exercised in the absence of knowledge, it contaminates rather than accelerates search. Direction effectively substitutes for knowledge transfer only when managers have valuable knowledge with which to direct subordinates. When search dictates extensive recombining of knowledge, it is quite unlikely that authority is well suited to governing solution search. If many actors possess the required knowledge to define an optimal search heuristic and a central authority is incapable of absorbing that knowledge in a timely manner, then the development of a commonly shared search heuristic is required. Hayek noted precisely this limit to authority when he commented, “We cannot expect that [the problem of coordinating knowledge] will be solved by first communicating all this knowledge to a central board which, after integrating all knowledge, issues its orders” (1945, p. 524, emphasis in original).
If firms could constrain managers to intervene only when they possess knowledge that improves the direction of search, then the contributions of authority could extend to searching highly complex problems. However, managers appear to have great difficulty constraining their use of authority within hierarchies. In part, this reflects managers’ inability to discern whether they possess information useful in directing search, and a strong bias to assume that they do. Managers are simply prone to meddle, wielding authority where they should delegate. Thus, we conclude that authority-based hierarchy provides an important advantage over markets in directing the search for solutions the more nondecomposable the problem. However, the limits to this governance solution are reached as the level of knowledge interactions escalate and the cognitive capacity of a single individual to assemble the required specialized knowledge reaches its limits.
The effectiveness of authority-based hierarchy relative to market-based control also diminishes rapidly as problems become more decomposable. When problems involve low interaction among design choices, using a manager’s authority to dictate search may again damage search performance. An authority is simply unlikely to develop the necessary specialized knowledge required to direct solution search for all relevant subproblems and design choices. Again, managers are prone to meddling, directing the path of search when they lack the knowledge to effectively do so. The low-powered incentives that accompany hierarchy are also detrimental when autonomous searching by independent actors is desired. The weak incentives of hierarchy limit specialized knowledge formation and thereby constrain the efficiency of search. Further, the use of hierarchy is simply more costly than markets due to the costs of paying and supporting the central authority. If no clear value is added, then the added costs discourage its use. Thus, as displayed in Table 2, the combination of greater cost and contaminated search renders authority-based hierarchy inefficient for decomposable problems.
3. Consensus-Based Hierarchy
Scholars also have argued for the knowledge-based effi- ciency of an alternative hierarchical form of governance, one that emphasizes extensive knowledge sharing and commonly shared search heuristics to guide decision making. It is this form of governance that Kogut and Zander (1992, 1996) emphasize when they describe the firm as providing a specialized social community that creates and transfers knowledge more quickly and efficiently than markets. In this conception of the firm, hierarchy is a device that supports knowledge transfer. However, this conception contrasts sharply with authority-based hierarchy, which minimizes knowledge transfer. Rather than hierarchy substituting direction for education as Demsetz (1988) and Conner and Prahalad (1996) suggest, hierarchy in this case substitutes education (i.e., knowledge transfer) for direction. Actors must first educate one another regarding knowledge relevant to defining a collective search heuristic.
This form of hierarchy achieves extensive knowledge transfer by enhancing the efficiency with which knowledge transfer occurs among actors within the firm. Extensive information sharing potentially permits actors within the firm to collectively agree on a path of search that is a consensus reflection of the specialized knowledge sets housed within the firm. However, as Arrow notes, consensus-based decision making, herein referred to as consensus-based hierarchy, is the near polar opposite of authority. Authority in this form of governance is used only to select a project, not the path of search. How then does a manager choose between the efficiency of these alternative internal governance choices of authority- and consensus- based hierarchy? The answer, of course, lies in the nature of the problem to which a manager seeks to find a solution. Consensus- based hierarchy is a potential solution to the failure of authority in governing heuristic search as problems increase in complexity.
Consensus-based hierarchy is supported by a distinct configuration of features that support active knowledge exchange. As Arrow (1974) argues, consensus can substitute for authority so long as knowledge transfer is inexpensive and actors within the group have “a sufficiently overriding commonly valued purpose…” (pp. 69-70). Under these circumstances, each agent within the firm perceives the correct decision based on his or her knowledge and interests. Because interests and knowledge are commonly shared, consensus is spontaneous.
Achieving consensus, however, first demands extensive knowledge sharing, and knowledge sharing requires a commonly shared language. Hierarchy possesses a distinct advantage over markets in promoting the formation of shared language. Arrow, in his treatise The Limits of Organization, recognizes the distinct advantages of hierarchy in facilitating knowledge transfer through firm- specific language and identity. He contends that a primary component of firm-specific capital formation is “learning the information channels within a firm and the codes for transmitting information through them” (Arrow 1974, p. 56). He further notes that such investments are not only individually specific, but that they in aggregate represent “irreversible capital accumulation for the organization” that leads to organizations having “distinct identities” (pp. 55- 56).10 Kogut and Zander similarly argue that communication costs are lower within the firm due to the presence of “higher order organizing principles” that “[establish] the context of discourse and coordination among individuals with disparate expertise” (Kogut and Zander 1996, p. 503). In their view, the boundaries of the firm define qualitative differences in the scope of social knowledge available to individuals. The shared identity that exists within a firm lowers the cost of communication and establishes “rules of coordination and influences the direction of search and learning” (Kogut and Zander 1996, p. 503).11
Incentives and dispute resolution mechanisms within consensus-based hierarchy are also configured to support knowledge transfer and consensus decision making. Very low-powered incentives are essential to consensus, because such incentives encourage (or more accurately do not discourage) knowledge sharing.12 High-powered incentives within the firm would encourage knowledge hoarding and strategic manipulation of search.
Dispute resolution in consensus-based hierarchies, of course, also differs from authority-based hierarchy. While forbearance by the courts remains the central characteristic of hierarchy compared to markets, dispute resolution in consensus-based hierarchy involves individual actors collectively deciding this path. Such group decision processes equate to Williamson’s (1985, pp. 246-247) relational team and approximate Ouchi’s (1980) clan form of organization. In this case, the firm “will engage in considerable social conditioning to help insure that employees understand and are dedicated to the purposes of the firm and employees be provided with considerable job security, which gives them assurance against exploitation” (Williamson 1985, p. 247). Such efforts build relationships among agents and facilitate the formation of a common identity. These relationships and shared identity in turn ease knowledge transfer, facilitate agreement, and discourage the exploitation by other agents of knowledge transferred into the firm (Allen 1967, Tushman 1978).13
While consensus-based hierarchies possess distinct advantages over authority-based hierarchies in facilitating heuristic search to highly complex problems, consensus fails in comparison to authority as problems diminish in complexity. The costs associated with supporting extreme levels of knowledge transfer are substantial and become unwarranted as problems diminish in complexity. Further, low-powered incentives constrain the motivation to develop specialized knowledge and dampen incentives for solution search. The scope of investment in shared language and socialization and the efforts involved in the transfer of knowledge can be excessive when problems are only moderately complex. Such investments and effort slow the accumulation of specialized knowledge necessary for effective directional search. Moreover, social attachments and idiosyncratic language that accompany consensus-based hierarchy can increase the cost of search by generating search heuristics that are limited in the knowledge sets that they incorporate. While firm-specific language and social attachments lower the cost of communication among coworkers, they encourage actors to oversearch their channels for knowledge rather than searching out knowledge not contained within the firm. Thus, not only are such social attachments costly to maintain (Hanson et al. 1999), but they also may misguide the process of search. Such over- socialization may reduce the infusion of new ideas and result in “parochialism and inertia” (Adler and Kwon 2002), or as Powell and Smith-Doerr (1994, p. 393) argue, the “ties that bind may become the ties that blind.” Individual choices of search trajectory may become increasingly determined by the knowledge that workers and their close friends possess. Thus, social attachments may bias decisions toward continuing patterns of search that extensively utilize existing knowledge sets within the firm and may limit the firm’s capacity to search and absorb new forms of knowledge.
This discussion suggests that the organizational costs of generating consensus are high. Hence, consensus- based hierarchy should only be adopted when the benefits of consensus are high—when problems are highly complex. Thus, for decomposable problems, consensus-based hierarchy is more costly than either market governance or authority-based hierarchy, as it is likely to take more time and trials and hence cost to identify a valuable solution. These costs increase, but slowly, as problems become increasingly nondecomposable, leaving consensus-based hierarchy as a more efficient governance solution for complex, nondecomposable problems.
Source: Nickerson Jack A., Zenger Todd R. (2004), “A Knowledge-Based Theory of the Firm: The Problem-Solving Perspective”, Organization Science, Vol. 15, No. 6 (Nov. – Dec., 2004), pp. 617-632, http://www.jstor.org/stable/30034765.