Software possesses an extremely strange property: it is possible to create high-value software products with effectively zero capital outlay. As Mozilla engineer Sam Penrose put it, software programming is labor that creates capital.
This characteristic make software radically different from engineering materials like steel, and much closer to artistic media such as paint.1 As a consequence, engineer and engineering are somewhat inappropriate terms. It is something of a stretch to even think of software as a kind of engineering “material.” Though all computing requires a physical substrate, the trillions of tiny electrical charges within computer circuits, the physical embodiment of a running program, barely seem like matter.
The closest relative to this strange new medium is paper. But even paper is not as cheap or evanescent. Though we can appreciate the spirit of creative abundance with which industrial age poets tossed crumpled-up false starts into trash cans, a part of us registers the wastefulness. Paper almost qualifies as a medium for true creative abundance, but falls just short.
Software though, is a medium that not only can, but must be approached with an abundance mindset. Without a level of extensive trial-and-error and apparent waste that would bankrupt both traditional engineering and art, good software does not take shape. From the earliest days of interactive computing, when programmers chose to build games while more “serious” problems waited for computer time, to modern complaints about “trivial” apps (which often turn out to be revolutionary), scarcity-oriented thinkers have remained unable to grasp the essential nature of software for fifty years.
The difference has a simple cause: unsullied purist visions have value beyond anxiety-alleviation and planning. They are also a critical authoritarian marketing and signaling tool — like formal dinners featuring expensive china — for attracting and concentrating scarce resources in fields such as architecture. In an environment of abundance, there is much less need for visions to serve such a marketing purpose. They only need to provide a roughly correct sense of direction to those laboring at software development to create capital using whatever tools and ideas they bring to the party — like potluck participants improvising whatever resources are necessary to make dinner happen.
Translated to technical terms, the dinnerware analogy is at the heart of software engineering. Purist visions tend to arise when authoritarian architects attempt to concentrate and use scarce resources optimally, in ways they often sincerely believe is best for all. By contrast, tinkering is focused on steady progress rather than optimal end-states that realize a totalizing vision. It is usually driven by individual interests and not obsessively concerned with grand and paternalistic “best for all” objectives. The result is that purist visions seem more comforting and aesthetically appealing on the outside, while pragmatic hacking looks messy and unfocused. At the same time purist visions are much less open to new possibilities and bricolage, while pragmatic hacking is highly open to both.
Within the world of computing, the importance of abundance-oriented approaches was already recognized by the 1960s. With Moore’s Law kicking in, pioneering computer scientist Alan Kay codified the idea of abundance orientation with the observation that programmers ought to “waste transistors” in order to truly unleash the power of computing.
But even for young engineers starting out today, used to routinely renting cloudy container-loads2 of computers by the minute, the principle remains difficult to follow. Devoting skills and resources to playful tinkering still seems “wrong,” when there are obvious and serious problems desperately waiting for skilled attention. Like the protagonist in the movie Brewster’s Millions, who struggles to spend $30 million within thirty days in order to inherit $300 million, software engineers must unlearn habits born of scarcity before they can be productive in their medium.
The principle of rough consensus and running code is perhaps the essence of the abundance mindset in software.
If you are used to the collaboration processes of authoritarian organizations, the idea of rough consensus might conjure up the image of a somewhat informal committee meeting, but the similarity is superficial. Consensus in traditional organizations tends to be brokered by harmony-seeking individuals attuned to the needs of others, sensitive to constraints, and good at creating “alignment” among competing autocrats. This is a natural mode of operation when consensus is sought in order to deal with scarcity. Allocating limited resources is the typical purpose of such industrial-age consensus seeking. Under such conditions, compromise represents a spirit of sharing and civility. Unfortunately, it is also a recipe for gridlock when compromise is hard and creative breakthroughs become necessary.
By contrast, software development favors individuals with an autocratic streak, driven by an uncompromising sense of how things ought to be designed and built, which at first blush appears to contradict the idea of consensus.
Paradoxically, the IETF philosophy of eschewing “kings, presidents and voting” means that rough consensus evolves through strong-minded individuals either truly coming to an agreement, or splitting off to pursue their own dissenting ideas. Conflicts are not sorted out through compromises that leave everybody unhappy. Instead they are sorted out through the principle futurist Bob Sutton identified as critical for navigating uncertainty: strong views, weakly held.
Pragmatists, unlike the authoritarian high-modernist architects studied by James Scott, hold strong views on the basis of having contributed running code rather than abstract visions. But they also recognize others as autonomous peers, rather than as unquestioning subordinates or rivals. Faced with conflict, they are willing to work hard to persuade others, be persuaded themselves, or leave.
Rough consensus favors people who, in traditional organizations, would be considered disruptive and stubborn: these are exactly the people prone to “breaking smart.” In its most powerful form, rough consensus is about finding the most fertile directions in which to proceed rather than uncovering constraints. Constraints in software tend to be relatively few and obvious. Possibilities, however, tend to be intimidatingly vast. Resisting limiting visions, finding the most fertile direction, and allying with the right people become the primary challenges.
In a process reminiscent of the “rule of agreement” in improv theater, ideas that unleash the strongest flood of follow-on builds tend to be recognized as the most fertile and adopted as the consensus. Collaborators who spark the most intense creative chemistry tend to be recognized as the right ones. The consensus is rough because it is left as a sense of direction, instead of being worked out into a detailed, purist vision.
This general principle of fertility-seeking has been repeatedly rediscovered and articulated in a bewildering variety of specific forms. The statements have names such as the principle of least commitment (planning software), the end-to-end principle (network design), the procrastination principle (architecture), optionality (investing), paving the cowpaths (interface design), lazy evaluation (language design) and late binding (code execution). While the details, assumptions and scope of applicability of these different statements vary, they all amount to leaving the future as free and unconstrained as possible, by making as few commitments as possible in any given local context.
The principle is in fact an expression of laissez-faire engineering ethics. Donald Knuth, another software pioneer, captured the ethical dimension with his version: premature optimization is the root of all evil. The principle is the deeper reason autonomy and creativity can migrate downstream to hands-on decision-making. Leaving more decisions for the future also leads to devolving authority to those who come later.
Such principles might seem dangerously playful and short-sighted, but under conditions of increasing abundance, with falling costs of failure, they turn out to be wise. It is generally smarter to assume that problems that seem difficult and important today might become trivial or be rendered moot in the future. Behaviors that would be short-sighted in the context of scarcity become far-sighted in the context of abundance.
The original design of the Mosaic browser, for instance, reflected the optimistic assumption that everybody would have high-bandwidth access to the Internet in the future, a statement that was not true at the time, but is now largely true in the developed world. Today, many financial technology entrepreneurs are building products based on the assumption that cryptocurrencies will be widely adopted and accepted. Underlying all such optimism about technology is an optimism about humans: a belief that those who come after us will be better informed and have more capabilities, and therefore able to make more creative decisions.
The consequences of this optimistic approach are radical. Traditional processes of consensus-seeking drive towards clarity in long-term visions but are usually fuzzy on immediate next steps. By contrast, rough consensus in software deliberately seeks ambiguity in long-term outcomes and extreme clarity in immediate next steps. It is a heuristic that helps correct the cognitive bias behind Amara’s Law. Clarity in next steps counteracts the tendency to overestimate what is possible in the short term, while comfort with ambiguity in visions counteracts the tendency to underestimate what is possible in the long term. At an ethical level, rough consensus is deeply anti-authoritarian, since it avoids constraining the freedoms of future stakeholders simply to allay present anxieties. The rejection of “voting” in the IETF model is a rejection of a false sense of egalitarianism, rather than a rejection of democratic principles.
In other words, true north in software is often the direction that combines ambiguity and evidence of fertility in the most alluring way: the direction of maximal interestingness.3
The decade after the dot com crash of 2000 demonstrated the value of this principle clearly. Startups derided for prioritizing “growth in eyeballs” (an “interestingness” direction) rather than clear models of steady-state profitability (a self-limiting purist vision of an idealized business) were eventually proven right. Iconic “eyeball” based businesses, such as Google and Facebook, turned out to be highly profitable. Businesses which prematurely optimized their business model in response to revenue anxieties limited their own potential and choked off their own growth.
The great practical advantage of this heuristic is that the direction of maximal interestingness can be very rapidly updated to reflect new information, by evolving the rough consensus. The term pivot, introduced by Eric Ries as part of the Lean Startup framework, has recently gained popularity for such reorientation. A pivot allows the direction of development to change rapidly, without a detailed long-term plan. It is enough to figure out experimental next steps. This ability to reorient and adopt new mental models quickly (what military strategists call a fast transient4) is at the heart of agility.
The response to new information is exactly the reverse in authoritarian development models. Because such models are based on detailed purist visions that grow more complex over time, it becomes increasingly harder to incorporate new data. As a result, the typical response to new information is to label it as an irrelevant distraction, reaffirm commitment to the original vision, and keep going. This is the runaway-train-wreck scenario. On the other hand, if the new information helps ideological opposition cohere within a democratic process, a competing purist vision can emerge. This leads to the stalled-train scenario.
The reason rough consensus avoids both these outcomes is that it is much easier to agree roughly on the most interesting direction than to either update a complex, detailed vision or bring two or more conflicting complex visions into harmony.
For this to work, an equally pragmatic implementation philosophy is necessary. One that is very different from the authoritarian high-modernist way, or as it is known in software engineering, the waterfall model (named for the way high-level purist plans flow unidirectionally towards low-level implementation work).
Not only does such a pragmatic implementation philosophy exist, it works so well that running code actually tends to outrun even the most uninhibited rough consensus process without turning into a train wreck. One illustration of this dynamic is that successful software tends to get both used and extended in ways that the original creators never anticipated – and are often pleasantly surprised by, and sometimes alarmed by. This is of course the well-known agile model. We will not get into the engineering details,5 but what matters are the consequences of using it.
The biggest consequence is this: in the waterfall model, execution usually lags vision, leading to a deficit-driven process. By contrast, in working agile processes, running code races ahead, leaving vision to catch up, creating a surplus-driven process.
Both kinds of gaps contain lurking unknowns, but of very different sorts. The surplus in the case of working agile processes is the source of many pleasant surprises: serendipity. The deficit in the case of waterfall models is the source of what William Boyd called zemblanity: “unpleasant unsurprises.”
In software, waterfall processes fail in predictable ways, like classic Greek tragedies. Agile processes on the other hand, can lead to snowballing serendipity, getting luckier and luckier, and succeeding in unexpected ways. The reason is simple: waterfall plans constrain the freedom of future participants, leading them to resent and rebel against the grand plan in predictable ways. By contrast, agile models empower future participants in a project, catalyzing creativity and unpredictable new value.
The engineering term for the serendipitous, empowering gap between running code and governing vision has now made it into popular culture in the form of a much-misunderstood idea: perpetual beta.
[1] See the essay by Paul Graham, Hackers and Painters.
[2] Modern cloud-computing datacenters often use modular architectures with racks of servers mounted within shipping containers. This allows them to be easily moved, swapped out or added.
[3] The importance of the “interestingness” of work extends far beyond software processes. As Edmund Phelps (see footnote 2 of Towards a Mass Flourishing) notes, based on data from the World Values Survey, that “How the survey respondents in a country valued the ‘interestingness of a job’ (c020 in the WVS classification) was significantly related to how well the country scored in several dimensions economic performance.”
[4] Fast transient is a term of art in a military doctrine known as maneuver warfare. Maneuver warfare is descended from a long tradition dating back to Sun Tzu’s Art of War and the German Blitzkrieg model in World War II. In its contemporary form, it was developed by Col. John Boyd of the US Air Force. The Lean Startup movement is in many ways a simplified version of a core concept in Boydian maneuver warfare: the OODA loop (Observe, Orient, Decide and Act). The Lean Startup notion of pivot corresponds roughly to the idea of a rapid reorientation via a fast transient. A good discussion of the application of maneuver warfare concepts for business environments can be found in Chet Richards’ excellent book, Certain to Win.
[5] For technologists interested in learning agile methodologies, there are many excellent books, such as The Principles of Product Development Flow: Second Generation Lean Product Development by Donald G. Reinertsen and active communities of practitioners constantly evolving the state of the art.