Code Read 8 - Eric S Raymond's Cathedral and Bazaar

The ever evolving "The Cathedral and the Bazaar" by Eric S Raymond (aka CatB) has become something of a lengthy read. Scott Rosenberg's Code Read 8 dives on in, and rightly describes it as a "classic essay" and says it "has proved its importance" in the literature of software development. I first read it several years ago, it was considerably pithier then. However, it is still full of important ideas.

Cathedrals and Bazaars

The most important idea is its title track - two different ways to build software. In the "cathedral" style, a master architect and a small group of hand-picked skilled craftsmen work toward a grand vision with lots of direction and coordination. This is the traditional model of software development. The "bazaar" model, which is common to many open-source projects, particularly Linux, is one in which there are no hand-selected craftsmen, and no master plan. Instead the people in power select the best contributions from those people interested and able enough to contribute something worthwhile. The direction in which the project evolves is determined by the availability of volunteers willing to push it in that direction, as well as an entity, often an elected committee, which acts as an editor.

CatB presents a strong case that high quality software can be produced quickly and efficiently using the bazaar approach. Actually, to many people who just came into the software world in the last five years or so, this seems self-evident - Linux, Apache, MySQL, PHP, and Mozilla are just a few of the high-quality, feature rich, and reliable projects built in the bazaar mode. They are part of the landscape of the Internet which many take for granted. Yet less than 10 years ago there were many thoughtful, intelligent people who had serious concerns about the utility of any of these. Today, of course, the only people who seriously argue that open source development can not produce good software are those who have a stake in commercial alternatives. So while CatB's argument is convincing, it has already been won.

Much of CatB is also devoted to describing how successful open source projects work. If you are planning on running an open source project, these are must-read sections. But even if you are not, the idea of project leader as editor instead of architect is a very interesting one, and reflects a common gem of an idea in leadership theory that is subtle and often misunderstood: some of the best leaders do not lead by inspiring others to follow the leader's ideas, but rather the leader finds and supports the best ideas of the people they are leading. What makes Linus Torvalds (the creator and benevolent-dictator-for-life of Linux) such a genius is not his ability to write code or convince others of the correctness of his ideas (which both are probably impressive), but rather his ability to pick and choose the best from among the many contributions to Linux. There is a great deal that goes along with that style of leadership that is difficult for a type-A ego, and CatB delves into all of it with great insight.

The Twainian Passing of Closed Source Software

The second main thrust of CatB is that traditional management styles and closed-source software will ultimately be washed away under the coming wave of high-quality open source software and its management processes. This is a bit more controversial - and in some cases I think it is just plain wrong. Certainly it is possible that Apache may become the only web server anybody uses. It is also possible, although a bit more of a stretch, that open-source databases will replace commercial databases, or that Linux will become the dominant operating system, or that OpenOffice will become the only office productivity suite. But it unlikely that anybody but E-Bay will ever see the software that runs eBay, or that Google will ever open-source their search software. Sometimes, software is so tied to the fundamental service a business provides that there is just not enough interest in an open-source equivalent. We would all like to be making money like eBay, but how many of us are actually trying to write on-line auction software to match eBay's? There are simply to few developers to support it - especially since any E-Bay competitors needs to distinguish itself, which will probably require substantially different software. So there are some markets in which there is simply not enough demand for open-source software to make it viable.

Another example is corporate web sites, which will always be paid for in the traditional sense, even if they are built using entirely open-source software, simply because nobody but Acme Widgets needs an Acme Widgets web site.

This is one of the things I think Cat B misses in its open-source evangelism. Open-source projects work well when many programmers need the same thing to support the businesses they work for - so we see web servers, operating systems, programming languages and tools, web-site management software, a shopping cart or two, image and photo editing software, an office productivity suite, and so on. But the success of an open-source project depends on having a legion of programmers who need it. Yet CatB argues that commercial software is going away, along with all of the management processes that come with it.

I just do not buy it. Instead, I see the comercial software market becoming smaller, and more individualized. Nobody buys compilers anymore. GCC, gmake, Ant, Eclipise, and dozens of other free products all work fine. Moreover, support for open source products is often better than support for commercial equivalents. (I speak from personal experience.) A lot of software is going open source. But there will always be a market for software to support the specific business processes of individual organizations, which will require one-off construction - even if it does use off-the-shelf open-source components.

The Corporate Embrace of Open-Source

Indeed, many open-source projects are now significantly supported by companies (like IBM, RedHat, and others) that make money building exactly that kind of custom software. The underlying components are no longer something that distinguishes one competitor from another in the marketplace, so companies that compete against each other are joining forces to make everyone's job easier. Companies like IBM, RedHat, Oracle, and countless others are paying programmers to write code which the company will then give to its competitors.

This is one of the most interesting facets of the open-source revolution - the evolution of business strategy and corporate intellectual property policy in the face of the commodification of software infrastructure. In some ways, the software market has become an interesting experiment in altruism. More and more technology companies are realizing that despite superficial appearances, their software is not the core value they provide. And in that case, it makes financial sense to cooperate with their competitors (and anyone else who is interested) to build and maintain that software, while they focus on the what real value that they do provide.

P.S.

One final note - people can be slow to accept change. Some managers and programmers are still protective of "their" code against people in their own organization. This seems to me to be doubly backward, and I hope that as more and more open-source software succeeds, IT departments will learn let more open-source software practices through their cathedral doors. It can only make our lives easier.