The Apache Foundation’s incredible rise

The Apache Software Foundation recently released its 28-page annual report for its 2015-2016 year, but here’s the TL;DR in one word: amazing.

What started as a simple HTTP server supported by a handful of developers in 1995 has become an army of 3,425 ASF committers and 5,922 Apache code contributors building 291 top-level projects.

Of course, during this same time, open source in general has grown exponentially. But the ASF has seen particularly impressive growth as it propels big data forward with dozens of popular projects, along with dev tools and more general fare. The reason, as board member Jim Jagielski explained in an interview, is the ASF’s emphasis on neutral, community-focused development.

Not bad for an organization that costs less than $1 million to run each year — especially compared to other open source foundations that put the needs of corporate interests above those of the developer community.

First, the data

By any metric, the ASF’s growth in 2015 is impressive:

• 20 new Apache Top-Level Projects (TLPs)
• A record 55 podlings under development in the Apache Incubator, plus 39 initiatives in the Apache Labs
• 743 repositories managed
• 33 percent increase in signed Individual Contributor License Agreements (CLAs)
• 3,425 ASF Committers and 5,922 Apache code contributors (a 21 percent year-over-year increase) added nearly 20 million lines of code, with an average 18,000 code commits per month
• 315,533,038 lines of code changed (65 percent year-over-year increase)
• Apache services ran 24/7/365 at near 100 percent uptime on an annual budget of less than $5,000 per project

Viewed in terms of code, it looks like this:

This is even more impressive when you consider how little money the ASF needs to operate. In its most recent fiscal year, the ASF required $874,000 to run, with much of its budget paid for by sponsors. The ASF has seven Platinum sponsors (Cloudera, Facebook, Google, LeaseWeb, Microsoft, Pivotal, and Yahoo) and eight Gold sponsors (ARM, Bloomberg, Comcast, Hewlett-Packard, Hortonworks, IBM, ODPi, and PhoenixNAP), not to mention a host of smaller ones.

But unlike other open source organizations, the strength of the ASF is its independence from corporate interests, as Jagielski told me in an interview. This independence has created a safe haven for a burgeoning open source developer population:

The Switzerland of open source

When I asked Jagielski to identify the primary reasons for the ASF’s growth, he made it clear that neutrality matters:

The ASF’s growth emphasizes the need, and the acknowledgement of the need, for a neutral, community-focused environment where everyone can work on, and contribute to, a project, without a pay-to-play governance model. In an ecosystem where companies are jockeying for positions of control over open source projects, the ASF provides a safe space where it is the community itself which has, and always will have, control over its own destiny.

Of course, some ASF projects are dominated by a single commercial entity. For example, though Cassandra was initially developed by Facebook, DataStax now accounts for nearly all of its ongoing development.

When I asked Jagielski about this phenomenon, he stressed that “even though some projects have a large, singular corporate ‘association,’ the ASF in general and the PMC [project management committee] take extreme care in ensuring that the influence and control rests on the shoulders of the individuals within the community, and not on the wants or needs of any companies.”

His comments are backed up by the ASF’s governing principles:

A PMC’s actions within their Apache project community and management of their project must be in the interest of that consensus and consistent with the ASF’s mission of producing software for the public good.

There are also certain expectations of diversity within a PMC; the board may apply extra scrutiny to PMCs with low diversity (that is, PMCs that are dominated by individuals with a common employer). Similarly, the ASF does not allow corporations to participate directly in Apache project management or other governance activities at the ASF, only individuals.

This requirement that “Apache projects … govern themselves independently of undue commercial influence” may not always live up to its ideal, but it’s telling that the ideal exists. In my experience ASF projects tend to live up to it, particularly as the projects attain a higher profile. Hadoop, for example, has broad participation that prevents any particular company from controlling its destiny (despite a years-long PR war over who contributed most).

This same ideal isn’t necessarily followed by all open source foundations, a fact called out by Simon Phipps. Far too many foundations are simply facades for corporations that want incoming code but don’t want to share that code freely.

This isn’t surprising. After all, open source has become big business. As Jake Flomenberg, an Accel partner styles it, “There is a massive shift going on in the ways technology is bought. Open source has gone from the exception to the rule.” Fortunately, that “rule” keeps getting support from the ASF, an organization that shepherds more and more of the industry’s most critical projects.