Many organisations are enjoying the benefits of using open-source software at various points throughout the IT infrastructure. Often this is in the form of Linux operating system and the various server tools that come bundled with it, for example the Apache webserver or Samba file and print server. There are an increasing range of companies out there offering specially packaged, easily maintained, and supported versions of these well known open-source components. However there is more to open-source than the contents of a Linux distribution and as many (perhaps more adventurous) IT managers look out into the vast sea of open-source projects they are left wondering how to evaluate the alternatives.

Open-source software can provide a challenge to the IT manager used to dealing mainly in proprietary products, since it often lacks the accoutrements of COTS packages which make choosing amongst them easier. Typical open-source projects fail to send you glossy promotional material pointing out product highlights and features, don't buy advertising, don't have salespeople to show you demos and point our the best features, don't get reviews in the mainstream IT press (although this is changing somewhat), and don't tell you about all the other people using the software (to make you feel comfortable).

Looking at open-source software as a whole there is a kind of anatomy - a common set of tools and systems that developers use to create the software. Understanding and tapping into the anatomy of an open-source can help you evaluate it and get a picture of the overall project quality.

Mailing lists

Mailing lists form the heart of most open-source projects - this is the main method that everyone in the project uses to communicate with each other. Most projects tend to split the mailing lists between user mailing lists which are about support issues, e.g. installation, maintenance, updates, bugs, etc, and developer mailing lists, about developer issues such as the development roadmap, fixing bugs, how to implement new features etc. Most mailing lists have a web archive of previous posts and this is a good place to start your evaluation.

If you decide to use an open-source product then the mailing list will typically be your first line of support so spend some time browsing the mailing list archives to see what the community around the project is like. Are there many people asking questions? And are the many people answering questions? Are the responses generally helpful? Do the existing project members actively encourage new contributions? As with any open internet communication there will always be a certain amount of noise, but behind that are people helping each other?

Release management & source code control

Since developers on an open-source project can be scattered around the world, they tend to rely on a source code control system to manage the code base and merge all of their work together. The most commonly used system is CVS, although more modern alternatives such as Subversion and Arch are starting to take over. The most notable counter example is the Linux kernel which uses the proprietary BitKeeper system.

Most projects have a browseable web archive of the CVS repository. Look at the frequency and dates of the changes to files - this can give and indication of the life of the project. Do changes happen daily, weekly, never? How many people are making changes? Bear in mind that many contributors wont be given access to CVS, and will submit code through more senior developers.

Similarly the project website usually has a download page which contains not only the latest release but a timeline of recent releases. How often does the project release new versions? If there are release notes or change logs with the releases, have a read to see what sort of changes happen between minor and major releases. Do the project maintainers keep rewriting huge sections of the software each release? Or do they have a mature architecture which they are making incremental improvements to?

Open-source projects tend to use a release numbering scheme similar to that of the Linux kernel. Big changes in the code base are indicated by incrementing the major version number, smaller changes by incrementing the minor number. In addition another convention is to use odd numbered point releases to indicate a development release and even numbered point release to indicate the stable release. So version 1.6 to 1.62 would indicate small changes, bugfixes and simple new features, 1.7 would be the development release leading to the stable 1.8 release which would have more significant updates to 1.62 - while all this is going on the project may also preparing a 2.0 release which would indicate significant rewrites and changes from the 1.x releases.

Bugs database

Whether or not a project uses a bug database is good indication of the size and seriousness of the project team. Two people working on a small codebase can get by without using a bug database, but as the complexity of the codebase and the project team grows a bugs database is a necessity.

When you hit bugs in the code, the bugs database is usually the best way to get your bug onto the developers todo list. How easy is it to report bugs? How easy is it to search for bugs? Have a look at other bugs on the database: how do the developers deal with them? Are the developers using the bugs database? Or do they have one because everyone else does? Mailing lists, release notes, and comments in the CVS archive can also give and indication as to how used the bugs database is.

Documentation

Documentation is typically the weakest area of open-source projects - coders don't like writing documentation - and often it's documentation that's the tie-breaker between otherwise equivalent projects.

Documentation in an open-source project can come in a few different forms. Some projects will have traditional manual style documentation, though it might be incomplete, and many supplement this with FAQs, Wikis (a form of user-created documentation), blogs or forums. These often contain all the content you want from documentation but in disparate pieces you have to find and assemble yourself.

Use the code, Luke

All other evaluation tools aside, the great advantage with open-source software is that you can download it and try it out free of charge - not a demo version or a time-limited trial, the full software, warts and all. Clearly this make the functional evaluation of the software easy, but it can also gives programmers an indication of the quality of the underlying code. Not every IT manager will have qualified programmers at their disposal but if you do have this kind of resource then a day spent getting familiar with the source code can give great insight into the real project: not the veneer of the UI, but the blood and guts code underneath.

Final thoughts

Like any top-level view there are exceptions and counter-examples. Some open-source projects may use a forum or bulletin board instead of a mailing list, some may use IRC or other chat protocols to provide support, some may use a human maintainer as the source code control tool (such as the Linux kernel did before using BitKeeper). There are many great open-source tools waiting to add value to your IT network if you have the time to use project resources to find out and make an evaluation.