Joe Poon

Often the most painful and time consuming part of a software project is the Last Mile, getting it deployed and working in production.   Why is this the case?  To understand this, it is useful to look at the underlying issues such as not having decent source control, lack of automated tests, excessive branching (and eventual merging), dedicated teams and infrequent releases.  These impede our productivity on a daily basis and when left unresolved, become magnified during the Last Mile.

Without shortening the Last Mile, our deployments are delayed and unpredictable events.  As a result, we are unable to quickly respond to changing business needs.

So how do we shorten the Last Mile? Enter continuous integration.  The beauty of continuous integration is that it is the cornerstone for resolving these issues - it forces us to get source control under control, brings visibility to the health of our code base, challenges us to think of multiple environments and configurations and encourages us to automate repetitive tasks.  True to it’s name, it encourages us to integrate early and often. With CI, we’re already resolving integration issues from day 1 thereby making deployment to production thankfully, a non-event.

Troy Gould and I presented The Last Mile at the Agile Vancouver 2009 Conference this week.  The presentation looks at the problems faced by teams trying to get software released, recommends good CI practices, introduces build pipelining with Cruise as an overall structure to manage the build-deploy-test-release process and encourages us not to stop with CI as a build server, but leverage it to automate deployments to environments including production.  Thanks to those who attended.  It was exciting to see all those actively using CI on their projects.  The slides can be downloaded here (17MB).

Thanks to Owen Rogers and the team of organizers for an excellent event and lining up a great set of speakers including Eric Evans, Michael Feathers and Martin Fowler.

Release early and often. On projects, we strive for small incremental changes deployed frequently to production. To achieve this, we do things such as:

• test-drive our code, build, check-in, repeat
• run continuous integration
• automate deployments

However, despite our best intentions, it all comes to a grinding halt if the business doesn’t want the code in production just yet.

Fear and Freeze
Often called a ‘code-freeze’, the reason for not deploying to production usually boils down to fear that the new code will introduce bugs and that any potential downtime is intolerable.

Delaying deployment out of fear causes problems and increases risk.

• End-users suffer: requested bug fixes/features are not seen for months.
• Ceremonious process: deployment to production becomes a “one-shot” deal.
• Golden window: developers, analysts and end-users feel the pressure to make this deployment perfect.
• Loss of focus: by the time deployment rolls around, developers may be working on other features or get pulled into other projects.
• Assumptions change: over time, requirements may change - your undeployed code is already obsolete. Or, perhaps the production infrastructure has changed.
• Loss ROI: why not capitalize on your software investment sooner and reap the value?
• Big bang: with infrequent deployments, the team will tend to adjust by accumulating code changes into larger deployments.

Deployment and the break fix
Ironically, after deployment to production finally occurs, if something goes awry, the business needs to fix it immediately. Usually this will be achieved by a last minute “break fix”, where the earlier qualms to deployment are temporarily forgotten. The business demands a quick code turnaround, the justification being that it is a “fixing a bug” and “not adding functionality.” It would make sense if this type of responsive mentality was the norm throughout the development cycle and not adopted only for emergencies.

Easing business qualms
As developers we take pride in iterative, well tested code and automated deployments. However, we also need to work with the business and understand why code can’t be pushed frequently into production, and minimize the delays where possible. Sure, there are applications where the nature is truly mission critical or it’s a legacy application where granular deployments are far from trivial. However, for all the other scenarios (which I suspect are the norm) where business fear is the driving force, we can work with the business to ease these qualms.

Quick and predictable
To do so, we need to demonstrate to the business that deployments to production can be a pleasant experience for both project managers and end users. Deployments should be:

• quick and predictable
• a graceful and seamless upgrade to the end user
• responsive and understanding to any errors/oversights that arise

The more frequently we deploy, the better we become at making it a good experience for the business.
In the end, if doesn’t matter if you can go from dev to staging in 60 seconds, if your code is left sitting in the garage.