Over the past few years my team have been on a journey toward Continuous Deployment – this has involved changing our architecture, our attitude towards testing, and our tolerance toward slow build times but most fundamentally we have noticed it required a change in philosophy and our attitude toward quality, responsibility and ownership.

In our previous model our mantra was “you release what you test” and with the right phase gates and environments in place it allowed new releases to flow through the hands of QA and into the responsibility of Production Operations. Despite having a huge body of automated tests, our attempts to refine this process to become faster always foundered:

1. we could patch additions into the release branch but this delayed the release while QA assured themselves we were ‘ready’

2. any late patches or bug fixes to the release branch had to be merged back into the Master branch and were sometimes forgotten causing regressions on the next release

3. the longer the delay to the branch release the more the product team tried to shoehorn more stuff into the release branch instead of Master and the vicious cycle continued

4. meanwhile QA were always absorbed getting the branch release out and work on Master was delayed or quality suffered

A recent pivot allowed us to become far more aggressive toward enforcing a strict “no junk in the trunk (master)” policy and we found that when a team embraces Continuous Deployment, they start to embrace a philosophy of personal responsibility and develop the following behaviours:

• ownership of services or code bases

• collaboration with the operations team (devops)

• making sure their code is well-tested and production ready

• releasing code in small releasable chunks

• thinking about backwards compatibility

• collaborating and communicating with your team and the business users that your code changes will affect

• accepting that you cannot live in isolation on your long-running branch without paying the price in conflicts

• realising that the QA team are not your bug finding monkeys

• checking your production logs for errors

• writing the correct level of automated tests to limit regressions without negatively impacting productivity

• monitoring your systems vital statistics (in all environments)

• deciding when to back-out and when to fix-forward (and having a plan for this)

• having a nuanced understanding of risk, quality assurance, testing and release strategies and that some features and bugs can be released immediately while others need in-depth testing

• understanding that you no longer need a “definition of done” because your work isn’t done until it’s successfully released into a stable production environment.

All this comes under the umbrella of personal responsibility and that comes from the knowledge that when a developer chooses to release a piece of code, that code will go to production and they will accept responsibility for it. Along with that responsibility is the pride and reward in knowing that one’s code is released – it’s live, in the wild, and changing users lives (hopefully for the better).

Recently we experimented with adding a traditional formal sign-off to our release process from business owners outside the product team and the results have been interesting. We were faced with either applying a moratorium to Master while sign-off was approved or creating a release branch and letting Master flow. Either way we were faced with similar issues:

1. Developers are no longer able to release their own code so Master starts to differ significantly from what is in production.

2. A “one size fits all” approach is used for release sign-off – whether it’s fixing a typo or refactoring your payment system you still need sign-off as developers are no longer trusted to weigh the risk themselves.

3. The QA team and release manager become responsible for the release, not the individual that wrote the code. The theme of personal responsibility is broken: the developer has moved onto a new task.

4. The release manager starts to manage what goes into the release: features and fixes are forced to languish on branches until the release manager is ready to merge them.

5. As the amount of code ready to deploy grows, release managers get nervous and need more time to test on the release branch.

6. More and more time is spent managing ‘the release’ than writing and delivering features

7. As stakeholders realise that releases take longer, they start to hold up the release for “one last fix” rather than wait another week. A full regression is then required by a nervous release manager and the vicious cycle continues.

8. When you push out your release it has batches of changes in it. If production is affected, it is far more difficult to work out which change caused the problem.

The end result: your release cycle is slower, your team is less productive and less engaged, your QA team is over-loaded and your releases are potentially more buggy and harder to fix.

Continuous Deployment is not just about releasing code fast – it’s about having a team that takes pride in their work and feels responsibility for the quality, stability and effectiveness of the live product.

Our focus now is on improving visibility, accountability and automation so we can better provide rich, descriptive release notifications to stakeholders and regain the capacity to release from within the team rather than have to ask permission from an approval body from outside the team.