the DevOps Handbook

is an excellent book by Gene Kim, Jez Humble, Patrick Debois, and John Willis (isbn 978-1-942788-00-3). As usual I'm going to quote from a few pages.

Make infrastructure easier to rebuild than to repair.
The average age of a Netflix AWS instance is twenty-four days.
Interrupting technology workers is easy, because the consequences are invisible to almost everyone.
In complex systems, adding more inspection steps and approval processes actually increases the likelihood of future failures.
Over the following year, they eliminated testing as a separate phase of work, instead integrating it into everyone's daily work. They doubled the features being delivered per month and halved the number of defects.
Bureaucracies are incredibly resilient and are designed to survive adverse conditions - one can remove half the bureaucrats, and the process will still survive.
When we have a tightly coupled architecture, small changes can result in large scale failure.
Our deployment pipeline infrastructure becomes as foundational for our development processes as our version control infrastructure.
If we find that unit or acceptance tests are too difficult and expensive to write and maintain, it's likely that we have an architecture that is too tightly coupled.
Any successful product or organization will necessarily evolve over its life cycle... eBay and Google are each on their fifth entire rewrite of their architecture from top to bottom.
... which can lead to the unfortunate metric of mean time until declared innocent.
The principle of small batch sizes also applies to code reviews.
80% of MTTR (mean time to recovery) is spent trying to determine what changed.
High performing DevOps organizations will fail and make mistakes more often... If high performers are performing thirty times more frequently but with only half the change failure rate, they're obviously having more failures. [Roy Rapoport, Netflix]
Spiders repair rips and tears in the web as they occur, not waiting for the failures to accumulate. [Dr Steven Spear]

NDC Does C++ Countdown!

It was my pleasure to run a small workshop style session at the excellent NDC-London conference. I ran a fun C++ game which parodies the popular UK TV gameshow Countdown.
  • In the TV version contestants take turns picking 9 random vowels/consonants and finding the longest word in 30 seconds.
  • In my version contestants take turns picking 7 random tokens from 5 categories: (keywords, identifiers, operators, punctuators, literals) and writing the shortest C++ program using all 7 tokens in 8 minutes.
Contestants write their code in customized cyber-dojo sessions which automatically:
  • checks which tokens have been used
  • tells you the size of the program
  • allows everyone to see all the submissions in the review
The rules:
  • tokens must be the correct type; eg you cannot write "." or ... for a dot operator
  • whitespace does not count towards the program's size
  • additional tokens are allowed
  • the program must compile
  • the program is not executed
  • warnings are allowed
In one round Phil Nash selected these 7 tokens:
const vector tokens =
    ".",                  // operator
    "switch",             // keyword
    "snafu",              // identifier
    ",",                  // punctuator
    "\"salmo\"",          // literal
    "goto",               // keyword
    "!",                  // operator
and the winning solution (54 characters long) was:
union X { X* x; };
X snafu() {
  l: switch (X().x,!"salmo"); goto l;
In another round Hulgar Frydrych selected these 7 tokens:
const vector tokens =
    "catch",              // keyword
    "->",                 // operator
    "[",                  // punctuator
    ";",                  // punctuator
    "--",                 // operator
    "foobar",             // identifier
    "operator",           // keyword
and the winning solution (53 characters long) was:
class c {
  c operator->(){ 
    catch(c x[]){
Can you create shorter versions?

the design and implementation of cyber-dojo

At the excellent Agile on the Beach conference in Cornwall I did a presentation outlining some of the history, design and implementation of cyber-dojo. The video has just gone live on youtube.

Continuous Delivery

Is an excellent book by Jez Humble and Dave Farley. As usual I'm going to quote from a few pages...
Software delivers no value until it is in the hands of its users.
The pattern that is central to this book is the deployment pipeline.
It should not be possible to make manual changes to testing, staging, and production environments.
If releases are frequent, the delta between releases will be small. This significantly reduces the risk associated with releasing and makes it much easier to to roll back.
Branching should, in most circumstances, be avoided.
Dashboards should be ubiquitous, and certainly at least one should be present in each team room.
One of the key principles of the deployment pipeline is that it is a pull system.
A corollary of having every version of every file in version control is that it allows you to be aggressive about deleting things that you don't think you need... The ability to weed out old ideas and implementations frees the team to try new things and to improve the code.
It should always be cheaper to create a new environment than to repair an old one.
The goal of continuous integration is that the software is in a working state all the time... Continuous is a practice not a tool... Continuously is more often than you think.
The most important practice for continuous integration to work properly is frequent check-ins to trunk or mainline.
Ideally, the compile and test process that you run prior to check-in and on your CI server should take no more than a few minutes. We think that ten minutes is about the limit, five minutes is better, and about 90 seconds is ideal.
Enabling developers to run smoke tests against a working system on a developer machine prior to each check-in can make a huge difference to the quality of your application.
Build breakages are a normal and expected part of the process. Our aim is to find errors and eliminate them as quickly as possible, without expecting perfection and zero errors.
Having a comprehensive test suite is essential to continuous integration.
You should also consider refactoring as a cornerstone of effective software development.

Building Microservices

Is an excellent book by Sam Newman. As usual I'm going to quote from a few pages...
Because microservices are primarily modeled around business domains, they avoid the problems of traditional tiered architectures.
Microservices should cleanly align to bounded contexts.
Another reason to prefer the nested approach could be to chunk up your architecture to simplify testing.
With an event-based collaboration, we invert things. Instead of a client initiating requests asking for things to be done, it instead says this thing happened and expects other parties to know what to do. We never tell anyone else what to do.
We always want to maintain the ability to release microservices independenty of each other.
A red build means the last change possibly did not intergrate. You need to stop all further check-ins that aren't involved in fixing the build to get it passing again.
The approach I prefer is to have a single CI build per microservice, to allow us to quickly make and validate a change prior to deployment into production.
No changes are ever made to a running server.
Rather than using a package manager like debs or RPMs, all software is installed as independent Docker apps, each running in its own container.
Flaky tests are the enemy. When they fail, they don't tell us much... A test suite with flaky tests can become a victim of what Diane Vaughan calls the normalization of deviance - the idea that over time we can become so accustomed to things being wrong that we start to accept them as being normal and not a problem.
All too often, the approach of accepting multiple services being deployed together drifts into a situation where services become coupled.
Most organizations that I see spending time creating functional test suites often expend little or no effort at all on better monitoring or recovering from failure.

Developer on Fire

Recently I was honoured to be interviewed by Dave Rael, the man behind Developer on Fire.
Click the image below to listen :-)

the design and evolution of cyber-dojo

I've talked about the design and evolution of cyber-dojo at two conferences this year. First at NorDevCon in Norwich and then also at Agile on the Beach in Falmouth. Here's the slides.