Covering tackles and end-to-end testing

The scrum process is named after a method of restarting play in the game of rugby. As anyone that has participated in a rugby scrum will tell you, the analogy is not a particularly good one. For example, very few daily stand-ups I have attended involved wedging your head between another two team mates’ backsides.

See if you think this rugby analogy is any better:

In rugby, if you miss a tackle or throw a pass that is intercepted, a team mate will need to make a covering tackle to stop the opposition from scoring. Often, the covering tackle will be made by the full-back because part of his or her role is to be the last line of defence for the team.

End-to-end integration tests (manual or automated) are like the full-back in software development. They are often the last line of defence against bugs getting through into production.

If everyone on the rugby team is making their tackles and completing their passes, the need for covering tackles is greatly reduced. If someone has to make a covering tackle it is because something has gone wrong. Similarly, if an end-to-end test catches a bad bug then something has gone wrong; some part of the process missed tackling the bug earlier. Maybe a requirement was misunderstood, an acceptance criteria missed, one or more tests cases were not identified, the code review was not thorough enough, static analysis issues were ignored, the design made a bad assumption, etc. End-to-end tests are meant to demonstrate that everything works together and provide a team with the confidence to release their product. Just as a full-back is not there to make all the tackles for a team, end-to-end tests are not there to find all the bugs for a team.

A good rugby team and good rugby players are always looking to improve their tackling skills and defensive tactics. They look at where and how their opponents have broken through their defensive line and covering tackles were needed. Then they adjust their tactics and train hard to try and strengthen those weaknesses. Good software development teams and good developers do something similar when a bug is caught by an end-to-end test. They look for ways to improve their process, improve their skills, and strengthen their team discipline to ensure they tackle those types of bug earlier in the process.

Why is this important? Covering tackles are an emergency measure that leaves team members out of position, needing to retreat quickly, to reshuffle positions, and reorganise rapidly. This requires extra effort and fast, clear communication by all involved. Some covering tackles in rugby games are sensationally desperate involving sprinting much of the length of the pitch or from one side to the other. The more desperate the covering tackle, the more effort needed from the team to recover and reorganise. Making such covering tackles successfully has great entertainment value for spectators and can give a player a huge buzz, but no player or team wants to have to do this too often during a match.

Similarly, if an end-to-end test finds a bad bug, a software team has to reorganise what they are doing currently and put in extra effort to address that bug. If this happens too frequently, it sucks time and energy from the team leaving less time and energy for developing new features.

Just like covering tackles in rugby, it is inevitable that end-to-end tests will catch the occasional bug but it should be exceptional, not a frequent occurrence, and the less sensational the better.

Two extreme scenarios:

  1. a software development team that relies only on end-to-end tests to catch bugs: this is like a rugby team that does not bother tackling, leaving their full-back to try and do all the tackling for them. Such a rugby team does not win matches. Such a software development team does not deliver high quality software.
  2. a software development team that has no end-to-end tests: this is like a rugby team that plays without a full-back believing they will never miss a tackle or throw a bad pass. This team does not deliver high quality software either.

Where between these two extremes does your team sit?