To automate or not to automate? That’s the question. I have tried to condense it into a list of pros and cons to see if automating your testing will be worth your time or not.


We have been living in a world where more and more things are automated and so we have started to take them for granted. However, sometimes it gets forgotten that something automatic requires time to be made and set up in order to work. This is especially tricky in the “Testing World”.

Often we hear of teams that have been asked to automate as much testing as possible. Also that the business is really looking forward to a test suite able to automatically validate their new and shiny systems. However, often it proves difficult to explain to “non-tech” people what are the implications of automating all the testing and the impacts on the team and their velocity. This is why I have tried to condense all the pros and cons of automating each level of testing (according to the testing pyramid) so it can be a bit easier to make the decision of what should be automated (or not) for a specific project.

Depending on the project we will need to decide what is the right approach to testing. Some projects will require to meet high-quality standards in order to be delivered (i.e. high profile app for a bank) while others may only require basic validation in order to be released (quick Prove Of Concept that will only reach a small subset of users).

The testing pyramid

Test Levels

Level 0 — UNIT TESTS

description on unit tests
Software life cycle stages for unit tests

Unit tests have two main functions. They make the developers think about how their code could break which leads them to write cleaner code. Also, they provide confidence in the moment of changing the existing code. Unit tests might seem really expensive at first sight. However, they really pay back the longer your project lives. So, automating them should be considered (almost a MUST have) for every piece of code that will leave longer than a Prove Of Concept.

Level 1 — INTEGRATION TESTS

description of integration tests
Software life cycles for integration tests

Integration tests are responsible to validate that 2 (or more systems) are able to communicate properly between them. It is very common that 2 “perfectly” working systems break because the integration between them is not right. It is for that reason why they should be automated and provide feedback as soon as possible in the SDLC. However, the approach to Integration Testing should be very pragmatic and make sure that they never test functionality but only “integration”. That will help to reduce the cost and improve their efficiency.

Level 2 — END TO END TESTING

End to end testing is usually a very controversial area to automate so I have decided to divide it into 3 different sections so it can be easier to identify where to put your efforts (and money).

Level 2.1 — FUNCTIONALITY TESTING

description for functionality testing
software life cycle for functionality testing

Functionality tests are usually expensive in time to create and run. That’s usually why they tent to be manually executed. At this stage there isn’t any rule on when to automate or not. It really depends on several factors as QA Team size and skills, change frequency expected around the functionality, etc. Automate functionality testing should be carefully studied and not taken for granted.

Level 2.2 — EXPLORATORY TESTING

description for exploratory testing
software life cycle stages for exploratory testing

The table speaks for itself and exploratory testing proves to be really difficult to automate with current tools. The return on it does not tend to be extremely high either.

Level 2.3 — SMOKE/REGRESSION TESTING

description for smoke/regression testing
software life cycle stages for smoke/regression testing

The idea of testing is to provide feedback as soon as possible to reduce the cost of fixing an issue. Regression testing is the easiest place for a “non-tech” person to understand what that exactly means. Automating these test scenarios will provide really quick feedback to a developer on a time frame that it will prove impossible for a manual tester (especially in big projects). On the other side, they have the same inconveniences to automate as the Functionality Tests. Again, the decision on automating these tests should be made considering almost the same bullet points as the functionality tests.

NOTE: All “saving time” benefits are related to the first round of development. Time will be impacted significantly once the development team starts facing issues.

Summary

In order to achieve high-quality standards, automation testing is a MUST. Why do I say that?

Even if in general, the development speed will be increased when working on new functionality. That increase in velocity is not just due to less work, but also to overlooking requirements or defects. If the overlooked requirements or defects are captured during a manual test round which happens right after the development process has finished, then extra work will be required from the development team and consequently reduce their capacity to deliver new functionality.

In the case that those missed details make all the way to production, they will have even a higher impact in the development cycle. Not only due to complexity to fix issues but also on customer satisfaction and business value.

So, even though the costs of your project might seem higher than expected at its very beginning, the return on the investment will be clear once the project reaches maturity. Anyway, always approach the testing pragmatically and adapt it to the needs of your project.

As Martin Fowler mention in his article about the DesignStaminaHypothesis, good design means higher velocity in the future. The same happens when the right level of testing is achieved.

David Ruiz

Senior Software Engineer in Test,Brisbane

David Ruiz

Senior Software Engineer in Test,Brisbane