What a nine-year automation estimate taught me

Before changing anything, I needed a baseline. Without a clear picture of the current state, you cannot measure the effect of a change or calculate its ROI. So I went through the last six months of testing statistics and put the main numbers in one place:

• About 700 end-to-end regression scenarios across all customer configurations.
• 20 to 30 minutes to run one scenario by hand, so running the whole suite manually took several hundred hours.
• 20 hours to automate one end-to-end test the conventional way, including writing, stabilizing and reviewing it.

The existing automation also shared a limited pool of real devices, so feedback on a build was slow. Flows that crossed several devices were the hardest to automate this way, and they were also the most important ones for customers.

The math is simple. 700 scenarios at 20 hours each is 14,000 hours. For capacity I used 132 productive hours per engineer per month, once meetings are taken out, which is 1,584 hours a year. 14,000 divided by 1,584 is about 8.8, so roughly nine years for one engineer doing nothing else. At that pace one engineer automates about 79 tests a year.

The usual answers are to hire more automation engineers or to improve the framework. We were already improving the framework from time to time. That is ongoing, iterative work: you find a slow spot, optimize it, and move on. But even if the next round brought 20 hours down to 15, that is a 25% saving, and it still leaves more than six years of work for one engineer. Hiring spreads the same 14,000 hours across more people, but every new engineer still pays 20 hours per test, and the suite keeps growing in the meantime.

The whole calculation turned on the 20 hours it took to create one automated test, so that was the number we had to change.

Besides the statistics, I asked the QA team what took up their week and what they had to repeat on every build.

The project had a large QA team, and about half of it were automation engineers. They covered one part of the regression suite in code. The other half covered a different part by hand, on every release. They knew which configurations tended to break and where, and which steps were easy to miss, but their time went into manual passes. Every new automated test was still written by the automation engineers, at 20 hours each.

So the suite was split in two: an automated part that grew slowly, and a manual part that had to be run again on every release. The question became whether the manual pass could produce automated tests at the same time. If it could, the manual part would turn into automated tests as the team worked through it, most of the 20 hours would go away, and the whole QA team would be creating automated tests, not only half of it.

Before building anything, I looked at the tools on the market that capture test scenarios. None of them fit, mostly for technical reasons. Typically they connected to a device running a production build and then worked in one of two ways. Some read the screen's UI tree as XML and saved every step as a full XPath to the UI element. Others saved X and Y coordinates together with the screen resolution. For our product neither approach was enough, and we wanted more control over the application we were testing. Execution time, support and the learning curve were still problems with these tools as well.

What I found even more useful was talking to customers. I asked to join meetings with some of them, to hear what caused them pain and what problems they had with their own test tools. Most of them said the same thing: when they requested a change in functionality, the UI changed, and their tests failed. Given how most tools on the market save each step, that was to be expected. So the goal was not only to make tests faster to create, but also to keep them up to date automatically when the UI changed.

Building our own tool was going to take real engineering time, so before starting I agreed it with the product's engineering manager as a planned task. Then I built a codeless testing tool around the needs of this product. A tester went through a scenario once, the way they would in a manual pass, and the tool turned that walkthrough into an automated test that could run again on later builds. Nobody had to write code.

A flow that moved between several devices stayed a single scenario from start to finish, as it is in real use. Captured tests could run on real devices or on virtual ones. With virtual devices, a scenario could include as many devices as the flow needed, and many scenarios could run at the same time.

The obvious doubt was whether results on virtual devices could be trusted as much as results on real ones, so I checked that before relying on them. For several months the existing automated suite ran side by side on physical and virtual devices. Both found the same defects, and the virtual devices raised no extra failures of their own.

The most important requirement was that a test captured on one build kept working on later builds, and that this held for every customer configuration. When a new build changed the UI, the tool updated the affected tests automatically, so they no longer failed on every release.

If captured tests had broken on each new release even though nothing was wrong with the product, testers would have stopped trusting the tool before they really started using it. So I put stability across builds ahead of new features.

I was also careful about what I promised. From the start I said that some tests cannot be captured by going through a scenario, and that those would stay with scripted automation. Calling the tool a replacement for scripted automation would have been proved wrong very quickly, and people would then have doubted everything else I said about it.

Manual QA engineers captured the scenarios they already ran by hand, and the captured suite was run on every release build. Scripted automation carried on alongside it for the tests that could not be captured this way.

The figures marked as modeled come from the same calculation as the nine-year estimate, with one hour per test instead of 20. The next section shows which inputs were measured.

In the model, that is a 20x difference per test. Across the whole suite it comes to about 700 hours of work instead of 14,000, or about 95% less effort to create the tests.

Measured on the project: 20 hours to automate one test the conventional way, 20 to 30 minutes to run a scenario by hand, and the time to capture a scenario with the tool, which was about the same as one manual run.

Modeled: I used one hour per captured scenario instead of the 20 to 30 minutes we measured, to leave a safety margin. At 30 minutes per scenario the ratio would be 40x, but I kept the conservative number for this post.

The other inputs are a suite of about 700 scenarios and 132 productive hours a month, which is 22 working days of 8 hours minus about 2 hours of meetings a day.

The model covers creating the tests and nothing else. Maintenance and new tests for changed configurations cost time with either approach, although the automatic updates made maintenance lighter with the tool. Building and supporting the tool costs time only with the new one, and that is not in the 95%.

When I look at a test automation problem now, the first number I ask for is how long it takes to create one automated test. On that project it explained our coverage problem better than flakiness, frameworks or CI capacity did.

The second thing I ask is how many people on the team actually create automated tests. In many QA teams I have worked with, a big part of the team spends its time running regression scenarios by hand on every release, while only the automation engineers create tests. On that project, turning the manual pass into test creation cut the modeled effort to build the suite by about 95%. Another round of framework optimization would have saved about 25%.

I expect the same on other products where each customer runs its own configuration. The suite grows with every new customer, while automation capacity grows only when you hire. If your coverage does not add up, look at the time it takes to create one test and at how many people on the team create them before you open another automation role.