Why do we discuss test automation so much?

Countdown to Testaify Functional for Web

I have been fortunate in my career. I had the opportunity to work at Ultimate Software for 12 years. We implemented Agile when most people were reading about it. We implemented Kanban when most people were still dipping their toes in Scrum. We were doing ATDD using Fitnesse. Most kids today do not even know what I am talking about. We did some fantastic stuff regarding performance testing. The great culture that Scott and Viv built allowed us to focus on quality and relentlessly pursue a better product. As such, we concentrated on testing a lot.

The Test Automation Sickness

But there is one thing we did not escape. We caught a sickness that persists even today across the software testing industry: We spent more time talking about test automation than talking about testing. From WinRunner back then, through QTP, to Selenium today, we spent an incredible amount of time talking, coding, fixing, and dealing with test automation issues. We even created our own web test automation framework at some point.

Test automation is a good idea until it goes too far. According to Wikipedia, test automation in software testing is the use of software separate from the software being tested to control the execution of tests and the comparison of actual outcomes with predicted outcomes. So far, so good.

Test Automation - Waterfall Days

In the case of Ultimate Software, GUI test automation was started as something an isolated team of specialists handled. I managed that team. A large group of testers created test cases, and some tests were sent to the Test Automation team to write and execute the scripts. Regression testing meant running those scripts, plus a lot of manual tests. After my promotion to run QA, which included my prior teams in test automation and performance testing, I wanted to figure out where we were regarding releases. How do we know we are ready to release?

For most of Ultimate’s existence up to that point, that was an exercise checking the gut feeling of a few people who knew the system very well. If most of them feel comfortable, you can release the latest version. The only exception was the performance testing team, which had a clear benchmark to meet before releasing. I wanted to quantify what we had. It took some work, but we started capturing the metrics of our functional test cases. We learned that less than 40% were automated.

Test Automation – Agile Days

We were implementing Scrum throughout the department and wanted cross-functional teams—no more specialist test automation people in their office, separate from the engineers and the testers. As such, we had metrics by team. According to the Agile sales pitch of the day, the one thing we have yet to achieve is increased productivity (around 2005). The consultant will highlight the need for more test automation and a continuous integration pipeline. He said it was time to learn XP (extreme programming). He told us about the testing pyramid.

I will not spend too much time on the testing pyramid, but let’s just say it is tough to do if the architecture of your product does not match the assumptions of the people who came up with the pyramid. Besides, the test trophy is better. Let’s get back to the test automation experience.

It was clear that our test automation level needed to be higher to allow us any increase in productivity. We have the numbers, and we can improve them. We had a goal to increase our test automation coverage from 40% to 80% in one year. We achieved that goal. We also noticed that productivity improvements only appeared once we reached around 60%. From that point on, the time to complete regression kept decreasing until what used to be measured in months started to be measured in weeks.

Too Many Scripts

The company kept growing. We added new products or bought another company or product. We tried to rearchitect what we built to be easier to test below the UI. Eventually, we had over 300,000 tests running every regression. But, the time it took to complete regression, even with 90%+ test automation coverage, was the same as when we were at 80%.

We had so many tests that we spent too much time reviewing and validating the results from the test automation. We know that UI tests are more fragile than below UI tests. While we achieved the results of more reliable releases, the productivity improvements completely stopped. As an organization, Ultimate Software decided to reduce the number of automated tests significantly. We discovered that many of the tests could have been more useful. Some were outright useless. The focus has been on test automation, not on testing. It was time to change that.

Test automation is needed, but it exists to help with testing.

On the one hand, the focus on test automation helped us increase productivity initially. It allowed us to become a lot more predictable. It also reduced bad releases. We used to have at least one or two bad releases per year. That number kept shrinking as time passed. When I left, we had one lousy release every other year. It did provide us with a solid foundation for predictability and consistency regarding quality.

On the other hand, we went too far. Test automation became so pervasive because of the large number of tests that we started delivering fewer and fewer features. We spent time developing frameworks for testing. While we created a testing class to help our teams and all new hires, we spent more time dealing with and talking about test automation than we did about testing.

Many companies wish they had 80% or 90% test automation coverage. Some might want 300,000 automated tests. But if you look carefully, we achieve better quality through brute force, not well-thought-out testing.

One of the reasons we are building Testaify Functional for Web is because we want to focus on designing solid tests. Most of the AI testing tools in the market help teams with their test automation scripts. We are tired of talking about test automation.

We want to use well-established software testing techniques to design and execute test cases that can provide a robust functional testing foundation for all our clients during integration, system, acceptance, and regression testing. We combine what we taught in our testing class and what we learned about test automation to create an AI-first software testing platform that comprehensively and autonomously tests your web-based applications. We want you to stop worrying about your test automation scripts and focus on what is essential: testing your application.

Join us on this journey. And heal from your test automation sickness.