Forrester is changing its software testing report!

TABLE OF CONTENTS

• The report is Evolving From Continuous Automation Testing to Autonomous Testing
  
  • Continuous Automation Testing Platforms are Failing
  • The Emergence of Agents is Essential in Autonomous Testing

The report is Evolving From Continuous Automation Testing to Autonomous Testing

As we mentioned in a previous blog post, the traction is there for autonomous testing. A recent example is the following blog post from Forrester Research announcing the evolution of their software testing report: 

We welcome this change, but given the conservative nature of research firms, we do not expect the report to meet our definition of autonomous testing. We discussed the issue with the autonomous testing definition in a previous blog post.

The new report is not out yet, so we should focus on the positive side of this Forrester update.

Continuous Automation Testing Platforms are Failing

Forrester does not put it that way in their blog post, but the following section is quite revealing:

These platforms have been instrumental in reducing the complexity of testing, especially in enterprise environments where applications span multiple channels, technologies, and packaged environments. But CAT platforms have struggled to move the industry beyond 23–25% of automated tests to significantly higher percentages of automation.

You have to admire the talent of these firms. They start by praising the CAT platforms, only to hit them with the painful reality of not moving the needle on a basic metric, such as test automation coverage.

We disagree with the first sentence, but it is better not to dwell on that at this moment. Still, that first sentence gives you an idea of the challenges regarding software testing in enterprise environments. 

The Emergence of Agents is Essential in Autonomous Testing

Forrester introduces the concept of what they like to call “TuringBots.” Specifically, they discuss the addition of “tester TuringBots” by autonomous testing platforms. The idea is bigger than testing, as the diagram from Forrester in the blog post clearly illustrates. Due to Forrester’s publication restrictions, we cannot show you the diagram, but we can describe it.

In Forrester's view of the product development lifecycle, you have several “TuringBots”. Here is the list:

• Analyze/Design TuringBot
• Coder TuringBot
• Tester TuringBot
• Delivery TuringBot
• Collaboration and Work Management TuringBot
• Development Insights TuringBot

I am sure Forrester will update this list in the future. Currently, it is a bot per role. I do not expect that to hold. From what we know of creative destruction, some existing roles will disappear, and new ones will emerge. The bots will not end up copying the human structures. We agree that there has been a long-standing need for a separate platform for testing.

According to Forrester, three factors are driving the emergence of autonomous testing platforms. Here is their list:

1. Generative AI in development. Developers are increasingly using genAI tools to write code, accelerating productivity and creating a need for testing platforms that can keep pace.
2. The proliferation of AI applications. Businesses are deploying AI-driven applications that leverage LLMs and genAI, which require specialized testing to address issues such as hallucinated outputs and complex behavior.
3. A shift to faster delivery cycles. Continuous delivery pipelines demand faster and more accurate testing to avoid bottlenecks in the development process.

Translation, there are only two factors:

1. Thanks to AI, delivering features is faster, so we need speedier and broader testing.
2. Someone needs to test these new AI features, and we assume it is the same group that tests today.

We agree with one (#1 and #3 in Forrester’s list.) From our perspective, number 1 was valid even before the arrival of GenAI. Teams have been struggling with testing for a considerable amount of time.

We are unsure about who and how exactly we will test AI features (#2 on the list). We are in the early stages, and we have already seen many approaches. Many teams are experimenting and trying to figure out that one.

My favorite line in this blog post is the last one:

The future of testing is autonomous, and the journey begins now.

We wholeheartedly agree with that statement. Testaify discovers a web application, designs functional test cases, and runs them. Finally, it shares its findings, including the capability to replay the finding steps, which helps engineers quickly identify the root cause.

Additionally, the setup is straightforward; simply provide us with the URL and credentials, and we'll take it from there.

Autonomous testing is here! Join our waiting list today!