First generation AI/ML tools made big promises, but they didn't solve testing's fundamental issues.
Post by Jul 1, 2024 9:36:02 AM · 4 min read

The AI Software Testing Market, Part I - The Early Days

When discussing the unique aspects of our Testaify product compared to others in the market, it's essential to highlight its innovative and distinctive features. These features are not just unique, but they also carry the potential to inspire a new era in the AI/ML software testing market, sparking optimism and excitement about the future of software testing.

Understanding the unique categories within the vast software testing market is crucial.

In this context, the AI/ML software testing market is not just significant; it's the future of software testing. Tools with AI/ML capabilities are expected to adapt and survive, making it a market that demands our attention.

I did some research to see if someone had done something similar. While different from what I was thinking, a blog post by a testing services company does a decent job covering some of the players in the market. Here is the blog post I am referring to.

This blog post indirectly covers some of the categories we see in the market and discusses some of the problems these tools are trying to solve. In our post, we will discuss each group's core approaches, the problems they are trying to solve, and the issues with each one.

As outlined in a previous blog post, the fundamental challenge regarding testing is the vast number of combinations required to test a product exhaustively.

The industry created GUI test automation tools to help with this issue. It tried to sell them by suggesting they were easy to use: You just need to record and playback. By the way, “record and playback” is one of the most hated terms in software testing. It does not work. To use a GUI test automation tool, you have to write code. It took many years and different patterns before the industry became effective at creating useful test scripts.

The advent of these tools created a new job: test automation developer or engineer. In other words, someone has to write these scripts and maintain them. Fragile test automation scripts became the shared pain of most testing groups.

Test Automation Helpers

One of the earliest attempts to incorporate AI/ML in software testing was the emergence of 'Test Automation Helpers.’ This group was primarily focused on addressing the issue of fragile test automation scripts.

The first generation of these AI tools implemented self-healing test automation. Gartner describes self-healing as follows:

If a test fails at runtime, AI-augmented tools can explore alternative ways to find the faulty component or information and then fix the broken test with the updated information.

Source: Gartner. (2022, November 28). Market Guide for AI-Augmented Software-Testing Tools.

They focused on freeing test automation engineers from constantly fixing and manually re-running test automation scripts. Some examples are Functionize, mabl, Leapwork, and Testim. Most of these first-generation AI testing tool companies started in the mid-2010s. Some have been acquired, like Testim.

Today, self-healing is a capability that all AI software testing tools must have. Many of these companies continue to evolve, add other capabilities, and expand their support to different aspects of the software testing process.

As you can see, their focus was on resolving the test automation execution problem. From our perspective, these tools are trying to fix the wrong problem. Instead of focusing on the core testing problem, they focus on improving the status quo fractionally. At best, the impact will be small.

Visual Testing

While most vendors took the self-healing path, one company became synonymous with visual testing: Applitools. 

According to Gartner,

Although an application may technically function, it may not render correctly in all instances. Thus, testers need the ability to rapidly perform accurate visual tests across a wide range of OS versions, browsers, and devices, especially for consumer-grade applications. AI can augment visual testing by using a variety of image recognition techniques that replicate a human looking at screens and comparing them. Leading visual tools can also aid with testing for compliance accessibility standards.

Source: Gartner. (2022, November 28). Market Guide for AI-Augmented Software-Testing Tools.

Unsurprisingly, visual testing emerged as an area where AI/ML can help. One particular area of early progress in AI/ML was computer vision. While this approach helps address specific problems, more is needed to help with the core issue around software testing. In essence, you are comparing images to see if something is different. That is a long way from knowing if an application works. Visual testing is a dumb approach to software testing. There is no intelligence behind it except to say these two pictures are different.

Visual testing has become a standard practice for mobile developers that must support many different devices. It is a standard capability on many of the AI testing platforms in the market.

Quality Analytics

A small group of vendors focuses on capturing data about testing to help predict potential problems, prioritize test cases (test selection), or provide trending information about releases. Analytics are essential to continuously improving the quality of your product. Sealights is an example of such a solution. Their product offers features like quality risk insights and test impact analytics. Other vendors like ACCELQ provide test selection features that optimize the test suite by removing duplicate tests or findings. Katalon provides test failure analysis by evaluating findings from previous test runs.

Analytics is an essential feature that most AI testing tool vendors provide one way or another. It is a standard expectation on AI software testing platforms.

Impact of early AI testing tools

Early AI testing tools focus on improving the status quo by automating niche testing practices (visual testing) or helping accelerate test automation execution (self-healing). The three areas we covered are converging. Instead of becoming sole self-healing test automation vendors, vendors are adding visual testing features to their product suites and expanding their analytics capabilities.

We also see vendors expanding their offerings to other testing specialties, such as performance, accessibility, and security, to provide comprehensive testing.

In the next blog post, we will move to the next generation. Specifically, we will discuss the efforts to achieve AI test design.

About the Author

Rafael E Santos is Testaify's COO. He's committed to a vision for Testaify: Delivering Continuous Comprehensive Testing through Testaify's AI-first testing platform.Testaify founder and COO Rafael E. Santos is a Stevie Award winner whose decades-long career includes strategic technology and product leadership roles. Rafael's goal for Testaify is to deliver comprehensive testing through Testaify's AI-first platform, which will change testing forever. Before Testaify, Rafael held executive positions at organizations like Ultimate Software and Trimble eBuilder.

Take the Next Step

Join the waitlist to be among the first to know when you can bring Testaify into your testing process.