Is it a finding or a defect, a butterfly or a cockroach? Whatever it is, savvy developers use their judgment to improve product quality.
Post by Feb 5, 2024 8:54:55 AM · 2 min read

Is it a Finding or a Defect?

Countdown to Testaify Functional for Web

We have significantly progressed in developing our first product, Testaify Functional for Web. Our alpha release is about to go out to partners, so I am focusing my blog posts on subjects related to Testaify Functional for Web.

How Testaify Functional for Web Works

One curious event recently brought to the forefront a conversation we have had internally for several months. The way Testaify Functional for Web works is as follows:

  • Setup: The user provides URL and authentication credentials for an application
  • Discovery: The Testaify Platform uses its AI Discovery Engine to build a model of the application
  • Design: Testaify Functional for Web uses the model to design test cases and generate the appropriate test data
  • Execution: Testaify Functional for Web executes the tests and captures the results
  • Report: Testaify Functional for Web provides these results within our product for the user to review

For some time, we reported test failures as defects. I have been arguing that they should be reported as findings instead. In a recent test run, we had one test failure. The actual finding generated some debate about it. It was not clear it was a defect. We discovered it was a defect, but not the one we thought we found.

What is a defect, anyway?

Many years ago, I shared with some of the team that while working on a web-based application, I made a mistake trying to address what I thought was a set of defects. Our .NET web app did not check at the field level for many different inputs that can generate the ugly .NET error page.

If you have worked in .NET, you know the page I am referring to. The one with the red title letters. The one that looks like this:

When you see "Server Error in '/' Application, what's the cause? Poor quality! What can you do? It's a judgement call.

It has many variations with different messages, but the general template is the same. Every time I saw this page, I got distraught. It meant poor quality, and a lack of attention to detail is a trademark of poor craftsmanship.

While interviewing a QA candidate, I asked her to test the application. She knew how to test at the field level, and it did not take her long to produce the .NET error page. Because of how upset I was, my head was about to explode. I had to do something about it.

I have the advantage of owning the backlog, so I reprioritize the backlog and move several epics to fix these defects. I researched and came up with all the requirements about different inputs that will not be allowed and what information we will present to the user.

We implemented the changes. For the most part, the changes improved the overall quality of the product, but I went too far. I added certain restrictions that immediately made life difficult for our users. What I consider a defect, our users consider a valuable feature. I did discuss the changes with users (customer advisory board). But discussing something and using something are two different things.

Determining if something is a defect takes time and can get complicated. Twice in my career, we have fixed a defect found by customer A, only to be told by customer B that we had just broken one of their favorite features. What one customer considers a defect, another considers a valuable enhancement.

Reporting Findings

Granted, achieving consensus about a defect is relatively easy. But each time, it requires a judgment call. More importantly, it requires consensus with the team and with the customers. Testing is about discovering potential issues and sharing that information.

As such, we will refer to our test failures as findings instead of defects. Our objective is to provide information to our users. They and their customers will determine whether a finding becomes a defect.

While all defects were findings first, not all findings become defects.

About the Author

Rafael E Santos is Testaify's COO. He's committed to a vision for Testaify: Delivering Continuous Comprehensive Testing through Testaify's AI-first testing platform.Testaify founder and COO Rafael E. Santos is a Stevie Award winner whose decades-long career includes strategic technology and product leadership roles. Rafael's goal for Testaify is to deliver Continuous Comprehensive Testing through Testaify's AI-first platform, which will change testing forever. Before Testaify, Rafael held executive positions at organizations like Ultimate Software and Trimble eBuilder.

Take the Next Step

Join the waitlist to be among the first to know when you can bring Testaify Functional for Web into your testing process.