Is it a Finding or a Defect?

Countdown to Testaify Functional for Web

We have significantly progressed in developing our first product, Testaify Functional for Web. Our alpha release is about to go out to partners, so I am focusing my blog posts on subjects related to Testaify.

How Testaify Works

One curious event recently brought to the forefront a conversation we have had internally for several months. The way Testaify works is as follows:

• Setup: The user provides URL and authentication credentials for an application
• Discovery: The Testaify Platform uses its AI Discovery Engine to build a model of the application
• Design: Testaify uses the model to design test cases and generate the appropriate test data
• Execution: Testaify executes the tests and captures the results
• Report: Testaify provides these results within our product for the user to review

For some time, we reported test failures as defects. I have been arguing that they should be reported as findings instead. In a recent test run, we had one test failure. The actual finding generated some debate about it. It was not clear it was a defect. We discovered it was a defect, but not the one we thought we found.

What is a defect, anyway?

Many years ago, I shared with some of the team that while working on a web-based application, I made a mistake trying to address what I thought was a set of defects. Our .NET web app did not check at the field level for many different inputs that can generate the ugly .NET error page.

If you have worked in .NET, you know the page I am referring to. The one with the red title letters. The one that looks like this:

It has many variations with different messages, but the general template is the same. Every time I saw this page, I got distraught. It meant poor quality, and a lack of attention to detail is a trademark of poor craftsmanship.

While interviewing a QA candidate, I asked her to test the application. She knew how to test at the field level, and it did not take her long to produce the .NET error page. Because of how upset I was, my head was about to explode. I had to do something about it.

I have the advantage of owning the backlog, so I reprioritize the backlog and move several epics to fix these defects. I researched and came up with all the requirements about different inputs that will not be allowed and what information we will present to the user.

We implemented the changes. For the most part, the changes improved the overall quality of the product, but I went too far. I added certain restrictions that immediately made life difficult for our users. What I consider a defect, our users consider a valuable feature. I did discuss the changes with users (customer advisory board). But discussing something and using something are two different things.

Determining if something is a defect takes time and can get complicated. Twice in my career, we have fixed a defect found by customer A, only to be told by customer B that we had just broken one of their favorite features. What one customer considers a defect, another considers a valuable enhancement.

Reporting Findings

Granted, achieving consensus about a defect is relatively easy. But each time, it requires a judgment call. More importantly, it requires consensus with the team and with the customers. Testing is about discovering potential issues and sharing that information.

As such, we will refer to our test failures as findings instead of defects. Our objective is to provide information to our users. They and their customers will determine whether a finding becomes a defect.

While all defects were findings first, not all findings become defects.