Can we agree on what every term means?

Let’s resolve this annoying software testing problem!

There are significant challenges with software testing, but for some reason, we make the whole thing worse by not having widely agreed-upon shared definitions. Why do different people use different definitions for the exact word in software testing? I do not see this problem on the coding side, except with unit testing. Curiously, the only ambiguous term on the coding side has “testing” in its name! You will not find two different definitions for strategy or command pattern. The single responsibility principle does not have multiple meanings.

I am ranting about this item because of a few blog posts I read while researching a subject. I was trying to figure out what people consider the most significant challenge with functional testing. Well, many consider functional test coverage the most crucial challenge. The problem is that these posts need to be clarified because they either incorrectly define functional test coverage or confuse the metric with the actual root causes.

Is it Wikipedia’s fault?

In one of the posts, the writer defined test coverage as the percentage of a software program exercised by tests. Well, that is code coverage. I found the strange reason behind this confused definition when I searched Wikipedia. First, Wikipedia does not have a separate definition for test coverage. Instead, it takes you to Fault coverage. The first clue this term might be a problem is Wikipedia does not have a page for it. Inside the fault coverage page, you will find the following definition for test coverage:

“The term test coverage used in the context of programming / software engineering, refers to measuring how much a software program has been exercised by tests. Coverage is a means of determining the rigour with which the question underlying the test has been answered. There are many kinds of test coverage:

• code coverage

• feature coverage,

• scenario coverage,

• screen item coverage,

• requirements coverage,

• model coverage.

Each of these coverage types assumes that some kind of baseline exists which defines the system under test. The number of types of test coverage therefore varies as the number of ways of defining the system.”

According to Wikipedia, there are many kinds of test coverage. In this strange world, code and feature coverage are types of “test coverage.” Also, there is something called scenario coverage. Plus, we have requirements coverage. My favorite part of this page is that code coverage is the only one of the so-called “test coverage types” that has a link. In other words, we do not know what the other bullet list items mean.

We're using different words to talk about the same thing.

It makes sense, as many of these “test coverage types” are the same. In behavior-driven development, a feature has scenarios. All these features are your requirements, so what is the difference between feature, scenario, and requirements coverage? Only the scope. They are all requirements.

I can guess what screen item coverage is, but I am unsure. It sounds like requirements coverage. I have seen UX designers reviewing (testing) the application to confirm the implementation of their work. That may be what it means; who knows!

One thing is certain: I do not know what they mean by model coverage. Coincidentally, in Testaify Functional for Web, we built a model for your application. An AI system builds the model as it discovers the application. Testaify uses the model to design test cases using specific testing methodologies. We are not planning to provide you with any model coverage number as we do not see any value.

What is test coverage?

Anyway, let’s get back to the subject of test coverage. Test coverage and code coverage are not the same thing. Code coverage measures explicitly the lines of code, branch statements, or paths taken through your application for a specific set of tests. It is a percentage. It is probably most beneficial for engineers who are validating their unit tests. It does not tell you if your tests are any good. It simply tells you they are touching certain parts of the code.

In a different post, the blog writer defines test coverage or functional test coverage as the percentage of tests executed against some baseline regarding your requirements (hopefully). If you created 100 tests to test feature A and executed only 90 tests, then your test coverage is 90%. It does not tell you if your tests are any good, if you touched 90% of the code, or if 90% of your tests passed or failed.

Is that a useful metric? No, it is not very useful. Suppose your regression suite is automated and running as part of your CI/CD pipeline. You get to 100% test coverage multiple times a day. I know not everyone lives in the Agile nirvana. In some contexts, it still takes weeks to complete regression testing. However, this metric has little value, even in that context. So, why is functional test coverage mentioned as one of the most critical challenges in software testing? I have a theory, but you must read my next blog post to find out. Sorry for the cliffhanger.

In the meantime, let’s develop a common language and definitions. Then, maybe we can start improving testing.