Time to Challenge the Testing Orthodoxy

Here are my issues with the White Box and Black Box testing terms.

As I have mentioned here, here, and here, I have an issue with terms and their use in software testing – sometimes with the definitions, but sometimes with the unexplainable limitations imposed on the term. In the case of White-box and Black-box testing, the definitions are acceptable. The imposed limitations, however, are another matter.

Definitions

Let's delve into the practical implications of these testing categories. White-box testing, as per Wikipedia, involves an internal perspective of the system to design test cases. That means you can see the code. On the other hand, Black-box testing is a method that examines the functionality of an application without delving into its internal structures or workings. In simpler terms, you cannot see the code in Black-box testing but can test its functionality. These definitions, at least, are clear and widely accepted.

The Problem

My issue is with the limitations imposed on each box. Usually, which techniques I can apply to one rather than the other. I will start with a simple example. Boundary value analysis is considered a Black-box testing technique. First, here is a quick primer on boundary value analysis.

Let's take a real-world example to understand the practical implications of our approach. Imagine you're designing software for a traffic ticketing system, a common scenario in today's world. These systems, found in many places worldwide, capture your speed on a scanner, take a photo of you and your plate number, and send you a ticket by email or regular mail.

According to the requirements, the speed limit is 55 mph (miles per hour). The requirements state the following information:

• Any car driving ten mph above the speed limit will be issued a fine of $100.
• For any additional ten mph above that number, the fine will be increased by $50.

The speed limit is what we need to test. The most important thing we need to determine is the boundary. Like most requirements, this one is okay but needs more precision. So, during standup, one of the engineers asked the following questions:

• If the speed limit is 55 mph, is the first fine at 65 mph, ten mph above the limit, or at 66 mph, eleven mph above the limit?
• I have a similar question to the increase in the fine: if the speed limit is 55 mph, is it $150 when it reaches 75 mph or 76 mph?

The product manager answers: In that example, the fine is issued when the vehicle reaches 65 mph and increases to $150 when it reaches 75 mph. The engineer's questions were simply to clarify the boundary. At 64 mph, there is no ticket; at 65 mph, there is a $100 fine. Plus, at 75 mph, the fine goes to $150, and at 85 mph, it will go to $200, and so on.

Let's say the engineer practices TDD (Test-Driven Development) and writes tests to verify the boundary. In TDD, you write the test, run it to see if it failed, and then write the code to pass it. You can use those tests to safely refactor your code if needed, as you can validate with your tests if the code still functions correctly. 

Is that White-box or Black-box testing? According to the software testing world, boundary value analysis is a Black-box testing technique. I wrote the test before I wrote the code; does that make it Black-box testing? But I am the one writing the code, and I can see it. Software testing literature says boundary value analysis is a Black-box testing technique; you cannot use it in your unit tests. Why not? The funny thing is that the definition of boundary value analysis in Wikipedia shows code to explain it. Yet dozens of software testing books tell you that this technique is a Black-box testing technique. Does it matter if it is White-box or Black-box testing? In my opinion, no, it does not matter. So why am I talking about it?

The Impact

Well, because we live in the world and have to deal with it as is. In our case, in the opposite direction. Testaify uses a so-called White-box technique in a Black-box context. Do not worry about us; the software testing gods are feeble and mostly powerless, so no lightning has hit us yet.

According to the software testing world, a technique called Basis Path Coverage (also known as Basis Path testing, Path testing, etc.) is a White-box testing technique. Basis Path Coverage falls within the family of Graph testing techniques, as does State Transition Coverage (also known as State Transition testing, etc.), which is considered a Black-box testing technique. And the nightmare continues.

According to Boris Beizer’s Software Testing Techniques book, a graph is an abstract model consisting of nodes (represented by circles or dots) and links (represented by lines with arrowheads), possibly with values associated with nodes and/or links. Beizer continues by adding that graphs represent the behavior or structure of many things, such as software, data, or relations.

Robert Binder’s Testing Object-Oriented Systems book states that path coverage is achieved when every possible entry-exit path has been exercised at least once. That makes sense, as you are using a graph as your model. According to Binder, a basis-path test model requires that a specific number of entry-exit paths be covered.

These well-known software testing books consider Basis Path Coverage a White-box testing technique and have a very narrow definition to accompany it, even though it is based on a graph. Graph models are not exclusive to software code.

The Testaify Way

At Testaify, we embrace a flexible approach to software testing. We build a graphical representation of your web application, enabling us to apply graph-based techniques to generate test cases. We use Basis Path Coverage to test your web application, a technique traditionally associated with White-box testing. We use Basis Path Coverage in a Black-box context. This flexibility sets us apart and allows us to challenge the software testing orthodoxy.

Yes, it is against software testing orthodoxy, but as I read somewhere, “Challenging orthodoxy is central to innovation.”

If you're intrigued by our approach and want to join the testing revolution, subscribe to our mailing list. By doing so, you'll be among the first to receive an invitation to use Testaify. Together, we can challenge the software testing orthodoxy and pave the way for innovative testing. The Software Testing Revolution Starts Here!