How Testaify Tests Software: State Transition Coverage

Testaify solves “the testing problem” by using agentic AI to automate the complex and often-overlooked practice of state transition coverage, ensuring thorough, up-to-date testing without human effort.

TABLE OF CONTENTS

• The Testing Problem
• State Transition Coverage
  • Why use State Transition Coverage?
  • How Many Testers Use State Transition Coverage?
  • Testaify and State Transition Coverage

One of the interesting aspects of speaking with VCs (venture capital firms) is that you often find yourself answering the same question multiple times. As you answer the question, you continue thinking about it and, in the process, discover new ways of responding. As we meet with VCs, a crucial question we need to answer is: What problem are we trying to solve? I recently started using the phrase “the testing problem” when answering this question.

The Testing Problem

When we say “the testing problem” at Testaify, we mean three related issues that are prevalent in the software testing community:

• Issue 1: Most testing is incomplete and poorly designed. Very few testing teams have comprehensive testing strategies in place. These strategies should cover all possible scenarios and edge cases. Achieving outstanding testing requires a large, well-trained team to perform it effectively.
• Issue 2: Test automation developed with poorly-designed tests is of little value. Plus, test automation becomes obsolete over time. UI test automation is expensive to maintain and run due to its inherent fragility.
• Issue 3: The objective of testing is to provide confidence. Most of the time, that means confidence you can release to production. Bringing all the data together to make that decision has become akin to a form of black magic in the industry.

In this blog post, we will illustrate issue 1 by explaining state transition coverage (aka state transition testing). Additionally, it will allow us to describe how Testaify models an application under test (AUT).

State Transition Coverage

In Louise Tamres' book, “Introducing Software Testing”, Tamres defines a "state machine as a behavioral model whose outcome depends upon both previous and current input." She goes further and discusses state transition diagrams. She describes a "state transition diagram as a pictorial representation of a state machine." She states that "its purpose is to depict the states that a system or component can assume, and it shows the events or circumstances that cause or result from a change from one state to another."

Louise describes the role of state transition diagrams and tables in software testing as follows:

A state transition diagram and its companion table, the state transition table (or state table for short), contain information that readily converts into test cases.

Usually, state transition diagrams look something like this:

Source: https://www.getsoftwareservice.com/state-transition-testing/

Or they might look like this:

Source: https://medium.com/analysts-corner/modeling-system-states-state-transition-diagrams-and-state-tables-d37fe908ff23

Inside Testaify, the state transition diagram looks like this:

In Testaify, a state represents a page that provides a unique behavior. The Testaify platform disregards the concept of pages within a web app and instead focuses on distinct and unique states. Each state represents a different user experience, based on the unique set of actions available from that state.

For example, I can take unique actions if I open the menu or access a form. The state changes each time I uncover a new set of actions.

A transition represents how I enter or exit a state, so it means the action taken. Building a model like this helps us understand how the system works.

Why use State Transition Coverage?

In other words, why is state transition coverage so important? The following is a snippet from Cem Kaner’s Testing Computer Software book that explains it:

Every interactive program moves from one visible state to another. If you do something that changes the range of available choices or makes the program display something different on the screen, you’ve changed the program’s state.

Kaner moved to an example next.

A menu system is a simple example. The program starts with an introductory menu. When you select an option, the program changes state and displays a new menu. Eventually, you get some information, a data input screen, or some other non-menu.

You must test each option in each menu. You should make sure that each selection you make takes you to the state that you should reach next.

Next, Kaner expands on his example, and it feels like he loses control due to the explosion of test cases.

Ideally, you will also check every pathway to every option in every menu. You might be able to reach Menu 15 from a choice on Menu 14 and from another on Menu 27. If so, you should test every choice in Menu 15 twice - once after reaching the menu from Menu 14, again after reaching it from Menu 27. If there are ten ways to get to Menu 14, there are at least ten ways to get to Menu 15, each a different route that takes you through Menu 14. If you can, test them all. Unfortunately, if there are many menu choices or levels, you’ll wear out your keyboard and your fingers before finishing all the possible tests.

Poor Kaner, I can visualize him writing this section and coming to the realization that very few will ever do what he just wrote. The first sign is when he writes, “If you can, test them all.”

The worst part is that on the same page, he writes:

State transitions can be much more complex than menu-to-menu. In some data entry systems, the form you get next depends on the numbers you entered into the present form. Numbers you enter into one field might also affect the range of choices available in another, or trigger the program to ask a series of further questions. 

As you delve into the details, the complexity of state transition coverage becomes increasingly apparent. It's a methodology that demands meticulous attention and a deep understanding of the application's behavior.

How Many Testers Use State Transition Coverage?

The reality is that not many testing professionals grasp or can effectively apply state transition coverage to their testing work. Creating this diagram and its associated table is a testing methodology that most testers struggle to execute. As you saw from the Kaner example, it's a time-consuming and arduous process that demands a thorough analysis to accurately document each possible iteration of a webpage based on the available actions. As such, very few testing teams truly understand their AUT.

If you're unaware of the number of states and transitions in your application, how can you ensure you have enough test cases to cover it? How can you determine the correct number of test cases to design? How can you accurately evaluate the test suite used by your team? These are crucial questions that underscore the importance of understanding your application's state transition coverage.

Most testing teams circumvent this problem by employing alternative techniques to gauge their testing coverage. The most common approach is to use a traceability matrix to map use cases (or features) to test cases. But even this approach is rarely used. While highly mature product organizations have a comprehensive inventory of use cases, most product organizations do not. The most common implementation is a spreadsheet that lists test cases per module.

Testaify and State Transition Coverage

At Testaify, we used state-based models to create a model of the AUT. That should not be a surprise to anyone well-versed in software. Alan Page states the reason in the book “How We Test Software at Microsoft”:

A great deal of software is state-based and can benefit from a state-based testing approach.

Alan goes further when he says:

A state model can clarify understanding of how the application works and what the different test permutations might be.

As such, Testaify builds a state-based model of the application every time the user runs a new test session. That way, we always have an up-to-date picture of the AUT and a clear understanding of the test permutations.

It is clear from Kaner’s example that the challenge of using state transition coverage stems from its complexity and the time-consuming nature of the approach. That is true for a human. Enter AI. Now, we can ask AI agents to build the state transition diagram for us. After that, they can design the test cases and execute them.

If an agentic AI platform like Testaify does everything, then “the testing problem” is gone. The Future of Testing is Here! Join our waitlist to be among the first to experience it!