Introducing the Test Cup Model, Powered by Autonomous Testing

TABLE OF CONTENTS

• The Evolution of Testing Continues
  • Agile Test Automation Models
    • The Test Pyramid Model
    • The Test Trophy Model
  • System Tester Pushback
  • AI and The Test Cup Model

The Evolution of Testing Continues

Agile testing introduced a significant shift in the test automation strategy. Before Agile, most teams primarily developed test automation at the UI level; in essence, this was at the system level. UI-level test automation had numerous issues, and we will not repeat them here. So, agile brought a welcome change. It shifted the emphasis to lower levels, such as unit and integration testing.

Still, AI is opening the door to a return to system testing and a new model that we like to call the Test Cup Model. Before we go into that model, let’s review some of the changes and reactions brought by the agile movement.

Agile Test Automation Models

The Test Pyramid Model

The first of the agile test automation models to gain popularity was Mike Cohn’s test pyramid. Here is the test pyramid:

This test pyramid model suggested a very different approach towards test automation. Instead of creating numerous UI test automation scripts, the engineer had to develop a large number of unit tests. As the figure above shows, unit tests are faster and more isolated than any other. If you add these tests to your continuous integration pipeline, they can indicate if something has broken before running any service or UI tests.

The test pyramid says that the team’s most significant investment should be on unit testing. The test pyramid also says that the team should create and automate more service tests (integration testing) than UI tests.

Due to its overreliance on unit testing, the test pyramid created significant blind spots during the testing process. Some in the testing community have suggested that it is no longer relevant and should be considered an outdated model.

Still, it is essential to recognize that, before the advent of agile and the test pyramid, most teams would not write a single unit or integration test. That approach also has significant issues, so no one wants to return to it.

The issues with the test pyramid, along with the emergence of alternative architecture approaches like microservices, have led to a set of alternatives to the test pyramid.

The Test Trophy Model

One of the most popular alternatives to the test pyramid is the test trophy, developed by Kent D. Dodds. The test trophy looks something like this:

The test trophy model makes two significant changes to the test pyramid. The first one is that it shifts the emphasis to integration testing, or what is referred to in the test pyramid figure as service tests. It also recognizes that a significant amount of development occurs today with interpreted languages like JavaScript, and it requires basic static tests that help identify typos and type errors. For most developers working with sophisticated IDEs, the tool handles most of these checks.

The Test Trophy and other models, such as the Test Honeycomb and the Test Diamond, emphasize integration testing over unit and system testing. That has generated some pushback.

System Tester Pushback

The pushback comes in two groups. The first group does not challenge the emphasis on test automation, but believes that the most important type of testing is system testing. One example model from this group is the Testing Crab. The model looks like this:

Source: https://changelog.com/posts/the-testing-pyramid-should-look-more-like-a-crab

As you can see, the model prioritizes system testing over all other types of testing. Still, it does recognize the importance of those types of testing. This model assumes that the team will automate these tests. It is that assumption that lies at the heart of the flaw in this model. System tests are costly to write, execute, and maintain. You are bringing back all the problems agile tries to solve.

Interestingly, some individuals in the testing community have found the emphasis on test automation over manual testing to be a problem. Some of them have embraced a model that the agile testing community considers an anti-pattern. It is sometimes referred to as the inverted pyramid, but my favorite name is the Testing Ice Cream Cone. Here it is:

Source: https://k-hartanto.medium.com/testing-pyramid-and-testing-ice-cream-cone-what-is-the-difference-6ddde3876c20

While many new development teams have adopted other models, the ice cream cone remains widely used across the industry. Many B2B product testing suites use a combination of manual testing and UI test automation. These teams added integration and unit tests as they jumped on the agile bandwagon, but they still spend considerable time conducting manual testing and creating UI test automation scripts. When you look at their testing suites, they look like an ice cream cone.

The problem is that, while real ice cream cones are delicious, this one sets you back and stops you from becoming a highly effective product development team. Believe me, it is an anti-pattern.

AI and The Test Cup Model

The arrival of AI, specifically autonomous software testing solutions like Testaify, brings the potential for a new model. Like the agile models that came before, we agree that the team should automate most of their tests. But we also agree with the pushback because system testing is the most important of the three types of testing. Still, you do need them all. In the Test Cup Model, AI plays a crucial role in generating and executing these tests, making the process more efficient and cost-effective.

As such, let me introduce the Test Cup Model:

Like the Test Trophy model, we agree to include static testing at the bottom, followed by unit testing at the base of the cup, with a layer of integration tests on top of the unit tests. In a world of APIs, integration tests are essential. Finally, the top layer consists of system tests, which are the most critical. If you need some manual testing, you can pour some beer, and the small foam layer at the top will be your manual exploratory tests. Like the foam, you run them and they go away.

The key difference between this model and the other models is that it utilizes AI to generate and execute these tests. You can implement the model without AI, but then system testing becomes too expensive. Here, AI eliminates the need to create, run, and maintain all those fragile UI test automation scripts.

Thanks to the emergence of autonomous testing products like Testaify, you can run many more system tests. Welcome to the autonomous testing revolution. Let’s embrace the Test Cup Model.