Agile testing is the most important innovation that emerged from the Agile movement. Granted, I am a testing nerd, so I am biased. Brian Marick, one of the original authors of the Agile Manifesto, proposed a matrix in 2003 to understand agile testing.
The matrix is above. At the top of the matrix, you have Business Facing, which refers to tests that deal with the business domain. Any business expert will understand these tests. At the matrix’s bottom, you have Technology Facing. Per the name, the focus here is on the technology or the verification of the technical implementation. Support Programming is at the left side of the matrix. This side includes the efforts that support the development of the specific product. On the right side, you have Critique Product. These are the tests trying to break things (the critique). Altogether, this matrix forms four distinct quadrants.
Quadrant 1, Support Programming/Technology Facing
The first quadrant at the bottom left is for Test-Driven Development (TDD). You write a unit test to help you think about the code you need (TDD) or check the code you wrote (old-fashioned unit testing). Most programming languages today have one or more unit testing frameworks to choose from to help in their development. IDEs (Integrated Development Environments like VisualStudio) support practices like TDD. This quadrant is now an established part of what we do in software development. At least it has increased the number of developers writing unit tests even if they are not implementing TDD.
Quadrant 2, Support Programming/Business Facing
The second quadrant describes the tests that help you understand the capabilities you need to develop. The critical difference between Q2 and Q1 is that these tests arise from the business perspective. These were called customer tests in the early days of XP (eXtreme Programming). They became ATDD (Acceptance Test Driven Development), usually implemented using FIT or Fitnesse. This quadrant has evolved even further to Behavior Driven Development (BDD). The tests read something like this:
| 1 |
Given I have $100 in my checking account |
|
2 |
And I have $10 in my savings account |
|
3 |
When I transfer $50 from my checking account to my savings account |
|
4 |
Then I have $50 in my checking account |
|
5 |
And I have $60 in my savings account |
Such tests are examples of business rules within a specific business domain. This quadrant needs to be understood. For many development organizations, making significant progress within Quadrant 2 is a pipe dream. Still, it’s a vital quadrant because of Shift Left. Quadrant 2 is an essential part of Shift Left testing.
Quadrant 3, Business Facing/Critique Product
Most of the testing in the third quadrant is about the functionality and usability of the application. This quadrant has all our regression functional tests. We verified that we did not break anything by adding the new functionality. This quadrant is also where we validate the usability of the product. If there’s enough time before release, we go into exploratory testing mode to find omissions and mistakes.
The third quadrant is the birthplace of test automation tools and test case management solutions; you’re trying to make all this testing go faster to cover more of your product. It is a very expensive quadrant, as we created a new role for developing test automation scripts and added tools to support this effort.
Quadrant 4, Critique Product/Technology Facing
The fourth quadrant is about performance (a big umbrella on its own), security, and accessibility. The focus of this quadrant is to see if the technical implementation meets the standards for a successful product.
It is also a very expensive quadrant as you have specialists handling the work. Those specialists use a set of expensive tools to do so. Before Agile, QA departments used the waterfall approach for a long time – addressing quadrants 3 and 4.
The thing about Agile testing that is unique is that you can define tests earlier, and you can drive development with tests. You can Shift Left. And you can make testing easier within quadrants 3 and 4. But what Agile thought leaders rarely discuss and never address are the issues in quadrants 3 and 4. We must design good tests instead of spending time fixing broken scripts that provide little value.
Brian Marick’s matrix and the agile testing quadrants help clarify what we mean by Shift Left.
At its most basic, Shift Left means doing more testing to support development and, in other words, reaching the goals of quadrants 1 and 2. Today, it mainly implies following practices like TDD and BDD (Test-Driven Development and Behavior Driven Development).
While it is great to see developers write more unit tests, surveys consistently find that TDD and BDD are rarely used, poorly used, and poorly understood practices. Why is that? If that is the case, are we succeeding at Shift Left testing? Are we producing better quality products?
There are a few reasons why the Shift Left movement seems stuck. The first reason is that Shift Left testing is complicated. TDD is hard. BDD is hard. You are trying to answer difficult questions. Sometimes, you need more research before you can finish the BDD scenarios. Despite their importance, initiatives that require significant time and effort, like TDD and BDD, often get set aside. A few technology executives understand the importance of working on the hard stuff. There is a reason why they gravitate towards “silver bullet” solutions.
Because selling hard stuff does not work well for the immediate bottom line, the industry focuses on other problems like “Scaling Agile.” As the consultants have taken over, they are paying attention to the people signing the checks. Their focus is on selling new silver bullets to these executives. You can see it in the names of some of these frameworks, like SAFe. Who does not want to be safe? It is a marketing triumph, but at what cost? Do you need to scale agile? Is that the most important problem? Answering these questions will require a separate blog post.
Another problem is that we forgot that the work of quadrants 3 and 4 still needs to happen! Just because they were the only things we did during the waterfall days does not mean they are unimportant. The actual QA practitioners know this work is essential. But with the arrival of Agile, we expanded their responsibilities. Now, they need to worry about three quadrants instead of two. And believe me, quadrants 3 and 4 are immense. Some agile thought leaders share the blame as they try to deemphasize quadrants 3 and 4. I made that mistake in the past, too. In a way, we made the problem worse as we never fixed the issue of poorly designed tests. We thought we could fix it in quadrant 2, but the low numbers of BDD practitioners suggest we need to improve.
No wonder many people in QA feel overwhelmed and dissatisfied with their jobs. We either increased their large and significant work or told them to forget about it and Shift Left.
The greatness of the four quadrants is that it gives us a framework to think about testing in all its complexity and glory.
We must consider all four quadrants as we develop any software product. You can only achieve comprehensive testing with all four quadrants.
Now, imagine a tool that will give you what you’ve always wanted: it can handle quadrants 3 and 4 for you. This tool will let your team cover all four quadrants within a sprint. With the power of our AI workers, you will design and execute all the tests you were never able to before.
That is Testaify. We are building Testaify to make it possible to achieve Continuous Comprehensive Testing (CCT). We are building it so you have time to Shift Left without feeling like you are abandoning your beloved product.
Welcome to the testing revolution!
About the Author
Testaify founder and COO Rafael E. Santos is a Stevie Award winner whose decades-long career includes strategic technology and product leadership roles. Rafael's goal for Testaify is to deliver comprehensive testing through Testaify's AI-first platform, which will change testing forever. Before Testaify, Rafael held executive positions at organizations like Ultimate Software and Trimble eBuilder.
Take the Next Step
Join the waitlist to be among the first to know when you can bring Testaify into your testing process.
References:
