Is software quality assurance better today than 20 years ago?

Is QA good enough? Part 1

While looking at Threads, I saw this message from Gergely Orosz:

It caught my attention immediately. Gergely writes The Pragmatic Engineer newsletter on Substack, and I have followed him for many years. He always provides valuable information.

QA is better now than 20 years ago, but …

As a big fan of Agile testing, we have made much progress in QA in the last 20+ years. The introduction of TDD (Test-Driven Development), CI/CD (Continuous Integration/Continuous Deployment), ATDD (Acceptance Test-Driven Development), and its most recent variation, BDD (Behavior-Driven Development), have had a significant impact on quality. For too long, we were focused only on the critique side of the matrix. In the last 20 years, we have begun to embrace the support programming side of the Marick Test Matrix. Quality is everyone’s responsibility.

So, the answer to the first question, Is QA better today than 20 years ago? Yes, QA is better today than it was 20 years ago. The second question is the problem question: Is QA good enough? Here, the answer is less clear.

Is QA good enough?

Agile consultants used the Standish Group’s CHAOS report to point out that we needed to change how we develop software. The CHAOS report does confirm that assertion based on this summary from the 2020 report:

These numbers tell us that we made progress from the Waterfall days. A success rate of 42% is way better than 13%. But I do not know anyone who will be happy to start a project and feel comfortable that the chances of success are less than 50%. In other words, the likelihood of a successful software project has worse odds than a coin flip. That is not good enough.

The Outsourcing Experience

The problems I see are all reflected in some of the replies to the Thread shown above. Here is one of the replies:

Fun fact, most companies have done away with any real QA departments and have off-shored it or have done away with "Quality" as a whole and focused on profit-making departments only, rather than anything but the backbone of their product development (if any). 

The outsourcing trend started before Agile software development became popular, but we saw signs that an outsourcing strategy would become the status quo for some time. As a CTO, I have experienced pressure from stakeholders to outsource some software development, especially QA. I have only faced this issue at places I have worked in in the last six years. These days, outsourcing is the norm.

In my experience, outsourcing is a good strategy for implementing a “Follow the Sun” approach. You can get products coded and tested 24 hours daily. However, it does require a clear understanding of the processes, standards, and practices you expect all your teams to follow. I usually build a development organization with people in three different time zones. That way, work can continue throughout most of the day.

The problem starts when you use outsourcing only to reduce costs. In that context, as that Thread reply implies, testing is treated as a cost center where executives can move the QA work to a cheaper place. I disagree with this approach. When leaders follow this path, it is because they do not care about testing. Consequently, product quality suffers.

Certain firms consider QA easy to outsource because of their greater focus on UI test automation. Let QA write and execute test automation scripts. Besides, test automation is not product code, and it is less critical. These organizations think you can do this work anywhere. Ironically, the QA community contributes to this attitude. QA team members must ask to participate at the beginning of the process. Too many in QA are happy to stay back and wait until they are needed. Unsurprisingly, executives think that QA is a group they can outsource.

You Always Need More, But You Never Know What You Are Getting Back

Several years ago, I met a CEO. I was pitching my company, which focused on testing design and requirements. During the conversation, this CEO told me, "With QA, you always need more people, but you never know what you are getting back.”

As an industry, we need to explain testing and its value better. Eventually, like Boeing is learning now, the market will tell you the value of quality; by then, it will be too late. Many companies continue to outsource or remove QA from the team altogether.

Finally, we need to explain the complex nature of testing better.

We can address all three points – what testing is, why it’s valuable, and why it’s so complex – by returning to Brian Marick’s four quadrants model.