There's Four, No, Five Schools of Testing

The Journey to Independent Testing via AI/ML - Part 1

As we keep working on our first product, I have been thinking about the evolution of testing. Many people know the story of the first computer bug. 

In September 1947, Dr. Grace Hopper and her colleagues at Harvard opened the computer hardware to figure out what was causing the computer issues they were experiencing when they found a moth trapped in a relay. Because this was the first reported bug in history, it might have been the beginning of testing software, even when it was a “hardware” issue.

In those days, the person writing the code was also the designer and the tester. To quote Flavien Huyn, who wrote the Software Quality: Origin Story post: “... work was done until the program ‘just ran.’ Quality meant ironing out all the bugs that popped out during development.”

Eventually, we built processes, defined techniques, and generally started defining testing. Putting this into a coherent story is problematic because it does not feel coherent. Even today, people use the same words to mean different things.

Bret Pettichord tried to describe the world of testing by using the concept of schools. One of the objectives was to clarify what different experts mean and set the stage for debate. I am unsure if he achieved it, but let’s use the schools as a starting point.

The following are the four schools of testing as defined by Pettichord:

Analytical School

• Software is a logical artifact
• Testing is a branch of Computer Science and Mathematics
• Testing techniques must have a logical-mathematical form
• Testing is technical
• Key question: Which techniques should we use?
• Contribution examples: many testing techniques, code coverage, and conducting tests against a specification

Factory School

• Software development is a project
• Testing is a measure of progress
• Testing must be managed
• Testing must be cost-effective
• Testing is following rules
• Key question: What metrics should we use?
• Contribution examples: requirements traceability

Quality Assurance School

• Software quality requires discipline
• Testing determines whether development processes are followed
• Testers may need to police developers to follow the rules
• Testers have to protect users from bad software
• Key question: Are we following a good process?
• Contribution examples: Quality Assurance over Testing, QA as the gatekeeper

Context-Driven School

• People create software
• Testing finds bugs. A bug is anything that could bug a stakeholder
• Testing provides information to the project
• Testing is a skilled, mental activity
• Testing is multidisciplinary
• Key question: What tests would be most valuable right now?
• Contribution example: Exploratory testing

While separate from Pettichord's original list, I will add a school that emerged after Pettichord defined the first four schools.

Agile School

• Testing can drive development
• Tests can be the specification
• Tests should be automated (mainly below the UI)
• Run tests as often as possible, ideally with every code check-in (keep them passing)
• Deliver as often as possible and reduce lead time as much as possible
• Focus on getting feedback sooner rather than later by collaborating with your users
• Practice ongoing exploratory testing to search for hidden assumptions
• Key question: Are the automated tests passing?
• Contribution examples: TDD, BDD, CI/CD

While reading Pettichord’s slide deck, the bias favoring the context-driven school is clear. It is not surprising as he refers to that one as his school. Instead of emphasizing the differences, I take another approach: I focus on their contributions. I see many of those contributions helping define something better.

We all owe enormous gratitude to the analytical school for codifying many testing techniques essential to any software testing toolkit. We owe the factory school the introduction of concepts like making testing cost-effective. We know you will always continue testing if you want to test everything. In some ways, they brought a sense of pragmatism and context awareness.

One school that many despise today is the quality assurance school. Still, I remember how proud certain members of our QA organization at Ultimate Software felt about having QA on their titles. They did see themselves as guardians of quality in the company. Most of them were former users of the product. At Testaify, we refer to these individuals as system thinkers. They look at the whole system. They feel personally responsible for ensuring users have a great experience using the product. When we implemented Scrum and started calling them testers, they felt offended.

It is not surprising that Apple still uses the title QA. It is always great to have someone who cares deeply about the product and the users in your team. We should be more understanding and let them use the title they feel represents them as they see themselves. Specific agile testing thought leaders (like Lisa Crispin) are on some sort of crusade to remove the quality assurance title and name from the world of software development.

Remember that you cannot use the same answer in every context. Some people do deserve the title QA. I apologize to all system thinkers for changing your title to “Tester” instead of “QA” many years ago. I was wrong.

The context-driven school brought people to the center. It recognizes the time limitations of testing. It pursues hidden assumptions. At Ultimate, I told my teams that assumptions are evil. You need to expose them. Plus, it brought exploratory testing to the world. Testing can be a very creative endeavor. You can use your imagination to find interesting issues.

Agile testing brought us the last 20+ years of improvements in QA. The focus on below-the-UI tests was a crucial development. It helped accelerate development and allowed us to implement continuous integration. The introduction of ATDD and BDD brought testing to everyone on the team (Shift Left).

I do not know any great agile team that does not use techniques or processes defined by the previous schools.

What is coming next is the age of autonomously testing applications via AI/ML. Join me in Part 2 to learn what this new age will look like.