Usability splash page on a computer monitor as a person gets to work.
Post by Jan 22, 2024 9:24:34 AM · 4 min read

Can AI Help with Usability Testing?

AI to the Rescue - Part 9

This blog post is the ninth in a long series. We recently introduced the concept of Continuous Comprehensive Testing (CCT), and we still need to discuss in depth what that means. This series of blog posts will provide a deeper understanding of CCT.

In our introductory CCT blog post, we said the following:

Our goal with Testaify is to provide a Continuous Comprehensive Testing (CCT) platform. The Testaify platform will enable you to evaluate the following aspects:

  • Functional
  • Usability
  • Performance
  • Accessibility
  • Security

While we cannot offer all these perspectives with the first release, we want you to know where we want to go as we reach for the CCT star.

Our previous blog post discussed Usability Testing and why it is so hard. We tried to preempt usability issues for a long time, but the best we got with the existing approaches were weak signals. We shifted right and became better at deploying and testing in production. Can we add something to help us avoid usability issues? Yes, we can. Can AI help us with that? Yes, it can.

The Objective of Usability Testing

While many books talk about methods, techniques, and processes, a few focus on the main objective of usability testing. Our previous blog post quoted the Handbook of Usability Testing: “The overall goal of usability testing is to identify and rectify usability deficiencies.” Like all testing, our goal is to identify potential issues or problems.

Testing can find issues, but it cannot rectify them. The reason this line from the book mentions “rectify usability deficiencies” is due to the assumption that you are conducting usability testing before releasing your product to production. Testing can identify, illuminate, and inform us of potential problems but cannot fix them.

If we go back to the fundamentals, usability testing should focus on identifying usability issues.

Shift-Right Usability Testing

The shift-right approach does allow us to monitor a specific group of users and compare them with a control group. It will enable us to test a hypothesis. Most cloud-based systems implement instrumentation to capture the activity within their products. Many cloud applications use tools like Google Analytics, Amplitude, or Pendo, and together with tools like Split and LaunchDarkly, you can genuinely design usability tests.

The feature flagging tools allow you to turn on a new design for a select group of customers. The app monitoring tools will let you see how these customers interact with the new design. You have a control group because the previous design is still in production. You can get additional information to understand the findings by using existing survey features or adding a survey tool like SurveyMonkey or Qualtrics.

Is this an improvement for usability testing? Yes. Is this something many teams do? According to this Essential UX Statistics blog post, only 55% of companies currently conduct UX testing. Many of these companies need to take a shift-right approach to usability testing. That means they are mostly getting informal feedback and capturing weak signals.

Usability Testing with AI

One of the most significant challenges with usability testing, even the shift-right approach, is data processing and analysis. This problem is one where AI can help considerably. AI/ML systems are excellent pattern-matching solutions, and they can review the data from a tool like Amplitude and come up with findings for the specific group. These vendors will likely work on adding AI/ML capabilities in the future. At least, we hope they do.

An AI capability for data crunching and analysis will significantly improve usability testing for all the teams using a shift-right approach. Can it help earlier in the process? Yes, and that is where Testaify comes in.

Usability Testing - Testaify to the rescue!

One essential platform aspect of Testaify is our discovery engine. The discovery engine builds a model of your application from a user perspective. This model is recreated every time you run a test session. As such, the model changes through time.

The Testaify model will enable our users to learn about potential usability problems. We will know if a path that used to take three steps now takes five. We will know if specific steps are slower or faster. We will provide you with early warning signals that can become the basis of your next hypothesis to test using a shift-right usability testing approach.

At the same time, data integration can provide us with a mechanism to learn from the results captured in the last experiment. The Testaify usability engine will become more intelligent with every test result. This virtuous cycle will improve Testaify’s usability engine prediction capabilities, taking your product to a higher quality standard with every iteration.

Final Thought - AI-generated Personas Usability Testing

The feature I am most excited about in the usability product is the ability to use AI-generated personas. Today, Big Tech uses the data they captured about us to sell us stuff. Testaify will use the data captured by usage monitoring tools to generate AI personas that simulate our users' behaviors.

Imagine using ten AI worker bees to conduct a usability test for a specific user persona like a realtor. Testaify can test the new design by generating ten unique AI worker bees to simulate ten realtors so you can test your new realtor app. Testaify creates, executes, and analyzes the usability testing simulation in minutes.

AI worker bees discover your app, warn you about potential issues, and become AI-generated personas for usability testing. The Future is always fantastic! I hope you join us as we build it.

About the Author

Rafael E Santos is Testaify's COO. He's committed to a vision for Testaify: Delivering Continuous Comprehensive Testing through Testaify's AI-first testing platform.Testaify founder and COO Rafael E. Santos is a Stevie Award winner whose decades-long career includes strategic technology and product leadership roles. Rafael's goal for Testaify is to deliver Continuous Comprehensive Testing through Testaify's AI-first platform, which will change testing forever. Before Testaify, Rafael held executive positions at organizations like Ultimate Software and Trimble eBuilder.

Take the Next Step

Join the waitlist to be among the first to know when you can bring Testaify Functional for Web into your testing process. 

Are you interested in the other blogs in this series?