Rethinking Quality Assurance in the Age of Generative AI

Generative AI is enabling software developers to create more code much faster than before. For software quality test engineers entrusted with assuring that applications get released on time and without bugs, GenAI presents both a challenge and a solution for addressing this issue. More specifically, it provides an opportunity to emphasize the importance of testing the user experience.

For too long, quality assurance engineers have been made to prioritize functionality, performance, and security testing over other aspects. Now is the dawn of a new era of testing that proves more comprehensive and delivers a broad range of benefits for engineers and testers alike.

The Focus of Software Testing Programs

Software quality has evolved to prioritize the following seven dimensions, listed in the order of their importance to business value:

1. Functionality: Ensuring an application’s fitness so that it meets the requirements of its design.
2. Usability: Evaluating an application’s user interface and workflow to ensure that it can be easily navigated and understood.
3. Performance: Testing the application’s performance under changing conditions to avoid latency issues and optimize response times.
4. Consistency: Validating an application’s output to ensure identical results and behavior, over time and in different locales.
5. Security: Assessing an application's vulnerability to potential attacks to ensure that data is always protected.
6. Resiliency: Ensuring systems remain operational under adverse and unpredictable conditions to recover gracefully and continue to function without critical failures.
7. Adoption to user feedback: Making sure an application adapts to and incorporates new information (as part of feature/functional maturity) based on the user’s feedback.

This list is already fairly daunting. Achieving full coverage of every dimension is a bit like trying to hit a moving target. Expectations can change rapidly, as can the availability of resources needed to meet them. GenAI offers an opportunity to do just that.

Functionality, Performance, and Security Testing Still Lead the Way

GenAI empowers software developers to produce highly creative applications (e.g., those that generate content, including artistic and musical compositions), and testing those applications requires a revised approach compared with traditional testing. The range of possible outputs from these applications is more fluid/flexible when compared with traditional apps, which results in increased:

• Randomness (less predictable output)
• Risk of unanticipated behaviors/outputs
• Performance bottlenecks from intensive algorithms resulting in a higher strain on hardware resources

The only way to meet the challenge presented by the list above is to fight fire with fire to deploy GenAI-assisted tools for test planning and execution. Automated testing tools should form the cornerstone of any testing plan. For the test engineer, this means more effort will be required to audit the software, design test plans, and execute them at scale. But, while the sheer volume and speed of code development will continue to demand considerable attention, test engineers will need to continue to analyze user interactions, both simulating and validating the application's responses and behavior, especially in edge cases where AI models might struggle to generate accurate output. 

GenAI demands a fresh approach to quality assurance, one that works smarter and faster. We need to open ourselves up to applying creative testing programs to adapt to this new way of building software. By rethinking some of our old methodologies, we can better anticipate issues early on and deliver higher-performing, more secure, and more user-friendly software.