#Using AI in QA: A Four-Step Loop With Verification Checkpoints
Summary: Bringing AI into testing fails for lack of process, not lack of tooling. A four-step loop — plan, write, debug, review — places a human checkpoint after each step. The allocation rule: give AI the determinate work, keep the work that requires judgement.
Contents
- Why AI adoption in QA usually fails
- What to delegate to AI, and what to keep
- The four-step loop and its checkpoints
- Why every step needs a checkpoint
- Four mistakes when rolling out the loop
- When this loop is the wrong fit
- Frequently asked questions
#1. Why AI adoption in QA usually fails
Most failed AI rollouts in testing are not caused by model quality. They are caused by nobody defining how far AI is allowed to decide.
The pattern repeats. A team starts with a test generation tool. Early results impress: hundreds of scenarios in minutes. A few weeks later the suite has bloated, nobody dares delete anything because nobody knows what still carries value, and the defect escape rate has not moved. The team concludes "AI isn't good enough for QA yet" and reverts.
The better diagnosis: the tool was dropped into a process that had no place for it. No step specified who validates the output, against what criteria, and what happens when the output is wrong. Absent those rules, everything the model produces is accepted by default — including the parts that are wrong.
This article describes a minimum process to avoid that. For the broader question — how the QA role itself changes between 2026 and 2030 — we covered that separately in the new era of QA in the age of AI.
#2. What to delegate to AI, and what to keep
The useful boundary is not "easy work versus hard work." It is determinate work versus work that requires judgement.
Determinate work has clear pass/fail criteria, narrow scope, and immediate verifiability: generating sample data, turning a specification into a test skeleton, finding recurring patterns in error logs, normalising naming against a convention. Judgement work has no single correct answer: should we ship this build, which risks are acceptable, what scope deserves testing given the resources at hand.
| Determinate work | Judgement work | |
|---|---|---|
| Characteristics | Narrow scope, clear criteria | Multiple trade-offs, context-dependent |
| QA examples | Sample data, test skeletons, log triage | Release decisions, scope selection, business risk assessment |
| Primary owner | AI | People |
| Secondary role | People verify | AI supplies inputs |
| Cost of misallocation | Wasted human time | Serious defects reaching production |
That boundary is moving, but far more slowly than intuition suggests. METR's study Measuring AI Ability to Complete Long Software Tasks (Kwa et al., NeurIPS 2025) measures capability as a "50% time horizon" — the task length, measured in how long humans typically take, that a model completes with a 50% success rate. The finding: that horizon has been doubling roughly every seven months since 2019.
Two practical consequences. First, short tasks are falling into the range AI handles well, so delegate them early. Second, the longer and more multi-step a task is, the lower the completion rate — so decompose before delegating, rather than expecting a model to carry a complex flow end to end.
#3. The four-step loop and its checkpoints
The loop below keeps AI in an execution role within each narrow step, and puts people at the decision points between steps.
| Step | What AI does | What the human verifies | Signal to stop |
|---|---|---|---|
| 1. Plan | Proposes test scope from the spec and codebase | Whether scope matches real business risk; what is missing | The proposal tracks the spec only, never mentions exception flows |
| 2. Write | Generates test cases following the team's existing conventions | Whether tests check actual behaviour, or merely restate the spec | Many tests assert what is trivially true |
| 3. Debug | Triages failures, finds recurring patterns, proposes causes | Whether the cause is evidenced, or merely a plausible guess | Fluent explanation with no specific log to point to |
| 4. Review | Re-evaluates the suite in a fresh, independent context | What the first three steps missed | The review pass agrees entirely with the writing pass |
Step 4 is the one most often cut, and the one most worth keeping. The critical detail: use a fresh context — do not continue the session that produced the suite. A model carrying its earlier reasoning tends to defend its earlier output. A clean context acts as a reviewer who has not read the draft.
The signal in the last column deserves attention on its own. When the review pass agrees entirely with the writing pass, that is not evidence of a good suite. It usually means both passes share the same wrong assumption.
#4. Why every step needs a checkpoint
Because the confidence of an output tells you nothing about its accuracy. A model presents a wrong result in exactly the tone it uses for a right one — there is no surface signal separating the two.
What makes this harder: users themselves tend to check less as they trust the tool more. A study by Microsoft Research and Carnegie Mellon University surveying 319 knowledge workers across 936 real usage episodes (Lee et al., CHI 2025) identified two predictors of critical thinking effort: greater confidence in the tool reduces verification; greater confidence in one's own expertise increases it.
For QA the implication is concrete. Whoever verifies AI output needs enough domain expertise to be willing to push back. Assigning review to junior staff on the grounds that "AI already did the hard part" inverts the order the work requires. This is also why the loop does not reduce the capability bar for a team — it relocates it.
#5. Four mistakes when rolling out the loop
- Cutting the review step to save time. It is the only step that catches systematic errors in the previous three. Remove it and the loop becomes a straight line.
- Reusing the same session for review. A model carrying its own prior reasoning will defend its own prior conclusion. Open a fresh context.
- Measuring by test cases generated. That number rises easily and says nothing about quality. Measure defect escape rate and the share of generated tests humans discard on review.
- Assigning the loop to people without domain grounding. Per the research in section 4, those least confident in their own expertise push back least on AI output — precisely what the loop exists to prevent.
#6. When this loop is the wrong fit
- Projects that are too small. If total test scope fits in a few person-days, running four steps costs more than it returns.
- Specifications changing daily. Step 1 will keep producing drifted scope. Stabilise requirements first, then apply process.
- Systems with absolute legal accountability. Healthcare, aviation, core financial systems — where every decision must trace to a named person. The loop remains usable at steps 1 and 3, but not at steps 2 and 4.
- Teams without written testing conventions. AI generates tests against existing conventions. Without them, output will vary between runs.
#7. Frequently asked questions
Where is the lowest-risk place to start applying this loop? Pick one narrowly scoped module with stable requirements and an existing suite to compare against. Run the loop in parallel with your current approach for two to three weeks, then compare defects found against time spent. Expand scope only once you have that comparison.
Does this loop require special tooling? No. Jira, TestRail, and Selenium keep their roles in storage and execution. The change sits in the layer above: how cases are generated, reviewed, and maintained. The one hard requirement is an AI tool that lets you open a genuinely fresh context for step 4.
How do we know the review step is actually working? Track step 4's own detection rate: of the issues found each cycle, what share came from the review pass. If that share stays near zero across cycles, step 4 is likely reusing prior context, or its review criteria are not sufficiently different from step 2's.
Does the loop reduce QA headcount? That is the wrong expectation to set. It reduces time spent on determinate work while raising the domain-expertise requirement at the verification points. Gains usually show up in time-to-runnable-suite and in defects caught early, not in headcount.
Is it safe to give project data to AI? Handle it at the process level, not only the tool level. Use on-premise models or enterprise services with contractual commitments not to train shared models on customer data, paired with a document classification rule stating which materials may leave for external tools and which may not.
#Conclusion and next step
AI's value in testing lies in compressing determinate work, not in deciding on people's behalf. The four-step loop exists to hold that boundary: AI executes within narrow scope, people decide between steps, and every step leaves somewhere for an error to surface before it travels further.
If your team is weighing how to bring AI into its testing workflow, the sensible first step is establishing where you stand: are testing conventions written down, which work is genuinely determinate, and which must stay with people. VAON offers a free QA process and system architecture audit, drawing on our system development and DX consulting work for Japanese enterprises.
References
- Thomas Kwa et al. Measuring AI Ability to Complete Long Software Tasks. arXiv:2503.14499, NeurIPS 2025. https://arxiv.org/abs/2503.14499
- Hao-Ping Lee et al. The Impact of Generative AI on Critical Thinking: Self-Reported Reductions in Cognitive Effort and Confidence Effects From a Survey of Knowledge Workers. CHI 2025. https://dl.acm.org/doi/full/10.1145/3706598.3713778
- David Ingraham. The Most Valuable QA Skill in the Age of AI Is Thinking. DEV Community. https://dev.to/cydavid/the-most-valuable-qa-skill-in-the-age-of-ai-is-thinking-3b8p