AI for Testing vs Testing AI Systems: Two Different Jobs

Summary: Using AI to test software and testing an AI system are two different jobs requiring two different skill sets. ISTQB made the split official: CT-GenAI covers using AI as a testing assistant, CT-AI v2.0 covers testing the AI system itself.

Contents

1. Why do these two ideas get confused?
2. How did ISTQB separate the two disciplines?
3. AI for testing (CT-GenAI): AI as an assistant
4. Testing AI systems (CT-AI v2.0): the AI is the thing under test
5. Why can't you have one AI test another?
6. How should a QA manager allocate these skills?
7. When does your team not need CT-AI?
8. Frequently asked questions

1. Why do these two ideas get confused?

The phrase "AI testing" carries two meanings that describe almost unrelated work: using AI to perform testing, and testing a product that is itself AI.

We meet the resulting confusion regularly in conversations with client QA teams. An engineer says the team "already does AI testing," and on closer questioning it turns out they use ChatGPT to generate Selenium code. That is useful work. It says nothing about the team's ability to test a chatbot, an AI agent, or a RAG system (Retrieval-Augmented Generation — answers generated from retrieved documents).

The confusion has real consequences in hiring and staffing. A QA manager recruits an engineer "with AI experience" for a chatbot testing project and discovers the candidate can prompt Copilot for test scripts and nothing more. In the other direction, an engineer who genuinely understands model evaluation gets assigned to routine automation work.

2. How did ISTQB separate the two disciplines?

The International Software Testing Qualifications Board (ISTQB) split these two areas into separate certifications, and the split completed during 2026.

In April 2026, ISTQB published the Certified Tester AI Testing (CT-AI) syllabus version 2.0 — the first major rewrite since 2021. The most significant change is not what was added but what was taken out: all material on using AI to assist testing has been removed from CT-AI (ISTQB, CT-AI v2.0 syllabus release, April 2026). CT-AI v2.0 is now exclusively about testing AI systems.

The removed material has its own home. Certified Tester – Testing with Generative AI (CT-GenAI) was approved by the ISTQB General Assembly on 25 July 2025 and is currently at version 1.1. It covers prompt engineering, the limitations of large language models, and integrating AI into test infrastructure.

CT-AI v1.0 is being retired: the English version remains available until 21 April 2027, other languages until 21 October 2027 (ISTQB, CT-AI certification page). Teams with engineers part-way through v1.0 study should plan against those dates.

3. AI for testing (CT-GenAI): AI as an assistant

The AI-for-testing branch uses AI as an assistant inside a conventional testing process. Three applications are in wide use:

1. Generating test cases from requirements. The model reads the specification and proposes test scenarios. The real value is the speed of reaching a first draft, not the quality of that draft.
2. Synthetic test data. The model produces sample data diverse enough to cover processing branches — particularly useful when production data contains personal information that cannot enter a test environment.
3. Self-healing scripts. When a UI change breaks a selector, the tool locates the corresponding element and updates the script instead of failing.

Self-healing is where the limitation needs stating plainly. The mechanism repairs scripts broken by interface change, but it cannot distinguish between "the Register button's id changed" and "the Register button was accidentally removed from the product." In the second case, a successful self-heal conceals a genuine defect. A suite that heals too well can stay green while the product is broken.

The operating rule that must accompany self-healing: every automatic repair is logged, and the log is reviewed on a schedule. Silent healing that leaves no trace is a risk, not a feature.

4. Testing AI systems (CT-AI v2.0): the AI is the thing under test

The testing-AI-systems branch applies when the product under test is a chatbot, an AI agent, a RAG system, or a machine learning model. Here the founding assumption of conventional testing breaks: the same input no longer yields the same output, so assertEquals stops working.

CT-AI v2.0 organises the work around two test levels specific to machine learning systems, plus a quality-characteristics layer:

Input data testing. Data determines model behaviour, so it must be tested as a software component: is the data representative of real usage, are the labels correct, is a bias being amplified, does the ingestion pipeline corrupt what passes through it.

Model testing. Evaluating the model itself with techniques absent from conventional testing: metamorphic testing (checking relationships between outputs rather than asserting one absolute output), adversarial testing, drift detection over time, and back-to-back comparison between versions.

AI quality characteristics. CT-AI v2.0 anchors this to ISO/IEC 25059 — the software quality standard extended for AI systems, covering characteristics such as robustness, transparency, and explainability.

This is work we do on our own product. VAON builds and operates OneBot, a RAG-based conversational assistant with LINE integration running in production. Testing OneBot, the hard part was never the main conversation flows. It was constructing evaluation criteria for non-deterministic output: whether an answer that is correct on the business rule but phrased differently counts as a pass, and at what threshold the model is judged to be fabricating. No assertion expresses that requirement. We had to build a dedicated evaluation set with human-defined scoring criteria — precisely the work CT-AI v2.0 describes, and precisely the work a strong automation engineer has typically never done.

5. Why can't you have one AI test another?

Asking one AI system to evaluate another produces a loop with no anchor: the system under test and the system judging it can fail in the same way, and nothing inside the loop detects it.

The problem is not that judge models score poorly. The problem is that the judge shares blind spots with the model it judges: overlapping training data, the same tendency to prefer fluent answers over correct ones, the same bias toward familiar phrasing. An answer that is wrong but well-presented can be accepted by both sides.

That does not mean abandoning model-based scoring. It means the test architecture has to be hybrid, in three layers:

1. The determinate layer. Checks with clear pass/fail criteria: output format, response time, whether sources are cited, whether personal data leaks. Fully automated.
2. The model-judging-model layer. Used for fast screening at volume, with accepted error, never as the basis for a release decision.
3. The human decision layer. People define what passing means, review random samples from layer 2, and decide on release. This is the final gate and it is not delegated to a tool.

The ratio between the three layers varies with product risk. An internal chatbot for looking up procedures is a different proposition from a customer-facing advisory assistant, even when both are technically RAG.

6. How should a QA manager allocate these skills?

The right allocation depends on what your team actually tests, not on whether the company is talking about AI.

One note on hiring: in the job description, name the thing under test instead of writing "AI experience." "Testing RAG systems, building evaluation criteria for non-deterministic output" filters candidates far more accurately than "AI experience preferred."

7. When does your team not need CT-AI?

If your product contains no AI component, CT-AI v2.0 offers your team no practical value yet. Input data testing and model testing are skills that only apply when there is a model to test.

Equally, if the team has no written testing conventions and no stable regression suite, introducing AI — on either branch — amplifies existing disorder rather than resolving it. Process foundations come before tooling. We covered that sequence in detail in our article on the four-step loop and verification checkpoints for using AI in testing.

8. Frequently asked questions

Should a QA engineer take CT-GenAI or CT-AI v2.0 first? It depends on the product being tested. If the product has no AI component, take CT-GenAI first because it applies immediately to daily work. If the team is about to test, or already tests, a chatbot, AI agent, or RAG system, CT-AI v2.0 is the right certification, and at least one team member should hold it before the project starts.

If we hold CT-AI v1.0, do we need to retake v2.0? CT-AI v2.0 is the first major rewrite since 2021, with a different structure and focus, so the older certification does not reflect the current content. ISTQB has also published the retirement schedule for v1.0: English until 21 April 2027, other languages until 21 October 2027.

Is using ChatGPT to write test scripts "AI testing"? No. That is using AI to assist testing, which falls under CT-GenAI. Testing AI means the thing under test is an AI system, requiring different techniques such as input data testing, metamorphic testing, and evaluation of non-deterministic output.

Can we use one language model to score another model's output? It can be used for fast screening at volume, but not as the basis for a release decision. The judge model shares blind spots with the model it judges, so a fluent but incorrect answer can be accepted by both. People still define the pass criteria and review samples.

How does testing a RAG system differ from testing a normal chatbot? A RAG system adds a layer that must be tested independently: retrieval. A wrong answer may come from the generation step, or from retrieval returning documents that are valid but irrelevant. These two causes need separate test sets; combined, they make it impossible to locate the fault.

Who should build the evaluation criteria for AI output? The people who know the business rules, not the people who know the tools. Deciding whether an answer passes is a business decision, not a technical one. Test engineers build the mechanism that runs and measures; domain experts define what correct means.

Conclusion and next step

These two career branches will keep diverging rather than converging. Using AI to test is a productivity skill that will be table stakes within a few years. Testing AI systems is a distinct specialisation, and the number of people who can do it currently trails the number of projects that need it.

For a QA manager, the immediate task is not choosing a certification. It is answering one question: does the product your team is accountable for contain an AI component, and if so, who on the team currently defines the pass criteria for its output? If nobody does, that gap comes before any training plan.

VAON tests AI systems on its own product and on projects for Japanese clients. If your team is preparing to take on an AI project, we offer a free QA process and system architecture audit, drawing on our experience in system development and DX consulting.

References

• ISTQB. ISTQB Releases Certified Tester AI Testing (CT-AI) Syllabus Version 2.0. April 2026. https://istqb.org/istqb-releases-certified-tester-ai-testing-ct-ai-syllabus-version-2-0/
• ISTQB. Certified Tester AI Testing (CT-AI) Version 2.0 — certification page (business outcomes, prerequisites, v1.0 retirement dates). https://istqb.org/certifications/certified-tester-ai-testing-ct-ai/
• ISTQB. Certified Tester Specialist Level – Testing with Generative AI (CT-GenAI) — certification page (approved 25 July 2025, version 1.1). https://istqb.org/certifications/gen-ai/