QA in the Age of AI: Don’t Just Test the System - Find Ways to Break It
AI is changing the way software is built. Features that once took hours or even days to implement can now be created with just a few prompts. But as coding speed increases, another problem is emerging: code is being generated faster than teams can test and verify it.
In this context, the role of QA is not disappearing. On the contrary, QA needs to become more proactive and adversarial: not only checking whether the system works according to the expected scenario, but also finding ways to push the system into states that the developer — whether human or AI — may not have anticipated.
How Much Code Is AI Writing?
There is no single answer to this question because different studies use different measurement methods. Some surveys measure the percentage of code that has been AI-assisted, while others only count code that was entirely generated by AI or actually deployed to production.
According to Sonar’s State of Code 2026 survey of more than 1,100 professional developers, respondents reported that AI currently contributes to approximately 42% of committed code. This figure is projected to increase to 65% by 2027. However, 96% of developers said they do not completely trust AI-generated code, and only 48% said they always verify AI-generated code before committing it.
Another survey by Fastly, covering 791 professional developers, found that 32% of senior developers said that more than half of the code they put into products is AI-generated.
Meanwhile, an analysis based on millions of developers estimated that the actual proportion of AI-generated code in production is lower, at around 26.9%. The difference highlights the importance of distinguishing between “AI-assisted code” and “AI-authored code.”
Regardless of which measurement method is used, the overall trend is clear: AI is becoming increasingly involved in the software development process.
And that leads to a more important question:
If AI is writing more and more code, who will find the cases that AI did not think of?
AI Usually Solves the Goal, Not the Entire Real World
An AI coding agent is usually given a specific goal:
- Create a screen.
- Add an API.
- Build an upload feature.
- Add a payment feature.
- Fix a bug in a repository.
- Write tests for a module.
AI can complete these tasks very quickly. But “completing the requirement” does not mean “covering every way a real user can interact with the system.”
An AI-generated flow may work perfectly like this:
Open the screen → Enter valid data → Click Submit once → Wait for the response → Receive a success notification
This is the happy path — the ideal usage flow that developers typically describe in a requirement.
But real users may:
- Click Submit two or three times in rapid succession.
- Refresh the page while a request is being processed.
- Open the same record in multiple tabs.
- Retry a request after a network interruption.
- Continue working with data that has already been deleted in another tab.
- Continue using the interface after their permissions have changed.
- Submit data at the minimum or maximum boundary.
- Combine multiple features in an order that was never anticipated.
An analysis of the QA coverage crisis surrounding AI-generated code also indicates that AI-generated implementations tend to focus on common scenarios while easily missing edge conditions, variations in user behavior, and exceptions in workflows.
This is exactly the gap that traditional testing can easily overlook when it focuses primarily on the happy path.
When All Tests Are Green but the Product Can Still Fail
A test suite can pass because it is testing exactly the situations that the developer had already thought about. If both the code and the tests are generated from the same limited description, they can also overlook the same group of problems.
For example, a delete feature may pass basic tests:
Open the record → Click Delete → Confirm → Record disappears
But what happens in these situations?
Click Delete twice at the same time Delete in Tab A while editing in Tab B Delete a record and then submit an old form Retry after the request times out Delete while a background job is processing
A testing guide for AI-generated code recommends checking boundary values, API contracts, integration failures, and adversarial inputs such as duplicate submissions, timeouts, invalid JSON, or declined payments.
Therefore, the status “test passed” only means that the system passed the scenarios that were tested. It does not prove that the system is safe against every possible real-world usage pattern.
QA Needs to Shift from Checking to Challenging
Edsger Dijkstra once said:
“Program testing can be used to show the presence of bugs, but never to show their absence!”
Testing can demonstrate the existence of bugs, but it cannot prove that software is completely free of bugs.
In the age of AI, this principle is even more important to remember. When AI helps generate code faster, QA should not simply try to run more and more similar tests. QA needs to change the question:
- Not only ask: “Does this flow work correctly?”
- But also ask: “How can I make this flow fail?”
This is the mindset behind Adversarial Testing.
Adversarial Testing does not mean random testing or deliberately breaking the system without control. It is a systematic testing approach in which QA intentionally plays the role of:
- an impatient user,
- a user who makes mistakes,
- a user with multiple tabs open,
- multiple users acting at the same time,
- an unstable network,
- requests completing in an unexpected order,
- or an actor intentionally pushing the system into an abnormal state.
Questions QA Should Ask
For every feature, QA should start with questions such as:
- What happens if the user repeats this action?
- What happens if the action is interrupted halfway through?
- What happens if two requests run concurrently?
- What happens if two tabs have different states?
- What happens if the data is changed by another user?
- What happens if the request succeeds but the UI thinks it failed?
- What happens if the UI shows success but the backend actually fails?
- What happens if an old request completes after a newer request?
- What happens if access permissions change between opening the form and submitting it?
- Which states must always remain valid, regardless of how the user interacts with the system?
These questions help QA test risk categories that are often overlooked:
- duplicate action,
- race condition,
- stale state,
- multi-tab conflict,
- out-of-order response,
- network failure,
- retry behavior,
- input boundary,
- authorization change,
- cross-feature interaction,
- data integrity,
- and partial completion.
A Simple Example
Suppose the system has a file upload feature.
Happy path
Select file → Upload completes → Save file ID to the database → Display file in the UI
Adversarial Testing expands this into:
Upload the same file twice Upload from two tabs simultaneously Refresh while the upload is in progress Close the modal while the upload is running Submit the form before the upload completes Retry after a timeout Delete the file while a background job is processing Upload a file with a valid extension but an incorrect content type
QA then needs to verify the following invariants:
- No orphaned files exist in storage.
- No database record points to a file that does not exist.
- No duplicate record is created unless explicitly intended.
- The UI does not show a successful upload when the backend fails.
- Retrying does not create unintended additional side effects.
This is the difference between “checking whether upload works” and “checking how upload can fail.”
From AI-Generated Code to Adversarial Testing
AI helps developers write code faster, but the speed of code generation does not guarantee product coverage. When most code is generated to solve a specific goal, QA needs to identify what was not described within that goal.
Key testing approaches include:
Repetition
- Rapidly clicking multiple times.
- Repeatedly submitting a form.
- Retrying after a timeout.
- Refreshing and performing the action again.
Interruption
- Losing network connectivity.
- Closing the tab.
- Navigating to another page.
- Logging out while a request is in progress.
- Deleting a related entity while processing is still ongoing.
Concurrency
- Two tabs editing the same data.
- Two users updating the same record.
- Two delete requests running simultaneously.
- A newer request completing before an older request.
Stale State
- Submitting a form that was opened earlier.
- Using data after permission has been revoked.
- Acting on a record that was deleted elsewhere.
- Overwriting new data with stale data.
Cross-Feature Interaction
- Uploading a file and submitting the form.
- Deleting data while a background job is running.
- Changing permissions while another screen is still open.
- Deleting a file while the profile still references that file.
The New Role of QA
In the traditional development process, QA is often viewed as the layer that checks the system after developers complete a feature. In an AI-assisted development process, this approach needs to change.
AI can generate code, tests, documentation, and even suggest bug fixes. Therefore, QA needs to focus more heavily on tasks that require independent reasoning:
- Identify hidden assumptions in the implementation.
- Find states that are not described in the requirements.
- Test interactions between multiple actors and sessions.
- Detect bugs that only appear when requests run concurrently.
- Verify invariants and data integrity.
- Design attacks that are not covered by happy-path tests.
QA is no longer just the person asking:
“Does this feature work correctly?”
QA becomes the person asking:
“Where will this system fail, under what conditions, and what will the consequences be?”
Conclusion
AI is helping software get built faster. But code generated faster does not mean that the product is ready for every real-world situation.
Current surveys show that AI already contributes to a significant portion of committed code, while many developers still do not completely trust AI-generated output and need to verify it before putting it into production.
Most AI-generated code can complete the happy path or the goal described in the prompt. But real users do not only follow the happy path. They repeat actions, interrupt operations, act concurrently, use stale data, and combine multiple features in ways that do not always appear in the original requirements.
Therefore, QA in the age of AI needs to shift from simply confirming that the system works to actively finding ways to make it fail.
Do not just ask:
“Does the system work?”
Instead, ask:
“If I wanted to break this system without changing the source code, what would I try next?”
I am currently using a set of skills to support Adversarial Testing and Bug Hunting in a more systematic way. If this article receives enough attention, I will share the complete skill set that I am currently using.