[Guides](https://featkpr.com/guides) / checked on 28 Sep 2026

# How to review AI-written tests: does each test check anything?

Reviewing an AI-written test means asking one question of it: would it fail if the behaviour it names were broken? A coding agent can write tests faster than anyone can read them, and make every one of them pass. Green says the code ran without complaint. It does not say that anything was checked.

Tim Derzhavets · published 28 Sep 2026 · checked on 28 Sep 2026

## Why a passing test proves less than it looks

A test has two halves: it drives the code, and it decides whether the answer was right. The second half is the hard one. Researchers call it the oracle problem ([Barr et al., IEEE TSE 2015](https://doi.org/10.1109/TSE.2014.2372785) · peer-reviewed ) : generating inputs is cheap, and knowing what the correct output should be is not. An agent asked to “add tests” is rewarded when the suite goes green, and the cheapest way to green is a weak oracle: an assertion on the status code, on “no exception was raised”, or on a value the test itself just wrote.

Meta measured what happens when a model writes tests without a gate. Its TestGen-LLM tool keeps a generated test only after it passes a set of filters. On Instagram’s Reels and Stories, 75% of the generated test cases built, 57% passed reliably and 25% increased coverage ([Alshahwan et al., Meta, FSE 2024](https://arxiv.org/abs/2402.09171) · peer-reviewed ) . The filters are the point: most of what a model writes has to be thrown away or fixed before it counts. You are the filter when nobody built one.

## Six checks for every AI-written test

Run them in order. The first matters most; the others catch what it can miss.

| Check | How to run it | What it catches |

|---|---|---|

| 1. It fails when the behaviour breaks | Break the code on purpose and run the test | Tests that pass whatever the code does |

| 2. It has a negative control | Send the same case where it must not succeed | Checks that were removed and nobody noticed |

| 3. The assertion checks the outcome | Read the assert lines before anything else | Status-only and “no error” tests |

| 4. One test, one claim | Read the name, then count the claims | Tests whose failure tells you nothing |

| 5. It gives the same answer every time | Run it ten times, and in a different order | Flaky tests that teach people to ignore red |

| 6. It covers a feature, not lines | Match it to a feature of the app | Coverage that ran code without checking it |

### 1. It fails when the behaviour is broken

Open the code the test claims to check and break it: delete the line that saves, flip the condition, return early. Run the test. It must go red. If it stays green, it checks nothing about that line, whatever its name says.

This is mutation testing done by hand. The idea is old ([DeMillo, Lipton and Sayward, 1978](https://doi.org/10.1109/C-M.1978.218136) · peer-reviewed ) and well studied ([Jia and Harman, IEEE TSE 2011](https://doi.org/10.1109/TSE.2010.62) · peer-reviewed ) : plant a small fault, a mutant, and see whether the suite notices. Catching mutants is a fair stand-in for catching real bugs: across 357 real faults in five open-source applications, a suite’s mutant score tracked its ability to find the real faults, independent of code coverage ([Just et al., FSE 2014](https://homes.cs.washington.edu/~rjust/publ/mutants_real_faults_fse_2014.pdf) · peer-reviewed ) . Google shows mutants to authors and reviewers inside code review, only on the lines a change touches, so the question is asked while the test is still fresh ([Petrović and Ivanković, Google, 2018](https://research.google/pubs/state-of-mutation-testing-at-google/) · peer-reviewed ) . Meta now points a model at the same idea: its ACH system writes the faults a suite misses first, and then the tests that catch them ([Foster et al., Meta, FSE 2025](https://arxiv.org/abs/2501.12862) · peer-reviewed ) .

You don’t need a tool to start. For each new test, make one mutation that the test’s name says it guards against.

### 2. It has a negative control

A test that only walks the happy path cannot tell a working check from a missing one. For every “this is allowed” test, write or ask for the case where it must not work: the user without the right, the malformed input, the record that doesn’t exist. If the happy-path test passes and the negative one passes too, the check between them is not there.

### 3. The assertion checks the outcome, not the status

Read the assert lines before the setup. `status == 200` and “no exception” describe the transport, not the result. The outcome is what the person wanted: the book exists with that name, the email left, the row is gone, the page now shows the new title. An agent reaches for the status because it is always there. Ask for the outcome instead, read back from the app, not from the test’s own variables.

### 4. One test, one claim

The name should state one claim, and the body should check that claim. A test that asserts ten things fails for ten reasons, so its failure tells you little, and one weak assertion hides among nine strong ones. Split it, or drop the assertions that belong to other tests.

### 5. It gives the same answer every time

Run the new test ten times, and once in a different order. A test that sometimes fails costs more than it saves: at Google about 16% of tests showed some flakiness, and 84% of the transitions from pass to fail they observed involved a flaky test ([Micco, Google, 2016](https://testing.googleblog.com/2016/05/flaky-tests-at-google-and-how-we.html) · engineering blog ) . In a survey of 335 developers and testers, 51% met flaky tests at least weekly, and losing trust in the suite was the effect they rated worst ([Gruber and Fraser, ICST 2022](https://arxiv.org/abs/2203.00483) · peer-reviewed ) . The largest single cause in fixes studied across 51 projects was waiting on asynchronous work ([Luo et al., FSE 2014](https://mir.cs.illinois.edu/lamyaa/publications/fse14.pdf) · peer-reviewed ) , so a fixed `sleep` in an agent’s test is the first line to question.

Brittleness is the slower version of the same problem. Tests tied to a screen’s structure break when the screen changes: across six open-source web apps moved to a newer release, 143 of 196 WebDriver tests needed repair ([Leotta et al., WCRE 2013](https://sepl.dibris.unige.it/publications/2013-leotta-WCRE.pdf) · peer-reviewed ) .

### 6. It covers a feature, not lines

Line coverage says code ran, not that anything was checked. In the largest study of the question, 31,000 test suites for five Java systems, coverage had a low to moderate correlation with how many faults a suite found once its size was controlled, and the authors advise against using it as a target ([Inozemtseva and Holmes, ICSE 2014](https://www.cs.ubc.ca/~rtholmes/papers/icse_2014_inozemtseva.pdf) · peer-reviewed ) . An agent can raise coverage without raising a single check.

Ask a different question: which features of the app have a test that would fail if the feature broke? That needs a list of the features first. The [feature inventory guide](https://featkpr.com/guides/feature-inventory) shows what to put on it.

## What to do with a test that fails a check

Fix the oracle when the test is aimed at the right behaviour: replace the status check with the outcome, add the negative control. Delete it when it duplicates another test. When you keep a test you could not check, mark it unchecked and keep it out of any count that claims coverage. An unchecked test still catches a crash; it just must not be counted as a check.

## One method: send each test again as the wrong user

Checks 1 and 2 can be made mechanical for one large class of tests: the ones about who may do what. In an app with accounts, most routes have someone who may use them and someone who may not. So each test is sent twice. Once as its own user, where it must pass. Then as the wrong user: someone who should get the opposite answer. A test that expects to reach a page is sent as a user without the right, and the app must refuse. A test that expects a refusal is sent as the admin, who must get through. Either way the test must now fail. A test that still passes is not checking the rule it names.

one test

Create a book on a shelf

`POST /shelves/{shelfSlug}/create-book`

- sign in
- open the shelf
- send the new book

**The test's own user: an Editor** holds book-create

BookStack answers **302 → /books/{new book}**

passed

**The wrong user: a reader** can read, has no book-create

BookStack **refused: sent back to the home page**

the test fails, as it must: the check exists

One BookStack test, drawn from featkpr's sample data. The counts under it are from the run of 20 Oct 2026, 03:18 UTC, not from this drawing. The same request leaves once and is sent as two users.

featkpr does this for every test it writes. On BookStack, the open-source wiki, the run of 20 Oct 2026, 03:18 UTC at commit 045c6a98 sent 311 tests, and all passed as their own user. Sent again as the wrong user, 276 of them failed, as they must. The other 35 stay unchecked: they all expect access, and the run had no user who lacked the right, so there was nobody to refuse.

BookStack · run of 20 Oct 2026, 03:18 UTC · commit 045c6a98 · featkpr's records, 20 Oct 2026

Sent as its own user must pass 311 of 311 passed Sent as the wrong user must fail 276 of 311 35 unchecked

held: the test did what it must unchecked: no wrong user to send it as

A test counts as checked only when both rows hold for it. The unchecked ones all expect access; the run had no user without the right to refuse.

This is one method, not the whole review. It proves that a permission check exists. It says nothing about whether the page shows the right content, which is still checks 3 and 6. The full method, with the code of one test, is in [the post on sending each test as the wrong user](https://featkpr.com/blog/2026-09-27-every-test-as-the-wrong-user) , and the run step by step is in [the BookStack report](https://featkpr.com/library/bookstack/reports/2026-09-26-bookstack-run) . On the wider list of web risks, broken access control is first in OWASP’s 2025 list ([OWASP, 2025](https://top10.owasp.org/2025/A01_2025-Broken_Access_Control) · community project ) .

## Where featkpr fits

featkpr starts before the tests. It [maps an app’s features](https://featkpr.com/services/feature-map) , the [flows](https://featkpr.com/library/flows) people take and the goals they reach, and a person approves or denies each piece. Tests sit on top of that map: each one is written for a case on it and counted against it, so check 6 has an answer before the first test runs. The [tests area of the product tour](https://featkpr.com/product/tests) shows the screens.

## Sources

each read on 28 Sep 2026; the date is when it was published, the mark says what kind of source it is

- E. Barr, M. Harman, P. McMinn, M. Shahbaz, S. Yoo, The Oracle Problem in Software Testing: A Survey, IEEE Transactions on Software Engineering 41(5), 2015. Cited here for the name of the problem: deciding whether an output is right. May 2015 · peer-reviewed, title and venue checked on Crossref [doi.org/10.1109/TSE.2014.2372785](https://doi.org/10.1109/TSE.2014.2372785)
- N. Alshahwan, J. Chheda, A. Finegenova et al. (Meta), Automated Unit Test Improvement using Large Language Models at Meta, arXiv 2402.09171, FSE 2024. On Instagram's Reels and Stories, 75% of TestGen-LLM's test cases built correctly, 57% passed reliably and 25% increased coverage. Written by the tool's makers. 14 Feb 2024 · peer-reviewed, industry paper on the authors' own tool [arxiv.org/abs/2402.09171](https://arxiv.org/abs/2402.09171)
- R. DeMillo, R. Lipton, F. Sayward, Hints on Test Data Selection: Help for the Practicing Programmer, IEEE Computer 11(4), 1978. The paper usually named as the start of mutation testing. Apr 1978 · peer-reviewed, title and venue checked on Crossref [doi.org/10.1109/C-M.1978.218136](https://doi.org/10.1109/C-M.1978.218136)
- Y. Jia, M. Harman, An Analysis and Survey of the Development of Mutation Testing, IEEE Transactions on Software Engineering 37(5), 2011. The field's survey. Sep 2011 · peer-reviewed, title and venue checked on Crossref [doi.org/10.1109/TSE.2010.62](https://doi.org/10.1109/TSE.2010.62)
- R. Just, D. Jalali, L. Inozemtseva, M. Ernst, R. Holmes, G. Fraser, Are Mutants a Valid Substitute for Real Faults in Software Testing?, FSE 2014. 357 real faults in 5 open-source applications: a statistically significant correlation between catching mutants and catching real faults, independent of code coverage. Nov 2014 · peer-reviewed [homes.cs.washington.edu/~rjust/publ/mutants_real_faults_fse_2014.pdf](https://homes.cs.washington.edu/~rjust/publ/mutants_real_faults_fse_2014.pdf) · [doi.org/10.1145/2635868.2635929](https://doi.org/10.1145/2635868.2635929)
- G. Petrović, M. Ivanković, State of Mutation Testing at Google, ICSE-SEIP 2018. Mutants shown to authors and reviewers inside Google's code review, only on the lines a change touched. May 2018 · peer-reviewed, Google's own system [research.google/pubs/state-of-mutation-testing-at-google/](https://research.google/pubs/state-of-mutation-testing-at-google/) · [doi.org/10.1145/3183519.3183521](https://doi.org/10.1145/3183519.3183521)
- C. Foster, A. Gulati, M. Harman et al. (Meta), Mutation-Guided LLM-based Test Generation at Meta, arXiv 2501.12862, FSE 2025 industry track. Meta's ACH writes faults a suite does not catch yet, then tests that catch them. Written by the tool's makers. 22 Jan 2025 · peer-reviewed, industry paper on the authors' own tool [arxiv.org/abs/2501.12862](https://arxiv.org/abs/2501.12862)
- J. Micco, Flaky Tests at Google and How We Mitigate Them, Google Testing Blog: "about 84% of the transitions we observe from pass to fail involve a flaky test"; "almost 16% of our tests have some level of flakiness". 27 May 2016 · engineering blog, Google's own data [testing.googleblog.com/2016/05/flaky-tests-at-google-and-how-we.html](https://testing.googleblog.com/2016/05/flaky-tests-at-google-and-how-we.html)
- M. Gruber, G. Fraser, A Survey on How Test Flakiness Affects Developers and What Support They Need To Address It, arXiv 2203.00483, ICST 2022. 335 developers and testers. Mar 2022 · peer-reviewed [arxiv.org/abs/2203.00483](https://arxiv.org/abs/2203.00483)
- Q. Luo, F. Hariri, L. Eloussi, D. Marinov, An Empirical Analysis of Flaky Tests, FSE 2014. Flaky-test fixes in 51 open-source projects: waiting on asynchronous work was the largest cause, 45% of the fixes classified. Nov 2014 · peer-reviewed [mir.cs.illinois.edu/lamyaa/publications/fse14.pdf](https://mir.cs.illinois.edu/lamyaa/publications/fse14.pdf) · [doi.org/10.1145/2635868.2635920](https://doi.org/10.1145/2635868.2635920)
- M. Leotta, D. Clerissi, F. Ricca, P. Tonella, Capture-Replay vs. Programmable Web Testing: An Empirical Assessment during Test Case Evolution, WCRE 2013. Six open-source web apps, each moved to a newer release 8 months to 4 years 8 months later; the counts per app from its Table III, summed by us. 2013 · peer-reviewed [sepl.dibris.unige.it/publications/2013-leotta-WCRE.pdf](https://sepl.dibris.unige.it/publications/2013-leotta-WCRE.pdf)
- L. Inozemtseva, R. Holmes, Coverage Is Not Strongly Correlated with Test Suite Effectiveness, ICSE 2014. 31,000 test suites for five Java systems: a low to moderate correlation between coverage and effectiveness once suite size is controlled; coverage "should not be used as a quality target". May 2014 · peer-reviewed [www.cs.ubc.ca/~rtholmes/papers/icse_2014_inozemtseva.pdf](https://www.cs.ubc.ca/~rtholmes/papers/icse_2014_inozemtseva.pdf) · [doi.org/10.1145/2568225.2568271](https://doi.org/10.1145/2568225.2568271)
- OWASP Top 10:2025, A01 Broken Access Control, first in the list. 2025 · community project [top10.owasp.org/2025/A01_2025-Broken_Access_Control](https://top10.owasp.org/2025/A01_2025-Broken_Access_Control)

How this guide was written and checked, and where its numbers come from: [how we write and check](https://featkpr.com/about/how-we-write) . Found an error? [Report a correction](https://featkpr.com/about/how-we-write#corrections) .

written by

[Tim Derzhavets](https://featkpr.com/about)

published 28 Sep 2026 · checked on 28 Sep 2026

this guide feeds

- [The tests on top of the map](https://featkpr.com/product/tests)
- [Service: tests on the map](https://featkpr.com/services/playwright-tests)
- [The post: each test as the wrong user](https://featkpr.com/blog/2026-09-27-every-test-as-the-wrong-user)
- [The BookStack report](https://featkpr.com/library/bookstack/reports/2026-09-26-bookstack-run)

## See featkpr map a real app

Half an hour on BookStack, your questions first. Your own app comes after.

[Ask for a private demo](https://featkpr.com/demo)

---

The page this twin stands for: https://featkpr.com/guides/review-ai-written-tests. Every page on this site has a `.md` twin, and answers `Accept: text/markdown`.