I planted twelve bugs in a game and my AI tester found four
Over the last year I have spent my spare time making small games, and for the last six months I have made them with AI coding agents. The agents write code quickly, they are just as quick to tell me it works, and by November I could no longer tell which of their green checkmarks to believe.
So I spent a week building checks around them, and then went back and tested the checks themselves.
The title comes from that second part. I planted twelve bugs in a build (a playable copy of a game) and gave it to my QA agent, the one that tests games for bugs. It found four. I fixed its instructions, moved the one bug that had cut its first run short and ran it again, and it found four again.
Two other things turned up the same week. My "blind" playtester, which is supposed to know nothing about the game it plays, was being started inside a project whose instruction file named the games. And in one session I launched twenty to thirty research agents to fill a catalogue of game cards, one note per game. Their web search tool stopped working partway through and they went on writing cards all the same. As the source for what they wrote, some of them put "knowledge of the genre".
I call the result GameStudio. It runs on Claude Code with Claude Haiku 5.5, Sonnet 5.5 and Opus 5.5, plus one more model, Fable 5.2, for a single task on a game's economy. This article is about that system, so the games show up only as things I tested it on. The numbers come from my repository and from two logs it keeps, a lessons file and the ticket journals, both described below.
What is in the repository
A vault of 1,361 Markdown notes in Obsidian holds what I have collected about the market from public sources, filed as games, mechanics, genres, sources, decisions and lessons. Next to it are about forty scripts that fetch data and run checks. Almost none of them call a language model. The exceptions are wrappers for image, 3D and sound generation, local Whisper transcription, and one script that starts a scheduled agent.

The third part is the agents themselves: fifteen roles, where a role is a job description for one kind of agent, and sixteen skills, which are reusable instructions for one kind of task. All of them are plain Markdown files that Claude Code loads.
Only a few of the fifteen run at once. There is one lead session, which is the main agent that hands out the work, then up to three coding workers and one reviewer, and read-only work goes out to batches of up to six agents. Even three coders turned out to be too many for one shared test file, which all of them edited and which conflicted on every merge, so now only the lead touches it.
State is kept in files, because a worker never sees the lead's conversation and gets only a ticket. A coding worker answers in a fixed format of at most 300 words: a status of done, partial or BLOCKED, the files it touched, and the exit code of each acceptance command. Each attempt goes into the game's ticket journal as a row, and anything that goes wrong gets a dated entry in the lessons file. A typical entry, translated: "Session launched about 16 heavy checks in parallel, the Mac froze (load 160)." The rule that came out of that one is a cap of four to six heavy processes per session.
The QA agent
The QA agent plays a build in a headless browser. It knows the specification. It has no shell, so it cannot run commands. It does have the Write tool, because the report has to go somewhere, which means only its instructions stop it from overwriting a file.
It kept coming back with no bugs, which would have been good news if I had any idea whether it could find one. So I took a copy of a game and wrote twelve bugs into it behind a test flag: a Continue button in the pause menu that did nothing, a counter with the wrong maximum, a pink strip across the screen, and nine more. I ran the agent against that. After the first run I changed its instructions (continue after a crash, check two window sizes, go through a checklist of bug classes on every screen), moved the planted crash that had stopped it to a later point in the game, and ran it a second time.
| Bug class | Planted | Found in run 1 | Found in run 2 |
|---|---|---|---|
| Crash or exception | 2 | 2 | 1 |
| Wrong number on screen | 2 | 1 | 1 |
| Visual artefact | 1 | 1 | 0 |
| Dead end | 1 | 0 | 1 |
| Control that does nothing | 1 | 0 | 1 |
| Text overflow | 2 | 0 | 0 |
| Gap in the tutorial | 1 | 0 | 0 |
| Memory leak | 1 | 0 | 0 |
| Untranslated text key | 1 | 0 | 0 |
| Total | 12 | 4 | 4 |
It filed five reports in the first run and four in the second, and all but one of them matched a planted bug. The odd one out I never checked, so I have no false-alarm rate to give. The reports it files are mostly right, and there are few of them.
Part of the shortfall is my own test design. In the first run the planted crash ended the session early with six of the twelve bugs still behind it. In the second a planted dead end hid two bugs and three others were on a screen the agent never reached. And all of this is two runs of one model on one game.
My target for recall, the share of real bugs the agent finds, was 50%, and I got 33% twice on a test that hid five or six of the twelve bugs from the agent each time. I would not read 33% as its recall. Of the bugs it could reach, it found four of six in the first run and four of seven in the second, and those are two small samples that I would not lean on either. The memory leak was in the count although the agent has no way to see one. All I showed is that I have no evidence its recall is high, and that one crash is enough to hide everything behind it.
The second four did not surprise me. I never believed an agent could be left to do the whole job, and this was the first time I had a number next to that belief, even a weak one. An agent reports "done" in the same confident tone whether the work is done or not. I use these tools every day and they still need checking. As the section on gates shows, I do not always do the checking myself.
I use the table for its breakdown by class. The agent caught crashes and exceptions, wrong numbers, a dead end and a control that does nothing, and it caught the visual artefact in one run out of two. It missed overflowing text both times, the first because it checked a single window size and the second because it ran out of actions before it reached the narrow window. A memory leak it cannot catch at all, having no tool that measures memory. That one needs a script and I have not written it.
After that I demoted QA to advisory. Its result is printed in the report of each gate (a checkpoint between development stages, described further down) and the verdict ignores it.
Before each real QA run a script now switches on three bugs from the game's list of planted bugs, and if the agent finds fewer than two, its report is voided. If the agent found each of the three independently at one in three, the only rate I have, the rule would void about three reports in four. In practice the outcome depends on which three are chosen, because four of the nine classes were never found in either run, and the rule repeats my own mistake whenever one of the three is a crash that hides the others. So a pass only rules out a lazy run. In its first real use the agent found none of the three and the report was thrown away; a fresh agent, given a hint about how to control the game, found two.
The bug list is stored in the game folder, which the QA agent can read, and all that stops it is a line in its instructions.
The blind playtest and its leaks
The blind playtest asks whether someone with no prior knowledge can understand the first screen. I treat it as a quick first check of onboarding, the way a game introduces itself to a new player, and nothing more, since I have never compared its verdicts with real players.
The blind tester's tool list has nine browser actions and that is all. This is the header of its role file, with the tool-server prefix removed from the names:
name: blind-tester
model: sonnet
tools: browser_take_screenshot, browser_mouse_click_xy, browser_mouse_move_xy,
browser_mouse_drag_xy, browser_mouse_down, browser_mouse_up,
browser_mouse_wheel, browser_press_key, browser_typeIt cannot read files, run code, open the browser console or go to another page. The lead opens the build and the tester gets the tab. A script strips that build first: source maps, source files, HTML comments and meta tags are removed, and the page title becomes "Game". The brief gives a persona (the kind of player to imitate), an episode number (an episode is one short play session of at most 15 actions) and the tester's own notes from earlier episodes. It does not give the game's name, genre or goal.
One persona is a seven-year-old who does not read, played by a model that reads every word in the screenshot. I have not found a way around that.
At the end of an episode the tester returns one JSON record. This is the template from the role file, shortened:
{
"episode": 1,
"persona": "seven-year-old who does not read",
"game_guess": "what I think this game is, in one phrase",
"goal_understood": true,
"goal_when": 3,
"first_success": "what worked first, and on which action",
"dead_ends": ["action 5: what I tried, what I expected, what happened"],
"would_quit": null,
"unclear": ["what I could not understand on screen"],
"hint_used": "none"
}Here goal_when is the number of the action on which the goal became clear, and would_quit is the action at which this player would give up, or null. The JSON is the tester's own account, and it is the part I trust least. The timings come from events that the game itself sends to the test harness: goal seen, first success, level complete. They are wall-clock seconds from page load and include the model's thinking time, so ten seconds for the model is not ten seconds for a child. Counting in actions would be more honest, and the record already has the action number.
On a toy game one episode of eight tool calls used 18,120 fresh Sonnet tokens (fresh meaning not read from cache). The first real run, three personas and eight episodes in total, used about 330,000, because real episodes are longer than the toy one.
The pass thresholds are a goal understood within 10 seconds, a first success within 60 seconds, and at most one hint in five minutes. The real run passed all of them, with both events coming about nine seconds after page load for every persona, which looked fine until I saw in the event log that the goal event had fired on the same move as the first success. Two of my thresholds were measuring one thing.
All three personas reported the same points of confusion. One of them, a piece landing off target, later turned out to be an artefact of how the tester drags the mouse. The game was fine.
The tester also had three ways to learn more than it should: my instruction files, the page itself, and progress saved by the previous persona. In Claude Code a subagent, meaning an agent started by another agent, is given the project's instruction file, and mine named the games. A critic agent, a role whose only job is to find faults, pointed this out while reviewing the setup, before I had any sign that a tester had used the information. I took the names out. While this article was being reviewed the other half of the same leak came up: a subagent also receives my personal instruction file, which still mentions the studio. That one is still open as I write.
The leak check had a hole of its own. It scanned files for the game's name, and although the strip script had set the tab title to "Game", the game set it again at runtime from the translation file, so the check never saw it, and the browser tool's replies can include the tab title. It missed the studio's server addresses in the build as well; the strip script now blanks them. And saved progress stayed in the browser profile, so one persona could inherit the progress of the one before.
Checks for the tab title and the saved progress are now steps in the test procedure, done by hand.
Gates, and a verdict called NOT ASSESSED
A gate is a checkpoint between two stages of a game. A script runs the gate's checks and returns one of four values, which from worst to best are FAIL, NOT ASSESSED, CONCERNS and PASS. The worst value among the checks becomes the verdict. A check whose report file is missing counts as NOT ASSESSED, and so does a report without a verdict line. Advisory checks are printed but left out of the verdict.
The same idea is written into eight of the fifteen role files: if you have no input, write NOT ASSESSED and list what is missing. Since I added it I have not seen "all good" in a report about something nobody had looked at.
There are eight gates between an idea and a release.
| Gate | What the script checks | Needs my decision |
|---|---|---|
| G0 idea | format of the concept note | yes |
| G1 specification | task-list check | yes |
| G2 skeleton | build succeeds, types, tests | no |
| G3 core | build, acceptance items for core gameplay | yes, I play for five minutes |
| G4 prototype | leak check of the blind-test build, blind test report, balance values within bounds | yes, go or no-go |
| G5 vertical slice | build, animation checks, a critic agent | no |
| G6 polish | build, all acceptance items, blind test report, balance values within bounds | yes |
| G7 release | build, all acceptance items, store checklist | yes |
In its first week one game went from idea to a public release, and only G1 and G2 got a PASS from the script. At G3 the script said FAIL with three acceptance items open, at G4 it said NOT ASSESSED because the balance check had no data, at G6 it said FAIL again, and all three times I overrode it with my own verdict. G0, G5 and G7 have no gate record at all.
The gates did what I built them for, and I went past them anyway. I was in the middle of a release. I knew the agents could not be left to finish on their own, and I let them carry on, half hoping the open items would sort themselves out. That is the habit I described in the QA section, and here it was mine. For now the table is a design I only partly follow.
At G0 the script only checks that the concept note is well formed. The idea itself is judged earlier by a "roast" agent, whose only job is to attack it, and then by me. G5 is the weakest gate, with no human sign-off and a judge that once scored the same change 7 out of 10 and then 5. Either I add my own sign-off or the critic becomes advisory, like QA.
Tickets
A game's specification lists its tasks in YAML, and each task is a ticket. This is one for a coding agent, shortened:
- id: T7
title: Hit and pickup effect sprites
lane: gfx
agent: game-coder
model: sonnet
paths: ["assets/fx/**", "src/fx/**"]
isolation: worktree
depends_on: [T5]
reviewer: art-director
max_iter: 2
budget_tokens: 1000000
acceptance:
- id: A1
what: "Effect sheets pass the animation gate"
oracle: command
run: "anim_gate.py <registry> <game>"
expect: "exit code 0"lane is the kind of work, here graphics. paths are the only files the ticket may touch. isolation: worktree asks for a separate working copy of the repository, max_iter is the number of attempts allowed, and oracle says how the result is judged, here by a command's exit code.
A script reads the task list before anything runs. It fails if two tasks in the same batch claim overlapping file paths, if a task is its own reviewer, if an acceptance item's expectation contains no number, or if a core gameplay task is assigned to a model below Opus. It also splits the tasks into batches of at most six, with at most three coders in a batch.
The budget is not enforced while the agent runs, because an agent has no reliable count of what it has spent, and the iteration cap is only a number the lead is told to respect. The lead reads the token usage after the attempt and writes it into the journal.
The thing that irritated me most all week was an agent telling me it had done and tested something when it had not. The research agents did exactly that when their search tool died. The two cases that became rules for coders are less clear-cut, because in both of them the hole was mine.
A coding agent once reported that all tests passed while its one failing test was being excluded by a skip flag inside the project's test command. I cannot blame the agent, which ran the command as given, and the flag was there for a reason; the hole was in my harness. Coders now run their own acceptance command directly, and the full suite without the flag, before they report.
A second loophole was found by a critic agent before any agent exploited it. Each ticket's acceptance items are hashed when I approve the specification, so that nobody can lower the bar later. The hash was being compared with a value stored in the same file as the items. An agent that edited an expectation and recomputed the hash would have passed. The comparison now uses the approved copy of the specification, which workers are still only told not to edit.
I also asked for worktree isolation and got it for the wrong repository. The games are in their own repositories, and the built-in isolation makes a worktree of the repository the session runs in, which is GameStudio. Three coders would have shared one working copy of the game, which is what I had asked for isolation to prevent. A small script now creates a branch and a worktree per ticket, the lead merges them one at a time, and the isolation field in the ticket is read by that script.
What Claude Code enforces
Claude Code enforces tool lists, so a role with no shell tool cannot run a command. Everything else is an instruction, checked afterwards or not at all: which files a ticket may touch, budgets, iteration caps, the ban on reading the bug list. The project's settings file has no hooks (scripts that Claude Code runs before or after a tool call) and no permission rules, and nothing in my user settings enforces a ticket's file scope.
That rule should be a pre-tool hook that refuses a write outside the ticket's paths before it happens, and I have not got to it. A hook on the edit tools will not see a write made through the shell, so coders with a shell need a second check. The reviewer and the author also come from the same model family, with different prompts, so I assume they share mistakes.
The knowledge base
The catalogue those research agents were filling is a set of game cards in the vault.
After that session I drew 15 cards at random with a fixed seed and checked about 45 facts in them by hand. Four facts were clearly wrong, in three cards. One card gave a game's all-time record from 2021 as its peak for the previous month. Another had audience: "yes", because a sentence from the body text had spilled into the field. Seven of the 15 cards had at least one problem of some kind, and across the vault twelve cards had the same field repeated three times.
Some of that is models inventing things, and some of it is that a tool stopped working and nothing told the agents to stop with it. The research role files now forbid "knowledge of the genre" as a source and require anything without a source to be marked UNVERIFIED. Research agents run in batches of six, with at most three or four searching at once. And I stopped using models to collect facts that a store or a feed already publishes. Scripts do it now:
| Script | What it reads |
|---|---|
steamdata.py | public Steam store data: price, date, developers, review summary, top reviews |
mobiledata.py | App Store and Google Play listings: rating, installs, lowest-rated reviews |
robloxdata.py | Roblox public game data: current players, visits, vote ratio, game passes |
steamnews.py | a game's news feed: how often the developer posted before and after release |
ingest.py | a video's subtitles, or a local Whisper transcript when there are none |
A separate script writes the results into cards, matching a card by normalised name or by a store link already in it and writing nothing if neither matches. It fills only empty fields and puts the source and date beside each value, which also means that whatever the agents invented and my audit missed is still in those cards.
Genre and style are links to notes that must exist. A linter enforces that for those two fields, along with a fixed list of audiences. Platform values are not checked, and 13 off-list values have crept into 14 cards.
Mechanic notes also carry appeal scores from 1 to 5 and a retention field. Those are a model's opinions written into structured fields, and I use them only to pick which notes to read.
I am happy with Obsidian, mostly for a reason that has little to do with agents: I can see the data. Game cards open as a gallery of covers, mechanics as a table I can filter, and the links between notes as a graph. I doubt it saves the agents a single token, since to them the vault is just a folder of Markdown files that they search as text. The pictures are for me.
To make lookup cheap, game, genre, mechanic and practice notes should each have a one-line summary of at most 160 characters. So far 727 notes have one. For 703 of them a script simply copied the opening of the note's overview section, and Haiku rewrote the 40 worst of those. An agent searches for a term as plain text, reads the summary line of each hit, and opens a note only after that. Whether this finds the right note, and what it saves in tokens, I cannot tell you.
Like a compiler, the linter exits with an error code when it finds errors. I run it by hand and it runs at gate G0, and nothing else forces it to run. Its two results today, translated and condensed:
games and mechanics: 657 files, 0 errors, 21 warnings GATE: CONCERNS
all linted notes: 1274 files, 10 errors, 136 warnings GATE: FAILOf the 136 warnings, 105 are links to notes that do not exist.
One count I watch is the number of UNVERIFIED and stub marks in the game cards. Six of the fifteen roles are told to mark any claim without a source, and the agent applies the mark to its own text, so the count tells me how well agents follow that rule at least as much as how much is unverified. In one day it rose from 1,668 to 1,857 as new cards arrived. Today it is 1,875.
What the week cost
My ticket journal for the week adds up to somewhere between $60 and $100 at API list prices, depending on how I sum its rows. Then I went through the session logs that Claude Code keeps on disk: three working lead sessions, four that lasted under a minute, and 650 subagent runs. They hold about 99 million fresh tokens (new input and output) and 3.88 billion tokens read from cache. At list prices, with cache writes and cache reads at their published rates, that is about $1,460 for the week, so the journal had caught about 5% of it.
None of this is a bill. I pay a flat subscription, and $1,460 is what the same week would cost if I paid per token, vault and research work included, since I could not separate those from the games in the logs.
| Role and model | Share of the $1,460 |
|---|---|
| Lead sessions, Opus | 39% |
| General-purpose helpers started by the lead, Opus | 28% |
| Coding agents, Opus | 15% |
| Specification writer, Opus | 5% |
| Coding agents, Sonnet | 2% |
| Research, art and general helpers, Sonnet | 7% |
| Blind tester, QA agent, librarian, video analyst | 1.5% |
| Everything else: Fable, critics, small roles | 4% |
Each share is rounded on its own, so the column adds up to 101.5%.
Every ticket had a budget of a million tokens and the largest used 445,000, so no ticket came near its budget, and the budget was pointed at the wrong thing anyway. The journal counts fresh tokens on ticketed runs. About half of the cost is tokens read back from cache, and two thirds of the total is in the lead sessions and in helpers that the lead started outside the ticket journal. On every model call the lead sends its whole conversation again. The cached part is billed at a tenth of the input price, but the two longest of my lead sessions averaged about 450,000 tokens of context per call, and that adds up. Shorter lead sessions, restarted at each stage and fed from files, should cut more than any hardware would.
The subscription quota stopped me before the cost did. Partway through a release the weekly Sonnet allowance ran out and the work moved to Opus, exactly backwards from my routing rule, which is a script or Haiku for filing, Sonnet for reading and non-core code, Opus for specifications, architecture and core gameplay code.
A local model comes next
I want part of the routine work to run on a model at home, and I did the sums before buying anything. The roles that could move, the ones that only read, file, summarise or test, plus research, were about 8% of the week's cost. I pay a flat $200 a month, so in cash a machine saves nothing.
The reason to buy one is quota, since those are Sonnet roles and Sonnet is the allowance that ran out, and repetition. My QA calibration is two runs and my blind test is eight episodes, because every run costs quota. A local model could replay the same episode fifty times overnight for the price of electricity.
The hardware, the model and the payback arithmetic are in a separate note, Doing the sums before buying a machine for a local model.
Smaller incidents
- I piped the worktree script into
head -1, the pipe closed, the script died with it, and every agent started without a worktree and came back BLOCKED. - All tests were green, yet one game's hub level could not be played, because the tests checked only numbers. A run in the real engine after every batch is now mandatory.
- In a separate case, after three rounds of fixes a critic's average score was still 7.1 out of 10. A new critic on a stronger model, asked how close the result was to the concept art, gave 4 to 5. After two or three rounds a person now decides, and the critic's list goes to the backlog.
If you are building something like this
All of this is one week old. It has carried one game to release with six of its eight gates skipped or overridden, and I did not compare it with working without it under controlled conditions, so I am not claiming it is faster or cheaper.
If I were starting again I would do the bug planting first. Before trusting "no bugs" from an AI tester, plant known bugs and count what comes back, read the result by bug class, and make sure one planted bug cannot hide another, which is the part I got wrong in both runs.
Give missing evidence a verdict of its own and rank it worse than a warning, so that a check that did not run can never read as a pass. Hash acceptance criteria when you approve them and compare against the approved copy; a hash stored beside the criteria protects nothing. Where a fact is available from an API, collect it with a script, and when a tool fails the task should stop and say so instead of carrying on from memory.
Count cost from the session logs, cache reads included. My ticket journal saw about 5% of it.
Two things are specific to Claude Code. A subagent receives your instruction files, the project one and the personal one, so if the subagent is an evaluator, read those files the way it would. And in my setup the tool lists were the only boundary that was enforced. Hooks and permission rules come next, and until they exist everything else in my role files is a request.