The free training label in your agent loop, and how much to trust it

If you are building an agent that takes actions, you probably already check whether each action worked. That check is the most underused asset in your system. It is a label, you are generating thousands of them, they cost you nothing extra, and almost nobody uses them for anything except deciding whether to retry.

This is about what happens when you do use them, what the label is actually worth once you measure it, and the specific ways an agent learning from its own output poisons itself. I have the numbers from my own system, and they are less flattering than the ones you usually see in posts like this.

The label hiding in your agent loop#

The loop in question is ordinary. A model looks at a screenshot and an instruction and returns a coordinate to click. The action executes. A second model looks at the before and after screenshots and answers one question: did the intended effect happen?

That second answer is the interesting one. Most systems use it to decide whether to retry, and then throw it away. But it is a per-action supervision signal on real traffic, produced for free as a side effect of doing the work. If you keep it, you can bank every action the verifier confirmed as a durable fact about how that specific app behaves, and hand those facts back to the planner and the grounder on the next run.

The effect is that first contact with an app is cold, and accuracy compounds from run two onward. That is the whole prize, and it is worth being precise about why it is achievable: not because the agent got smarter, but because you stopped discarding the only labelled data you had.

What the label is actually worth#

Here is the part usually left out. From my grounding baseline, re-measured against production on the morning this post went out: 109 grounding attempts across 44 executions, of which the verifier confirmed 97. A hit rate of 0.89.

An update, and the post earned it. I first wrote this against a baseline captured in June: 80 attempts from a single app, a 0.8625 hit rate, and a drag success rate of zero. Re-running it against production the same morning moved almost every figure, so the numbers here are the current ones and the June set stays in for contrast. That is inconvenient for a post arguing that a measurement has a half-life, and also the best evidence for it I could have asked for.

Read that carefully, because the obvious misreading is the one I nearly published. The 89% is the grounder being right. The verifier is the thing that caught the other twelve. So the label is not 89% reliable; the label is what tells you about the 11%. Those are different quantities and conflating them will make you trust the wrong component.

Broken down by action, the same 109 samples:

• type: 32 of 32. Typing into a located field is essentially solved.
• click: 47 of 57, so 0.825. This is the volume case and it sets the headline number.
• scroll: 14 of 15.
• drag: 4 of 5. In June this was 0 of 4.

The drag number is the one that moved, and it is worth being exact about what moved. Dragging fails because the drop target frequently does not exist until the gesture is already in progress, so there is nothing on screen to ground against at the moment you need to aim. The machinery around it in my codebase is elaborate, pressing, nudging to reveal slots, re-grounding the drop mid-gesture, and dispatching the native HTML5 sequence alongside the pointer one, precisely because the naive version scored zero. Elaborate code is usually a confession, and here it is also the fix: that is what took drag from never working to mostly working. Five attempts is nowhere near enough to call it solved, though, and I would not quote 80% at you as if it were stable.

One figure in that run matters more than the hit rate, and I nearly walked past it. Across all 109 attempts the grounder failed to locate a target exactly zero times. It always returned a coordinate. It has no way to say it cannot see something, which means roughly one attempt in nine is a confident answer pointing at the wrong place, and nothing in the output separates those from the other eight. That is the whole argument for a separate checker rather than a more careful actor. Relatedly, of the four times the agent escalated to a second look at a failed step, all four recovered.

And one caveat that matters more than any of the above, which has improved. The June figures came entirely from one domain, making them a measurement of one app rather than an accuracy claim. The current run spans ten, from a marketing site to a data dashboard to a preview environment, and the spread between them is wide: six sit at 100%, one sits at 0 of 2. Anyone quoting a number like this at you without saying how many apps it covers is telling you very little. The baseline file records its own timestamp and commit sha for the same reason. A number like this has a half-life, as the top of this section now admits.

The last thing worth knowing: my verifier is not clean either. I have a filed, still-open case where a confirmation was simply wrong, an accessibility check that scanned a client-rendered control before it finished rendering, called it unnamed, and had that finding confirmed downstream at critical severity. The label is good. It is not ground truth. Design for that, not for the version where it is.

Five rules for learning from a label you do not trust#

Everything below is a direct consequence of the label being imperfect. If the verifier were an oracle, none of this would be necessary, and the naive version would be fine.

• 1. Never learn from a failure. Only steps the verifier confirmed become facts. This sounds obvious and is easy to violate accidentally, because the tempting move is to learn from failures too, as negative examples. Do not: a failure tells you the attempt was wrong, not what the right answer was, so you end up banking a guess.
• 2. Make the verifier a different model family from the actor. This is the rule I would defend hardest. If the same family both acts and checks, its errors correlate, and a correlated verifier is not a verifier, it is an echo. Mine grounds with one model, plans with a second, and verifies with a third from a deliberately different family, so that a blind spot in the actor is not automatically a blind spot in the check.
• 3. Score confidence by source, and make repeat evidence the only way up. A single verified step seeds at 0.6. A step recovered by a replan seeds lower, at 0.55, because a recovery is weaker evidence than a clean pass. Human approval seeds at 0.9 and outranks both. Repeat sightings add 0.1, capped just under 1, and the bump is idempotent per run so one chatty run cannot promote its own guess. Nothing below a 0.4 floor is ever injected.
• 4. Merge fuzzily, because your own planner will rephrase. The same intention comes back as "navigate to dashboards tab" one run and "navigate to the dashboards tab" the next. Key on exact text and you accumulate near-duplicates until the injected context is mostly noise. I fold a candidate into an existing sibling when token overlap crosses 0.6 Jaccard.
• 5. Let a human veto, and make the veto permanent. Anything a human rejects is never resurrected, no matter how much later evidence accumulates for it. This is the only hard stop in the system and it is the one that lets you sleep.

There is a sixth that is more subtle. When a failed step is later recovered by a replanned retry, the steps the replan inserted in between are worth banking as a precondition of the retry, not discarded as noise. "Search for the field" before "drag the field" is the actual knowledge. Learn the atomic action alone and you will faithfully reproduce a step that cannot work yet.

The failure nobody predicts#

Everything above is about protecting the store from bad entries. The failure that actually bit me was the opposite: the store was fine, and injecting it broke the planner.

Feed a large block of learned app knowledge into a planning prompt and past some threshold the planner stops planning and returns an empty list. Not a worse plan. No plan. The knowledge was correct, the prompt was well formed, and the output was nothing.

Two things came out of that. The block is now kept deliberately tight: effects stripped, deduplicated by action and target, capped in length, and bare drags and scrolls excluded entirely because they only make sense with their precondition attached. And planning retries without any learned context when the first attempt comes back empty, which both dodges the failure and sidesteps a cached empty response, while emitting a visible event so the operator can see it happened.

The general lesson is not about prompt size. It is that a self-improving system has a new failure mode the non-learning version did not: the improvement path itself can break the thing it was improving, and it will do so silently, because an empty plan is not an error. Whatever you inject, keep the un-injected path alive as a fallback and instrument the switch.

What to build first#

If you want the compounding without the six months, the order matters, and it is roughly the reverse of what is fun to build:

• Persist the verifier verdict per action, with the instruction and the outcome. Just store it. You cannot measure anything until this exists, and it is an afternoon.
• Measure your hit rate by action type before you build any learning. Mine told me drag was zero, which changed what I built next, and it is not zero any more. An aggregate number would have hidden both the problem and the fix.
• Split actor and verifier across model families, even if it costs more. This is the cheapest correctness win available and it gets harder to retrofit the longer you wait.
• Only then bank confirmed actions as priors, behind a confidence floor, and make the prior a hint rather than an instruction. The grounder should still look. The prior only steers.
• Keep a human veto and a visible list of what the system believes. If you cannot show someone what your agent thinks it learned, you cannot debug it and neither can they.

The takeaway#

Self-improving agents get discussed as though the hard part is the learning algorithm. In my experience the learning is the easy half. The hard half is that your supervision signal is wrong some of the time, you cannot tell which times, and every mechanism worth building is a hedge against that rather than a use of the signal.

So measure the label before you trust it, split the actor from the checker so their mistakes do not agree, and keep a path that works when the learned context is empty. Do that and the free label is genuinely free. Skip it and you have built a system that gets more confident and no more correct, which is strictly worse than the one you started with.