A prompt cannot stop a model from inventing a number

The failure

On 1 August 2026 we found our own advertising generator producing proof-points nobody had measured: 42%, 63%, 40% and 34% — four different figures for the same claim — alongside a reference to an "EU financial client" that does not exist.

Four different numbers for one claim is a better illustration than one wrong number, because it shows what is actually happening. The figure was not mistaken. It was arbitrary.

The instruction not to invent figures was already in the prompt. It had been there from the first version, phrased several different ways over several revisions.

This is not a post about a model behaving badly. It is about a category error we made, and how we corrected it — because the same error sits in a large share of the LLM systems shipping right now, wherever the safeguard against a false claim is a sentence in a prompt.

Why the instruction cannot work

A language model does not have a register of things it knows to be true and things it is making up. Producing 42% and producing the are the same operation: the next token, conditioned on everything before it. When the surrounding context is an advertisement for a technical product, a percentage is an extremely likely continuation, because that is what such sentences contain in the text the model learned from.

"Do not invent statistics" competes with that pressure. It does not override it. Instruction-following is itself probabilistic — a strong prior, not a constraint — and it is weakest exactly where the sentence structure most strongly wants a number.

So the instruction fails in the case it exists for, and it fails silently. There is no error, no low-confidence flag, no refusal. The output is a well-formed advertisement, and the only signal that something is wrong is a human happening to recognise the figure as fictional.

The general rule underneath: a prompt is a request, not a control. Anything that must not happen has to be prevented by something that is not the model.

What replaced it

Three parts, and the interesting thing about them is that they make opposite trade-offs.

1. A record of what may be said

The generator does not write from general knowledge about the company. It writes from a structured record of permitted claims, and a claim that is not in that record is not available to the generator at all. Not weaker — absent.

Entries take two forms, deliberately. Some are plain lines somebody typed and personally vouches for. Others carry structure: the claim, who asserts it, and where it is written down. Both answer the same question — may this be said — and the structured form additionally answers on whose authority.

The most useful thing in that record today is what is missing from it. The list of permitted metrics is empty, and so is the list of nameable customers. Not "sparse": empty. Which means the honest current state of the system is that it may make no quantitative claim at all, and may not refer to a customer, because there is no measurement and no customer to refer to.

Follow that through and it says something sharper than "the gate is strict". With the permitted set empty, every figure fails, because there is nothing for it to be matched against. The check is not tuned to be severe today; it is as severe as it can be, by construction — and it will loosen exactly as fast as somebody measures something and writes it down.

That record earned its keep in an unexpected way. One entry carried, for months, an evidence line reading "no separate repository in the workspace; no deployment, no client." It was accurate. It was also unread — the entry sat there while our own website described the same thing as ready to deploy, until an unrelated check finally compared the two. The information was never missing. The reader was.

Hence a second rule: an evidence note only works if something mechanical reads it. A field only a human consults is documentation, and documentation is where wrong claims go to survive.

2. A deterministic check, and it runs before the image exists

Generated copy passes a check with no model in it: a regular expression and a substring comparison against the permitted claims. It looks for three shapes, each one seen live:

• a number or percentage (the 42/63/40/34 case);
• a quantity in words — "thousands of…", "hundreds of…";
• an implied customer — "our clients", when the nameable-customers list is empty.

Anything it catches must appear verbatim in a permitted claim, or the creative is discarded.

Two things about this are worth stating precisely, because both are easy to overclaim.

It runs before the image is generated. Not after, and this is the difference between rejection and non-existence: a creative carrying an unverified number is never built. No image is paid for, no asset is stored, nothing enters a review queue. The cheapest gate is the one that fires before the expensive step.

It is narrow, and the narrowness is the honest part. It reads the headline and the body text, and it looks for those three shapes. An invented claim without a number passes it. "Certified under ISO 27001", "built on the same stack the banks use" — the check does not see them. Those are held by the prompt and by the person who approves publication, which is to say: by exactly the mechanism this post argues is not a control.

That is not a hole we are hiding. It is a scope we chose, and naming it is the difference between a gate and a claim about a gate.

3. The image is a claim too — and this check is a model

A generated advertisement is not only text. The picture carries claims, and a generated picture carries artefacts no text check will ever see.

Here the trade-off inverts. Over claims, a model must not be used: asking a second LLM "does this look accurate?" reproduces the original problem one layer up, with the added disadvantage of sounding authoritative. Over pixels there is no deterministic alternative — an image has no list of permitted claims to be compared against — so a vision model does the judging.

What makes that acceptable is not trust. It is a bounded mandate: the vision check can only reject. It can never push anything into publication; that still requires a named human decision. And when it cannot answer, nothing ships.

Here is its verdict on a creative it rejected, translated verbatim from our internal record:

"Next to the inserted logo in the lower right corner there is a visible isolated letter fragment 'L' that belongs neither to the logo nor to the headline text — it looks like an AI artefact (a truncated or random text remnant)."

"Several empty orange and grey 'badge' shapes on the server boards have no legible content; they appear to be decorative AI artefacts without function."

The same report recorded what was sound, and it is only fair to quote both halves:

"The headline text 'AI Incident Triage, Zero Cloud Exposure' is legible and correctly spelled, the colours mostly match the palette (primary, accent, tech_dark), there are no prohibited claims."

That creative was rejected, not repaired. The copy was clean and the claims had resolved — which we can assert as a property of the ordering rather than an impression, since a creative that has an image at all has already passed the deterministic check. The image had a letter fragment floating beside the logo and empty badges on a rendered server rack: the visual equivalent of a confident sentence about nothing.

The day the checker hallucinated

A post arguing that models invent things owes you the case where our own checker did it.

On a correctly rendered Serbian image, the vision gate returned a failure with the justification "'ć' instead of 'ć'" — the same character on both sides of the sentence. At that resolution the model cannot reliably tell ć from č from ę, so it invented a difference and then wrote a rationale for it.

The cost was not abstract. Every failure triggers a regeneration — roughly five cents an image, up to the retry limit — after which the creative is marked failed. A language with diacritics would therefore have burned double the budget and produced nothing. And the conclusion sitting there waiting to be drawn was that the image generator cannot handle diacritics, when the thing that could not handle them was the checker. A wrong verdict does not only cost the retry; it hands you a plausible explanation pointing at the wrong component.

The fix was not a better prompt for the checker either. It was a structural separation: the instructions now distinguish a verdict from an observation. A letter that is visibly broken, doubled, or has become a different word is a failure. "I cannot make out which diacritic that is" is the model's limitation, not a defect in the image — it gets written into the notes and leaves the verdict untouched.

Which is the same lesson one level down: you do not fix a model's overconfidence by asking it to be less confident. You change what it is allowed to decide.

What it costs

Creatives die, and the fastest ones die before they are made. That is the intended behaviour, and it makes the system slower and less prolific than the version with a prompt instruction. Permanently, not as a tuning problem.

Fail-closed, but recoverable. When the vision check cannot run, the creative does not ship — but it is parked rather than discarded, and can be judged later against the stored image. A terminal failure and an unavailable checker are both "nothing publishes"; only one of them throws away an image you already paid for.

You need the record first. The gate is only as good as what it resolves against. Building that record is unglamorous, and keeping it read is a separate problem from keeping it correct.

It does not make the copy good. Everything here is about claims, not quality. A dull, accurate advertisement passes every check described. This is a floor, not a standard.

And the honest headline number: the claim check has never fired in production. Zero rejections on record. That is not evidence that it works — a gate that has never said no is not yet proven live. The most likely reading is the one that follows from the empty metrics list: with nothing quantitative permitted, the generator has not attempted a number since the record existed. Two other checks have fired — a novelty check twice, and the vision gate on the creative quoted above.

What generalises

• The dangerous output is the plausible one. A model that fails visibly is a solved problem; a model that fabricates a well-formatted figure is not.
• Move the control out of the prompt. If a rule must hold, something outside the model has to enforce it — and enforcement means the output does not ship, not that it is labelled.
• Put the cheap check before the expensive step. Rejecting after generation costs what generation cost.
• Where you must use a model to check a model, bound its mandate. Ours may only reject, may never publish, and its uncertainty is recorded as an observation rather than a verdict.
• Fail closed, but keep the work. Unavailable is not the same as failed; treat both as "does not publish" and only one as "throw away".
• State where the gate stops. A check described as complete when it covers three shapes is a claim of the kind it exists to prevent.
• Make something mechanical read your own records. A correct note nobody reads has the same effect as a wrong one.

The four percentages from 1 August were caught by a person. Everything described here exists so the next one is caught by a check — and so that when the check itself is the thing that is wrong, that is visible too.

The system described here is marketar, our own marketing automation, running on our own campaigns. This is engineering guidance drawn from that work; where a claim of yours carries regulatory weight, the obligations are yours and your counsel's to apply.