The strongest model in a new benchmark of LLM ordering agents produced perfectly valid JSON on every attempt, and still got the order wrong about one time in five S¹. That 20-point gap between a clean schema check and a correct transaction is the finding that should make anyone building agent systems pause, because it sits in the exact layer most teams treat as solved.

My read: This is the first benchmark I've seen that cleanly separates the three failure modes most teams conflate: broken syntax, invalid schema, and wrong semantics. The 100%-vs-80% gap in the strongest model is the headline, but the double-digit unsafe acceptances in weaker models are the scarier signal. Those are orders that look perfectly valid to any downstream system and would execute without raising an alarm. I'm skeptical of generalising beyond restaurant ordering, and the paper doesn't name the four models, which makes independent reproduction hard. But the core warning, that structured output is plumbing not verification, is correct and overdue.

What "schema valid" actually means

The paper, posted to arXiv on 22 July by Yin Li at the University of Birmingham, introduces OrderBench, a deterministic benchmark for restaurant ordering agents S¹P². The word "deterministic" matters: every test case has a single correct answer, not a panel of judges scoring vibes.

OrderBench splits correctness into six layers S¹:

  • Syntactic validity: is the output well-formed JSON?
  • Schema validity: does the JSON match the expected structure (right fields, right types)?
  • Status decisions: did the agent correctly decide to accept, clarify, or reject the order?
  • Exact item semantics: did it pick the right menu items?
  • Constraint preservation: did it honour the customer's modifications and exclusions?
  • Unsafe acceptances: did it accept an order it should have rejected or clarified?

Most production teams check the first two and stop. OrderBench checks all six.

How the test ran

The study made 2,400 calls through Nebius Token Factory to four open models S¹P⁵. Each model ran in two configurations: prompt-only, where the model receives instructions to produce JSON with no enforcement mechanism, and JSON-schema mode, where the provider's structured-output capability forces the model to generate JSON that conforms to a given schema S¹.

The distinction matters because structured-output modes have become the default answer to "how do I make my LLM agent reliable." OpenAI, Anthropic, Google and others all offer some version of it. The promise is that the model can't produce malformed output. OrderBench tests whether that promise extends to correctness.

The gap that matters

In the strongest model tested, both prompt-only and JSON-schema modes achieved 100% schema validity S¹. No broken JSON, no missing fields, no wrong types. By the measure most teams use, the agent was perfect.

Semantic success, though, remained near 80% S¹. One in five orders was wrong in some way the schema couldn't catch: the wrong menu item, or an order accepted when it should have been rejected.

Strongest model: schema validity vs semantic success

In weaker models, the picture is worse. Schema-valid unsafe acceptances, orders that passed every structural check but should never have been accepted, occurred in double digits S¹. These are the most dangerous failures: the downstream system sees a clean, valid object and executes it. Nobody gets an alert.

The paper's central finding is stark: output that passes schema validation can still contain meaningful errors at high rates S¹. While JSON Schema and provider-level structured-output features eliminate many formatting and parsing failures, they cannot on their own guarantee that the resulting transaction is safe or accurate S¹.

Why this connects to the wider agent problem

LLM agents are increasingly serving as transaction compilers: a user describes what they want in natural language, and the model generates a structured object that an API can act on S¹. The restaurant domain is a stand-in for any setting where a model translates fuzzy human language into a precise, executable instruction.

The OrderBench finding exposes a pattern worth noting: the model produces something that looks right structurally but is wrong semantically. The shape of the problem is consistent even when the domain changes.

The authors caution that while structured output is an essential interface mechanism, it cannot replace domain-specific verification and execution that refuses to proceed when checks fail S¹. In plain terms: the schema check tells you the output is shaped right. It does not tell you the output is right.

What to do about it

Consider a pharmacy chain building an agent that takes prescription refill requests by chat. The customer types "I need my metformin refilled, the 500mg ones, and skip the vitamin D this time." The agent emits a JSON object with the drug name, dosage, and a flag to exclude the supplement. The schema validates. The API executes.

But what if the agent picked the 1000mg dose instead of 500mg? What if it included the vitamin D anyway? What if it accepted a refill for a medication the customer hasn't been prescribed in over a year? The schema would catch none of those. Only a domain verification layer, checking the dosage against the patient's record, confirming the exclusion, and refusing to process anything that doesn't match, would catch them.

The practical takeaway: treat structured output as the floor, not the ceiling. After the schema check passes, run a second layer of validation that checks the actual content against domain rules. If the content doesn't match, fail closed. Don't execute. This week, audit one agent workflow you run: find the point where schema validation ends and ask what, if anything, checks the semantics after that.

What we don't know yet

The paper is an arXiv preprint and has not been peer-reviewed S¹. The four open models tested are not named in the abstract, which makes the results hard to verify independently. The "near 80%" and "double digits" figures are approximate, and the full distribution across models is not available from the abstract alone.

The benchmark covers only restaurant ordering. Whether the 20-point gap between schema validity and semantic success holds in other transaction types, such as financial transfers, medical scheduling, or legal filings, is an open question. The paper introduces a benchmark and a warning but does not propose a technical fix for the semantic gap it exposes.

The next signal: a closer reading of the full paper, available on arXiv P², should reveal the four model names and exact per-layer error rates. We'll check the semantic gap claim against those numbers. If you want the follow-up, subscribe now and you'll get it.

Sources: S1 — When JSON Is Not Enough: Semantic Reliability of Schema-Constrained LL · P2 — When JSON Is Not Enough: Semantic Reliability of Schema-Constrained LL · P3 — [2604.25149v1] Semantic Layers for Reliable LLM-Powered Data Analytics · P4 — krasserm/bot-with-plan · P5 — nebius/token-factory-cookbook

More from Not A Tech Guy

Generated from an audited evidence pack with primary-source research. Social-media items are discussion signals, not verified facts. Nothing here is financial, legal or medical advice.