AI Structured Output: How to Stop Malformed JSON From Agents

You ask a model for a JSON object, or a set of parameters for a function call. Most of the time it comes back exactly right. That's what makes the times it doesn't so easy to miss: a malformed answer reads exactly like a correct one, right up until whatever runs next tries to use it.

LinkedIn built a defensive parser that runs on every response, checking a model's structured output against the schema before anything downstream sees it. It repairs what it recognizes and stops what it can't fix. Their error rate went from about 10% to roughly 1 in 10,000.

If you've read when to use an AI agent instead of one prompt, you already know a model returns free text by default, with nothing built in that forces a shape on it.

Why doesn't structured output work every time?

An LLM samples one token at a time from a probability distribution. Asking it for JSON, or for a specific set of function-call parameters, gives it a target. Nothing at generation time forces every response to land on that target.

LinkedIn measured this directly on their own product: "about ~90% of the time, the LLM responses contained the parameters in the right format." [1] That means about 1 call in 10 came back some other shape: a missing field, a wrong type, a string where a number belonged.

Ninety percent sounds close to solved. It's the other 10% that a real system has to plan for, because nothing in the model's output tells you which calls landed in that group.

What did LinkedIn build instead of a better prompt?

A defensive parser that runs on every response before anything downstream reads it:

• Checks the response against known error patterns
• Repairs what it recognizes
• Stops anything it can't fix before it moves downstream

LinkedIn doesn't re-prompt it.

LinkedIn was direct about why not: re-prompting "adds a non-trivial amount of latency and also consumes precious GPU capacity due to the additional LLM call." [1] A second call costs the same as the first, on top of whatever the first one already cost, and the user is still waiting on both.

One more choice tucked inside their parser: YAML instead of JSON for the format itself. Their reasoning was plain: YAML "is less verbose, and hence consumes fewer tokens than JSON." [1] YAML generates fewer tokens, which costs less per call and gives the model a smaller surface to get the shape wrong on.

The result: LinkedIn's parser brought the malformed-output rate down from about 10% to roughly 0.01%, about 1 in 10,000. [1]

How does Ramp check a returned code's shape?

Ramp classifies businesses into NAICS and SIC industry codes, a much narrower kind of structured output than a JSON blob, but the same failure mode applies: a response that looks fine and isn't.

Their setup is bigger than a single field. They map "100+ internal levels to thousands of codes," and one internal category alone could reach "50 potential NAICS codes." [2]

Rather than ask one prompt to pick correctly out of thousands of options, Ramp split the job in two: a first prompt runs over a long list of thin descriptions to narrow the field to a short list of plausible codes, then a second prompt reads that short list with fuller context and picks one. [2]

When to use an AI agent instead of one prompt covers the fuller version of this retrieve-then-select split and the accuracy numbers Ramp published on each half. The retrieval cutoff guide covers the general version of scoring a retrieval step before you trust what it hands the next one.

What matters here is the step after both prompts finish: "We validate that the output NAICS codes from each LLM prompt are valid." [2] That confirms a returned code is real and well-formed. It says nothing about whether the code is correct.

At Ramp, the same check runs on one field instead of a whole payload.

Ramp published no volume, latency, cost, or absolute accuracy number for any of this. The acc@k and fuzzy-accuracy figures below are relative gains, not a scale you can compare against your own system.

	LinkedIn	Ramp
What it checks	The whole response's shape against the schema	One returned code's validity
The fix	A defensive parser that repairs or stops	A validity check after a two-prompt narrowing
The number	~10% malformed cut to ~0.01%	Up to 60% acc@k improvement, 5%-15% fuzzy-accuracy gain per parameter [2], both relative, no baseline published

Why does Ramp let some wrong answers through?

A model constrained to pick only from a retrieved shortlist sometimes lands on a wrong answer, one it would have gotten right on its own if the first pass had surfaced the correct code as a candidate.

Their own words: "we've also found cases where the LLM predicts the correct code despite it not being present in the recommendations." [2]

So Ramp's validity check confirms the output is a real, well-formed code. It does not confirm the code came from the shortlist retrieval handed over. A schema check stops a malformed answer. It doesn't limit every answer to whatever one earlier step put on the shortlist.

What's the rule underneath both fixes?

A malformed response that comes back looking like an ordinary, well-formed answer is worse than one that visibly fails. LinkedIn's own ~10% malformed rate is what that looks like at production scale: about one response in ten moved through looking right, with nothing in the output to flag it.

Neither team's fix was a better prompt, and neither team just trusted the model's output more. LinkedIn's parser handles the whole payload, every field checked against the schema. Ramp's check is narrower: one returned code, confirmed real and well-formed.

When do you not need a defensive parser?

Most projects don't need one on day one. Skip it, for now, when any of these hold:

1. A person reads the output before anything else touches it: a Gmail draft, a Notion page, a row in a sheet you open yourself. They'll see a wrong shape the moment they look.
2. The volume is low enough to spot-check by hand: a job that runs a few dozen times a day, or one you trigger yourself rather than one that fires on its own.
3. You haven't run the job by hand yet, so you don't know what breaks. A parser built before that means guessing at error patterns instead of catching real ones.

Build the check once a person stops reading every output, or once the volume makes hand-checking a real cost. That's usually the same moment a JSON blob or a function call starts feeding straight into another system with nobody in between.

What should you check on your own build?

If you're calling a model for any structured output, a function call, a JSON object, an extracted field, log a parse-success and parse-failure counter on every call for a week before you assume your own conformance rate.

LinkedIn's ~90% is LinkedIn's own number for their own product on their own prompts. Run your own one-week counter log before you treat either number as a baseline for your system.

Read the primary sources yourself: LinkedIn's post for the parser reasoning, Ramp's for the two-prompt narrowing and the validity check. Both are linked below.

Quick recap

• Asking a model for a shape gives it a target. Nothing at generation time forces the response to land on it.
• LinkedIn measured about 90% correctly-formatted parameters out of the box, about 1 in 10 wrong.
• Their fix: a parser that ran after generation and either repaired what it recognized or stopped what it couldn't. It brought the error rate to about 0.01%.
• Ramp's validity check confirms a returned code is real and well-formed. It doesn't confirm the code is the right one, or that it came from the model's own shortlist.
• Over-constraining a model to only pick from retrieved candidates can force a wrong answer over one the model would have produced correctly on its own.
• Build the check once a person stops reading every output by hand, or once the volume makes that hand-checking a real cost.

Start Here

At daisyguti.ai/work-with-me there's a short intake form: about nine questions, about two minutes. I read every one myself. I'll tell you whether a schema-checking layer like this, or a custom workflow automation, fits what you're building, and what to do next. I'm a 20+ year engineer and I build these systems for small business owners.

Sources

1. LinkedIn Engineering, "Musings on Building a Generative AI Product," Juan Pablo Bottaro (April 25, 2024) - https://www.linkedin.com/blog/engineering/generative-ai/musings-on-building-a-generative-ai-product - source of the ~90% correctly-formatted figure, the ~0.01% post-parser rate, the YAML-over-JSON reasoning, and the latency/GPU cost of re-prompting.
2. Ramp Engineering, "From RAG to Richness: How Ramp Revamped Industry Classification," Ryne Carbone (January 15, 2025) - https://engineering.ramp.com/post/industry_classification - source of the two-prompt narrowing, the validity check, the deliberate-hallucination admission, and the acc@k / fuzzy-accuracy figures (both stated as relative gains only).