Serving an AI Agent to Real People: What Breaks in Production

An agent that answers well in a demo can still fall over the day real people start hitting it at once. LangChain found that out with its own web-serving library.

It retired LangServe on November 18, 2024, and said exactly why: "We recommend using LangGraph Platform rather than LangServe for new projects." [1] The repository went read-only on May 5, 2026. [1]

A serving layer for an agent has to do three things a plain wrapper cannot:

1. Hold state across turns.
2. Survive a dropped connection.
3. Answer while the model is still generating it.

LangServe could do none of them. Its own deprecation notice names exactly why.

LangServe did one job: take a chain and put it behind an HTTP endpoint. That's also the first shape most teams reach for the moment an agent has to serve real traffic instead of running in a notebook.

What LangServe's own notice said was missing

LangChain didn't call LangServe broken. The deprecation notice names eight specific gaps, and each one maps to something that goes wrong the moment a demo becomes a product.

What the notice named	What breaks without it
Persistence	Nothing survives past the response. The moment the answer is sent, the conversation is gone.
Memory	Every request starts from zero. There's no shared record of what the last three turns said.
Double-texting	A user sends a second message before the first reply finishes. A request-response wrapper has no queue to put it in.
Human-in-the-loop	Nothing to pause on when a step needs a person's yes. The response either finishes on its own or it doesn't finish.
Cron and webhooks	Nothing fires unless a request triggers it. A job that has to run on a schedule, or react to something happening elsewhere, has nowhere to live.
High-load management	No concurrency controls beyond whatever your web server ships with by default.
Advanced streaming	Token-by-token still assumes one open connection. Nothing coordinates several concurrent streams or picks a dropped one back up.
Long-running tasks	An HTTP connection has a timeout. A task that takes minutes outlives it.

A plain wrapper treats every request as if it starts and ends with the HTTP call. An agent doesn't work that way:

• It holds a conversation across turns.
• It can take minutes to finish a step.
• It sometimes needs a person's approval before it moves.

How streaming changes what an agent can do mid-answer

LinkedIn's engineering team put a production generative AI feature in front of members and wrote up what changed under load. [2]

They fixed how the response left the server before touching the model: "a full answer might take minutes to complete, so we make all our requests stream to reduce perceived latency." Nobody was going to sit through a two-minute spinner.

Streaming also changes what a response can do while it's still arriving. Their system decides which internal APIs to call based on what the model is saying, rather than waiting for it to finish talking.

"We fire API calls as soon as parameters are ready, without waiting for the full LLM response." A call fires the moment its own parameters are complete, while the rest of the answer is still generating.

How LinkedIn took a 10% malformed-output rate to 0.01%

Ask a model for structured output and read enough of them, and some will come back wrong. LinkedIn measured it directly: "~90% of the time, the LLM responses contained the parameters in the right format, ~10% of the time the LLM would make mistakes."

A re-prompt costs a full extra round trip in latency and GPU time on every failure. LinkedIn built an in-house defensive parser instead, one that patches the common mistakes without asking the model to try again, and took the error rate down to about 0.01%.

The parser reads YAML on purpose: "We picked YAML because it is less verbose, and hence consumes fewer tokens than JSON." Fewer tokens per response, at their volume, is real money, and it's a formatting choice made before a single line of parsing logic got written.

How LinkedIn tuned throughput and hit a latency wall

Under load, you can buy more throughput. LinkedIn's own number: "it's not uncommon to get 2x/3x the TokensPerSecond (TPS) if you are willing to sacrifice" time-to-first-token and time-between-tokens, the two numbers that decide how fast a response feels to the person reading it.

Here's what that costs in practice, in their own math: "for a 200-token reasoning step, even a 10ms TBT increase means an extra 2s of latency." Ten milliseconds a token doesn't sound like anything until you multiply it by every token in the answer.

LinkedIn optimized the wrong half of it first. They tuned for time-to-first-token, the number that maps directly to how fast a reply feels to start: "we initially only bounded TTFT as that mapped to member latency directly."

Then chain-of-thought reasoning steps got added to the prompts, and time-between-tokens started costing more than they'd priced in: "as Chain of Thought became prominent in our prompts, we neglected that TBT would hurt us much more," and "some tasks were hitting timeouts."

One latency metric got tuned. A different one, left alone, pushed requests past the point where they finished at all.

Gradio defaults to 1 concurrent worker

Gradio, the framework more than a million developers use every month to put a screen on an AI model, [3] ships a queue in front of every app so it can handle concurrent users without falling over.

The setting that decides how many of those users get served at once is default_concurrency_limit, and it defaults to 1: one worker running your function, no matter how many people are waiting. [4]

Two more defaults sit next to it:

Setting	Default	What it controls
default_concurrency_limit	1	Workers running your function at once
max_threads	40	Total size of the thread pool
max_batch_size	4	Requests grouped into one model call

If your queue is backing up and default_concurrency_limit is already as high as your thread pool allows, Gradio's own guidance is to batch: group several requests into one model call with max_batch_size, rather than keep raising the concurrency number. The hardware handles one batched call more efficiently than the same work spread across parallel workers.

What a serving layer looks like under real traffic

Two companies published the shape of a serving layer built to survive real traffic, with no dramatic story attached, just the numbers.

Uber built a GenAI Gateway: one HTTP/JSON interface, shaped like OpenAI's own API, sitting in front of every model it uses, vendor and in-house alike. [5] Code written against LangChain or LlamaIndex doesn't have to change when the model behind it does.

By July 2024 it was carrying more than 60 use cases across close to 30 teams, 16 million queries a month, with a peak of 25 queries per second.

Wealthsimple runs the same pattern at a smaller scale. Its internal LLM gateway launched in April 2023, and by September 2024 it had handled more than 72,000 requests. [6] Same idea as Uber's, one gateway in front of the models instead of every team wiring up its own client.

What a serving layer owes

LangServe got replaced for exactly the gaps its own notice named. Run the three requirements from the top of this guide against whatever tool shows up in the next proposal before you build on it.

If you're still translating the tool names on a proposal before you get to any of this, the builder-stack guide covers what Gradio, LangServe and LangGraph each are.

And if the agent itself isn't built yet, how my AI office runs on LangGraph is the architecture this guide assumes you have running: the loop, the state it holds, and the gate that pauses it.

If you're not sure the job needs an agent at all, start with the test for whether it does before you build anything to serve.

Quick recap

• LangChain retired LangServe on November 18, 2024, and named eight gaps: persistence, memory, double-texting, human-in-the-loop, cron and webhooks, high-load management, advanced streaming, long-running tasks.
• LinkedIn fires an API call the moment its parameters are ready, without waiting for the full model response. That only works because streaming is wired into the architecture from the start.
• A defensive parser beats a retry. LinkedIn's own YAML parser took a ~10% malformed-output rate to about 0.01%.
• Under load you can trade time-to-first-token and time-between-tokens for more throughput, but tuning one without the other is how a system starts timing out.
• Gradio's default_concurrency_limit defaults to 1. Its own guidance for most bottlenecks is to batch requests into one model call rather than raise that number.
• Uber and Wealthsimple both built one gateway in front of every model instead of wiring each team's own client to it.

Start Here

The intake at daisyguti.ai/work-with-me is about nine questions and takes a few minutes, held as a live conversation with an AI. It turns your answers into a brief with the specifics that matter: what you're trying to solve, what you've already tried, what a working version looks like to you.

I read that and reply with whether a system like this, or a piece of it, fits your business.

Sources

1. LangServe, official repository and deprecation notice - https://github.com/langchain-ai/langserve - "We recommend using LangGraph Platform rather than LangServe for new projects," deprecated November 18, 2024, archived read-only May 5, 2026.
2. LinkedIn Engineering, "Musings on Building a Generative AI Product" - https://www.linkedin.com/blog/engineering/generative-ai/musings-on-building-a-generative-ai-product
3. Hugging Face, "Gradio Hits a Million Developers" - https://huggingface.co/blog/gradio-1m - published April 4, 2025.
4. Gradio, "Setting Up a Demo for Maximum Performance" - https://www.gradio.app/guides/setting-up-a-demo-for-maximum-performance
5. Uber Engineering, "Navigating the LLM Landscape: Uber's Innovation with GenAI Gateway" - https://www.uber.com/en-US/blog/genai-gateway/ - published July 11, 2024.
6. Wealthsimple Engineering, "Get to Know Our LLM Gateway" - https://engineering.wealthsimple.com/get-to-know-our-llm-gateway-and-how-it-provides-a-secure-and-reliable-space-to-use-generative-ai - published September 17, 2024.