AI Dialog State: How Chatbots Remember What You Said Mid-Conversation

Character.AI serves more than 20,000 inference queries a second, about a fifth of Google Search's own request volume. The average reply on their platform carries 180 prior messages of conversation history behind it. They published both numbers themselves, in June 2024, alongside how they make holding that much history affordable. [1]

Dialog state is what a system carries forward between turns, the prior messages and results a model needs on its next call. Character.AI caches that cheaply, at scale. Microsoft Research and Salesforce found that today's leading models answer 39% less reliably once a conversation runs several turns deep. Prefix caching does not touch that reliability gap.

You need dialog state once you are building anything that holds a conversation past one exchange:

• a support bot
• a coding assistant
• a research agent someone comes back to

Then you decide what to keep. A quick test: watch how your bill or latency moves as conversations get longer. If either climbs steeply past ten or fifteen turns, keep a managed window instead of the full transcript.

What dialog state means, and what it doesn't

Dialog state is the running memory of one conversation, updated after every turn. A chatbot needs it to resolve "switch me to that one" against whatever it named a few messages earlier. An AI agent needs it to avoid re-asking a question someone already answered.

It is not the same mechanism as pausing an agent mid-task to wait for a person's approval. Dialog state is the conversation a model gets handed back on its next turn. LangGraph's interrupt() is a different mechanism: it freezes a single run mid-task until a person answers yes or no.

It is also not the same thing as an agent reading your own documents. An agent using retrieval pulls in something new: a document, a database row. Dialog state is only what the two sides have already said.

How does Character.AI cache a long conversation?

The general technique is prefix caching: instead of recomputing a model's internal representation of everything sitting in its context window on every turn, the system stores that representation and reuses whatever prefix of the conversation has not changed since the last call.

Character.AI's own version, from their June 2024 writeup: [1]

• Cached KV tensors (each token's key/value pair from the attention layers) sit in a tree-structured LRU cache.
• Each entry is indexed by a rolling hash of prefix tokens, a running fingerprint of everything said so far.
• A new request looks up the cache on longest match: the most conversation history it can reuse without recomputing.

The result: a 95% cache hit rate, and serving costs down 33x against where they started in late 2022.

Why didn't a shorter attention window hurt accuracy?

Cutting the attention horizon (how many tokens back a layer can attend to) to 1,024 tokens on most layers did not move Character.AI's eval scores, including a long-context needle-in-haystack test. Only 1 layer in 6 keeps attention across the full history. [1]

Most of a 180-message conversation is not read at most layers, and their own benchmarks did not catch a difference. A different architecture may behave differently, and most providers do not publish their per-layer attention design the way Character.AI just did.

What you can check yourself: whether your own long-context eval scores hold steady as conversations grow past your typical length. If they do, the model may not be reading as much of that history as you are paying to store.

How do you persist conversation state?

LangGraph, the open-source framework for building agents as graphs, has a documented mechanism for this: a checkpointer. It saves a snapshot of the graph's state at every step, keyed to a thread_id you pass in on each call. [2]

from langgraph.checkpoint.memory import InMemorySaver

checkpointer = InMemorySaver()
graph = builder.compile(checkpointer=checkpointer)

result = graph.invoke(
    {"messages": [{"role": "user", "content": "Hi, my name is Bob."}]},
    {"configurable": {"thread_id": "thread-1"}},
)

On a later call with the same thread_id, the graph reads its own prior state back before it does anything else, the same message history Character.AI's cache is built to reuse. PostgresSaver writes that state to Postgres instead of memory, so the conversation picks up after a restart instead of starting over. [2]

None of this is Character.AI's specific cache design. LangGraph's checkpointer is what a smaller system reaches for to hold the same conversation across calls.

Why do multi-turn conversations get less reliable?

Microsoft Research and Salesforce ran 15 models from eight providers through 200,000+ simulated conversations across six tasks: coding, SQL, function calling, math, data-to-text generation, and summarization. Same information as a single-turn prompt, delivered one requirement per turn instead of all at once.

Average result: a 39% drop in performance, multi-turn versus single-turn, on identical underlying tasks. The paper won Best Paper at ICLR 2026. [3]

Their own words: [3]

"When LLMs take a wrong turn in a conversation, they get lost and do not recover."

Salesforce found the same shape on a different task, running their own CRMArena-Pro benchmark: agents handling customer-service workflows scored roughly 58% single-turn, dropping to roughly 35% multi-turn on the same underlying work. [4]

Is a faster cache the same as a more reliable model?

No. Character.AI cut serving cost 33x with prefix caching, and its paper never studied accuracy. MSR and Salesforce measured the 39% accuracy drop and have not published a fix for it. The two are different problems with different causes.

Problem	What it's about	What addresses it
Serving cost	How expensive it is to hold 180 messages of history	Prefix caching. Character.AI cut this 33x.
Reasoning reliability	Whether the model uses that history correctly	A different mechanism. Multi-turn accuracy is still an open problem.

What can you check on your own system?

Two checks, and neither needs Character.AI's cache or MSR's benchmark. They run on what you already have.

1. If you are on Claude with prompt caching on: read usage.cache_read_input_tokens off your own API responses. That is your own cache-hit-rate number, the same measurement Character.AI published for theirs, just smaller. [5]
2. On your own agent: log turn count against task success for a week. If success drops as conversations run longer, you are looking at the same shape MSR and Salesforce measured, whatever the exact percentage turns out to be.

Quick recap

• Dialog state is the conversation a model can be handed back on its next turn, not the same thing as an agent pausing for a person's approval.
• Character.AI holds 180 messages of history per average reply and caches it with a rolling-hash-keyed, tree-structured LRU, at a 95% hit rate and 33x lower serving cost than its 2022 baseline.
• Cutting most layers' attention to 1,024 tokens did not move their eval scores. That is their model, not a universal rule.
• LangGraph persists state per conversation with a checkpointer keyed to a thread_id.
• Multi-turn accuracy dropped 39% on average across 15 models and 200,000+ simulated conversations, per MSR and Salesforce's ICLR 2026 Best Paper.
• Character.AI cut serving cost 33x with prefix caching. Nobody has published a fix for the 39% reliability drop MSR and Salesforce measured.

Start Here

The intake at daisyguti.ai/work-with-me is about nine questions and takes a few minutes, held as a live conversation with an AI. It turns your answers into a brief with the specifics that matter: what you are trying to solve, what you have already tried, what a working version looks like to you.

I read that and reply with whether an agent like the ones in this guide, or a different kind of automation, would fit.

Sources

1. Character.AI, "Optimizing AI Inference at Character.AI" - https://blog.character.ai/optimizing-ai-inference-at-character-ai-2/ - 20,000+ queries per second, 180-message average dialogue history, tree-structured LRU cache keyed by a rolling hash of prefix tokens, 95% cache rate, serving cost down 33x since late 2022, 1,024-token attention horizon on most layers with 1 in 6 keeping global attention and no significant impact on eval metrics. Published June 20, 2024.
2. LangGraph persistence documentation - https://docs.langchain.com/oss/python/langgraph/persistence - the checkpointer concept, the thread_id config key, and InMemorySaver / PostgresSaver.
3. Laban, Hayashi, Zhou, Neville, "LLMs Get Lost In Multi-Turn Conversation" - https://arxiv.org/abs/2505.06120 - 15 models, 200,000+ simulated conversations, six tasks, 39% average multi-turn performance drop. Best Paper Award, ICLR 2026.
4. Salesforce AI Research, CRMArena-Pro - https://arxiv.org/abs/2505.18878 - 58% single-turn success dropping to 35% multi-turn on the same underlying CRM tasks.
5. Anthropic, prompt caching documentation - https://platform.claude.com/docs/en/build-with-claude/prompt-caching - the cache_read_input_tokens usage field.