RAG Confidence Thresholds: When an AI Should Say I Don't Know

Google Research ran a test on RAG, the pattern where a model answers by pulling in your own documents first. I cover that mechanism in full in how RAG gives an agent access to your own documents.

Gemma got 10.2% of answers wrong with no context at all. Handed a passage that looked related but wasn't, it hallucinated 66.1% of the time. [1] A passage that looks related but isn't does more damage than no passage at all.

You should set your system to return nothing when the number behind an answer, a similarity score on a retrieved passage or a self-scored confidence number on the answer itself, falls below a threshold you set on purpose. Below that line, wrong answers outnumber right ones.

What decides whether an answer comes back?

The mechanism is a score. You can score any answer or source with a similarity comparison, a logprob, or a second model call, then set a threshold ahead of time for how good is good enough. Anything that scores below it gets refused.

Google's ASPIRE, published at Findings of EMNLP 2023, trains a model to grade its own correctness and answer "I don't know" when that grade comes in low. [2]

On the CoQA benchmark, a conversational question-answering dataset, ASPIRE moved the AUROC score, a measure of how well a confidence number separates right answers from wrong ones, from 51.3% to 80.3%.

A 2.7-billion-parameter OPT model, one of Meta's open language models, running ASPIRE matched or beat a 30-billion-parameter OPT model with no ASPIRE, at ten times the size.

Before training anything new, check what your own build already exposes:

1. Your retrieval step already returns a cosine similarity for its top match.
2. Your model call already carries a per-token logprob you can read off the response.
3. Your reranker, if you run one, already returns its own relevance score.

Any of those three can become a threshold before you build ASPIRE's self-grading step. How to choose a retrieval cutoff is the workflow for turning a similarity score into a real number, from real questions rather than a guess.

Why is a threshold also a spend decision?

Google's Gemini API ships a real version of this dial. Every prompt that requests grounding, Google's term for search-backed answers, gets a prediction score between 0 and 1. Below a threshold that defaults to 0.3, Google skips the search and lets the model answer on its own. [3]

That's a different mechanism from ASPIRE:

Mechanism	What crosses the threshold	Below the threshold
Gemini's dynamic retrieval	A prediction score on whether the prompt needs a search	The model still answers, just without a fetched source
ASPIRE	A confidence score on the answer itself	The system returns "I don't know" instead of an answer

The bill is real. Google's own pricing page lists $35 per 1,000 grounded queries on the Gemini 2.5 family, checked 2026-09-14, and $14 per 1,000 on the newer Gemini 3 family. [4] A stricter threshold means fewer grounding calls and a smaller invoice, on top of whatever your own retrieval layer charges per lookup.

Why doesn't any company publish its abstention rate?

I looked for one number before writing this: what share of queries a real production RAG system refuses. I searched abstention rate, refusal rate, IDK rate, coverage-vs-accuracy, selective prediction, and escalation threshold, across legal, clinical, and enterprise search.

I found first-party research (ASPIRE and the sufficient-context paper above) and third-party audits of named products. I could not find a single company saying "we say I don't know on X% of our own queries." No named company has published that number, and no benchmark for it exists to go find.

What happens when there's no refusal rule at all?

Stanford's RegLab, the university's regulation and technology lab, ran the first preregistered empirical evaluation of commercial legal AI tools. Preregistered means the test questions and scoring method were locked before anyone saw a result, so the test couldn't be tuned to a favorable answer after the fact. [5]

LexisNexis and Thomson Reuters legal research tools each hallucinated between 17% and 33% of the time, against LexisNexis's own marketing claim of "100% hallucination-free" linked citations.

More specific per-tool numbers circulate alongside that study: a 65% accuracy figure for Lexis+ AI, 42% for Westlaw, incompleteness rates of 18%, 25%, and 62%. Those came from search results and a company landing page, not from the paper itself. Treat them as unconfirmed rather than as RegLab's finding until you've read the paper directly.

Is there a refusal rate for summarization too?

Vectara, a company that runs a public hallucination benchmark, publishes something adjacent: an "Answer Rate" per model, the share of documents a model summarized instead of declining to. Phi-4 sits at 80.7%. Gemini 2.5 Flash Lite sits at 99.5%. [6]

Vectara's Answer Rate tells you whether a model will summarize a document at all, not whether the retrieved source was good enough to answer from.

What should you check in your own build?

Three questions, in order:

1. Does your retrieval or generation step expose any score at all? A cosine similarity, a logprob, a reranker score, a self-scored confidence number. If nothing is exposed, that's the first thing to add, before a threshold means anything.
2. Is there a floor anywhere in the pipeline, or does every query get answered regardless of the score? I walk through building that floor from real questions, instead of a guess, in how to choose a retrieval cutoff.
3. What would a stricter floor cost you? Gemini's published $35 per 1,000, or $14 on its newer models, is one real number to reason from, even on a system that runs on a completely different mechanism.

Before you buy a tool that touches customer-facing answers, ask what its abstention mechanism is and what threshold triggers it. LexisNexis marketed "100% hallucination-free" citations while the RegLab audit found a 17-33% hallucination rate behind that claim.

Quick recap

• Retrieval that returns something irrelevant is worse than retrieval that returns nothing: 10.2% wrong with no context, 66.1% wrong with insufficient context (Google Research, ICLR 2025).
• The mechanism is a score plus a threshold. ASPIRE moved CoQA AUROC from 51.3% to 80.3% by training a model to grade its own correctness.
• Gemini's threshold decides whether a search fires at all; below 0.3, the model still answers without one. ASPIRE's threshold decides whether an answer ships at all; below it, the system stays quiet instead of guessing.
• A threshold is a spend dial: Gemini charges $35 per 1,000 grounded queries on its 2.5 models, $14 on Gemini 3.
• No named company publishes its own abstention rate.
• Skipping the mechanism has a measured cost: LexisNexis and Thomson Reuters legal AI tools each hallucinate 17-33% of the time, against marketing that claimed otherwise.

Start Here

The intake at daisyguti.ai/work-with-me is about nine questions and takes a few minutes, held as a live conversation with an AI. It turns your answers into a brief with the specifics that matter: what you're trying to solve, what you've already tried, what a working version looks like to you.

I read that and reply with whether a threshold like this, or a different fix inside your retrieval pipeline, is the right next step.

Sources

1. Google Research, "Sufficient Context: A New Lens on Retrieval Augmented Generation Systems," ICLR 2025 - https://arxiv.org/abs/2411.06037 - Gemma gave incorrect answers 10.2% of the time with no context, rising to 66.1% with insufficient context.
2. Google Research, "Introducing ASPIRE for selective prediction in LLMs" (paper: Chen et al., Findings of EMNLP 2023) - https://research.google/blog/introducing-aspire-for-selective-prediction-in-llms/ - ASPIRE improves CoQA AUROC from 51.3% to 80.3%; a 2.7B-parameter OPT model with ASPIRE matched or beat a 30B-parameter OPT model.
3. Google Developers Blog, "Gemini API and Google AI Studio now offer Grounding with Google Search" (October 31, 2024) - https://developers.googleblog.com/en/gemini-api-and-ai-studio-now-offer-grounding-with-google-search/ - dynamic retrieval's 0-1 prediction score and 0.3 default threshold.
4. Gemini Developer API pricing page, checked 2026-09-14 - https://ai.google.dev/gemini-api/docs/pricing - $35 per 1,000 grounded prompts on Gemini 2.5 models, $14 per 1,000 on Gemini 3 models.
5. Magesh, Surani, Dahl, Suzgun, Manning, Ho (Stanford RegLab), "Hallucination-Free? Assessing the Reliability of Leading AI Legal Research Tools," May 2024 - https://arxiv.org/abs/2405.20362 - the first preregistered empirical evaluation of legal AI research tools; 17-33% hallucination rate against a "100% hallucination-free" marketing claim.
6. Vectara, Hallucination Leaderboard - https://github.com/vectara/hallucination-leaderboard - Answer Rate: Phi-4 at 80.7%, Gemini 2.5 Flash Lite at 99.5%.