I Built AI That Answers From My Own Documents

On August 8, writing this guide, I opened memory/client.py to read my own retrieval query. I expected a floor on it — a score a passage has to clear before my agents are allowed to see it. There wasn't one. ORDER BY distance LIMIT 5, no WHERE. Every question got five paragraphs back, however far off they were.

I run nine AI agents inside my LLC, and they answer out of my real files — a charter, an authority matrix, and 61 dated decision records. Here's the machinery under that, the floor I added, and the case where I'd tell you not to build any of it.

The buzzwords guide translated the word for this — RAG — and has the table comparing pasting a document in against searching it. This one is the build.

How does AI answer from your own documents?

It searches them first. Your files get cut into passages, each passage becomes a list of numbers, and those numbers are stored so a search can rank by meaning instead of by keyword. Ask a question and the closest few passages go into the prompt. The model answers out of those passages.

Four parts, and every product in this category has all four whether or not the sales deck names them:

• A loader reads your files and cuts them into chunks.
• An embedder turns each chunk into a fixed-length list of numbers.
• A store holds those numbers and returns the nearest ones on request.
• A prompt hands the retrieved passages over with an instruction to answer out of them.

Most of the damage happens in the loader. Mine packs whole paragraphs up to 1,500 characters, then carries the last 200 characters forward into the next chunk so a thought cut at a boundary still shows up whole somewhere.

A loader that counts characters and ignores paragraphs will cut a price table between the header row and the numbers. Those numbers get stored with no labels on them. They still rank.

Something to check today. Ask for the timestamp on the newest chunk in your index, then look at the newest file in the folder it reads. Months apart means the loader ran once at install and nobody wired the second run.

What does that look like in a real database?

Postgres 16 with the pgvector extension switched on. One table named chunks holds the passage text with a vector(1024) column beside it. A unique constraint on (document_id, chunk_index) keeps a re-run from storing the same passage twice. An HNSW index on that column using vector_cosine_ops is what makes the search fast.

CREATE TABLE chunks (
    id          uuid PRIMARY KEY DEFAULT gen_random_uuid(),
    document_id uuid NOT NULL REFERENCES documents (id) ON DELETE CASCADE,
    chunk_index int  NOT NULL DEFAULT 0,   -- order within its document
    content     text NOT NULL,             -- the passage a person can read
    embedding   vector(1024) NOT NULL,     -- must match what the embedder emits
    UNIQUE (document_id, chunk_index)      -- re-run the loader, get no duplicates
);

CREATE INDEX chunks_embedding_hnsw
    ON chunks USING hnsw (embedding vector_cosine_ops);

Three things in there do real work:

• The text sits in the same row as the numbers. Nobody can read 1,024 floats. Keeping content beside embedding is what lets you open the source passage and check the answer against it.
• The unique constraint is the duplicate guard. Run the loader twice without it and every passage has a twin. Both come back, and the model reads the same paragraph as two independent sources.
• vector_cosine_ops picks the distance measure. The bge-m3, Voyage and OpenAI docs all point to cosine for normalized text embeddings.

I left the index tuning alone. pgvector defaults m to 16 and ef_construction to 64. [1] My schema file says to revisit past roughly 100,000 chunks, and I am nowhere near that.

Which embedding model should you run?

Mine is bge-m3 on Ollama, running on my Mac. It emits 1,024 numbers per passage, reads up to 8,192 tokens at a time, covers more than 100 languages, and carries the MIT license. [2][3] Every embed costs $0 and the text never leaves the machine. I picked it to keep company text off other people's servers.

Embedder	Dimensions	Indexes in pgvector	Cost	Where your text goes
bge-m3 on Ollama	1,024	Plain vector	$0	Stays on your machine
Qwen3-Embedding-0.6B (Ollama)	up to 1,024	Plain vector	$0	Stays on your machine
voyage-4-lite	1,024 (default)	Plain vector	Metered per token	Voyage's servers
text-embedding-3-large	3,072	Needs halfvec	Metered per token	OpenAI's servers

Count what your embedder emits before anyone creates the column. Ollama answers on POST /api/embed: [4]

# Ask the local model for one embedding and count the numbers in it.
curl -s http://localhost:11434/api/embed \
  -d '{"model": "bge-m3", "input": "how do refunds work"}' \
  | python3 -c "import json,sys; print(len(json.load(sys.stdin)['embeddings'][0]))"
# 1024   <- this is the N that goes in vector(N)

Why does 1024 keep showing up?

Four embedders land on exactly that number: bge-m3, Qwen3-Embedding-0.6B, voyage-4-lite and voyage-4. [2][5][6] My June 22, 2026 decision file picked 1,024 as the interop width for that reason — moving to hosted Voyage, or to a stronger local model, then needs no schema migration.

The width is fixed the moment you run CREATE TABLE. A vector(1024) column takes 1,024 numbers and rejects anything else. So swapping bge-m3 for voyage-4-lite is two lines of .env and the table never moves. Swap in a model of a different width and you are altering the column and re-embedding every row in it.

# .env — the embedder is chosen here, never in the schema
EMBEDDING_PROVIDER=ollama              # ollama | voyage
EMBEDDING_MODEL=bge-m3                 # ollama: bge-m3 | voyage: voyage-4-lite
OLLAMA_BASE_URL=http://localhost:11434 # only the local path reads this

pgvector has two size limits and they are different numbers.

• A vector column stores up to 16,000 dimensions. [1]
• HNSW and IVFFlat only index up to 2,000 of them. [1] Above that the column still holds your data, and every search reads every row.
• text-embedding-3-large emits 3,072. [7] Starting on OpenAI puts you over the index limit on day one.

What halfvec is. A second pgvector column type that stores each number as a half-precision float — 16 bits instead of 32, 2 * dimensions + 8 bytes per row against 4 * dimensions + 8 for a plain vector. [1] Indexes accept it up to 4,000 dimensions. [1] You can also leave the column alone and cast at index time:

-- index a 3,072-dim column by casting it down to half precision
CREATE INDEX ON chunks USING hnsw ((embedding::halfvec(3072)) halfvec_cosine_ops);

That costs you three things:

• Precision. Every number gets rounded to the nearest 16-bit float. Rankings move a little. How much depends on your data, and nobody can tell you the number without measuring on yours.
• A column change. Either the type migration or the cast above, on a table you have already filled.
• A re-embed, usually. Not for the cast itself. But the reason you are here is a 3,072-dim model, and moving to a different model means embedding every passage again from scratch.

The other route is OpenAI's dimensions parameter, which shortens the vectors before they reach you. [7]

Ask what width a vendor's embedder emits and whether their index can hold it.

What does changing the embedding model cost you?

Every passage you have stored, embedded again. Old vectors and new ones describe different spaces, so a table holding both ranks nonsense. A model with a different output width means re-creating the column on top of that. This is the lock-in under every product in this category, hosted ones included.

Two questions belong on the proposal:

• Which embedding model, and what dimension? Get the model name in writing. "An industry-standard embedding model" is not an answer.
• What does embedding my whole corpus cost, in hours and in dollars? You pay that bill twice. Once on the initial load, before a single question works, and again the day anybody changes the model. Get both numbers before you sign.

How do you keep a weak match from becoming a hallucination?

A hallucination is confident invention — the buzzwords guide has the $5,000 court case. Retrieval narrows the gap by putting real passages in front of the model. It does not close it. A nearest-neighbour search always returns rows, and a weak row reads exactly like a strong one once the model has written it up.

Mine sends the cosine similarity to the model with every passage, so a 0.91 and a 0.34 arrive labelled. Every hit also carries the document_id it came from. Checking an answer means opening one file rather than reading the folder.

Picking a cutoff is measurement work. My duplicate check scores a new observation against what is already on file, and I set the line it has to cross. On July 19, 2026 I measured that on bge-m3 against real rows:

• Same idea, reworded on a later run: 0.72 to 0.86.
• Two related but genuinely different observations: 0.59 to 0.62.
• The cutoff sits at 0.70, inside the gap between them.

You can't borrow that gap off my numbers. Run your own questions through your own index and write down three things for each one:

• The question, worded the way somebody really asked it. Take them from the support inbox, the Slack channel, the site search box.
• The top similarity score that came back.
• Whether that passage was any use. Yes or no, your call.

Sort the list by score. The seam between the yes rows and the no rows is your cutoff.

On how many questions to run. It scales with the size of your corpus and with how varied the questions are, and it is a sample-size problem with real math behind it. Sample-size calculators for a proportion are a search away.

Pick a margin of error you can live with, take the count the calculator gives you, then keep adding rows until the seam stops moving.

Mine had no floor at all, and yes, that was bad

Until August 8 my retrieval path had no cutoff. Ask my agents something my company has never written down and five paragraphs came back anyway — the least-unrelated rows in the table, scored, but never held back. The No matches. refusal in agents/anthropic_runner.py only fires on zero rows, and zero rows means the table came back empty. A weak match never reached it.

Why that matters: the score gets passed along and nothing acts on it. A 0.31 lands in the prompt as a passage to answer from, same as a 0.91. That is the exact spot where a plausible sentence gets written out of nothing related.

memory/client.py::search_chunks now carries a floor:

SELECT content, 1 - (embedding <=> %s) AS similarity
FROM chunks
WHERE 1 - (embedding <=> %s) >= %s   -- the floor. Default 0.45.
ORDER BY embedding <=> %s
LIMIT %s

Where 0.45 comes from. The 0.70 above answers a different question — whether two notes say the same thing. A search wants related-but-different material, which is the band that same July 19 calibration measured at 0.59 to 0.62. So the floor sits underneath that band and keeps it.

It is set low on purpose. 0.45 rejects what is plainly unrelated and nothing else. A floor set too high hides real answers and says nothing while it does it, which is worse than the noise it removes. MEMORY_RETRIEVAL_FLOOR moves it the day measurement on my own corpus says it should move.

What a wrong answer costs depends on who reads it.

Where the answer lands	What a wrong one costs	Who finds out, and when
An internal draft you'd rewrite anyway	A few minutes	You, on the spot
A number inside a quote you send	The gap, plus the conversation about it	The customer, after they've read it
A refund quoted off the policy you retired	A refund! One you never agreed to give	The customer, who will hold you to it

The review is where I got burned. I wrote the duplicate check so an unreadable queue took the same path as an empty one. Both meant go. Six duplicate drafts went out over two runs and I deleted them by hand.

A check that cannot answer has to refuse. "I couldn't read it" is never "go ahead."

When is a chat window still the better answer?

Three conditions. The documents fit in one prompt, they rarely change, and the person asking can spot a bad answer. Paste them in and the model reads all of them on every run. Prompt caching keeps the repeat cost down, and the buzzwords guide shows the one field that switches it on.

While all three hold, a chat tab is the whole system.

When one of them stops holding:

• The pile outgrows the prompt. Now somebody has to write a rule for what the model sees. That rule is the build.
• The files start changing weekly. A pasted copy goes stale the day you edit the original. Nothing warns you.
• Someone starts asking who can't check the answer. That's where a wrong answer travels.

Before you point anything at that folder, count the files in it that contradict each other. Retrieval over three versions of one policy returns all three, ranked by closeness, with nothing on them marking which is live. Move the dead ones somewhere the loader can't reach, or the index goes on offering them.

What changes when this gets bigger?

Retrieval quality first. My June 22, 2026 decision file grades bge-m3 as strong and puts it below the very top hosted models, then names two ways up. Voyage at the same 1,024 dimensions is a config change plus a re-embed, and it sends company text off the machine. A bigger local model is the other route.

The large Qwen embedders pass 2,000 dimensions, so that one costs a halfvec column on top of the re-embed. The store here is pgvector, an extension on the Postgres this Office was already running. A proposal may name a standalone vector database instead; the question to put to it is what that buys over the database already in the stack.

More than one company's files in one table is mostly a query change. You add a tenant column and a WHERE on it. The schema barely moves. That WHERE then behaves differently from one on an ordinary table.

Filtering happens after the index is scanned. With HNSW and the default hnsw.ef_search of 40, a filter matching 10% of your rows returns about 4 of them. [1] You asked for 5 and got 4. No error, no warning, and the rows you missed are sitting right there in the table.

Three documented ways around it, all in the pgvector README: [1]

• Iterative index scans, added in pgvector 0.8.0. SET hnsw.iterative_scan = strict_order; keeps scanning until enough rows clear the filter.
• A partial index, when you filter on a handful of values: CREATE INDEX ON chunks USING hnsw (embedding vector_cosine_ops) WHERE (tenant_id = 123);
• Partitioning by list, when you filter on many. One partition per tenant, each carrying its own index.

NVIDIA's Deep Learning Institute runs a course on the build side of this: Building RAG Agents with LLMs, course ID course-v1:DLI+S-FX-15+V1. [8] That's where to go if you want to write the pipeline instead of buying one.

Quick Recap

• Four parts: a loader, an embedder, a store, and a prompt that hands the passages over. Every vendor has all four.
• Ask for the timestamp on the newest chunk. Months behind your folder means the loader ran once and nobody wired the second run.
• Mine is Postgres 16 plus pgvector — one chunks table, a vector(1024) column, an HNSW index on vector_cosine_ops, m 16 and ef_construction 64 left at the defaults.
• Chunks run to 1,500 characters and break on paragraphs. The last 200 characters carry into the next one.
• bge-m3 on Ollama emits 1,024 dimensions, reads 8,192 tokens, MIT licensed, $0 per embed, text stays on the machine.
• 1024 is an interop width. Four models emit it, so switching providers is a config line and no schema change.
• pgvector indexes up to 2,000 dimensions. OpenAI's 3,072-dim model needs halfvec — 16-bit storage, a column change, and a little precision — or shortened vectors.
• Embedding the whole corpus is a bill you pay twice. Once on the first load and again on any model change. Get both numbers in hours and dollars.
• A search always returns something. Pass the similarity score along and set a floor, or a weak match reads like an answer. Mine had no floor until August 8; it defaults to 0.45 now.
• Filtering runs after the index scan. A WHERE matching 10% of your rows hands back about 4 of the 5 you asked for. Iterative scans, a partial index, or partitioning.
• A chat window wins while the set is small, stable, and read by someone who can check it.

Start Here

You can follow every step here and still not know whether your business needs it yet, or which folder to point it at first.

At daisyguti.ai/work-with-me there's an AI intake assessment that maps where your business stands before anything gets built. It takes a few minutes and gives you a short list of what to hand off first.

Sources

1. pgvector, open-source vector search for Postgres - https://github.com/pgvector/pgvector - "Vectors can have up to 16,000 dimensions"; HNSW/IVFFlat support vector up to 2,000 dimensions and halfvec up to 4,000; "Each vector takes 4 * dimensions + 8 bytes of storage... single-precision floating-point number" against "Each half vector takes 2 * dimensions + 8 bytes... half-precision floating-point number"; HNSW defaults are m 16 and ef_construction 64; vector_cosine_ops is the cosine operator class. On filtering: "filtering is applied after the index is scanned. If a condition matches 10% of rows, with HNSW and the default hnsw.ef_search of 40, only 4 rows will match on average"; "Starting with 0.8.0, you can enable iterative index scans"; partial indexes for a few distinct values and PARTITION BY LIST for many.
2. BAAI/bge-m3 model card, Hugging Face - https://huggingface.co/BAAI/bge-m3 - "Dimension: 1024", "Sequence Length: 8192", "License: mit", "more than 100 working languages".
3. bge-m3 on Ollama - https://ollama.com/library/bge-m3 - the local model server used here; 567M parameters, 1.2GB download, inputs up to 8,192 tokens.
4. Ollama embeddings endpoint - https://docs.ollama.com/api/embed - POST /api/embed takes {"model", "input"} and returns {"embeddings": [[...]]}.
5. Voyage AI embeddings documentation - https://docs.voyageai.com/docs/embeddings - voyage-4, voyage-4-lite and voyage-4-large default to 1,024 dimensions, with 256, 512, 1,024 and 2,048 selectable.
6. Qwen3-Embedding-0.6B model card, Hugging Face - https://huggingface.co/Qwen/Qwen3-Embedding-0.6B - output dimensions from 32 to 1,024, Apache-2.0 license.
7. OpenAI embeddings guide - https://developers.openai.com/api/docs/guides/embeddings - "the length of the embedding vector is 1536 for text-embedding-3-small or 3072 for text-embedding-3-large", shortenable via the dimensions parameter.
8. NVIDIA Deep Learning Institute, "Building RAG Agents with LLMs" - https://learn.nvidia.com/courses/course-detail?course_id=course-v1%3ADLI+S-FX-15+V1