Multi-Tenant AI Systems: What Breaks When You Add Customers

Notion's active workspaces grew 15x over about two years while infrastructure spend fell more than 90% in the same stretch [1].

Both numbers sit in the same engineering post as the failure that forced the rebuild: their first architecture, dual-path indexing on dedicated pods, ran out of capacity within a month of launch [1]. Build for a second customer before you find yours.

Each of the three breaks below shares the same root cause: the build was written and tested against one tenant, so nothing in it distinguishes a second from the first until something collides.

• One shared index that lets one tenant's writes degrade another's reads
• One queue with no fairness rule that lets one tenant starve everyone else's requests
• A bottleneck nobody measured before assuming what it was

What Breaks First in a Multi-Tenant Build?

This explanation comes from Qdrant's own engineering docs. It's a vendor's account of its own system: Qdrant is explaining its own product, and no outside party audited the numbers.

They describe two symptoms of one collection shared by every tenant. On the write side: "indexing speed can become a bottleneck when many tenants share a collection" [2].

On the read side, once a query has to filter down to one tenant's rows inside a graph built for everyone: on 5 million vectors with two filters at roughly 4% selectivity, standard HNSW held only 53.34% accuracy at 1.25ms [2].

Turning on Qdrant's ACORN algorithm, which walks past a filtered-out neighbor instead of stopping at it, brought that same query to 97.20% accuracy, at 13.86ms [2]. That's roughly eleven times the latency, spent to buy back the accuracy the shared graph lost in the first place.

That cost lands on every query, not just once at index-building time.

How Do You Partition a Shared Index?

Notion's fix names the exact field: workspace ID. Every query and write routes by workspace ID to the shard that owns it, using range-based partitioning, so one workspace's load never lands on another workspace's shard [1].

They paired that with a page-state cache, a hash of each page's content deciding whether it needs re-embedding at all, which cut the volume of data getting reprocessed by 70% [1].

Qdrant ships a version of the same idea for the filtering failure above. A multitenant collection "can consist of a shared 'fallback' shard for small tenants and multiple dedicated shards for large tenants" [2].

A tenant that outgrows the fallback shard gets promoted "to their own dedicated shard when they grow large enough," live, with both reads and writes continuing through the move and no reindex required [2].

Check your own build against one question: is there a tenant ID anywhere in the routing path, the way a filtered vector search needs one, or does every query hit the same index no matter who asked?

What Breaks in the Queue?

The second break has nothing to do with the index. It's the queue. One tenant's large batch job can starve every other tenant's requests if nothing in the queue tells the difference between them. It's the same shared-state problem parallel agents run into inside one stage: more than one writer touching a shared resource with nothing arbitrating between them.

Cohere's production system runs a request through admission control, then an SLA tier, then Deficit Round Robin (DRR), then a priority queue inside each tenant's own line [3]. DRR gives every tenant a turn and a budget for that turn; once a tenant's budget runs out for the round, its next request waits and the next tenant gets a turn.

The choice that matters is what DRR budgets by. For ordinary chat traffic, Cohere budgets by request count; for batched embedding and reranking work, it budgets by token count instead [3].

A 10-token chat turn and a 50,000-token batch job aren't "one request" each when tokens are what maps to GPU cost. Budgeting by request count for batched work gives you a queue that looks fair while one tenant burns most of the GPU without anyone noticing.

Does the Same Wall Show Up in Serving?

You hit the same wall one layer down, in the model serving the requests.

Microsoft Research's DiskANN shows what "solved" looks like at the index layer: more than 5,000 queries a second at under 3ms mean latency, with 95%+ recall, on a billion points, on a single node [5]. That's the ceiling a tenant-aware index is reaching toward, without giving up the isolation Qdrant's numbers show you pay for.

DeepSeek published something rarer than a benchmark: real 24-hour production telemetry. Across 608 billion input tokens that day, 56.3% hit the cache. Actual infrastructure cost for those 24 hours: $87,072. The same traffic billed at list API pricing would have cost $562,027 [4].

Cloudflare's own story cuts the other way. They replaced vLLM with a Rust engine, Infire, and the reason wasn't GPU throughput. Their real bottleneck was CPU: 25% utilization on Infire against vLLM's 140% on bare metal, rising to 250% inside their sandbox [6].

They measured which resource was saturated instead of assuming it was the GPU. The same discipline works for picking a retrieval cutoff by testing it against real queries instead of guessing.

What Should You Check Before It Grows?

Run these four checks against your own build before a second customer arrives:

1. Is there one shared index across every customer right now, with no tenant field anywhere in the routing path?
2. If a heavy customer sends a large batch job, does anything stop it from starving everyone else's requests?
3. Is your queue's fairness rule based on request count or token count, and does that match where your real cost comes from?
4. Have you measured your bottleneck under load, or are you assuming it's the GPU?

Run it under load and read the CPU and GPU utilization numbers yourself. Cloudflare's wall was CPU, and they only found that by looking.

Quick Recap

• Notion's workspaces grew 15x and costs fell over 90%, but only after their first architecture hit a capacity wall a month in [1].
• One index shared by every tenant is the first break: filtered multi-tenant search on Qdrant's own numbers fell to 53.34% accuracy before a fix brought it back to 97.20%, at roughly eleven times the latency [2].
• Partition by tenant. Notion routes by workspace ID; Qdrant promotes a growing tenant to its own shard live, with no reindex [1][2].
• An unfair queue is the second break. Cohere budgets by token count for batched work, because token count is what maps to GPU cost [3].
• The same wall shows up in serving: DiskANN shows the index-layer ceiling, DeepSeek published a rare real cost number, and Cloudflare found their bottleneck was CPU, not GPU, only because they measured [4][5][6].
• Before a second customer arrives, run the four checks above.

Start Here

The intake at daisyguti.ai/work-with-me is about nine questions and takes a few minutes, held as a live conversation with an AI. I read what comes back and reply with whether your build needs tenant-aware partitioning, a fairer queue, or neither yet.

Sources

1. Notion Engineering, "Two years of vector search at Notion: 10x scale, 1/10th cost" (February 19, 2026) - https://www.notion.com/blog/two-years-of-vector-search-at-notion - the 15x growth in active workspaces, the 90%+ embeddings-cost reduction, the 60% search-spend cut, the 70% reprocessing reduction, workspace-ID range-based partitioning, the page-state cache, and the capacity-wall failure of their first architecture.
2. Qdrant, "Qdrant 1.16 - Tiered Multitenancy & Disk-Efficient Vector Search" (November 19, 2025) - https://qdrant.tech/blog/qdrant-1.16.x/ - vendor engineering docs, not an audited customer case: the 53.34%/1.25ms standard HNSW vs. 97.20%/13.86ms ACORN figures on 5M vectors at ~4% filter selectivity, and the tiered-sharding, tenant-promotion fix. The indexing-bottleneck line is from Qdrant's multitenancy documentation - https://qdrant.tech/documentation/manage-data/multitenancy/.
3. Cohere, "LLM Serving Fairness: No More Noisy Neighbors" (June 17, 2026) - https://cohere.com/blog/serving-fairness - the admission-control, SLA-tier, Deficit Round Robin, priority-queue pipeline, and DRR's token-count budgeting for batched embedding and reranking work.
4. DeepSeek, "Day 6: One More Thing, DeepSeek-V3/R1 Inference System Overview" (February 2025) - https://github.com/deepseek-ai/open-infra-index/blob/main/202502OpenSourceWeek/day_6_one_more_thing_deepseekV3R1_inference_system_overview.md - the 608-billion-input-token, 56.3%-cache-hit, $87,072-actual-vs-$562,027-theoretical 24-hour production telemetry.
5. Microsoft Research, "DiskANN: Fast Accurate Billion-point Nearest Neighbor Search on a Single Node" (NeurIPS 2019) - https://www.microsoft.com/en-us/research/publication/diskann-fast-accurate-billion-point-nearest-neighbor-search-on-a-single-node/ - the greater-than-5,000-queries-per-second, under-3ms mean latency, 95%+ recall figures on a billion points, single node.
6. Cloudflare, "How we built the most efficient inference engine for Cloudflare's network" (August 27, 2025) - https://blog.cloudflare.com/cloudflares-most-efficient-ai-inference-engine/ - the Infire engine replacing vLLM, and the 25% vs. 140% (bare metal) vs. 250% (sandboxed) CPU-utilization figures.