← Back to the factory

Architecture planner · free, runs in your browser

How RAG runs at a million users.

The demo on the home page answers one question at a time from a handful of files. Real systems search terabytes for millions of people. Set your corpus and traffic, and the planner draws the factory you would need: shards, caches, GPUs, KV cache and a cost range, with the reasoning shown.

Every number here is an estimate from editable assumptions. Use it to learn the shape of the problem, not to sign a contract.

Planner picksLarge-scale tiered RAG
Vectors37.5Mestimate 61.2 GB index
Peak load174/sestimate questions per second
GPUs if self-hosted28estimate Llama 3.3 70B
Monthly cost$126k – $265kestimate self-hosted LLM

Hybrid + rerank, engineered for millions of users and billions of vectors: sharded, quantized, cached at every layer.

Your factory: Large-scale tiered RAG

Large-scale tiered RAG architectureIngest: sources → cdc stream → parse → chunk → embed. Stores: vector vault, bm25 (1 shard × 2 replicas). Query: question → sem. cache → embed → search → fuse → rerank → llm → answer. The LLM keeps a prefix/KV cache of the shared prompt.cache hit: answer straight from the cache, no LLMINGEST LINE · once per documentQUERY LINE · every questionSOURCES100 GBCDC STREAMchanges in secondsPARSE30 GB textCHUNK800 charsEMBED1024-d int8?QUESTION174/s peakSEM. CACHE20% hitsEMBEDquestionSEARCHtop 20 + 20FUSERRFRERANK20 → 5LLM×28KV$28 GPUs1ANSWERcitedBM25 INDEXVECTOR VAULT1 shard × 2 replicas

Ingest: sources → cdc stream → parse → chunk → embed. Stores: vector vault, bm25 (1 shard × 2 replicas). Query: question → sem. cache → embed → search → fuse → rerank → llm → answer. The LLM keeps a prefix/KV cache of the shared prompt.

In plain English
The top belt runs once per document: files are parsed into text, cut into chunks, turned into vectors and shelved. The bottom belt runs for every question: it searches the shelves, keeps the best passages and hands them to the LLM, which writes a cited answer. Everything else on the drawing exists to make that fast, cheap or safe at your scale.

Which architecture, and why

The planner picks from your settings. Pick a card to override it.

Why Large-scale tiered RAG

  1. At 1M daily users, about 174 questions per second at peak, one index on one machine and one LLM endpoint will not keep up. You need sharding, replicas and caching at every layer.
  2. A semantic cache answers repeated questions without touching the LLM, and prefix caching avoids recomputing the shared system prompt for every request.

Layered on top:sharded + replicated indexsemantic answer cacheprefix / KV cachestreaming CDC ingestionasync evaluation

Use it when

  • Millions of users or hundreds of QPS
  • Corpora from ~100 GB to petabytes
  • When cost per question decides whether the product survives

Avoid it when

  • Small teams with modest traffic (the moving parts cost more than they save)
  • Early prototypes

Parts list

  • Streaming CDC ingestion
  • Sharded + replicated ANN index
  • int8 / binary quantization with rescoring
  • Semantic answer cache
  • Prefix / KV cache in the inference server
  • Autoscaled GPU pool (vLLM) or managed API
  • Async evaluation on sampled traffic
  • Tracing and cost dashboards

Sizing: from files to vectors to traffic

In plain English
A PDF is mostly layout, images and fonts; only about a third is text you can search. That text is cut into chunks, and each chunk becomes one vector: a list of 1024 numbers. Storing them as 4-byte floats is precise but large; 1-byte integers (int8) or single bits (binary) shrink the index 4× or 32× for a small loss in recall, usually won back by re-scoring the top hits at full precision.
Sizing estimates
QuantityValue estimateHow it is worked out
Files100 GByour corpus slider
Extracted text30 GBfiles × 0.3 text ratio (PDF layout, images and markup are dropped)
Text tokens7.5Btext ÷ 4 characters per token
Chunks = vectors37.5Mtext ÷ 800-character chunks (overlap ignored; adds ~15%)
One vector4,096 B / 1,024 B / 128 B1024 dims × 4 B float32 / 1 B int8 / 1 bit binary
All raw vectors154 GB / 38.4 GB / 4.8 GBfloat32 / int8 / binary
Index in memory61.2 GBint8 vectors + 64 B metadata each, × 1.5 HNSW overhead
Shards × replicas1 × 2 = 2 nodesindex ÷ 64 GB per shard; replicas for failover and 139 searches/s at peak (500/s each)
Embed the corpus once7.5B tokens · 41.7 hat 50k tokens/s batch throughput
Re-embed per day75M tokensdaily freshness churn
Questions per day5Mdaily users × questions per user
Average / peak QPS57.9 / 173.6per day ÷ 86,400 s; peak × 3
LLM calls/s at peak138.9after 20% semantic-cache hits
LLM tokens per day10B in · 1.4B out2.5k prompt + 350 output tokens per call

The KV cache: why GPUs run out of memory

In plain English
While a model writes, it keeps a note (a key and a value vector) for every token it has seen, in every layer, so it does not have to re-read the prompt for each new word. That is the KV cache. It grows with prompt length and with the number of people being answered at the same moment, and it often needs more GPU memory than the model itself. Two caches help: a semantic cache skips the LLM for repeated questions, and prefix caching stores the shared system prompt’s KV once instead of once per request.
80 layers · 8 KV heads · head dim 128 · 70.6B params · the model behind the live demo (on Groq)

Worked out live from your inputs estimate

  1. KV per token2 × 80 layers × 8 KV heads × 128 dims × 2 B327,680 B
  2. KV per request× 2.85k tokens (prompt + answer)934 MB
  3. Requests in flight138.9 calls/s × 9.25 s each (Little’s law)1.28k
  4. KV for all of themshared 600-token prefix stored once per replica949 GB
  5. Model weights70.6B params × 2 B141 GB
  6. GPUs7 replicas × 4 GPUs (80 GB, 90% usable). Memory floor alone: 16.28
One replica: 4 × 80 GB, up to 198 requests at once estimate
weights 141 GB shared prefix 197 MB request KV 146 GB
Without semantic cache or prefix sharing44 GPUs
With this plan’s caches28 GPUs

Caching saves about 16 GPUs here. FP8 KV (set KV bytes to 1 in Assumptions) halves KV memory.

What it costs per month

In plain English
With an API you pay per token, so cost rises in a straight line with traffic. With your own GPUs you pay per hour whether they are busy or not, so self-hosting only wins once traffic keeps them busy (or when data may not leave your network). The vector database is usually a small part of the bill; the LLM is the big one.

Managed APIs

LLM on a hosted open 70B API, pay per token

$210k – $451k/ month estimate

  • LLM API (hosted open 70B)10.0B in + 1.4B out per day$125k – $267k
  • Vector database1 shard × 2 replicas = 2 nodes$600 – $1.8k
  • Embeddings (questions + changed docs)query embeddings plus re-embedding churn$578 – $1.24k
  • Rerankingper-question cross-encoder calls$84k – $180k

Self-hosted LLM

suggested

Llama 3.3 70B on your own GPUs (vLLM), pay per GPU-hour

$126k – $265k/ month estimate

  • GPUs for Llama 3.3 70B28 × 80 GB GPUs, always on$41k – $82k
  • Vector database1 shard × 2 replicas = 2 nodes$600 – $1.8k
  • Embeddings (questions + changed docs)query embeddings plus re-embedding churn$578 – $1.24k
  • Rerankingper-question cross-encoder calls$84k – $180k

One-time

  • Embed the corpus once7.5B tokens$525 – $1.13k

Prices are placeholders, not quotes. Check current pricing and edit them in Assumptions. Ranges multiply point estimates by 0.7–1.5; excludes people, networking, storage of raw files and observability tooling. Cost per question at the low end: $0.84 per 1,000.

Parts catalogue: what to build it from

In plain English
You will store about 37.5M vectors. Under ~10 million, a Postgres extension is plenty. Up to about a billion, a dedicated vector database earns its keep. Beyond that, or when nothing may leave your racks, pick something built to shard.
OptionWhyTrade-off
pgvector (Postgres)One database for rows, metadata and vectors; transactions and joins for ACLs.HNSW indexes get slow and memory-hungry past ~10M vectors per table.
Pinecone / Qdrant / Weaviatefits your planPurpose-built ANN, metadata filters, namespaces, hybrid search; managed or self-hosted.Another vendor and bill; managed tiers price per read/write units or pods.
Milvus / Vespa / sharded OpenSearchDesigned for billions of vectors across many nodes; GPU indexes (Milvus), rich ranking (Vespa).Real operational weight: more services, tuning, and people who know them.