Hybrid + rerank, engineered for millions of users and billions of vectors: sharded, quantized, cached at every layer.
Your factory: Large-scale tiered RAG
Ingest: sources → cdc stream → parse → chunk → embed. Stores: vector vault, bm25 (1 shard × 2 replicas). Query: question → sem. cache → embed → search → fuse → rerank → llm → answer. The LLM keeps a prefix/KV cache of the shared prompt.
Which architecture, and why
The planner picks from your settings. Pick a card to override it.
Why Large-scale tiered RAG
- At 1M daily users, about 174 questions per second at peak, one index on one machine and one LLM endpoint will not keep up. You need sharding, replicas and caching at every layer.
- A semantic cache answers repeated questions without touching the LLM, and prefix caching avoids recomputing the shared system prompt for every request.
Layered on top:sharded + replicated indexsemantic answer cacheprefix / KV cachestreaming CDC ingestionasync evaluation
Use it when
- Millions of users or hundreds of QPS
- Corpora from ~100 GB to petabytes
- When cost per question decides whether the product survives
Avoid it when
- Small teams with modest traffic (the moving parts cost more than they save)
- Early prototypes
Parts list
- Streaming CDC ingestion
- Sharded + replicated ANN index
- int8 / binary quantization with rescoring
- Semantic answer cache
- Prefix / KV cache in the inference server
- Autoscaled GPU pool (vLLM) or managed API
- Async evaluation on sampled traffic
- Tracing and cost dashboards
Sizing: from files to vectors to traffic
| Quantity | Value estimate | How it is worked out |
|---|---|---|
| Files | 100 GB | your corpus slider |
| Extracted text | 30 GB | files × 0.3 text ratio (PDF layout, images and markup are dropped) |
| Text tokens | 7.5B | text ÷ 4 characters per token |
| Chunks = vectors | 37.5M | text ÷ 800-character chunks (overlap ignored; adds ~15%) |
| One vector | 4,096 B / 1,024 B / 128 B | 1024 dims × 4 B float32 / 1 B int8 / 1 bit binary |
| All raw vectors | 154 GB / 38.4 GB / 4.8 GB | float32 / int8 / binary |
| Index in memory | 61.2 GB | int8 vectors + 64 B metadata each, × 1.5 HNSW overhead |
| Shards × replicas | 1 × 2 = 2 nodes | index ÷ 64 GB per shard; replicas for failover and 139 searches/s at peak (500/s each) |
| Embed the corpus once | 7.5B tokens · 41.7 h | at 50k tokens/s batch throughput |
| Re-embed per day | 75M tokens | daily freshness churn |
| Questions per day | 5M | daily users × questions per user |
| Average / peak QPS | 57.9 / 173.6 | per day ÷ 86,400 s; peak × 3 |
| LLM calls/s at peak | 138.9 | after 20% semantic-cache hits |
| LLM tokens per day | 10B in · 1.4B out | 2.5k prompt + 350 output tokens per call |
The KV cache: why GPUs run out of memory
Worked out live from your inputs estimate
- KV per token2 × 80 layers × 8 KV heads × 128 dims × 2 B327,680 B
- KV per request× 2.85k tokens (prompt + answer)934 MB
- Requests in flight138.9 calls/s × 9.25 s each (Little’s law)1.28k
- KV for all of themshared 600-token prefix stored once per replica949 GB
- Model weights70.6B params × 2 B141 GB
- GPUs7 replicas × 4 GPUs (80 GB, 90% usable). Memory floor alone: 16.28
Caching saves about 16 GPUs here. FP8 KV (set KV bytes to 1 in Assumptions) halves KV memory.
What it costs per month
Managed APIs
LLM on a hosted open 70B API, pay per token
$210k – $451k/ month estimate
- LLM API (hosted open 70B)10.0B in + 1.4B out per day$125k – $267k
- Vector database1 shard × 2 replicas = 2 nodes$600 – $1.8k
- Embeddings (questions + changed docs)query embeddings plus re-embedding churn$578 – $1.24k
- Rerankingper-question cross-encoder calls$84k – $180k
Self-hosted LLM
suggestedLlama 3.3 70B on your own GPUs (vLLM), pay per GPU-hour
$126k – $265k/ month estimate
- GPUs for Llama 3.3 70B28 × 80 GB GPUs, always on$41k – $82k
- Vector database1 shard × 2 replicas = 2 nodes$600 – $1.8k
- Embeddings (questions + changed docs)query embeddings plus re-embedding churn$578 – $1.24k
- Rerankingper-question cross-encoder calls$84k – $180k
One-time
- Embed the corpus once7.5B tokens$525 – $1.13k
Prices are placeholders, not quotes. Check current pricing and edit them in Assumptions. Ranges multiply point estimates by 0.7–1.5; excludes people, networking, storage of raw files and observability tooling. Cost per question at the low end: $0.84 per 1,000.
Parts catalogue: what to build it from
| Option | Why | Trade-off |
|---|---|---|
| pgvector (Postgres) | One database for rows, metadata and vectors; transactions and joins for ACLs. | HNSW indexes get slow and memory-hungry past ~10M vectors per table. |
| Pinecone / Qdrant / Weaviatefits your plan | Purpose-built ANN, metadata filters, namespaces, hybrid search; managed or self-hosted. | Another vendor and bill; managed tiers price per read/write units or pods. |
| Milvus / Vespa / sharded OpenSearch | Designed for billions of vectors across many nodes; GPU indexes (Milvus), rich ranking (Vespa). | Real operational weight: more services, tuning, and people who know them. |