When moving RAG systems to production, the first question is usually "which embedding model—because token cost?" Wrong question. The right one: "if retrieval precision drops below 0.85, what percentage of user queries turn into hallucinations?" RAG's cost structure isn't like batch inference—poor retrieval creates exponential token waste downstream and erodes user trust. Embedding model selection, chunking strategy, and evaluation setup must be approached through this lens.

Embedding model: Latent space quality outweighs cost/token metrics

When selecting an embedding model, prioritize metrics in this order: retrieval precision → semantic drift → latency → cost/token. OpenAI's text-embedding-3-large has 3072 dimensions, Cohere's embed-v3 1024, Voyage AI's voyage-2 1536—these numbers determine latent space granularity. But the real difference isn't in benchmarks; it's in domain-specific query behavior. On an e-commerce platform, the query "black leather jacket size M" caused text-embedding-3-large to generate 12% more false positives because it encoded "leather" more as a style than a material. Voyage AI's domain fine-tuning option becomes relevant here—fine-tuning with 5,000 query-document pairs over 2 weeks increased precision by 18% over baseline.

Cost math works like this: text-embedding-3-large costs $0.13 per 1M tokens, Cohere $0.10. But low precision means wrong context goes to the LLM—GPT-4o costs $0.30 per 10K tokens, and poor retrieval wastes 3K extra tokens per query = $0.09 extra. At 100K queries/month, that's $9K waste. Saving $30 on embedding while losing $9K downstream is irrational. Latency shows a similar pattern: Cohere runs at 45ms, Voyage at 62ms—but Voyage's retrieval quality reduces reranking overhead by 40%, bringing total pipeline latency from 180ms to 140ms.

Track semantic drift with temporal queries in your evaluation set. Run the same user query 3 months apart and compare retrieved document sets. Drift above 15% signals that the embedding model faces production concept drift—retrain or switch needed. Without this tracking, embedding selection becomes a blind decision.

Chunking strategy: The fixed-size fallacy and overlap tradeoffs

The most common mistake: 512-token fixed-size chunks + 50-token overlap. This naive approach ignores semantic boundaries. It splits markdown headings, code blocks, and tables mid-stream, creating context loss in retrieval. Alternative: semantic chunking—use sentence embeddings to set dynamic chunk boundaries based on semantic similarity threshold (e.g., cosine 0.75). LangChain's SemanticChunker does this, but it adds 30% latency overhead. For pipeline-critical latency constraints, a hybrid approach—recursive character splitting + heading-aware parsing—is more pragmatic.

The overlap tradeoff: 0% overlap = information loss at chunk boundaries; 50% overlap = 1.5x index size + 25% query latency increase. The sweet spot varies by domain. For technical documentation, 25% overlap (128 tokens @ 512-token chunks) works; for conversational data, 10% (50 tokens) suffices. Test with a "chunk boundary query" subset in your evaluation set—questions whose answers span two chunks. How does increased overlap affect retrieval precision on these? In our testing, 25% overlap raised precision from 0.68 to 0.81 on boundary queries. Pushing to 50% gained 0.83, but the 2% improvement didn't justify the latency penalty.

Chunk size isn't binary either. 256-token chunks enable more granular retrieval; 1024-token chunks provide more context per chunk. But when the LLM context window fills: 1024-token chunks = 4 chunks × 4K tokens; 256-token chunks = 16 chunks × 4K tokens—same context, but 256-token chunking offers 4× more semantic options. The tradeoff: 4× embedding cost, but better retrieval diversity. In production, use a hybrid: FAQ and short-form content get 256-token chunks, long-form articles get 768. Tracking chunk performance via data analytics infrastructure helps determine which chunk size performs best for which query type.

Chunk metadata: JSON field injection

Injecting metadata into every chunk is critical for retrieval filtering. Fields like {category, created_at, author, content_type} enable metadata filters alongside vector search. Example: querying "Python tutorials from 2025" applies both semantic matching and the created_at > 2025-01-01 filter. This hybrid approach increased retrieval precision by 22%. Pinecone, Weaviate, and Qdrant all support metadata filtering, but query syntax differs—using an abstraction layer like LlamaIndex provides flexibility.

Evaluation setup: Offline metrics don't predict production hallucinations

For RAG evaluation, offline metrics include retrieval precision, recall, MRR (mean reciprocal rank), and NDCG. Necessary, but insufficient. In production, the real problem is: retrieved context is correct, yet the LLM still hallucinates. End-to-end evaluation fixes this—comparing retrieved chunks + LLM response against ground truth. The Ragas framework does this with LLM-as-judge metrics like faithfulness, answer relevance, and context precision. We use GPT-4o as the judge and run batch evaluations—1,000 queries evaluated in 24 hours.

Composition of the evaluation set: 60% real user queries (from production logs), 20% edge cases (deliberately ambiguous), 20% adversarial (outdated information, deprecated docs). Real user queries reflect production distribution. Edge cases test the model's uncertainty handling. The adversarial set simulates temporal drift—a 2023 query against 2026 docs should include an "outdated" warning in the response.

For continuous evaluation, add 200 new queries to the evaluation set every sprint (2 weeks). These come from production logs (random sample) plus curated edge cases. We A/B test model, chunking, and retrieval config changes against this set. Precision drop above 5% triggers a rollback. The evaluation pipeline runs on AWS Step Functions—embedding, retrieval, LLM inference, scoring, Slack alerts. Total runtime: 45 minutes, cost: $12 per run. Pushing RAG changes to production without this is blind deployment.

Reranking and query expansion: Neglected layers of the retrieval pipeline

Vector search alone isn't enough. After top-K retrieval (e.g., K=20), a reranking model (Cohere Rerank, bge-reranker) sorts by semantic relevance and passes the final K=5 to the LLM, improving retrieval precision by 30%. Reranking adds 80ms latency, but by preventing wrong context from reaching the LLM, overall pipeline reliability increases. Cost: Cohere Rerank runs $1 per 1K queries—$100/month at 100K queries—but reduced downstream LLM waste from $9K to $3K.

Query expansion: A simple user query like "how to set up RAG" should also match "retrieval-augmented generation implementation" in semantic space. HyDE (hypothetical document embedding) does this: the LLM writes an ideal response to the query, embeds it, and searches with that embedding. This provides implicit query expansion. Production testing showed 15% precision gain but 120ms latency cost. For latency-critical systems, classical query expansion (synonym injection) achieves similar gains in 40ms.

Production monitoring: Retrieval quality can't be optimized if unobservable

In a RAG system, monitor: retrieval latency p50/p95/p99, embedding cache hit rate, retrieved chunk relevance score distribution, LLM faithfulness score (computed via LLM-as-judge), and user feedback (thumbs up/down). Push these to Datadog as custom metrics. If retrieval latency p95 exceeds 200ms, alert—because total user-facing latency has a 500ms SLA, and retrieval above 200ms plus LLM inference breaches it.

Retrieved chunk relevance scores: Log the cosine similarity scores of the top-5 chunks per retrieval. Distribution shift (e.g., mean score dropping from 0.78 to 0.65) signals embedding model drift or corpus quality issues. Tracking this within your first-party data architecture enables proactive retrieval quality management.

When cost truly matters: What to do

After stabilizing retrieval quality, if cost optimization is next: (1) Embedding cache—repeat queries return cached results (6-hour TTL). Hit rate reached 40%, cutting embedding cost by 40%. (2) Quantized embeddings—use int8 instead of float32; index size dropped 75%, retrieval precision loss was 2%—acceptable. (3) Hybrid search—combine sparse (BM25) + dense (vector); sparse costs 70% less. Route queries via a classifier: 30% sparse, 70% vector—total cost dropped 20%.

These cost optimizations only work after retrieval quality stabilizes. Otherwise, blind cost cuts increase downstream LLM waste and raise net costs. RAG economics: embedding $500/month, retrieval infrastructure $1,200/month, LLM inference $8,000/month. Cutting $100 from embedding while tanking retrieval quality and adding $2,000 to LLM waste is irrational. But quantizing embeddings and saving $125 while adding only $50 in LLM waste—when retrieval precision is locked at 90%—is rational.

Production RAG systems are becoming critical in marketing automation, customer support, and content generation. But all depend on retrieval quality—poor retrieval tanks the trustworthiness of AI output to zero. Setting up embedding models, chunking, evaluation, and monitoring correctly before chasing cost savings is the only rational path. Start now: measure retrieval precision in your current RAG pipeline; if you don't have it, add it. Then consider cost.