RAG in Production: Retrieval Quality Comes Before Cost

When RAG systems go to production, the most common problem you encounter is this: if retrieval quality is low, no amount of LLM power will help — the answer will be garbage. OpenAI's text-embedding-3-large costs $0.00013 per token, Cohere's embed-english-v3.0 costs $0.0001 — a 30% difference — but if you're retrieving the wrong chunks, the result is the same: hallucination. If you cut embedding costs while degrading retrieval quality, downstream LLM costs spike 200% (re-ranking, prompt padding, retries). This article shows how embedding selection, chunking, and evaluation setup are prioritized in production RAG pipelines.

Embedding Model Selection: Latency × Recall Matrix

When choosing an embedding model, two metrics are critical: retrieval recall@k (correct information in the first k chunks) and p99 latency. The difference between Ada v2 and text-embedding-3-small is not just cost — it's semantic granularity. If your domain is narrow with heavy terminology (law, finance), a fine-tuned Sentence-BERT variant (768 dims) often outperforms OpenAI's 1536-dim model on recall.

In production, here are the numbers we've seen: with text-embedding-3-large, you get a 64.6 retrieval score on MTEB benchmarks, but on your domain-specific eval set (e.g., e-commerce product documentation), it drops to 58.2. When we tested Cohere's embed-multilingual-v3.0 on Turkish content, recall@5 came in 12% higher — because Cohere invested more non-English corpus in multilingual training. There's no single metric: batch size 128 gives embedding latency of 230ms, but a single request takes 45ms. For real-time search, latency is the priority; for offline indexing, recall is.

In practice, we test like this: take your eval set (100–200 questions + ground truth chunks), index with 3 models, calculate recall@1/3/5 and MRR (mean reciprocal rank) for each. After picking the winner, decide whether fine-tuning is worth it — if recall@5 is below 75%, fine-tuning ROI is positive. Roibase's data analytics work includes the metric infrastructure needed to build this evaluation pipeline.

Chunking Strategy: Fixed vs Semantic vs Recursive

Chunk size is RAG's most critical hyperparameter. The difference between a 512-token chunk and a 2048-token chunk is this: small chunks deliver more specific retrieval but lose context; large chunks preserve context but add noise. And chunk overlap ratio (e.g., 10%) affects retrieval precision too.

Fixed-size chunking (cutting every 512 tokens) is the simplest approach, but cutting mid-paragraph breaks semantic coherence. Langchain's RecursiveCharacterTextSplitter works like this: first split by \n\n (paragraph), if too large, split by \n (line), if still too large, split by period. This method gives 18% better recall@3 because chunk boundaries follow natural text structure.

Semantic chunking goes one step further: you build chunks by looking at embedding similarity. For example, when a topic shift is detected in a document (cosine similarity drops below 0.6), a new chunk starts. LlamaIndex's SemanticSplitterNodeParser uses this method. In production, the tradeoff we see: semantic chunking increases indexing time by 40% (every sentence gets embedded) but improves retrieval quality by 9%.

Chunk Overlap: How Much Is Enough?

Overlap ratio is typically kept between 10–20%. A 512-token chunk with 50-token overlap means a sentence might appear in two chunks. As overlap increases, index size grows (storage cost), but edge-case retrieval quality improves. Our testing found 15% overlap is the sweet spot: anything more gives diminishing returns.

Overlap strategy matters too: sliding window (every chunk shifts by 50 tokens) or paragraph-aware overlap (overlap only at paragraph boundaries)? Paragraph-aware overlap produces 7% smaller index sizes while preserving the same retrieval quality.

Evaluation Setup: Offline Metrics Must Represent Production

The biggest trap in RAG evaluation is this: offline metrics look good, but you hit a hallucination wave in production. The reason is your eval set doesn't represent production query distribution. Our recommendation: pull 200 random queries from production logs and manually annotate ground truth chunks. This 4-hour job gives you correct guidance for 6 months.

Metrics that need measuring:

To measure context precision and faithfulness, we use LLM-as-judge: ask GPT-4o-mini "Is this chunk relevant to the question?" and get a 0–1 score. This method shows 89% correlation with human eval (in our internal testing) and costs 1/50th of human eval.

In production, you need continuous evaluation: every 1,000 queries, randomly sample 10 and run them through the eval pipeline; if you detect a recall drop, get an alert. This setup is easy to build with Prometheus + Grafana — retrieval latency, chunk count, and LLM token usage metrics on the same dashboard.

Hybrid Search: Combining Dense + Sparse Retrieval

Pure dense retrieval (embedding similarity alone) sometimes misses exact term matches. For example, when a user asks "Q3 2025 revenue," the chunk "third quarter 2025 earnings" is semantically close but has no exact terms — sparse retrieval like BM25 works better here. Hybrid search combines both: dense retrieval fetches top-50 chunks, sparse retrieval fetches top-50 chunks, and RRF (reciprocal rank fusion) merges them.

Vector DBs like Weaviate and Qdrant natively support hybrid search. In our testing, hybrid search gives 6% better recall@10 than pure dense but adds 18% latency (two separate index queries). In production, you can toggle hybrid search by query complexity: if the query is under 3 words, use sparse only; over 10 words, use dense only; in between, use hybrid.

The alpha parameter (dense vs sparse weight) varies by domain: in e-commerce, sparse is more important (product codes, SKUs); in technical documentation, dense is more important (conceptual similarity). Our default is alpha 0.7 (dense-weighted), but it should be optimized via A/B testing.

Re-Ranking: Precision Gains After Retrieval

The first retrieval fetches 50 chunks, but feeding all of them as context to the LLM is both expensive and noisy. A re-ranking model (like Cohere's rerank-english-v3.0) rescores those 50 chunks by query relevance and picks the top 5–10. The re-ranker's job is different: the embedding model measures general semantic similarity; the re-ranker measures query-chunk relevance.

In production, re-ranking gives 15% better context precision but adds 80ms latency. The tradeoff is this: if your downstream LLM cost is high (using GPT-4), re-ranking ROI is positive; if you're on GPT-4o-mini, the latency cost weighs heavier. In our setup, critical queries (SLA <500ms) skip re-ranking; analytical queries (dashboards, reports) use it.

Re-ranker selection also matters: Cohere's model is cross-encoder-based, higher latency but better accuracy. Jina AI's re-ranker is bi-encoder-based, lower latency but 4% lower accuracy. In production, test both and decide based on your latency/accuracy tradeoff.

Cost Profiling: Token Economics Start With Embedding

In a RAG pipeline, costs break down like this (average production case):

• Embedding: 8%
• Vector search: 2% (compute)
• Re-ranking: 5%
• LLM inference: 85%

Embedding cost looks small, but it compounds across large volumes during indexing. 1M documents, avg 1,000 tokens/doc, OpenAI text-embedding-3-large = 1B tokens = $130. If you reindex monthly (not incremental, full reindex), annual embedding cost is $1,560. Switch to Cohere and it's $1,200. That's 23% savings.

But here's the real cost: if retrieval quality is low, the LLM retries, pads context, corrects hallucinations — that's 200% token overhead. 1M queries/month, avg 2,000 tokens/query, GPT-4o at $10/1M tokens = $20K/month. Retrieval quality drops 10%, retry rate climbs 15%, cost jumps to $23K. You're trying to save $30 on embedding while losing $3K downstream.

That's why when you say "RAG in production," the first question should be: do you have an eval setup? If not, choosing an embedding model is premature. Roibase's first-party data architecture includes building the logging infrastructure that feeds this evaluation pipeline — production queries, retrieval results, LLM responses must be stored structurally so you can analyze them later.

Incremental Indexing: How to React to Changing Data

In production, the document set isn't static — new blog posts, product pages, documentation appear every day. Full reindexing is expensive and requires downtime. Incremental indexing works like this: you re-embed only the changed documents and add them to the vector DB. Qdrant and Pinecone natively support incremental inserts.

The challenge is this: when a document changes, do you update just that chunk or the entire document? If chunk boundaries shifted (new paragraph added, chunk size changed), you need to recalculate all the document's chunks. Our strategy: we version documents (hash), and if the version changes, we delete all chunks and re-add them. This approach does 3% extra reindexing but guarantees consistency.

Deletion strategy matters too: if you don't delete old chunks from the vector DB, the index gets polluted and relevance drops. But adding TTL to every chunk is overhead. Our solution: we add doc_id and version metadata to each chunk, and when a document updates, we bulk-delete the old version's chunks by doc_id + version. This takes 200ms in Qdrant, 450ms in Pinecone (for 10K chunks).

The most critical step in taking a RAG system to production is measuring and continuously monitoring retrieval quality beforehand. Embedding model selection, chunking strategy, evaluation setup — these aren't independent; they affect the whole pipeline. Cost optimization doesn't start with embedding; it starts with retrieval precision. A system that can't fetch the right chunk on the first try compounds in cost downstream.