How to Monitor a RAG Pipeline in Production
A RAG pipeline is a multi-stage system where a failure at any stage can produce a wrong answer, and the stage that failed is rarely obvious from the final output alone. The retriever might return irrelevant documents, the prompt template might inject context in a way the model ignores, the model might hallucinate beyond the retrieved context, or a re-ranker might filter out the most relevant result. Without stage-by-stage monitoring, debugging is guesswork and regressions go undetected for days or weeks. The approach below instruments each stage independently so that a quality drop localizes to the responsible component automatically.
Step 1: Instrument Each Pipeline Stage as a Span
The foundation of RAG monitoring is creating a separate observability span for each stage of the pipeline. A typical RAG pipeline has five stages, and each should be its own span within the request trace.
The query processing span captures query transformation: the raw user query, any query rewriting or expansion that happens before retrieval, and the generation of the query embedding. Record the original query text, the rewritten query if applicable, the embedding model and version, the embedding latency, and the vector dimensions.
The retrieval span captures the search operation. Record the collection or index searched, the query vector or text, the top-k parameter, any metadata filters applied, the number of results returned, the similarity scores for each result, the document IDs or titles of retrieved results, and the retrieval latency. If you use hybrid search (combining vector similarity with keyword matching), record both the vector and keyword queries and the fusion method (reciprocal rank fusion, weighted combination).
The re-ranking span, if your pipeline includes a re-ranker, captures the re-ordering of retrieved results. Record the re-ranking model, the input documents and their original scores, the output documents and their re-ranked scores, and the re-ranking latency. This span is critical for debugging because a re-ranker can promote or demote relevant documents, and without visibility into its decisions, re-ranker bugs are extremely difficult to diagnose.
The prompt assembly span captures how retrieved context is injected into the prompt template. Record the template name or version, the number of context chunks injected, the total token count of the assembled prompt, and any truncation that occurred (if the context exceeded the model's context window and was trimmed). Truncation is a common source of quality regression that is invisible without this span.
The generation span captures the LLM call. Record the model name and version, the full assembled prompt (or a reference to it), the completion text, input and output token counts, generation parameters, finish reason, time-to-first-token, and total generation latency.
Step 2: Track Retrieval Quality Metrics
Retrieval quality determines the ceiling of your RAG system's output quality, because the model cannot produce a correct answer if the relevant information was not retrieved. The following metrics should be tracked continuously from your retrieval spans.
Retrieval hit rate measures the percentage of queries where at least one retrieved document is above your relevance threshold. A healthy RAG system typically has a hit rate of 85 to 95 percent. A drop in hit rate indicates that user queries are drifting away from what the index covers, that the index was not updated after a content change, or that the embedding model was updated and the index needs re-embedding.
Average similarity score tracks the mean similarity score of the top-k retrieved documents over time. A declining average indicates embedding drift (the query embeddings are shifting relative to the document embeddings), index staleness, or a change in query distribution. Track this as a daily rolling average and alert on drops greater than 5 percent from the 30-day baseline.
Empty result rate measures the percentage of queries that return zero results above threshold. This should be near zero for a well-maintained RAG system. A spike in empty results usually means the index is down, the embedding service failed, or a metadata filter is too restrictive.
Index freshness tracks the age of the newest document in the index. If your content is updated regularly (knowledge base articles, product documentation, policy documents), index freshness tells you whether the indexing pipeline is keeping up. An index that has not been updated in 48 hours when content changes daily is a data freshness problem that will cause incorrect answers before any other metric moves.
Retrieval latency percentiles (p50, p95, p99) track how long retrieval takes. Retrieval is often the second-largest latency contributor after generation, and latency spikes in retrieval usually indicate index performance problems (too many vectors, insufficient resources) or network issues between your application and the vector database.
Step 3: Monitor Generation Faithfulness
Faithfulness measures whether the model's response is grounded in the retrieved context rather than hallucinated. This is the most important quality metric for RAG systems because the entire purpose of RAG is to ground the model's output in authoritative sources, and a failure of faithfulness defeats that purpose.
Run an automated faithfulness evaluator on a sample of production traffic. The evaluator takes the retrieved context and the model's response and uses a separate LLM call to judge whether each claim in the response is supported by the context. Platforms like Langfuse, Arize Phoenix, and Ragas provide pre-built faithfulness evaluators. The evaluation runs asynchronously on sampled traces (10 to 25 percent of traffic is typical) and attaches a faithfulness score to each traced request.
Track the faithfulness score distribution as a daily metric. A healthy RAG system should have a median faithfulness score above 0.85 (on a 0 to 1 scale). Alert on two conditions: a drop in median faithfulness greater than 0.05 from the 14-day baseline (which indicates a systemic regression), and any individual trace with a faithfulness score below 0.5 (which indicates a severe hallucination that should be reviewed immediately).
Complement faithfulness with a relevance evaluator that checks whether the response actually answers the user's question. A response can be perfectly faithful to the retrieved context but completely irrelevant if the wrong documents were retrieved. Tracking both metrics separately lets you distinguish retrieval failures (low relevance, acceptable faithfulness) from generation failures (good relevance, low faithfulness).
Step 4: Track Cost and Latency per Stage
Break down cost and latency by pipeline stage rather than tracking them only at the request level. This breakdown is what makes optimization actionable, because it tells you which stage to target.
For cost, the primary driver is token usage in the generation span. Track input tokens (which include the system prompt, retrieved context, and user message) and output tokens separately, because they often have different per-token prices and different optimization strategies. Input token reduction comes from shortening prompts, reducing the number of retrieved chunks, or enabling prompt caching. Output token reduction comes from adjusting max_tokens or adding instructions that encourage concise responses.
For latency, create a stacked breakdown that shows how the total request time is distributed across embedding, retrieval, re-ranking (if applicable), prompt assembly, and generation. This breakdown immediately identifies the bottleneck. In most RAG systems, generation is the largest latency contributor (50 to 70 percent of total time), followed by retrieval (15 to 30 percent), with embedding and prompt assembly contributing the remainder. If retrieval is taking more than 30 percent of total time, investigate index performance. If generation pre-fill time is high, investigate prompt length.
Track cost and latency trends over time and correlate them with deploys and configuration changes. A cost spike that coincides with a deploy points to a prompt template change or a retrieval configuration change. A latency spike that coincides with an index update points to an indexing issue. These correlations are only possible when you have continuous, stage-by-stage monitoring.
Step 5: Set Alerts on Key Metrics
Alerts turn monitoring from a dashboard you check occasionally into an automated system that notifies you when something goes wrong. For RAG pipelines, set alerts on the following metrics.
Retrieval hit rate: alert when the 1-hour hit rate drops more than 10 percentage points below the 7-day average. This catches index failures, embedding service outages, and sudden query distribution shifts.
Faithfulness score: alert when the 6-hour rolling average faithfulness score drops more than 0.05 below the 14-day baseline. This catches model version changes, prompt template regressions, and retrieval quality degradations that manifest as hallucinations.
Cost per request: alert when the 1-hour average cost per request exceeds twice the 7-day average. This catches prompt template changes that dramatically increased token usage, retrieval configuration changes that inject too much context, and retry loops that make duplicate model calls.
P95 latency: alert when the 15-minute p95 latency exceeds 150 percent of the 7-day p95 baseline. This catches vector database performance issues, model provider latency spikes, and pipeline changes that added new sequential operations.
Error rate: alert when more than 1 percent of requests produce an error (model timeout, retrieval failure, content filter rejection) in any 15-minute window. This catches infrastructure failures and model provider outages.
Use a two-tier alerting strategy: warning alerts notify the team channel for investigation, and critical alerts page the on-call engineer. Retrieval hit rate and error rate alerts should be critical because they indicate the system is not functioning. Faithfulness and cost alerts should be warnings because they indicate degradation that needs investigation but does not necessarily mean the system is down.
Step 6: Feed Production Failures Back into Evaluation
The final step closes the loop between production monitoring and offline evaluation. When a production trace triggers an alert, receives negative user feedback, or scores below threshold on an automated evaluator, capture that trace and add its inputs and expected outputs to your offline evaluation dataset. This means the regression that slipped through your evaluation suite once will be caught automatically on every future change, because it is now part of the test set.
Over time, this feedback loop builds an evaluation dataset that reflects the real distribution of production traffic, including the edge cases and adversarial inputs that no manually curated dataset captures. Teams that run this loop for six months typically have evaluation datasets that are dramatically more comprehensive and representative than teams that build evaluation datasets from scratch, because production traffic is the most creative test generator available.
Implement this loop by tagging traces that meet the capture criteria (alert triggered, low evaluation score, negative user feedback), exporting them to your evaluation dataset on a scheduled basis, and including the evaluation dataset run as a gate in your CI/CD pipeline. This creates a system where production quality monitoring directly improves pre-deployment evaluation, which in turn reduces the number of regressions that reach production.
Monitor each RAG pipeline stage independently with its own metrics and alerts, because the whole-pipeline view only tells you something is wrong while the stage-level view tells you which component to fix. The minimum viable monitoring setup is retrieval hit rate, generation faithfulness, cost per request, and p95 latency, with alerts on each.