RAG Query Routing: Send Queries to the Right Source

Why Single-Index RAG Breaks Down

The simplest RAG architecture has one vector index containing all documents, and every query searches that single index. This works when the knowledge base is homogeneous: all product documentation, all support articles, or all internal wiki pages. It breaks down when the knowledge base spans multiple types of information with different retrieval needs.

Consider a customer support RAG system that needs to answer questions about product features (documentation), billing (database records), service status (real-time API), and company policies (policy documents). Putting everything in one vector index creates three problems. First, the embedding model cannot equally represent such diverse content, so semantic similarity scores become unreliable across content types. Second, the index grows large enough that retrieval latency and relevance both degrade. Third, some questions cannot be answered by vector search at all, a billing question like "how much did I spend last month?" requires a SQL query against the billing database, not a similarity search against document embeddings.

Query routing solves this by adding a classification step before retrieval. The router analyzes the incoming query and decides which retrieval source (or combination of sources) to use. This keeps each index focused on its content type, uses the right retrieval method for each question type, and avoids wasting compute on indexes that cannot possibly contain the answer.

Classifier-Based Routing

The fastest and most predictable routing approach uses a trained classifier that maps queries to predefined route categories. The classifier is typically a small model (a fine-tuned BERT variant, a logistic regression on sentence embeddings, or even a simple keyword matcher) that takes the query text as input and outputs the route category with a confidence score.

To build a classifier-based router: define your route categories (one per retrieval source), collect 100 to 500 example queries per category, train a classifier, and deploy it as the first step in the retrieval pipeline. The classifier adds 5 to 20 milliseconds of latency, which is negligible compared to the retrieval and generation steps.

The strengths of classifier-based routing are speed, cost, and determinism. The router runs locally, costs nothing per query (no LLM call), and produces consistent results for similar queries. The weakness is rigidity: adding a new route category requires collecting training examples and retraining the classifier, and edge cases that do not cleanly fit any category get misrouted. For systems with 3 to 8 stable route categories, classifier-based routing is the production standard.

A practical implementation using sentence embeddings: embed 50 to 100 example queries per category using your text embedding model, store the embeddings with their category labels, and at query time, embed the incoming query and find the nearest neighbor among the example embeddings. The category of the nearest neighbor is the predicted route. This approach does not require training a separate model, uses the embedding model you already have, and can be updated by adding new examples without retraining.

LLM-Based Routing

LLM-based routing uses a language model to classify the query by interpreting a system prompt that describes the available routes and their intended content. The prompt typically lists each route with a description and examples, and the model outputs the route name (or a JSON object with the route and confidence).

The advantage is flexibility: adding a new route is as simple as adding a line to the system prompt, and the LLM can handle ambiguous queries by reasoning about which route best matches the user's intent. The model can also decompose complex queries that span multiple routes, returning a list of routes to search in parallel rather than picking one.

The disadvantage is cost and latency. An LLM routing call adds 200 to 800 milliseconds and costs $0.001 to $0.005 per query depending on the model. For high-volume systems serving thousands of queries per minute, this cost and latency are unacceptable for a routing step. The solution is to use a small, fast model (GPT-4o mini at $0.15 per million input tokens, or Claude 3.5 Haiku) dedicated to routing, not the expensive model used for generation.

A hybrid approach combines both: use the classifier for queries where it has high confidence (above 0.85), and fall back to LLM-based routing for queries where the classifier is uncertain. This gives you the speed and cost of classifier routing for the 80% of queries that are straightforward, and the flexibility of LLM routing for the 20% that are ambiguous or complex.

Multi-Index Architecture

The retrieval sources that a router dispatches to are not limited to vector indexes. A production RAG system typically includes several source types, each with its own retrieval mechanism.

Vector indexes handle semantic search over unstructured text. You might have separate indexes for product documentation, support articles, engineering specs, and internal communications, each with chunking and embedding strategies optimized for that content type.

SQL and structured databases handle questions about specific records, aggregations, and filtered lookups. A question like "how many orders shipped last week?" requires a SQL query, not a vector search. The routing step identifies these queries and dispatches them to a text-to-SQL pipeline that generates and executes the appropriate database query.

API endpoints handle real-time information that is not stored in your knowledge base. Service status, live inventory, current weather, stock prices, and other dynamic data come from API calls that the routing step triggers when the query requires up-to-the-minute information.

Knowledge graphs handle relationship queries that vector search struggles with. "What teams report to the VP of Engineering?" and "Which products use the same supplier?" are relationship traversals that a graph database answers directly. The router identifies graph-shaped queries by detecting entity names and relationship keywords.

The router's job is to map each query to one or more of these sources and merge the results into a unified context for the LLM. When multiple sources are selected, the retrieval runs in parallel and the results are interleaved with source labels so the LLM knows where each piece of context came from.

Query Decomposition and Multi-Hop Routing

Some queries cannot be answered by a single retrieval source because they require combining information from multiple sources. "Compare the pricing of product X in our database with what competitors charge according to our market research docs" needs both a SQL lookup and a document search. Simple routing picks one source; decomposition splits the query into sub-queries that each get routed independently.

The decomposition step typically uses an LLM to analyze the original query and produce a list of sub-queries, each tagged with the source it should target. The sub-queries are executed in parallel, their results are assembled into a combined context, and the original query is answered using the full context. This is the foundation of agentic RAG, where an agent orchestrates multiple retrieval steps to answer complex questions.

Multi-hop routing extends decomposition to sequential queries where the result of one retrieval informs the next. "Find all customers who purchased product X and check if any of them filed support tickets in the last 30 days" requires first querying the sales database for customer IDs, then searching the support ticket index filtered by those IDs. The router must recognize this sequential dependency and execute the retrievals in order rather than in parallel.

Implementing a Production Router

A practical production router combines query classification with fallback logic and monitoring. The implementation has five components.

Route definitions: Each route specifies a name, a description, a retrieval function, and example queries. The description is used by LLM-based routing, and the example queries are used by classifier-based routing.

The classifier: A lightweight model or embedding similarity lookup that maps queries to routes with confidence scores. Train on the example queries and update monthly as new query patterns emerge from production traffic.

Fallback logic: When the classifier confidence is below the threshold (typically 0.7 to 0.85), fall back to LLM-based routing. When LLM routing also fails to produce a clear route (the model outputs multiple routes with similar confidence), search all relevant indexes in parallel and let the reranker sort out relevance.

Monitoring: Log every routing decision with the query text, the selected route, the confidence score, and whether the user found the answer helpful (if you have feedback signals). Queries that consistently get routed to the wrong source reveal gaps in the training data or missing route categories.

Self-correction: When the retrieval results from the selected route are poor (low similarity scores, empty results), the system can automatically re-route to the next most likely source rather than returning a bad answer. This retry logic catches routing errors without requiring the user to rephrase the question.

Routing Signals Beyond Query Text

The query text is the primary routing signal, but production systems benefit from additional context. User role and permissions determine which sources the user is allowed to access, which narrows the routing options. Conversation history provides context for ambiguous follow-up questions: if the previous three questions were about billing, a follow-up "and what about last year?" should route to the billing database, not the documentation index. The application context (which page or feature the user is currently viewing) provides intent signals that disambiguate queries.

These additional signals can be incorporated as features in the classifier or as context in the LLM routing prompt. The key is to use them to disambiguate, not to override: if the query text clearly indicates a documentation search, the user's current page should not redirect it to the billing database just because the user happens to be on the billing page.