Query Routing for Retrieval Augmented Generation

The Problem with General Retrieval

CHUD began with a single retrieval pipeline. Every user query went through the same vector search process against the document index. This worked for simple factual questions. It failed when users asked synthesis-based questions requiring broad context. A narrow vector search often misses the relevant documents needed for high-level summaries. We were treating every intent as a lookup task. This led to incomplete answers and unnecessary token usage.

Implementing Intent Classification

We implemented a lightweight classification layer between the user input and the document database. This layer categorizes the intent of a query before any search is executed. We identified three primary categories: factual lookup, analytical synthesis, and procedural instruction. Factual lookups rely on standard vector similarity. Synthesis queries trigger a multi-pass retrieval that aggregates chunks from disparate sections of the library. Procedural instructions bypass the vector database entirely to pull relevant system documentation. This ensures that the context provided to the model is optimized for the specific task at hand.

Managing Routing Latency

Adding a classification step introduces a latency overhead. We optimized this by training a small, distilled model to act as the router. This model sits on the edge and finishes its inference in under ten milliseconds. By keeping the model size minimal, we avoid the bottleneck that would occur with a larger, more general-purpose model. The router does not generate text. It simply outputs a label that dictates the search path. The search path then initiates the appropriate retrieval function immediately. This approach allows us to maintain the snappy feel of the interface while increasing the relevance of the underlying data.

Tradeoffs in Retrieval Precision

This architecture is not perfect. It introduces a risk of misclassification. If the router misidentifies a synthesis query as a factual lookup, the model receives insufficient context. We monitor these edge cases by analyzing queries that receive a low confidence score from the router. These queries are flagged for manual review to help tune the classification weights. We prioritize precision over recall for the factual path to prevent hallucination. For synthesis tasks, we favor recall by widening the retrieval window if the router indicates ambiguity. The result is a system that adapts its retrieval strategy based on what the student is actually trying to accomplish.