RAGRouter: Learning to Route Queries to Multiple Retrieval-Augmented Language Models

Fan Wu (University of Illinois at Urbana-Champaign) · Chaoyue Niu (Shanghai Jiao Tong University) · Guihai Chen (Shanghai Jiao Tong University) · Jiarui Zhang (SUN YAT-SEN UNIVERSITY) · Xiangyu Liu (WeChat, Tencent) · Yong Hu (WeChat)

contrastive learningdedicated router modeldiverse knowledge-intensive tasksdocument embeddingsexternal documentsinformed routing decisionsintelligent routing mechanismsknowledge representation shiftslow-latency constraintsperformance-efficiency trade-offsrag capability embeddingsragrouterretrieval-augmented generationretrieval-augmented llm routing problemstatic parametric knowledge representations

Retrieval-Augmented Generation (RAG) significantly improves the performance of Large Language Models (LLMs) on knowledge-intensive tasks. However, varying response quality across LLMs under RAG necessitates intelligent routing mechanisms, which select the most suitable model for each query from multiple retrieval-augmented LLMs via a dedicated router model. We observe that external documents dynamically affect LLMs' ability to answer queries, while existing routing methods, which rely on static parametric knowledge representations, exhibit suboptimal performance in RAG scenarios. To address this, we formally define the new retrieval-augmented LLM routing problem, incorporating the influence of retrieved documents into the routing framework. We propose RAGRouter, a RAG-aware routing design, which leverages document embeddings and RAG capability embeddings with contrastive learning to capture knowledge representation shifts and enable informed routing decisions. Extensive experiments on diverse knowledge-intensive tasks and retrieval settings, covering open and closed-source LLMs, show that RAGRouter outperforms the best individual LLM and existing routing methods. With an extended score-threshold-based mechanism, it also achieves strong performance-efficiency trade-offs under low-latency constraints. The code and data are available at https://github.com/OwwO99/RAGRouter.