Monitoring Embedding Drift in RAG Pipelines

ShopMax India uses a RAG pipeline to answer customer queries about products, pricing, and policies. Over time, the product catalog changes - new items are added, prices are updated, and discontinued products are removed. If user queries drift significantly from what the retriever was built on, retrieval quality degrades silently. Embedding drift monitoring detects this degradation before it impacts customers.

Drift is measured by comparing the cosine similarity distribution of current query embeddings against a baseline set recorded at deployment time. When the distribution shifts - average similarity drops or variance increases - it signals that user queries have changed or that the knowledge base no longer covers the topics users are asking about. ShopMax India computes this metric daily and alerts if average similarity drops by more than 10 percent from baseline.

The example below establishes a baseline similarity distribution at deployment and detects drift when a new batch of queries is evaluated against it.

It gives the following output,

Average max similarity to baseline: 0.3812
DRIFT DETECTED: Query distribution has shifted significantly.
Action: Review knowledge base coverage for new query types.

Run drift checks daily using a scheduled job and store the average similarity score in a time-series database to track gradual trends. When drift is detected, sample the low-similarity queries and manually review whether the knowledge base covers those topics. Use the drifted query clusters to prioritise new document ingestion into the RAG pipeline. Set a secondary alert if drift persists for more than three consecutive days without remediation.