RAG Regression Testing: What to Track Before Retrieval Changes

Before a retrieval change, define which tasks depend on freshness, exact citation, domain-specific phrasing, or long-tail coverage.

Keep representative queries and expected source behavior stable enough that a new retriever can be compared against the previous path.

A bad answer may come from missing documents, stale context, irrelevant chunks, broken ranking, context assembly, prompt policy, or the model's final response.

The eval should record where the failure first appears. Otherwise the team fixes the visible answer while the retrieval defect remains.

Release gates should name blockers such as stale retrieval on regulated content, missing source coverage for high-value tasks, or unsafe synthesis when context is incomplete.

Warnings can cover lower-risk drift that needs monitoring but should not block every release.