RAG EVALUATION FOR THE ENTERPRISE: FAITHFULNESS, ABSTENTION, AND THE METRICS THAT SURVIVE SECURITY REVIEW
How to evaluate retrieval-augmented generation in production: faithfulness, citation support, honest abstention, regression suites, and why 'accuracy' alone fails procurement.
Enterprise RAG evaluation means measuring whether answers are supported by retrieved sources, whether the system abstains when evidence is missing, and whether quality holds as the corpus changes - not whether a demo looked fluent on launch day. If you only track 'accuracy' on a static quiz, you will ship a system that invents confidently under distribution shift.
The three metrics that matter
- →Faithfulness: is every claim in the answer entailed by the retrieved passages?
- →Citation support: can a reviewer click from claim → source without archaeology?
- →Abstention quality: does the system say 'not found' when it should, instead of improvising?
Fluency and latency matter operationally, but they are not proof of truthfulness. Security and legal reviewers care about the first three - see also /products/knowledge and /solutions/ai-automation/rag-systems.
Build an eval set from real failures
Start with questions your users already ask, plus adversarial ones: missing documents, conflicting sources, out-of-scope topics, and near-miss retrievals. Label expected behavior: answer-with-citations, refuse, or escalate. Re-run the suite on every corpus or prompt change - that regression habit is the difference between a pilot and a product.
Retrieval quality before generation quality
Most 'bad answers' are bad retrieval wearing a fluent coat. Measure hit-rate@k, MRR, and section diversity before you tune the LLM. Hybrid lexical + vector retrieval (as in Symbiosis Knowledge) usually beats either alone on enterprise corpora with IDs, acronyms, and policy codes.
What 'good enough' looks like for go-live
Agree thresholds with the business owner before the pilot: e.g. faithfulness ≥ X on the golden set, abstention on known-missing queries ≥ Y, and zero critical fabrications on a red-team set. Publish the harness with the numbers - the same honesty rule we use on /resources/benchmarks.
If you can't reproduce the score, you don't have a score - you have a slide.