/ THE SHORT ANSWER
- 01Test retrieval separately from answer fluency.
- 02Include conflicting, stale, absent, and permissioned cases.
- 03Price the ongoing content and evaluation operation, not only the software.
/ dotSuper point of view
RAG quality is an operating property of sources, retrieval, permissions, evaluation, and ownership. A fluent demo can hide weakness in every one of those layers.
What the evidence says
OWASP’s LLM application guidance highlights risks including prompt injection, sensitive-information disclosure, vector and embedding weaknesses, excessive agency, and unbounded consumption.
A practical decision framework
It is designed to make the decision inspectable, not to imitate a platform ranking formula, certification checklist, or legal test.
- Sources: provenance, versioning, deletion, freshness, and ownership.
- Retrieval: relevance, coverage, conflict handling, and permission enforcement.
- Generation: grounded answers, citations, refusal, and uncertainty.
- Operations: evaluation set, monitoring, change control, cost, and support.
| Step | Decision to record |
|---|---|
| 01 | Sources: provenance, versioning, deletion, freshness, and ownership. |
| 02 | Retrieval: relevance, coverage, conflict handling, and permission enforcement. |
| 03 | Generation: grounded answers, citations, refusal, and uncertainty. |
| 04 | Operations: evaluation set, monitoring, change control, cost, and support. |
How to put it into practice
Include straightforward, multi-source, ambiguous, unanswerable, stale, and restricted questions, then score retrieval and final answers separately.
Ask the vendor to demonstrate an update, a permission change, a source deletion, a model change, and a rollback.
These lifecycle actions determine whether the system can be safely owned.
- Name the accountable owner and the decision this work must enable.
- Record the current evidence, assumptions, exclusions, and next review trigger.
- Measure a useful outcome rather than treating publication or deployment as success.
What this page cannot conclude
- 01A short acceptance set cannot predict every production interaction.
- 02Sector-specific privacy, records, security, and residency requirements need separate review.
- 03Publication, technical eligibility, or good practice cannot guarantee ranking, referral traffic, citation, adoption, or a business outcome.
Sources
- 01Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence ProfileNational Institute of Standards and Technology · accessed Aug 30, 2026
- 02OWASP Top 10 for LLM Applications 2025OWASP GenAI Security Project · accessed Aug 30, 2026
Our editorial standard · Found an error? Send a correction with its source.
/ CITE OR SHARE THIS GUIDE
Make the evidence easy to verify.
When you reference this guide, link to its canonical URL. That gives readers one stable place for the evidence, limitations and future updates.
dotSuper Research Desk. (August 30, 2026). How to Evaluate a RAG Vendor: The SME Buyer Scorecard. dotSuper. https://dotsuper.net/feeds/market-intelligence/rag-vendor-evaluation-for-smes