Start with the business decision
There is no universal AI quality score. Document search needs evidence coverage and citation accuracy; classification needs per-class errors; an operator assistant needs usefulness and handling time; an agent also needs correct tool selection and safe action. Define critical scenarios, the cost of failure and a minimum acceptable result.
Build a representative reference set
Include common requests, difficult edge cases, unsafe requests, real user phrasing and different document formats. For each item, record expected facts, acceptable alternatives, required sources and prohibited actions. Split data into a development set and a hidden control set so prompt tuning does not simply memorize the test.
Evaluate the system in layers
For RAG, score retrieval separately from the final answer. For agents, inspect the plan, chosen tool, call arguments and outcome. Deterministic checks work well for structure, required facts, citations and forbidden actions; expert review remains necessary for usefulness, completeness and professional judgment. Also measure latency, cost, failures and behavior under concurrent load.
Establish release gates
Run every model, prompt, index, tool or parameter change against the same versioned suite. Compare it with the current production configuration. A higher average must not hide a failure in a critical scenario, which may require a perfect pass rate. Pin model and configuration versions and retain results with the release record.
Keep learning after launch
Collect privacy-safe failure examples from production, review them and add durable cases to the suite. Report results by scenario and error type rather than as one attractive number. Business owners accept usefulness, security owners accept constraints, and engineering accepts reproducibility, performance and rollback. That turns AI change into a controlled release process.