What is an AI evaluation framework for production systems?

Where should AI evaluation start?

Every AI evaluation should start from the decision the system serves, not from the model. A recommendation system, a document router, a fraud flag, a customer answer, a stock level: each exists to improve a specific, named decision, and that decision defines what better means. Write the decision down before choosing a model: That paragraph becomes the standard every later measure answers to. When the evaluation set, the offline metrics, and the live dashboards all trace back to a named decision, disagreements become checkable instead of political. The reverse pattern is common and expensive: a model is chosen, a benchmark is found, and the benchmark becomes the goal. Teams optimize the metric for quarters while the decision it was meant to serve stays exactly where it was.

Who makes this decision today, and what information do they use?
What does a good decision look like, in the terms the business already tracks?
What would make the result worse in a way the business would notice first?

Why does a baseline matter more than a benchmark?

A baseline matters more than a benchmark because it prices sophistication in the currency of the actual decision. A lexical retrieval baseline, a rules engine, a straightforward regression, or the current human process gives the floor that any learned system must beat on the same cases, the same latency budget, and the same cost boundary. The baseline does three jobs: It prices sophistication. A more complex model is worth its operating cost only if it beats the baseline by enough to matter on the decision, not on the leaderboard. It localizes failures. When the complex system regresses, running the same cases through the baseline shows whether the world changed or the model did. It keeps the upgrade conversation honest. The question is never whether a larger model scores better in isolation. The question is whether it beats this baseline, on these slices, at this latency, for this cost per decision. On retrieval work we start with a TF-IDF baseline on the real question log for exactly this reason. It is fast, inspectable, and cheap to evaluate, and every later retriever has to beat it on the same queries and judgments.

What should you measure after an AI system goes live?

After go-live, measure outcomes tied to the decision, performance on the slices that carry risk or margin, latency and cost per decision at the expected traffic shape, and a drift signal that says when live inputs have moved away from the evaluation set. Offline measures on a curated set are where evaluation starts, not where it ends. Concretely: A dashboard nobody is accountable for is a museum, not a control. Name an owner for each measure.

Outcome measures tied to the decision. Acceptance rate on routed documents, margin on recommended categories, resolution without escalation, forecast error at the shelf and category levels where planners act.
Slice measures for the segments that carry risk or margin. Customer types, languages, document categories, store formats, new versus returning users. Aggregate scores hide exactly the slices that matter.
Latency and cost per decision at projected traffic. A system that meets its latency target at 10 requests per minute and collapses at 200 is not evaluated, it is demoed.
A drift signal with a named recipient. Define the statistics that will tell you live traffic has walked away from the evaluation set, and who receives that alarm.

How do you keep an evaluation set honest over time?

Keep an evaluation set honest by versioning it like code, sampling real traffic into it on a schedule, adding each live failure as a case, and measuring how much human judges agree on the labels. Below reasonable agreement, the metric measures the labeling instructions, not the system. The practices that matter most: Quarterly, review the set for staleness: retire questions customers no longer ask, add new intents with agreed judgments, and record what changed.

Hold out a live-traffic slice. Sample current requests, judge them, and score the system on them. Review sampling, labels, instrumentation, and user behavior with the slice so it remains a useful check on current use.
Track judgment agreement. Where humans label gold answers, report inter-rater agreement alongside the score. When agreement drops, fix the instructions before blaming the model.

What failure modes belong in the evaluation plan?

The failure modes that belong in the plan are the low-confidence fallback path, the user-visible behavior when the system is unsure, the logging that makes a failure reproducible, the escalation route to a human, and the tested rollback. Each is a requirement with cases in the evaluation set, not a paragraph in a design document. The happy path is the easy half of the corpus. The other half is what the operating team will remember about the system.

What is the fallback path when confidence is low, and what does the user see?
What is logged so a failure can be reproduced and fixed?
What escapes to a human, and how fast?
How is the system rolled back, and who can do it at 2 a.m.?

What gates should an AI change pass before release?

A change should reach users only after it beats the baseline on decision measures, shows no slice regressing beyond tolerance, stays within latency and cost budgets at projected traffic, adds its new failure cases to the evaluation set, and documents a tested rollback. These gates turn the model registry into the control point of the system. Beats the baseline on the decision measures. No slice regresses beyond its agreed tolerance. Latency and cost per decision within budget at projected traffic. New failure cases from this change are in the evaluation set. Rollback tested and documented. Every deployed version points at the evaluation report that justified it, and any auditor, executive, or new team member can trace why the live system is the live system.

How much should evaluation cost relative to the model?

Evaluation should be budgeted as a first-class workstream alongside the model and the pipeline: the baseline, the slice measures, the live-data collection path, the review cadence, and the failure corpus. Systems remain useful when their teams can show, with evidence the operating business accepts, that the decision improved. Everything else is a demonstration, and demonstrations are eventually switched off.

How does BluePi apply this framework in delivery?

BluePi delivers AI systems with evaluation designed alongside the system: decision framing, baseline, slice measures, drift signals, and release gates, owned with the client team so the measures survive the handover. The evaluation and measurement capability page describes the practice, and the guides on retrieval baselines and hybrid recommendation evaluation show the framework applied to specific system types.

Where to read next on the same subject.