Build a release decision from versioned evidence

Ask what the pass rate leaves out

A hypothetical release review receives a dashboard showing a high pass rate. Before discussing the number, ask what entered its denominator. Were timed-out runs counted? Were failed setups discarded? Did multiple retries become separate successes? Were high-risk actions exercised, or mostly read-only conversations? Different answers can produce the same headline while supporting very different decisions.

Report at least the planned scenarios, attempted executions, completed executions, passes, failures and inconclusive outcomes. Distinguish setup errors from behavioral failures without making either disappear. State how retries and repeated runs are counted. Keep unique scenario coverage separate from execution volume: rerunning one easy scenario does not cover a new permission boundary.

Define acceptance criteria before examining a candidate's results. Otherwise teams can unintentionally select a favorable denominator after the fact. This is not a demand for one universal metric. It is a requirement that the release decision can be reconstructed from a stable definition rather than a persuasive chart.

Map evidence to the authority being released

Coverage should follow the permitted actions and their consequences. Identify workflow states, actor roles, tool operations, policy exceptions and dependency failures. Then connect each release-critical risk to scenarios and checks. A simple matrix exposes gaps more honestly than a composite readiness score.

Record untested cells explicitly. If a candidate lacks evidence for autonomous refunds, a read-only release may still be considered under separately reviewed controls. Coverage is relative to scope, not proof that all possible behaviors have been explored. Pairwise combinations can reduce test cost, but they may miss higher-order interactions around a critical transition.

Version the evidence, not just the model

A model identifier alone does not identify the evaluated system. Prompts, tool adapters, schemas, authorization rules, retrieval sources and orchestration can all change behavior. Preserve an evidence manifest that connects a candidate to the exact environment and checks used to assess it.

Illustrative evidence manifest fields:
candidate: model, prompt, orchestration, tool versions
boundary: permissions, policy, schema, dependency contracts
inputs: scenario revision, snapshot ID, fault schedule
checks: invariant definitions, evaluator version, rubric
results: attempted, completed, failed, inconclusive
records: run IDs, trace locations, state-diff locations
decision: scope, exceptions, approvers, invalidation triggers

Use immutable identifiers or content digests where practical, and restrict access to sensitive records. OpenTelemetry provides trace and span context for connecting work across a request path [1]. Those identifiers can help join execution records, but a trace does not supply the manifest automatically or prove a business invariant. Retain the authoritative state evidence separately, with provenance sufficient to explain how a verdict was produced.

Keep uncertainty visible during comparison

Compare a candidate and its baseline on the same scenario revisions and starting conditions where feasible. Report changed outcomes by risk family, not only their net total. Fixing several harmless wording failures does not cancel a new unauthorized write. Review new failures and unexplained differences before summarizing aggregate progress.

Repeated executions can expose variability, but runs sharing a snapshot, prompt template or model service are not necessarily independent observations. A finite set of zero observed failures does not imply zero production risk. Do not translate test counts into a reliability probability without a defensible sampling model and an explanation of the target population.

Preserve inconclusive runs in the report and investigate whether they cluster around a candidate or dependency. Excluding unstable infrastructure can make comparison cleaner, but exclusion also removes evidence about operating conditions. Publish both the exclusion rule and its effect on denominators. Where environment parity is weak, narrow the conclusion instead of compensating with more decimal places.

Make signoff a decision with an owner

An evidence packet supports judgment; it does not authorize itself. Name the accountable human approver and the reviewers needed for the proposed authority, such as operations, security or the relevant policy owner. Record the allowed workflow, permission boundary, unresolved issues, accepted exceptions and reasons for accepting them.

NIST describes the AI Risk Management Framework as voluntary guidance for considering trustworthiness throughout design, development, use and evaluation [2]. Referencing that framework does not certify a system, establish regulatory compliance or replace obligations specific to a deployment. Likewise, human signoff is an accountability mechanism, not a guarantee that the underlying judgment is correct.

For a bounded rollout, agree on monitoring signals, stop conditions, escalation coverage and who can disable an action. Reverting a model or prompt does not undo money moved, messages sent or records disclosed. Prepare reconciliation and incident procedures for irreversible effects rather than describing rollback as universal recovery.

Define when the evidence stops applying

A release decision concerns a configuration in an operating context. A new tool schema, changed approval policy, different model revision or expanded permission can invalidate its assumptions without changing the headline product name. List these triggers in the decision record and identify which checks each trigger requires.

Distinguish evidence retention from evidence freshness. Keeping a prior report supports auditability, but does not make it applicable to today's dependency behavior. Schedule review according to risk and observed change, and rerun relevant contract checks when external systems drift. Preserve superseded decisions so reviewers can follow why authority expanded or contracted.

When an incident occurs, turn its causal mechanism into a regression scenario without exposing sensitive production material. Confirm that the scenario fails under the old behavior and that its checker can detect the defect. This closes the loop between operation and evaluation. The result is not a permanent certificate of safety, but a maintained argument: this configuration may perform these actions under these controls, based on this evidence, with these limits.