An evaluation of a learned robot policy should accept or reject a bounded claim about a defined task, robot configuration and operating population. Before counting successes, agree what constitutes an episode, completion, failure, intervention and an invalid attempt. Then connect the measurements to criteria authorised for the intended decision, with an uncertainty plan and separate evidence for system constraints and safety. A benchmark score alone does not make that contract.
“The robot picked it up” sounds clear until a reviewer asks what happened next. Was the object placed in the right fixture? Did someone correct the grasp? Was the attempt within the declared conditions? Did the robot violate a constraint while completing the task? Different answers can produce the same headline success rate while supporting different claims.
The evaluation contract makes those meanings explicit before the team sees the results. It is a suggested engineering record that lets another reviewer reconstruct what was attempted, what was observed and what decision the evidence can support.
Begin with the claim and the decision
A useful claim names the task outcome, embodiment, configuration and conditions. For example: “This policy can transfer the specified tray from the loading surface to the fixture without assistance under the declared supervised-cell conditions.” That is already more informative than “the policy is reliable”, but it still needs an operational definition and acceptance criteria.
Record the decision the evaluation is intended to inform. Comparing candidate policies, continuing development, authorising a restricted trial and accepting an operational application require different evidence and authority. A result useful for model selection may leave the operational decision unresolved.
NIST's voluntary AI Risk Management Framework offers a general basis for that discipline. Its MEASURE Playbook calls for documented test sets, metrics and tools, and for documenting operating conditions, validity and limitations on generalisation. It does not set a robotics release threshold or establish product conformity. (NIST framework status, MEASURE 2.1 and 2.5)
Fix the task, embodiment and episode boundary
The same policy label can be evaluated through different sensors, action conversions, grippers, controllers or fixtures. Retain the configuration actually used. An evaluator needs to know which observations reach the policy, what its actions mean and which surrounding system executes or rejects them.
The original paper Robot Policy Evaluation for Sim-to-Real Transfer: A Benchmarking Perspective discusses inconsistent task definitions, different action spaces and cross-embodiment comparison. Its proposed framework considers single-manipulator tasks and separates task completion, sub-task measures, failure modes, trajectory and temporal measures. Those distinctions are useful prompts for an evaluation contract, within the paper's scope. (Sections II-C and III)
Specify when an episode begins and ends. For a transfer task, it might begin when the task request is accepted after a documented reset and end at confirmed completion, a defined timeout, a disqualifying event or an intervention. Define any permitted autonomous retry within that episode. Otherwise, resetting after a bad grasp can quietly turn one failed attempt into several favourable attempts.
Reset rules matter too. State how the object and fixture are prepared, what controller or policy memory is cleared and what carries across attempts. If the system learns during evaluation, describe that behaviour and the resulting claim explicitly. Evidence from an evolving sequence answers a different question from evidence about a frozen policy configuration.
A fictional tray-transfer evaluation contract
The following example is a design exercise. No robot evaluation has been executed, no score exists and no acceptance threshold or decision authority has been supplied. The labels identify teaching records rather than real project artefacts.
| Contract field | Proposed content | What must be resolved before use |
|---|---|---|
| Decision and claim | Decide whether the candidate supports further supervised tray-transfer evaluation without task assistance | Name the decision owner and required evidence |
| Task TASK-TRAY-01 | Transfer the designated tray from loading surface L1 to fixture F1, then release it and withdraw the gripper | Define fixture acceptance geometry and endpoint observation |
| Embodiment and configuration | Fixed arm, named gripper, camera and calibration, controller, policy checkpoint, observation/action adapter and fixture revisions | Record actual identities and controlled settings |
| Episode start | A valid task request is accepted after the specified reset | Agree the reset, ready state and recorded start event |
| Episode end | Confirmed completion, authorised timeout, disqualifying event or intervention | Supply the timing limit and its authority |
| Completion meaning | Tray is in the designated accepted fixture state, released, with the gripper withdrawn, within the episode limit | Agree the measurement and tolerances |
| Assistance boundary | No person corrects motion, repositions the tray or resets the task during an autonomous completion claim | Define interventions and any separate assisted-use claim |
| Operating population | Declared trays, allowed starting placements, fixture condition, lighting range and camera installation | Supply limits and a justified sampling plan |
| Required slices | Report designated tray and placement groups separately, including difficult in-scope combinations identified in planning | Agree coverage and comparison rules |
| Constraints | Retain actual system constraints and their required responses alongside task outcomes | Derive and authorise them through the relevant system and hazard work |
| Criterion and uncertainty | Apply an authorised decision rule to each required claim, with its uncertainty treatment | Authority, numerical thresholds and analysis method remain unassigned |
| Evidence state | Protocol and evaluation records are proposed | Execution, result assessment and acceptance remain outstanding |
The missing tolerances and limits are specific decision inputs. A team can use the record to assign their resolution before testing. Filling them with plausible-looking numbers would create precision without a justified criterion.
Classify an attempt before adding it to a rate
Define validity separately from outcome. A valid attempt meets the protocol's initial conditions and has the evidence required to judge the relevant result. A policy producing a poor action is an outcome of the system under test. It is not automatically an invalid experiment.
| Classification | Proposed rule for this example | Treatment in the record |
|---|---|---|
| Valid, unassisted completion | Meets the task endpoint within the limit with no intervention | Record completion and assess constraints separately |
| Valid failure | Ends without meeting the endpoint, including a defined timeout, dropped tray or infeasible policy action | Retain the failure and diagnostic category |
| Intervened attempt | A person or another assistance mechanism changes the task execution, including a safety stop under the agreed rule | Retain the attempt and intervention reason. It cannot support the unassisted completion claim |
| Invalid experiment | A predeclared protocol defect prevents the relevant inference, such as the wrong initial fixture or loss of the required outcome record | Preserve the reason and disposition. Do not silently remove it |
| Autonomous recovery | Policy retries within the permitted episode and finishes or fails under the same contract | Retain the full episode, recovery and timing evidence |
An intervention can be appropriate and still prevent an unassisted task-success claim. A protective system stopping hazardous motion might behave as required while the task remains incomplete. These facts can coexist in the record.
Loss of an outcome log may prevent classification of task completion. It does not erase other observations, such as a documented constraint violation or an intervention. Invalid-attempt rules should state which inference is affected and who reviews the disposition.
Agree whether an assisted application is actually the intended use. If it is, define what assistance is allowed and measure that application honestly. Keep its results separate from a claim of autonomous completion. Detailed denominator calculations and trial-count selection follow from these rules, once their meanings are settled.
Measure the intended population and its important slices
A population is the set of conditions for which the claim is intended to hold. A test set is the set actually evaluated. Record how one represents the other, including exclusions and gaps. A large collection of easy placements can describe those placements precisely while offering little evidence about a difficult in-scope fixture approach.
For the tray example, the contract might require separate reporting for designated tray types and approach positions. Identify those slices before execution, state how they are sampled and preserve combinations that matter to the application. If an important slice lacks evidence, mark the relevant claim unresolved instead of letting an aggregate conceal the gap.
NIST MEASURE 2.3 discusses limits of computational performance estimates for understanding real-world context and impacts. MEASURE 2.5 calls for defined operating conditions, validity assessment and documentation of system variance. These are general guidance, with the robot's actual population and decision rule supplied by its project. (NIST MEASURE Playbook)
Avoid two common changes after seeing the results: narrowing the population to remove inconvenient failures and choosing a new success definition that flatters the policy. Either may be a legitimate revised development question, but it creates a revised contract. Preserve the original result and explain the new scope before gathering or interpreting evidence against it.
Separate desired success from constraints and safety
Task completion measures whether the desired outcome occurred. Constraints describe behaviour that must remain within the authorised system boundary. Safety acceptance needs the relevant hazard, protective-measure and system evidence. A single task-success number cannot carry all three meanings.
For example, a tray can reach F1 after a disallowed movement. Record the task endpoint and the constraint breach separately, then apply the authorised acceptance rule. Conversely, a correctly triggered protective stop can demonstrate one required response while leaving the transfer incomplete. The evaluation record should preserve both outcomes so reviewers can answer both questions.
This is where the learned policy connects to the surrounding robot requirements. Define which constraints are enforced outside the policy, what the policy is expected to expose and which integrated observations demonstrate the required response. A favourable model metric does not establish the behaviour of that complete arrangement. The companion guide on hazard-derived safety requirements explains the separate hazard-to-control chain.
State what simulation evidence can support
The SIMPLER research reports paired simulation and real-world policy evaluations across two embodiments and eight task families, with correlated performance in those reported manipulation setups. Its abstract identifies visual and control disparities as challenges the work addresses. That is evidence about the reported comparison, not a demonstrated transfer claim for any new robot. (Li and colleagues, PMLR 2025 abstract)
The Yang and colleagues benchmarking paper proposes ways to compare simulation and physical performance. Its Section IV still describes paired real-world experiments as planned. Use the proposed metrics as evaluation-design ideas, not as completed validation of transfer or deployment safety. (Proposed framework, Section IV)
For each simulation result in your contract, record the medium, implemented task, simulated embodiment and the claim being tested. If a physical claim is intended, state the evidence needed to justify that inference, including relevant discrepancies and matched physical comparison. A simulation result can be useful while that broader claim remains unresolved.
Make the acceptance decision reconstructable
Before execution, the authorised owner needs to approve the claim, criteria, required slices and uncertainty plan. Record how the analysis will handle limited observations, repeated conditions, invalid experiments and interventions. Define what outcome leaves the decision inconclusive. No uncertainty method can repair a task definition that measures the wrong outcome.
After execution, retain the protocol version, actual configuration, attempted-episode log, observations, classifications, exclusions, slice results and analysis limitations. Connect them to the criterion version and the recorded decision. The existing AI system assurance guide explains the shared configuration and evidence chain. The evaluation contract adds the specific meaning of each attempted robot task and the population being claimed.
Begin by asking two reviewers to classify the same proposed episode from its definition alone. If they disagree about completion, intervention or validity, resolve that rule before collecting a headline score. Then assign the missing criterion authority and limits. The resulting contract makes an evaluation capable of answering its intended question.