Robotics verification evidence should identify the requirement and procedure revisions, the configuration actually assessed, the conditions, the result and the reviewer's applicability decision. Preserve what a historical result established, then assess whether it supports the configuration now under review. Test execution, result interpretation, evidence acceptance and programme-baseline acceptance are separate records.
A green result needs a configuration next to it
A perception model changes and the requirement sentence stays the same. The old timing result is still in the folder. Can the team use it? That question begins with the report's original claim and the implementation it assessed. It cannot be answered by retaining a green status beside the requirement.
An evidence record needs to explain both what happened and why it is relevant now. NASA's product-verification guidance connects the assessment with the product, requirements, plan and enabling products. It also calls for analysis of data integrity, discrepancies and captured work products. The ledger below makes that context visible for a robot replay claim.
Keep CFG-A's history beside CFG-B's open question
WR-01 is a fictional supervised indoor warehouse replay bench. Its labels, 80 ms target and evidence states are teaching choices, with no customer result, stopping-distance meaning, physical operating release or safety acceptance. Actual execution, measurement and reviewer decisions are not supplied.
The accepted historical teaching state is CFG-A = H1/L1/F1.4/M1/CAL-07/MAP-03. CR-014 proposes candidate CFG-B = H1/L1/F1.4/M2/CAL-07/MAP-03. Only M1 → M2 is declared to change.
| Manifest element | Historical teaching CFG-A | Candidate teaching CFG-B |
|---|---|---|
| Hardware | H1 | H1 |
| Lidar | L1 | L1 |
| Firmware | F1.4 | F1.4 |
| Perception model | M1 | M2 |
| Calibration | CAL-07 | CAL-07 |
| Map | MAP-03 | MAP-03 |
SYS-PER-01 v1 remains:
For DSET-07 v1 under LOAD-02, elapsed time from replayed sensor-frame arrival to detection delivery at IF-DET-01 v1 shall not exceed 80 ms.
TEST-LAT-01 v1 names the teaching procedure. EVID-LAT-A is an assumed fictional historical pass for CFG-A, without a measured value or execution date. EVID-LAT-B is Pending for CFG-B, with no candidate run or result supplied. RUN-042 is a teaching run label only, rather than execution provenance or an Arc ID.
The diagram preserves two different questions. CFG-A has an assumed historical teaching outcome. CFG-B needs evidence and an applicability judgement. Unchanged labels alone do not prove artifact identity, dependency independence or evidence applicability.
Record the rig's actual state separately from the programme baseline — A programme baseline records the state authorised for a purpose. An execution record identifies what the rig actually assessed. A candidate can be tested before baseline acceptance, and the rig can differ from its planned manifest. Keep the planned, actual and authorised states distinguishable.
NASA's configuration-management guidance supports consistent product and version information. Its verification and validation plan outline asks for test-article pedigree, support equipment and facilities, and distinguishes informal integrated activities from formal verification. For WR-01, that means identifying the assessed medium and artifacts before using a declaration as an execution fact. The as-designed, as-built and as-tested guide explains the general record distinctions.
The fourteen-group result and applicability ledger
Use one record for each result and preserve its proposed uses. The groups below keep provenance subfields explicit and distinguish a plan from an observation. Real accountable owners are roles to assign. No reviewer name, execution time or report digest is invented to complete a cell.
| Field group | Historical teaching EVID-LAT-A | Candidate teaching EVID-LAT-B | Real evidence or owner action needed |
|---|---|---|---|
| Requirement revision | SYS-PER-01 v1, motivated by NEED-01, with IF-DET-01 v1 as the delivery boundary. | SYS-PER-01 v1 proposed for the same claim. | Requirement owner identifies the actual item/revision, criterion and source/interface relationships. |
| Method | Test of the replay timing claim, by teaching assumption. | Test planned, with no execution supplied. | Verification owner states the method and supported claim, including any contribution from analysis, inspection or demonstration. |
| Procedure revision | TEST-LAT-01 v1 named. Measurement details unknown. | TEST-LAT-01 v1 proposed, with suitability for M2 unresolved. | Verification owner supplies controlled identity, observation points, capture, pairing, comparison and invalid-data rules. |
| Actual configuration and separate programme baseline | Teaching assessed CFG-A: H1/L1/F1.4/M1/CAL-07/MAP-03. Historical teaching baseline CFG-A. Actual rig and programme revision unknown. | Planned CFG-B: H1/L1/F1.4/M2/CAL-07/MAP-03. Candidate under CR-014, with no observed execution or accepted baseline. | Configuration owner records actual hardware/software and test-medium identity, links the programme/source baseline separately and reconciles differences from the plan. |
| Artifact identities and digests | Actual model, firmware, runtime and fixture references/digests unknown. | Candidate references/digests unknown. | Configuration and evidence owners supply resolvable identities and any digests used. Labels alone do not identify assessed bytes. |
| Dataset, fixture and test medium | DSET-07 v1 under LOAD-02 on the supervised warehouse replay bench. Content, load and actual medium details unknown. | Same labels proposed, with no run confirming them. | Dataset/fixture owners define content/version identity, load, replay medium and limits. |
| Calibration and map | CAL-07/MAP-03 named, with actual references and applicability facts unknown. | CAL-07/MAP-03 retained, with M2 applicability unresolved. | Calibration/localisation owner identifies assessed references, validity and relevant preprocessing/transform dependencies. |
| Interface, clocks and units | IF-DET-01 v1. Sensor-frame arrival to detection delivery in milliseconds. Actual clock basis/resolution, capture, pairing and frame facts unknown. | Same boundary proposed, with measurement facts unknown. | Endpoint and verification owners define meanings, units, clocks/events and relevant spatial frame/axes, including any justified conversions. |
| Execution and report provenance | No real run/date/time or measured execution supplied. Raw/report references/digests, producing system, tools and instruments/versions unknown. | RUN-042 is a teaching label. No real execution, raw/report/digest or producing-system/tool/instrument facts supplied. | Evidence owner records actual run/system identity, execution date/time, raw data and report identities/digests, tools/versions and instrument identity/version/calibration. Justify N/A for an absent instrument. |
| Actual outcome | Assumed fictional historical pass for the CFG-A bench claim. No measured latency supplied. | Pending, with no candidate run or result supplied. | Verification owner records actual values/units and interpretation against the criterion, or the precise outstanding state. Pending is neither pass nor fail. |
| Anomaly or deviation | No actual issue record supplied. This does not establish anomaly-free execution. | No execution supplied, so observed issues and dispositions unknown. | Verification and relevant authority record observed issues, affected claims, status, deviation authority and disposition. |
| Applicability, rationale and limits | Historical teaching claim retained. Proposed use for CFG-B requires model-dependency reassessment. | Candidate applicability unresolved. | Reviewer states applicable, conditional, reassessment needed or unresolved, with reasoning, covered conditions/configurations and exclusions. |
| Reviewer, date and authority | No real reviewer/date or completed decision. | Verification owner and baseline authority are roles to assign. No real review/date supplied. | Record reviewer, date, inspected revision and decision authority, separately from any baseline-acceptance decision. |
| Open action | Preserve history and assess its contribution to the candidate argument. | Resolve measurement/provenance gaps and obtain candidate evidence or an adequate applicability argument. | Assign a role, evidence request and next decision for each gap. Record supplement, repeat or hold and an agreed due date when supplied. |
The ledger should make an incomplete record easier to use. An unknown load definition goes to the fixture owner. A missing producing-system or report identity goes to the evidence owner. A valid report with a changed model dependency goes to the verification reviewer for an applicability decision. Those are different problems with different remedies.
NIST's voluntary MEASURE guidance supports documenting test sets, metrics and tools, examining deployment-like conditions and recording limits on generalisation. A reachable report is a useful artifact reference, but its content and context still need review.
Model Cards offers prompts for model versions, intended use, metrics and evaluation data. Datasheets for Datasets adds prompts for dataset motivation, composition and lifecycle, including explicit unknowns. These documentation proposals can inform the context fields. They do not establish satisfaction of a robot system requirement or operational safety.
Read the status as several facts — Test execution records that an activity ran. The result records what was observed and how it compares with the criterion. Evidence acceptance records an authorised judgement that the evidence supports a defined claim. Applicability records the configurations and conditions that judgement covers. Programme-baseline acceptance is a further decision about an authorised state.
The general continuous-verification guide explains the broader lifecycle. In this ledger, keep these facts beside each other. A completed test can still have an unresolved anomaly. A valid historical pass can remain useful for its original context while its candidate applicability is unresolved. A Pending state needs a reason, and EVID-LAT-B's reason is specific: no candidate execution or result is supplied.
Reconcile the actual configuration before proposing coverage
Compare the observed artifacts and conditions with the planned manifest. Identify the procedure and requirement revisions used. Record each discrepancy and ask the configuration and verification owners which claim the activity actually assessed.
If a real rig used an artifact different from planned CFG-B, preserve that actual identity. The evidence may support a claim about the state assessed, depending on procedure validity and sufficiency. It cannot silently be relabelled as a result for the intended candidate. Any proposed applicability to another state needs its own rationale.
This reconciliation also protects useful negative information. An event-capture problem, invalid calibration or missing data can limit a claim even when the reported value meets a threshold. Keep the original values and issue record so the reviewer can decide what to correct, supplement or repeat.
Decide what to reuse, supplement, repeat or hold
M2 lies on the timing path, so the old label “pass” cannot settle its evidence question. The verification owner can propose a repeated timing test, a supplement for a defined gap or an applicability argument addressing the changed dependency. No such candidate evidence or argument is supplied in WR-01.
| Evidence or trigger | Question for the reviewer | Proposed action and limit |
|---|---|---|
| EVID-LAT-A considered for CFG-B | Does the historical timing claim cover the changed M2 path? | Preserve CFG-A history and reassess. Repeat, supplement or a justified applicability argument remains an open action. |
| Potential physical-label inspection | Is the inspected label/property independent of M1 → M2, with the same relevant item and conditions? | Potential reuse only for that property after actual evidence and review. No inspection or result is supplied. |
| A criterion changes in a real successor | Does the earlier result support the revised obligation and comparison basis? | Reassess against the controlled revision while preserving the original result and old criterion. |
| Calibration or provenance is found invalid | Can the affected measurement still support the claim? | Hold affected closure, investigate scope and supplement or repeat as justified. Preserve the validity finding. |
| Evidence supports part of a claim | Which conditions or configurations are supported, and which remain open? | Record conditional or partial applicability with exclusions and actions. Whole-requirement closure remains a separate judgement. |
The last four rows illustrate possible decisions, rather than events observed in WR-01. A physical-label inspection can be a useful contrast because its assessed property may be independent of the model. Even then, the owner needs the inspection record, item identity, criterion, relevant conditions, anomaly status and independence argument before reuse is accepted.
Keep the criterion visible through rework. Weakening SYS-PER-01 to retain a pass changes the obligation, so it requires an explicit authorised revision and a new evidence assessment. Preserving history should help a reviewer explain a changed decision.
One question about limited evidence
Can one result have conditional applicability?
Yes. A reviewer can identify a supported subset of conditions, configurations or claims, with explicit exclusions and remaining actions. Record those boundaries alongside the result. A whole-requirement pass flag would hide them. No conditional acceptance is asserted for the WR-01 candidate.
Check one result against two configurations
Take one real report and two manifests. Complete the fourteen groups, reconcile the actual assessed state and ask the verification owner what remains applicable, what needs evidence and which decision is open. Use that question in a requirements-workflow evaluation, including an Arc evaluation, without assuming a particular product representation, retention guarantee or automatic acceptance rule.
The model-change dossier owns the wider dependency sweep. The CI mapping guide owns preparation of a reviewable evidence proposal from an external run.