Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,12 @@ The project follows [Keep a Changelog](https://keepachangelog.com/) conventions.

## Unreleased

- Added a captured two-run qualification workflow. Candidate adapters can now
report optional revision and snapshot metadata, candidate-facing commands can
enforce a task-pinned identity before the first event, and `evidence-init` /
`evidence-verify` prepare clean roots and emit a grading-ready manifest only
for unchanged, complete, canonical, byte-identical qualification bundles.

## 0.6.0 - 2026-07-27

- Fixed conformance CLI exit classification across single runs, suites,
Expand Down
9 changes: 9 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -189,6 +189,15 @@ these are separate version boundaries. `fifo-limit-v1` qualification does not
run STP or pro-rata cases, so unsupported features stay explicit without
becoming false failures.

For release evidence, do not rely on one successful workspace. Start a captured
pair with `tracebook-conformance evidence-init`, run the canonical qualification
once in each generated clean root with the plan's pinned candidate name,
revision, and snapshot, then run `tracebook-conformance evidence-verify`. The
verifier emits a compact `evidence-manifest.json` only when both source trees
remain unchanged and both qualification bundles agree on terminal result,
candidate metadata, deterministic IDs, counts, coverage, and every artifact
byte. See the [captured evidence workflow](https://github.com/Taz33m/tracebook/blob/main/docs/conformance.md#captured-two-run-evidence).

[Read the research-grounded roadmap and adoption experiment](https://github.com/Taz33m/tracebook/blob/main/docs/research-roadmap.md).

## Architecture
Expand Down
5 changes: 5 additions & 0 deletions docs/api-stability.md
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,11 @@
normalization, or hashing requires an explicit protocol/schema version
decision. `suite_hash` binds case configuration and fixture identity; an
intentional suite edit must update it.
- Protocol-v1 engine metadata may add the optional `revision` and `snapshot_id`
identity fields. Ordinary adapters may omit them; a command using the three
task-pinned `--candidate-*` flags requires an exact match before any event is
sent. Evidence plan schema v1 and evidence manifest schema v1 are public
artifacts for the canonical captured two-run qualification workflow.
- Campaign artifact schema version 1 is public. Campaign generator version 2,
the built-in versioned profile definitions, seed derivation, and trace hashes
are reproducibility contracts. An intentional generation change requires a
Expand Down
21 changes: 21 additions & 0 deletions docs/commands.md
Original file line number Diff line number Diff line change
Expand Up @@ -132,6 +132,27 @@ automatically minimized failure. Use this command for a first integration or a
profile-level CI claim; use `suite` when intentionally comparing every broader
fixed semantic surface.

Prepare and verify a release-grade two-run evidence pair:

```bash
tracebook-conformance evidence-init /path/to/candidate \
--workspace .tracebook/release-evidence \
--candidate-name owner/repository \
--candidate-revision REVISION

# Run the canonical qualification from evidence-plan.json in both generated
# roots, with all three --candidate-* identity flags, then:
tracebook-conformance evidence-verify \
.tracebook/release-evidence/evidence-plan.json
```

`evidence-init` refuses an existing workspace and creates independent candidate,
adapter, build, cache, and qualification paths for `run-1` and `run-2`.
`evidence-verify` writes `evidence-manifest.json` only after both canonical
bundles are qualified, byte-identical, and bound to the unchanged pinned
candidate. See [Captured Two-Run Evidence](conformance.md#captured-two-run-evidence)
for the exact qualification command and trust boundary.

Copy and run the standard suite:

```bash
Expand Down
78 changes: 76 additions & 2 deletions docs/conformance.md
Original file line number Diff line number Diff line change
Expand Up @@ -135,6 +135,64 @@ not exchange certification and does not imply support for unselected suite
cases. Selection version 1 is part of the artifact identity and will not change
silently.

## Captured Two-Run Evidence

Use the evidence workflow when a qualification result will support a release,
benchmark, or external claim. It pins the candidate before adapter readiness and
requires the same canonical qualification from two prepared roots.

```bash
REVISION=$(git -C /path/to/candidate rev-parse HEAD)
WORKSPACE=$PWD/.tracebook/release-evidence

tracebook-conformance evidence-init /path/to/candidate \
--workspace "$WORKSPACE" \
--candidate-name owner/repository \
--candidate-revision "$REVISION"
```

`evidence-init` strips `.git` from the captured source identity, copies that
exact tree into `runs/run-1/candidate` and `runs/run-2/candidate`, and creates a
separate empty `adapter`, `build`, and `cache` directory for each run. It writes
`evidence-plan.json` last and refuses an existing or source-nested workspace.
Use each run's paths for that run's adapter and build; do not share generated
files between the two runs.

The command prints the candidate snapshot ID. The adapter's `ready` frame must
report that ID and the task-pinned name and revision. Run the plan's canonical
contract in each root, changing only the run number and adapter command:

```bash
SNAPSHOT=$(python -c \
'import json,sys; print(json.load(open(sys.argv[1]))["candidate"]["snapshot_id"])' \
"$WORKSPACE/evidence-plan.json")

tracebook-conformance qualify \
--profile fifo-limit-v1 --suite-version v2 \
--seed 42 --traces 25 --events-per-trace 200 --max-minimize-runs 100 \
--candidate-name owner/repository \
--candidate-revision "$REVISION" \
--candidate-snapshot "$SNAPSHOT" \
--candidate-cmd "$WORKSPACE/runs/run-1/adapter/launch" \
--output-dir "$WORKSPACE/runs/run-1/qualification"
```

Repeat for `run-2`, using only its candidate, adapter, build, cache, and
qualification paths. Then verify the pair:

```bash
tracebook-conformance evidence-verify "$WORKSPACE/evidence-plan.json"
```

Verification fails closed if either captured tree changed; either bundle is
missing, noncanonical, incomplete, or unqualified; candidate identity differs;
or terminal result, deterministic IDs, counts, semantic coverage, or artifact
bytes differ between runs. Success writes one exclusive
`evidence-manifest.json` at the workspace root. The workflow prepares and checks
the filesystem contract, but it does not sandbox a compiler or prove that an
adapter avoided undeclared external caches; CI or a disposable host remains the
strongest execution boundary.

## Differential Campaigns

Campaigns generate stateful traces, compare them one at a time, and stop at the
Expand Down Expand Up @@ -346,9 +404,15 @@ Host to candidate, once:
Candidate to host:

```json
{"type":"ready","protocol":"tracebook.conformance","protocol_version":1,"engine":{"name":"my-engine","version":"1.4.2","language":"Rust"}}
{"type":"ready","protocol":"tracebook.conformance","protocol_version":1,"engine":{"name":"owner/repository","version":"1.4.2","language":"Rust","revision":"8f31c2a","snapshot_id":"sha256:..."}}
```

`revision` and `snapshot_id` are optional for ordinary comparisons. When
`--candidate-name`, `--candidate-revision`, and `--candidate-snapshot` are
supplied, all three flags are required together and the `ready` metadata must
match before Tracebook sends the first event. Captured two-run evidence always
uses the pinned form.

Host to candidate for each event:

```json
Expand Down Expand Up @@ -429,7 +493,13 @@ from tracebook.conformance import EngineMetadata, serve_stdio

class MyAdapter:
def __init__(self, config):
self.metadata = EngineMetadata("my-engine", "1.0", "Python")
self.metadata = EngineMetadata(
"owner/repository",
"1.0",
"Python",
revision="8f31c2a",
snapshot_id="sha256:...",
)
self.config = config

def apply(self, event, index):
Expand All @@ -447,6 +517,10 @@ class MyAdapter:
raise SystemExit(serve_stdio(MyAdapter))
```

The last two fields are needed only when the invoking command pins candidate
identity. In a captured evidence run, populate them from the immutable task or
`evidence-plan.json`, not by inspecting the host after the adapter starts.

[`examples/conformance_adapter.py`](../examples/conformance_adapter.py) is a
runnable reference. Non-Python adapters implement the same frames directly.

Expand Down
84 changes: 82 additions & 2 deletions src/tracebook/conformance/cli.py
Original file line number Diff line number Diff line change
Expand Up @@ -20,11 +20,12 @@
write_campaign_corpus,
)
from .compare import run_conformance
from .evidence import prepare_evidence_workspace, write_evidence_manifest
from .external import AdapterProtocolError, ExternalProcessAdapterFactory
from .exit_codes import exit_code_for_artifact
from .junit import write_junit
from .minimize import minimize_failing_trace
from .model import ConformanceConfig, ConformanceError
from .model import ConformanceConfig, ConformanceError, PinnedCandidateIdentity
from .qualification import _QualificationOutputReservation, run_qualification
from .reproduce import (
discover_failure_metadata,
Expand Down Expand Up @@ -54,6 +55,18 @@ def _add_config_arguments(parser: argparse.ArgumentParser) -> None:

def _add_candidate_arguments(parser: argparse.ArgumentParser) -> None:
parser.add_argument("--timeout", type=float, default=5.0)
parser.add_argument(
"--candidate-name",
help="Task-pinned candidate name expected in the ready frame.",
)
parser.add_argument(
"--candidate-revision",
help="Task-pinned revision expected in the ready frame.",
)
parser.add_argument(
"--candidate-snapshot",
help="Task-pinned snapshot ID expected in the ready frame.",
)
candidate = parser.add_mutually_exclusive_group(required=True)
candidate.add_argument(
"--candidate-cmd",
Expand Down Expand Up @@ -163,6 +176,23 @@ def _build_parser() -> argparse.ArgumentParser:
reproduce.add_argument("--junit-output")
_add_config_arguments(reproduce)
_add_candidate_arguments(reproduce)

evidence_init = commands.add_parser(
"evidence-init",
help="Create two clean roots for a captured qualification pair.",
)
evidence_init.add_argument("candidate_source")
evidence_init.add_argument("--workspace", required=True)
evidence_init.add_argument("--candidate-name", required=True)
evidence_init.add_argument("--candidate-revision", required=True)
evidence_init.add_argument("--expected-snapshot")

evidence_verify = commands.add_parser(
"evidence-verify",
help="Verify two captured qualification bundles and write one manifest.",
)
evidence_verify.add_argument("plan")
evidence_verify.add_argument("--output")
return parser


Expand All @@ -174,7 +204,32 @@ def _candidate_factory(args) -> ExternalProcessAdapterFactory:
command = command[1:]
if not command:
raise ConformanceError("--candidate requires a command")
return ExternalProcessAdapterFactory(command, timeout_seconds=args.timeout)
identity_values = (
args.candidate_name,
args.candidate_revision,
args.candidate_snapshot,
)
if any(value is not None for value in identity_values) and not all(
value is not None for value in identity_values
):
raise ConformanceError(
"--candidate-name, --candidate-revision, and --candidate-snapshot "
"must be provided together"
)
expected_identity = (
PinnedCandidateIdentity(
name=args.candidate_name,
revision=args.candidate_revision,
snapshot_id=args.candidate_snapshot,
)
if all(value is not None for value in identity_values)
else None
)
return ExternalProcessAdapterFactory(
command,
timeout_seconds=args.timeout,
expected_identity=expected_identity,
)


def _config(args) -> ConformanceConfig:
Expand Down Expand Up @@ -253,6 +308,31 @@ def main(argv: Optional[List[str]] = None) -> int:
print(f"Suite id: {suite.suite_id}")
print(f"Cases: {len(suite.cases)}")
return 0
if args.command == "evidence-init":
plan_path = prepare_evidence_workspace(
args.candidate_source,
args.workspace,
candidate_name=args.candidate_name,
candidate_revision=args.candidate_revision,
expected_snapshot=args.expected_snapshot,
)
plan = json.loads(plan_path.read_text(encoding="utf-8"))
print(f"Evidence plan written: {plan_path}")
print(f"Candidate snapshot: {plan['candidate']['snapshot_id']}")
for run in plan["runs"]:
print(
f"{run['run_id']}: candidate={run['candidate_root']} "
f"qualification={run['qualification_dir']}"
)
return 0
if args.command == "evidence-verify":
manifest_path = write_evidence_manifest(args.plan, args.output)
manifest = json.loads(manifest_path.read_text(encoding="utf-8"))
print(f"Evidence manifest written: {manifest_path}")
print(f"Manifest: {manifest['manifest_id']}")
print(f"Qualification: {manifest['runs'][0]['qualification_id']}")
print("Evidence pair: PASS")
return 0
if args.command == "run":
_require_distinct_paths(args.events, args.output, args.junit_output)
single_report = run_conformance(
Expand Down
Loading