Skip to content

Freeze the v1.7 Operator contract - #1101

Draft
jeffhuber wants to merge 5 commits into
mainfrom
codex/1085-operator-contract
Draft

jeffhuber wants to merge 5 commits into
mainfrom
codex/1085-operator-contract

Conversation

@jeffhuber

@jeffhuber jeffhuber commented Sep 21, 2026

Copy link
Copy Markdown
Contributor

The v1.7 Operator work needs a stable authority and recovery boundary before any service or mutation runtime can safely consume it. This PR adds closed, versioned schemas, a normative semantic and policy-binding validator, and canonical accepted/rejected/executable transition fixtures for policy, provider qualification, work lifecycle, singleton lease fencing, durable mutation intent, response certainty, reconciliation, bounded budgets, and metadata-only projection.

The contract distinguishes stale prepared intents from already-dispatched unknown effects: takeover preserves the immutable work generation, lease identity, dispatch fence, and target while a separately fenced current holder reconciles without redispatch. Dispatch, original-holder retry, takeover-holder retry, reconciliation, and result commit fail closed against an independently supplied live lease, tenant/repository scope, current generation, and exact head. A confirmed failure after takeover may retry under the same intent only when current reconciliation authority matches the independently read lease; the original dispatch fence remains stale and immutable.

policy_binding_errors binds state records to the owner policy: tenant and repository admission, mutation authority, action attempts/reconciliations/time/spend, cumulative work time/spend, owner escalations, and lease cadence. Lease renewal and expiry are absolute offsets from the durable current-interval acquired_at anchor, so shifting both deadlines cannot extend authority. Recovery binds each event to its allowed source and destination action tuples, requires work last_transition to equal the actual before-to-after edge, preserves scope and identity, and rejects chronology, timestamp, and cumulative-counter rollback.

Terminal work state/reason pairs and projected event/identifier/state/reason families are closed in the schema. The executable catalog covers restart, duplicate delivery, takeover, stale head, timeout, partial success, budget exhaustion, escalation redelivery, and owner stop, plus named negative transitions for known-failure timeout regression, known-success stale-head abandonment, and false work-source edges.

The canonical contract documents human-only merge approval and remains contract-only. This PR does not add an Operator loop, active dispatch, service container, GitHub App mutations, remote Board ingress, or cloud schema changes.

Validation on exact head 531c4530680e53fcf9855a4e6aef4ac2cc2f6f54:

  • 22 focused Operator contract tests and 4,266 generated subtests pass under pytest; the unittest discovery path also passes.
  • Official Draft 2020-12 validation accepts both schemas, all 16 canonical accepted fixtures, and all 61 schema-valid required, forbidden, and rejected-transition records.
  • The proactive matrices exhaust every recovery action source/target tuple, every recovery work source/target state, every work state/reason pair, every projection event/state/reason pair and identifier family, every declared work transition, every lease state, mutation authority dimensions, policy ceilings, takeover scope/chronology, and cumulative rollback dimensions.
  • Ruff, privacy scan, documentation lifecycle (141/141 declared documents), package workflow guard, committed package-manifest regeneration, and git diff --check pass.
  • Fresh wheel and sdist builds, Twine checks, and artifact inclusion pass for the semantic validator, both schemas, fixture catalog, and canonical contract document.
  • The earlier full repository suite ran 4,874 tests. Six Board stop tests failed identically on the base commit when this Mac's installed managed Board service was detected. Four packaging failures caused by a pip-less test venv/concurrent build were rerun serially after seeding pip and passed.

Closes #1085

@jeffhuber jeffhuber added needs-claude-audit builder:codex Code Mower generated label labels Sep 21, 2026
@jeffhuber

Copy link
Copy Markdown
Contributor Author

Codex exact-head audit — BLOCKED

Audited exact head caa20e62d94e5eb2b82b3308e2bff4461b0e5fd2 independently and read-only.

[P1] Lease takeover makes an already-dispatched unknown result unrepresentable

src/code_mower/operator_state_v1.schema.json:773-807 requires certainty: unknown to remain dispatched, uncertain, or reconciling, with reconciliation/owner action as the only next step. But :894-927 forces every stale fence or stale head to state: abandoned and next_action: abandon.

After takeover, the old immutable intent's fence is necessarily stale. If that intent was dispatched and its response was lost, it must remain unknown and be reconciled by the new holder; abandoning it can lose a remote success and permit a later duplicate. The current schema has no valid record for that required situation. Reproduction: start from accepted fixture timeout_is_uncertain_and_reconciles_before_retry, change only fence_status to stale, and Draft 2020-12 validation rejects it because the two then branches conflict. This contradicts docs/operator-contract-v1.md:79-81,100-114 and the #1085 takeover/restart criterion.

Please distinguish a stale pre-dispatch intent, which can be abandoned, from a stale-fenced post-dispatch unknown intent, which the current holder must reconcile without redispatch. Preserve the original immutable dispatch fence while representing the current reconciler's authority, and add accepted takeover-after-unknown plus rejected blind-retry fixtures.

[P2] Qualification records are not internally fail-closed

src/code_mower/operator_policy_v1.schema.json:493-514 constrains pending/failed/stale records only to decision: denied; it does not constrain their reason or evidence. Consequently a record with status: pending, decision: denied, reason: evidence_current, and full evidence validates. Likewise :350-381 permits expires_at <= observed_at, and a qualified/allowed/evidence_current record with reverse expiry still validates. The canonical document promises current, unexpired evidence at docs/operator-contract-v1.md:145-149.

Add status-specific reason/evidence invariants and executable semantic validation for cross-field/time checks (including observed_at < expires_at, age relative to policy, and current time). Add rejected fixtures for contradictory status/reason/evidence and invalid or stale evidence. JSON Schema alone cannot establish wall-clock freshness, so dependent runtimes need one normative validation rule/API rather than each inventing it.

[P2] The documented owner-stop/escalation contract cannot be encoded durably

The canonical document says stale head and stale lease stop in awaiting_owner and that escalations use a durable deduplication key (docs/operator-contract-v1.md:59-64). The work schema does allow those two stop reasons (src/code_mower/operator_state_v1.schema.json:229-255), but policy owner_action.stop_reasons omits both (src/code_mower/operator_policy_v1.schema.json:257-283). The work item stores only owner_escalation_count (operator_state_v1.schema.json:120-175); no record carries the promised durable escalation deduplication key.

Align the policy stop list with the state/doc contract and add a durable dedupe identity (on the work item or a closed escalation record), with duplicate-delivery/restart fixtures proving one owner notification within the configured count.

[P2] Mutation intents are not bound to the work generation they authorize

The work item has a monotonically increasing generation, and the canonical document says a generation mismatch rejects a write (operator_state_v1.schema.json:141-145; docs/operator-contract-v1.md:46-57). The action-intent required/properties block (operator_state_v1.schema.json:610-743) binds only work_id; it carries no work generation or equivalent immutable version. After cancellation/re-admission or any later generation, recovery cannot prove whether a persisted intent belongs to the current work generation before reconciling or retrying it.

Bind each intent to work_generation (and reject stale-generation dispatch/retry), then add accepted current-generation and rejected stale-generation fixtures. This is needed for the stated CAS/generation boundary to protect remote mutations rather than only work-item writes.

[P2] Several required failure fixtures are labels, not executable contract cases

failure_scenarios stores three strings per scenario, while tests/test_operator_contract_v1.py:252-273 only asserts scenario names and selected string values. Restart-after-dispatch, duplicate delivery, takeover, partial success, and owner stop therefore do not exercise a schema-valid before/after sequence or rejection invariant. The P1 conflict above survives because the takeover and timeout fixtures are never composed.

Represent the recovery scenarios as structured records/sequences validated against the contract (or a small normative transition validator) and test their required and forbidden transitions. That makes the fixtures consumable by dependent runtime/provider PRs as #1085 requires.

Validation performed

  • Remote PR head verified as caa20e62d94e5eb2b82b3308e2bff4461b0e5fd2 immediately before this comment.
  • 9 focused Operator contract tests pass.
  • Official Draft 2020-12 validation accepts all 13 accepted fixtures and rejects all 11 rejected fixtures; it also reproduces the unknown-plus-stale conflict and confirms the contradictory qualification/reverse-expiry cases currently validate.
  • Ruff and privacy scan pass.
  • Documentation lifecycle passes with 141/141 declared documents.
  • Committed package manifest exactly matches a fresh generated manifest.
  • No runtime or backward-compatibility changes are introduced by this PR; the findings are contract correctness gaps that should be fixed before dependent Operator implementations consume v1.

@github-actions

github-actions Bot commented Sep 21, 2026

Copy link
Copy Markdown

Claude audit (merge-authority lane)

Head SHA: caa20e62d94e5eb2b82b3308e2bff4461b0e5fd2
Verdict: PASS
Review details remain in the local audit artifact.
Publication workflow: .github/workflows/local-audit-publication.yml at c38c137353e0d58ef60b66d47aebd2a73668ab62

@jeffhuber

Copy link
Copy Markdown
Contributor Author

Codex exact-head audit — BLOCKED

Audited exact head 75de57efe8b375b2797235199dbd6ce211802f6b independently and read-only.

The five findings from the previous exact-head audit are materially addressed at the record/fixture level: takeover can now retain an unknown dispatched intent under separate reconciliation authority; qualification status/reason/evidence and time checks exist; owner stop reasons and an escalation key are durable; intents bind work generation; and recovery cases are structured executable sequences. Three remaining authority gaps prevent a PASS.

[P1] The normative intent check still approves a stale holder for dispatch or retry

src/code_mower/operator_contract_v1.py:110-131 compares current_lease_epoch and current_fence_token only when the record already contains reconciliation_authority. A prepared intent and a confirmed-failure intent with next_action: retry have no reconciliation authority, so the helper returns no errors even when the caller supplies a completely different current lease epoch/token. It also accepts the record's self-asserted fence_status: current and head_status: current without comparing the durable lease scope/state/expiry or current head.

That contradicts docs/operator-contract-v1.md:75-96,110-112: a stale holder must not dispatch or retry, and the durable store/adapter must compare the immutable dispatch fence immediately before mutation. A later runtime consuming this advertised normative helper can pass validation and duplicate a mutation after takeover.

Please make the semantic authority check operation-aware and fail closed: compare the intent's original epoch/token, tenant/repository scope, work generation, exact head, and a live current lease before dispatch/retry/result commit; compare reconciliation_authority to that same independently supplied lease for reconciliation. Add rejected stale-fence/head and expired/wrong-scope lease fixtures rather than trusting status labels stored on the intent.

[P1] Owner policy is not bound to work/action records, so valid records can exceed authorization and budgets

The policy schemas define the allowlist, mutation set, and ceilings, but no normative check relates them to an action or work record. action_intent_semantic_errors receives no policy. With the checked-in pilot policy (max_attempts_per_action: 3, reconciliations 2, action seconds 300, spend $25.00, owner escalations 2), I changed a prepared intent to schema-valid counters 5/5/604800/$999999.99; official Draft 2020-12 validation and action_intent_semantic_errors(... current_work_generation=4) both accepted it. A work item with owner_escalation_count: 3 likewise validates. A repository outside the policy allowlist or an operation absent from authority.mutations has the same gap.

This violates the #1085 budget/authority scope and docs/operator-contract-v1.md:132-139, where only the owner may enlarge those limits. Add one policy-binding semantic API (and accepted/rejected fixtures) that checks tenant and repository admission, operation authority, attempt/reconciliation/time/spend ceilings, owner-escalation ceiling, and applicable lease cadence. Concurrency can remain a durable-store/runtime check, but the record-level ceilings must not be independently reinvented by each dependent PR.

[P1] Recovery validation permits cross-scope takeover and cumulative-counter rollback

recovery_transition_errors says it validates counters and immutable identity (operator_contract_v1.py:172-181), but its generic pairing checks only selected fields. I changed the takeover's new lease to a different tenant, repository, and lease ID while preserving epoch+1/token; the function returned (). I also changed the restart-after-dispatch result from elapsed_seconds: 300, spend_usd: 0.10 to 1/0.00; it returned (). Reconciliation counts can be reset in the same way. The takeover branch checks only epoch/token and attempt count (:230-257), and the other branches generally inspect only attempt count (:188-228).

A lease from another scope cannot authorize recovery of this intent, and rolling counters backward bypasses the cumulative budget promised at docs/operator-contract-v1.md:103-104,134-138. Require tenant/repository/lease identity continuity, bind the intent to the lease scope, require monotonic timestamps and all cumulative counters (using decimal comparison for spend), and add forbidden cross-scope/counter-reset fixtures. Lease chronology also needs a semantic invariant (acquired_at <= renew_by < expires_at); a reversed active lease currently passes the official schema with no semantic error.

Validation performed

  • Remote PR head re-verified as 75de57efe8b375b2797235199dbd6ce211802f6b immediately before publication.
  • 12 focused Operator tests pass under both unittest and pytest (54 subtests under pytest).
  • Official Draft 2020-12 check_schema passes; all 14 accepted fixtures validate and all 20 rejected fixtures fail schema or normative semantic validation.
  • Ruff, privacy scan, docs lifecycle tests, package-manifest/release-hygiene checks, and git diff --check pass.
  • The adversarial reproductions above remain individually schema-valid and return no semantic/recovery errors.

@github-actions

github-actions Bot commented Sep 21, 2026

Copy link
Copy Markdown

Claude audit (merge-authority lane)

Head SHA: 75de57efe8b375b2797235199dbd6ce211802f6b
Verdict: PASS
Review details remain in the local audit artifact.
Publication workflow: .github/workflows/local-audit-publication.yml at c38c137353e0d58ef60b66d47aebd2a73668ab62

@jeffhuber

Copy link
Copy Markdown
Contributor Author

Codex exact-head audit — BLOCKED

Audited exact head 4d3ecc35393f3810ab54e4c2bd48d6da69f7cb76 independently and read-only.

Counts: P0=0, P1=3, P2=1.

The three P1s from the preceding audit are materially improved: mutation checks now bind the supplied generation, tenant/repository, lease identity, live fence and exact head; policy binding covers record-level authority and budget ceilings; and recovery rejects the previously demonstrated scope, identity, chronology and cumulative-counter rollbacks. Four remaining contract gaps prevent a PASS.

[P1] A takeover-confirmed failure cannot perform the contract's same-intent retry

action_intent_semantic_errors uses reconciliation_authority to compare the new holder's durable lease, but lines 157-161 independently require the intent's original fence_status to be current for every retry. The accepted takeover shape correctly keeps the immutable dispatch fence stale and records the new holder only in reconciliation_authority. Starting from takeover_reconciles_unknown_without_redispatch, changing the outcome to schema-valid confirmed_failure / matched_failure / next_action: retry produces only dispatch or retry requires a current dispatch fence.

That leaves no truthful record for the documented rule at docs/operator-contract-v1.md:124-130 that a confirmed absence/failure may retry under the same durable intent after reconciliation. Marking the original fence current would contradict the accepted takeover fixture and immutable-fence contract; creating a new intent contradicts the same-intent rule. Bind retry to the independently read current reconciliation authority after takeover while retaining the stale original dispatch fence, and add an accepted/rejected exact matrix for original-holder and takeover-holder retry.

[P1] Recovery events do not bind the required source state or the actual work edge

The pair helpers at operator_contract_v1.py:307-370 preserve identities and monotonic counters, but the event branches at :416-526 generally validate only the destination. As a result, all of these schema-valid corruptions return no recovery errors:

  • provider_timeout can turn an already confirmed_failure intent back into unknown;
  • stale_head can turn an already confirmed_success intent into abandoned;
  • budget_exhaustion can move an observed work item directly to awaiting_owner while the after record claims last_transition: executing:awaiting_owner.

The test called a full transition matrix at tests/test_operator_contract_v1.py:608-641 validates isolated destination records; it never composes the source and destination named by last_transition, and there is no equivalent complete action transition matrix. This can erase a known result or accept an impossible lifecycle edge, violating the atomic state/certainty guarantees at docs/operator-contract-v1.md:56-63,117-137. Define each event's allowed source tuple and require new.last_transition == f"{old.state}:{new.state}"; exhaustively reject all other source/destination combinations for work and action records.

[P1] Lease cadence binding allows authority beyond the owner TTL and renewal deadline

policy_binding_errors lines 286-294 checks only expires_at - renew_by == lease_ttl_seconds - lease_renewal_seconds. Shifting both deadlines equally leaves that difference unchanged. On the accepted policy/lease, adding 50 seconds to both renew_by and expires_at remains schema-valid and returns (), even though it changes the configured 30-second renewal point to 80 seconds and the 90-second TTL to 140 seconds.

This materially widens mutation authority beyond the owner policy and contradicts docs/operator-contract-v1.md:98-103,141-146. Bind each deadline to acquired_at (renew_by - acquired_at and expires_at - acquired_at) and validate the policy relation itself, with shifted-early and shifted-late negative fixtures.

[P2] Closed records still admit contradictory state/reason projections

The work schema accepts state: completed, terminal: true, last_transition: awaiting_review:completed, and reason: work_failed. The projection schema likewise accepts event_type: lease_state, state: qualified, and reason: remote_success. Both pass official Draft 2020-12 validation because terminal work reasons and projection event/state/reason combinations are not bound.

These contradictions are externally visible metadata and undermine the fail-closed status contract. Add state-specific work reasons and event-specific projection ID/state/reason constraints (or one normative semantic validator), with exhaustive accepted/rejected combinations.

Validation performed

  • Remote head re-verified as 4d3ecc35393f3810ab54e4c2bd48d6da69f7cb76 immediately before publication; worktree remained clean.
  • 18 focused Operator tests pass under unittest.
  • Official Draft 2020-12 check_schema passes, all 14 accepted fixtures and every required recovery record validate, and the adversarial examples above are independently schema-valid where applicable.
  • Ruff, privacy scan, documentation lifecycle (141/141), package-manifest check, and git diff --check pass.
  • Fresh wheel and sdist builds contain the semantic module, both schemas, the fixture catalog, and canonical document.

needs-codex-audit remains in place pending a new exact-head audit.

@github-actions

github-actions Bot commented Sep 21, 2026

Copy link
Copy Markdown

Claude audit (merge-authority lane)

Head SHA: 4d3ecc35393f3810ab54e4c2bd48d6da69f7cb76
Verdict: BLOCKED
Review details remain in the local audit artifact.
Publication workflow: .github/workflows/local-audit-publication.yml at 052d49452605eaf668fb07f385673fe5e25b303f

@github-actions

github-actions Bot commented Sep 21, 2026

Copy link
Copy Markdown

Claude audit (merge-authority lane)

Head SHA: 531c4530680e53fcf9855a4e6aef4ac2cc2f6f54
Verdict: PASS
Review details remain in the local audit artifact.
Publication workflow: .github/workflows/local-audit-publication.yml at 052d49452605eaf668fb07f385673fe5e25b303f

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

v1.7.0 contract: freeze Operator authority, state, recovery, and privacy

1 participant