Skip to content

Harden recovery, posture checks, and replay handling - #24

Merged
itswl merged 6 commits into
mainfrom
fix/agent-recovery-safety
Oct 27, 2026
Merged

itswl merged 6 commits into
mainfrom
fix/agent-recovery-safety

Conversation

@itswl

@itswl itswl commented Oct 27, 2026

Copy link
Copy Markdown
Owner

Summary

  • Persist remediation step checkpoints and report interrupted outcomes as completed, unknown, or not started.
  • Bound and redact captured output; omit oversized output to avoid leaking secrets cut at the truncation boundary.
  • Report incomplete credential measurements honestly and reject declared writing postures that cannot be verified.
  • Serialize identical signed requests with renewable SQLite leases and prevent concurrent requests from both entering the pipeline.
  • Document recovery limits and add regression tests for AWS simulation failures, lease expiry/concurrency, and interrupted execution.

Verification

  • bash scripts/gate.sh — passed, including all component tests, cross-service checks, mutation checks, and Docker builds.
  • Estate identifier scan was skipped because .estate-identifiers is not present locally. CI may enforce this check using the repository secret.

Limitations

A crash after an external side effect but before its completion record can still result in that side effect being repeated after lease expiry. Pattern-based redaction cannot guarantee removal of every secret.

itswl added 5 commits October 27, 2026 14:35
…ethods nothing calls go instead

The first version deleted 24 comment lines from the inbound handler to fit the 6,650-line ceiling, including reasoning this change does not touch: the storm fuse's two stages, why the adapter parses before the storm path, why a failure leaves the sender's retry processable. They are back, and the replay comment now says the same thing in terms of the lease. signature_seen and claim_signature had no caller left; they go, their three tests ask the same questions of acquire/complete/release, and security.py's docstrings name the lease instead of the removed method. The pipe stays inside its ceiling (6,650 of 6,650) without raising it.
With nothing naming an AWS credential, the first version reported unverifiable unless AWS_EC2_METADATA_DISABLED=true. On a laptop and on the probes' internal network, where no metadata service can exist, that turned the quickstart's no-credentials and a read-only node's readonly-confirmed into unverifiable, paged at every start, and made a writing node with a declared radius refuse to start. No compose file set the variable.

Now the service is asked directly, one second and no proxy (the egress proxy answers for addresses it refuses): one that answers is measured like any other source, none answering is no credential, and the CLI's 'Unable to locate credentials' is an absence. A profile name with no file behind it is no longer a credential source. A shared test fixture keeps every test off the network, because a cloud CI runner has a metadata service at the same address. Each behaviour has a test that fails without it.
…characters

The first version omitted any output past 10,000 bytes, tail included, to avoid storing a secret the cut split in half. Redacting the whole output before cutting gives the same guarantee — the redactor sees every secret whole — and keeps the tail, where a failing command says why. Past a megabyte nothing is held and the output is omitted, so memory stays bounded. Tested with a secret straddling the cut and with a 1.1 MB flood; the straddle test fails when the cut comes first.
…note records the amendments and the rollback line

The README and the containment page described the assumed metadata credentials and the omitted output of the first version. The decision note says what was amended in review and why, and adds the rollback consequence of the new pending_signatures table: an earlier pipe ignores it, and a lease left in it expires.
@itswl

itswl commented Oct 27, 2026

Copy link
Copy Markdown
Owner Author

Review and amendments by Claude (claude-opus-5-5), at the repository owner's request.

The remediation checkpoints, the signature lease, the non-finite timestamp check and the stricter AWS simulation parsing are kept as written. Three things were amended on this branch before merging.

  1. Instance metadata is asked, not assumed. With nothing naming an AWS credential, the first version reported unverifiable unless AWS_EC2_METADATA_DISABLED=true. Run on four shapes, main against this PR as opened: the quickstart with no credentials went from no-credentials to unverifiable; a read-only watch node from readonly-confirmed to unverifiable, paging at start; the work runner with a profile name and no credentials file from recorded to unverifiable, paging; and a writing node with a declared radius from starting to refusing to start. That was on a laptop and on an internal network, where no metadata service can exist, and no compose file sets the variable. Now the service is asked directly, one second and no proxy: one that answers is measured like any other source, none answering is no credential, and the CLI's Unable to locate credentials is an absence. A profile name with no file behind it is no longer a credential source. The four shapes read as they do on main again.
  2. A step's output is redacted whole, then cut to its last 10,000 characters, instead of omitted past 10,000 bytes. The tail is where a failing command says why, and a secret the cut would split is seen whole by the redactor. Past a megabyte the output is omitted, as before.
  3. The pipe's explanatory comments are back: the replay order, now in terms of the lease; the storm fuse's two stages; why the adapter parses before the storm path; why a failure leaves the retry processable. They had been cut to fit the 6,650-line ceiling. The two signature methods nothing calls any more (signature_seen, claim_signature) went instead, their tests moved to the lease API, and security.py's docstrings name the lease. The pipe stays inside its ceiling without raising it.

Each amendment has a test that fails without it. Also run: the estate-identifier scan this PR's description says was skipped (clean), and the full gate (STACK GATE GREEN). The decision note records the amendments and the rollback consequence of the new pending_signatures table.

Not changed, worth a follow-up: a failed lease renewal cancels the request mid-pipeline rather than letting complete_signature answer 503; the lease is taken, a write, before the in-memory fuse check, so a bulk replay of one capture costs a write per copy; the lease length is fixed at 120 seconds.

… another vendor's answer is not

CI's image boot and stack smoke refused to start the investigator: GitHub's runners are Azure VMs, Azure's metadata service answers at the same address, and the probe counted any answer as an AWS service. The image has no aws CLI, so the readonly node was unmeasurable and refused under enforce — what a deployment on any other vendor's VM would have hit too.

The probe now asks what the AWS CLI asks: a token (IMDSv2), then the role listing, with the token when there is one. A role name is a credential; a 404 is an instance with no role; any other answer is no AWS service; nothing at the address is nothing. Tested against a stand-in answering as IMDSv2 with a role, IMDSv1 with a role, an instance with no role, and another vendor's service, with a bogus proxy in the environment; counting any answer, or asking through the proxy, fails those tests. Renamed metadata_reachable to metadata_offers_credentials, which is the question it answers.
@itswl

itswl commented Oct 27, 2026

Copy link
Copy Markdown
Owner Author

Follow-up by Claude (claude-opus-5-5). CI caught a hole in amendment 1: GitHub's runners are Azure VMs, Azure's metadata service answers at the same address, and the probe counted any answer as AWS, so the image without an aws CLI refused to start there, as it would on any other vendor's VM. The probe now asks what the AWS CLI asks: a token (IMDSv2), then the role listing. A role name is a credential; a 404 is an instance with no role; any other answer is no AWS service; nothing at the address is nothing. Tested against a stand-in answering as IMDSv2 and IMDSv1 with a role, with no role, and as another vendor's service, with a bogus proxy in the environment. Renamed to metadata_offers_credentials.

@itswl
itswl merged commit 0c129c8 into main Oct 27, 2026
10 checks passed
@itswl
itswl deleted the fix/agent-recovery-safety branch October 27, 2026 07:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant