Skip to content

Fail closed on hook failures - #1097

Draft
shunichironomura wants to merge 4 commits into
mainfrom
hook-outcome-fail-closed
Draft

shunichironomura wants to merge 4 commits into
mainfrom
hook-outcome-fail-closed

Conversation

@shunichironomura

@shunichironomura shunichironomura commented Jul 7, 2026

Copy link
Copy Markdown
Member

Warning

We should not merge this PR until #1099 is merged.

Summary

This PR changes hook execution from a "best effort / fail open" model to a "fail closed after collecting diagnostics" model.

Internally, it introduces HookOutcome<T> so hooks can distinguish:

  • Ok(HookOutcome::Succeeded(output)): the hook ran and its configured success condition passed
  • Ok(HookOutcome::Failed { output, reason }): the hook ran and captured useful output, but its configured success condition failed
  • Err(error): the hook could not complete because of an operational error

Hook::run now returns CapsulaResult<HookOutcome<Self::Output>>, while Captured is only responsible for JSON serialization. All in-tree hooks, including the newly added capture-json and capture-toml hooks, use the new outcome API. The server preserves policy-failure reasons through upload, database storage, API responses, and the run-detail UI.

AI assistance

  • AI used: yes — AI coding assistants helped implement and test the changes and update this PR description.
  • Human review of AI-assisted code: pending

Breaking changes

  • No breaking changes
  • Breaking changes; the breaking change label is applied

User-facing behavior changes

Pre-run hook failures now stop the wrapped command

Before this PR, hook errors were recorded in pre-run.json as success: false, but the wrapped command still ran. That meant safety gates could fail open.

After this PR:

  1. Capsula still runs all configured pre-run hooks to collect as much context as possible.
  2. Capsula writes pre-run.json with every hook result.
  3. If any pre-run hook fails or errors, Capsula exits with the hook-failure exit code before running the wrapped command.

This affects both policy failures and operational errors:

  • dirty git repo when capture-git-repo.allow_dirty = false
  • unpushed git commit when capture-git-repo.require_pushed = true
  • a pre-run hook returning an error, such as a missing guard command or failed artifact-dir creation

Post-run hook failures now make Capsula fail after recording results

Post-run hooks cannot stop the wrapped command because it has already run. They now fail closed at the Capsula layer instead:

  1. Capsula still runs all configured post-run hooks.
  2. Capsula writes post-run.json with every hook result.
  3. If any post-run hook fails or errors:
    • capsula run exits with the hook-failure exit code when the wrapped command succeeded.
    • capsula run preserves the wrapped command's non-zero exit code when the wrapped command already failed.
    • capsula run-end exits with the hook-failure exit code after writing post-run.json.

This makes post-run capture/notification failures visible to automation without masking an already-failing wrapped command.

Hook policy failures are now visible in hook metadata

When a hook completes but its configured condition fails, the hook output is still recorded, but __meta.success is now false and __meta.failure_reason explains why.

For example, a dirty repo with allow_dirty = false is recorded as a failed hook outcome rather than a successful capture with an out-of-band abort flag.

Operational hook errors continue to be recorded with __meta.error.

capture-command now treats non-zero exit status as failure by default

Before this PR, capture-command only requested abort on non-zero status when abort_on_failure = true.

After this PR, the default successful status set is [0]. A non-zero status is therefore a hook failure unless configured otherwise.

To intentionally check for a failing command, configure the expected status explicitly:

[[pre-run.hooks]]
id = "capture-command"
command = ["test", "-f", "missing-file.txt"]
success_codes = [1]

For compatibility, existing configs that explicitly set abort_on_failure = false still accept any exit status. New configs should prefer success_codes.

Notes for hook developers

Hook implementations now need to return CapsulaResult<HookOutcome<Self::Output>> from Hook::run.

Use the three outcome categories intentionally:

  • Return Ok(HookOutcome::success(output)) when the hook completed and the captured state satisfies the hook's configured policy.
  • Return Ok(HookOutcome::failure(output, reason)) when the hook completed and produced useful output, but the captured state violates the hook's configured policy.
    • Example: capture-git-repo successfully captured repo state, but allow_dirty = false and the repo is dirty.
    • Example: capture-command successfully captured stdout/stderr/status, but the exit status is not in success_codes.
  • Return Err(error) when the hook itself could not complete the operation it was asked to perform.
    • Example: command spawn failure, missing artifact directory, unreadable file, libgit2 operation failure, Slack API failure.

Captured no longer has abort_requested(). Abort/failure policy should be expressed through HookOutcome::failure(...), not hidden inside captured output.

The orchestrator will still attempt every hook in the phase. Hook developers do not need to short-circuit other hooks; they only need to classify their own result correctly.

For hook configuration design, prefer explicit success criteria over ad-hoc abort booleans. For example, capture-command now uses success_codes so expected non-zero statuses can be represented directly.

Notes for reviewers

  • This is intentionally both an internal API refactor and a user-facing behavior change.
  • Hooks in both phases are still all attempted before Capsula decides the phase failed.
  • capsula run does not mask a wrapped command's non-zero exit code with a post-run hook failure code.

Tests

  • just lint
  • just test

Closes #1062

@codecov-commenter

codecov-commenter commented Jul 7, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 86.33880% with 25 lines in your changes missing coverage. Please review.
✅ Project coverage is 63.31%. Comparing base (43ead84) to head (0d7ac32).

Files with missing lines Patch % Lines
crates/capsula-orchestration/src/hooks.rs 82.00% 9 Missing ⚠️
crates/capsula-tui/src/app.rs 0.00% 5 Missing ⚠️
crates/capsula-notify-slack/src/lib.rs 0.00% 4 Missing ⚠️
crates/capsula-capture-command/src/lib.rs 89.65% 3 Missing ⚠️
crates/capsula-cli/src/main.rs 87.50% 2 Missing ⚠️
crates/capsula-server/src/lib.rs 33.33% 2 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main    #1097      +/-   ##
==========================================
+ Coverage   62.49%   63.31%   +0.81%     
==========================================
  Files          45       44       -1     
  Lines        4994     5097     +103     
==========================================
+ Hits         3121     3227     +106     
+ Misses       1873     1870       -3     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@shunichironomura

Copy link
Copy Markdown
Member Author

@shunsuke-shimomura FYI

@shunsuke-shimomura

Copy link
Copy Markdown
Member

Heads-up on the server side: capsula-server's HookMeta in crates/capsula-server/src/models.rs does not have a failure_reason field yet, so the new __meta.failure_reason emitted by this PR will be silently dropped when the server ingests pre-run.json / post-run.json.

__meta.error (operational error, Err case) is already covered, but failure_reason (policy-violation case, Ok(HookOutcome::Failed { reason })) is a distinct field carrying information the server currently cannot surface — e.g. "aborted because repo was dirty" vs "aborted because git tag creation errored" would look the same in the UI/API without it.

Suggest adding:

pub struct HookMeta {
    pub id: String,
    pub config: Option<JsonValue>,
    pub success: bool,
    pub error: Option<String>,
    pub failure_reason: Option<String>, // new
}

and threading it through RunOutputRow / the DB schema / the response types so the failure reason survives the round-trip. Doesn't need to block this PR, but worth a follow-up so the extra diagnostic isn't lost on the server.

@shunichironomura
shunichironomura marked this pull request as draft July 7, 2026 13:47
@shunichironomura
shunichironomura force-pushed the hook-outcome-fail-closed branch from 93a5223 to 085d153 Compare July 7, 2026 14:24
@shunichironomura
shunichironomura force-pushed the hook-outcome-fail-closed branch 3 times, most recently from e36bd14 to 6a82a85 Compare July 22, 2026 06:49
@shunichironomura
shunichironomura force-pushed the hook-outcome-fail-closed branch 2 times, most recently from 985d14e to 7d291d9 Compare July 22, 2026 22:53
@shunichironomura
shunichironomura force-pushed the hook-outcome-fail-closed branch 2 times, most recently from 0df4332 to 709e249 Compare September 16, 2026 06:17

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Abort gates fail open: a hook *error* never aborts the run (allow_dirty/require_pushed/abort_on_failure bypassed)

3 participants