Successful Deep Scan completion promotes findings from unconsumed worker checkpoints

[msapelov]

Summary

A successfully completed Deep Scan can promote findings from incomplete discovery artifacts belonging to workers whose results were never consumed by a successful reducer.

For example:

final successful reducer:
  [A, B]

unconsumed discovery worker:
  status: canceled
  merge_state: none
  complete: false
  checkpoint findings: [C]

sealed result:
  [A, B, C]

The final semantic reducer selected A and B, but saved-result reconciliation later promotes C into the top-level report.

Preserving partial worker output is useful. The issue is that, after a successful Deep Scan already has a valid final reducer result, an unconsumed partial finding can bypass that reducer and become an authoritative reportable finding.

Observed example

I originally observed this in a completed Deep Scan using bundled plugin 0.1.37:

• 36 discovery workers completed successfully and were consumed by reducers;
• four other discovery workers terminated on replaceable policy_refusal failures and were never consumed by a successful reducer;
• the final successful reducer contained 45 findings;
• those four unconsumed workers had produced 18 incomplete finding records;
• the sealed report contained 63 findings.

The 18-record difference was attributable to those incomplete worker artifacts.

Current main removes an unvalidated worker's current output/result.json, but the underlying issue remains reproducible at 233186073adb45c188a04f570202cc2c766599b0 using current or archived checkpoints, or an archived incomplete result.

The deterministic current-main regression produced:

Artifact containing C	Current output/result.json	Sealed result
Current checkpoint	absent	[A, B, C]
Archived checkpoint	absent	[A, B, C]
Archived incomplete result	absent	[A, B, C]

The corresponding failed, interrupted, and canceled parent controls also retained [A, B, C], as intended.

All six deterministic cases passed without model calls.

Current behavior

Current main removes the current result when discovery validation does not succeed:

if (!discoveryValidated) {
  await fs.rm(files.resultPath, { force: true });
}

worker-runner.ts:197-208

However, this cleanup does not remove the worker's current or archived checkpoints.

During successful completion, all Deep Scan worker rows are passed to saved-result reconciliation:

workbench_db.py:1528-1548

For discovery workers, reconciliation considers current and archived result/checkpoint artifacts without first requiring that the worker:

status == succeeded
merge_state == merged

or otherwise establishing that its result was consumed by a successful reducer:

workbench_saved_results.py:304-360

An archived or checkpoint source is suppressed only when the same worker has a successfully read current complete result.

Because an unvalidated canceled worker's current result has been removed, no such complete source exists and its surviving checkpoint is not considered superseded:

workbench_saved_results.py:569-598

If an incomplete artifact contains a valid finding whose saved-result finding key or worker-local candidate representation is not already represented by the canonical parent, that finding can be appended as a new top-level finding:

workbench_saved_results.py:620-738

So an otherwise successful Deep Scan can end with a sealed finding set that differs from the finding set produced by its final successful reducer.

Why this matters

The recovered record still passes normal saved-result and finalization validation.

What it has not passed through is the Deep semantic reducer that reconciles findings across discovery workers.

That means the additional finding has not participated in the reducer's cross-finding decisions around:

• semantic duplication;
• remediation subsumption;
• canonical identity selection;
• overlapping evidence;
• independently reachable instances.

The partial finding does not need to be discarded. It can remain available as saved worker evidence, diagnostic output, or deferred/incomplete coverage.

It just should not silently become a new canonical top-level finding after successful semantic reduction has finished.

Expected behavior

When a parent Deep Scan completes successfully with a valid final reducer:

1. The final reducer should remain authoritative for the top-level finding set.
2. Worker artifacts may still preserve evidence, provenance, and history for findings represented by that reducer.
3. Unmatched findings from canceled, replaceable, omitted, or otherwise unconsumed workers should remain inspectable, but should not create new top-level findings.

For a failed, canceled, or interrupted parent scan, the current loss-preserving behavior should remain. In that case there may be no completed canonical reducer result, so preserving partial worker findings as recoverable output is useful.

The intended distinction is:

successful parent + valid final reducer:
  unconsumed partial finding C
  -> preserve as partial evidence
  -> do not promote top-level

failed / canceled / interrupted parent:
  partial finding C
  -> preserve in recoverable partial output

Regression coverage

A deterministic regression could use a successful parent with canonical findings A and B, plus a canceled/unconsumed discovery worker whose only remaining artifact contains C.

Expected:

canonical reducer: [A, B]
sealed result:     [A, B]

The test should cover:

• current checkpoints;
• archived checkpoints;
• archived incomplete results.

Corresponding stopped-parent tests should continue producing:

[A, B, C]

to preserve the loss-recovery guarantees introduced by PR #617.

Related

PR #617 fixed a neighboring successful-completion case where findings from superseded checkpoints could reappear in the final canonical result.

This appears to be the remaining worker-level case: incomplete artifacts from an unconsumed discovery worker can still introduce new top-level findings after a successful final reducer has completed.