[RFC] Archetypes for long-running tasks: name Oracle/Generator/Executor/Gate onto the staged convergence layer

[clubanderson]

Hold — design discussion, not a task. This is a proposal for maintainer
discussion. Please do not implement it, decompose it into sub-issues, or relay it
to contributors until the design is settled. The hold label should stay applied.

Status: Proposal
Scope: pkg/convergence rollout framing, pkg/worksource adapter surface, pkg/planning vocabulary, progress/burndown UX
Out of scope (deferred): implementing a wavefront adapter, wiring the staged convergence packages, any crustify-side change
Prior art / related: #4246, #4263 (convergence rollout), #7281 (staged-and-unwired marker), #6111 (per-repo ACMM)

Summary

Adopt an explicit archetype vocabulary — Oracle, Generator, Executor, Gate — as hive's public way of describing any long-running task, so that work which is not "GitHub issue → PR → merge" can still get hive's checkpointing, burndown, and progress UX for free.

The proposal is not to build new machinery. Hive already has all four archetypes; three of them are sitting in pkg/convergence staged and deliberately unwired. The proposal is to (1) name them, (2) make the naming the acceptance criterion for finishing the convergence rollout, and (3) prove the vocabulary is genuinely general by driving one non-PR workload through it.

Motivation

This came out of a question from Nick Mitchell about crustify-rs/crustify — an LLM harness for migrating C/C++ to Rust — framed as: could crustify either (a) adopt hive's archetypes so that crustify management becomes just hive management, or (b) could hive learn from crustify's Oracle/Generator/Executor/Gate archetypes, so any complex task can be expressed in archetypal terms and gets checkpointing, burndown and UX tracking?

Two findings reframe the question.

Finding 1: the four archetypes do not exist in crustify's code

Read in full at 90b6d4890f6b6584720c001bd2d23d204e0589af (106 files, ~7.7k LOC). There is no Oracle/Generator/Executor/Gate trait, class, or Protocol. The complete class inventory is Layout, ClaudeCliBackend, CodexCliBackend, Backend, InstrumentSpec, AuditAgent, DriverUnavailable, Route, AgentLog, SessionLog, TranslateAgent, SkillSpec, CrustifyAgent, SessionBase, ArtifactStore, Node, Unit, Batch, Stage. The only abstract type is Backend (src/crustify/agents/backends/__init__.py:26); there are zero uses of ABC/abstractmethod.

What the words denote today:

Term	What it actually is
Oracle	Two passive data sources: the external wavefront CodeQL tool, and crates.json — "the placement oracle" (src/crustify/crates.py:3)
Generator	TranslateAgent, the only subclass of CrustifyAgent (src/crustify/agents/translate.py:46)
Executor	Free functions _schedule.run / _isolated_step, plus stdlib ThreadPoolExecutor (src/crustify/_schedule.py:152-228)
Gate	Shell commands declared in specs/build.json, enforced by prompt text and exit codes

So the archetypes are an aspirational vocabulary, not an implementation. That is the good news: nothing has to be migrated away from, and the naming is still free.

Finding 2: the asymmetry is state, not vocabulary

Crustify is deliberately stateless. From src/crustify/artifact_store.py:4-16:

"Stage completion is data-driven: agents check for the existence of their output artifacts... There is no state.json — each stage's done-ness is the existence of its on-disk artifact."

The consequences are exactly the things Nick wants:

Concern	crustify	hive
Unit of work	a C type or free symbol, packed into a Batch	beads.Bead with DependsOn (src/pkg/beads/beads.go:107)
Burndown	// crustify:todo anchors in Rust source, replaced by // Wraps: / // Replaces: (src/crustify/anchors.py:14-33)	Store.Ready(actor) over a dependency DAG (src/pkg/beads/beads.go:580)
Checkpoint / resume	none — checkpoint() is a log line, "accounting only" (src/crustify/agentlog.py:96-100); a translate run cannot be resumed	durable bead store + snapshot state
Decomposition	performed outside the repo by wavefront schedule	planning.Decompose: epic → ordered child-bead DAG
Gate verdict	an exception list; the command exits non-zero	tri-state admission decision
UX	CLI text, dry-run wave listing, cost report	dashboard, /api/plans, TUI

Therefore option (a) is backwards. Crustify is not missing archetypes; it is missing durable work-item state — which is precisely what hive is. Option (b) is the right direction, and it is also the direction hive is already heading for its own reasons.

Proposal

1. Name the four archetypes, mapping each to code that already exists

• Oracle → worksource.WorkSource (src/pkg/worksource/worksource.go). Already the "Step 01 seam" that enumerates source-neutral work items with dependency edges, already multi-backend (GitHub Issues, GitHub Projects, Linear, Jira). An oracle like wavefront schedule becomes one more adapter, not a special case.
• Generator → the existing agent / lane / role model. Unchanged.
• Executor → convergence/mutation.Executor (src/pkg/convergence/mutation/executor.go:29-45) — claim ledger + operation journal + epoch fencing around a single EffectFunc, where an error means the effect is UNCERTAIN and must be reconciled before any retry.
• Gate → convergence.Evaluate (src/pkg/convergence/convergence.go) — a pure, level-triggered admission judgment whose verdict is deliberately tri-state: True / False / Unknown.

Unknown is the load-bearing part and the single biggest thing hive has that crustify does not. Per the package doc, it never authorises a mutating transition and never blocks anything beyond the affected candidate, so a duplicate, out-of-order, or entirely missed event cannot leave admission wedged — the next evaluation recomputes from current source state. That is the property a long-running task needs most, and an exception list cannot express it.

Two further pieces have no crustify analogue at all and should be named as part of the vocabulary:

• convergence/proof — the assumptions a decision rested on, and the invalidation rules that retire a proof when they stop holding.
• convergence/outcome — the durable record of what a decision predicted versus what actually happened.

2. "Instantiate a Long Running Task in archetypal terms" — already exists, should be made the front door

planning.Decompose (src/pkg/planning/decompose.go) turns an epic bead into an ordered DAG of child beads tagged parent_epic, execution=agent-suitable|human-required, and plan_status=draft. Store.Ready() then hides those children until a human approves the plan via planning.ApprovePlan / POST /api/plan/{epicID}/approve.

That is exactly the requested shape — the model proposes the plan, humans gate it, the deterministic runtime executes it — and it already has UX (/api/plans, /api/plan/{epicID}, /approve, /reject, /child/{childID}). The RFC proposes treating this as the public entry point for any long-running task, rather than a planning-intelligence feature specific to hive's own epics.

3. Make the archetypes the acceptance criterion for the convergence rollout

pkg/convergence/{mutation,proof,outcome} are staged and not wired into any hive binary (#7281), tracked for activation by #4246 / #4263. The marker is enforced, not decorative: TestStagedConvergencePackagesAreMarkedAndUnwired fails if the packages become reachable from a binary while still claiming to be unwired.

The ask is to add one requirement to that rollout: the wired result must be expressible against a workload that produces no pull requests at all. A migration campaign is the natural candidate. If the vocabulary only fits "issue → PR → merge", it is not archetypal and we should find that out before wiring, not after.

What hive should learn from crustify

These are genuine improvements independent of whether any crustify integration ever happens.

1. Progress should live in the artifact, not only in a side database. Anchors-in-source cannot desync from reality; a bead store can. Worth asking what hive's equivalent of a // crustify:todo anchor is for each workload type.
2. Worktree-per-agent isolation. From src/crustify/worktree.py:1-12: "isolation is not a parallelism optimisation, it is what makes an agent's scoped cargo check mean anything." A Gate verdict is only meaningful over an isolated workspace. Verified there at 8 concurrent agents landing by lock-free atomic push with rebase-on-rejection.
3. Refuse to substitute a number. crustify-audit returns DriverUnavailable and counts: null rather than a fallback, because "A number absent is recoverable; a number that looks like crustify's but was produced differently is not" (src/crustify_audit/driver.py:10-17). This is the same instinct as convergence's Unknown, and hive's dashboards and fleet reports should adopt it explicitly.
4. Absolute started_at / ended_at in usage records, not just durations, so overlap is computable — how many agents were genuinely live at once, which a duration alone cannot answer (src/crustify/core/agentlog.py:142-148).
5. Explicit wave barriers as a layer-completion checkpoint, with elapsed-so-far flushed as each layer completes so a killed run still accounts for the layers that finished.
6. A stale-plan provenance guard. _validate_oracle_provenance (src/crustify/wave.py:124-147) SHA-256s the oracle config and aborts if it changed since scheduling. Hive's approved plans have the same hazard: an approved plan_status can outlive the state it was approved against.

Open questions

1. Are Oracle/Generator/Executor/Gate the right four, or does hive need Proof and Outcome as first-class peers rather than supporting cast? Six archetypes is less tidy but matches what the code actually distinguishes.
2. Should worksource.Issue grow a non-issue-shaped sibling, or should a migration unit be coerced into Issue? Coercion keeps steps 03-07 source-agnostic but the naming becomes a lie.
3. Does the plan-approval gate scale to a DAG with thousands of leaves, where per-child human approval is not viable? Probably needs approval at layer granularity.
4. Burndown UX: hive currently answers "what is ready", not "how much is left". What is the minimum needed for a real burndown chart — and should it derive from artifacts rather than bead status?
5. Is there appetite for a wavefront worksource adapter as the proving workload, or is there a better non-PR workload already in the fleet?

Non-goals

• Not proposing any change to crustify.
• Not proposing to wire the staged convergence packages in this RFC — only to constrain what "wired" must mean.
• Not proposing a new scheduler, planner, or router. pkg/convergence explicitly keeps admission judgment separate from decomposition, replanning, governor policy, prioritisation, and lane routing; this RFC preserves that separation.

[joshyorko]

@clubanderson Andy, I like using a genuinely non-PR workload as the acceptance test. After looking more closely at the convergence implementation and Crustify/Wavefront, I think the important question is less “do we have the four archetypes?” and more “what exact contract lets an external workload safely consume the machinery?”

I would keep the four names provisionally, but not treat their package mappings as completed interfaces. WorkSource still describes tracker-originated work against a repository. More importantly, the proof validator requires a real PR and GitHub-checks evidence, and the mutation effect validator accepts only PR creation. Those are reasonable boundaries for the first vertical, but a Wavefront adapter alone cannot make the proof/effect path non-PR.

I would separate admission, verification, and publication rather than draw one Gate in a fixed pipeline. An agent can be authorized to generate a candidate in an isolated workspace; compiler/audit/review evidence establishes particular predicates; promotion needs another current decision against the exact candidate and target.

The loop should close through re-observation, but re-observation alone is not writer fencing. “The artifact is absent” does not prove an old worker cannot publish it later. The adapter needs a defined conditional-write, idempotency, fencing, or termination contract. The #4002 handoff investigation is relevant because it already identifies overlapping journals and ownership questions; I would avoid building a third mechanism here.

Wavefront should remain authoritative for its semantic graph. It already owns dependency closure, cycle handling, waves, and batching. Hive should be able to validate and admit an imported versioned plan without requiring an LLM to redecompose it. Internal beads are fine as an execution ledger; fabricated GitHub issues and PRs should not be required.

For approval, bind authorization to a plan revision, allowed scope/effects, and load-bearing assumptions—not every leaf. But expected output changes must not invalidate the entire campaign after each successful batch.

For progress, distinguish current satisfied obligations, known remaining work, unknown evidence, and scope added or retired. A wrapped symbol, a ported symbol, and a reviewed/promoted symbol are not interchangeable “done” counts.

The first proving run I would ask for is a small migration fixture without GitHub credentials, exercising crashes around the effect boundary, a late old worker, stale evidence, partial discovery, and unrelated ready work beside a blocked dependency. Passing that would demonstrate a meaningful non-PR contract—not universal support for every long-running task, which is fine.

I would keep this hold-gated and settle that bounded adapter/effect/proof contract before wiring or freezing the public vocabulary.

The source I checked against is Hive v4 92e0056, with a spot-check of the executor/proof surfaces on v5 9deda34. This is source/component evidence, not a claim about a live Hive failure.

[Danathar]

This makes a lot more sense to me when I map it to the kind of work I’m already doing with Hive (I think?).

A possible example is the large audit/campaign work I’ve been doing: full repo bug/security sweeps, reviewing the contributor relay and backend for issues, validating findings, checking for duplicates, and then filing good issues for contributors to pick up.

Today that usually lives inside one very large agent session (and replicating that process after I close my window is a pain when I forgot what I typed last time) . The model has to remember what it inspected, what it found, what was disproven, what still needs verification, and what has already been filed. If the context window ends, the model changes, or the work moves to another contributor, a lot of that continuity has to be reconstructed.

That is where I see the real value in #7620, if I'm reading it right. I'm still trying to digest it.

For that sort of workload, the Oracle could define the units or subsystems that still need inspection, the Generator could be whatever model is doing the analysis, verification could establish whether a finding is real and not a duplicate, the Gate could decide whether the evidence is sufficient, and the Executor could file the issue exactly once. Hive would own the durable state and progress instead of the model session owning it.

This also connects directly to the idea I raised about contributed ClankeRs helping when an operator runs short on model budget #7629 (maybe?). If the campaign state is durable (lives somewhere persistent) and ownership/effects are properly fenced, work could move from one contributor agent to another without depending on the previous agent’s context, while also preventing an old worker from waking up later and publishing stale or duplicate work.

The same applies to the Bluefin triage work I do now: finding actionable issues with no PR, deciding what is suitable for an agent, reviewing the result, and moving on to the next item. A lot of that is effectively a manually maintained work graph already.

So for me, the interesting part of this RFC is less the four names and more whether Hive can make the long-running campaign itself the durable object: known work, remaining work, blocked work, uncertain evidence, verified findings, completed effects, and progress/burndown, independent of any one model session or GitHub PR lifecycle.

A concrete proving workload could be something like “audit these repos/components, validate findings, and file issues.” That is real work Hive users are already doing, it can span many agents, it needs idempotency/fencing, and its output is not fundamentally a PR pipeline.

[joshyorko]

@Danathar @clubanderson — the audit example makes this much more concrete. I think it’s a credible first non-PR workload because the durable object becomes the campaign itself: what was inspected, what was disproven, what is still uncertain, and what has already been published.

A few boundaries I’d want to preserve:

1. Inspection, finding, and publication should be separate state. A component can be fully inspected and produce no findings. A finding can be valid without being published. And filing an issue does not mean the underlying condition is fixed. That also gives burndown a sane meaning: completion of the approved inspection scope, not “percentage of bugs found.”

2. “Exactly once” is really two problems. Retrying one publication must not create another issue, but two workers independently discovering the same defect also need semantic deduplication. An operation ID solves the first, not necessarily the second. Hive already has useful issue dedupe/rejected-twin machinery, but title/file heuristics should not become the universal definition of finding identity. An uncertain create must remain Unknown until reconciled rather than blindly retrying.

3. Verification and permission to publish are different gates. “Another model agreed” is not enough by itself; the campaign should preserve what evidence was actually checked and against which revision. Security findings make this especially obvious: Hive’s own policy requires private disclosure, so a technically confirmed finding cannot automatically mean “open a public issue.”

4. [RFC] [v6] Standby contributors: a lane paused for budget hands its queue to volunteers, behind a model floor #7629 fits, but task takeover and in-flight handoff are different guarantees. Reassigning untouched work is easy. Resuming a checkpoint is harder. Taking over work with an uncertain external effect is harder again. I’d start with the first two and make the third an explicit recovery test rather than assume durable storage solves it.

For the pilot, I’d actually start with a report-only audit over a pinned scope, then optionally add an authorized issue publisher. That lets us test clean inspections, disproven findings, handoff, stale evidence, duplicate findings, and lost publication responses without making GitHub issue creation the definition of progress.

The success metric also shouldn’t be “more issues filed.” It should be less repeated investigation and less avoidable reviewer work without losing evidence or creating duplicate effects.

Andy, I’d be interested in using this bounded audit campaign as one proving workload and keeping Crustify/Wavefront as the second, more domain-structured one. The two would stress different parts of the same convergence contract.