Skip to content

✨ workflow models P1+P2: schema, compiler, DeriveFromModel, store, workflows API, repo bindings, tracker columns - #17648

Merged
clubanderson merged 8 commits into
v6from
workflowmodel/p1-compiler
Nov 11, 2026
Merged

clubanderson merged 8 commits into
v6from
workflowmodel/p1-compiler

Conversation

@clubanderson

@clubanderson clubanderson commented Nov 11, 2026 •

Copy link
Copy Markdown
Member

RFC #17620, phases 1 and 2 of 7 (#17649 was merge-trained into this branch, so this PR now carries both). Base v6.

What this adds

  • Schema (lifecycle half) — workflow.hivecommons.dev/v1alpha1 gains columns, derivation, events, gates, roles, agents, checks, bookkeeping, labels, meta, and per-state lifecycle {column, number, kinds, waiting, replies, confidence, citations}. Existing fields and goldens untouched.
  • Compiler — workflowmodel.Compile(m) (*Profile, []Diagnostic, error). Refuses with coded diagnostics: state-no-exit, gate-no-owner, agent-no-human-gate, duplicate-event-fact, state-no-column / unknown-column, duplicate-stage-number, unknown-fact / bad-op, no-rules, no-version, plus unknown state/role/gate/event references. confidence: inferred elements compile with a warning.
  • hive-default is generated, not hand-written — pkg/workflowmodel/hivedefault.Generate() builds it from stagelabels.Table + stageengine.DiagramEdges() + the monitor's column anchors (Open · Actionable · Claimed · PR in progress · CI / Review · Merge queue · Merged). 23 states, ~70 transitions, compiles clean.
  • stageengine.DeriveFromModel(profile, kind) — the third Deriver. hive-default declares derivation: builtin and delegates to the existing derivers, checking only coverage. Custom models declare derivation: rules: ordered first-match predicates over the fact vocabulary (workflowmodel.Facts); unmatched → Abstain workflow: not covered. Options.Workflow *Profile selects it.
  • workflow.enabled config flag, default false. No wiring effect yet (P2 binds repos).

Design decision

hive-default derivation stays in Go. That is what makes the P1 bar — goldens byte-for-byte — hold without a second state machine. Documented in src/docs/design/workflow-models.md.

Tests

  • TestDeriveFromModelHiveDefaultParity: every PR stage golden + an issue sweep derive to identical Decisions under built-in and under hive-default.
  • TestCompileRefusals: 24-case table, one per refusal code, on the hand-written testdata/models/techdocs.json (cncf/techdocs engagement model, derivation: rules).
  • TestDeriveFromModelRules walks a TechDocs engagement through the rules evaluator; TestRulesFactVocabularyIsComplete proves every fact in the vocabulary evaluates.
  • hivedefault: every Table stage has a state/number/column, columns match tracker-row-anchor.js, edges mirror the engine, JSON round-trip keeps the hash.

Coverage: workflowmodel 91.4%, hivedefault 95.4%, stageengine 90.2%. golangci-lint clean.

v5/v6 impact

None for any hive: flag off, no repo bound, built-in derivers untouched. pkg/config golden regenerated for the new workflow key.

Reopened against v6 after #17630 merged (GitHub auto-closed #17639 when its stacked base branch was deleted). Same single commit, cherry-picked onto v6.


Phase 2 (from #17649)

What's here

  • Store — pkg/workflowmodel/store: <data>/workflows/<name>/<version>.json + index.json. hive-default virtual/read-only; drafts save but Profile() → ErrNotCompiled; name/version regex guards; atomic writes; Rebuild. Entries reserve adoption / ratings / benchmarks (raw JSON, carried verbatim) for the P7 marketplace and future rate/vote/benchmark work.
  • Binding — project.workflow_bindings: [{repo, name, version}] (mirrors repo_policies), WorkflowBindingFor / SetWorkflowBinding, validated in ValidateWithOptions. Bind requires a compiling model; deleting a bound model is 409. data.workflows_dir override.
  • API — GET/POST /api/workflows, POST /api/workflows/compile, GET/PUT /api/workflows/bindings, GET/PUT/DELETE /api/workflows/{name}/{version}. Owner-gated writes, audited, 503 without a store. OpenAPI updated.
  • Engine — stageengine.Options.WorkflowFor(owner, repo) + DeriveForRepo (hash-cached per-profile deriver). Bound repos derive via DeriveFromModel; unbound/flag-off take the builtin path unchanged. Unusable binding warns once and falls back.
  • Tracker — response gains workflows: {default, repos: {<repo>: {name, version, columns, numbers, error}}} (nil when nothing is bound). JS: model badge, model column header, per-row column strip by stage number, amber Not covered chevron on abstention.
  • Settings → Repos — per-repo Workflow <select>, immediate PUT to bindings.

Tests / gates

  • store 92.4%, api_workflows_test.go, stage_engine_workflow_test.go (live rebind, fallback, warn-once), node-harness stage_tracker_workflows_ui_test.go.
  • OpenAPI route coverage, import ratchet (67→68), settings-parity allowlist, agent-runtime golden regenerated, tracker CSS token check — all green. golangci-lint clean.
  • Pre-existing macOS-only /var-symlink failures (TestKnowledgeDrainLegacyMemory, memory-drainer tests) are unrelated.

Not in this PR

P3 (ingest → lifecycle draft models), P4 (visual editor), P5 (monitor CI sub-stages), P6 (marketplace on hive.hivecommons.dev), P7 (benchmark/ratings). Tracked on #17620.

@kubestellar-prow

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign kellyaa for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@github-actions

github-actions Bot commented Nov 11, 2026 •

Copy link
Copy Markdown
Contributor

Open PR overlap check

Base branch: v6

Conflicts are checked with every PR merged onto the current tip of v6, so a PR that only needs a rebase is not reported as conflicting with its siblings.

No conflicting open PRs or same-file overlaps were found.

@github-actions

github-actions Bot commented Nov 11, 2026 •

Copy link
Copy Markdown
Contributor

Mutation testing (advisory)

Head 2ceee43c0.

No surviving mutants reported in the packages that finished.

⏱ Time budget (1020s) exhausted: 3 of 6 changed package(s) were not fully mutated, so this report is partial:

  • src/pkg/dashboard — unfinished
  • src/pkg/stageengine — not started
  • src/pkg/workflowmodel — not started

Advisory only; this job never blocks the PR (#17349).

@hivecommons-hive

hivecommons-hive Bot commented Nov 11, 2026 •

Copy link
Copy Markdown
Contributor

Hive status — updated in place; edits do not notify. Full history is in the Activity Monitor.

clubanderson this PR is held at stage/071-needs-author and the hive will not merge it yet.

Why:

  • the review swarm returned requires_human: it has a question for you, not a fix for the hive
  • the reviewer asked you: correctness requires human: Stage-engine slice verified: deriver composition keeps the Reconciler the sole writer, binding lookup is locked consistently with its writer, unusable binding falls back to hive-default with a once-per-key warning. One low perf finding (per-derivation ReadFile+Unmarshal+Compile for bound repos). Compiler, store API handlers, ingest and tracker UI (~60 files) unread and need a human reviewer.
  • the reviewer asked you: intent-alignment requires human: Stage-engine slice matches RFC 📝 RFC: Workflow models — ingest, store, edit and select a repository's workflow without changing the built-in one #17620 phase 2 (per-repo binding, built-in lifecycle unchanged when unbound). Whole-PR scope (P1+P2 in one XXL change) not assessed.
  • the reviewer asked you: security requires human: New dashboard write endpoints (api_workflows.go, api_workflows_ingest.go: model store, repo bindings) were not read in this pass; their owner-gating and input bounds need a human check. The stage-engine slice adds no new write surface.

Next: the hive does not push to your branch. Push a new head (or reply here); CI and the review swarm re-run automatically and the PR returns to the normal path.

Review findings from adjudicator-2 (folded here because this head already carries its hive review):

Security delta re-review at 2b97f5b (delta from 0dca60c). The knowledge/gitsource hunks in the compare are base v6 merges, not this PR's work, and were not judged here. Owner gating (requireOwnerRole, api_workflows.go:116,141,189), the store's name/version regexes (store/store.go:52-53, no traversal possible), atomic writes, and the tracker/config JS escaping (tracker-workflows.js, config-workflows.js) all check out.

Still present — raw store errors echoed to clients (src/pkg/dashboard/api_workflows.go:54, :211; also src/pkg/dashboard/stage_tracker_workflows.go:89)

workflowStoreError's default branch returns err.Error() with a 500, and Store.Get/Save/writeIndex pass through os.ReadFile/os.WriteFile/os.Rename errors verbatim (store/store.go:181,239,246,403-407), so an EACCES/ENOSPC/corrupt-index condition answers with the absolute data path (open /data/workflows/<name>/<v>.json: permission denied). The same string is now also surfaced through the unauthenticated tracker response as workflows.repos[<repo>].error (stage_tracker_workflows.go:89). Low severity, but it is the one security note from the previous round and it was not addressed.

Suggested fix — log, return a generic message:

--- a/src/pkg/dashboard/api_workflows.go
+++ b/src/pkg/dashboard/api_workflows.go
@@
-func workflowStoreError(w http.ResponseWriter, err error) {
+func (s *Server) workflowStoreError(w http.ResponseWriter, err error) {
 	switch {
 	case errors.Is(err, wfstore.ErrNotFound):
 		jsonError(w, err.Error(), http.StatusNotFound)
 	case errors.Is(err, wfstore.ErrReadOnly):
 		jsonError(w, err.Error(), http.StatusForbidden)
 	case errors.Is(err, wfstore.ErrBadName), errors.Is(err, wfstore.ErrBadVersion), errors.Is(err, wfstore.ErrNotCompiled):
 		jsonError(w, err.Error(), http.StatusBadRequest)
 	default:
-		jsonError(w, err.Error(), http.StatusInternalServerError)
+		s.logger.Error("workflow store", "error", err)
+		jsonError(w, "workflow store error", http.StatusInternalServerError)
 	}
 }
@@ handleWorkflowBindingPut
-			jsonError(w, "save config: "+err.Error(), http.StatusInternalServerError)
+			s.logger.Error("workflow binding: save config", "error", err)
+			jsonError(w, "save config failed", http.StatusInternalServerError)
--- a/src/pkg/dashboard/stage_tracker_workflows.go
+++ b/src/pkg/dashboard/stage_tracker_workflows.go
@@
 		p, err := lookup(b.Name, b.Version)
 		if err != nil {
-			out.Repos[repo] = StageTrackerWorkflow{Name: b.Name, Version: b.Version, Error: err.Error()}
+			msg := "workflow model unavailable"
+			if errors.Is(err, wfstore.ErrNotFound) || errors.Is(err, wfstore.ErrNotCompiled) {
+				msg = err.Error()
+			}
+			out.Repos[repo] = StageTrackerWorkflow{Name: b.Name, Version: b.Version, Error: msg}
 			continue
 		}

(ErrNotFound/ErrNotCompiled carry only name@version + diagnostic codes, so they stay safe to pass through.)

Review findings from ci-maintainer (folded here because this head already carries its hive review):

Intent-alignment (delta re-review, head 0dca60c8)

PR title/body no longer describe the diff. The only new commit since f2a631fb is the squash of #17649 (phase 2: store, /api/workflows, repo bindings, engine wiring, tracker UI, +3.2k lines), merged into this branch. The body still says "P1 … workflow.enabled config flag, default false. No wiring effect yet (P2 binds repos)", but at this head:

  • src/cmd/hive/stage_engine.go:47 sets WorkflowFor: stageEngineWorkflowFor(cfg, wfstore.Open(...), logger) and stage_engine.go:58-86 resolves per-repo bindings from cfg.WorkflowBindingFor at derivation time.
  • src/pkg/stageengine/reconciler.go:96-99 wraps o.Issue/o.PR in DeriveForRepo whenever WorkflowFor is set; src/pkg/stageengine/workflow.go:16-37 is the new per-repo deriver.
  • src/cmd/hive/main.go:1089 wires WorkflowStore into the dashboard; src/pkg/dashboard/api_workflows.go adds owner-gated save/delete/compile routes; src/pkg/config/workflow_binding.go adds project.workflow_bindings.

Whatever lands on v6 from this PR is P1+P2, so the squash title ("P1: lifecycle schema, compiler, hive-default generator, DeriveFromModel") and body will misdescribe the merged commit. Concrete fix (pick one):

  1. Retitle to cover both phases and replace the "No wiring effect yet (P2 binds repos)" line with the ✨ workflow models P2: store, /api/workflows, repo bindings, engine wiring, tracker columns (#17620) #17649 summary (flag off ⇒ stageEngineWorkflowFor returns nil at stage_engine.go:65, so "nothing changes for unbound repos / flag off" remains true). Both changelog fragments are already present.
  2. Or revert 0dca60c8 here and re-open ✨ workflow models P2: store, /api/workflows, repo bindings, engine wiring, tracker columns (#17620) #17649 against v6 after this merges, keeping the P1 scope the body promises.

Prior findings (src/pkg/workflowmodel/compile.go is byte-identical between heads, blob ce30c1ac):

  • compile.go:198 final state placed only via lifecycle.column hits continue before the StateFinal → Terminal mark at :203 — still present.
  • compile.go:215/:240 numbers()/exits() iterate m.Spec.States only, so region states skip checks — still present.

Review findings from reviewer-2 (folded here because this head already carries its hive review):

Scoped review: the stage-engine integration only. This PR is 6460+/732− over 70 files; I read the slice that touches every item's stage/* labels (src/cmd/hive/stage_engine.go, pkg/stageengine/reconciler.go, pkg/stageengine/workflow.go DeriveForRepo, pkg/config/workflow_binding.go, pkg/workflowmodel/store/store.go) against the v5 tree. The workflows API handlers, ingest, compiler and tracker UI are not covered here and still need a human reviewer.

What checks out in the slice:

  • New() composition (reconciler.go:86-99): Workflow replaces the built-in derivers, then WorkflowFor wraps whatever is in place, so an unbound repo falls through to the unchanged IssueDeriver()/PRDeriver(). The Reconciler stays the only writer (ADR-0025 intact).
  • WorkflowBindingFor (workflow_binding.go:30) reads under repoPauseMu, the same lock SetWorkflowBinding takes, and stageEngineWorkflowFor re-reads cfg per call — consistent with the in-place mutation applyConfigOverrides (main.go:6839) already relies on, so a binding edit takes effect on the next derivation as the comment says.
  • Unusable binding → warn once per key|err → nil → hive-default fallback (stage_engine.go:77-83). Fail-open to the built-in labels rather than unlabelled is the right default for a label writer.

Low — a bound repository re-reads and recompiles its model on every derivation. stageEngineWorkflowFor calls st.Profile(b.Name, b.Version) per lookup (stage_engine.go:76); Profile → Get does os.ReadFile + json.Unmarshal under the store-wide s.mu (store.go:177-187) and then workflowmodel.Compile (store.go:212), with no cache. DeriveForRepo's cache[p.Hash] (workflow.go:29-35) only memoises the Deriver built from the profile; the load+compile that produces p runs first, for every issue and PR the engine reconciles, serialised across goroutines by s.mu. The built-in hive-default short-circuits (store.go:205), so this only bites real bindings, and model versions look immutable once saved, so a name@version → *Profile cache in the store (invalidated by Save/Delete) would remove the cost without changing semantics. Not a correctness defect; flagging because the stage engine runs on every item in every sweep.

I have not changed or closed anything.

Review findings from reviewer (folded here because this head already carries its hive review):

Style / conventions re-review at 0dca60c (delta from f2a631f). Prior correctness findings at src/pkg/workflowmodel/compile.go:198 and :215 are still present — compile.go is not in the delta, and the continue at compile.go:198 still skips the StateFinal → Terminal marking at :202.

The delta turns this "P1" PR into P1+P2 (store, /api/workflows, bindings, engine wiring, tracker UI). The code follows local handler idioms (jsonError/requireOwnerRole/auditFromRequest in api_workflows.go), but it misses several repository ratchets and the PR text now misdescribes the change.

1. PR title/body contradict the diff (medium)

Body: "workflow.enabled config flag … No wiring effect yet (P2 binds repos)". The delta wires it: src/cmd/hive/stage_engine.go:48 sets Options.WorkflowFor, src/pkg/stageengine/reconciler.go:96-99 composes DeriveForRepo, and changelog.d/added-workflowmodel-store-bindings.md is literally titled "RFC phase 2". Please retitle (e.g. "workflow models P1+P2: … store, bindings, /api/workflows, engine wiring") and replace the "No wiring effect yet" bullet with what the second fragment says, or split P2 back out. Reviewers and the changelog compiler should not have to discover scope from the diff.

2. Repo ratchets not updated — CI red at this head (medium)

  • internal/testutil/dashboard_import_allowlist.txt lacks the new dependency; TestDashboardImportRatchet fails ("pkg/dashboard gained a new dependency on pkg/workflowmodel"). src/pkg/dashboard/import_count_test.go:79 was bumped but the testutil allowlist was not:
    github.com/hivecommons/hive/pkg/webassets
    github.com/hivecommons/hive/pkg/workflowmodel
    github.com/hivecommons/hive/pkg/workflowmodel/store
    github.com/hivecommons/hive/pkg/worksource
    
  • Inserting s.registerWorkflowRoutes() at src/pkg/dashboard/api.go:38 shifted every api.go:N citation under src/docs/api-reference/*.md (664 drifted) and the new /api/workflows* routes have no api-reference entry. Run bash src/scripts/check-api-reference-citations.sh --fix and add the Workflows routes.
  • src/pkg/dashboard/static/index.html edits dropped inline styles below the baseline; TestStyleRatchet asks to lower src/pkg/dashboard/webstatic/style_ratchet_test.go:45:
    		inlineStyles:   928,
    

3. reconciler.go is not gofmt-clean (low)

gofmt -l flags src/pkg/stageengine/reconciler.go at head; lines 57-58:

	Logger      *slog.Logger
	Metrics     Metrics

4. Import grouping (low)

src/cmd/hive/main.go:8 and src/pkg/dashboard/deps.go:5 drop wfstore "github.com/hivecommons/hive/pkg/workflowmodel/store" into the stdlib import block. Every other file in the delta (stage_engine.go:11-12, api_workflows.go:9-11) puts it in the module group after a blank line — move it there.

Advisory

wfstore.Open(wfstore.Root(cfg.Data.WorkflowsDir)) is constructed twice (main.go:1089 for the dashboard, stage_engine.go:48 for the engine). Store carries a sync.Mutex (store/store.go:84-87), so two instances give a false impression of serialization; today only the dashboard writes, so it is harmless, but building it once in boot and sharing it would make that explicit.

Review findings from adjudicator (folded here because this head already carries its hive review):

Correctness re-review at 37caad9 (delta since 0dca60c). Prior correctness-relevant notes: PR title/body P1-only → fixed (now P1+P2; note the body's "Not in this PR: P3" is contradicted by commit 37caad9, which lands P3a api_workflows_ingest.go / ingest/lifecycle.go). reconciler.go Options gofmt, main.go import grouping, and the double wfstore.Open for the same root are still present (style backlog, not re-filed here).

1. Tracker workflows block ignores workflow.enabled; UI and engine disagree when the flag is off (medium)

stage_tracker_workflows.go:68 emits the block whenever cfg.Project.WorkflowBindings is non-empty, and tracker-workflows.js then swaps in the model's columns/badge for that repo. But the engine only honours a binding when the flag is on: stage_engine.go:67 (if !cfg.Workflow.Enabled { return nil }) → the repo is labelled through hive-default. Nothing stops a binding from existing with the flag off — PUT /api/workflows/bindings (api_workflows.go:186-222) never checks Workflow.Enabled, and project.workflow_bindings can be set in hive.yaml. Result: the SPA places hive-default stage/NNN numbers into a custom model's numbers arrays — rows land in the wrong column or show the amber "Not covered" chevron while the engine has in fact labelled them normally. This contradicts the body's "unbound/flag-off take the builtin path unchanged" and the default workflow.enabled: false safety story.

func stageTrackerWorkflows(cfg *config.Config, lookup workflowProfileLookup, qualify func(string) string) *StageTrackerWorkflows {
	if cfg == nil || !cfg.Workflow.Enabled || len(cfg.Project.WorkflowBindings) == 0 {
		return nil
	}

(stage_tracker_workflows.go:67-70; add a flag-off case to TestStageTrackerWorkflowsBlock.)

2. Project.WorkflowBindings read without the lock that guards its writes (low)

SetWorkflowBinding replaces the slice under repoPauseMu (workflow_binding.go:56-76), and WorkflowBindingFor reads under RLock (:30). The dashboard side bypasses it: api_workflows.go:149 (delete 409 check), api_workflows.go:168 (workflowBindings(), marshalled straight into responses), and stage_tracker_workflows.go:68,75 iterate the field directly. A concurrent PUT /api/workflows/bindings + tracker poll is a -race report. Mirrors the pre-existing unlocked RepoPolicies reads, but this PR adds the mutex and the writer, so the new readers should use it:

+// WorkflowBindings returns a copy of the bindings, safe against SetWorkflowBinding.
+func (c *Config) WorkflowBindings() []WorkflowBinding {
+	if c == nil {
+		return nil
+	}
+	repoPauseMu.RLock()
+	defer repoPauseMu.RUnlock()
+	return append([]WorkflowBinding(nil), c.Project.WorkflowBindings...)
+}

and replace the three direct cfg.Project.WorkflowBindings reads in api_workflows.go / stage_tracker_workflows.go with cfg.WorkflowBindings().

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Correctness review at f2a631f.

type: final state placed only via lifecycle.column is never marked terminal → spurious state-no-exit refusal (src/pkg/workflowmodel/compile.go:195-204)

Profile.Terminal is documented as "states in a terminal column or of final type" (compile.go:68), but the second pass in columns() hits continue at line 198 when a state's column comes from lifecycle.column rather than columns[].states, so the s.Type == StateFinal marking at 202-204 is skipped (and the col.Terminal marking only ever runs in the first loop, 178-180). Consequences for such a state: exits() (compile.go:246-251) refuses the model with state-no-exit because a final state has no outgoing transition; and even if it did compile, rules-mode derivation would abstain at stageengine/workflow.go:102-107 instead of returning Terminal: true. Neither fixture exercises this path — hivedefault.go lists merged/closed in both columns[].states and lifecycle.column, and testdata/models/techdocs.json uses columns[].states only — so the 24-case TestCompileRefusals table does not catch it.

		if _, ok := c.p.ColumnOf[s.ID]; !ok {
			if s.Lifecycle.Column != "" && cols[s.Lifecycle.Column] {
				c.p.ColumnOf[s.ID] = s.Lifecycle.Column
				if terminalCols[s.Lifecycle.Column] {
					c.p.Terminal[s.ID] = true
				}
			} else {
				c.errorf(CodeNoColumn, s.ID, "state %q is not in any column", s.ID)
			}
		}
		if s.Type == StateFinal {
			c.p.Terminal[s.ID] = true
		}

(with terminalCols := map[string]bool{} populated alongside cols[col.ID] = true in the first loop). A compile test with a final state carrying only lifecycle.column would pin it.

Low — region states are resolvable but skip per-state checks (compile.go:140-148 vs 215, 246, 438): flatten registers Regions in p.States, so rules/agents/columns may reference them and decisionFor will use their Lifecycle.Number, yet numbers()/exits()/inferred() iterate only m.Spec.States, so a region state escapes duplicate-stage-number, state-no-exit and the inferred-element warning. defaults.go:48 already produces a parallel state with regions. Either iterate the flattened map in those passes or document that region states must not carry a number.

Could not run go test in this environment; findings are from reading the head tree.

— hive: agent=reviewer-2 backend=copilot model=claude-fable-5.1 copilot=1.0.88

@hivecommons-hive

Copy link
Copy Markdown
Contributor

Review follow-ups (below review.backlog.min_severity: P2)

Non-blocking findings below the backlog filing line are listed here instead of as issues, one comment per PR, updated in place. Tick them off as they are fixed; set review.backlog.min_severity: P3 to file every finding.

  • P3 info correctness — Per-derivation Store.Profile re-reads and recompiles the model; DeriveForRepo hash cache saves nothing (src/pkg/workflowmodel/store/store.go:204): stageEngineWorkflowFor calls st.Profile on every derivation; Profile does ReadFile + readIndex (JSON unmarshal of the catalogue) + workflowmodel.Compile under the store mutex each call (store.go:167-…

@hivecommons-hive

Copy link
Copy Markdown
Contributor

CI triage at head 0dca60c — every red check is PR-local and deterministic (v6 is green on all of them); each one prints its own fix:

check cause fix
test (rest 2/4) TestDashboardImportRatchet (internal/testutil/dashboard_import_ratchet_test.go:59): pkg/dashboard gained a direct import of pkg/workflowmodel route it through an existing seam, or add github.com/hivecommons/hive/pkg/workflowmodel to internal/testutil/dashboard_import_allowlist.txt with the justification in this PR
test (rest 3/4) TestStyleRatchet (pkg/dashboard/webstatic/style_ratchet_test.go:91): hub static pages now have 928 style= attributes, below the 939 baseline lower styleRatchetBaselines["hub static pages"].inlineStyles to 928 so the ratchet keeps biting
relative links and anchors resolve 660+ DRIFT lines: src/docs/api-reference.md file:line citations into pkg/dashboard/api.go shifted bash src/scripts/check-api-reference-citations.sh --fix and commit
changelog-fragment-guard same api-reference drift surfaced through the guard step same --fix run
native-queue-policy / DCO DCO: action_required — at least one commit on the branch is missing Signed-off-by git rebase --signoff (or amend) and force-push

coverage will re-evaluate once the two test shards pass. Contributor-authored branch, so the hive does not push to it.


🐝 Hive Agent: quality | Instance: hosted-available-oke-11-placeholder-r05x | SHA: unknown

— hive: agent=quality backend=copilot model=claude-fable-5.1 copilot=1.0.88

@hivecommons-hive

Copy link
Copy Markdown
Contributor

CI triage at head 0dca60c — every red check is PR-local (v6 at a59684c is green on all of them; this head is 2 ahead / 1 behind, no shared incident). Five distinct causes, all fixable in one push:

  1. DCO (action_required) → also fails native-queue-policy. Commit 0dca60c is authored as Andy Anderson <andy@clubanderson.com> but signed off as Andrew Anderson <andy@clubanderson.com>; the DCO app requires the author name to match the trailer (f2a631f is fine). git commit --amend --author="Andrew Anderson <andy@clubanderson.com>" --no-edit and force-push with lease. native-queue-policy just surfaces Error: DCO: action_required and clears on its own.

  2. test (rest 2/4) → TestDashboardImportRatchet (src/internal/testutil/dashboard_import_ratchet_test.go:59): pkg/dashboard gained a new dependency on pkg/workflowmodel. Either route the /api/workflows handlers through an existing seam, or add github.com/hivecommons/hive/pkg/workflowmodel to src/internal/testutil/dashboard_import_allowlist.txt with the justification in this PR body.

  3. test (rest 3/4) → TestStyleRatchet (src/pkg/dashboard/webstatic/style_ratchet_test.go:91): hub static pages has 928 style= attributes, below the baseline of 939. Lower styleRatchetBaselines["hub static pages"].inlineStyles to 928 so the ratchet keeps biting.

  4. changelog-fragment-guard / relative links and anchors resolve: not the fragments themselves (both changelog.d/added-workflowmodel-*.md are at the root) — the api-reference citation guard reports DRIFT api-reference.md:33 GET /api/version -> pkg/dashboard/api.go:45 (line does not register it) and onward, because the new routes shifted api.go line numbers. Run bash src/scripts/check-api-reference-citations.sh --fix and commit the result.

  5. coverage only aggregates shard results; it goes green once (2) and (3) pass.

Note: the title says P1 but 0dca60c is "workflow models P2: store, /api/workflows, repo bindings, engine wiring" — if P2 is meant to land here, the title/body and Closes line should say so; otherwise P2 wants its own PR against this one.

Contributor-authored branch, so the hive does not push to it.


🐝 Hive Agent: scanner | Instance: hosted-available-oke-11-placeholder-r05x | SHA: unknown

— hive: agent=scanner backend=copilot model=claude-fable-5.1 copilot=1.0.88

…DeriveFromModel (#17620 P1)

Extends the v1alpha1 schema with columns, derivation, events, gates,
roles, agents, checks, bookkeeping, labels and meta; adds Compile with
coded refusals; generates hive-default from the stage table and the
engine's expected edges; adds stageengine.DeriveFromModel (builtin
delegation + rules evaluator) and Options.Workflow; adds the
workflow.enabled flag (default false, no wiring effect yet).

The PR stage-golden corpus derives identically under hive-default.

Signed-off-by: Andrew Anderson <andy@clubanderson.com>
(cherry picked from commit 43eb8b2)
…ring, tracker columns (#17620) (#17649)

* ✨ workflowmodel: store, /api/workflows, repo bindings (RFC #17620 P2a)

Signed-off-by: Andrew Anderson <andy@clubanderson.com>

* ✨ stageengine: per-repo workflow binding, tracker workflows block (RFC #17620 P2b)

Signed-off-by: Andrew Anderson <andy@clubanderson.com>

* workflow models P2: Settings/tracker UI, openapi, gates, docs (P2c)

Per-repo Workflow select on Settings → Repos, tracker workflows block
rendering (model column header, per-row strip, Not covered chevron),
openapi routes/schemas, import ratchet and settings-parity allowlist,
regenerated agent-runtime golden, phase-2 design notes and fragment.

Refs #17620

Signed-off-by: Andrew Anderson <andy@clubanderson.com>

---------

Signed-off-by: Andrew Anderson <andy@clubanderson.com>
…ding, lower style ratchet baselines

Signed-off-by: Andrew Anderson <andy@clubanderson.com>
@clubanderson
clubanderson force-pushed the workflowmodel/p1-compiler branch from 0dca60c to cfd333a Compare November 11, 2026 06:02
@clubanderson clubanderson changed the title ✨ workflow models P1: lifecycle schema, compiler, hive-default generator, DeriveFromModel (#17620) ✨ workflow models P1+P2: schema, compiler, DeriveFromModel, store, /api/workflows, repo bindings, tracker columns (#17620) Nov 11, 2026
@clubanderson clubanderson changed the title ✨ workflow models P1+P2: schema, compiler, DeriveFromModel, store, /api/workflows, repo bindings, tracker columns (#17620) ✨ workflow models P1+P2: schema, compiler, DeriveFromModel, store, workflows API, repo bindings, tracker columns Nov 11, 2026
…RFC #17620 P2)

Signed-off-by: Andrew Anderson <andy@clubanderson.com>

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Correctness re-review at 2b97f5b (delta since 0dca60c: the branch was rebased onto v6, so the GitHub compare is noisy; the PR-side delta is commits cfd333a5 + 2b97f5ba — api-reference resync, import-allowlist entry, CSS token swap, style-ratchet baselines). No Go logic changed in the workflow code, so the three in-scope correctness findings from the previous round are still present:

1. Tracker shows bound-model columns the engine will not use when workflow.enabled is false — src/pkg/dashboard/stage_tracker_workflows.go:68 gates only on len(cfg.Project.WorkflowBindings) == 0, while the engine resolver at src/cmd/hive/stage_engine.go:67 returns nil whenever !cfg.Workflow.Enabled. A hive with bindings in hive.yaml but the flag off gets per-repo columns in the SPA while every item is still derived by hive-default.

	if cfg == nil || !cfg.Workflow.Enabled || len(cfg.Project.WorkflowBindings) == 0 {
		return nil
	}

2. Saving a non-compiling draft over a bound version breaks the live binding — src/pkg/dashboard/api_workflows.go:129 calls st.Save, and store.Save (src/pkg/workflowmodel/store/store.go:233-276) deliberately persists drafts with Compiled=false. Nothing checks Project.WorkflowBindings on the save path the way handleWorkflowDelete does at :147-154, so after the save st.Profile (store.go:212-214) returns ErrNotCompiled, stageEngineWorkflowFor falls back to hive-default (stage_engine.go:78-83), and the tracker reports Error for that repo. Suggested guard, mirroring the delete path:

 	e, err := st.Save(&m, requestUser(r), time.Now().UTC())
+	// refuse to break a live binding with a draft: compile first when bound

better placed before the save:

+	if s.deps.Config != nil {
+		for _, b := range s.deps.Config.Project.WorkflowBindings {
+			if b.Name == m.Metadata.Name && b.Version == m.Spec.Meta.Version {
+				if _, _, cerr := workflowmodel.Compile(&m); cerr != nil {
+					jsonError(w, fmt.Sprintf("%s is bound to %s and must compile: %v", b.Key(), b.Repo, cerr), http.StatusConflict)
+					return
+				}
+			}
+		}
+	}
 	e, err := st.Save(&m, requestUser(r), time.Now().UTC())

3. Unlocked reads of Project.WorkflowBindings — SetWorkflowBinding mutates the slice under repoPauseMu (src/pkg/config/workflow_binding.go:56-75), but api_workflows.go:148 (delete), :167 (workflowBindings()), and stage_tracker_workflows.go:68,75 range over it with no lock. A dashboard PUT concurrent with a tracker build or list is a data race (go test -race on a concurrent handler test would flag it). Expose a locked accessor in pkg/config:

// WorkflowBindings returns a copy of the bindings under the binding lock.
func (c *Config) WorkflowBindings() []WorkflowBinding {
	if c == nil { return nil }
	repoPauseMu.RLock()
	defer repoPauseMu.RUnlock()
	return append([]WorkflowBinding(nil), c.Project.WorkflowBindings...)
}

and use it at the three dashboard sites.

Advisory (unchanged, not blocking): Store.Profile (store.go:204-217) re-reads and recompiles the model on every call, so the per-profile hash cache in DeriveForRepo never hits across derivations.

— hive: agent=adjudicator backend=copilot model=claude-fable-5.1 copilot=1.0.88

…rvey compile stats (RFC #17620 P3a) (#17652)

ingest.Lifecycle composes hive-default's compiled lifecycle with a
repository's pipelines: PR-verify pipelines on the CI / Review states with
citations, the post-merge chain as inferred columns after Merged,
provenance ingested-unreviewed. New pipeline.status fact (engine reports
unknown until post-merge runs are observed). ingest.ReadTree selects CI
files from a checkout. POST /api/workflows/ingest fetches a repo's default
branch read-only (or takes pasted files), compiles, optionally saves at the
next integer version. workflow-survey compiles every corpus repo's
lifecycle model: 317/317 compile, 0 refusals.

Signed-off-by: Andrew Anderson <andy@clubanderson.com>
@clubanderson
clubanderson force-pushed the workflowmodel/p1-compiler branch from b1627e0 to 37caad9 Compare November 11, 2026 06:24

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Intent alignment — delta re-review at 37caad96

The body now covers P2 (previous P1-only gap: fixed for P2), but head commit 37caad96 merges #17652 (P3a) into this branch while the body still says under Not in this PR: "P3 (ingest → lifecycle draft models)". The diff contradicts that:

  • src/pkg/workflowmodel/ingest/lifecycle.go / tree.go — new ingest package
  • src/pkg/dashboard/api_workflows_ingest.go:36 — POST /api/workflows/ingest, owner-gated, fetches the forge on the hive credential and can write the store (:100-117)
  • src/cmd/hive/workflow_ingest.go:20 + src/cmd/hive/main.go:1090 — WorkflowRepoFiles wired into the live server
  • src/cmd/workflow-survey/main.go — compile stats
  • changelog.d/added-workflowmodel-ingest-lifecycle.md — fragment titled "RFC phase 3a"

Either drop 37caad96 from this branch (land P3a as its own PR against this one, as #17652 originally was), or make the description match what ships. Suggested body edit:

-RFC #17620, phases 1 and 2 of 7 (#17649 was merge-trained into this branch, so this PR now carries both). Base `v6`.
+RFC #17620, phases 1, 2 and 3a of 7 (#17649 and #17652 were merge-trained into this branch, so this PR now carries all three). Base `v6`.
...
+# Phase 3a (from #17652)
+- `pkg/workflowmodel/ingest`: `Lifecycle` composes hive-default with the repo's pipelines; `ReadTree` selects CI files.
+- `POST /api/workflows/ingest` (owner-gated, `{repo}` or `{files}`, `save: true` stores next integer version).
+- `cmd/workflow-survey` compiles every repo model and reports refusal counts.
...
 ## Not in this PR
-P3 (ingest → lifecycle draft models), P4 (visual editor), ...
+P3b (reviewed/promoted ingest drafts), P4 (visual editor), ...

and retitle to ✨ workflow models P1+P2+P3a: ….

Still present from the earlier review (not re-filed): hive.yaml.example has no workflow.enabled / data.workflows_dir / project.workflow_bindings keys; src/docs/env-vars.md has no HIVE_WORKFLOWS_DATA_DIR row (declared at src/pkg/config/config.go:301); reconciler.go not gofmt-clean; main.go:8 wfstore import in the stdlib group; store opened in both main.go:1089 and stage_engine.go:49.

— hive: agent=ci-maintainer backend=copilot model=claude-fable-5.1 copilot=1.0.88

@clubanderson
clubanderson merged commit 1217d64 into v6 Nov 11, 2026
82 of 83 checks passed
@kubestellar-prow
kubestellar-prow Bot deleted the workflowmodel/p1-compiler branch November 11, 2026 07:33
@github-actions

Copy link
Copy Markdown
Contributor

Thank you for your contribution! Your PR has been merged.

We'd love to hear how your experience was: share feedback

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Read the store, the /api/workflows* handlers (including ingest), the engine wiring (DeriveForRepo / DeriveFromModel / stageEngineWorkflowFor) and the binding config against the tree at 2ceee43. The parts I read look correct to me; nothing blocking. Three notes, two low and one on the PR body.

1. DeriveFromModel under derivation: builtin uses the package-internal deriver, not the one the engine was configured with (low, latent).
src/pkg/stageengine/workflow.go:56 builds modelDeriver{inner: builtinDeriver(kind)} = issueDeriverF, and DeriveForRepo (workflow.go:17-36) never hands its fallback into it. Production wires Issue: github.IssueDeriver() (src/cmd/hive/stage_engine.go:45), which wraps stageengine.DeriveIssue with resolveIssueLabelFacts (src/pkg/github/issue_stage_facts.go:164-171). So for a repo bound to hive-default, the label-resolution wrapper is bypassed. Today this is harmless because every Facts producer already resolves before Reconcile (issue_stage_facts.go:39, stage_facts_sink.go:174), and TestDeriveFromModelHiveDefaultParity compares against stageengine.IssueDeriver(), so it passes. But the parity the PR promises ("delegates to the existing derivers") is to the raw engine deriver, not to whatever Options.Issue/Options.PR was set to — the next caller that relies on the configured wrapper gets different derivation only in bound repos. Suggestion: let DeriveForRepo pass fallback as the builtin-mode inner (e.g. an unexported deriveFromModelWith(p, kind, inner)), and point the parity test at github.IssueDeriver() so the test guards the production pair.

2. Binding dedupe key differs from the lookup key (low).
src/pkg/config/workflow_binding.go:81-91 rejects duplicates on strings.ToLower(b.Repo), while WorkflowBindingFor (:27-36) matches on repoPauseKey(org, repo), which normalises bare vs org-qualified names. repo: hive and repo: hivecommons/hive therefore validate as two bindings but resolve to one by slice order. SetWorkflowBinding would collapse them on the next edit, so a hand-edited hive.yaml is the only way to get there; using repoPauseKey in the validator closes it.

3. PR body is stale about scope. It says "Not in this PR: P3 (ingest …)", but 37caad9 (#17652, P3a) is merge-trained into this branch: src/pkg/workflowmodel/ingest/, src/pkg/dashboard/api_workflows_ingest.go, src/cmd/hive/workflow_ingest.go and changelog.d/added-workflowmodel-ingest-lifecycle.md all land here. The title says P1+P2. Worth a one-line body/title update so the squash message matches what merges.

Checked and fine: name/version regexes in store.go:53-54 exclude / and a leading ., so no traversal from the {name}/{version} path params; writes are atomic; all mutating routes are requireOwnerRole-gated and audited; compile is unauthenticated but read-only and bounded by decodeBody's MaxBytesReader; handleWorkflowDelete 409s on a bound model; ingest's forge fetch reuses the existing tarball path in complexity_source.go. Everything is behind workflow.enabled: false plus an explicit binding, so unbound hives are untouched.

Not read: the compiler internals (compile.go), the hivedefault generator, and the tracker/settings JS — CI covers those and I did not re-derive them.

— hive: agent=reviewer backend=copilot model=claude-fable-5.1 copilot=1.0.88

hivecommons-hive Bot added a commit that referenced this pull request Nov 11, 2026
…tore, workflows API, repo bindings, tracker columns (#17679)

* ✨ workflowmodel: phase-1 ingest (GitHub Actions, Tekton), renderers and survey harness (#17620) (#17630)

* ✨ workflowmodel: phase-1 ingest (GitHub Actions, Tekton), renderers and survey harness (#17620)

Deterministic ingest of a repository's GitHub Actions workflows and Tekton /
OpenShift Pipelines resources into one content-hashed workflow model bound
onto the hive default lifecycle; Mermaid and Activity Monitor column
renderers; cmd/workflow-survey proving harness; golden fixtures from six
real repositories. Nothing wires into the stage engine yet.

Signed-off-by: Andrew Anderson <andy@clubanderson.com>

* workflow-survey: make main testable, drop unused const

golangci-lint flagged maxFileBytes as unused and the coverage gate
hard-fails any new cmd package without tests. Split main into
realMain(args, stdout, stderr) and run over the curated fixture corpus
in-process (96.6% statement coverage).

Signed-off-by: Andrew Anderson <andy@clubanderson.com>

---------

Signed-off-by: Andrew Anderson <andy@clubanderson.com>
(cherry picked from commit a59684c)

* ✨ workflow models P1+P2: schema, compiler, DeriveFromModel, store, workflows API, repo bindings, tracker columns (#17648)

* ✨ workflowmodel: lifecycle schema, compiler, hive-default generator, DeriveFromModel (#17620 P1)

Extends the v1alpha1 schema with columns, derivation, events, gates,
roles, agents, checks, bookkeeping, labels and meta; adds Compile with
coded refusals; generates hive-default from the stage table and the
engine's expected edges; adds stageengine.DeriveFromModel (builtin
delegation + rules evaluator) and Options.Workflow; adds the
workflow.enabled flag (default false, no wiring effect yet).

The PR stage-golden corpus derives identically under hive-default.

Signed-off-by: Andrew Anderson <andy@clubanderson.com>
(cherry picked from commit 43eb8b2)

* ✨ workflow models P2: store, /api/workflows, repo bindings, engine wiring, tracker columns (#17620) (#17649)

* ✨ workflowmodel: store, /api/workflows, repo bindings (RFC #17620 P2a)

Signed-off-by: Andrew Anderson <andy@clubanderson.com>

* ✨ stageengine: per-repo workflow binding, tracker workflows block (RFC #17620 P2b)

Signed-off-by: Andrew Anderson <andy@clubanderson.com>

* workflow models P2: Settings/tracker UI, openapi, gates, docs (P2c)

Per-repo Workflow select on Settings → Repos, tracker workflows block
rendering (model column header, per-row strip, Not covered chevron),
openapi routes/schemas, import ratchet and settings-parity allowlist,
regenerated agent-runtime golden, phase-2 design notes and fragment.

Refs #17620

Signed-off-by: Andrew Anderson <andy@clubanderson.com>

---------

Signed-off-by: Andrew Anderson <andy@clubanderson.com>

* workflowmodel P2: refresh api-reference citations, tokenize badge padding, lower style ratchet baselines

Signed-off-by: Andrew Anderson <andy@clubanderson.com>

* testutil: allow pkg/dashboard → pkg/workflowmodel in import ratchet (RFC #17620 P2)

Signed-off-by: Andrew Anderson <andy@clubanderson.com>

* ✨ workflowmodel: ingest → lifecycle models, /api/workflows/ingest, survey compile stats (RFC #17620 P3a) (#17652)

ingest.Lifecycle composes hive-default's compiled lifecycle with a
repository's pipelines: PR-verify pipelines on the CI / Review states with
citations, the post-merge chain as inferred columns after Merged,
provenance ingested-unreviewed. New pipeline.status fact (engine reports
unknown until post-merge runs are observed). ingest.ReadTree selects CI
files from a checkout. POST /api/workflows/ingest fetches a repo's default
branch read-only (or takes pasted files), compiles, optionally saves at the
next integer version. workflow-survey compiles every corpus repo's
lifecycle model: 317/317 compile, 0 refusals.

Signed-off-by: Andrew Anderson <andy@clubanderson.com>

---------

Signed-off-by: Andrew Anderson <andy@clubanderson.com>
Co-authored-by: hivecommons-hive[bot] <280983584+hivecommons-hive[bot]@users.noreply.github.com>
(cherry picked from commit 1217d64)

---------

Signed-off-by: Andrew Anderson <andy@clubanderson.com>
Co-authored-by: hivecommons-hive[bot] <280983584+hivecommons-hive[bot]@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

dco-signoff: yes Indicates the PR's author has signed the DCO. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant