diff --git a/changelog.d/added-workflowmodel-compiler.md b/changelog.d/added-workflowmodel-compiler.md new file mode 100644 index 0000000000..84df9ed9d8 --- /dev/null +++ b/changelog.d/added-workflowmodel-compiler.md @@ -0,0 +1 @@ +- Workflow models, RFC phase 1 ([#17620](https://github.com/hivecommons/hive/issues/17620)): the `workflow.hivecommons.dev/v1alpha1` schema gains the lifecycle half of the RFC — `columns`, `derivation`, `events`, `gates`, `roles`, `agents`, `checks`, `bookkeeping`, `labels`, `meta` and per-state `lifecycle` (column, stage number, item kinds, replies) — and `workflowmodel.Compile` turns a model into a `Profile` or refuses it with coded diagnostics (a non-terminal state with no exit, a gate with no releasing role, an agent-granted state with no human-released gate before a terminal outcome, two events bound to one fact, a state in no or two columns, duplicate stage numbers, an unknown fact or operator). `pkg/workflowmodel/hivedefault` generates the `hive-default` model from the live stage table and the engine's expected-edge table, so the monitor's seven columns and the stage numbers are now a compiled model rather than constants. `stageengine.DeriveFromModel` derives under a compiled profile: `hive-default` declares `derivation: builtin` and delegates to the existing derivers (the whole PR stage-golden corpus and the issue sweep derive identically), custom models declare `derivation: rules` (ordered first-match predicates over the fact vocabulary, uncovered items abstain `workflow: not covered`). New `workflow.enabled` config flag, default false; nothing changes for any hive until a repository is bound (phase 2). diff --git a/changelog.d/added-workflowmodel-ingest-lifecycle.md b/changelog.d/added-workflowmodel-ingest-lifecycle.md new file mode 100644 index 0000000000..7906f340b3 --- /dev/null +++ b/changelog.d/added-workflowmodel-ingest-lifecycle.md @@ -0,0 +1 @@ +- Workflow models, RFC phase 3a ([#17620](https://github.com/hivecommons/hive/issues/17620)): an ingested repository now yields a model that compiles. `ingest.Lifecycle` composes the hive-default lifecycle (states, columns, builtin derivation, roles, gates, events) with the repository's pipelines — PR-verify pipelines bound to the CI / Review states with file citations, the post-merge chain (main CI, release, per-environment deploys, done) as new columns after Merged — every ingested element marked inferred and the model's provenance `ingested-unreviewed`. New fact `pipeline.status` names the post-merge run the chain waits on (the engine reports `unknown` until it observes post-merge runs, so nothing moves on it yet). New owner-gated `POST /api/workflows/ingest` takes `{repo}` (default branch fetched read-only through the hive's GitHub client, CI files selected by `ingest.ReadTree`) or pasted `{files}`, returns the model, compile result and diagnostics, and with `save: true` stores it at the next integer version. `cmd/workflow-survey` compiles every repository's lifecycle model and reports pass/refusal counts by code; over the 317-repository CNCF corpus 317/317 compile with 1,117 inferred-element warnings and no refusals. diff --git a/changelog.d/added-workflowmodel-ingest.md b/changelog.d/added-workflowmodel-ingest.md new file mode 100644 index 0000000000..b15e2939e4 --- /dev/null +++ b/changelog.d/added-workflowmodel-ingest.md @@ -0,0 +1 @@ +- Workflow models, phase 1 ([#17620](https://github.com/hivecommons/hive/issues/17620)): a new `pkg/workflowmodel` package ingests a repository's GitHub Actions workflows and Tekton / OpenShift Pipelines resources (`pkg/workflowmodel/githubactions`, `pkg/workflowmodel/tekton`, composed by `pkg/workflowmodel/ingest`) into one deterministic, content-hashed model of the repo's delivery process — pipelines with their trigger roles (PR verify, push verify, release, deploy, scheduled, manual), execution DAGs with topological levels, matrices, reusable workflows, environments and concurrency — bound onto the hive default lifecycle. `pkg/workflowmodel/render` turns a model into a Mermaid diagram and the Activity Monitor column set it implies (CI sub-stages under *CI / Review*, post-merge *Main CI*, *Release* and *Deploy: * columns), and a model with no pipelines renders today's seven columns unchanged. Nothing is wired into the v5/v6 stage engine yet; `cmd/workflow-survey` is the proving harness that produced the CNCF-landscape survey posted on the issue (317 repositories, 4,389 pipelines, 317 valid models). Golden fixtures under `pkg/workflowmodel/testdata/corpus` pin real repositories' results. diff --git a/changelog.d/added-workflowmodel-store-bindings.md b/changelog.d/added-workflowmodel-store-bindings.md new file mode 100644 index 0000000000..c369578e87 --- /dev/null +++ b/changelog.d/added-workflowmodel-store-bindings.md @@ -0,0 +1 @@ +- Workflow models, RFC phase 2 ([#17620](https://github.com/hivecommons/hive/issues/17620)): models now have a home and a way to reach the engine. `pkg/workflowmodel/store` keeps versioned models under `/workflows//.json` (plus `index.json`); `hive-default` is virtual and read-only, drafts save but do not compile, every entry reserves `adoption`/`ratings`/`benchmarks` fields for the marketplace and carries them verbatim across saves. New owner-gated, audited `/api/workflows` routes list, save, read, delete and compile models; `project.workflow_bindings: [{repo, name, version}]` binds a repository to a compiling model (the binding is refused for a draft, and a bound model cannot be deleted). When `workflow.enabled` is on, the stage reconciler derives bound repositories through the bound model and everything else through the built-in derivers exactly as before. Settings → Repos gains a per-repo Workflow select, and the Activity Monitor response gains a `workflows` block (default columns plus per-repo columns and stage numbers) that the tracker uses to label the model column and to render a per-row column strip, with an amber "Not covered" chevron when a bound model abstains. Nothing changes for unbound repositories. diff --git a/dashboard/openapi.json b/dashboard/openapi.json index a895906118..d4fae455f6 100644 --- a/dashboard/openapi.json +++ b/dashboard/openapi.json @@ -35262,7 +35262,472 @@ } } } - } + }, + "/api/workflows": { + "get": { + "tags": [ + "Workflows" + ], + "summary": "List workflow models", + "description": "Every stored model version plus the virtual read-only hive-default, the current repository bindings, and whether model-driven derivation (workflow.enabled) is on.", + "responses": { + "200": { + "description": "Library", + "content": { + "application/json": { + "schema": { + "type": "object", + "properties": { + "workflows": { + "type": "array", + "items": { + "$ref": "#/components/schemas/WorkflowEntry" + } + }, + "bindings": { + "type": "array", + "items": { + "$ref": "#/components/schemas/WorkflowBinding" + } + }, + "enabled": { + "type": "boolean" + } + } + } + } + } + }, + "503": { + "description": "No workflow store" + } + } + }, + "post": { + "tags": [ + "Workflows" + ], + "summary": "Save a workflow model", + "description": "Owner only. Compiles the model; a draft that fails compilation is still stored (compiled=false) so it can be edited, but cannot be bound. Audited as workflow_save.", + "requestBody": { + "required": true, + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/WorkflowModel" + } + } + } + }, + "responses": { + "201": { + "description": "Stored", + "content": { + "application/json": { + "schema": { + "type": "object", + "properties": { + "entry": { + "$ref": "#/components/schemas/WorkflowEntry" + } + } + } + } + } + }, + "400": { + "description": "Bad name/version or invalid model" + }, + "403": { + "description": "Not an owner, or the version is read-only (hive-default)" + }, + "503": { + "description": "No workflow store" + } + } + } + }, + "/api/workflows/compile": { + "post": { + "tags": [ + "Workflows" + ], + "summary": "Compile a workflow model without storing it", + "description": "Returns the compiler diagnostics and, when it compiles, the profile. Used by the editor and by ingest previews.", + "requestBody": { + "required": true, + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/WorkflowModel" + } + } + } + }, + "responses": { + "200": { + "description": "Compilation result", + "content": { + "application/json": { + "schema": { + "type": "object", + "properties": { + "compiled": { + "type": "boolean" + }, + "diagnostics": { + "type": "array", + "items": { + "$ref": "#/components/schemas/WorkflowDiagnostic" + } + }, + "profile": { + "$ref": "#/components/schemas/WorkflowProfile" + } + } + } + } + } + }, + "400": { + "description": "Malformed body" + }, + "503": { + "description": "No workflow store" + } + } + } + }, + "/api/workflows/bindings": { + "get": { + "tags": [ + "Workflows" + ], + "summary": "List repository → workflow bindings", + "description": "project.workflow_bindings from hive.yaml plus the live workflow.enabled flag. Unbound repositories derive with hive-default.", + "responses": { + "200": { + "description": "Bindings", + "content": { + "application/json": { + "schema": { + "type": "object", + "properties": { + "bindings": { + "type": "array", + "items": { + "$ref": "#/components/schemas/WorkflowBinding" + } + }, + "enabled": { + "type": "boolean" + } + } + } + } + } + }, + "503": { + "description": "No workflow store or config" + } + } + }, + "put": { + "tags": [ + "Workflows" + ], + "summary": "Bind or unbind a repository", + "description": "Owner only. Body {repo, name, version}; an empty name unbinds. The model must exist and compile. Persists hive.yaml and takes effect on the next derivation pass. Audited as workflow_bind / workflow_unbind.", + "requestBody": { + "required": true, + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/WorkflowBinding" + } + } + } + }, + "responses": { + "200": { + "description": "Bindings after the change", + "content": { + "application/json": { + "schema": { + "type": "object", + "properties": { + "changed": { + "type": "boolean" + }, + "bindings": { + "type": "array", + "items": { + "$ref": "#/components/schemas/WorkflowBinding" + } + } + } + } + } + } + }, + "400": { + "description": "Missing repo, or the model does not compile" + }, + "403": { + "description": "Not an owner" + }, + "404": { + "description": "Model not found" + }, + "503": { + "description": "No workflow store or config" + } + } + } + }, + "/api/workflows/{name}/{version}": { + "get": { + "tags": [ + "Workflows" + ], + "summary": "Fetch one workflow model version", + "parameters": [ + { + "name": "name", + "in": "path", + "required": true, + "schema": { + "type": "string" + } + }, + { + "name": "version", + "in": "path", + "required": true, + "schema": { + "type": "string" + } + } + ], + "responses": { + "200": { + "description": "Model", + "content": { + "application/json": { + "schema": { + "type": "object", + "properties": { + "entry": { + "$ref": "#/components/schemas/WorkflowEntry" + }, + "model": { + "$ref": "#/components/schemas/WorkflowModel" + }, + "profile": { + "$ref": "#/components/schemas/WorkflowProfile" + } + } + } + } + } + }, + "404": { + "description": "Not found" + }, + "503": { + "description": "No workflow store" + } + } + }, + "put": { + "tags": [ + "Workflows" + ], + "summary": "Save a workflow model at an explicit name@version", + "description": "Owner only. Same as POST /api/workflows but the path must agree with the body name/version. Audited as workflow_save.", + "parameters": [ + { + "name": "name", + "in": "path", + "required": true, + "schema": { + "type": "string" + } + }, + { + "name": "version", + "in": "path", + "required": true, + "schema": { + "type": "string" + } + } + ], + "requestBody": { + "required": true, + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/WorkflowModel" + } + } + } + }, + "responses": { + "201": { + "description": "Stored", + "content": { + "application/json": { + "schema": { + "type": "object", + "properties": { + "entry": { + "$ref": "#/components/schemas/WorkflowEntry" + } + } + } + } + } + }, + "400": { + "description": "Path and body disagree, or invalid model" + }, + "403": { + "description": "Not an owner, or read-only" + }, + "503": { + "description": "No workflow store" + } + } + }, + "delete": { + "tags": [ + "Workflows" + ], + "summary": "Delete a workflow model version", + "description": "Owner only. Refused with 409 while any repository is bound to it; hive-default cannot be deleted. Audited as workflow_delete.", + "parameters": [ + { + "name": "name", + "in": "path", + "required": true, + "schema": { + "type": "string" + } + }, + { + "name": "version", + "in": "path", + "required": true, + "schema": { + "type": "string" + } + } + ], + "responses": { + "200": { + "description": "Deleted", + "content": { + "application/json": { + "schema": { + "type": "object", + "properties": { + "ok": { + "type": "boolean" + } + } + } + } + } + }, + "403": { + "description": "Not an owner, or read-only" + }, + "404": { + "description": "Not found" + }, + "409": { + "description": "Still bound" + }, + "503": { + "description": "No workflow store" + } + } + } + }, + "/api/workflows/ingest": { + "post": { + "tags": [ + "Workflows" + ], + "summary": "Ingest a repository's CI definitions into a lifecycle workflow model", + "description": "Reads the repository's default-branch GitHub Actions workflows and Tekton resources (or pasted files) and returns the hive-default lifecycle with the repository's pipelines bound: PR-verify pipelines on the CI / Review states, the post-merge chain as new columns. Provenance ingested-unreviewed. With save=true the model is stored as the next integer version under name. Owner role required.", + "requestBody": { + "required": true, + "content": { + "application/json": { + "schema": { + "type": "object", + "properties": { + "repo": { + "type": "string", + "description": "owner/name to fetch through the forge client; exactly one of repo or files" + }, + "files": { + "type": "object", + "additionalProperties": { + "type": "string" + }, + "description": "repo-relative path to file contents, for pasted or uploaded definitions" + }, + "save": { + "type": "boolean" + }, + "name": { + "type": "string", + "description": "store name (default: slug of the repository)" + } + } + } + } + } + }, + "responses": { + "200": { + "description": "Ingested (not saved)", + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/WorkflowIngestResult" + } + } + } + }, + "201": { + "description": "Ingested and saved", + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/WorkflowIngestResult" + } + } + } + }, + "400": { + "description": "Bad request" + }, + "403": { + "description": "Owner role required" + }, + "502": { + "description": "Repository fetch failed" + }, + "503": { + "description": "No forge client or no workflow store" + } + } + } + } }, "components": { "schemas": { @@ -35566,7 +36031,138 @@ } } } - } + }, + "WorkflowEntry": { + "type": "object", + "description": "Index row for one stored model version.", + "properties": { + "name": { + "type": "string" + }, + "version": { + "type": "string" + }, + "hash": { + "type": "string", + "description": "sha256 of the stored model document" + }, + "title": { + "type": "string" + }, + "provenance": { + "type": "string", + "enum": [ + "builtin", + "local", + "ingested", + "imported" + ] + }, + "compiled": { + "type": "boolean", + "description": "false for a draft that does not compile; drafts can be stored but not bound" + }, + "builtin": { + "type": "boolean", + "description": "true for the read-only hive-default model" + }, + "created_at": { + "type": "string", + "format": "date-time" + }, + "updated_at": { + "type": "string", + "format": "date-time" + }, + "ratings": { + "description": "Reserved for the marketplace (benchmarks, votes); preserved verbatim across saves.", + "nullable": true + } + } + }, + "WorkflowBinding": { + "type": "object", + "properties": { + "repo": { + "type": "string", + "description": "owner/repo or bare repo name" + }, + "name": { + "type": "string" + }, + "version": { + "type": "string" + } + }, + "required": [ + "repo", + "name", + "version" + ] + }, + "WorkflowDiagnostic": { + "type": "object", + "properties": { + "severity": { + "type": "string", + "enum": [ + "error", + "warning" + ] + }, + "code": { + "type": "string" + }, + "message": { + "type": "string" + }, + "path": { + "type": "string" + } + } + }, + "WorkflowModel": { + "type": "object", + "description": "A workflow.hivecommons.dev/v1alpha1 model document (states, transitions, columns, derivation, events, gates, roles, agents, checks, bookkeeping, labels, meta).", + "additionalProperties": true + }, + "WorkflowProfile": { + "type": "object", + "description": "Compiled, derivation-ready form of a model (hash, states, columns, ordered rules). Never trusted from the wire: the spoke recompiles on save and on bind.", + "additionalProperties": true + }, + "WorkflowIngestResult": { + "type": "object", + "properties": { + "model": { + "$ref": "#/components/schemas/WorkflowModel" + }, + "compiled": { + "type": "boolean" + }, + "diagnostics": { + "type": "array", + "items": { + "$ref": "#/components/schemas/WorkflowDiagnostic" + } + }, + "profile": { + "type": "object" + }, + "ref": { + "type": "string" + }, + "files": { + "type": "integer" + }, + "pipelines": { + "type": "integer" + }, + "entry": { + "$ref": "#/components/schemas/WorkflowEntry" + } + } + } } } } diff --git a/src/cmd/hive/main.go b/src/cmd/hive/main.go index 3467cd1173..33a3ce9ea3 100644 --- a/src/cmd/hive/main.go +++ b/src/cmd/hive/main.go @@ -5,6 +5,7 @@ import ( "crypto/rand" "encoding/json" "fmt" + wfstore "github.com/hivecommons/hive/pkg/workflowmodel/store" "io" "log/slog" "net/url" @@ -1088,6 +1089,8 @@ func (b *boot) wireBootClosures() { } return &dashboard.Dependencies{ Config: b.cfg, + WorkflowStore: wfstore.Open(wfstore.Root(b.cfg.Data.WorkflowsDir)), + WorkflowRepoFiles: workflowRepoFiles(func() *github.Client { return b.ghClient }), AgentMgr: b.agentMgr, CruxRuntimeHealth: cruxRuntimeHealthFunc(b.cruxSupervision), // Provider gateways (#5565 slice 3): concrete openrouter/watsonx/ diff --git a/src/cmd/hive/stage_engine.go b/src/cmd/hive/stage_engine.go index 58b5ae5e5d..72d5f41066 100644 --- a/src/cmd/hive/stage_engine.go +++ b/src/cmd/hive/stage_engine.go @@ -8,6 +8,8 @@ import ( "github.com/hivecommons/hive/pkg/github" "github.com/hivecommons/hive/pkg/stageengine" "github.com/hivecommons/hive/pkg/stagelabels" + "github.com/hivecommons/hive/pkg/workflowmodel" + wfstore "github.com/hivecommons/hive/pkg/workflowmodel/store" ) // The central stage engine (ADR-0025, hivecommons/hive#12999) is built once @@ -38,17 +40,52 @@ func stageEngineOptions(cfg *config.Config, logger *slog.Logger) *stageengine.Op logger = slog.Default() } stageEngine.opts = &stageengine.Options{ - Mode: stageEngineConfiguredMode(cfg), - Store: stagelabels.Shared(), - Issue: github.IssueDeriver(), - PR: stageengine.PRDeriver(), - Logger: logger, - Metrics: stageengine.DefaultCounters(), + Mode: stageEngineConfiguredMode(cfg), + Store: stagelabels.Shared(), + Issue: github.IssueDeriver(), + PR: stageengine.PRDeriver(), + Logger: logger, + Metrics: stageengine.DefaultCounters(), + WorkflowFor: stageEngineWorkflowFor(cfg, wfstore.Open(wfstore.Root(cfg.Data.WorkflowsDir)), logger), } }) return stageEngine.opts } +// stageEngineWorkflowFor resolves the workflow model bound to a repository +// (hivecommons/hive#17620). cfg is read on every call — hive.yaml reloads +// and dashboard binding edits mutate it in place — so a binding or the +// workflow.enabled flag takes effect on the next derivation. A bound model +// that is missing or no longer compiles is logged once per key and the +// repository falls back to hive-default rather than going unlabelled. +func stageEngineWorkflowFor(cfg *config.Config, st *wfstore.Store, logger *slog.Logger) func(owner, repo string) *workflowmodel.Profile { + if cfg == nil || st == nil { + return nil + } + var warned sync.Map + return func(owner, repo string) *workflowmodel.Profile { + if !cfg.Workflow.Enabled { + return nil + } + full := repo + if owner != "" { + full = owner + "/" + repo + } + b, ok := cfg.WorkflowBindingFor(full) + if !ok { + return nil + } + p, err := st.Profile(b.Name, b.Version) + if err != nil { + if _, dup := warned.LoadOrStore(b.Key()+"|"+err.Error(), true); !dup && logger != nil { + logger.Warn("workflow binding unusable; repository falls back to hive-default", "repo", full, "workflow", b.Key(), "err", err) + } + return nil + } + return p + } +} + func stageEngineConfiguredMode(cfg *config.Config) stageengine.Mode { if cfg == nil { return config.DefaultStageEngineMode diff --git a/src/cmd/hive/stage_engine_workflow_test.go b/src/cmd/hive/stage_engine_workflow_test.go new file mode 100644 index 0000000000..a41bc124de --- /dev/null +++ b/src/cmd/hive/stage_engine_workflow_test.go @@ -0,0 +1,64 @@ +package main + +import ( + "encoding/json" + "os" + "testing" + "time" + + "github.com/hivecommons/hive/pkg/config" + "github.com/hivecommons/hive/pkg/workflowmodel" + wfstore "github.com/hivecommons/hive/pkg/workflowmodel/store" +) + +func TestStageEngineWorkflowFor(t *testing.T) { + if stageEngineWorkflowFor(nil, nil, nil) != nil { + t.Fatal("nil inputs") + } + st := wfstore.Open(t.TempDir()) + b, err := os.ReadFile("../../pkg/workflowmodel/testdata/models/techdocs.json") + if err != nil { + t.Fatal(err) + } + var m workflowmodel.Model + if err := json.Unmarshal(b, &m); err != nil { + t.Fatal(err) + } + if _, err := st.Save(&m, "test", time.Now()); err != nil { + t.Fatal(err) + } + cfg := &config.Config{Project: config.ProjectConfig{Org: "cncf"}} + if _, err := cfg.SetWorkflowBinding("techdocs", "cncf-techdocs", "1"); err != nil { + t.Fatal(err) + } + if _, err := cfg.SetWorkflowBinding("ghost", "nope", "1"); err != nil { + t.Fatal(err) + } + lookup := stageEngineWorkflowFor(cfg, st, discardLogger()) + + if lookup("cncf", "techdocs") != nil { + t.Fatal("flag off must resolve nothing") + } + cfg.Workflow.Enabled = true + if p := lookup("cncf", "techdocs"); p == nil || p.Name != "cncf-techdocs" { + t.Fatalf("%+v", p) + } + if lookup("", "techdocs") == nil { + t.Fatal("bare repo should qualify with the org") + } + if lookup("cncf", "other") != nil { + t.Fatal("unbound repo resolved a model") + } + // Twice: the second call exercises the warn-once path. + for i := 0; i < 2; i++ { + if lookup("cncf", "ghost") != nil { + t.Fatal("missing model must fall back") + } + } + if _, err := cfg.SetWorkflowBinding("techdocs", "", ""); err != nil { + t.Fatal(err) + } + if lookup("cncf", "techdocs") != nil { + t.Fatal("unbinding must take effect live") + } +} diff --git a/src/cmd/hive/workflow_ingest.go b/src/cmd/hive/workflow_ingest.go new file mode 100644 index 0000000000..00f80f6d45 --- /dev/null +++ b/src/cmd/hive/workflow_ingest.go @@ -0,0 +1,37 @@ +package main + +import ( + "context" + "errors" + + "github.com/hivecommons/hive/pkg/github" + "github.com/hivecommons/hive/pkg/workflowmodel/ingest" +) + +// errNoForgeClient is returned by the workflow ingest fetcher before a +// GitHub client exists. +var errNoForgeClient = errors.New("no GitHub client") + +// workflowRepoFiles is the dashboard's WorkflowRepoFiles dependency +// (POST /api/workflows/ingest, RFC #17620 P3): it downloads the repo's +// default branch read-only, keeps only the CI definition files and +// removes the checkout. client is read per call so a credential rebuild is +// followed. +func workflowRepoFiles(client func() *github.Client) func(ctx context.Context, repo string) (map[string][]byte, string, error) { + return func(ctx context.Context, repo string) (map[string][]byte, string, error) { + c := client() + if c == nil { + return nil, "", errNoForgeClient + } + co, err := c.FetchDefaultBranchTree(ctx, repo) + if err != nil { + return nil, "", err + } + defer co.Cleanup() + files, err := ingest.ReadTree(co.Dir) + if err != nil { + return nil, "", err + } + return files, co.Ref, nil + } +} diff --git a/src/cmd/hive/workflow_ingest_test.go b/src/cmd/hive/workflow_ingest_test.go new file mode 100644 index 0000000000..b10e3e219c --- /dev/null +++ b/src/cmd/hive/workflow_ingest_test.go @@ -0,0 +1,16 @@ +package main + +import ( + "context" + "errors" + "testing" + + "github.com/hivecommons/hive/pkg/github" +) + +func TestWorkflowRepoFilesWithoutClient(t *testing.T) { + fetch := workflowRepoFiles(func() *github.Client { return nil }) + if _, _, err := fetch(context.Background(), "acme/widgets"); !errors.Is(err, errNoForgeClient) { + t.Fatalf("err = %v", err) + } +} diff --git a/src/cmd/workflow-survey/main.go b/src/cmd/workflow-survey/main.go new file mode 100644 index 0000000000..31ad3e2752 --- /dev/null +++ b/src/cmd/workflow-survey/main.go @@ -0,0 +1,492 @@ +// Command workflow-survey ingests every repository in a corpus directory +// into a workflow model and writes, per repository, the model JSON, the +// Mermaid diagrams and the Activity Monitor columns, plus a corpus-wide +// summary. It is the proving harness for hivecommons/hive#17620. +// +// Corpus layout (as collected by the survey script): /repos/ +// __/ with an index.json per repository. +package main + +import ( + "encoding/json" + "flag" + "fmt" + "io" + "os" + "path/filepath" + "sort" + "strings" + + "github.com/hivecommons/hive/pkg/workflowmodel" + "github.com/hivecommons/hive/pkg/workflowmodel/ingest" + "github.com/hivecommons/hive/pkg/workflowmodel/render" +) + +const ( + // worstN is how many repositories the summary lists by finding count. + worstN = 20 + // topN is how many repositories the summary lists by pipeline count. + topN = 20 + // repoDirSeparator joins owner and repo in a corpus directory name. + repoDirSeparator = "__" + // defaultForge is recorded in every model's scope. + defaultForge = "github.com" + // exitUsage and exitFailure are the process exit codes. + exitUsage = 2 + exitFailure = 1 +) + +// repoIndex is the per-repository index the collector writes. +type repoIndex struct { + Repo string `json:"repo"` + Project string `json:"project,omitempty"` + Files []struct { + Path string `json:"path"` + Tag string `json:"tag,omitempty"` + } `json:"files"` +} + +// repoResult is one row of the summary. +type repoResult struct { + Repo string `json:"repo"` + Project string `json:"project,omitempty"` + Files int `json:"files"` + Parsed int `json:"parsed"` + Pipelines int `json:"pipelines"` + GHAPipelines int `json:"ghaPipelines"` + TektonPipeline int `json:"tektonPipelines"` + Roles map[workflowmodel.Role]int `json:"roles"` + Findings int `json:"findings"` + FindingCodes map[string]int `json:"findingCodes,omitempty"` + Constructs map[string]int `json:"constructs"` + Columns []string `json:"columns"` + ColumnDiff []string `json:"columnDiff,omitempty"` + CISubstages int `json:"ciSubstages"` + MaxLevels int `json:"maxLevels"` + MaxNodes int `json:"maxNodes"` + Valid bool `json:"valid"` + ValidateError string `json:"validateError,omitempty"` + Hash string `json:"hash"` + Severities map[workflowmodel.Severity]int `json:"severities,omitempty"` + // Compiled says whether the lifecycle model (ingest.Lifecycle) compiles; + // CompileCodes counts its diagnostics by code, errors and warnings. + Compiled bool `json:"compiled"` + CompileError string `json:"compileError,omitempty"` + CompileCodes map[string]int `json:"compileCodes,omitempty"` + LifecycleCols int `json:"lifecycleColumns"` +} + +type summary struct { + Repos int `json:"repos"` + ReposWithGHA int `json:"reposWithGHA"` + ReposWithTekton int `json:"reposWithTekton"` + Files int `json:"files"` + Parsed int `json:"parsed"` + Pipelines int `json:"pipelines"` + Valid int `json:"valid"` + Compiled int `json:"compiled"` + CompileCodes map[string]int `json:"compileCodes"` + Roles map[workflowmodel.Role]int `json:"roles"` + Constructs map[string]int `json:"constructs"` + FindingCodes map[string]int `json:"findingCodes"` + Severities map[workflowmodel.Severity]int `json:"severities"` + ColumnSets map[string]int `json:"columnSets"` + ReposChangedCols int `json:"reposWithChangedColumns"` + ByProject map[string]*projectStats `json:"byProject"` + Worst []repoResult `json:"worst"` + Top []repoResult `json:"top"` + Repos_ []repoResult `json:"results"` +} + +type projectStats struct { + Repos int `json:"repos"` + Pipelines int `json:"pipelines"` + Findings int `json:"findings"` +} + +func main() { + os.Exit(realMain(os.Args[1:], os.Stdout, os.Stderr)) +} + +// realMain parses args and runs the survey, returning the exit code. It +// is separated from main so tests can drive the binary in-process. +func realMain(args []string, stdout, stderr io.Writer) int { + fs := flag.NewFlagSet("workflow-survey", flag.ContinueOnError) + fs.SetOutput(stderr) + corpus := fs.String("corpus", "", "corpus root (contains repos/)") + out := fs.String("out", "", "output root (default /out)") + only := fs.String("only", "", "comma-separated owner/repo filter") + if err := fs.Parse(args); err != nil { + return exitUsage + } + if *corpus == "" { + fmt.Fprintln(stderr, "usage: workflow-survey -corpus DIR [-out DIR] [-only owner/repo,...]") + return exitUsage + } + if *out == "" { + *out = filepath.Join(*corpus, "out") + } + filter := map[string]bool{} + for _, r := range strings.Split(*only, ",") { + if r = strings.TrimSpace(r); r != "" { + filter[r] = true + } + } + if err := run(*corpus, *out, filter, stdout, stderr); err != nil { + fmt.Fprintln(stderr, err) + return exitFailure + } + return 0 +} + +// loadProjects maps owner/repo to project maturity from the collector's +// projects.json, when present. +func loadProjects(corpus string) map[string]string { + out := map[string]string{} + b, err := os.ReadFile(filepath.Join(corpus, "projects.json")) + if err != nil { + return out + } + var projects []struct { + Project string `json:"project"` + Repos []string `json:"repos"` + } + if err := json.Unmarshal(b, &projects); err != nil { + return out + } + for _, p := range projects { + for _, r := range p.Repos { + out[strings.ToLower(r)] = p.Project + } + } + return out +} + +func run(corpus, out string, filter map[string]bool, stdout, stderr io.Writer) error { + dirs, err := os.ReadDir(filepath.Join(corpus, "repos")) + if err != nil { + return err + } + maturity := loadProjects(corpus) + sum := &summary{Roles: map[workflowmodel.Role]int{}, Constructs: map[string]int{}, FindingCodes: map[string]int{}, Severities: map[workflowmodel.Severity]int{}, CompileCodes: map[string]int{}, ColumnSets: map[string]int{}, ByProject: map[string]*projectStats{}} + for _, d := range dirs { + if !d.IsDir() { + continue + } + repo := strings.Replace(d.Name(), repoDirSeparator, "/", 1) + if len(filter) > 0 && !filter[repo] { + continue + } + res, err := surveyRepo(filepath.Join(corpus, "repos", d.Name()), repo, filepath.Join(out, d.Name())) + if res.Project == "" { + res.Project = maturity[strings.ToLower(repo)] + } + if err != nil { + fmt.Fprintf(stderr, "%s: %v\n", repo, err) + continue + } + fmt.Fprintf(stdout, "%-50s files=%-3d parsed=%-3d pipelines=%-3d findings=%-3d cols=%d%s\n", repo, res.Files, res.Parsed, res.Pipelines, res.Findings, len(res.Columns), ternary(res.Valid, "", " INVALID")) + accumulate(sum, res) + } + finish(sum) + if err := os.MkdirAll(out, 0o755); err != nil { + return err + } + if err := writeJSON(filepath.Join(out, "summary.json"), sum); err != nil { + return err + } + return os.WriteFile(filepath.Join(out, "summary.md"), []byte(markdown(sum)), 0o644) +} + +func surveyRepo(dir, repo, outDir string) (repoResult, error) { + var idx repoIndex + if b, err := os.ReadFile(filepath.Join(dir, "index.json")); err == nil { + _ = json.Unmarshal(b, &idx) + } + files := map[string][]byte{} + err := filepath.WalkDir(dir, func(path string, d os.DirEntry, err error) error { + if err != nil || d.IsDir() || d.Name() == "index.json" { + return err + } + data, err := os.ReadFile(path) + if err != nil { + return err + } + rel, _ := filepath.Rel(dir, path) + files[filepath.ToSlash(rel)] = data + return nil + }) + if err != nil { + return repoResult{}, err + } + m := ingest.Repository(workflowmodel.Scope{Forge: defaultForge, Repository: repo}, files) + pipelines := m.Spec.Pipelines + cov := m.Spec.Coverage + var ghaCount, tknCount int + for _, p := range pipelines { + if p.System == workflowmodel.SystemTekton { + tknCount++ + } else { + ghaCount++ + } + } + + res := repoResult{Repo: repo, Project: idx.Project, Files: cov.Files, Parsed: cov.Parsed, Pipelines: len(pipelines), GHAPipelines: ghaCount, TektonPipeline: tknCount, Roles: map[workflowmodel.Role]int{}, FindingCodes: map[string]int{}, Constructs: map[string]int{}, Severities: map[workflowmodel.Severity]int{}, Findings: len(cov.Findings), Hash: m.Hash()} + if err := m.Validate(); err != nil { + res.ValidateError = err.Error() + } else { + res.Valid = true + } + cov = m.Spec.Coverage + res.Findings = len(cov.Findings) + for _, f := range cov.Findings { + res.FindingCodes[f.Code]++ + res.Severities[f.Severity]++ + } + for _, p := range pipelines { + res.Roles[p.Role]++ + countConstructs(res.Constructs, p) + if p.Levels > res.MaxLevels { + res.MaxLevels = p.Levels + } + if len(p.Nodes) > res.MaxNodes { + res.MaxNodes = len(p.Nodes) + } + } + cols := render.Columns(m) + for _, c := range cols { + res.Columns = append(res.Columns, c.Label) + res.CISubstages += len(c.Substages) + } + res.ColumnDiff = render.Diff(m) + + lm := ingest.Lifecycle(workflowmodel.Scope{Forge: defaultForge, Repository: repo}, files) + res.LifecycleCols = len(lm.Spec.Columns) + res.CompileCodes = map[string]int{} + _, diags, cerr := workflowmodel.Compile(lm) + for _, d := range diags { + res.CompileCodes[d.Code]++ + } + if cerr != nil { + res.CompileError = cerr.Error() + } else { + res.Compiled = true + } + + if err := os.MkdirAll(outDir, 0o755); err != nil { + return res, err + } + if err := writeJSON(filepath.Join(outDir, "lifecycle.json"), lm); err != nil { + return res, err + } + if err := writeJSON(filepath.Join(outDir, "model.json"), m); err != nil { + return res, err + } + if err := writeJSON(filepath.Join(outDir, "columns.json"), cols); err != nil { + return res, err + } + var mm strings.Builder + mm.WriteString("```mermaid\n" + render.Mermaid(m) + "```\n") + for i := range m.Spec.Pipelines { + mm.WriteString("\n```mermaid\n" + render.PipelineMermaid(&m.Spec.Pipelines[i]) + "```\n") + } + if err := os.WriteFile(filepath.Join(outDir, "diagram.md"), []byte(mm.String()), 0o644); err != nil { + return res, err + } + return res, writeJSON(filepath.Join(outDir, "result.json"), res) +} + +func countConstructs(c map[string]int, p workflowmodel.Pipeline) { + c["pipelines"]++ + if p.Concurrency != nil { + c["concurrency"]++ + } + for _, t := range p.Triggers { + c["trigger:"+t.Event]++ + if len(t.Paths) > 0 { + c["trigger-paths"]++ + } + } + for _, n := range p.Nodes { + c["nodes"]++ + c["node:"+string(n.Kind)]++ + if len(n.Needs) > 0 { + c["needs"]++ + } + if n.Matrix != nil { + c["matrix"]++ + if n.Matrix.Dynamic { + c["matrix-dynamic"]++ + } + if n.Matrix.Include > 0 || n.Matrix.Exclude > 0 { + c["matrix-include-exclude"]++ + } + } + if n.Environment != "" { + c["environment"]++ + } + if n.If != "" { + c["if"]++ + } + if len(n.When) > 0 { + c["when"]++ + } + } +} + +func accumulate(s *summary, r repoResult) { + s.Repos++ + s.Files += r.Files + s.Parsed += r.Parsed + s.Pipelines += r.Pipelines + if r.GHAPipelines > 0 { + s.ReposWithGHA++ + } + if r.TektonPipeline > 0 { + s.ReposWithTekton++ + } + if r.Valid { + s.Valid++ + } + if r.Compiled { + s.Compiled++ + } + for k, v := range r.CompileCodes { + s.CompileCodes[k] += v + } + if len(r.ColumnDiff) > 0 { + s.ReposChangedCols++ + } + for k, v := range r.Roles { + s.Roles[k] += v + } + for k, v := range r.Constructs { + s.Constructs[k] += v + } + for k, v := range r.FindingCodes { + s.FindingCodes[k] += v + } + for k, v := range r.Severities { + s.Severities[k] += v + } + s.ColumnSets[strings.Join(r.Columns, " | ")]++ + proj := r.Project + if proj == "" { + proj = "unknown" + } + ps := s.ByProject[proj] + if ps == nil { + ps = &projectStats{} + s.ByProject[proj] = ps + } + ps.Repos++ + ps.Pipelines += r.Pipelines + ps.Findings += r.Findings + s.Repos_ = append(s.Repos_, r) +} + +func finish(s *summary) { + sort.Slice(s.Repos_, func(i, j int) bool { return s.Repos_[i].Repo < s.Repos_[j].Repo }) + worst := append([]repoResult(nil), s.Repos_...) + sort.SliceStable(worst, func(i, j int) bool { + if worst[i].Valid != worst[j].Valid { + return !worst[i].Valid + } + return worst[i].Findings > worst[j].Findings + }) + s.Worst = worst[:min(worstN, len(worst))] + top := append([]repoResult(nil), s.Repos_...) + sort.SliceStable(top, func(i, j int) bool { return top[i].Pipelines > top[j].Pipelines }) + s.Top = top[:min(topN, len(top))] +} + +func markdown(s *summary) string { + var b strings.Builder + fmt.Fprintf(&b, "# Workflow model survey\n\n") + fmt.Fprintf(&b, "| | |\n|---|---|\n") + fmt.Fprintf(&b, "| Repositories | %d |\n| with GitHub Actions pipelines | %d |\n| with Tekton pipelines | %d |\n", s.Repos, s.ReposWithGHA, s.ReposWithTekton) + fmt.Fprintf(&b, "| Files read | %d |\n| Files parsed | %d (%.1f%%) |\n| Pipelines | %d |\n| Models valid | %d / %d |\n| Lifecycle models compiled | %d / %d |\n| Repos whose columns change | %d |\n\n", s.Files, s.Parsed, pct(s.Parsed, s.Files), s.Pipelines, s.Valid, s.Repos, s.Compiled, s.Repos, s.ReposChangedCols) + b.WriteString("## Lifecycle compile diagnostics\n\n| Code | Count |\n|---|---|\n") + for _, k := range sortedKeys(s.CompileCodes) { + fmt.Fprintf(&b, "| %s | %d |\n", k, s.CompileCodes[k]) + } + b.WriteString("\n") + b.WriteString("## Pipeline roles\n\n| Role | Pipelines |\n|---|---|\n") + for _, k := range sortedKeys(s.Roles) { + fmt.Fprintf(&b, "| %s | %d |\n", k, s.Roles[workflowmodel.Role(k)]) + } + b.WriteString("\n## Constructs seen\n\n| Construct | Count |\n|---|---|\n") + for _, k := range sortedKeys(s.Constructs) { + fmt.Fprintf(&b, "| %s | %d |\n", k, s.Constructs[k]) + } + b.WriteString("\n## Findings by code\n\n| Code | Count |\n|---|---|\n") + for _, k := range sortedKeys(s.FindingCodes) { + fmt.Fprintf(&b, "| %s | %d |\n", k, s.FindingCodes[k]) + } + b.WriteString("\n## Findings by severity\n\n| Severity | Count |\n|---|---|\n") + for _, k := range sortedKeys(s.Severities) { + fmt.Fprintf(&b, "| %s | %d |\n", k, s.Severities[workflowmodel.Severity(k)]) + } + b.WriteString("\n## Column sets (Activity Monitor)\n\n| Repos | Columns |\n|---|---|\n") + type kv struct { + k string + v int + } + var sets []kv + for k, v := range s.ColumnSets { + sets = append(sets, kv{k, v}) + } + sort.Slice(sets, func(i, j int) bool { return sets[i].v > sets[j].v || (sets[i].v == sets[j].v && sets[i].k < sets[j].k) }) + for _, e := range sets { + fmt.Fprintf(&b, "| %d | %s |\n", e.v, e.k) + } + b.WriteString("\n## By project maturity\n\n| Maturity | Repos | Pipelines | Findings |\n|---|---|---|---|\n") + for _, k := range sortedKeys(s.ByProject) { + p := s.ByProject[k] + fmt.Fprintf(&b, "| %s | %d | %d | %d |\n", k, p.Repos, p.Pipelines, p.Findings) + } + fmt.Fprintf(&b, "\n## Top %d repositories by pipelines\n\n| Repo | Pipelines | Max nodes | Max levels | CI substages | Columns |\n|---|---|---|---|---|---|\n", topN) + for _, r := range s.Top { + fmt.Fprintf(&b, "| %s | %d | %d | %d | %d | %s |\n", r.Repo, r.Pipelines, r.MaxNodes, r.MaxLevels, r.CISubstages, strings.Join(r.Columns, " › ")) + } + fmt.Fprintf(&b, "\n## Worst %d repositories by findings\n\n| Repo | Valid | Findings | Codes |\n|---|---|---|---|\n", worstN) + for _, r := range s.Worst { + var codes []string + for _, k := range sortedKeys(r.FindingCodes) { + codes = append(codes, fmt.Sprintf("%s×%d", k, r.FindingCodes[k])) + } + fmt.Fprintf(&b, "| %s | %v | %d | %s |\n", r.Repo, r.Valid, r.Findings, strings.Join(codes, ", ")) + } + return b.String() +} + +func sortedKeys[V any, K ~string](m map[K]V) []string { + out := make([]string, 0, len(m)) + for k := range m { + out = append(out, string(k)) + } + sort.Strings(out) + return out +} + +func pct(n, d int) float64 { + if d == 0 { + return 0 + } + return float64(n) * 100 / float64(d) +} + +func ternary(c bool, a, b string) string { + if c { + return a + } + return b +} + +func writeJSON(path string, v any) error { + b, err := json.MarshalIndent(v, "", " ") + if err != nil { + return err + } + return os.WriteFile(path, append(b, '\n'), 0o644) +} diff --git a/src/cmd/workflow-survey/main_test.go b/src/cmd/workflow-survey/main_test.go new file mode 100644 index 0000000000..59d40c9f32 --- /dev/null +++ b/src/cmd/workflow-survey/main_test.go @@ -0,0 +1,249 @@ +package main + +import ( + "bytes" + "encoding/json" + "os" + "path/filepath" + "strings" + "testing" +) + +// fixtureCorpus is the curated ingest corpus shared with +// pkg/workflowmodel/ingest. The test copies it (symlinks would fail the +// collector layout's IsDir check) into the layout the collector writes. +const fixtureCorpus = "../../pkg/workflowmodel/testdata/corpus" + +// buildCorpus assembles a /repos/__ tree from the +// fixture directories plus a projects.json mapping one repo to a maturity. +func buildCorpus(t *testing.T) string { + t.Helper() + root := t.TempDir() + repos := filepath.Join(root, "repos") + entries, err := os.ReadDir(fixtureCorpus) + if err != nil { + t.Fatal(err) + } + var n int + for _, e := range entries { + if !e.IsDir() { + continue + } + src, err := filepath.Abs(filepath.Join(fixtureCorpus, e.Name())) + if err != nil { + t.Fatal(err) + } + if err := os.MkdirAll(repos, 0o755); err != nil { + t.Fatal(err) + } + if err := os.CopyFS(filepath.Join(repos, e.Name()), os.DirFS(src)); err != nil { + t.Fatal(err) + } + n++ + } + if n == 0 { + t.Fatalf("no fixture repositories under %s", fixtureCorpus) + } + // A stray file in repos/ must be ignored, as must a repo not in projects.json. + if err := os.WriteFile(filepath.Join(repos, "README.md"), []byte("x"), 0o644); err != nil { + t.Fatal(err) + } + projects := `[{"project":"graduated","repos":["ArgoProj/argo-cd"]},{"project":"incubating","repos":["cilium/cilium"]}]` + if err := os.WriteFile(filepath.Join(root, "projects.json"), []byte(projects), 0o644); err != nil { + t.Fatal(err) + } + return root +} + +func TestRealMainSurveysCorpus(t *testing.T) { + corpus := buildCorpus(t) + var stdout, stderr bytes.Buffer + if code := realMain([]string{"-corpus", corpus}, &stdout, &stderr); code != 0 { + t.Fatalf("exit %d, stderr: %s", code, stderr.String()) + } + if stderr.Len() != 0 { + t.Errorf("unexpected stderr: %s", stderr.String()) + } + for _, repo := range []string{"argoproj/argo-cd", "tektoncd/pipeline", "konflux-ci/konflux-ci"} { + if !strings.Contains(stdout.String(), repo) { + t.Errorf("stdout missing %s:\n%s", repo, stdout.String()) + } + } + if strings.Contains(stdout.String(), "INVALID") { + t.Errorf("fixture model reported invalid:\n%s", stdout.String()) + } + + out := filepath.Join(corpus, "out") + raw, err := os.ReadFile(filepath.Join(out, "summary.json")) + if err != nil { + t.Fatal(err) + } + var sum summary + if err := json.Unmarshal(raw, &sum); err != nil { + t.Fatal(err) + } + if sum.Repos != 6 || sum.Valid != sum.Repos { + t.Errorf("repos=%d valid=%d, want 6 valid", sum.Repos, sum.Valid) + } + if sum.ReposWithGHA == 0 || sum.ReposWithTekton == 0 { + t.Errorf("expected both systems in the corpus: gha=%d tekton=%d", sum.ReposWithGHA, sum.ReposWithTekton) + } + if sum.ReposChangedCols == 0 { + t.Error("expected at least one repo whose columns differ from the hive default") + } + if got := sum.ByProject["graduated"]; got == nil || got.Repos != 1 { + t.Errorf("projects.json maturity join (case-insensitive) not applied: %+v", sum.ByProject) + } + if got := sum.ByProject["unknown"]; got == nil || got.Repos != 4 { + t.Errorf("repos without a maturity should land in unknown: %+v", sum.ByProject) + } + if len(sum.Worst) != sum.Repos || len(sum.Top) != sum.Repos { + t.Errorf("worst/top should list every repo when under the cap: %d/%d", len(sum.Worst), len(sum.Top)) + } + for i := 1; i < len(sum.Top); i++ { + if sum.Top[i-1].Pipelines < sum.Top[i].Pipelines { + t.Errorf("top not sorted by pipelines desc at %d", i) + } + } + + md, err := os.ReadFile(filepath.Join(out, "summary.md")) + if err != nil { + t.Fatal(err) + } + for _, want := range []string{"# Workflow model survey", "## Pipeline roles", "## Column sets (Activity Monitor)", "| graduated | 1 |", "trigger:pull_request"} { + if !strings.Contains(string(md), want) { + t.Errorf("summary.md missing %q", want) + } + } + + for _, f := range []string{"model.json", "columns.json", "diagram.md", "result.json"} { + if _, err := os.Stat(filepath.Join(out, "argoproj__argo-cd", f)); err != nil { + t.Errorf("per-repo artifact missing: %v", err) + } + } + diagram, _ := os.ReadFile(filepath.Join(out, "konflux-ci__konflux-ci", "diagram.md")) + if !strings.Contains(string(diagram), "stateDiagram-v2") || !strings.Contains(string(diagram), "flowchart") { + t.Error("diagram.md should carry the state diagram and per-pipeline flowcharts") + } +} + +func TestRealMainOnlyFilterAndOutDir(t *testing.T) { + corpus := buildCorpus(t) + out := filepath.Join(t.TempDir(), "nested", "out") + var stdout, stderr bytes.Buffer + code := realMain([]string{"-corpus", corpus, "-out", out, "-only", " cilium/cilium , envoyproxy/envoy"}, &stdout, &stderr) + if code != 0 { + t.Fatalf("exit %d, stderr: %s", code, stderr.String()) + } + lines := strings.Split(strings.TrimSpace(stdout.String()), "\n") + if len(lines) != 2 { + t.Fatalf("filter should leave two repos, got %d:\n%s", len(lines), stdout.String()) + } + raw, err := os.ReadFile(filepath.Join(out, "summary.json")) + if err != nil { + t.Fatal(err) + } + var sum summary + if err := json.Unmarshal(raw, &sum); err != nil { + t.Fatal(err) + } + if sum.Repos != 2 { + t.Errorf("repos=%d, want 2", sum.Repos) + } + if _, err := os.Stat(filepath.Join(out, "argoproj__argo-cd")); !os.IsNotExist(err) { + t.Error("filtered-out repo should not be written") + } +} + +func TestRealMainErrors(t *testing.T) { + cases := []struct { + name string + args []string + code int + want string + }{ + {"no corpus flag", nil, exitUsage, "usage:"}, + {"unknown flag", []string{"-bogus"}, exitUsage, "flag provided but not defined"}, + {"missing corpus dir", []string{"-corpus", filepath.Join(t.TempDir(), "nope")}, exitFailure, "no such file"}, + } + for _, tc := range cases { + t.Run(tc.name, func(t *testing.T) { + var stdout, stderr bytes.Buffer + if code := realMain(tc.args, &stdout, &stderr); code != tc.code { + t.Errorf("exit %d, want %d", code, tc.code) + } + if !strings.Contains(stderr.String(), tc.want) { + t.Errorf("stderr %q should contain %q", stderr.String(), tc.want) + } + }) + } +} + +func TestRunReportsUnreadableRepo(t *testing.T) { + corpus := buildCorpus(t) + // A repo directory containing an unreadable file is reported on + // stderr and skipped rather than aborting the survey. + bad := filepath.Join(corpus, "repos", "acme__broken", ".github", "workflows") + if err := os.MkdirAll(bad, 0o755); err != nil { + t.Fatal(err) + } + if err := os.WriteFile(filepath.Join(bad, "ci.yml"), []byte("on: push\njobs: {}\n"), 0o000); err != nil { + t.Fatal(err) + } + if os.Getuid() == 0 { + t.Skip("root can read mode 0000 files") + } + var stdout, stderr bytes.Buffer + if code := realMain([]string{"-corpus", corpus}, &stdout, &stderr); code != 0 { + t.Fatalf("exit %d: %s", code, stderr.String()) + } + if !strings.Contains(stderr.String(), "acme/broken") || !strings.Contains(stderr.String(), "permission denied") { + t.Errorf("stderr should name the skipped repo: %q", stderr.String()) + } + if strings.Contains(stdout.String(), "acme/broken") { + t.Error("skipped repo must not be counted") + } +} + +func TestRunFailsWhenOutputUnwritable(t *testing.T) { + if os.Getuid() == 0 { + t.Skip("root ignores directory permissions") + } + corpus := buildCorpus(t) + out := filepath.Join(t.TempDir(), "ro") + if err := os.MkdirAll(out, 0o500); err != nil { + t.Fatal(err) + } + var stdout, stderr bytes.Buffer + if code := realMain([]string{"-corpus", corpus, "-out", out}, &stdout, &stderr); code != exitFailure { + t.Fatalf("exit %d, want %d; stderr: %s", code, exitFailure, stderr.String()) + } + if !strings.Contains(stderr.String(), "permission denied") { + t.Errorf("stderr %q should explain the write failure", stderr.String()) + } +} + +func TestHelpers(t *testing.T) { + if pct(1, 0) != 0 || pct(1, 4) != 25 { + t.Error("pct") + } + if ternary(true, "a", "b") != "a" || ternary(false, "a", "b") != "b" { + t.Error("ternary") + } + if got := sortedKeys(map[string]int{"b": 1, "a": 2}); strings.Join(got, ",") != "a,b" { + t.Errorf("sortedKeys = %v", got) + } + if loadProjects(t.TempDir()) == nil { + t.Error("loadProjects without projects.json should return an empty map") + } + dir := t.TempDir() + if err := os.WriteFile(filepath.Join(dir, "projects.json"), []byte("{not json"), 0o644); err != nil { + t.Fatal(err) + } + if m := loadProjects(dir); len(m) != 0 { + t.Errorf("malformed projects.json should yield no entries, got %v", m) + } + if err := writeJSON(filepath.Join(dir, "x.json"), make(chan int)); err == nil { + t.Error("writeJSON should surface marshal errors") + } +} diff --git a/src/cmd/workflow-survey/workflow-survey b/src/cmd/workflow-survey/workflow-survey new file mode 100755 index 0000000000..faf72a53d2 Binary files /dev/null and b/src/cmd/workflow-survey/workflow-survey differ diff --git a/src/docs/api-reference/agents-and-controls.md b/src/docs/api-reference/agents-and-controls.md index 1523bc34b8..086d85bb17 100644 --- a/src/docs/api-reference/agents-and-controls.md +++ b/src/docs/api-reference/agents-and-controls.md @@ -6,43 +6,43 @@ Part of the [Dashboard REST API reference](../api-reference.md); see that page f | Method | Path | Auth | Purpose | Source | |---|---|---|---|---| -| `POST` | `/api/kick/{agent}` | Owner only | Kick — asynchronous; answers `202` once queued (see below) | `pkg/dashboard/api.go:138` | -| `GET` | `/api/kick/{agent}/status` | Dashboard auth/session | Outcome of the most recent kick | `pkg/dashboard/api.go:142` | -| `POST` | `/api/switch/{agent}/{backend}` | Owner only | Switch | `pkg/dashboard/api.go:143` | -| `POST` | `/api/model/{agent}/{model}` | Owner only | Model Set | `pkg/dashboard/api.go:144` | -| `POST` | `/api/effort/{agent}/{effort}` | Owner only | Set launch-only reasoning effort after validating against the agent's live backend (including runtime override); persists config/agent overlay, restarts the session, and `default` clears the stored effort | `pkg/dashboard/api.go:145` | -| `POST` | `/api/pause/{agent}` | Owner only | Pause | `pkg/dashboard/api.go:146` | -| `POST` | `/api/resume/{agent}` | Owner only | Resume | `pkg/dashboard/api.go:147` | -| `GET` | `/api/agent-state/{agent}` | Dashboard auth/session | Agent State | `pkg/dashboard/api.go:148` | -| `GET` | `/api/breaker` | Dashboard auth/session | Breaker State | `pkg/dashboard/api.go:149` | -| `POST` | `/api/breaker/engage` | Owner only | Breaker Engage | `pkg/dashboard/api.go:150` | -| `POST` | `/api/breaker/release` | Owner only | Breaker Release | `pkg/dashboard/api.go:151` | -| `POST` | `/api/pin/{agent}/{dimension}` | Owner only | Pin | `pkg/dashboard/api.go:152` | -| `POST` | `/api/unpin/{agent}/{dimension}` | Owner only | Unpin | `pkg/dashboard/api.go:153` | -| `POST` | `/api/restart/{agent}` | Owner only | Restart | `pkg/dashboard/api.go:157` | -| `GET` | `/api/model-advisor` | Dashboard auth/session | Model Advisor | `pkg/dashboard/api.go:174` | -| `GET` | `/api/governor/pr-models` | Dashboard auth/session | Agent-authored PR distribution by normalized attribution model/backend for `window=7d`, `30d`, or `all`, including per-model rework stats and the top 10 most-reworked PRs | `pkg/dashboard/api.go:175` | -| `GET` | `/api/reviewer/accuracy` | Dashboard auth/session | Reviewer verdict calibration for the configured window: false-approve/false-block rates and confidence buckets per perspective and reviewer model | `pkg/dashboard/api.go:238` | -| `GET` | `/api/review/queue` | Dashboard auth/session | PR review queue: every open PR (agent and contributor, actionable and held) ranked by triage class, confidence band, CI state and age, with per-PR `reasons`; unreviewed items carry `pending_since` and `redispatch_count` from the review dispatch state; each entry carries `stage` `{stage, entered_at, last_activity_at, quiet_for, stalled, waiting_on_hive, past_dwell, history[]}` from the stage-label store (`stalled`/`past_dwell` follow `governor.labels.stage.dwell` and `stalled.require_quiet`; `history[]` is `{from, to, at, reason, regressed}`, newest last, capped at 8) and the top-level `hive_held` counts PRs past their stage dwell while waiting on the hive (`waiting_on_hive` and `past_dwell` both true); paged with `limit` (1-200, default 50) and `offset` | `pkg/dashboard/api.go:514` | -| `POST` | `/api/review/queue/{owner}/{repo}/{number}/superseded/close` | Owner only | Close a superseded PR now (grace window or not): posts the close comment, adds `hive/superseded`, closes it | `pkg/dashboard/api.go:517` | -| `POST` | `/api/review/queue/{owner}/{repo}/{number}/superseded/keep-open` | Owner only | Keep a superseded PR open: adds `hive/keep-open`, which the supersession sweep never auto-closes | `pkg/dashboard/api.go:518` | -| `GET` | `/api/review/pipeline` | Dashboard auth/session | Review pipeline: one card per review-queue PR placed in its stage (`unreviewed`, `reviewing`, `changes_requested`, `fixing`, `human_hold`, `approved`, `merged`, `abandoned`) with `since`, `reviewers`, P0–P3 `severity` counts, `loop_count`/`loop_cap`, `next_action` and `reasons`; filtered by `repo` and `stage`, paged with `limit` (1-200, default 50) and `offset` | `pkg/dashboard/api.go:522` | +| `POST` | `/api/kick/{agent}` | Owner only | Kick — asynchronous; answers `202` once queued (see below) | `pkg/dashboard/api.go:139` | +| `GET` | `/api/kick/{agent}/status` | Dashboard auth/session | Outcome of the most recent kick | `pkg/dashboard/api.go:143` | +| `POST` | `/api/switch/{agent}/{backend}` | Owner only | Switch | `pkg/dashboard/api.go:144` | +| `POST` | `/api/model/{agent}/{model}` | Owner only | Model Set | `pkg/dashboard/api.go:145` | +| `POST` | `/api/effort/{agent}/{effort}` | Owner only | Set launch-only reasoning effort after validating against the agent's live backend (including runtime override); persists config/agent overlay, restarts the session, and `default` clears the stored effort | `pkg/dashboard/api.go:146` | +| `POST` | `/api/pause/{agent}` | Owner only | Pause | `pkg/dashboard/api.go:147` | +| `POST` | `/api/resume/{agent}` | Owner only | Resume | `pkg/dashboard/api.go:148` | +| `GET` | `/api/agent-state/{agent}` | Dashboard auth/session | Agent State | `pkg/dashboard/api.go:149` | +| `GET` | `/api/breaker` | Dashboard auth/session | Breaker State | `pkg/dashboard/api.go:150` | +| `POST` | `/api/breaker/engage` | Owner only | Breaker Engage | `pkg/dashboard/api.go:151` | +| `POST` | `/api/breaker/release` | Owner only | Breaker Release | `pkg/dashboard/api.go:152` | +| `POST` | `/api/pin/{agent}/{dimension}` | Owner only | Pin | `pkg/dashboard/api.go:153` | +| `POST` | `/api/unpin/{agent}/{dimension}` | Owner only | Unpin | `pkg/dashboard/api.go:154` | +| `POST` | `/api/restart/{agent}` | Owner only | Restart | `pkg/dashboard/api.go:158` | +| `GET` | `/api/model-advisor` | Dashboard auth/session | Model Advisor | `pkg/dashboard/api.go:175` | +| `GET` | `/api/governor/pr-models` | Dashboard auth/session | Agent-authored PR distribution by normalized attribution model/backend for `window=7d`, `30d`, or `all`, including per-model rework stats and the top 10 most-reworked PRs | `pkg/dashboard/api.go:176` | +| `GET` | `/api/reviewer/accuracy` | Dashboard auth/session | Reviewer verdict calibration for the configured window: false-approve/false-block rates and confidence buckets per perspective and reviewer model | `pkg/dashboard/api.go:239` | +| `GET` | `/api/review/queue` | Dashboard auth/session | PR review queue: every open PR (agent and contributor, actionable and held) ranked by triage class, confidence band, CI state and age, with per-PR `reasons`; unreviewed items carry `pending_since` and `redispatch_count` from the review dispatch state; each entry carries `stage` `{stage, entered_at, last_activity_at, quiet_for, stalled, waiting_on_hive, past_dwell, history[]}` from the stage-label store (`stalled`/`past_dwell` follow `governor.labels.stage.dwell` and `stalled.require_quiet`; `history[]` is `{from, to, at, reason, regressed}`, newest last, capped at 8) and the top-level `hive_held` counts PRs past their stage dwell while waiting on the hive (`waiting_on_hive` and `past_dwell` both true); paged with `limit` (1-200, default 50) and `offset` | `pkg/dashboard/api.go:515` | +| `POST` | `/api/review/queue/{owner}/{repo}/{number}/superseded/close` | Owner only | Close a superseded PR now (grace window or not): posts the close comment, adds `hive/superseded`, closes it | `pkg/dashboard/api.go:518` | +| `POST` | `/api/review/queue/{owner}/{repo}/{number}/superseded/keep-open` | Owner only | Keep a superseded PR open: adds `hive/keep-open`, which the supersession sweep never auto-closes | `pkg/dashboard/api.go:519` | +| `GET` | `/api/review/pipeline` | Dashboard auth/session | Review pipeline: one card per review-queue PR placed in its stage (`unreviewed`, `reviewing`, `changes_requested`, `fixing`, `human_hold`, `approved`, `merged`, `abandoned`) with `since`, `reviewers`, P0–P3 `severity` counts, `loop_count`/`loop_cap`, `next_action` and `reasons`; filtered by `repo` and `stage`, paged with `limit` (1-200, default 50) and `offset` | `pkg/dashboard/api.go:523` | | `GET` | `/api/review/dispatch/events` | Dashboard auth/session | Webhook review dispatch queue: `enabled`, `webhooks_configured`, `debounce_s`, `pending` (soonest first) and `recent` (newest first) dispatches with repo, number, head SHA, action, timestamps and coalesced count, plus `stats` counters; in memory, never calls GitHub | `pkg/dashboard/api_review_event_dispatch.go:16` | -| `GET` | `/api/review/evidence` | Merger or owner | [Review evidence bundle](../review-evidence.md) for one PR head. Query `repo` (`owner/name`), `number`, optional `head` (full SHA or unique prefix of at least 7 characters; `409` when ambiguous; default is the newest head), `format=json\|zip` (default `json`) and `download=1` for an attachment. `json` returns the bundle bytes exactly as sealed (verdicts, posted reviews, policy, CI check runs, sentinel findings, human actions, merge event, `hash`, optional `signature`); `zip` adds `verdicts.json`, `review-links.json` and a `manifest.json` of SHA-256s plus the signing public key when readable. `400` for a bad `repo`/`number`; `404` with `status: "expired"`/`expired: true` when retention removed a bundle Hive can prove existed, otherwise `status: "not_found"` | `pkg/dashboard/api.go:526` | -| `GET` | `/api/review/evidence/list` | Merger or owner | Every retained evidence bundle id and head for one PR (`repo`, `number`), oldest first, with the newest marked latest; `400` for a bad `repo`/`number` | `pkg/dashboard/api.go:527` | -| `GET` | `/api/evidence/write-counters` | Merger or owner | Durable attempted vs blocked agent GitHub write counters ([#12069](https://github.com/hivecommons/hive/issues/12069)): `totals` by intent tier (ACMM mode) with `blocked_by_reason`, and `weeks` keyed by the UTC Monday of each ISO week with per-tier and per-reason counts. Persisted across restarts; observation only, never gates a write or merge. Empty `totals`/`weeks` before anything is recorded | `pkg/dashboard/api.go:529` | -| `GET` | `/api/evidence/summary` | Dashboard auth/session | Read-only aggregate over `/data/evidence` bundles (schema v1) plus reviewer accuracy. Query `hours` (default 24, capped like `/api/pr-throughput`; `0` = all time) or a custom `from`/`to` range (RFC3339 or `YYYY-MM-DD`, date-only `to` inclusive, `to` defaults to now; `400` when `to` precedes `from`, the span exceeds 3 × 366 days, or `to` is given without `from`; the response echoes `from`/`to`), `repo` (`owner/name`) and `author_kind=human\|agent\|bot`; `400` for a bad value. A PR falls in the window by its open time (earliest bundle `created_at` or `merge.opened_at`), never by when a bundle was written; all bundles of a PR are folded so the newest supplies verdicts and the merge / `reverted_by` / backfilled close come from whichever record knows them. Returns `available_windows` (`{"24","168","720","all"}` → PR count under the same filters), `outcomes_by_author_kind` (merged/closed/open; `closed` comes from backfilled records, `reverted_14d` is null until bundles record it), `reviewer_calibration` buckets per perspective, `reviewer_accuracy_by_model` split by author kind (blocks split into `enforced_blocks` / `advisory_blocks` by `policy.require_approval`; advisory blocks that preceded a merge are `dissent_before_merge`, not false blocks), `sentinel_hits_per_week`, `shadow_gate` (`n`, `would_have_held`, `would_have_blocked`, and per gate `review_gate` / `intent` the evaluated `n` with `held` / `passed` outcomes `{n, merged, closed, open, reverted}`, plus `review_gate_by_lane` `{merge_eligible, self_merge}`), `write_counters` (per intent tier `{attempted, blocked, blocked_by_reason}` from the proxy write counter file; omitted when absent or unreadable), `write_activity` (blocked ÷ every agent write in the window: `total_writes`, `blocked`, `direct` `{recorded, since, partial_coverage, attempted, blocked, by_tier, blocked_by_reason}` from the proxy's weekly counter buckets under `/data/hive-metrics/write-counters.json`, `mediated` `{n, refused, by_action}` from the audit log), `claim_vs_verdict` (per ISO week of agent-authored PRs: `n`, `claimed`, `ci_captured`, `ci_not_green`, `verdict_not_approve`, `both`; records with empty CI are excluded from the CI column and disclosed under `missing.records_empty_ci`), `ci_recovery` (`policy` `{self_authored_automerge, required_checks}`, `n`, `at_claim` `{green, red, pending, none}`, `red` `{n, recovered_merged, green_open, abandoned, pending, outcome_unknown, enriched, not_enriched, median_pushes_to_green, median_hours_to_green, same_head_recovered}`, `gate` `{merged_n, with_checks, green, green_from_bundles, non_required_red, required_unknown, breaches, breach_prs[], suspected, suspected_prs[], pending, unknown, green_to_merge_n, median_green_to_merge_hours}` over every merged PR by GitHub-confirmed state of the *required* checks on the merged head, as the merge gate judges (`breach_prs[]` carry `red`, `merged_by`; a red non-required check is disclosed, not a breach; a red review-time snapshot, a required check with no run, or a red read without a resolvable required set is only suspected), and `convergence` `{n, passes {"1","2","3","4+","never"}, settled, in_flight, unknown, median_passes, from_github, from_bundles, by_week[]}` over agent PRs by CI passes to first green, from the durable `/data/hive-metrics/ci-recovery.json` GitHub check-run trail rollup), `work_balance` (per ISO week: `n`, `opened` by actor kind `{human, agent, bot}`, `closed`, `prs_opened`, `prs_merged`, `prs_closed`, `queue_issues`, `queue_prs`, `issue_source` `github|audit`, `throttled`) with `work_balance_sources` (`issues_github_since`, `issues_fetched_at`, `issues_known`, `audit_only_weeks`, `issue_series_weeks`, `queue_samples`, `queue_retention_days`; issues from the durable GitHub rollup `/data/hive-metrics/issue-activity.json` for weeks at or after `issues_github_since` and, for earlier weeks, from the backfilled issue series `///issues/.json` written by the issue backfill ([#12527](https://github.com/hivecommons/hive/pull/12527)) when its coverage spans the requested portion of the week (`issue_series_weeks` counts the repo-weeks it supplied), else the audit log (`audit_only_weeks`), PRs from folded bundles, queue depth from governor snapshots; `missing.issues_human_opens_unobserved` is true while no GitHub rollup exists unless the issue series covers every row, and whenever any row fell back to the audit log, `missing.throttle_windows_unavailable` discloses that no governor throttle windows are reported), `intent_counters` (per intent tier `{attempted, blocked}` from `/data/hive-metrics/intent-counters.json` — evaluations and the subset the intent gate would have blocked with enforcement on, [#12362](https://github.com/hivecommons/hive/issues/12362); omitted when absent or unreadable), `complexity_by_repo` (per managed repo the newest 12 monthly complexity snapshots, oldest first, each `{month, ref, captured_at, files, functions, mean_cyclomatic, median_cyclomatic, p90_cyclomatic, max_cyclomatic, functions_over_threshold, threshold, static_warnings, static_warnings_by_tool}`; omitted when no repo has a snapshot), the `/api/reviewer/accuracy` summary, and `n` plus `missing` counts (unknown-model verdicts, empty CI, unreadable records). Never consulted by merge decisions. | `pkg/dashboard/api.go:532` | -| `GET` | `/api/agents` | Dashboard auth/session | Agents List | `pkg/dashboard/api.go:319` | -| `POST` | `/api/agents` | Owner only | Agent Create | `pkg/dashboard/api.go:320` | -| `POST` | `/api/agents/import` | Owner only | Agent Import | `pkg/dashboard/api.go:321` | -| `DELETE` | `/api/agents/{name}` | Owner only | Agent Delete | `pkg/dashboard/api.go:322` | - -| `GET` | `/api/agents/{name}/log` | Dashboard auth/session | Agent Full Log | `pkg/dashboard/api.go:125` | -| `GET` | `/api/agents/{name}/terminal-urls` | Dashboard auth/session | Agent Terminal URLs | `pkg/dashboard/api.go:129` | -| `GET` | `/api/agents/{name}/kicks` | Dashboard auth/session | Agent Kick Log List | `pkg/dashboard/api.go:132` | -| `GET` | `/api/agents/{name}/kicks/{id}` | Dashboard auth/session | Agent Kick Log Get | `pkg/dashboard/api.go:133` | -| `GET` | `/agents/{name}/kicks` | Dashboard auth/session | Agent Kick History Page (HTML) | `pkg/dashboard/api.go:134` | -| `POST` | `/api/agents/{name}/login-code` | Owner only | Agent Login Code | `pkg/dashboard/api.go:156` | +| `GET` | `/api/review/evidence` | Merger or owner | [Review evidence bundle](../review-evidence.md) for one PR head. Query `repo` (`owner/name`), `number`, optional `head` (full SHA or unique prefix of at least 7 characters; `409` when ambiguous; default is the newest head), `format=json\|zip` (default `json`) and `download=1` for an attachment. `json` returns the bundle bytes exactly as sealed (verdicts, posted reviews, policy, CI check runs, sentinel findings, human actions, merge event, `hash`, optional `signature`); `zip` adds `verdicts.json`, `review-links.json` and a `manifest.json` of SHA-256s plus the signing public key when readable. `400` for a bad `repo`/`number`; `404` with `status: "expired"`/`expired: true` when retention removed a bundle Hive can prove existed, otherwise `status: "not_found"` | `pkg/dashboard/api.go:527` | +| `GET` | `/api/review/evidence/list` | Merger or owner | Every retained evidence bundle id and head for one PR (`repo`, `number`), oldest first, with the newest marked latest; `400` for a bad `repo`/`number` | `pkg/dashboard/api.go:528` | +| `GET` | `/api/evidence/write-counters` | Merger or owner | Durable attempted vs blocked agent GitHub write counters ([#12069](https://github.com/hivecommons/hive/issues/12069)): `totals` by intent tier (ACMM mode) with `blocked_by_reason`, and `weeks` keyed by the UTC Monday of each ISO week with per-tier and per-reason counts. Persisted across restarts; observation only, never gates a write or merge. Empty `totals`/`weeks` before anything is recorded | `pkg/dashboard/api.go:530` | +| `GET` | `/api/evidence/summary` | Dashboard auth/session | Read-only aggregate over `/data/evidence` bundles (schema v1) plus reviewer accuracy. Query `hours` (default 24, capped like `/api/pr-throughput`; `0` = all time) or a custom `from`/`to` range (RFC3339 or `YYYY-MM-DD`, date-only `to` inclusive, `to` defaults to now; `400` when `to` precedes `from`, the span exceeds 3 × 366 days, or `to` is given without `from`; the response echoes `from`/`to`), `repo` (`owner/name`) and `author_kind=human\|agent\|bot`; `400` for a bad value. A PR falls in the window by its open time (earliest bundle `created_at` or `merge.opened_at`), never by when a bundle was written; all bundles of a PR are folded so the newest supplies verdicts and the merge / `reverted_by` / backfilled close come from whichever record knows them. Returns `available_windows` (`{"24","168","720","all"}` → PR count under the same filters), `outcomes_by_author_kind` (merged/closed/open; `closed` comes from backfilled records, `reverted_14d` is null until bundles record it), `reviewer_calibration` buckets per perspective, `reviewer_accuracy_by_model` split by author kind (blocks split into `enforced_blocks` / `advisory_blocks` by `policy.require_approval`; advisory blocks that preceded a merge are `dissent_before_merge`, not false blocks), `sentinel_hits_per_week`, `shadow_gate` (`n`, `would_have_held`, `would_have_blocked`, and per gate `review_gate` / `intent` the evaluated `n` with `held` / `passed` outcomes `{n, merged, closed, open, reverted}`, plus `review_gate_by_lane` `{merge_eligible, self_merge}`), `write_counters` (per intent tier `{attempted, blocked, blocked_by_reason}` from the proxy write counter file; omitted when absent or unreadable), `write_activity` (blocked ÷ every agent write in the window: `total_writes`, `blocked`, `direct` `{recorded, since, partial_coverage, attempted, blocked, by_tier, blocked_by_reason}` from the proxy's weekly counter buckets under `/data/hive-metrics/write-counters.json`, `mediated` `{n, refused, by_action}` from the audit log), `claim_vs_verdict` (per ISO week of agent-authored PRs: `n`, `claimed`, `ci_captured`, `ci_not_green`, `verdict_not_approve`, `both`; records with empty CI are excluded from the CI column and disclosed under `missing.records_empty_ci`), `ci_recovery` (`policy` `{self_authored_automerge, required_checks}`, `n`, `at_claim` `{green, red, pending, none}`, `red` `{n, recovered_merged, green_open, abandoned, pending, outcome_unknown, enriched, not_enriched, median_pushes_to_green, median_hours_to_green, same_head_recovered}`, `gate` `{merged_n, with_checks, green, green_from_bundles, non_required_red, required_unknown, breaches, breach_prs[], suspected, suspected_prs[], pending, unknown, green_to_merge_n, median_green_to_merge_hours}` over every merged PR by GitHub-confirmed state of the *required* checks on the merged head, as the merge gate judges (`breach_prs[]` carry `red`, `merged_by`; a red non-required check is disclosed, not a breach; a red review-time snapshot, a required check with no run, or a red read without a resolvable required set is only suspected), and `convergence` `{n, passes {"1","2","3","4+","never"}, settled, in_flight, unknown, median_passes, from_github, from_bundles, by_week[]}` over agent PRs by CI passes to first green, from the durable `/data/hive-metrics/ci-recovery.json` GitHub check-run trail rollup), `work_balance` (per ISO week: `n`, `opened` by actor kind `{human, agent, bot}`, `closed`, `prs_opened`, `prs_merged`, `prs_closed`, `queue_issues`, `queue_prs`, `issue_source` `github|audit`, `throttled`) with `work_balance_sources` (`issues_github_since`, `issues_fetched_at`, `issues_known`, `audit_only_weeks`, `issue_series_weeks`, `queue_samples`, `queue_retention_days`; issues from the durable GitHub rollup `/data/hive-metrics/issue-activity.json` for weeks at or after `issues_github_since` and, for earlier weeks, from the backfilled issue series `///issues/.json` written by the issue backfill ([#12527](https://github.com/hivecommons/hive/pull/12527)) when its coverage spans the requested portion of the week (`issue_series_weeks` counts the repo-weeks it supplied), else the audit log (`audit_only_weeks`), PRs from folded bundles, queue depth from governor snapshots; `missing.issues_human_opens_unobserved` is true while no GitHub rollup exists unless the issue series covers every row, and whenever any row fell back to the audit log, `missing.throttle_windows_unavailable` discloses that no governor throttle windows are reported), `intent_counters` (per intent tier `{attempted, blocked}` from `/data/hive-metrics/intent-counters.json` — evaluations and the subset the intent gate would have blocked with enforcement on, [#12362](https://github.com/hivecommons/hive/issues/12362); omitted when absent or unreadable), `complexity_by_repo` (per managed repo the newest 12 monthly complexity snapshots, oldest first, each `{month, ref, captured_at, files, functions, mean_cyclomatic, median_cyclomatic, p90_cyclomatic, max_cyclomatic, functions_over_threshold, threshold, static_warnings, static_warnings_by_tool}`; omitted when no repo has a snapshot), the `/api/reviewer/accuracy` summary, and `n` plus `missing` counts (unknown-model verdicts, empty CI, unreadable records). Never consulted by merge decisions. | `pkg/dashboard/api.go:533` | +| `GET` | `/api/agents` | Dashboard auth/session | Agents List | `pkg/dashboard/api.go:320` | +| `POST` | `/api/agents` | Owner only | Agent Create | `pkg/dashboard/api.go:321` | +| `POST` | `/api/agents/import` | Owner only | Agent Import | `pkg/dashboard/api.go:322` | +| `DELETE` | `/api/agents/{name}` | Owner only | Agent Delete | `pkg/dashboard/api.go:323` | + +| `GET` | `/api/agents/{name}/log` | Dashboard auth/session | Agent Full Log | `pkg/dashboard/api.go:126` | +| `GET` | `/api/agents/{name}/terminal-urls` | Dashboard auth/session | Agent Terminal URLs | `pkg/dashboard/api.go:130` | +| `GET` | `/api/agents/{name}/kicks` | Dashboard auth/session | Agent Kick Log List | `pkg/dashboard/api.go:133` | +| `GET` | `/api/agents/{name}/kicks/{id}` | Dashboard auth/session | Agent Kick Log Get | `pkg/dashboard/api.go:134` | +| `GET` | `/agents/{name}/kicks` | Dashboard auth/session | Agent Kick History Page (HTML) | `pkg/dashboard/api.go:135` | +| `POST` | `/api/agents/{name}/login-code` | Owner only | Agent Login Code | `pkg/dashboard/api.go:157` | ### Kick is asynchronous diff --git a/src/docs/api-reference/auth.md b/src/docs/api-reference/auth.md index e70dfa1d62..e4114ce752 100644 --- a/src/docs/api-reference/auth.md +++ b/src/docs/api-reference/auth.md @@ -5,13 +5,13 @@ Part of the [Dashboard REST API reference](../api-reference.md); see that page f | Method | Path | Auth | Purpose | Source | |---|---|---|---|---| -| `GET` | `/api/gh-auth` | Dashboard auth/session | GitHub Auth | `pkg/dashboard/api.go:179` | -| `GET` | `/api/gh-rate-limits` | Dashboard auth/session | GitHub Rate Limits | `pkg/dashboard/api.go:180` | -| `GET` | `/api/gh-user-auth/status` | Public | GitHub User Auth Status | `pkg/dashboard/api.go:181` | -| `POST` | `/api/gh-user-auth/start` | Public | GitHub User Auth Start | `pkg/dashboard/api.go:182` | -| `POST` | `/api/gh-user-auth/poll` | Public | GitHub User Auth Poll | `pkg/dashboard/api.go:183` | -| `POST` | `/api/gh-user-auth/logout` | Public | GitHub User Auth Logout | `pkg/dashboard/api.go:184` | -| `GET` | `/api/gh-user-auth/session` | Public | GitHub User Auth Session | `pkg/dashboard/api.go:185` | +| `GET` | `/api/gh-auth` | Dashboard auth/session | GitHub Auth | `pkg/dashboard/api.go:180` | +| `GET` | `/api/gh-rate-limits` | Dashboard auth/session | GitHub Rate Limits | `pkg/dashboard/api.go:181` | +| `GET` | `/api/gh-user-auth/status` | Public | GitHub User Auth Status | `pkg/dashboard/api.go:182` | +| `POST` | `/api/gh-user-auth/start` | Public | GitHub User Auth Start | `pkg/dashboard/api.go:183` | +| `POST` | `/api/gh-user-auth/poll` | Public | GitHub User Auth Poll | `pkg/dashboard/api.go:184` | +| `POST` | `/api/gh-user-auth/logout` | Public | GitHub User Auth Logout | `pkg/dashboard/api.go:185` | +| `GET` | `/api/gh-user-auth/session` | Public | GitHub User Auth Session | `pkg/dashboard/api.go:186` | | `GET` | `/api/claude-auth/status` | Dashboard auth/session | Claude Auth Status | `pkg/dashboard/claude_auth.go:54` | | `POST` | `/api/claude-auth/start` | Dashboard auth/session | Claude Auth Start | `pkg/dashboard/claude_auth.go:55` | | `POST` | `/api/claude-auth/exchange` | Dashboard auth/session | Claude Auth Exchange | `pkg/dashboard/claude_auth.go:56` | diff --git a/src/docs/api-reference/beads.md b/src/docs/api-reference/beads.md index 5b1fa09a82..b67e37d79d 100644 --- a/src/docs/api-reference/beads.md +++ b/src/docs/api-reference/beads.md @@ -5,8 +5,8 @@ Part of the [Dashboard REST API reference](../api-reference.md); see that page f | Method | Path | Auth | Purpose | Source | |---|---|---|---|---| -| `GET` | `/api/beads` | Dashboard auth/session | Beads List | `pkg/dashboard/api.go:498` | -| `GET` | `/api/beads/{agent}` | Dashboard auth/session | Beads List | `pkg/dashboard/api.go:499` | -| `POST` | `/api/beads/{agent}` | Owner only | Beads Create | `pkg/dashboard/api.go:500` | -| `POST` | `/api/beads/reset` | Owner only | Beads Reset | `pkg/dashboard/api.go:501` | -| `POST` | `/api/beads/reset/{agent}` | Owner only | Beads Reset Agent | `pkg/dashboard/api.go:502` | +| `GET` | `/api/beads` | Dashboard auth/session | Beads List | `pkg/dashboard/api.go:499` | +| `GET` | `/api/beads/{agent}` | Dashboard auth/session | Beads List | `pkg/dashboard/api.go:500` | +| `POST` | `/api/beads/{agent}` | Owner only | Beads Create | `pkg/dashboard/api.go:501` | +| `POST` | `/api/beads/reset` | Owner only | Beads Reset | `pkg/dashboard/api.go:502` | +| `POST` | `/api/beads/reset/{agent}` | Owner only | Beads Reset Agent | `pkg/dashboard/api.go:503` | diff --git a/src/docs/api-reference/configuration.md b/src/docs/api-reference/configuration.md index cfc998bdb2..fb22892d16 100644 --- a/src/docs/api-reference/configuration.md +++ b/src/docs/api-reference/configuration.md @@ -5,115 +5,115 @@ Part of the [Dashboard REST API reference](../api-reference.md); see that page f | Method | Path | Auth | Purpose | Source | |---|---|---|---|---| -| `GET` | `/api/themes` | Dashboard auth/session | Embedded dashboard/contributor theme catalog with swatches for Settings → Appearance; `?scope=contributor` filters profile themes | `pkg/dashboard/api.go:47` | -| `GET` | `/api/theme.css` | Dashboard auth/session | Effective dashboard theme stylesheet with strong ETag / 304 support; `?theme=` previews an embedded preset; `?scope=contributor` emits contributor token aliases and layers hive custom CSS | `pkg/dashboard/api.go:48` | -| `GET` | `/api/config` | Dashboard auth/session | Config | `pkg/dashboard/api.go:49` | +| `GET` | `/api/themes` | Dashboard auth/session | Embedded dashboard/contributor theme catalog with swatches for Settings → Appearance; `?scope=contributor` filters profile themes | `pkg/dashboard/api.go:48` | +| `GET` | `/api/theme.css` | Dashboard auth/session | Effective dashboard theme stylesheet with strong ETag / 304 support; `?theme=` previews an embedded preset; `?scope=contributor` emits contributor token aliases and layers hive custom CSS | `pkg/dashboard/api.go:49` | +| `GET` | `/api/config` | Dashboard auth/session | Config | `pkg/dashboard/api.go:50` | | `GET` | `/api/config/triggers` | Verified owner only | Ordered Event triggers, opaque `version`/ETag, existing agents and managed repository values/labels; applies on next evaluation without restart | `pkg/dashboard/api_event_triggers.go` | | `PUT` | `/api/config/triggers` | Verified owner only | Compile-to-bool and durably compare-and-save `{rules:[{name,agent,expr,priority}],preset?:{index,kind,repo}}`; exact `If-Match` required (428 missing, 412 stale), validation 400, persistence failure 500 leaves live rules unchanged. No enabled field or source/webhook installation | `pkg/dashboard/api_event_triggers.go` | -| `GET` | `/api/config/dashboard/theme` | Owner only | Dashboard theme selection, overrides, effective theme, and built-in catalog | `pkg/dashboard/api.go:58` | -| `PUT` | `/api/config/dashboard/theme` | Owner only | Persist `dashboard.theme` and `dashboard.theme_overrides` with validation, CSS/background guardrails, and audit logging | `pkg/dashboard/api.go:59` | -| `GET` | `/api/config/download` | Owner only | Config Download | `pkg/dashboard/api.go:50` | -| `GET` | `/api/config/sla` | Owner only | Delivery SLA targets: `{targets:{start_to_pr_h,pr_to_merge_h,open_to_merge_h}, effective_targets:{...}, defaults:{...}}`, hours, from `sla.targets` in `hive.yaml` with defaults 4/24/72 | `pkg/dashboard/api.go:261` | -| `PUT` | `/api/config/sla` | Owner only | Replace `sla.targets` (`{targets:{start_to_pr_h,pr_to_merge_h,open_to_merge_h}}`; each 0 (default) to 8760, validated before anything is persisted); the Throughput Delivery SLA panel recolours on the next status poll | `pkg/dashboard/api.go:262` | +| `GET` | `/api/config/dashboard/theme` | Owner only | Dashboard theme selection, overrides, effective theme, and built-in catalog | `pkg/dashboard/api.go:59` | +| `PUT` | `/api/config/dashboard/theme` | Owner only | Persist `dashboard.theme` and `dashboard.theme_overrides` with validation, CSS/background guardrails, and audit logging | `pkg/dashboard/api.go:60` | +| `GET` | `/api/config/download` | Owner only | Config Download | `pkg/dashboard/api.go:51` | +| `GET` | `/api/config/sla` | Owner only | Delivery SLA targets: `{targets:{start_to_pr_h,pr_to_merge_h,open_to_merge_h}, effective_targets:{...}, defaults:{...}}`, hours, from `sla.targets` in `hive.yaml` with defaults 4/24/72 | `pkg/dashboard/api.go:262` | +| `PUT` | `/api/config/sla` | Owner only | Replace `sla.targets` (`{targets:{start_to_pr_h,pr_to_merge_h,open_to_merge_h}}`; each 0 (default) to 8760, validated before anything is persisted); the Throughput Delivery SLA panel recolours on the next status poll | `pkg/dashboard/api.go:263` | | `GET` | `/api/config/export` | Owner only | Redacted JSON export of the effective config, layer files, side files, and provenance | `pkg/dashboard/api_config_export.go` | -| `GET` | `/api/config/provenance` | Owner only | Config Provenance | `pkg/dashboard/api.go:53` | -| `GET` | `/api/config/variables` | Dashboard auth/session | Variables List | `pkg/dashboard/api.go:54` | -| `GET` | `/api/config/authorized-users` | Dashboard auth/session | Authorized Users List | `pkg/dashboard/api.go:57` | -| `PUT` | `/api/config/variables/{name}` | Owner only | Variable Upsert | `pkg/dashboard/api.go:60` | -| `DELETE` | `/api/config/variables/{name}` | Owner only | Variable Delete | `pkg/dashboard/api.go:61` | +| `GET` | `/api/config/provenance` | Owner only | Config Provenance | `pkg/dashboard/api.go:54` | +| `GET` | `/api/config/variables` | Dashboard auth/session | Variables List | `pkg/dashboard/api.go:55` | +| `GET` | `/api/config/authorized-users` | Dashboard auth/session | Authorized Users List | `pkg/dashboard/api.go:58` | +| `PUT` | `/api/config/variables/{name}` | Owner only | Variable Upsert | `pkg/dashboard/api.go:61` | +| `DELETE` | `/api/config/variables/{name}` | Owner only | Variable Delete | `pkg/dashboard/api.go:62` | -| `GET` | `/api/config/agent/{name}` | Dashboard auth/session | Agent Config Get | `pkg/dashboard/api.go:194` | -| `PUT` | `/api/config/agent/{name}/general` | Owner only | Agent Config General | `pkg/dashboard/api.go:195` | -| `PUT` | `/api/config/agent/{name}/cadences` | Owner only | Agent Config Cadences | `pkg/dashboard/api.go:196` | -| `PUT` | `/api/config/agent/{name}/models` | Owner only | Agent Config Models | `pkg/dashboard/api.go:197` | -| `PUT` | `/api/config/agent/{name}/pipeline` | Owner only | Agent Config Pipeline | `pkg/dashboard/api.go:198` | -| `PUT` | `/api/config/agent/{name}/hooks` | Owner only | Agent Config Hooks | `pkg/dashboard/api.go:199` | -| `PUT` | `/api/config/agent/{name}/restrictions` | Owner only | Agent Config Restrictions | `pkg/dashboard/api.go:200` | -| `PUT` | `/api/config/agent/{name}/stats` | Owner only | Agent Config Stats | `pkg/dashboard/api.go:201` | -| `GET` | `/api/config/agent/{name}/prompt` | Dashboard auth/session | Agent Prompt | `pkg/dashboard/api.go:202` | -| `PUT` | `/api/config/agent/{name}/prompt` | Owner only | Agent Prompt Save | `pkg/dashboard/api.go:203` | -| `GET` | `/api/config/agent/{name}/export` | Dashboard auth/session | Agent Export | `pkg/dashboard/api.go:204` | -| `PUT` | `/api/config/agent/{name}/channels` | Owner only | Agent Config Channels | `pkg/dashboard/api.go:205` | -| `PUT` | `/api/config/agent/{name}/tools` | Owner only | Agent Config Tools | `pkg/dashboard/api.go:206` | -| `PUT` | `/api/config/agent/{name}/connections` | Owner only | Agent Config Connections | `pkg/dashboard/api.go:207` | +| `GET` | `/api/config/agent/{name}` | Dashboard auth/session | Agent Config Get | `pkg/dashboard/api.go:195` | +| `PUT` | `/api/config/agent/{name}/general` | Owner only | Agent Config General | `pkg/dashboard/api.go:196` | +| `PUT` | `/api/config/agent/{name}/cadences` | Owner only | Agent Config Cadences | `pkg/dashboard/api.go:197` | +| `PUT` | `/api/config/agent/{name}/models` | Owner only | Agent Config Models | `pkg/dashboard/api.go:198` | +| `PUT` | `/api/config/agent/{name}/pipeline` | Owner only | Agent Config Pipeline | `pkg/dashboard/api.go:199` | +| `PUT` | `/api/config/agent/{name}/hooks` | Owner only | Agent Config Hooks | `pkg/dashboard/api.go:200` | +| `PUT` | `/api/config/agent/{name}/restrictions` | Owner only | Agent Config Restrictions | `pkg/dashboard/api.go:201` | +| `PUT` | `/api/config/agent/{name}/stats` | Owner only | Agent Config Stats | `pkg/dashboard/api.go:202` | +| `GET` | `/api/config/agent/{name}/prompt` | Dashboard auth/session | Agent Prompt | `pkg/dashboard/api.go:203` | +| `PUT` | `/api/config/agent/{name}/prompt` | Owner only | Agent Prompt Save | `pkg/dashboard/api.go:204` | +| `GET` | `/api/config/agent/{name}/export` | Dashboard auth/session | Agent Export | `pkg/dashboard/api.go:205` | +| `PUT` | `/api/config/agent/{name}/channels` | Owner only | Agent Config Channels | `pkg/dashboard/api.go:206` | +| `PUT` | `/api/config/agent/{name}/tools` | Owner only | Agent Config Tools | `pkg/dashboard/api.go:207` | +| `PUT` | `/api/config/agent/{name}/connections` | Owner only | Agent Config Connections | `pkg/dashboard/api.go:208` | | `GET` | `/api/config/agent/{name}/variables` | Dashboard auth/session | Per-agent Variables tab: the agent's own `${VAR}` definitions plus the hive-level ones it inherits (each flagged `overridden` when the agent redefines it) and the seed-only exec/http gate flags. Never returns values — only name, type, scope and a non-secret source hint | `pkg/dashboard/api_variables.go:298` | | `PUT` | `/api/config/agent/{name}/variables/{var}` | Owner only | Create or update one per-agent variable (`static`/`env` only; `script`/`http` are 403 and seed-only; scope `template` or `both`; secret-looking static values rejected). Persisted on the agent entry; overrides a same-named hive-level variable in this agent's kick prompt | `pkg/dashboard/api_variables.go:299` | | `DELETE` | `/api/config/agent/{name}/variables/{var}` | Owner only | Delete one per-agent variable (404 for inherited hive-level variables — delete those via `/api/config/variables/{name}`) | `pkg/dashboard/api_variables.go:300` | -| `GET` | `/api/config/stat-sources` | Dashboard auth/session | Stat Sources | `pkg/dashboard/api.go:208` | -| `GET` | `/api/config/governor` | Dashboard auth/session | Governor Config Get | `pkg/dashboard/api.go:210` | +| `GET` | `/api/config/stat-sources` | Dashboard auth/session | Stat Sources | `pkg/dashboard/api.go:209` | +| `GET` | `/api/config/governor` | Dashboard auth/session | Governor Config Get | `pkg/dashboard/api.go:211` | | `PATCH` | `/api/config/governor` | Owner only | Update `merge_train_stall_warn_s`; omitted values stay unchanged, 0 disables warnings; returns the effective interval | `pkg/dashboard/api_governor_merge_train.go` | -| `PUT` | `/api/config/governor/sensing` | Owner only | Governor Sensing | `pkg/dashboard/api.go:214` | -| `PUT` | `/api/config/governor/thresholds` | Owner only | Governor Thresholds | `pkg/dashboard/api.go:215` | -| `PUT` | `/api/config/governor/labels` | Owner only | Governor Labels | `pkg/dashboard/api.go:222` | -| `PUT` | `/api/config/governor/budget` | Owner only | Governor Budget | `pkg/dashboard/api.go:223` | -| `PUT` | `/api/config/governor/notifications` | Owner only | Governor Notifications | `pkg/dashboard/api.go:227` | -| `PUT` | `/api/config/governor/health` | Owner only | Governor Health | `pkg/dashboard/api.go:228` | -| `PUT` | `/api/config/governor/logging` | Owner only | Governor Logging | `pkg/dashboard/api.go:240` | -| `PUT` | `/api/config/governor/attribution` | Owner only | Governor Attribution | `pkg/dashboard/api.go:241` | -| `PUT` | `/api/config/governor/hub` | Owner only | Governor Hub | `pkg/dashboard/api.go:242` | -| `PUT` | `/api/config/governor/litellm` | Owner only | Governor Lite LLM | `pkg/dashboard/api.go:243` | -| `PUT` | `/api/config/governor/trajectory` | Owner only | Governor Trajectory | `pkg/dashboard/api.go:244` | +| `PUT` | `/api/config/governor/sensing` | Owner only | Governor Sensing | `pkg/dashboard/api.go:215` | +| `PUT` | `/api/config/governor/thresholds` | Owner only | Governor Thresholds | `pkg/dashboard/api.go:216` | +| `PUT` | `/api/config/governor/labels` | Owner only | Governor Labels | `pkg/dashboard/api.go:223` | +| `PUT` | `/api/config/governor/budget` | Owner only | Governor Budget | `pkg/dashboard/api.go:224` | +| `PUT` | `/api/config/governor/notifications` | Owner only | Governor Notifications | `pkg/dashboard/api.go:228` | +| `PUT` | `/api/config/governor/health` | Owner only | Governor Health | `pkg/dashboard/api.go:229` | +| `PUT` | `/api/config/governor/logging` | Owner only | Governor Logging | `pkg/dashboard/api.go:241` | +| `PUT` | `/api/config/governor/attribution` | Owner only | Governor Attribution | `pkg/dashboard/api.go:242` | +| `PUT` | `/api/config/governor/hub` | Owner only | Governor Hub | `pkg/dashboard/api.go:243` | +| `PUT` | `/api/config/governor/litellm` | Owner only | Governor Lite LLM | `pkg/dashboard/api.go:244` | +| `PUT` | `/api/config/governor/trajectory` | Owner only | Governor Trajectory | `pkg/dashboard/api.go:245` | | `GET` | `/api/config/governor/backup` | Owner only | Backup Key Status (presence + safe source label; never the key value) | `pkg/dashboard/backup_key.go` | | `PUT` | `/api/config/governor/backup` | Owner only | Backup Key Set (64-hex AES-256 key; stored 0600, path-only in `hive.yaml`) | `pkg/dashboard/backup_key.go` | | `DELETE` | `/api/config/governor/backup` | Owner only | Backup Key Clear (backups are refused again) | `pkg/dashboard/backup_key.go` | -| `GET` | `/api/config/governor/bob` | Dashboard auth/session | Governor Bob Status | `pkg/dashboard/api.go:290` | -| `PUT` | `/api/config/governor/bob` | Owner only | Governor Bob Key | `pkg/dashboard/api.go:291` | -| `DELETE` | `/api/config/governor/bob` | Owner only | Governor Bob Key Clear | `pkg/dashboard/api.go:292` | +| `GET` | `/api/config/governor/bob` | Dashboard auth/session | Governor Bob Status | `pkg/dashboard/api.go:291` | +| `PUT` | `/api/config/governor/bob` | Owner only | Governor Bob Key | `pkg/dashboard/api.go:292` | +| `DELETE` | `/api/config/governor/bob` | Owner only | Governor Bob Key Clear | `pkg/dashboard/api.go:293` | -| `GET` | `/api/config/governor/cadence-scope` | Owner only | Governor Cadence Scope Get (`aggregate` or `per_repo`) | `pkg/dashboard/api.go:216` | -| `PUT` | `/api/config/governor/cadence-scope` | Owner only | Governor Cadence Scope Set (`cadenceScope`/`cadence_scope`: `aggregate`, `per_repo`, or empty for default) | `pkg/dashboard/api.go:217` | -| `POST` | `/api/config/governor/bob/test` | Dashboard auth/session | Governor Bob Key Test | `pkg/dashboard/api.go:294` | -| `POST` | `/api/config/governor/litellm/test` | Dashboard auth/session | Governor Lite LLMTest | `pkg/dashboard/api.go:295` | -| `GET` | `/api/config/governor/gateways` | Dashboard auth/session | Governor Gateways List | `pkg/dashboard/api.go:298` | -| `PUT` | `/api/config/governor/gateways` | Owner only | Governor Gateways Upsert | `pkg/dashboard/api.go:299` | -| `DELETE` | `/api/config/governor/gateways/{name}` | Owner only | Governor Gateways Delete | `pkg/dashboard/api.go:300` | -| `POST` | `/api/config/governor/gateways/{name}/test` | Dashboard auth/session | Governor Gateways Test | `pkg/dashboard/api.go:301` | -| `POST` | `/api/config/governor/gateways/discover` | Owner only | Governor Gateways Discover | `pkg/dashboard/api.go:302` | -| `POST` | `/api/config/governor/agents` | Owner only | Governor Add Agent | `pkg/dashboard/api.go:307` | -| `DELETE` | `/api/config/governor/agents/{name}` | Owner only | Governor Remove Agent | `pkg/dashboard/api.go:308` | -| `PUT` | `/api/config/governor/repos` | Owner only | Governor Repos | `pkg/dashboard/api.go:309` | -| `POST` | `/api/config/governor/repos/check-access` | Dashboard auth/session | Governor Repo Check Access | `pkg/dashboard/api.go:313` | -| `PUT` | `/api/config/github` | Owner only | Config GitHub | `pkg/dashboard/api.go:314` | -| `GET` | `/api/config/github/forge-apps` | Dashboard auth/session | Config GitHub Forge Apps | `pkg/dashboard/api.go:317` | -| `GET` | `/api/config/sidebar` | Dashboard auth/session | Sidebar Get | `pkg/dashboard/api.go:372` | -| `PUT` | `/api/config/sidebar` | Dashboard auth/session | Sidebar Set | `pkg/dashboard/api.go:373` | -| `GET` | `/api/config/backends` | Dashboard auth/session | Backends | `pkg/dashboard/api.go:374` | -| `GET` | `/api/compliance/status` | Merger or owner | Evaluated compliance controls for the frameworks in `compliance.frameworks` (`compliance.Report`, JSON): per control `meets`/`deviates`/`off`/`not_covered` with current vs recommended value per mapped setting, a summary, and the non-certification disclaimer — see [Compliance](../compliance.md) ([#11078](https://github.com/hivecommons/hive/issues/11078)) | `pkg/dashboard/api.go:364` | -| `GET` | `/api/compliance/posture` | Merger or owner | Posture-check catalogue, effective interval / window / history retention, and the latest run (`compliance.PostureRun`: schedule/manual trigger, `triggered_by` identity, and per-check `pass`/`fail`/`skip`/`error`, detail, control ids, evidence refs); `?since=` (RFC 3339, `YYYY-MM-DD` or a look-back such as `168h`) adds every run since then, oldest first, capped at 1000 — see [Compliance](../compliance.md#posture-checks-catalogue) ([#11079](https://github.com/hivecommons/hive/issues/11079)) | `pkg/dashboard/api.go:365` | -| `POST` | `/api/compliance/posture/run` | Merger or owner | Run the posture checks now, attributed to the authenticated user, and record the pass in the history; audited as `compliance_posture_run`; `409` when no framework is selected or a pass is already running ([#11079](https://github.com/hivecommons/hive/issues/11079)) | `pkg/dashboard/api.go:366` | -| `GET` | `/api/compliance/posture/history` | Owner only | Posture history for `?since=`/`?until=` (default the last 30 days) as one series per check: points (`at`, `status`), status counts, `last_fail_at` and the latest result with detail and evidence refs; at most the 1000 most recent runs — see [Compliance](../compliance.md#evidence-exports-and-attestations) ([#11081](https://github.com/hivecommons/hive/issues/11081)) | `pkg/dashboard/api.go:367` | -| `GET` | `/api/compliance/export` | Owner only | Evidence download for `?since=`/`?until=`: `kind=controls` (`json`/`md`), `posture` (`json`/`csv`), `audit` (`json`), `config` (redacted effective config + sha256, `json`), `attestations` (`json`/`csv`) or `bundle` (`json`). Every format includes source, period, generator, selected profile versions, attesters, retention and a per-section SHA-256 manifest; audited as `compliance_export` with the SHA-256 of the delivered bytes ([#11081](https://github.com/hivecommons/hive/issues/11081), [#13582](https://github.com/hivecommons/hive/issues/13582)) | `pkg/dashboard/api.go:368` | -| `GET` | `/api/compliance/attestations` | Owner only | Owner attestations, newest first (`?framework=` filters) ([#11081](https://github.com/hivecommons/hive/issues/11081)) | `pkg/dashboard/api.go:369` | -| `POST` | `/api/compliance/attestations` | Owner only | Record `{"framework", "reviewed_on", "note"}` for the signed-in owner; audited as `compliance_attestation`; `201` with the attestation, `400` for an unknown framework, a bad or future date, or a note over 2000 characters ([#11081](https://github.com/hivecommons/hive/issues/11081)) | `pkg/dashboard/api.go:370` | -| `PUT` | `/api/config/governor/compliance` | Owner only | Settings → Compliance framework picker: `{"frameworks": [...]}` sets `compliance.frameworks` (empty list deselects all, absent key leaves it unchanged; unknown/duplicate ids are a `400` before any change); audited as `config_compliance`; returns the re-evaluated `compliance.Report` ([#11080](https://github.com/hivecommons/hive/issues/11080)) | `pkg/dashboard/api.go:539` | +| `GET` | `/api/config/governor/cadence-scope` | Owner only | Governor Cadence Scope Get (`aggregate` or `per_repo`) | `pkg/dashboard/api.go:217` | +| `PUT` | `/api/config/governor/cadence-scope` | Owner only | Governor Cadence Scope Set (`cadenceScope`/`cadence_scope`: `aggregate`, `per_repo`, or empty for default) | `pkg/dashboard/api.go:218` | +| `POST` | `/api/config/governor/bob/test` | Dashboard auth/session | Governor Bob Key Test | `pkg/dashboard/api.go:295` | +| `POST` | `/api/config/governor/litellm/test` | Dashboard auth/session | Governor Lite LLMTest | `pkg/dashboard/api.go:296` | +| `GET` | `/api/config/governor/gateways` | Dashboard auth/session | Governor Gateways List | `pkg/dashboard/api.go:299` | +| `PUT` | `/api/config/governor/gateways` | Owner only | Governor Gateways Upsert | `pkg/dashboard/api.go:300` | +| `DELETE` | `/api/config/governor/gateways/{name}` | Owner only | Governor Gateways Delete | `pkg/dashboard/api.go:301` | +| `POST` | `/api/config/governor/gateways/{name}/test` | Dashboard auth/session | Governor Gateways Test | `pkg/dashboard/api.go:302` | +| `POST` | `/api/config/governor/gateways/discover` | Owner only | Governor Gateways Discover | `pkg/dashboard/api.go:303` | +| `POST` | `/api/config/governor/agents` | Owner only | Governor Add Agent | `pkg/dashboard/api.go:308` | +| `DELETE` | `/api/config/governor/agents/{name}` | Owner only | Governor Remove Agent | `pkg/dashboard/api.go:309` | +| `PUT` | `/api/config/governor/repos` | Owner only | Governor Repos | `pkg/dashboard/api.go:310` | +| `POST` | `/api/config/governor/repos/check-access` | Dashboard auth/session | Governor Repo Check Access | `pkg/dashboard/api.go:314` | +| `PUT` | `/api/config/github` | Owner only | Config GitHub | `pkg/dashboard/api.go:315` | +| `GET` | `/api/config/github/forge-apps` | Dashboard auth/session | Config GitHub Forge Apps | `pkg/dashboard/api.go:318` | +| `GET` | `/api/config/sidebar` | Dashboard auth/session | Sidebar Get | `pkg/dashboard/api.go:373` | +| `PUT` | `/api/config/sidebar` | Dashboard auth/session | Sidebar Set | `pkg/dashboard/api.go:374` | +| `GET` | `/api/config/backends` | Dashboard auth/session | Backends | `pkg/dashboard/api.go:375` | +| `GET` | `/api/compliance/status` | Merger or owner | Evaluated compliance controls for the frameworks in `compliance.frameworks` (`compliance.Report`, JSON): per control `meets`/`deviates`/`off`/`not_covered` with current vs recommended value per mapped setting, a summary, and the non-certification disclaimer — see [Compliance](../compliance.md) ([#11078](https://github.com/hivecommons/hive/issues/11078)) | `pkg/dashboard/api.go:365` | +| `GET` | `/api/compliance/posture` | Merger or owner | Posture-check catalogue, effective interval / window / history retention, and the latest run (`compliance.PostureRun`: schedule/manual trigger, `triggered_by` identity, and per-check `pass`/`fail`/`skip`/`error`, detail, control ids, evidence refs); `?since=` (RFC 3339, `YYYY-MM-DD` or a look-back such as `168h`) adds every run since then, oldest first, capped at 1000 — see [Compliance](../compliance.md#posture-checks-catalogue) ([#11079](https://github.com/hivecommons/hive/issues/11079)) | `pkg/dashboard/api.go:366` | +| `POST` | `/api/compliance/posture/run` | Merger or owner | Run the posture checks now, attributed to the authenticated user, and record the pass in the history; audited as `compliance_posture_run`; `409` when no framework is selected or a pass is already running ([#11079](https://github.com/hivecommons/hive/issues/11079)) | `pkg/dashboard/api.go:367` | +| `GET` | `/api/compliance/posture/history` | Owner only | Posture history for `?since=`/`?until=` (default the last 30 days) as one series per check: points (`at`, `status`), status counts, `last_fail_at` and the latest result with detail and evidence refs; at most the 1000 most recent runs — see [Compliance](../compliance.md#evidence-exports-and-attestations) ([#11081](https://github.com/hivecommons/hive/issues/11081)) | `pkg/dashboard/api.go:368` | +| `GET` | `/api/compliance/export` | Owner only | Evidence download for `?since=`/`?until=`: `kind=controls` (`json`/`md`), `posture` (`json`/`csv`), `audit` (`json`), `config` (redacted effective config + sha256, `json`), `attestations` (`json`/`csv`) or `bundle` (`json`). Every format includes source, period, generator, selected profile versions, attesters, retention and a per-section SHA-256 manifest; audited as `compliance_export` with the SHA-256 of the delivered bytes ([#11081](https://github.com/hivecommons/hive/issues/11081), [#13582](https://github.com/hivecommons/hive/issues/13582)) | `pkg/dashboard/api.go:369` | +| `GET` | `/api/compliance/attestations` | Owner only | Owner attestations, newest first (`?framework=` filters) ([#11081](https://github.com/hivecommons/hive/issues/11081)) | `pkg/dashboard/api.go:370` | +| `POST` | `/api/compliance/attestations` | Owner only | Record `{"framework", "reviewed_on", "note"}` for the signed-in owner; audited as `compliance_attestation`; `201` with the attestation, `400` for an unknown framework, a bad or future date, or a note over 2000 characters ([#11081](https://github.com/hivecommons/hive/issues/11081)) | `pkg/dashboard/api.go:371` | +| `PUT` | `/api/config/governor/compliance` | Owner only | Settings → Compliance framework picker: `{"frameworks": [...]}` sets `compliance.frameworks` (empty list deselects all, absent key leaves it unchanged; unknown/duplicate ids are a `400` before any change); audited as `config_compliance`; returns the re-evaluated `compliance.Report` ([#11080](https://github.com/hivecommons/hive/issues/11080)) | `pkg/dashboard/api.go:540` | -| `GET` | `/api/config/governor/threshold-scaling` | Owner only | Governor Threshold Scaling Get | `pkg/dashboard/api.go:219` | -| `PUT` | `/api/config/governor/threshold-scaling` | Owner only | Governor Threshold Scaling Set | `pkg/dashboard/api.go:220` | -| `PUT` | `/api/config/governor/coverage-target` | Owner only | Governor Coverage Target | `pkg/dashboard/api.go:221` | -| `POST` | `/api/config/governor/budget/reset` | Owner only | Governor Budget Reset | `pkg/dashboard/api.go:224` | -| `PUT` | `/api/config/governor/watchdog` | Owner only | Governor Watchdog | `pkg/dashboard/api.go:229` | -| `GET` | `/api/config/escalation` | Owner only | Escalation Config Get | `pkg/dashboard/api.go:232` | -| `PUT` | `/api/config/escalation` | Owner only | Escalation Config Set | `pkg/dashboard/api.go:233` | -| `GET` | `/api/config/write-surface` | Owner only | Lane write allowlist (`write_surface.allowlist`, [#9587](https://github.com/hivecommons/hive/issues/9587)): `allowlist`, `ops`, `warnings` | `pkg/dashboard/api.go:535` | -| `PUT` | `/api/config/write-surface` | Owner only | Replace the lane write allowlist; `{"allowlist": {}}` clears it; unknown operation or bad lane name is a 400 with nothing changed | `pkg/dashboard/api.go:536` | -| `GET` | `/api/config/review` | Dashboard auth/session | Review Config Get | `pkg/dashboard/api.go:236` | -| `PUT` | `/api/config/review` | Owner only | Review Config Set | `pkg/dashboard/api.go:239` | -| `PUT` | `/api/config/governor/features` | Owner only | Governor Features | `pkg/dashboard/api.go:245` | -| `GET` | `/api/config/governor/general-advanced` | Owner only | Governor General Advanced Get | `pkg/dashboard/api.go:246` | -| `PUT` | `/api/config/governor/general-advanced` | Owner only | Governor General Advanced Set | `pkg/dashboard/api.go:247` | -| `GET` | `/api/config/auto-merge` | Owner only | Auto-Merge Config Get | `pkg/dashboard/api.go:251` | -| `PUT` | `/api/config/auto-merge` | Owner only | Auto-Merge Config Set | `pkg/dashboard/api.go:252` | +| `GET` | `/api/config/governor/threshold-scaling` | Owner only | Governor Threshold Scaling Get | `pkg/dashboard/api.go:220` | +| `PUT` | `/api/config/governor/threshold-scaling` | Owner only | Governor Threshold Scaling Set | `pkg/dashboard/api.go:221` | +| `PUT` | `/api/config/governor/coverage-target` | Owner only | Governor Coverage Target | `pkg/dashboard/api.go:222` | +| `POST` | `/api/config/governor/budget/reset` | Owner only | Governor Budget Reset | `pkg/dashboard/api.go:225` | +| `PUT` | `/api/config/governor/watchdog` | Owner only | Governor Watchdog | `pkg/dashboard/api.go:230` | +| `GET` | `/api/config/escalation` | Owner only | Escalation Config Get | `pkg/dashboard/api.go:233` | +| `PUT` | `/api/config/escalation` | Owner only | Escalation Config Set | `pkg/dashboard/api.go:234` | +| `GET` | `/api/config/write-surface` | Owner only | Lane write allowlist (`write_surface.allowlist`, [#9587](https://github.com/hivecommons/hive/issues/9587)): `allowlist`, `ops`, `warnings` | `pkg/dashboard/api.go:536` | +| `PUT` | `/api/config/write-surface` | Owner only | Replace the lane write allowlist; `{"allowlist": {}}` clears it; unknown operation or bad lane name is a 400 with nothing changed | `pkg/dashboard/api.go:537` | +| `GET` | `/api/config/review` | Dashboard auth/session | Review Config Get | `pkg/dashboard/api.go:237` | +| `PUT` | `/api/config/review` | Owner only | Review Config Set | `pkg/dashboard/api.go:240` | +| `PUT` | `/api/config/governor/features` | Owner only | Governor Features | `pkg/dashboard/api.go:246` | +| `GET` | `/api/config/governor/general-advanced` | Owner only | Governor General Advanced Get | `pkg/dashboard/api.go:247` | +| `PUT` | `/api/config/governor/general-advanced` | Owner only | Governor General Advanced Set | `pkg/dashboard/api.go:248` | +| `GET` | `/api/config/auto-merge` | Owner only | Auto-Merge Config Get | `pkg/dashboard/api.go:252` | +| `PUT` | `/api/config/auto-merge` | Owner only | Auto-Merge Config Set | `pkg/dashboard/api.go:253` | | `PATCH` | `/api/config/auto-merge` | Owner only | Partially update auto-merge settings, including nested `merge_train`; omitted fields stay unchanged; returns effective settings and bot-author metadata | `pkg/dashboard/api_config_automerge.go` | -| `GET` | `/api/config/convergence` | Owner only | Convergence Config Get | `pkg/dashboard/api.go:258` | -| `PUT` | `/api/config/convergence` | Owner only | Convergence Config Set | `pkg/dashboard/api.go:260` | -| `GET` | `/api/config/governor/advisory` | Owner only | Governor Advisory Get | `pkg/dashboard/api.go:264` | -| `PUT` | `/api/config/governor/advisory` | Owner only | Governor Advisory Set | `pkg/dashboard/api.go:265` | -| `GET` | `/api/config/governor/replan` | Owner only | Governor Replan Get | `pkg/dashboard/api.go:266` | -| `PUT` | `/api/config/governor/replan` | Owner only | Governor Replan Set | `pkg/dashboard/api.go:267` | -| `GET` | `/api/config/governor/work-source` | Owner only | Governor Work Source Get (Jira secrets/TLS fields reported as set/unset only) | `pkg/dashboard/api.go:277` | -| `PUT` | `/api/config/governor/work-source` | Owner only | Governor Work Source Set (including Jira Data Center TLS settings) | `pkg/dashboard/api.go:278` | -| `PUT` | `/api/config/governor/security` | Owner only | Governor Security | `pkg/dashboard/api.go:279` | -| `GET` | `/api/config/governor/project-observability` | Owner only | Governor Project Observability Get | `pkg/dashboard/api.go:280` | -| `PUT` | `/api/config/governor/project-observability` | Owner only | Governor Project Observability Set | `pkg/dashboard/api.go:281` | -| `GET` | `/api/config/governor/inference-auth` | Owner only | Governor Inference Auth Get | `pkg/dashboard/api.go:296` | -| `PUT` | `/api/config/governor/inference-auth` | Owner only | Governor Inference Auth Set | `pkg/dashboard/api.go:297` | +| `GET` | `/api/config/convergence` | Owner only | Convergence Config Get | `pkg/dashboard/api.go:259` | +| `PUT` | `/api/config/convergence` | Owner only | Convergence Config Set | `pkg/dashboard/api.go:261` | +| `GET` | `/api/config/governor/advisory` | Owner only | Governor Advisory Get | `pkg/dashboard/api.go:265` | +| `PUT` | `/api/config/governor/advisory` | Owner only | Governor Advisory Set | `pkg/dashboard/api.go:266` | +| `GET` | `/api/config/governor/replan` | Owner only | Governor Replan Get | `pkg/dashboard/api.go:267` | +| `PUT` | `/api/config/governor/replan` | Owner only | Governor Replan Set | `pkg/dashboard/api.go:268` | +| `GET` | `/api/config/governor/work-source` | Owner only | Governor Work Source Get (Jira secrets/TLS fields reported as set/unset only) | `pkg/dashboard/api.go:278` | +| `PUT` | `/api/config/governor/work-source` | Owner only | Governor Work Source Set (including Jira Data Center TLS settings) | `pkg/dashboard/api.go:279` | +| `PUT` | `/api/config/governor/security` | Owner only | Governor Security | `pkg/dashboard/api.go:280` | +| `GET` | `/api/config/governor/project-observability` | Owner only | Governor Project Observability Get | `pkg/dashboard/api.go:281` | +| `PUT` | `/api/config/governor/project-observability` | Owner only | Governor Project Observability Set | `pkg/dashboard/api.go:282` | +| `GET` | `/api/config/governor/inference-auth` | Owner only | Governor Inference Auth Get | `pkg/dashboard/api.go:297` | +| `PUT` | `/api/config/governor/inference-auth` | Owner only | Governor Inference Auth Set | `pkg/dashboard/api.go:298` | diff --git a/src/docs/api-reference/cost-tokens-telemetry.md b/src/docs/api-reference/cost-tokens-telemetry.md index 10f470cace..e80975236d 100644 --- a/src/docs/api-reference/cost-tokens-telemetry.md +++ b/src/docs/api-reference/cost-tokens-telemetry.md @@ -5,16 +5,16 @@ Part of the [Dashboard REST API reference](../api-reference.md); see that page f | Method | Path | Auth | Purpose | Source | |---|---|---|---|---| -| `GET` | `/api/trends` | Dashboard auth/session | Trends | `pkg/dashboard/api.go:86` | -| `GET` | `/api/token-access` | Dashboard auth/session | Token Access | `pkg/dashboard/api.go:160` | -| `GET` | `/api/tokens` | Dashboard auth/session | Tokens | `pkg/dashboard/api.go:161` | -| `GET` | `/api/cost` | Dashboard auth/session | Cost; `merged_prs` / `closed_issues` come from the webhook-incremented ForgeState counters reconciled hourly against search, with `count_stale`, `count_stale_since` and `count_source` (`forgestate` / `search`) when last-good counts are being served (#13376), `count_partial` when only some configured repos have an attributed baseline yet, and `count_error` naming why the latest attributed refresh failed or was partial (the reconcile asks GitHub about a budgeted slice of repos per pass so a many-repo hive stays inside search's rate limit); `daily_rollup` lists per-UTC-day estimated spend (`day`, `cost_usd`, `input_tokens`, `output_tokens`, `samples`) kept for ~2 years so the Cost chart can reach past the 30-day snapshot ring; `recording` reports `agent_models`, `unrecorded_models`, `recent_models`, `last_priced_growth_at`, `last_agent_model_growth_at`, `stale_after_ms` and `stale` so the card can flag stale live usage recording (#13485) | `pkg/dashboard/api.go:162` | +| `GET` | `/api/trends` | Dashboard auth/session | Trends | `pkg/dashboard/api.go:87` | +| `GET` | `/api/token-access` | Dashboard auth/session | Token Access | `pkg/dashboard/api.go:161` | +| `GET` | `/api/tokens` | Dashboard auth/session | Tokens | `pkg/dashboard/api.go:162` | +| `GET` | `/api/cost` | Dashboard auth/session | Cost; `merged_prs` / `closed_issues` come from the webhook-incremented ForgeState counters reconciled hourly against search, with `count_stale`, `count_stale_since` and `count_source` (`forgestate` / `search`) when last-good counts are being served (#13376), `count_partial` when only some configured repos have an attributed baseline yet, and `count_error` naming why the latest attributed refresh failed or was partial (the reconcile asks GitHub about a budgeted slice of repos per pass so a many-repo hive stays inside search's rate limit); `daily_rollup` lists per-UTC-day estimated spend (`day`, `cost_usd`, `input_tokens`, `output_tokens`, `samples`) kept for ~2 years so the Cost chart can reach past the 30-day snapshot ring; `recording` reports `agent_models`, `unrecorded_models`, `recent_models`, `last_priced_growth_at`, `last_agent_model_growth_at`, `stale_after_ms` and `stale` so the card can flag stale live usage recording (#13485) | `pkg/dashboard/api.go:163` | | `GET` | `/api/repo-activity` | Dashboard auth/session | Per-repo audited activity | `pkg/dashboard/api.go` | | `GET` | `/api/repo-cost` | Dashboard auth/session | Per-repo estimated token cost (interval join, cached) | `pkg/dashboard/api.go` | | `GET` | `/api/launch-cost` | Dashboard auth/session | Per-issue / PR / head-SHA estimated token cost from kick launch attribution (cached; `?ref=owner/repo#N` narrows) | `pkg/dashboard/api.go` | | `GET` | `/api/launch-cost/history` | Dashboard auth/session | Daily per-ref rollup of launch attribution (`daily_rollup[]`: `day`, `by_ref[]` with `ref`, `kind`, `launches`, `share_usd`), kept ~2 years on the `/data` PVC so cost per PR can be measured past the `/api/launch-cost` window | `pkg/dashboard/api.go` | -| `GET` | `/api/cost/history` | Dashboard auth/session | Cost History | `pkg/dashboard/api.go:168` | -| `GET` | `/api/trend/history` | Dashboard auth/session | Trend History | `pkg/dashboard/api.go:171` | -| `GET` | `/api/timeseries` | Dashboard auth/session | Time Series | `pkg/dashboard/api.go:173` | -| `GET` | `/api/providers/headroom` | Dashboard auth/session | Providers Headroom | `pkg/dashboard/api.go:63` | -| `GET` | `/api/budget/history` | Dashboard auth/session | Budget History | `pkg/dashboard/api.go:170` | +| `GET` | `/api/cost/history` | Dashboard auth/session | Cost History | `pkg/dashboard/api.go:169` | +| `GET` | `/api/trend/history` | Dashboard auth/session | Trend History | `pkg/dashboard/api.go:172` | +| `GET` | `/api/timeseries` | Dashboard auth/session | Time Series | `pkg/dashboard/api.go:174` | +| `GET` | `/api/providers/headroom` | Dashboard auth/session | Providers Headroom | `pkg/dashboard/api.go:64` | +| `GET` | `/api/budget/history` | Dashboard auth/session | Budget History | `pkg/dashboard/api.go:171` | diff --git a/src/docs/api-reference/dashboard-misc.md b/src/docs/api-reference/dashboard-misc.md index f893a521f4..404a030c9b 100644 --- a/src/docs/api-reference/dashboard-misc.md +++ b/src/docs/api-reference/dashboard-misc.md @@ -24,49 +24,49 @@ The monitor renders CI and Review concurrently on one shared PR row; see | Method | Path | Auth | Purpose | Source | |---|---|---|---|---| -| `GET` | `/api/audit` | Read-write role | Audit Log — `{"entries": [...]}` envelope, newest first, capped at 200; response shape and the serve-time `user_name` field in [audit-log.md](../audit-log.md#get-apiaudit) | `pkg/dashboard/api.go:62` | -| `GET` | `/api/watchdog/activity` | Read-write role | Watchdog activity readout for the Health tab (#7254): `watchdog-*` audit actions over `?days=` (default 30, clamped to 90) — total / taken / observed / `byAction`, a zero-filled per-day `daily` histogram, per-agent `agents` liveness, and the Observe → Heal `promotion` hint; see [agent-watchdog.md](../agent-watchdog.md#watchdog-activity-strip) | `pkg/dashboard/api.go:509` | -| `POST` | `/api/presence` | Dashboard auth/session | Presence | `pkg/dashboard/api.go:65` | -| `POST` | `/api/upgrade/announce` | Owner role | Announce a pod roll before it happens ([#15788](https://github.com/hivecommons/hive/issues/15788)): body `{"target": "", "source": "hub"|"operator", "reason": "..."}` (`version` is an alias of `target`; all fields optional). Marks `/api/status`/`/api/version` `upgradeProgress` as `state: pulling, phase: announced` with the target and publishes an SSE `upgrading` event so every open dashboard shows the upgrade animation; a SIGTERM afterwards (or with no announce at all) publishes `phase: restarting` from the same record. The hub sends the same announce through the heartbeat response (`upgrade_announce`) for hub-managed hives; an announce with no kill expires after 15 minutes. Returns `{ok, upgradeProgress}` | `pkg/dashboard/api.go:68` | -| `GET` | `/api/prompt-history` | Dashboard auth/session | Prompt History | `pkg/dashboard/api.go:66` | -| `POST` | `/api/self-upgrade` | Owner only | Self Upgrade | `pkg/dashboard/api.go:67` | -| `GET` | `/api/backup/status` | Owner only | Backup Status — `available:false` with a reason when no encryption key is configured | `pkg/dashboard/api.go:72` | -| `POST` | `/api/backup` | Owner only | Backup Download — `412` when no encryption key is configured (never an unencrypted archive) | `pkg/dashboard/api.go:73` | -| `POST` | `/api/banner-dismissed` | Dashboard auth/session | Banner Dismissed | `pkg/dashboard/api.go:74` | -| `GET` | `/api/history` | Dashboard auth/session | History | `pkg/dashboard/api.go:85` | -| `GET` | `/api/timeline` | Dashboard auth/session | Timeline | `pkg/dashboard/api.go:87` | -| `GET` | `/api/lifecycle-timeline` | Dashboard auth/session | Issue→PR lifecycle journeys plus derived stage timeline. Each journey keeps the dense `stages` map and adds a backwards-compatible `details` object for expandable rows: chronological stages with durations, current-stage age, agent, classification attrs, issue/PR links, blocked reason, and kick/run identifiers when recorded. | `pkg/dashboard/api.go:88` | -| `GET` | `/api/pr-throughput` | Dashboard auth/session | Throughput: pull/merge requests and issues across tracked forges. Counts hive-created PRs/issues, hive/observed reviews, observed PR merges/closures, and issue closes over `?hours=N` (default 24, capped at audit retention; `hours=0` = all time from durable counters in `/data/pr-throughput-counters.json`) with optional exact case-insensitive `?repo=owner/name` and `?role=created\|reviewed\|merged` for the actor trend. The response keeps legacy `opened`/`merged`/`closed`, `merged_by_path`, `buckets`, velocity metrics, previous-window trends, top agents/repos/reasons, and `recorded_since`, and adds `by_actor` (`pr`/`issue` × `created`/`reviewed`/`merged`/`closed` × `hive`/`human`/`other`) plus `series` points (`t`, `hive`, `human`, `other`) for the selected role. | `pkg/dashboard/api.go:89` | +| `GET` | `/api/audit` | Read-write role | Audit Log — `{"entries": [...]}` envelope, newest first, capped at 200; response shape and the serve-time `user_name` field in [audit-log.md](../audit-log.md#get-apiaudit) | `pkg/dashboard/api.go:63` | +| `GET` | `/api/watchdog/activity` | Read-write role | Watchdog activity readout for the Health tab (#7254): `watchdog-*` audit actions over `?days=` (default 30, clamped to 90) — total / taken / observed / `byAction`, a zero-filled per-day `daily` histogram, per-agent `agents` liveness, and the Observe → Heal `promotion` hint; see [agent-watchdog.md](../agent-watchdog.md#watchdog-activity-strip) | `pkg/dashboard/api.go:510` | +| `POST` | `/api/presence` | Dashboard auth/session | Presence | `pkg/dashboard/api.go:66` | +| `POST` | `/api/upgrade/announce` | Owner role | Announce a pod roll before it happens ([#15788](https://github.com/hivecommons/hive/issues/15788)): body `{"target": "", "source": "hub"|"operator", "reason": "..."}` (`version` is an alias of `target`; all fields optional). Marks `/api/status`/`/api/version` `upgradeProgress` as `state: pulling, phase: announced` with the target and publishes an SSE `upgrading` event so every open dashboard shows the upgrade animation; a SIGTERM afterwards (or with no announce at all) publishes `phase: restarting` from the same record. The hub sends the same announce through the heartbeat response (`upgrade_announce`) for hub-managed hives; an announce with no kill expires after 15 minutes. Returns `{ok, upgradeProgress}` | `pkg/dashboard/api.go:69` | +| `GET` | `/api/prompt-history` | Dashboard auth/session | Prompt History | `pkg/dashboard/api.go:67` | +| `POST` | `/api/self-upgrade` | Owner only | Self Upgrade | `pkg/dashboard/api.go:68` | +| `GET` | `/api/backup/status` | Owner only | Backup Status — `available:false` with a reason when no encryption key is configured | `pkg/dashboard/api.go:73` | +| `POST` | `/api/backup` | Owner only | Backup Download — `412` when no encryption key is configured (never an unencrypted archive) | `pkg/dashboard/api.go:74` | +| `POST` | `/api/banner-dismissed` | Dashboard auth/session | Banner Dismissed | `pkg/dashboard/api.go:75` | +| `GET` | `/api/history` | Dashboard auth/session | History | `pkg/dashboard/api.go:86` | +| `GET` | `/api/timeline` | Dashboard auth/session | Timeline | `pkg/dashboard/api.go:88` | +| `GET` | `/api/lifecycle-timeline` | Dashboard auth/session | Issue→PR lifecycle journeys plus derived stage timeline. Each journey keeps the dense `stages` map and adds a backwards-compatible `details` object for expandable rows: chronological stages with durations, current-stage age, agent, classification attrs, issue/PR links, blocked reason, and kick/run identifiers when recorded. | `pkg/dashboard/api.go:89` | +| `GET` | `/api/pr-throughput` | Dashboard auth/session | Throughput: pull/merge requests and issues across tracked forges. Counts hive-created PRs/issues, hive/observed reviews, observed PR merges/closures, and issue closes over `?hours=N` (default 24, capped at audit retention; `hours=0` = all time from durable counters in `/data/pr-throughput-counters.json`) with optional exact case-insensitive `?repo=owner/name` and `?role=created\|reviewed\|merged` for the actor trend. The response keeps legacy `opened`/`merged`/`closed`, `merged_by_path`, `buckets`, velocity metrics, previous-window trends, top agents/repos/reasons, and `recorded_since`, and adds `by_actor` (`pr`/`issue` × `created`/`reviewed`/`merged`/`closed` × `hive`/`human`/`other`) plus `series` points (`t`, `hive`, `human`, `other`) for the selected role. | `pkg/dashboard/api.go:90` | -| `GET` | `/api/convergence/soak` | Owner only | Convergence Soak Status | `pkg/dashboard/api.go:263` | -| `POST` | `/api/plan/from-issue` | Dashboard auth/session | Plan From Issue | `pkg/dashboard/api.go:461` | -| `GET` | `/api/plan/{epicID}` | Dashboard auth/session | Plan Tree | `pkg/dashboard/api.go:464` | -| `POST` | `/api/plan/{epicID}/approve` | Owner only | Plan Approve | `pkg/dashboard/api.go:465` | -| `POST` | `/api/plan/{epicID}/reject` | Owner only | Plan Reject | `pkg/dashboard/api.go:467` | -| `POST` | `/api/plan/{epicID}/child/{childID}` | Owner only | Plan Child Action | `pkg/dashboard/api.go:468` | +| `GET` | `/api/convergence/soak` | Owner only | Convergence Soak Status | `pkg/dashboard/api.go:264` | +| `POST` | `/api/plan/from-issue` | Dashboard auth/session | Plan From Issue | `pkg/dashboard/api.go:462` | +| `GET` | `/api/plan/{epicID}` | Dashboard auth/session | Plan Tree | `pkg/dashboard/api.go:465` | +| `POST` | `/api/plan/{epicID}/approve` | Owner only | Plan Approve | `pkg/dashboard/api.go:466` | +| `POST` | `/api/plan/{epicID}/reject` | Owner only | Plan Reject | `pkg/dashboard/api.go:468` | +| `POST` | `/api/plan/{epicID}/child/{childID}` | Owner only | Plan Child Action | `pkg/dashboard/api.go:469` | | `POST` | `/api/linear/agent/install` | Owner only | Linear Agent Install (OAuth start) | `pkg/dashboard/api_linear_agent.go:215` | | `GET` | `/api/linear/agent/status` | Owner only | Linear Agent Status | `pkg/dashboard/api_linear_agent.go:216` | | `POST` | `/api/linear/agent/disconnect` | Owner only | Linear Agent Disconnect | `pkg/dashboard/api_linear_agent.go:217` | -| `GET` | `/api/upstream-watch` | Owner only | Upstream-watch divergence: per watched fork the upstream followed, the watermark and last-run time, surfaced/ported/dismissed/skipped counts, and the recent upstream refs with their upstream URL, diff URL, fork issue number and state. Read-only — it reads the config and the watch state file, never GitHub. Also carries `interval`, `project_repos` and each repo's configured entry (`config`) for the editor. | `pkg/dashboard/api.go:305` | -| `PUT` | `/api/config/upstream-watch` | Owner only | Upstream Watch Config Update: edit `upstream_watch` from the Features tab. Body `{enabled?, interval?, repos?: {: {upstream?, sources?, pr_labels?, max_issues_per_run?, label?} \| null}}` — absent keys are unchanged, `null` removes a repo, keys are stored as the bare repo name. Validated with the loader's rules before the config changes (400 on failure), then persisted and audited as `config_upstream_watch`. | `pkg/dashboard/api.go:306` | +| `GET` | `/api/upstream-watch` | Owner only | Upstream-watch divergence: per watched fork the upstream followed, the watermark and last-run time, surfaced/ported/dismissed/skipped counts, and the recent upstream refs with their upstream URL, diff URL, fork issue number and state. Read-only — it reads the config and the watch state file, never GitHub. Also carries `interval`, `project_repos` and each repo's configured entry (`config`) for the editor. | `pkg/dashboard/api.go:306` | +| `PUT` | `/api/config/upstream-watch` | Owner only | Upstream Watch Config Update: edit `upstream_watch` from the Features tab. Body `{enabled?, interval?, repos?: {: {upstream?, sources?, pr_labels?, max_issues_per_run?, label?} \| null}}` — absent keys are unchanged, `null` removes a repo, keys are stored as the bare repo name. Validated with the loader's rules before the config changes (400 on failure), then persisted and audited as `config_upstream_watch`. | `pkg/dashboard/api.go:307` | -| `GET` | `/api/hotspots` | Dashboard auth/session | Merge-hotspot advisor report (`?repo=owner/name` optional): files repeatedly touched by concurrent PRs over the last `governor.hotspot_window_days`, each with `prs` (distinct PRs), `pr_numbers`, `overlaps` (PR pairs open at the same time), `conflicts` (rebase conflicts blamed on the file), `lines`, `kind` (`code`/`doc`/`lock`/`generated`/`data`), `score`, `symbol_groups`, `recommendation` (`split-code`/`split-doc`/`split-source`/`report-only`/`none`), `reason`, `generated_from`, `source_files` (files under `generated_from`, when known), `issue_number`/`issue_url` of the modularize issue filed, `last_updated`; top-level `count`/`actionable`/`filed`. Derived from the merge train's own PR file lists and the rebase follow-through's conflict records — never lists PRs or files from GitHub. See [operator-reference.md § Merge-hotspot advisor](../operator-reference.md#merge-hotspot-advisor-governorhotspot_). | `pkg/dashboard/api.go:546` | -| `GET` | `/api/widget` | Dashboard auth/session | Widget | `pkg/dashboard/api.go:117` | -| `GET` | `/api/pane/{agent}` | Dashboard auth/session | Pane | `pkg/dashboard/api.go:122` | -| `GET` | `/api/role` | Dashboard auth/session | Role | `pkg/dashboard/api.go:136` | -| `POST` | `/api/reset-restarts/{agent}` | Owner only | Reset Restarts | `pkg/dashboard/api.go:158` | -| `GET` | `/api/budget-ignore` | Dashboard auth/session | Budget Ignore Get | `pkg/dashboard/api.go:176` | -| `POST` | `/api/budget-ignore` | Owner only | Budget Ignore Set | `pkg/dashboard/api.go:177` | -| `GET` | `/api/summaries` | Dashboard auth/session | Summaries | `pkg/dashboard/api.go:189` | -| `GET` | `/api/stage-tracker` | Dashboard auth/session | Stage Tracker: every open issue and pull request of the watched repos for the Hive activity monitor — actionable, held, claimed, in-progress, covered-by-PR, filtered (`needs-*`), draft and exempt-labelled items alike, each with `stage`, `stage_since`, `labels`, `created_at`, `author_kind`, `modifiers`, `claimant: {kind: hive-agent|clanker|person, login, agent, hive, since, source: claim-ledger|claim-comment|assignee|kick-history|author}` naming who holds a claimed / in-progress issue or a PR under review or rework (absent when nobody is named; mirrored onto `linked_prs[]`), (PRs) `sentinel_alert` / `sentinel_findings` / `sentinel_reason` when the sentinel flagged the PR, `decision_needed` on a row parked on a person (085 / 045 / 015, the needs-human pill; mirrored onto the issue's `linked_prs[]` entry): `{question, asked_by, since, blocker, blocker_reason, options: [{action, label, enabled, reason_if_disabled}]}` — the reviewer's own decision question when the verdict record has one, the blocker in plain words, and the applicable actions under the row context menu's action ids (`decide:delegate` first), (PRs) `lineage` on the hive's copy of an adopted PR — `{kind: adopted|superseded|rebased-by-hive, from_number, from_url, from_author, from_author_kind, to_number, to_url, at, reason}`, derived from the stage store's adoption record or the `hive/adopt-` head branch, mirrored onto the issue's `linked_prs[]` entry; absent on every other row — and (issues) `linked_prs`; items with no stage yet report stage `0` "Open"; `needs_human_labels` is the "a person must act" label set behind the monitor's Needs human facet (the same list as `/api/status.labelPresets[needs-human]`); `recently_merged` lists the PRs (`repo`, `number`, `merged_at`) whose `pr_merged` audit entry is under 30 minutes old, so a client diffing two payloads can tell a merge from a close; a PR in a conflict-aware merge train carries `merge_train` `{id, state, front, position, size, attempt, max_attempts, overlap_files, ahead, length, head_number, head_state, head_reason, head_since, eta_s, eta_at, eta_basis, late, stalled, stalled_s}` — `head_state` one of `needs-human` / `ci` / `soaking` / `ready` / `merging` / `fixing` / `unknown` with `head_reason` in plain words; `eta_s` / `eta_at` the estimated wait for this member's own merge attempt (own remaining soak + each PR ahead's remaining soak or one sweep + the next sweep tick; absent while the head waits on a person or CI), `eta_at` anchored to the member's first estimate at its position so a countdown survives polls, `eta_basis` its short reading ("2 ahead · head soaking 12m · next sweep in 40s"), `late` once now has overrun the first estimate twice over, `stalled` / `stalled_s` once the head has sat longer than `governor.merge_train_stall_warn_s` ([#15395](https://github.com/hivecommons/hive/issues/15395)); a PR the queue holds because another open PR declares the same `Fixes #N` carries no `merge_train` and a `merge_gate` of `{state: "duplicate-of:", duplicate_of, duplicate_issue}` plus the `needs-decision` modifier (the queue keeps one PR per issue: green first, then the earlier-opened); a PR GitHub reports conflicting on its own, or one under repair (stage/070, 071, 072, 074, 085, or a red required check), carries no `merge_train` and is not counted in any member's `size` / `length` — a repaired member the view dropped carries `left_train: {why: fixing|needs-author|returned-to-contributor|adopted-by-hive|needs-decision|checks-failing, note}` instead (mirrored onto the issue's `linked_prs[]` entry), and a train left with one member reports no `merge_train` at all; a PR short of the merge queue (below stage/080) the planner ordered rides no train either — it carries `overlap: {behind, with, files, note}` ("overlaps #15637 (sidebar.html, index.html) — will queue behind it") and no `merge_train`, and counts in no member's `size` — the state slugs (`front`, `queued-behind`) are payload values only, the SPA renders ordinal places ("Queued for merge · 1st of 2", with the train's tag when more than one train is on the page). Served from a server-side cache rebuilt at most every `dashboard.stage_tracker.cache_ttl_s` (default `governor.eval_interval_s`) from the spoke's existing snapshot — a request never calls GitHub. Carries `generated_at`, `source` (`cache` or `fresh`), `cache_ttl_s`, an `ETag` and `Cache-Control: private, max-age=`; `If-None-Match` yields 304. Each hit is recorded under REST caller `dashboard:stage-tracker` so `/api/gh-rate-limits` shows it charged nothing. | `pkg/dashboard/api.go:190` | -| `GET` | `/api/matrix/lines` | Dashboard auth/session | Matrix Lines (deprecated name, see [Published-checkpoint regions](#published-checkpoint-regions)): serve bounded backlog of pushed lines behind dashboard.matrix with repo allowlist filtering. `max` counts only lines the repo allowlist admits, so a page is never emptied by filtered-out lines. A client paging with `max` should use the last returned line's `seq` as its next `since`, not the top-level `seq`, which is the source's latest sequence number and would skip lines still past the page. `more` is true only while another line the caller may see remains past the page; stop paging when it is false. | `pkg/dashboard/api.go:191` | -| `POST` | `/api/prs/{owner}/{repo}/{number}/queue-automerge` | Merger/owner role | Queue PRAuto Merge | `pkg/dashboard/api.go:192` | -| `GET` | `/api/inference/models/{backend}` | Dashboard auth/session | Inference Models | `pkg/dashboard/api.go:377` | -| `GET` | `/api/hive-id` | Dashboard auth/session | Hive IDGet | `pkg/dashboard/api.go:440` | -| `PUT` | `/api/hive-id` | Owner only | Hive IDSet | `pkg/dashboard/api.go:441` | -| `POST` | `/api/chat` | Dashboard auth/session | Chat | `pkg/dashboard/api.go:470` | -| `GET` | `/api/auth/token` | Public | Auth Token | `pkg/dashboard/api.go:504` | +| `GET` | `/api/hotspots` | Dashboard auth/session | Merge-hotspot advisor report (`?repo=owner/name` optional): files repeatedly touched by concurrent PRs over the last `governor.hotspot_window_days`, each with `prs` (distinct PRs), `pr_numbers`, `overlaps` (PR pairs open at the same time), `conflicts` (rebase conflicts blamed on the file), `lines`, `kind` (`code`/`doc`/`lock`/`generated`/`data`), `score`, `symbol_groups`, `recommendation` (`split-code`/`split-doc`/`split-source`/`report-only`/`none`), `reason`, `generated_from`, `source_files` (files under `generated_from`, when known), `issue_number`/`issue_url` of the modularize issue filed, `last_updated`; top-level `count`/`actionable`/`filed`. Derived from the merge train's own PR file lists and the rebase follow-through's conflict records — never lists PRs or files from GitHub. See [operator-reference.md § Merge-hotspot advisor](../operator-reference.md#merge-hotspot-advisor-governorhotspot_). | `pkg/dashboard/api.go:547` | +| `GET` | `/api/widget` | Dashboard auth/session | Widget | `pkg/dashboard/api.go:118` | +| `GET` | `/api/pane/{agent}` | Dashboard auth/session | Pane | `pkg/dashboard/api.go:123` | +| `GET` | `/api/role` | Dashboard auth/session | Role | `pkg/dashboard/api.go:137` | +| `POST` | `/api/reset-restarts/{agent}` | Owner only | Reset Restarts | `pkg/dashboard/api.go:159` | +| `GET` | `/api/budget-ignore` | Dashboard auth/session | Budget Ignore Get | `pkg/dashboard/api.go:177` | +| `POST` | `/api/budget-ignore` | Owner only | Budget Ignore Set | `pkg/dashboard/api.go:178` | +| `GET` | `/api/summaries` | Dashboard auth/session | Summaries | `pkg/dashboard/api.go:190` | +| `GET` | `/api/stage-tracker` | Dashboard auth/session | Stage Tracker: every open issue and pull request of the watched repos for the Hive activity monitor — actionable, held, claimed, in-progress, covered-by-PR, filtered (`needs-*`), draft and exempt-labelled items alike, each with `stage`, `stage_since`, `labels`, `created_at`, `author_kind`, `modifiers`, `claimant: {kind: hive-agent|clanker|person, login, agent, hive, since, source: claim-ledger|claim-comment|assignee|kick-history|author}` naming who holds a claimed / in-progress issue or a PR under review or rework (absent when nobody is named; mirrored onto `linked_prs[]`), (PRs) `sentinel_alert` / `sentinel_findings` / `sentinel_reason` when the sentinel flagged the PR, `decision_needed` on a row parked on a person (085 / 045 / 015, the needs-human pill; mirrored onto the issue's `linked_prs[]` entry): `{question, asked_by, since, blocker, blocker_reason, options: [{action, label, enabled, reason_if_disabled}]}` — the reviewer's own decision question when the verdict record has one, the blocker in plain words, and the applicable actions under the row context menu's action ids (`decide:delegate` first), (PRs) `lineage` on the hive's copy of an adopted PR — `{kind: adopted|superseded|rebased-by-hive, from_number, from_url, from_author, from_author_kind, to_number, to_url, at, reason}`, derived from the stage store's adoption record or the `hive/adopt-` head branch, mirrored onto the issue's `linked_prs[]` entry; absent on every other row — and (issues) `linked_prs`; items with no stage yet report stage `0` "Open"; `needs_human_labels` is the "a person must act" label set behind the monitor's Needs human facet (the same list as `/api/status.labelPresets[needs-human]`); `recently_merged` lists the PRs (`repo`, `number`, `merged_at`) whose `pr_merged` audit entry is under 30 minutes old, so a client diffing two payloads can tell a merge from a close; a PR in a conflict-aware merge train carries `merge_train` `{id, state, front, position, size, attempt, max_attempts, overlap_files, ahead, length, head_number, head_state, head_reason, head_since, eta_s, eta_at, eta_basis, late, stalled, stalled_s}` — `head_state` one of `needs-human` / `ci` / `soaking` / `ready` / `merging` / `fixing` / `unknown` with `head_reason` in plain words; `eta_s` / `eta_at` the estimated wait for this member's own merge attempt (own remaining soak + each PR ahead's remaining soak or one sweep + the next sweep tick; absent while the head waits on a person or CI), `eta_at` anchored to the member's first estimate at its position so a countdown survives polls, `eta_basis` its short reading ("2 ahead · head soaking 12m · next sweep in 40s"), `late` once now has overrun the first estimate twice over, `stalled` / `stalled_s` once the head has sat longer than `governor.merge_train_stall_warn_s` ([#15395](https://github.com/hivecommons/hive/issues/15395)); a PR the queue holds because another open PR declares the same `Fixes #N` carries no `merge_train` and a `merge_gate` of `{state: "duplicate-of:", duplicate_of, duplicate_issue}` plus the `needs-decision` modifier (the queue keeps one PR per issue: green first, then the earlier-opened); a PR GitHub reports conflicting on its own, or one under repair (stage/070, 071, 072, 074, 085, or a red required check), carries no `merge_train` and is not counted in any member's `size` / `length` — a repaired member the view dropped carries `left_train: {why: fixing|needs-author|returned-to-contributor|adopted-by-hive|needs-decision|checks-failing, note}` instead (mirrored onto the issue's `linked_prs[]` entry), and a train left with one member reports no `merge_train` at all; a PR short of the merge queue (below stage/080) the planner ordered rides no train either — it carries `overlap: {behind, with, files, note}` ("overlaps #15637 (sidebar.html, index.html) — will queue behind it") and no `merge_train`, and counts in no member's `size` — the state slugs (`front`, `queued-behind`) are payload values only, the SPA renders ordinal places ("Queued for merge · 1st of 2", with the train's tag when more than one train is on the page). Served from a server-side cache rebuilt at most every `dashboard.stage_tracker.cache_ttl_s` (default `governor.eval_interval_s`) from the spoke's existing snapshot — a request never calls GitHub. Carries `generated_at`, `source` (`cache` or `fresh`), `cache_ttl_s`, an `ETag` and `Cache-Control: private, max-age=`; `If-None-Match` yields 304. Each hit is recorded under REST caller `dashboard:stage-tracker` so `/api/gh-rate-limits` shows it charged nothing. | `pkg/dashboard/api.go:191` | +| `GET` | `/api/matrix/lines` | Dashboard auth/session | Matrix Lines (deprecated name, see [Published-checkpoint regions](#published-checkpoint-regions)): serve bounded backlog of pushed lines behind dashboard.matrix with repo allowlist filtering. `max` counts only lines the repo allowlist admits, so a page is never emptied by filtered-out lines. A client paging with `max` should use the last returned line's `seq` as its next `since`, not the top-level `seq`, which is the source's latest sequence number and would skip lines still past the page. `more` is true only while another line the caller may see remains past the page; stop paging when it is false. | `pkg/dashboard/api.go:192` | +| `POST` | `/api/prs/{owner}/{repo}/{number}/queue-automerge` | Merger/owner role | Queue PRAuto Merge | `pkg/dashboard/api.go:193` | +| `GET` | `/api/inference/models/{backend}` | Dashboard auth/session | Inference Models | `pkg/dashboard/api.go:378` | +| `GET` | `/api/hive-id` | Dashboard auth/session | Hive IDGet | `pkg/dashboard/api.go:441` | +| `PUT` | `/api/hive-id` | Owner only | Hive IDSet | `pkg/dashboard/api.go:442` | +| `POST` | `/api/chat` | Dashboard auth/session | Chat | `pkg/dashboard/api.go:471` | +| `GET` | `/api/auth/token` | Public | Auth Token | `pkg/dashboard/api.go:505` | | `GET` | `/api/v1/` | GitHub token | Contributor v1 API dispatcher | `pkg/dashboard/api_contribute.go:350` | | `POST` | `/api/v1/` | GitHub token | Contributor v1 API dispatcher | `pkg/dashboard/api_contribute.go:351` | | `GET` | `/api/docs` | Dashboard auth/session | APIDocs | `pkg/dashboard/api_contribute.go:352` | diff --git a/src/docs/api-reference/health-status-events.md b/src/docs/api-reference/health-status-events.md index d1217f0209..e992c4eff0 100644 --- a/src/docs/api-reference/health-status-events.md +++ b/src/docs/api-reference/health-status-events.md @@ -21,9 +21,9 @@ Part of the [Dashboard REST API reference](../api-reference.md); see that page f | Method | Path | Auth | Purpose | Source | |---|---|---|---|---| -| `GET` | `/api/version` | Dashboard auth/session | Build/version metadata; includes `upgradeMarker` (`target`, `current`, `attempts`, `maxAttempts`, `failed`, `requestedAt`, `lastError`) while a self-upgrade is in flight or has failed ([#6765](https://github.com/hivecommons/hive/issues/6765)), the hub-delivered `upgradePolicy` (`hub_managed`, `spoke_managed`, `schedule`, `schedule_hour`, `schedule_timezone`, `schedule_weekday`, `paused`, `branch`, `channel`, `target_sha`, `target_resolved`, `armed_target`, and `next_update_at` — the hub's RFC3339 ETA for the next promotion into the `stable` channel, omitted when unknown ([#10256](https://github.com/hivecommons/hive/issues/10256))) when available, and an `autoUpdate` object (`enabled`, `state` — one of `disabled`/`up_to_date`/`behind`/`retrying`/`failed`/`unknown` — `healthy`, `period`, `targetBranch`, `targetCommit`, `currentCommit`, `commitsBehind`, `lastAttemptAt`, `lastError`, `detail`, and `nextUpdateAt` — the hub's RFC3339 ETA for the next update, omitted when unknown, disabled or paused ([#10257](https://github.com/hivecommons/hive/issues/10257))) that never reports a failed or unknown update as healthy ([#6962](https://github.com/hivecommons/hive/issues/6962), [#6963](https://github.com/hivecommons/hive/issues/6963)). Also `startedAt` (RFC3339 process start — changes on every pod roll even when the build does not, so the dashboard can say "Restarted" vs "Upgraded") and, until the first status is published, `startupPhase`/`startupPhaseLabel` (the boot phase a fresh pod is in) ([#15788](https://github.com/hivecommons/hive/issues/15788)) | `pkg/dashboard/api.go:44` | -| `GET` | `/api/version/release-notes` | Dashboard auth/session | Changelog gap between two revisions: `?from=&to=` returns `{from:{sha,version?}, to:{sha,ref,version?}, sections:[{version,date,categories:{added,changed,fixed,security,deprecated}}], unreleased?, truncated, source, error?}` — the `CHANGELOG.md` release sections present at `to` and absent at `from`, newest first, plus (when `to` is not a release tag) the `changelog.d/` fragments new at `to` under an `Unreleased (in this build)` section. Fetch failures return `source: "unavailable"` with `error` rather than an error status; results are cached per (from, to) for an hour ([#11065](https://github.com/hivecommons/hive/issues/11065)) | `pkg/dashboard/api.go:45` | -| `POST` | `/api/release-channel` | Owner only | Hosted spoke self-service release-channel selector; relays `stable`/`candidate`/`edge` to the hub's existing switch-branch endpoint with the spoke dashboard-token proof, and reports the requested channel as pending until the Deployment image lands | `pkg/dashboard/api.go:69` | +| `GET` | `/api/version` | Dashboard auth/session | Build/version metadata; includes `upgradeMarker` (`target`, `current`, `attempts`, `maxAttempts`, `failed`, `requestedAt`, `lastError`) while a self-upgrade is in flight or has failed ([#6765](https://github.com/hivecommons/hive/issues/6765)), the hub-delivered `upgradePolicy` (`hub_managed`, `spoke_managed`, `schedule`, `schedule_hour`, `schedule_timezone`, `schedule_weekday`, `paused`, `branch`, `channel`, `target_sha`, `target_resolved`, `armed_target`, and `next_update_at` — the hub's RFC3339 ETA for the next promotion into the `stable` channel, omitted when unknown ([#10256](https://github.com/hivecommons/hive/issues/10256))) when available, and an `autoUpdate` object (`enabled`, `state` — one of `disabled`/`up_to_date`/`behind`/`retrying`/`failed`/`unknown` — `healthy`, `period`, `targetBranch`, `targetCommit`, `currentCommit`, `commitsBehind`, `lastAttemptAt`, `lastError`, `detail`, and `nextUpdateAt` — the hub's RFC3339 ETA for the next update, omitted when unknown, disabled or paused ([#10257](https://github.com/hivecommons/hive/issues/10257))) that never reports a failed or unknown update as healthy ([#6962](https://github.com/hivecommons/hive/issues/6962), [#6963](https://github.com/hivecommons/hive/issues/6963)). Also `startedAt` (RFC3339 process start — changes on every pod roll even when the build does not, so the dashboard can say "Restarted" vs "Upgraded") and, until the first status is published, `startupPhase`/`startupPhaseLabel` (the boot phase a fresh pod is in) ([#15788](https://github.com/hivecommons/hive/issues/15788)) | `pkg/dashboard/api.go:45` | +| `GET` | `/api/version/release-notes` | Dashboard auth/session | Changelog gap between two revisions: `?from=&to=` returns `{from:{sha,version?}, to:{sha,ref,version?}, sections:[{version,date,categories:{added,changed,fixed,security,deprecated}}], unreleased?, truncated, source, error?}` — the `CHANGELOG.md` release sections present at `to` and absent at `from`, newest first, plus (when `to` is not a release tag) the `changelog.d/` fragments new at `to` under an `Unreleased (in this build)` section. Fetch failures return `source: "unavailable"` with `error` rather than an error status; results are cached per (from, to) for an hour ([#11065](https://github.com/hivecommons/hive/issues/11065)) | `pkg/dashboard/api.go:46` | +| `POST` | `/api/release-channel` | Owner only | Hosted spoke self-service release-channel selector; relays `stable`/`candidate`/`edge` to the hub's existing switch-branch endpoint with the spoke dashboard-token proof, and reports the requested channel as pending until the Deployment image lands | `pkg/dashboard/api.go:70` | | `GET` | `/api/health` | Public | Basic health probe | `pkg/dashboard/server.go:1655` | | `GET` | `/api/health/deep` | Public | Deep health probe | `pkg/dashboard/server.go:1656` | | `GET` | `/api/livez` | Public | Kubernetes liveness probe | `pkg/dashboard/server.go:1657` | @@ -44,30 +44,30 @@ Part of the [Dashboard REST API reference](../api-reference.md); see that page f | `PUT` | `/api/swarm/themes` | Owner only | Set a per-repo swarm theme (`event_name`, `call_to_arms`, `leaderboard_title`). | `pkg/dashboard/swarm.go:316` | | `PATCH` | `/api/swarm/objectives` | Owner only | Update the active swarm SDLC objective checklist completion state. | `pkg/dashboard/swarm.go:317` | | `GET` | `/api/leaderboard/swarm` | Public | Stable public JSON record of swarm history, leaderboard, top players, and achievements. | `pkg/dashboard/swarm.go:318` | -| `GET` | `/api/runs` | Dashboard auth/session | Active staged runs projected from task leases, plans, and lifecycle timeline. Returns an array of `Run` objects: canonical `key` (`owner/repo#number`), current-stage `lease_key`, `title`, `repo` (`owner/repo`), `stage`, `gen`, `stage_started_at`, `waiting_on` (`agent`, `remote`, `human`, `ci`, `none`), `waiting_since`, `assignee`, `last_receipt`, `plan_epic_id`, and `stages[]` (`name`, `status`, `gen`, `receipt`). | `pkg/dashboard/api.go:90` | -| `GET` | `/api/runs/audit` | Owner only | Read-only cross-repo audit query over existing retained artifacts. Query `repo`, canonical `run` (`owner/repo#number`), RFC3339 `since`/`until`, `kind`, `limit` (default 100, max 500), and `page_token`. Returns `{items,next_page_token,retention}`; each item has `source` (`audit_log`, `timeline`, `lease_receipt`, `plan_epic`), `kind`, `run`, `repo`, `at`, `actor`, `artifact_id`, `attrs`, or an explicit `kind=expired,status=expired,state=Unknown` marker when requested history may have aged out. | `pkg/dashboard/api.go:91` | -| `POST` | `/api/runs/audit` | Owner only | Activate the convergence audit campaign over a scope directory. Body `{"campaign_key":"audit-campaign","scope_dir":"/data/convergence/audit/scope","store":"audit","generation":1,"run_key":"","run_url":""}`; `scope_dir` defaults to `/data/convergence/audit/scope`, `store` defaults to `audit`, `auditor`, `scanner`, `supervisor`, then the first configured bead store. Runs inspection even when `publication.enabled` is false, but then skips the publisher and returns `publication_skipped:true` with reason `publication.disabled`. | `pkg/dashboard/api.go:92` | -| `GET` | `/api/runs/{key}/checkpoint` | Dashboard auth/session | Compact mobile/chat checkpoint review payload for a held run. Returns `run_key`, `stage`, lease `gen`, `repo`, `title`, capped plain-text `summary`, approve/reject `decisions`, dashboard deep link, `lease_gen` staleness fence, and verified-owner approver rule. | `pkg/dashboard/api.go:94` | -| `POST` | `/api/runs/{key}/checkpoint` | Owner only | Resolve a compact checkpoint decision. Body `{"action":"approve"|"reject","gen":}`; refuses stale generations with 409 and non-owner callers with 403 before mutating the plan. | `pkg/dashboard/api.go:95` | -| `GET` | `/api/runs/{key}` | Dashboard auth/session | One run by URL-escaped canonical run key (`owner/repo%23number`); early lease-shaped keys (`repo!run-key:stage`) and queued Wavefront run-stage keys (`owner/repo!graph:node`) are also accepted. Includes timeline-derived stage history in the same `Run` shape as `/api/runs` when a lease/timeline exists. History entries that came from an owner reset carry the owner's `reason`. When the key belongs to a wired convergence campaign, the detail response also includes an optional `burndown` block with `source`, `satisfied`, `remaining`, `unknown`, and `scope`; the list endpoint omits this block. | `pkg/dashboard/api.go:96` | -| `POST` | `/api/runs/{key}/reset` | Owner only | Move a run's lease back to an earlier stage (for example `implement` to `plan` after a rejected plan). Body `{"to": "", "reason": ""}`, both required. Mints a new generation so the relay holding the old one can no longer resume, persists before answering, and records the reason on the agent audit sink, the lifecycle timeline, and the `stage_completed` hook payload (`attrs.reset = "true"`). Returns `{ok, key, stage_from, stage, gen, reason}`. 400 for the same or a later stage or a missing reason, 404 when no live staged lease holds the key, 409 when the lease expired, 500 when the registry could not be written. | `pkg/dashboard/api.go:97` | -| `GET` | `/api/campaigns` | Dashboard auth/session | List resumable campaigns from archived Inception sessions and retained Spektacular (Spek) runs. Query `search`/`q`, `repo`, `stage`, `status`, and `owner`; each campaign is keyed by stable spec/session id and includes title, source, repos, current stage/step, artifact links, linked issues/PRs, contributors, last activity, status, engine, type, run link when applicable, `superseded_by` when a newer Revise revision exists, and recheck status (`enabled`, `interval`, `last_at`, `next_at`, `in_flight`, `last_delta_count`, `external_count`, `sources_failed`). | `pkg/dashboard/api.go:98` | -| `GET` | `/api/campaigns/{id}` | Dashboard auth/session | Return one campaign by URL-escaped stable campaign id or run key, using the same campaign shape as `/api/campaigns`; recheck revisions may include a `drift` block with `codebase_changed`, `prior_revision`, `delta_count`, `recheck_reason`, `external[]` (`source`, `kind`, `title`, `url`, `published_at`, `summary`), `external_count`, and `sources_failed`. | `pkg/dashboard/api.go:99` | -| `POST` | `/api/campaigns/{id}/resume` | Owner only | Resume or pick up a campaign. Archived Inception campaigns take a durable campaign lease, restore their saved state and wiki working files; Spek campaigns return the retained run detail plus a CLI status command that reloads state from Spek artifacts. Body may include `{ "surface": "chat"|"contributor"|"cli"|"dashboard" }`; 409 means another user holds the lease. | `pkg/dashboard/api.go:100` | -| `POST` | `/api/campaigns/{id}/release` | Owner only | Release the caller's Inception campaign lease and write a `campaign_release` audit entry so another user can continue it. Run-backed Spektacular campaigns return `409 Conflict`; this endpoint never revokes their run-lifecycle stage leases. | `pkg/dashboard/api.go:101` | -| `POST` | `/api/campaigns/{id}/revise` | Owner only | Create a linked revision campaign (`revision_of`, `revision`) for revisiting shipped or parked Inception/Spek work without overwriting the previous campaign. The new revision is leased to the caller and audit-logged as `campaign_revise`. | `pkg/dashboard/api.go:102` | -| `DELETE` | `/api/campaigns/{id}` | Owner only | Delete a campaign record: removes the archived Inception state and wiki snapshot for the stable campaign id and audit-logs `campaign_delete`. Knowledge-base facts, Jam threads, scaffolded repo files and other revisions in the same chain are kept; deleting the live L1 session clears it. Run-backed Spektacular campaigns return `409 Conflict`; unknown ids `404`. | `pkg/dashboard/api.go:103` | -| `POST` | `/api/campaigns/{id}/recheck` | Owner only | Create an immediate linked Spek recheck revision using the cadence path. Returns `409 Conflict` when a recheck revision is already in flight, `404 Not Found` when recheck is disabled unless `?force=true` is supplied, and records `campaign_recheck`. | `pkg/dashboard/api.go:104` | -| `GET` | `/api/campaigns/{id}/jam` | Dashboard auth/session | Return the per-campaign Jam workspace: section-anchored threads, comments, suggestions, polls, poll decisions, current spec revision and attributed revision history. | `pkg/dashboard/api.go:105` | -| `POST` | `/api/campaigns/{id}/jam` | Read-write role | Record an attributed spec revision for a campaign. Body includes `spec_content`, optional `reason`, and optional agent/model attribution fields. | `pkg/dashboard/api.go:106` | -| `GET` | `/api/campaigns/{id}/jam/threads` | Dashboard auth/session | List async Jam threads anchored to spec sections, including comments and author attribution. | `pkg/dashboard/api.go:107` | -| `POST` | `/api/campaigns/{id}/jam/threads` | Read-write role | Create a section-anchored Jam thread or add a comment to an existing thread. Body includes `section` and `body`, or `thread_id` and `body`. | `pkg/dashboard/api.go:108` | -| `POST` | `/api/campaigns/{id}/jam/agents` | Maintainer only | Invite Spektacular or a configured hive agent into a Jam thread. The agent reply is posted as an attributed thread comment and an open suggestion for human accept/reject; no spec revision is auto-applied. | `pkg/dashboard/api.go:109` | -| `GET` | `/api/campaigns/{id}/jam/suggestions` | Dashboard auth/session | List Jam suggestions with open/accepted/rejected status, proposed text, attribution and applied revision id when accepted. | `pkg/dashboard/api.go:110` | -| `POST` | `/api/campaigns/{id}/jam/suggestions` | Read-write role; maintainer for accept | Create a suggestion (`section`, `proposed_text`) or resolve it with `action: "accept"`/`"reject"` and `suggestion_id`. Accepting records a new attributed spec revision. | `pkg/dashboard/api.go:111` | -| `GET` | `/api/campaigns/{id}/jam/polls` | Dashboard auth/session | List Jam polls, advisory vote tallies and maintainer decisions stored against spec revisions. | `pkg/dashboard/api.go:112` | -| `POST` | `/api/campaigns/{id}/jam/polls` | Read-write role; maintainer for decide | Create polls (`section`, `question`, `options`), cast/update advisory votes (`action: "vote"`), or record a maintainer decision with rationale (`action: "decide"`). | `pkg/dashboard/api.go:113` | -| `GET` | `/api/campaigns/{id}/jam/project-sync` | Dashboard auth/session | Return the campaign's GitHub Projects sync configuration, last published items, last status, and retry guidance for failures. | `pkg/dashboard/api.go:114` | -| `POST` | `/api/campaigns/{id}/jam/project-sync` | Maintainer only | Enable/disable opt-in GitHub Projects sync, publish campaign specs/decisions/derived issues to the linked project, or record inbound project status updates without losing local Jam decisions. | `pkg/dashboard/api.go:115` | -| `GET` | `/api/campaigns/{id}/jam/ws` | Dashboard auth/session | Upgrade to a Jam WebSocket for live participant presence, section focus, and conflict-aware spec co-edit messages. Live edits require read-write role and stale `base_revision_id` values are rejected with a conflict message. | `pkg/dashboard/api.go:116` | -| `GET` | `/api/runs/{key}/trace` | Dashboard auth/session | Resolve `Hive-Run`, `Hive-Plan`, and `Hive-Spec` commit trailers for `?sha=...`, returning the linked plan section, spek clause, approval audit record, rationale audit entries, and typed no-linkage results when trailers are missing. | `pkg/dashboard/api.go:93` | +| `GET` | `/api/runs` | Dashboard auth/session | Active staged runs projected from task leases, plans, and lifecycle timeline. Returns an array of `Run` objects: canonical `key` (`owner/repo#number`), current-stage `lease_key`, `title`, `repo` (`owner/repo`), `stage`, `gen`, `stage_started_at`, `waiting_on` (`agent`, `remote`, `human`, `ci`, `none`), `waiting_since`, `assignee`, `last_receipt`, `plan_epic_id`, and `stages[]` (`name`, `status`, `gen`, `receipt`). | `pkg/dashboard/api.go:91` | +| `GET` | `/api/runs/audit` | Owner only | Read-only cross-repo audit query over existing retained artifacts. Query `repo`, canonical `run` (`owner/repo#number`), RFC3339 `since`/`until`, `kind`, `limit` (default 100, max 500), and `page_token`. Returns `{items,next_page_token,retention}`; each item has `source` (`audit_log`, `timeline`, `lease_receipt`, `plan_epic`), `kind`, `run`, `repo`, `at`, `actor`, `artifact_id`, `attrs`, or an explicit `kind=expired,status=expired,state=Unknown` marker when requested history may have aged out. | `pkg/dashboard/api.go:92` | +| `POST` | `/api/runs/audit` | Owner only | Activate the convergence audit campaign over a scope directory. Body `{"campaign_key":"audit-campaign","scope_dir":"/data/convergence/audit/scope","store":"audit","generation":1,"run_key":"","run_url":""}`; `scope_dir` defaults to `/data/convergence/audit/scope`, `store` defaults to `audit`, `auditor`, `scanner`, `supervisor`, then the first configured bead store. Runs inspection even when `publication.enabled` is false, but then skips the publisher and returns `publication_skipped:true` with reason `publication.disabled`. | `pkg/dashboard/api.go:93` | +| `GET` | `/api/runs/{key}/checkpoint` | Dashboard auth/session | Compact mobile/chat checkpoint review payload for a held run. Returns `run_key`, `stage`, lease `gen`, `repo`, `title`, capped plain-text `summary`, approve/reject `decisions`, dashboard deep link, `lease_gen` staleness fence, and verified-owner approver rule. | `pkg/dashboard/api.go:95` | +| `POST` | `/api/runs/{key}/checkpoint` | Owner only | Resolve a compact checkpoint decision. Body `{"action":"approve"|"reject","gen":}`; refuses stale generations with 409 and non-owner callers with 403 before mutating the plan. | `pkg/dashboard/api.go:96` | +| `GET` | `/api/runs/{key}` | Dashboard auth/session | One run by URL-escaped canonical run key (`owner/repo%23number`); early lease-shaped keys (`repo!run-key:stage`) and queued Wavefront run-stage keys (`owner/repo!graph:node`) are also accepted. Includes timeline-derived stage history in the same `Run` shape as `/api/runs` when a lease/timeline exists. History entries that came from an owner reset carry the owner's `reason`. When the key belongs to a wired convergence campaign, the detail response also includes an optional `burndown` block with `source`, `satisfied`, `remaining`, `unknown`, and `scope`; the list endpoint omits this block. | `pkg/dashboard/api.go:97` | +| `POST` | `/api/runs/{key}/reset` | Owner only | Move a run's lease back to an earlier stage (for example `implement` to `plan` after a rejected plan). Body `{"to": "", "reason": ""}`, both required. Mints a new generation so the relay holding the old one can no longer resume, persists before answering, and records the reason on the agent audit sink, the lifecycle timeline, and the `stage_completed` hook payload (`attrs.reset = "true"`). Returns `{ok, key, stage_from, stage, gen, reason}`. 400 for the same or a later stage or a missing reason, 404 when no live staged lease holds the key, 409 when the lease expired, 500 when the registry could not be written. | `pkg/dashboard/api.go:98` | +| `GET` | `/api/campaigns` | Dashboard auth/session | List resumable campaigns from archived Inception sessions and retained Spektacular (Spek) runs. Query `search`/`q`, `repo`, `stage`, `status`, and `owner`; each campaign is keyed by stable spec/session id and includes title, source, repos, current stage/step, artifact links, linked issues/PRs, contributors, last activity, status, engine, type, run link when applicable, `superseded_by` when a newer Revise revision exists, and recheck status (`enabled`, `interval`, `last_at`, `next_at`, `in_flight`, `last_delta_count`, `external_count`, `sources_failed`). | `pkg/dashboard/api.go:99` | +| `GET` | `/api/campaigns/{id}` | Dashboard auth/session | Return one campaign by URL-escaped stable campaign id or run key, using the same campaign shape as `/api/campaigns`; recheck revisions may include a `drift` block with `codebase_changed`, `prior_revision`, `delta_count`, `recheck_reason`, `external[]` (`source`, `kind`, `title`, `url`, `published_at`, `summary`), `external_count`, and `sources_failed`. | `pkg/dashboard/api.go:100` | +| `POST` | `/api/campaigns/{id}/resume` | Owner only | Resume or pick up a campaign. Archived Inception campaigns take a durable campaign lease, restore their saved state and wiki working files; Spek campaigns return the retained run detail plus a CLI status command that reloads state from Spek artifacts. Body may include `{ "surface": "chat"|"contributor"|"cli"|"dashboard" }`; 409 means another user holds the lease. | `pkg/dashboard/api.go:101` | +| `POST` | `/api/campaigns/{id}/release` | Owner only | Release the caller's Inception campaign lease and write a `campaign_release` audit entry so another user can continue it. Run-backed Spektacular campaigns return `409 Conflict`; this endpoint never revokes their run-lifecycle stage leases. | `pkg/dashboard/api.go:102` | +| `POST` | `/api/campaigns/{id}/revise` | Owner only | Create a linked revision campaign (`revision_of`, `revision`) for revisiting shipped or parked Inception/Spek work without overwriting the previous campaign. The new revision is leased to the caller and audit-logged as `campaign_revise`. | `pkg/dashboard/api.go:103` | +| `DELETE` | `/api/campaigns/{id}` | Owner only | Delete a campaign record: removes the archived Inception state and wiki snapshot for the stable campaign id and audit-logs `campaign_delete`. Knowledge-base facts, Jam threads, scaffolded repo files and other revisions in the same chain are kept; deleting the live L1 session clears it. Run-backed Spektacular campaigns return `409 Conflict`; unknown ids `404`. | `pkg/dashboard/api.go:104` | +| `POST` | `/api/campaigns/{id}/recheck` | Owner only | Create an immediate linked Spek recheck revision using the cadence path. Returns `409 Conflict` when a recheck revision is already in flight, `404 Not Found` when recheck is disabled unless `?force=true` is supplied, and records `campaign_recheck`. | `pkg/dashboard/api.go:105` | +| `GET` | `/api/campaigns/{id}/jam` | Dashboard auth/session | Return the per-campaign Jam workspace: section-anchored threads, comments, suggestions, polls, poll decisions, current spec revision and attributed revision history. | `pkg/dashboard/api.go:106` | +| `POST` | `/api/campaigns/{id}/jam` | Read-write role | Record an attributed spec revision for a campaign. Body includes `spec_content`, optional `reason`, and optional agent/model attribution fields. | `pkg/dashboard/api.go:107` | +| `GET` | `/api/campaigns/{id}/jam/threads` | Dashboard auth/session | List async Jam threads anchored to spec sections, including comments and author attribution. | `pkg/dashboard/api.go:108` | +| `POST` | `/api/campaigns/{id}/jam/threads` | Read-write role | Create a section-anchored Jam thread or add a comment to an existing thread. Body includes `section` and `body`, or `thread_id` and `body`. | `pkg/dashboard/api.go:109` | +| `POST` | `/api/campaigns/{id}/jam/agents` | Maintainer only | Invite Spektacular or a configured hive agent into a Jam thread. The agent reply is posted as an attributed thread comment and an open suggestion for human accept/reject; no spec revision is auto-applied. | `pkg/dashboard/api.go:110` | +| `GET` | `/api/campaigns/{id}/jam/suggestions` | Dashboard auth/session | List Jam suggestions with open/accepted/rejected status, proposed text, attribution and applied revision id when accepted. | `pkg/dashboard/api.go:111` | +| `POST` | `/api/campaigns/{id}/jam/suggestions` | Read-write role; maintainer for accept | Create a suggestion (`section`, `proposed_text`) or resolve it with `action: "accept"`/`"reject"` and `suggestion_id`. Accepting records a new attributed spec revision. | `pkg/dashboard/api.go:112` | +| `GET` | `/api/campaigns/{id}/jam/polls` | Dashboard auth/session | List Jam polls, advisory vote tallies and maintainer decisions stored against spec revisions. | `pkg/dashboard/api.go:113` | +| `POST` | `/api/campaigns/{id}/jam/polls` | Read-write role; maintainer for decide | Create polls (`section`, `question`, `options`), cast/update advisory votes (`action: "vote"`), or record a maintainer decision with rationale (`action: "decide"`). | `pkg/dashboard/api.go:114` | +| `GET` | `/api/campaigns/{id}/jam/project-sync` | Dashboard auth/session | Return the campaign's GitHub Projects sync configuration, last published items, last status, and retry guidance for failures. | `pkg/dashboard/api.go:115` | +| `POST` | `/api/campaigns/{id}/jam/project-sync` | Maintainer only | Enable/disable opt-in GitHub Projects sync, publish campaign specs/decisions/derived issues to the linked project, or record inbound project status updates without losing local Jam decisions. | `pkg/dashboard/api.go:116` | +| `GET` | `/api/campaigns/{id}/jam/ws` | Dashboard auth/session | Upgrade to a Jam WebSocket for live participant presence, section focus, and conflict-aware spec co-edit messages. Live edits require read-write role and stale `base_revision_id` values are rejected with a conflict message. | `pkg/dashboard/api.go:117` | +| `GET` | `/api/runs/{key}/trace` | Dashboard auth/session | Resolve `Hive-Run`, `Hive-Plan`, and `Hive-Spec` commit trailers for `?sha=...`, returning the linked plan section, spek clause, approval audit record, rationale audit entries, and typed no-linkage results when trailers are missing. | `pkg/dashboard/api.go:94` | diff --git a/src/docs/api-reference/inception.md b/src/docs/api-reference/inception.md index 9446f8926b..0260a12e83 100644 --- a/src/docs/api-reference/inception.md +++ b/src/docs/api-reference/inception.md @@ -5,17 +5,17 @@ Part of the [Dashboard REST API reference](../api-reference.md); see that page f | Method | Path | Auth | Purpose | Source | |---|---|---|---|---| -| `POST` | `/api/inception/start` | Owner only | Inception Start | `pkg/dashboard/api.go:443` | -| `POST` | `/api/inception/scan` | Owner only | Inception Scan | `pkg/dashboard/api.go:444` | -| `GET` | `/api/inception/state` | Dashboard auth/session | Inception State | `pkg/dashboard/api.go:445` | -| `POST` | `/api/inception/questions` | Owner only | Inception Set Questions | `pkg/dashboard/api.go:446` | -| `POST` | `/api/inception/answer` | Owner only | Inception Answer | `pkg/dashboard/api.go:447` | -| `POST` | `/api/inception/facts` | Owner only | Inception Record Facts | `pkg/dashboard/api.go:448` | -| `GET` | `/api/inception/scaffold` | Dashboard auth/session | Inception Scaffold | `pkg/dashboard/api.go:449` | -| `POST` | `/api/inception/approve` | Owner only | Inception Approve | `pkg/dashboard/api.go:450` | -| `POST` | `/api/inception/reset` | Owner only | Inception Reset | `pkg/dashboard/api.go:451` | -| `GET` | `/api/inception/ideation-facts` | Dashboard auth/session | Inception Ideation Facts | `pkg/dashboard/api.go:452` | -| `GET` | `/api/inception/download` | Dashboard auth/session | Inception Download | `pkg/dashboard/api.go:453` | -| `GET` | `/api/inception/has-files` | Dashboard auth/session | Inception Has Files | `pkg/dashboard/api.go:454` | -| `PUT` | `/api/inception/wiki-name` | Owner only | Inception Rename Wiki | `pkg/dashboard/api.go:455` | -| `POST` | `/api/inception/import` | Owner only | Inception Import | `pkg/dashboard/api.go:456` | +| `POST` | `/api/inception/start` | Owner only | Inception Start | `pkg/dashboard/api.go:444` | +| `POST` | `/api/inception/scan` | Owner only | Inception Scan | `pkg/dashboard/api.go:445` | +| `GET` | `/api/inception/state` | Dashboard auth/session | Inception State | `pkg/dashboard/api.go:446` | +| `POST` | `/api/inception/questions` | Owner only | Inception Set Questions | `pkg/dashboard/api.go:447` | +| `POST` | `/api/inception/answer` | Owner only | Inception Answer | `pkg/dashboard/api.go:448` | +| `POST` | `/api/inception/facts` | Owner only | Inception Record Facts | `pkg/dashboard/api.go:449` | +| `GET` | `/api/inception/scaffold` | Dashboard auth/session | Inception Scaffold | `pkg/dashboard/api.go:450` | +| `POST` | `/api/inception/approve` | Owner only | Inception Approve | `pkg/dashboard/api.go:451` | +| `POST` | `/api/inception/reset` | Owner only | Inception Reset | `pkg/dashboard/api.go:452` | +| `GET` | `/api/inception/ideation-facts` | Dashboard auth/session | Inception Ideation Facts | `pkg/dashboard/api.go:453` | +| `GET` | `/api/inception/download` | Dashboard auth/session | Inception Download | `pkg/dashboard/api.go:454` | +| `GET` | `/api/inception/has-files` | Dashboard auth/session | Inception Has Files | `pkg/dashboard/api.go:455` | +| `PUT` | `/api/inception/wiki-name` | Owner only | Inception Rename Wiki | `pkg/dashboard/api.go:456` | +| `POST` | `/api/inception/import` | Owner only | Inception Import | `pkg/dashboard/api.go:457` | diff --git a/src/docs/api-reference/knowledge.md b/src/docs/api-reference/knowledge.md index dacadb8729..ac9bc8f73a 100644 --- a/src/docs/api-reference/knowledge.md +++ b/src/docs/api-reference/knowledge.md @@ -5,38 +5,38 @@ Part of the [Dashboard REST API reference](../api-reference.md); see that page f | Method | Path | Auth | Purpose | Source | |---|---|---|---|---| -| `GET` | `/api/knowledge` | Dashboard auth/session | Paginated knowledge facts: `type` filter, `limit` (default 50, max 200; invalid/non-positive values use the default), `offset` (default 0). Returns `enabled`, `facts`, `count` (this page), `total`, `limit`, `offset`, `has_more`; ordered by layer then slug. Offsets are not snapshot cursors: concurrent edits may shift pages. | `pkg/dashboard/api.go:379` | -| `GET` | `/api/knowledge/export` | Dashboard auth/session | Knowledge Export | `pkg/dashboard/api.go:380` | -| `POST` | `/mcp/knowledge` | Public (read-only) when the dashboard setting or `HIVE_PUBLIC_KNOWLEDGE` is on; otherwise 404 to a signed-in caller and 401 to one who is not | Public Knowledge MCP endpoint — `initialize`, `tools/list`, `tools/call` for `knowledge_search`/`knowledge_toc`/`knowledge_get`/`knowledge_export` ([docs](../public-knowledge-mcp.md)) | `pkg/dashboard/api.go:389` | -| `GET` | `/api/knowledge/search` | Dashboard auth/session | Knowledge Search | `pkg/dashboard/api.go:381` | -| `GET` | `/api/knowledge/toc` | Dashboard auth/session | Scoped knowledge table of contents: capped, body-free entries (`id`, `title`, `type`, `layer`, `repo`, `tags`, `status`, `confidence`, `updated`, `size_bytes`, `source`) plus `total`/`returned`/`truncated`. Query: `layers`, `repos`, `types`, `tags` (comma-separated), `limit` (default 50, max 200), `include_states` (operator opt-in for `draft`/`deprecated`/`superseded`/`all`), `format=prompt` with `max_chars` for a bounded markdown rendering, `agent` to apply that agent's `knowledge.agent_scopes` entry (the other parameters can only narrow it). Approved knowledge only by default ([docs](../knowledge-toc.md)) | `pkg/dashboard/api.go:382` | -| `GET` | `/api/knowledge/effective` | Dashboard auth/session | Effective knowledge for one agent (`agent` required): the agent's `knowledge.agent_scopes` entry (unrestricted when it has none) intersected with the request scope (approved only unless `include_states` is set). Returns `scope` (resolved scope applied, with `configured`), `included` TOC entries, `excluded` entries each with a `reason` (`lifecycle`, `layer`, `repo`, `type` or `tag` — first failing check, in that order), `included_total`/`excluded_total`/`included_returned`/`excluded_returned` and `truncated`; `limit` caps each list (default 50, max 200) ([docs](../knowledge-toc.md#effective-knowledge)) | `pkg/dashboard/api.go:383` | -| `GET` | `/api/knowledge/entry/{id}` | Dashboard auth/session | Full read of one knowledge entry: `markdown` (front-matter plus body) and the fact. Takes the same scope, `include_states` and `agent` query as the table of contents and returns 404 for an entry outside it ([docs](../knowledge-toc.md)) | `pkg/dashboard/api.go:384` | -| `POST` | `/api/knowledge/cli-memory/drain-legacy` | Owner only | One-shot drain of the shared home's legacy Claude memory (`/data/home/.claude/projects/*/memory`) into `state: draft` entries in `bead-synth-wiki`, tagged `cli-memory` and `agent:shared`, deduplicated by content hash. Body `{"truncate": bool}` (default false) empties only files whose every fact is now stored. Returns `store`, `agent`, `sources`, `drained`, `dupes`, `skipped`, `truncate`, `truncated`, `kept`, `errors`; 409 when the vault is not connected. Audited as `knowledge_drain_legacy_memory` ([docs](../knowledge-lifecycle.md#auto-memory-drafts)) | `pkg/dashboard/api.go:386` | -| `GET` | `/api/knowledge/public` | Dashboard auth/session | Public Knowledge MCP sharing status (`enabled`, `tags`, `source`, endpoint URL) | `pkg/dashboard/api.go:390` | -| `PUT` | `/api/knowledge/public` | Owner only | Persist public Knowledge MCP sharing setting; overrides env vars until changed | `pkg/dashboard/api.go:391` | -| `GET` | `/api/knowledge/health` | Dashboard auth/session | Knowledge Health | `pkg/dashboard/api.go:392` | -| `GET` | `/api/knowledge/stats` | Dashboard auth/session | Knowledge Stats | `pkg/dashboard/api.go:393` | -| `GET` | `/api/knowledge/graph` | Dashboard auth/session | Knowledge Graph | `pkg/dashboard/api.go:394` | -| `GET` | `/api/knowledge/fact-history` | Dashboard auth/session | Fact History | `pkg/dashboard/api.go:395` | -| `POST` | `/api/knowledge/create` | Dashboard auth/session | Create one fact (`title`, `body`, `type`, `tags`, `layer`, `confidence`, `state`). `state` is `approved` (default) or `draft`; a draft is written with `state: draft` into the local channel named by `layer` (400 when there is none, 409 when its title already names an entry). A request tagged `cli-memory` must be a draft carrying only `cli-memory`, `agent:`, `cli:`, `memhash:` and `repo:/` tags (400 otherwise) | `pkg/dashboard/api.go:396` | -| `POST` | `/api/knowledge/import` | Dashboard auth/session | Knowledge Import | `pkg/dashboard/api.go:397` | -| `POST` | `/api/knowledge/promote` | Dashboard auth/session | Knowledge Promote | `pkg/dashboard/api.go:403` | -| `GET` | `/api/knowledge/subscriptions` | Dashboard auth/session | Knowledge Subs List | `pkg/dashboard/api.go:404` | -| `POST` | `/api/knowledge/subscriptions` | Dashboard auth/session | Knowledge Subs Add | `pkg/dashboard/api.go:405` | -| `DELETE` | `/api/knowledge/subscriptions` | Dashboard auth/session | Knowledge Subs Remove | `pkg/dashboard/api.go:406` | -| `PUT` | `/api/knowledge/{layer}/{slug}` | Dashboard auth/session | Knowledge Update | `pkg/dashboard/api.go:407` | -| `DELETE` | `/api/knowledge/{layer}/{slug}` | Dashboard auth/session | Knowledge Delete | `pkg/dashboard/api.go:408` | -| `GET` | `/api/knowledge/{layer}` | Dashboard auth/session | Knowledge Layer | `pkg/dashboard/api.go:409` | -| `GET` | `/api/knowledge/{layer}/{slug}` | Dashboard auth/session | Knowledge Fact | `pkg/dashboard/api.go:410` | -| `PUT` | `/api/knowledge/enabled` | Owner only | Knowledge Toggle | `pkg/dashboard/api.go:411` | -| `GET` | `/api/knowledge/bead-synthesizer` | Dashboard auth/session | Bead Synth Status | `pkg/dashboard/api.go:412` | -| `PUT` | `/api/knowledge/bead-synthesizer/enabled` | Owner only | Bead Synth Toggle | `pkg/dashboard/api.go:413` | -| `GET` | `/api/knowledge/vaults` | Dashboard auth/session | Vaults List | `pkg/dashboard/api.go:414` | -| `POST` | `/api/knowledge/vaults` | Owner only | Vaults Connect | `pkg/dashboard/api.go:415` | -| `DELETE` | `/api/knowledge/vaults` | Dashboard auth/session | Vaults Disconnect | `pkg/dashboard/api.go:416` | -| `POST` | `/api/knowledge/vaults/reindex` | Dashboard auth/session | Vaults Reindex | `pkg/dashboard/api.go:417` | -| `GET` | `/api/knowledge/vaults/{name}/facts` | Dashboard auth/session | Vault Facts | `pkg/dashboard/api.go:418` | +| `GET` | `/api/knowledge` | Dashboard auth/session | Paginated knowledge facts: `type` filter, `limit` (default 50, max 200; invalid/non-positive values use the default), `offset` (default 0). Returns `enabled`, `facts`, `count` (this page), `total`, `limit`, `offset`, `has_more`; ordered by layer then slug. Offsets are not snapshot cursors: concurrent edits may shift pages. | `pkg/dashboard/api.go:380` | +| `GET` | `/api/knowledge/export` | Dashboard auth/session | Knowledge Export | `pkg/dashboard/api.go:381` | +| `POST` | `/mcp/knowledge` | Public (read-only) when the dashboard setting or `HIVE_PUBLIC_KNOWLEDGE` is on; otherwise 404 to a signed-in caller and 401 to one who is not | Public Knowledge MCP endpoint — `initialize`, `tools/list`, `tools/call` for `knowledge_search`/`knowledge_toc`/`knowledge_get`/`knowledge_export` ([docs](../public-knowledge-mcp.md)) | `pkg/dashboard/api.go:390` | +| `GET` | `/api/knowledge/search` | Dashboard auth/session | Knowledge Search | `pkg/dashboard/api.go:382` | +| `GET` | `/api/knowledge/toc` | Dashboard auth/session | Scoped knowledge table of contents: capped, body-free entries (`id`, `title`, `type`, `layer`, `repo`, `tags`, `status`, `confidence`, `updated`, `size_bytes`, `source`) plus `total`/`returned`/`truncated`. Query: `layers`, `repos`, `types`, `tags` (comma-separated), `limit` (default 50, max 200), `include_states` (operator opt-in for `draft`/`deprecated`/`superseded`/`all`), `format=prompt` with `max_chars` for a bounded markdown rendering, `agent` to apply that agent's `knowledge.agent_scopes` entry (the other parameters can only narrow it). Approved knowledge only by default ([docs](../knowledge-toc.md)) | `pkg/dashboard/api.go:383` | +| `GET` | `/api/knowledge/effective` | Dashboard auth/session | Effective knowledge for one agent (`agent` required): the agent's `knowledge.agent_scopes` entry (unrestricted when it has none) intersected with the request scope (approved only unless `include_states` is set). Returns `scope` (resolved scope applied, with `configured`), `included` TOC entries, `excluded` entries each with a `reason` (`lifecycle`, `layer`, `repo`, `type` or `tag` — first failing check, in that order), `included_total`/`excluded_total`/`included_returned`/`excluded_returned` and `truncated`; `limit` caps each list (default 50, max 200) ([docs](../knowledge-toc.md#effective-knowledge)) | `pkg/dashboard/api.go:384` | +| `GET` | `/api/knowledge/entry/{id}` | Dashboard auth/session | Full read of one knowledge entry: `markdown` (front-matter plus body) and the fact. Takes the same scope, `include_states` and `agent` query as the table of contents and returns 404 for an entry outside it ([docs](../knowledge-toc.md)) | `pkg/dashboard/api.go:385` | +| `POST` | `/api/knowledge/cli-memory/drain-legacy` | Owner only | One-shot drain of the shared home's legacy Claude memory (`/data/home/.claude/projects/*/memory`) into `state: draft` entries in `bead-synth-wiki`, tagged `cli-memory` and `agent:shared`, deduplicated by content hash. Body `{"truncate": bool}` (default false) empties only files whose every fact is now stored. Returns `store`, `agent`, `sources`, `drained`, `dupes`, `skipped`, `truncate`, `truncated`, `kept`, `errors`; 409 when the vault is not connected. Audited as `knowledge_drain_legacy_memory` ([docs](../knowledge-lifecycle.md#auto-memory-drafts)) | `pkg/dashboard/api.go:387` | +| `GET` | `/api/knowledge/public` | Dashboard auth/session | Public Knowledge MCP sharing status (`enabled`, `tags`, `source`, endpoint URL) | `pkg/dashboard/api.go:391` | +| `PUT` | `/api/knowledge/public` | Owner only | Persist public Knowledge MCP sharing setting; overrides env vars until changed | `pkg/dashboard/api.go:392` | +| `GET` | `/api/knowledge/health` | Dashboard auth/session | Knowledge Health | `pkg/dashboard/api.go:393` | +| `GET` | `/api/knowledge/stats` | Dashboard auth/session | Knowledge Stats | `pkg/dashboard/api.go:394` | +| `GET` | `/api/knowledge/graph` | Dashboard auth/session | Knowledge Graph | `pkg/dashboard/api.go:395` | +| `GET` | `/api/knowledge/fact-history` | Dashboard auth/session | Fact History | `pkg/dashboard/api.go:396` | +| `POST` | `/api/knowledge/create` | Dashboard auth/session | Create one fact (`title`, `body`, `type`, `tags`, `layer`, `confidence`, `state`). `state` is `approved` (default) or `draft`; a draft is written with `state: draft` into the local channel named by `layer` (400 when there is none, 409 when its title already names an entry). A request tagged `cli-memory` must be a draft carrying only `cli-memory`, `agent:`, `cli:`, `memhash:` and `repo:/` tags (400 otherwise) | `pkg/dashboard/api.go:397` | +| `POST` | `/api/knowledge/import` | Dashboard auth/session | Knowledge Import | `pkg/dashboard/api.go:398` | +| `POST` | `/api/knowledge/promote` | Dashboard auth/session | Knowledge Promote | `pkg/dashboard/api.go:404` | +| `GET` | `/api/knowledge/subscriptions` | Dashboard auth/session | Knowledge Subs List | `pkg/dashboard/api.go:405` | +| `POST` | `/api/knowledge/subscriptions` | Dashboard auth/session | Knowledge Subs Add | `pkg/dashboard/api.go:406` | +| `DELETE` | `/api/knowledge/subscriptions` | Dashboard auth/session | Knowledge Subs Remove | `pkg/dashboard/api.go:407` | +| `PUT` | `/api/knowledge/{layer}/{slug}` | Dashboard auth/session | Knowledge Update | `pkg/dashboard/api.go:408` | +| `DELETE` | `/api/knowledge/{layer}/{slug}` | Dashboard auth/session | Knowledge Delete | `pkg/dashboard/api.go:409` | +| `GET` | `/api/knowledge/{layer}` | Dashboard auth/session | Knowledge Layer | `pkg/dashboard/api.go:410` | +| `GET` | `/api/knowledge/{layer}/{slug}` | Dashboard auth/session | Knowledge Fact | `pkg/dashboard/api.go:411` | +| `PUT` | `/api/knowledge/enabled` | Owner only | Knowledge Toggle | `pkg/dashboard/api.go:412` | +| `GET` | `/api/knowledge/bead-synthesizer` | Dashboard auth/session | Bead Synth Status | `pkg/dashboard/api.go:413` | +| `PUT` | `/api/knowledge/bead-synthesizer/enabled` | Owner only | Bead Synth Toggle | `pkg/dashboard/api.go:414` | +| `GET` | `/api/knowledge/vaults` | Dashboard auth/session | Vaults List | `pkg/dashboard/api.go:415` | +| `POST` | `/api/knowledge/vaults` | Owner only | Vaults Connect | `pkg/dashboard/api.go:416` | +| `DELETE` | `/api/knowledge/vaults` | Dashboard auth/session | Vaults Disconnect | `pkg/dashboard/api.go:417` | +| `POST` | `/api/knowledge/vaults/reindex` | Dashboard auth/session | Vaults Reindex | `pkg/dashboard/api.go:418` | +| `GET` | `/api/knowledge/vaults/{name}/facts` | Dashboard auth/session | Vault Facts | `pkg/dashboard/api.go:419` | | `GET` | `/api/config/knowledge/connectors` | Dashboard auth/session | Knowledge connectors config (name, type, layer, scope, env/file auth reference) and registered types | `pkg/dashboard/api.go` | | `PUT` | `/api/config/knowledge/connectors` | Owner only | Replace `knowledge.connectors` after validation; inline secrets rejected | `pkg/dashboard/api.go` | | `GET` | `/api/config/knowledge/connectors/status` | Dashboard auth/session | Per-connector status pill, last sync, pages/facts, last error, plus the `publish` mirror status | `pkg/dashboard/api.go` | @@ -46,19 +46,19 @@ Part of the [Dashboard REST API reference](../api-reference.md); see that page f | `PUT` | `/api/config/knowledge/agent-scopes` | Owner only | Replace `knowledge.agent_scopes` after validating layer and state names; persisted and audited as `config_knowledge_agent_scopes` | `pkg/dashboard/api.go` | | `GET` | `/api/config/knowledge/pipeline` | Owner only | Scalar bead synthesizer, curator, code map and primer settings, plus the read-only engine, vault paths and primer merge strategy ([docs](../settings.md#knowledge)) | `pkg/dashboard/api.go` | | `PUT` | `/api/config/knowledge/pipeline` | Owner only | Update those settings after `config.ValidateKnowledgePipeline`; 0 or empty restores the default; the bead synthesizer switch applies at once, the rest after a restart; audited as `config_knowledge_pipeline` | `pkg/dashboard/api.go` | -| `GET` | `/api/knowledge/git-sources` | Dashboard auth/session | Git Sources List | `pkg/dashboard/api.go:428` | -| `POST` | `/api/knowledge/git-sources` | Owner only | Git Sources Connect | `pkg/dashboard/api.go:429` | -| `DELETE` | `/api/knowledge/git-sources` | Owner only | Git Sources Disconnect | `pkg/dashboard/api.go:430` | -| `POST` | `/api/knowledge/obsidian/sync` | Dashboard auth/session | Obsidian Sync | `pkg/dashboard/api.go:431` | -| `GET` | `/api/knowledge/documents` | Dashboard auth/session | Documents List | `pkg/dashboard/api.go:432` | -| `POST` | `/api/knowledge/documents` | Dashboard auth/session | Documents Import | `pkg/dashboard/api.go:433` | -| `GET` | `/api/knowledge/documents/{slug}` | Dashboard auth/session | Document Get | `pkg/dashboard/api.go:434` | -| `DELETE` | `/api/knowledge/documents/{slug}` | Dashboard auth/session | Document Delete | `pkg/dashboard/api.go:435` | -| `POST` | `/api/knowledge/documents/{slug}/reimport` | Dashboard auth/session | Document Reimport | `pkg/dashboard/api.go:436` | -| `GET` | `/api/knowledge/context7/search` | Dashboard auth/session | Context7 Search | `pkg/dashboard/api.go:437` | -| `POST` | `/api/knowledge/cleanup-orphans` | Dashboard auth/session | Cleanup Orphans | `pkg/dashboard/api.go:438` | -| `GET` | `/api/knowledge/channels` | Dashboard auth/session | Knowledge Channels List | `pkg/dashboard/api.go:401` | -| `POST` | `/api/knowledge/channels` | Dashboard auth/session | Knowledge Channel Create | `pkg/dashboard/api.go:402` | +| `GET` | `/api/knowledge/git-sources` | Dashboard auth/session | Git Sources List | `pkg/dashboard/api.go:429` | +| `POST` | `/api/knowledge/git-sources` | Owner only | Git Sources Connect | `pkg/dashboard/api.go:430` | +| `DELETE` | `/api/knowledge/git-sources` | Owner only | Git Sources Disconnect | `pkg/dashboard/api.go:431` | +| `POST` | `/api/knowledge/obsidian/sync` | Dashboard auth/session | Obsidian Sync | `pkg/dashboard/api.go:432` | +| `GET` | `/api/knowledge/documents` | Dashboard auth/session | Documents List | `pkg/dashboard/api.go:433` | +| `POST` | `/api/knowledge/documents` | Dashboard auth/session | Documents Import | `pkg/dashboard/api.go:434` | +| `GET` | `/api/knowledge/documents/{slug}` | Dashboard auth/session | Document Get | `pkg/dashboard/api.go:435` | +| `DELETE` | `/api/knowledge/documents/{slug}` | Dashboard auth/session | Document Delete | `pkg/dashboard/api.go:436` | +| `POST` | `/api/knowledge/documents/{slug}/reimport` | Dashboard auth/session | Document Reimport | `pkg/dashboard/api.go:437` | +| `GET` | `/api/knowledge/context7/search` | Dashboard auth/session | Context7 Search | `pkg/dashboard/api.go:438` | +| `POST` | `/api/knowledge/cleanup-orphans` | Dashboard auth/session | Cleanup Orphans | `pkg/dashboard/api.go:439` | +| `GET` | `/api/knowledge/channels` | Dashboard auth/session | Knowledge Channels List | `pkg/dashboard/api.go:402` | +| `POST` | `/api/knowledge/channels` | Dashboard auth/session | Knowledge Channel Create | `pkg/dashboard/api.go:403` | | `GET` | `/api/knowledge/connectors` | Dashboard auth/session | Knowledge connector statuses and publish mirror status (`{"connectors": [...], "publish": {...}\|null}`, see [knowledge-connectors.md](../knowledge-connectors.md)) | `pkg/dashboard/api_knowledge_connectors.go:23` | | `POST` | `/api/knowledge/connectors/{name}/sync` | Owner only | Start one connector sync now (202; 404 unknown, 409 running) | `pkg/dashboard/api_knowledge_connectors.go:24` | | `POST` | `/api/knowledge/publish/sync` | Owner only | Run a publish mirror batch now (202 with publish status; 503 mirror not running, 409 batch in progress) | `pkg/dashboard/api_knowledge_connectors.go:25` | diff --git a/src/docs/api-reference/nous.md b/src/docs/api-reference/nous.md index c0249cf03f..6dce3749dd 100644 --- a/src/docs/api-reference/nous.md +++ b/src/docs/api-reference/nous.md @@ -5,28 +5,28 @@ Part of the [Dashboard REST API reference](../api-reference.md); see that page f | Method | Path | Auth | Purpose | Source | |---|---|---|---|---| -| `GET` | `/api/nous/status` | Dashboard auth/session | Nous Status — includes `experimentLoop` (`dir`, `lastWriteAt`, `running`, `idleAfterSeconds`): whether the external experiment loop, the only producer of proposals, has written under `/var/run/nous/governor` within 24 evaluation cycles; ready cards then carry `experimentLoopIdle` ([#12541](https://github.com/hivecommons/hive/issues/12541)) | `pkg/dashboard/api.go:472` | -| `GET` | `/api/nous/ledger` | Dashboard auth/session | Nous Ledger | `pkg/dashboard/api.go:473` | -| `GET` | `/api/nous/principles` | Dashboard auth/session | Nous Principles | `pkg/dashboard/api.go:474` | -| `POST` | `/api/nous/approve` | Owner only | Nous Approve | `pkg/dashboard/api.go:475` | -| `POST` | `/api/nous/abort` | Owner only | Nous Abort | `pkg/dashboard/api.go:476` | -| `PUT` | `/api/nous/mode` | Owner only | Nous Mode | `pkg/dashboard/api.go:477` | -| `PUT` | `/api/nous/scope` | Owner only | Nous Scope | `pkg/dashboard/api.go:478` | -| `GET` | `/api/nous/phase` | Dashboard auth/session | Nous Phase | `pkg/dashboard/api.go:479` | -| `PUT` | `/api/nous/gate-decision` | Owner only | Nous Gate Decision | `pkg/dashboard/api.go:480` | -| `GET` | `/api/nous/gate-pending` | Dashboard auth/session | Nous Gate Pending | `pkg/dashboard/api.go:481` | -| `POST` | `/api/nous/gate-respond` | Owner only | Nous Gate Respond | `pkg/dashboard/api.go:482` | -| `GET` | `/api/nous/gate-response` | Dashboard auth/session | Nous Gate Response | `pkg/dashboard/api.go:483` | -| `GET` | `/api/nous/config` | Dashboard auth/session | Nous Config Get | `pkg/dashboard/api.go:484` | -| `PUT` | `/api/nous/config/goals` | Owner only | Nous Config Goals — legacy; the dashboard no longer exposes the research question in the ⚙ dialog, use `PATCH /api/nous/campaigns/{id}` | `pkg/dashboard/api.go:485` | -| `PUT` | `/api/nous/config/repos` | Owner only | Nous Config Repos — the allow-list of repos experiments may target (empty = any) | `pkg/dashboard/api.go:486` | -| `PUT` | `/api/nous/config/output` | Owner only | Nous Config Output | `pkg/dashboard/api.go:487` | -| `PUT` | `/api/nous/config/fast-fail` | Owner only | Nous Config Fast Fail | `pkg/dashboard/api.go:488` | -| `PUT` | `/api/nous/config/schedule` | Owner only | Nous Config Schedule | `pkg/dashboard/api.go:489` | -| `PUT` | `/api/nous/config/controllables` | Owner only | Nous Config Controllables | `pkg/dashboard/api.go:490` | -| `PUT` | `/api/nous/config/principles` | Owner only | Nous Config Principles | `pkg/dashboard/api.go:491` | -| `DELETE` | `/api/nous/principles/{id}` | Owner only | Nous Delete Principle | `pkg/dashboard/api.go:492` | -| `POST` | `/api/nous/campaign/start` | Owner only | Nous Campaign Start — archives the active campaign and starts a new experiment whose baseline restarts at 0 ([#12501](https://github.com/hivecommons/hive/issues/12501)) | `pkg/dashboard/api.go:493` | -| `GET` | `/api/nous/campaigns` | Any | Nous Campaigns List — every experiment with its per-campaign baseline, scope, results and stage ([#12532](https://github.com/hivecommons/hive/issues/12532)) | `pkg/dashboard/api.go:494` | -| `POST` | `/api/nous/campaigns` | Owner only | Nous Campaigns Create — starts an experiment alongside the running ones; 409 `scope_conflict` when a running experiment shares a repo target or both are hive-wide ([#12532](https://github.com/hivecommons/hive/issues/12532)) | `pkg/dashboard/api.go:495` | -| `PATCH` | `/api/nous/campaigns/{id}` | Owner only | Nous Campaign Patch — pause / resume / archive, or edit question, hypothesis and scope. Hypothesis/tag edits save in place (`status: "updated"`); a changed question or repo set needs `restart: true` (else 409 `restart_required`) and archives the campaign as version N while creating version N+1 with `restartedFrom` (`status: "restarted", campaign, archived`); resume, scope widening and restarts re-run the conflict guard ([#12532](https://github.com/hivecommons/hive/issues/12532)) | `pkg/dashboard/api.go:496` | +| `GET` | `/api/nous/status` | Dashboard auth/session | Nous Status — includes `experimentLoop` (`dir`, `lastWriteAt`, `running`, `idleAfterSeconds`): whether the external experiment loop, the only producer of proposals, has written under `/var/run/nous/governor` within 24 evaluation cycles; ready cards then carry `experimentLoopIdle` ([#12541](https://github.com/hivecommons/hive/issues/12541)) | `pkg/dashboard/api.go:473` | +| `GET` | `/api/nous/ledger` | Dashboard auth/session | Nous Ledger | `pkg/dashboard/api.go:474` | +| `GET` | `/api/nous/principles` | Dashboard auth/session | Nous Principles | `pkg/dashboard/api.go:475` | +| `POST` | `/api/nous/approve` | Owner only | Nous Approve | `pkg/dashboard/api.go:476` | +| `POST` | `/api/nous/abort` | Owner only | Nous Abort | `pkg/dashboard/api.go:477` | +| `PUT` | `/api/nous/mode` | Owner only | Nous Mode | `pkg/dashboard/api.go:478` | +| `PUT` | `/api/nous/scope` | Owner only | Nous Scope | `pkg/dashboard/api.go:479` | +| `GET` | `/api/nous/phase` | Dashboard auth/session | Nous Phase | `pkg/dashboard/api.go:480` | +| `PUT` | `/api/nous/gate-decision` | Owner only | Nous Gate Decision | `pkg/dashboard/api.go:481` | +| `GET` | `/api/nous/gate-pending` | Dashboard auth/session | Nous Gate Pending | `pkg/dashboard/api.go:482` | +| `POST` | `/api/nous/gate-respond` | Owner only | Nous Gate Respond | `pkg/dashboard/api.go:483` | +| `GET` | `/api/nous/gate-response` | Dashboard auth/session | Nous Gate Response | `pkg/dashboard/api.go:484` | +| `GET` | `/api/nous/config` | Dashboard auth/session | Nous Config Get | `pkg/dashboard/api.go:485` | +| `PUT` | `/api/nous/config/goals` | Owner only | Nous Config Goals — legacy; the dashboard no longer exposes the research question in the ⚙ dialog, use `PATCH /api/nous/campaigns/{id}` | `pkg/dashboard/api.go:486` | +| `PUT` | `/api/nous/config/repos` | Owner only | Nous Config Repos — the allow-list of repos experiments may target (empty = any) | `pkg/dashboard/api.go:487` | +| `PUT` | `/api/nous/config/output` | Owner only | Nous Config Output | `pkg/dashboard/api.go:488` | +| `PUT` | `/api/nous/config/fast-fail` | Owner only | Nous Config Fast Fail | `pkg/dashboard/api.go:489` | +| `PUT` | `/api/nous/config/schedule` | Owner only | Nous Config Schedule | `pkg/dashboard/api.go:490` | +| `PUT` | `/api/nous/config/controllables` | Owner only | Nous Config Controllables | `pkg/dashboard/api.go:491` | +| `PUT` | `/api/nous/config/principles` | Owner only | Nous Config Principles | `pkg/dashboard/api.go:492` | +| `DELETE` | `/api/nous/principles/{id}` | Owner only | Nous Delete Principle | `pkg/dashboard/api.go:493` | +| `POST` | `/api/nous/campaign/start` | Owner only | Nous Campaign Start — archives the active campaign and starts a new experiment whose baseline restarts at 0 ([#12501](https://github.com/hivecommons/hive/issues/12501)) | `pkg/dashboard/api.go:494` | +| `GET` | `/api/nous/campaigns` | Any | Nous Campaigns List — every experiment with its per-campaign baseline, scope, results and stage ([#12532](https://github.com/hivecommons/hive/issues/12532)) | `pkg/dashboard/api.go:495` | +| `POST` | `/api/nous/campaigns` | Owner only | Nous Campaigns Create — starts an experiment alongside the running ones; 409 `scope_conflict` when a running experiment shares a repo target or both are hive-wide ([#12532](https://github.com/hivecommons/hive/issues/12532)) | `pkg/dashboard/api.go:496` | +| `PATCH` | `/api/nous/campaigns/{id}` | Owner only | Nous Campaign Patch — pause / resume / archive, or edit question, hypothesis and scope. Hypothesis/tag edits save in place (`status: "updated"`); a changed question or repo set needs `restart: true` (else 409 `restart_required`) and archives the campaign as version N while creating version N+1 with `restartedFrom` (`status: "restarted", campaign, archived`); resume, scope widening and restarts re-run the conflict guard ([#12532](https://github.com/hivecommons/hive/issues/12532)) | `pkg/dashboard/api.go:497` | diff --git a/src/docs/api-reference/packs-and-acmm.md b/src/docs/api-reference/packs-and-acmm.md index 8ecae2c22b..0c15c8d508 100644 --- a/src/docs/api-reference/packs-and-acmm.md +++ b/src/docs/api-reference/packs-and-acmm.md @@ -5,29 +5,29 @@ Part of the [Dashboard REST API reference](../api-reference.md); see that page f | Method | Path | Auth | Purpose | Source | |---|---|---|---|---| -| `GET` | `/api/packs` | Dashboard auth/session | Packs List | `pkg/dashboard/api.go:324` | -| `POST` | `/api/packs/{level}/apply` | Owner only | Pack Apply | `pkg/dashboard/api.go:325` | -| `PUT` | `/api/packs/level` | Owner only | Set the ACMM level. Body `{"level": N}` persists and applies the pack; optional `{"release_level_holds": true}` performs a deliberate one-off release of Hive App level-applied `hold` labels for the transition. Level holds are otherwise never auto-released. When the change makes the L6 self-authored merge sweep eligible, the success body adds `self_merge_sweep_active`, `level_changed_at`, `level_holds_pending: [{repo, number, title, url, agent}]`, and `repos: [{repo, auto_merge}]` so the dashboard can show the owner an informational notice and repo toggles. | `pkg/dashboard/api.go:326` | +| `GET` | `/api/packs` | Dashboard auth/session | Packs List | `pkg/dashboard/api.go:325` | +| `POST` | `/api/packs/{level}/apply` | Owner only | Pack Apply | `pkg/dashboard/api.go:326` | +| `PUT` | `/api/packs/level` | Owner only | Set the ACMM level. Body `{"level": N}` persists and applies the pack; optional `{"release_level_holds": true}` performs a deliberate one-off release of Hive App level-applied `hold` labels for the transition. Level holds are otherwise never auto-released. When the change makes the L6 self-authored merge sweep eligible, the success body adds `self_merge_sweep_active`, `level_changed_at`, `level_holds_pending: [{repo, number, title, url, agent}]`, and `repos: [{repo, auto_merge}]` so the dashboard can show the owner an informational notice and repo toggles. | `pkg/dashboard/api.go:327` | | `POST` | `/api/repos/rescan` | Dashboard auth/session | Re-enumerate watched repositories' open issues/PRs for the dashboard Projects cards. The scan is read-only with respect to agents/governor actions, collapses concurrent requests, and debounces repeated presses for 30 seconds (`repoRescanDebounce`) so the button cannot hammer GitHub | `pkg/dashboard/api.go`, `pkg/dashboard/api_repos_rescan.go` | | `POST` | `/api/repos/pause` | Owner | Quiet ONE repository without stopping the hive: agents stop writing to it and stop being handed work on it, while it keeps its dashboard card and ACMM eval. Body `{"repo": "...", "reason": "..."}` — the repo travels in the body because a `project.repos` entry may be an explicit cross-org `owner/name`. Records who/when/why; `changed: false` marks a no-op re-pause and leaves the original provenance intact; `persisted: false` means the pause is in force but could not be written to config. Rejects a repo outside `project.repos`. See [per-repo agent pause](../repo-pause.md) | `pkg/dashboard/api_repo_pause.go` | | `POST` | `/api/repos/resume` | Owner | Lift a repository's pause. Unlike pause, does not require the repo to be in `project.repos`, so a pause left behind by a removed repo can still be cleared | `pkg/dashboard/api_repo_pause.go` | | `GET` | `/api/repos/pauses` | Dashboard auth/session | Every repo pause with its provenance. Read-only — the same state is already on the repository cards | `pkg/dashboard/api_repo_pause.go` | -| `GET` | `/api/repos/{owner}/{repo}/hold-permission` | Dashboard auth/session | Reports whether the signed-in user can toggle hold chips for a repository card. Owners are allowed; otherwise the server checks GitHub `repos/{owner}/{repo}/collaborators/{user}/permission` and briefly caches the effective permission | `pkg/dashboard/api.go:341`, `pkg/dashboard/api_repo_hold.go` | +| `GET` | `/api/repos/{owner}/{repo}/hold-permission` | Dashboard auth/session | Reports whether the signed-in user can toggle hold chips for a repository card. Owners are allowed; otherwise the server checks GitHub `repos/{owner}/{repo}/collaborators/{user}/permission` and briefly caches the effective permission | `pkg/dashboard/api.go:342`, `pkg/dashboard/api_repo_hold.go` | | `GET` | `/api/repos/{owner}/{repo}/items/{number}` | Dashboard session | Per-item detail for the activity monitor's decision badge: `{repo, number, stage, stage_slug, operator_actions[], operator_action_count, last_operator_action}`. `operator_actions[]` is the server-side ledger of every row action submitted from the dashboard (hold/release, decision, decide-delegated, send-to-human, accept-triage — recorded by `recordOperatorAction` around those routes) as `{action, actor, at, note, target, outcome: pending|applied|no-op|failed, detail, head_sha, stage}`; `last_operator_action` adds the derived `state` (`waiting|addressed|stuck|pending|failed`), `stuck_after_seconds` and `followed[]` (the stage hops after the action). The same `last_operator_action` / `operator_action_count` ride every stage-tracker row and linked PR. Persisted in `/data/operator-actions.json`; `HIVE_OPERATOR_ACTION_STUCK_AFTER` (default `30m`) sets the stuck window. Rows also carry `rebase_merge: {state: approved|rebasing|ci-pending|merged|failed, actor, base, text, fail_reason, conflict_files[], armed_at}` while a *Rebase, then merge as maintainer* follow-through is live or just settled (persisted in `/data/rebase-merge-followups.json`; `fail_reason: foreign-head` when the head moved after the approval by someone other than the hive, so nothing merged), and `rebase: {attempt, max, exhausted, head, text}` on a conflicting PR the fixer is rebasing under the two-attempt loop guard (`/data/rebase-attempts.json`). | `pkg/dashboard/api.go`, `pkg/dashboard/operator_actions.go` | -| `POST` | `/api/repos/{owner}/{repo}/items/{number}/hold` | Owner or repo write/maintain/admin | Add or remove a dashboard hold chip for an issue or PR. Body `{"held": true}` adds the hive's exact-match canonical `hive-pause/` label (shortened with a hash suffix when the plain spelling would exceed GitHub's 50-character label cap, [#17037](https://github.com/hivecommons/hive/issues/17037)); `{"held": false}` removes exactly the labels causing the hold (`hive-pause/` and/or generic hold labels) and never removes `hive/` provenance. A GitHub failure answers `424` with the step and GitHub's reason — never `502`/`503`, which the hosted Ingress replaces with its generic "hive unavailable" page. Every toggle writes an audit-log entry; the #8851 upgrade migration writes `/data/hive-hold-migration-.json` with audit-backed, provenance-only, and ambiguous legacy `hive/` classifications. | `pkg/dashboard/api.go:345`, `pkg/dashboard/api_repo_hold.go` | -| `POST` | `/api/repos/{owner}/{repo}/items/{number}/decision` | Owner or repo write/maintain/admin, signed-in GitHub user | Record an operator decision from the activity monitor's row menu / selection bar. Body `{"decision": "approve"|"direction"|"close-superseded"|"close-wontfix"|"close-read"|"close-issue"|"adopt"|"reroute"|"re-author"|"merge-now"|"queue-merge", "comment": "…", "type": "issue"|"pr", "superseded_by": "#458", "force": true, "to": ""|"any"}` (`force` only with `adopt`, `to` only with `reroute`; an unknown `decision` is a 400). On an issue parked at 045, `approve`/`direction` act exactly as a maintainer's `/hive approve` / `/hive decision ` comment would (labels, `approved-direction`, stage-engine Unparked fact, reply comment naming the operator). On a PR parked at 085 (`type: "pr"`), `approve` posts the `/lgtm` and `/approve` commands as two comments attributed to the operator, records the approval against the head as the human answer to a `requires_human` / decision-requested verdict (the lanes and the stage engine honour it like a trusted merger's Approve review), lifts `needs-human`, clears the sentinel label and acknowledges the bundle's sentinel findings for that head, re-derives the stage and drops the tracker cache — the response adds `released` (`approval-recorded`, `needs-human`, `sentinel-label`, `sentinel-acknowledged`) and `stage`; the merge gate decides from there (no review as the operator, no merge) — and `direction` records the text as a human change request against the head — the `/hive decision` fix lane — on a hive-authored PR, and on a contributor's clanker PR (`author/clanker` / hive attribution markers) where the text is posted to the contributor's hive and the PR moves to `stage/072-returned-to-contributor` (`status: "direction-returned-to-contributor"`); a person's PR, which no hive can carry direction to, answers 409. The close variants first post a required comment naming the operator (and the superseding PR when `superseded_by` is given); a failed comment prevents closure. `close-superseded` and `close-wontfix` close as **not planned**; `close-read` acknowledges an advisory issue and closes it **as completed**. `close-issue` (`type: "issue"` only) closes the single named open issue **as completed**, at any stage, leaving linked PRs and other issues untouched; it returns 400 for a PR (including one sent as an issue), 404 when the issue is not found, and 409 when already closed. It retains the existing permission and mutation gates, including the paused-repo `status: "refused"` result without writes. `adopt` and `re-author` require `type: "pr"` and queue adoption via the `adoption/supersede` label (409 unless `governor.labels.stage.adoption` is enabled). `adopt` with `"force": true` is *Adopt now* ([#15038](https://github.com/hivecommons/hive/issues/15038)): it bypasses the author's presence, the grace window and `maintainer_policy`, marks the stage record and nudges the automerge sweep so the adoption engine supersedes the head at once (responds `status: "already-adopted"` when nothing is left to do; 400 on the hive's own PR). `reroute` (`type: "pr"`, `"to": ""|"any"`) hands the PR's follow-up work to another online clanker through the contributor fix queue, posts a `` note and arms a pickup timer that falls back to Adopt now after `governor.reroute.pickup_timeout`; answers `status: "rerouted"`, 409 when no clanker is online or adoption is disabled. On a PR at 090, `queue-merge` (*Approve and let hive merge*, the lead action) records an approval attributed to the operator and applies the automerge label, returning `status: "queued-for-merge"` (on the hive's own PR, which the App cannot review, the queue note is a comment and the operator's approval is recorded as `approve` records it — `released` lists `approval-recorded`, `admitted`); hive's merge sweep merges on its next tick once CI is green and the head has soaked, and that recorded operator approval is the one thing that lets the sweep merge an untrusted-author PR (the merge comment names the operator). `merge-now` (*Merge now as maintainer*, secondary) posts an attributed comment and merges immediately through the hive's GitHub App, returning `status: "merged"`. Both require `type: "pr"` and a verified owner or `merger` role in addition to the route's repo permission check; both refuse closed/merged PRs and the operator's own PR. `merge-now` on a PR whose `mergeable_state` is `dirty`/`behind` instead answers `202` with `status: "rebase-queued"` (approval recorded, rebase kicked, merge follows via the follow-through once the hive's own rebase is green; a head pushed by anyone else fails the follow-through closed; `head-not-pushable` refusal on a fork without maintainer edits). `approve` on such a PR adds `rebase: {attempt, max, exhausted, head, text}` for the rebase episode it opened. Every decision writes an audit-log entry (`repo_item_decision_*`, e.g. `repo_item_decision_adopt_now`, `repo_item_decision_reroute`). | `pkg/dashboard/api.go:346`, `pkg/dashboard/api_repo_item_decision.go` | -| `GET` | `/api/notes` | Signed-in GitHub user (or verified owner) | Maintainer notes — the private per-item notes maintainers leave each other on Activity Monitor rows ([#maintainer-notes](../operator-reference.md#maintainer-notes)). `?repo=owner/name&number=N` lists one item's live notes oldest first; with no query, every live note in the hive (the SPA's count badges). `{notes: [{id, repo, number, author, text, created_at}]}`. Never read from or written to GitHub. | `pkg/dashboard/api.go:350`, `pkg/dashboard/api_maintainer_notes.go` | -| `POST` | `/api/notes` | Owner or repo write/maintain/admin, signed-in GitHub user | Append a maintainer note. Body `{repo, number, text}` — `text` is plain text up to 2000 characters; the author is the dashboard session's GitHub login, never a body field. `401` signed out, `403` without write on the repo, `400` for an empty/oversized note or a bad item. `201 {ok, note}`; appended to `/data/maintainer-notes.jsonl`, audited as `maintainer_note_add`, and broadcast as the `maintainer-notes` SSE event `{seq, repo, number, count, action}` so other maintainers' browsers refresh the row. | `pkg/dashboard/api.go:351`, `pkg/dashboard/api_maintainer_notes.go` | -| `DELETE` | `/api/notes/{id}` | The note's author, or a verified owner | Tombstone a maintainer note (append-only store: a `deleted_at` record, never a rewrite). `404` unknown or already deleted, `403` for anyone else. `{ok, note}`; audited as `maintainer_note_delete`, broadcast as `maintainer-notes`. | `pkg/dashboard/api.go:352`, `pkg/dashboard/api_maintainer_notes.go` | +| `POST` | `/api/repos/{owner}/{repo}/items/{number}/hold` | Owner or repo write/maintain/admin | Add or remove a dashboard hold chip for an issue or PR. Body `{"held": true}` adds the hive's exact-match canonical `hive-pause/` label (shortened with a hash suffix when the plain spelling would exceed GitHub's 50-character label cap, [#17037](https://github.com/hivecommons/hive/issues/17037)); `{"held": false}` removes exactly the labels causing the hold (`hive-pause/` and/or generic hold labels) and never removes `hive/` provenance. A GitHub failure answers `424` with the step and GitHub's reason — never `502`/`503`, which the hosted Ingress replaces with its generic "hive unavailable" page. Every toggle writes an audit-log entry; the #8851 upgrade migration writes `/data/hive-hold-migration-.json` with audit-backed, provenance-only, and ambiguous legacy `hive/` classifications. | `pkg/dashboard/api.go:346`, `pkg/dashboard/api_repo_hold.go` | +| `POST` | `/api/repos/{owner}/{repo}/items/{number}/decision` | Owner or repo write/maintain/admin, signed-in GitHub user | Record an operator decision from the activity monitor's row menu / selection bar. Body `{"decision": "approve"|"direction"|"close-superseded"|"close-wontfix"|"close-read"|"close-issue"|"adopt"|"reroute"|"re-author"|"merge-now"|"queue-merge", "comment": "…", "type": "issue"|"pr", "superseded_by": "#458", "force": true, "to": ""|"any"}` (`force` only with `adopt`, `to` only with `reroute`; an unknown `decision` is a 400). On an issue parked at 045, `approve`/`direction` act exactly as a maintainer's `/hive approve` / `/hive decision ` comment would (labels, `approved-direction`, stage-engine Unparked fact, reply comment naming the operator). On a PR parked at 085 (`type: "pr"`), `approve` posts the `/lgtm` and `/approve` commands as two comments attributed to the operator, records the approval against the head as the human answer to a `requires_human` / decision-requested verdict (the lanes and the stage engine honour it like a trusted merger's Approve review), lifts `needs-human`, clears the sentinel label and acknowledges the bundle's sentinel findings for that head, re-derives the stage and drops the tracker cache — the response adds `released` (`approval-recorded`, `needs-human`, `sentinel-label`, `sentinel-acknowledged`) and `stage`; the merge gate decides from there (no review as the operator, no merge) — and `direction` records the text as a human change request against the head — the `/hive decision` fix lane — on a hive-authored PR, and on a contributor's clanker PR (`author/clanker` / hive attribution markers) where the text is posted to the contributor's hive and the PR moves to `stage/072-returned-to-contributor` (`status: "direction-returned-to-contributor"`); a person's PR, which no hive can carry direction to, answers 409. The close variants first post a required comment naming the operator (and the superseding PR when `superseded_by` is given); a failed comment prevents closure. `close-superseded` and `close-wontfix` close as **not planned**; `close-read` acknowledges an advisory issue and closes it **as completed**. `close-issue` (`type: "issue"` only) closes the single named open issue **as completed**, at any stage, leaving linked PRs and other issues untouched; it returns 400 for a PR (including one sent as an issue), 404 when the issue is not found, and 409 when already closed. It retains the existing permission and mutation gates, including the paused-repo `status: "refused"` result without writes. `adopt` and `re-author` require `type: "pr"` and queue adoption via the `adoption/supersede` label (409 unless `governor.labels.stage.adoption` is enabled). `adopt` with `"force": true` is *Adopt now* ([#15038](https://github.com/hivecommons/hive/issues/15038)): it bypasses the author's presence, the grace window and `maintainer_policy`, marks the stage record and nudges the automerge sweep so the adoption engine supersedes the head at once (responds `status: "already-adopted"` when nothing is left to do; 400 on the hive's own PR). `reroute` (`type: "pr"`, `"to": ""|"any"`) hands the PR's follow-up work to another online clanker through the contributor fix queue, posts a `` note and arms a pickup timer that falls back to Adopt now after `governor.reroute.pickup_timeout`; answers `status: "rerouted"`, 409 when no clanker is online or adoption is disabled. On a PR at 090, `queue-merge` (*Approve and let hive merge*, the lead action) records an approval attributed to the operator and applies the automerge label, returning `status: "queued-for-merge"` (on the hive's own PR, which the App cannot review, the queue note is a comment and the operator's approval is recorded as `approve` records it — `released` lists `approval-recorded`, `admitted`); hive's merge sweep merges on its next tick once CI is green and the head has soaked, and that recorded operator approval is the one thing that lets the sweep merge an untrusted-author PR (the merge comment names the operator). `merge-now` (*Merge now as maintainer*, secondary) posts an attributed comment and merges immediately through the hive's GitHub App, returning `status: "merged"`. Both require `type: "pr"` and a verified owner or `merger` role in addition to the route's repo permission check; both refuse closed/merged PRs and the operator's own PR. `merge-now` on a PR whose `mergeable_state` is `dirty`/`behind` instead answers `202` with `status: "rebase-queued"` (approval recorded, rebase kicked, merge follows via the follow-through once the hive's own rebase is green; a head pushed by anyone else fails the follow-through closed; `head-not-pushable` refusal on a fork without maintainer edits). `approve` on such a PR adds `rebase: {attempt, max, exhausted, head, text}` for the rebase episode it opened. Every decision writes an audit-log entry (`repo_item_decision_*`, e.g. `repo_item_decision_adopt_now`, `repo_item_decision_reroute`). | `pkg/dashboard/api.go:347`, `pkg/dashboard/api_repo_item_decision.go` | +| `GET` | `/api/notes` | Signed-in GitHub user (or verified owner) | Maintainer notes — the private per-item notes maintainers leave each other on Activity Monitor rows ([#maintainer-notes](../operator-reference.md#maintainer-notes)). `?repo=owner/name&number=N` lists one item's live notes oldest first; with no query, every live note in the hive (the SPA's count badges). `{notes: [{id, repo, number, author, text, created_at}]}`. Never read from or written to GitHub. | `pkg/dashboard/api.go:351`, `pkg/dashboard/api_maintainer_notes.go` | +| `POST` | `/api/notes` | Owner or repo write/maintain/admin, signed-in GitHub user | Append a maintainer note. Body `{repo, number, text}` — `text` is plain text up to 2000 characters; the author is the dashboard session's GitHub login, never a body field. `401` signed out, `403` without write on the repo, `400` for an empty/oversized note or a bad item. `201 {ok, note}`; appended to `/data/maintainer-notes.jsonl`, audited as `maintainer_note_add`, and broadcast as the `maintainer-notes` SSE event `{seq, repo, number, count, action}` so other maintainers' browsers refresh the row. | `pkg/dashboard/api.go:352`, `pkg/dashboard/api_maintainer_notes.go` | +| `DELETE` | `/api/notes/{id}` | The note's author, or a verified owner | Tombstone a maintainer note (append-only store: a `deleted_at` record, never a rewrite). `404` unknown or already deleted, `403` for anyone else. `{ok, note}`; audited as `maintainer_note_delete`, broadcast as `maintainer-notes`. | `pkg/dashboard/api.go:353`, `pkg/dashboard/api_maintainer_notes.go` | | `POST` / `GET` | `/api/repos/{owner}/{repo}/items/{number}/decide-delegated` | Same as `…/hold` (owner or repo write), signed-in GitHub user; `decisions.delegate.enabled` (403 when off) | *Let hive decide*: enqueue a delegated-decision job in which the hive decides the parked item itself as the owner's delegate under `pkg/delegatedecision.PolicyText` and records it as a human decision (`/hive decision` comment with the `` marker, park labels off, `triage/accepted`, queue approval on a PR head). Body `{"type": "issue"|"pr", "reason", "caption", "sentinel_reason"}`. `POST` answers 202 with the job (`{id, status: "running"|"done"|"failed", decision: {verdict, rule, text, rationale, actions, keeps_park, ladder?: {decision, reason, ruled_out[]}}, notes, error}`; `ladder` is set on contributor-PR decisions and names the rung that decided — `already-fixed` → `mergeable-as-is` → `rebase-then-merge` → `return-to-contributor` → `contributor-offline` — and why each rung above it did not apply); 409 with the running job while one is in flight; `GET` reads it. Rails: never merges, refuses items wearing `hold`, never grants secrets/permissions (`needs-owner` + `hold` instead), recommends only on hive policy/governance questions (keeps 085). Audited as `repo_item_decision_delegated[_failed]`. See [dashboard-sections.md](../dashboard-sections.md#let-hive-decide). | `pkg/dashboard/api.go:329`, `pkg/dashboard/api_repo_item_decide_delegated.go` | -| `POST` | `/api/repos/{owner}/{repo}/items/{number}/decide-delegated/plan` | Same as `…/decide-delegated` | Preview for *Let hive decide*: runs the same owner-delegate policy with **no side effects** (nothing written, no job) and answers `{would, rule, acting, title, summary, why, consequences[], ladder, mergeable_as_is, alternatives[], writes: false, prior, notes}` — `summary` is one verb-first sentence with no "Hive would:" prefix (*Send #15332 back to its author with one instruction: make CI green, nothing else.*), `why` one short line (*CI is red.*), `consequences` at most two items, `ladder` the rung that decided with its `ruled_out` list, and `mergeable_as_is` whether an as-is merge is honest (the queue-merge and merge-now alternatives are dropped when it is not). The dialog shows **Cancel** / **Let hive do that**, plus **Approve and let hive merge** (or **Merge now as maintainer** when the hive would approve anyway) only when `mergeable_as_is`; only then does the SPA post `…/decide-delegated` with `chosen` naming the button. A transient GitHub failure (5xx, rate limit, timeout) is retried twice with backoff before the plan gives up. A failure answers `{ok: false, error, reason, retryable, attempts, elapsed_ms, hive, hint}` under **409** when the item is no longer open (`reason: not-open`) or **424** when a read it depends on failed (`github`), is not configured (`config`) or the plan itself failed (`plan`) — never 502/503, which the hosted Ingress intercepts and replaces with its generic *hive unavailable* page (hivecommons/hive#16776). See [dashboard-sections.md](../dashboard-sections.md#let-hive-decide). | `pkg/dashboard/api.go:355`, `pkg/dashboard/api_repo_item_decide_delegated_plan.go` | -| `POST` | `/api/repos/{owner}/{repo}/items/{number}/blocker/recheck` | Same as `…/hold` (owner or repo write), signed-in GitHub user | *Re-check permissions* on an issue parked on a capability blocker (#15099): probes the GitHub App installation's permissions; when the missing permission is granted, clears every ledger entry blocked on it and un-parks the item (`{result: {outcome: "released", …}}`), otherwise `{result: {outcome: "no-effect", reason: "blocker", options: [...]}}`; 404 when the item has no blocker, 409 when the blocker is not a capability gap | `pkg/dashboard/api.go:356` | -| `POST` | `/api/repos/{owner}/{repo}/items/{number}/blocker/close-partial` | Same as `…/hold` (owner or repo write), signed-in GitHub user | Option (c) on a capability blocker (#15099): labels the issue `hive/partially-addressed`, posts the operator's `{"comment"}` naming what remains and closes it; the ledger entry is cleared | `pkg/dashboard/api.go:357` | -| `POST` | `/api/repos/{owner}/{repo}/items/{number}/accept-triage` | Owner or repo write/maintain/admin | Accept triage on an issue from the activity monitor's row menu — what a maintainer accepting triage on GitHub does by hand: adds the reporter-trust triage label (`triage/accepted`, or the first of `issue_filter.reporter_trust.untrusted_require_labels`; the label is created in the repo if missing) and removes the awaiting label (`needs-triage` / `reporter_trust.awaiting_label`) and a Hive-written `stage/015-triage-needed`. Empty body. Returns `{ok, repo, number, acceptLabel, added, removed, alreadyAccepted, minStatusSeq}`; an issue already carrying the accept label with nothing left to remove is a no-op (`alreadyAccepted: true`, no write, no audit). 404 for a pull request or an unknown item. The status snapshot's labels are patched so the next reconcile reads 010 Actionable before the engine's own 015 → 010 derivation. Every write is audited as `repo_item_accept_triage` under the dashboard user. | `pkg/dashboard/api.go:553`, `pkg/dashboard/api_repo_item_accept_triage.go` | -| `POST` | `/api/repos/{owner}/{repo}/items/{number}/sentinel-confirm` | Owner or repo write/maintain/admin | **Confirm safe** on the activity monitor's human-gate pill ([#17470](https://github.com/hivecommons/hive/issues/17470)): a person confirms a sentinel finding is intended. Body `{comment?, type?}` (`type` is `pr` or `issue`, comment ≤ 4000 characters). Posts the confirmation as a comment first — ``, `@user confirmed the sentinel finding safe from the dashboard.`, then the comment text — and removes every label matching the configured sentinel label (`sentinel.label`, default `sentinel-alert`, case-insensitive). Returns `{ok, repo, number, label, removed, alreadyClear, result}`; with no sentinel label on the item the comment is still posted and `alreadyClear` is true. 403 for a repository the hive does not watch. Audited as `repo_item_sentinel_confirm` under the dashboard user; the operator-action ledger records it as `confirm-sentinel`. | `pkg/dashboard/api.go:347`, `pkg/dashboard/api_repo_item_sentinel_confirm.go` | -| `GET` | `/api/acmm/evaluation` | Dashboard auth/session | ACMMEvaluation — combined codebase + operational result, cached server-side for 1 hour (`acmmEvalTTL`). `?refresh=1` (the dashboard's "🔄 Re-evaluate" button, #5877) bypasses the hourly TTL but is debounced server-side: requests within 1 minute of the last evaluation (`acmmRefreshDebounce`) still serve the cache, since a full refresh costs up to ~29 GitHub GetContents calls per repo. The response's `last_evaluated_at` timestamp reports when the cached evaluation was computed | `pkg/dashboard/api.go:359`, `pkg/dashboard/api_acmm_eval.go` | -| `POST` | `/api/acmm/issue` | Owner only | ACMMCreate Issue — files on GitHub or, with `governor.acmm.issue_tracker: work_source` / body `tracker: "work_source"` on a Linear-sourced hive, on Linear; response `tracker` says which. See [ACMM policy matrix](../acmm-policy-matrix.md#where-acmm-gap-issues-are-filed) | `pkg/dashboard/api.go:360` | -| `GET` | `/api/acmm-recommendation` | Dashboard auth/session | Advisory level-up recommendation (`acmmadvisor.Recommendation`, JSON): never changes the applied level — see [ACMM advisor](../acmm-advisor.md) | `pkg/dashboard/api.go:361` | -| `GET` | `/api/hive-advice` | Dashboard auth/session | Weekly owner advice (`hiveadvisor.Result`, JSON) shared with the advisory digest: recommendations with `items`, `links` (Overview export paths) and `bands`; `bands[]` rule text; `counts` naming the governor actionable queue vs. the Overview total. Which recommendations show is frozen for 7 days; their numbers refresh every call — see [Owner advice](../advisory.md#owner-advice) | `pkg/dashboard/api.go:363`, `pkg/dashboard/api_hive_advice.go` | +| `POST` | `/api/repos/{owner}/{repo}/items/{number}/decide-delegated/plan` | Same as `…/decide-delegated` | Preview for *Let hive decide*: runs the same owner-delegate policy with **no side effects** (nothing written, no job) and answers `{would, rule, acting, title, summary, why, consequences[], ladder, mergeable_as_is, alternatives[], writes: false, prior, notes}` — `summary` is one verb-first sentence with no "Hive would:" prefix (*Send #15332 back to its author with one instruction: make CI green, nothing else.*), `why` one short line (*CI is red.*), `consequences` at most two items, `ladder` the rung that decided with its `ruled_out` list, and `mergeable_as_is` whether an as-is merge is honest (the queue-merge and merge-now alternatives are dropped when it is not). The dialog shows **Cancel** / **Let hive do that**, plus **Approve and let hive merge** (or **Merge now as maintainer** when the hive would approve anyway) only when `mergeable_as_is`; only then does the SPA post `…/decide-delegated` with `chosen` naming the button. A transient GitHub failure (5xx, rate limit, timeout) is retried twice with backoff before the plan gives up. A failure answers `{ok: false, error, reason, retryable, attempts, elapsed_ms, hive, hint}` under **409** when the item is no longer open (`reason: not-open`) or **424** when a read it depends on failed (`github`), is not configured (`config`) or the plan itself failed (`plan`) — never 502/503, which the hosted Ingress intercepts and replaces with its generic *hive unavailable* page (hivecommons/hive#16776). See [dashboard-sections.md](../dashboard-sections.md#let-hive-decide). | `pkg/dashboard/api.go:356`, `pkg/dashboard/api_repo_item_decide_delegated_plan.go` | +| `POST` | `/api/repos/{owner}/{repo}/items/{number}/blocker/recheck` | Same as `…/hold` (owner or repo write), signed-in GitHub user | *Re-check permissions* on an issue parked on a capability blocker (#15099): probes the GitHub App installation's permissions; when the missing permission is granted, clears every ledger entry blocked on it and un-parks the item (`{result: {outcome: "released", …}}`), otherwise `{result: {outcome: "no-effect", reason: "blocker", options: [...]}}`; 404 when the item has no blocker, 409 when the blocker is not a capability gap | `pkg/dashboard/api.go:357` | +| `POST` | `/api/repos/{owner}/{repo}/items/{number}/blocker/close-partial` | Same as `…/hold` (owner or repo write), signed-in GitHub user | Option (c) on a capability blocker (#15099): labels the issue `hive/partially-addressed`, posts the operator's `{"comment"}` naming what remains and closes it; the ledger entry is cleared | `pkg/dashboard/api.go:358` | +| `POST` | `/api/repos/{owner}/{repo}/items/{number}/accept-triage` | Owner or repo write/maintain/admin | Accept triage on an issue from the activity monitor's row menu — what a maintainer accepting triage on GitHub does by hand: adds the reporter-trust triage label (`triage/accepted`, or the first of `issue_filter.reporter_trust.untrusted_require_labels`; the label is created in the repo if missing) and removes the awaiting label (`needs-triage` / `reporter_trust.awaiting_label`) and a Hive-written `stage/015-triage-needed`. Empty body. Returns `{ok, repo, number, acceptLabel, added, removed, alreadyAccepted, minStatusSeq}`; an issue already carrying the accept label with nothing left to remove is a no-op (`alreadyAccepted: true`, no write, no audit). 404 for a pull request or an unknown item. The status snapshot's labels are patched so the next reconcile reads 010 Actionable before the engine's own 015 → 010 derivation. Every write is audited as `repo_item_accept_triage` under the dashboard user. | `pkg/dashboard/api.go:554`, `pkg/dashboard/api_repo_item_accept_triage.go` | +| `POST` | `/api/repos/{owner}/{repo}/items/{number}/sentinel-confirm` | Owner or repo write/maintain/admin | **Confirm safe** on the activity monitor's human-gate pill ([#17470](https://github.com/hivecommons/hive/issues/17470)): a person confirms a sentinel finding is intended. Body `{comment?, type?}` (`type` is `pr` or `issue`, comment ≤ 4000 characters). Posts the confirmation as a comment first — ``, `@user confirmed the sentinel finding safe from the dashboard.`, then the comment text — and removes every label matching the configured sentinel label (`sentinel.label`, default `sentinel-alert`, case-insensitive). Returns `{ok, repo, number, label, removed, alreadyClear, result}`; with no sentinel label on the item the comment is still posted and `alreadyClear` is true. 403 for a repository the hive does not watch. Audited as `repo_item_sentinel_confirm` under the dashboard user; the operator-action ledger records it as `confirm-sentinel`. | `pkg/dashboard/api.go:348`, `pkg/dashboard/api_repo_item_sentinel_confirm.go` | +| `GET` | `/api/acmm/evaluation` | Dashboard auth/session | ACMMEvaluation — combined codebase + operational result, cached server-side for 1 hour (`acmmEvalTTL`). `?refresh=1` (the dashboard's "🔄 Re-evaluate" button, #5877) bypasses the hourly TTL but is debounced server-side: requests within 1 minute of the last evaluation (`acmmRefreshDebounce`) still serve the cache, since a full refresh costs up to ~29 GitHub GetContents calls per repo. The response's `last_evaluated_at` timestamp reports when the cached evaluation was computed | `pkg/dashboard/api.go:360`, `pkg/dashboard/api_acmm_eval.go` | +| `POST` | `/api/acmm/issue` | Owner only | ACMMCreate Issue — files on GitHub or, with `governor.acmm.issue_tracker: work_source` / body `tracker: "work_source"` on a Linear-sourced hive, on Linear; response `tracker` says which. See [ACMM policy matrix](../acmm-policy-matrix.md#where-acmm-gap-issues-are-filed) | `pkg/dashboard/api.go:361` | +| `GET` | `/api/acmm-recommendation` | Dashboard auth/session | Advisory level-up recommendation (`acmmadvisor.Recommendation`, JSON): never changes the applied level — see [ACMM advisor](../acmm-advisor.md) | `pkg/dashboard/api.go:362` | +| `GET` | `/api/hive-advice` | Dashboard auth/session | Weekly owner advice (`hiveadvisor.Result`, JSON) shared with the advisory digest: recommendations with `items`, `links` (Overview export paths) and `bands`; `bands[]` rule text; `counts` naming the governor actionable queue vs. the Overview total. Which recommendations show is frozen for 7 days; their numbers refresh every call — see [Owner advice](../advisory.md#owner-advice) | `pkg/dashboard/api.go:364`, `pkg/dashboard/api_hive_advice.go` | diff --git a/src/docs/api-reference/snapshots-and-style.md b/src/docs/api-reference/snapshots-and-style.md index 0951aa89f6..49513410a0 100644 --- a/src/docs/api-reference/snapshots-and-style.md +++ b/src/docs/api-reference/snapshots-and-style.md @@ -5,8 +5,8 @@ Part of the [Dashboard REST API reference](../api-reference.md); see that page f | Method | Path | Auth | Purpose | Source | |---|---|---|---|---| -| `GET` | `/api/style` | Public | Sanitized custom dashboard CSS | `pkg/dashboard/api.go:46` | +| `GET` | `/api/style` | Public | Sanitized custom dashboard CSS | `pkg/dashboard/api.go:47` | | `GET` | `/branding/custom.css` | Dashboard auth/session | Operator branding stylesheet override, read per request (see [branding](../branding.md)) | `pkg/dashboard/server.go:1745` | -| `GET` | `/api/snapshot/frame-ancestors` | Public | Snapshot framing allowlist | `pkg/dashboard/api.go:82` | -| `GET` | `/api/snapshot` | Public | Snapshot data | `pkg/dashboard/api.go:83` | -| `GET` | `/snapshot` | Public | Public read-only snapshot page | `pkg/dashboard/api.go:84` | +| `GET` | `/api/snapshot/frame-ancestors` | Public | Snapshot framing allowlist | `pkg/dashboard/api.go:83` | +| `GET` | `/api/snapshot` | Public | Snapshot data | `pkg/dashboard/api.go:84` | +| `GET` | `/snapshot` | Public | Public read-only snapshot page | `pkg/dashboard/api.go:85` | diff --git a/src/docs/design/workflow-models.md b/src/docs/design/workflow-models.md new file mode 100644 index 0000000000..c98e896da6 --- /dev/null +++ b/src/docs/design/workflow-models.md @@ -0,0 +1,157 @@ +# Workflow models + +**Status: phase 1 implemented** (ingest, model, renderers and the proving harness — +[#17620](https://github.com/hivecommons/hive/issues/17620)); phases 2–4 (editor, stage-engine +binding, marketplace) are design only. Lands on the `v6` branch; the v5 and v6 lifecycles are +untouched until phase 2 opts a hive in. + +## What this is + +A deterministic reading of how a repository already delivers software — its GitHub Actions +workflows and its Tekton / OpenShift Pipelines resources — expressed as one model +(`workflow.hivecommons.dev/v1alpha1`) that hive can render, diff, store and, later, run as a +hive's lifecycle. No LLM is involved in the ingest: two YAML definitions that mean the same thing +produce the same content hash. + +The model has two layers: + +- **Process graph** — the states an issue moves through (open → actionable → claimed → PR in + progress → review ∥ CI → merge queue → merged, then optionally main CI → release → deploy → + done), with guards on the transitions. With no pipelines bound this is exactly the hive default + (`HiveDefault` in `pkg/workflowmodel/defaults.go`) and renders today's Activity Monitor columns. +- **Pipelines** — each workflow or Pipeline as an execution DAG (jobs / tasks / reusable + workflows as nodes, `needs` / `runAfter` / result references as edges), with matrix, + concurrency, environment, `finally` and `when` recorded on the nodes and the triggers that + start it. + +`Bind` joins them: a pipeline's **role** (derived from its triggers — `pull_request` / +`merge_group` ⇒ PR verify, `push` to branches ⇒ push verify, tags or `release` ⇒ release, +environments or deployment events ⇒ deploy, otherwise scheduled / manual / other) decides which +state it fills. One workflow serving several triggers binds to each role its triggers imply +(`BindingRoles`), so `on: [pull_request, push]` appears under both *CI / Review* and *Main CI*. + +## Packages + +| Package | Role | +|---|---| +| `pkg/workflowmodel` | types, `Validate`, `Hash`, `HiveDefault`, `Bind`, topological `Levels`, `Compile` → `Profile` | +| `pkg/workflowmodel/hivedefault` | `Generate()` the `hive-default` model from `stagelabels.Table` + `stageengine.DiagramEdges()`; `Profile()` compiled | +| `pkg/stageengine` (`workflow.go`) | `DeriveFromModel(profile, kind)`: builtin delegation or the rules evaluator; `Options.Workflow` | +| `pkg/workflowmodel/githubactions` | `.github/workflows/*.y*ml` → pipelines | +| `pkg/workflowmodel/tekton` | Pipeline / PipelineRun / Task (+ Pipelines-as-Code annotations) → pipelines | +| `pkg/workflowmodel/ingest` | `Repository(scope, files)`: route files to ingesters, compose, bind | +| `pkg/workflowmodel/render` | `Columns` (Activity Monitor), `Diff` against the default, `Mermaid` | +| `cmd/workflow-survey` | walk a corpus of repositories, write per-repo model / columns / diagram and a summary | + +Everything the ingest could not represent is a **finding** on `spec.coverage` with a stable code +(`reusable-only`, `secrets-inherit`, `no-trigger`, `environment-dynamic`, `yaml-parse`, …) and a +severity, never a guess. + +## Column derivation + +- PR-verify pipelines fill the `ci` region; their merged topological levels become sub-stages of + *CI / Review* (at most six, names truncated past three per level). +- Push-verify adds *Main CI*; release adds *Release*; deploy adds one *Deploy: \* column per + literal environment (or a single *Deploy* when the environment is an expression); the chain + ends in *Done*. *Merged* stops being the final state only when a post-merge chain exists. +- Scheduled, manual and other pipelines bind to no state and are listed, not rendered. + +## Proving corpus + +`pkg/workflowmodel/testdata/corpus/__/` holds real CI files copied from public +repositories (argo-cd, cilium, envoy, konflux-ci, tektoncd/pipeline, serverless-operator) with a +golden summary per repository; `go test ./pkg/workflowmodel/ingest -update` rewrites the goldens +after an intended change. The full 317-repository CNCF-landscape survey and its collector live +outside the repository; the results are on #17620. + +## Lifecycle half and the compiler (RFC phase 1) + +The RFC (#17620) asks for one engine that derives under any model. The decision that keeps v5/v6 +byte-identical is that **`hive-default` derivation stays in Go**: `hivedefault.Generate()` builds +the model from the stage table, the engine's expected edges and the monitor's column anchors, and +declares `derivation: builtin`. `stageengine.DeriveFromModel` then delegates to `IssueDeriver` / +`PRDeriver` and only checks coverage — a stage the model has no state for abstains with +`workflow: not covered` instead of writing a label the model does not know. The parity test +(`TestDeriveFromModelHiveDefaultParity`) runs every PR stage golden and an issue sweep through both +and requires identical `Decision`s. + +Custom models declare `derivation: rules`: ordered rules, each a conjunction of predicates over the +fact vocabulary in `workflowmodel.Facts` (`open`, `merged`, `draft`, `labels has stage/*`, +`author.kind is human`, `pr.checks is green`, `pr.review is approved`, `issue.covered`, …); the +first rule whose predicates all hold names the state, and an item no rule matches is not covered. +Rules reach only the facts the engine already computes; no LLM is ever in the derivation path. +`testdata/models/techdocs.json` is the hand-written reference model (cncf/techdocs engagement). + +`Compile` refuses (error diagnostics, no profile): `state-no-exit`, `gate-no-owner`, +`agent-no-human-gate` (every path from an agent-granted state to a terminal outcome must cross a +transition gated by a role with `human: true`; closure edges — the item closed unmerged — are +observations and exempt), `duplicate-event-fact`, `state-no-column` / `unknown-column`, +`duplicate-stage-number`, `unknown-fact` / `bad-op`, `no-rules`, `no-version`. Elements marked +`confidence: inferred` compile with an `inferred-element` warning the editor shows. + +Stage numbers in a model are Hive's label numbers (`stagelabels.Table`) for now, so a bound repo's +labels, tracker anchors and replies keep working; a state whose number does not apply to the item +kind is not covered. Models are whole (no inheritance) through phase 4. + +## Known gaps (phase 1) + +- Prow (`kubernetes/*`) is a third syntax not read; those repos render the default columns. +- `pull_request_target`-only bot workflows (labelers, welcome bots) count as PR verify and inflate + the first CI level; the editor (phase 2) is where a maintainer deselects them. +- Pipelines-as-Code `on-cel-expression` triggers are recorded as `cel` events and not classified + into roles, so a tag-push expressed in CEL does not add *Release*. +- Bare Tekton `Pipeline` library files (catalogs) have no trigger and bind nowhere (`no-trigger`). + +## Store, bindings and engine wiring (RFC phase 2) + +- **Store** (`pkg/workflowmodel/store`): `/workflows//.json` plus an + `index.json` rebuilt on write; names/versions are regex-guarded, writes are atomic. + `hive-default` is virtual and read-only. Drafts (models that do not compile) save but + `Profile()` returns `ErrNotCompiled`. Entries reserve `adoption`, `ratings`, `benchmarks` + (raw JSON, carried verbatim) so the phase-7 marketplace can add them without a migration. +- **Binding** (`pkg/config/workflow_binding.go`): `project.workflow_bindings: [{repo, name, + version}]`, mirroring `repo_policies`. `WorkflowBindingFor` is keyed like repo pauses + (lowercased, org-qualified) and reads the live config, so a PUT takes effect without a + restart. Binding requires the model to compile; deleting a bound model is a 409. +- **API** (`pkg/dashboard/api_workflows.go`): `GET/POST /api/workflows`, `POST + /api/workflows/compile`, `GET/PUT /api/workflows/bindings`, `GET/PUT/DELETE + /api/workflows/{name}/{version}`. Writes need the owner role and are audited; 503 when no + store is configured. +- **Engine**: `stageengine.Options.WorkflowFor(owner, repo) *Profile` and `DeriveForRepo` (a + per-profile, hash-cached deriver) route bound repositories through `DeriveFromModel`; unbound + repositories and `workflow.enabled: false` take the builtin path unchanged. An unusable + binding (missing or non-compiling model) warns once and falls back. +- **Tracker**: the response gains `workflows: {default: {...}, repos: {: {name, version, + columns, numbers, error}}}` (nil when nothing is bound). The JS labels the model column, + renders a per-row column strip by stage number, and shows an amber "Not covered" chevron for + abstentions. Settings → Repos has a per-repo Workflow select that PUTs the binding. + +## Ingest → lifecycle (RFC phase 3a) + +`ingest.Repository` (phase 0) is unchanged and still produces the process half. `ingest.Lifecycle` +composes on top of it: + +1. Start from `hivedefault.Generate()` — the compiled hive-default lifecycle, `derivation: builtin`. +2. Copy the ingested pipelines, sources and coverage; `meta.source` is the system(s) read, + `meta.provenance` is `ingested-unreviewed`, `meta.version` is `1` (the store assigns later + versions). +3. PR-verify pipelines are bound to every state in the CI / Review column, each with a citation to + the file the pipeline came from. +4. Every post-merge state the process half produced (main CI, release, `deploy-`, done) + becomes a state in its own column after Merged, `confidence: inferred`, with the environment + guards carried over. The chain's transitions fire on `pipeline-succeeded` = + `pipeline.status is green`. The engine does not observe post-merge runs yet and reports + `unknown` for that fact, so a bound ingested model derives exactly like hive-default today; + the extra columns render, nothing lands in them until post-merge observation exists. + +The compiler warns once per inferred element; a human confirming them in the editor (phase 4) +is what moves provenance to `local`/`curated`. + +`POST /api/workflows/ingest` is the spoke entry point (owner role): `{repo}` fetches the default +branch through `FetchDefaultBranchTree`, `ingest.ReadTree` keeps GitHub Actions workflows, YAML +under Tekton directories and pipeline-named YAML that looks like Tekton (the survey collector's +rules), and the checkout is removed. `{files}` accepts pasted definitions. `save: true` stores the +result at the next integer version under `name` (default: the repository slug). + +`cmd/workflow-survey` now compiles every corpus repository's lifecycle model (`lifecycle.json` +alongside `model.json`) and reports pass/refusal counts by diagnostic code. diff --git a/src/internal/testutil/dashboard_import_allowlist.txt b/src/internal/testutil/dashboard_import_allowlist.txt index ee1fcff2c7..43b8a642cf 100644 --- a/src/internal/testutil/dashboard_import_allowlist.txt +++ b/src/internal/testutil/dashboard_import_allowlist.txt @@ -7,6 +7,7 @@ # hivecommons/hive#13536) from pkg/dashboard must route through an existing # seam or be added here with PR justification; removed imports must be # deleted so the ratchet tightens. Entries are sorted. +# workflowmodel: /api/workflows handlers compile lifecycle models (Compile, Profile diagnostics) and the tracker emits per-repo workflow columns; the store and bindings already route through Dependencies (hivecommons/hive#17620 P2). github.com/hivecommons/hive/crux github.com/hivecommons/hive/crux/agentparse github.com/hivecommons/hive/crux/claude @@ -73,5 +74,6 @@ github.com/hivecommons/hive/pkg/tokens github.com/hivecommons/hive/pkg/turn github.com/hivecommons/hive/pkg/upstreamwatch github.com/hivecommons/hive/pkg/webassets +github.com/hivecommons/hive/pkg/workflowmodel github.com/hivecommons/hive/pkg/worksource github.com/hivecommons/hive/pkg/writeback diff --git a/src/pkg/config/config.go b/src/pkg/config/config.go index b9f388ad4d..a061c6a702 100644 --- a/src/pkg/config/config.go +++ b/src/pkg/config/config.go @@ -110,9 +110,14 @@ type Config struct { // Dedupe tunes the dedupe lane's follow-through on its checklists. Dedupe DedupeLaneConfig `yaml:"dedupe,omitempty" json:"dedupe,omitempty"` Contribute ContributeConfig `yaml:"contribute,omitempty" json:"contribute,omitempty"` - HiveID string `yaml:"hive_id"` - ACMMLevel *int `yaml:"acmm_level,omitempty" json:"acmm_level"` - Variables VariablesConfig `yaml:"variables,omitempty"` + // Workflow gates the workflow-model engine (hivecommons/hive#17620). + // Off by default: every repository derives stages with the built-in + // derivers exactly as before. On, repositories bound to a workflow + // model derive through it; unbound repositories are unchanged. + Workflow WorkflowConfig `yaml:"workflow,omitempty" json:"workflow,omitempty"` + HiveID string `yaml:"hive_id"` + ACMMLevel *int `yaml:"acmm_level,omitempty" json:"acmm_level"` + Variables VariablesConfig `yaml:"variables,omitempty"` // OTel configures standards-based OTLP trace export. It is the preferred // operator-facing block; Tracing is retained as a legacy alias. OTel OTelConfig `yaml:"otel,omitempty" json:"otel,omitempty"` @@ -277,6 +282,11 @@ type Config struct { ReviewerAutoscaleActive int `yaml:"-" json:"-"` } +// WorkflowConfig is the `workflow:` block (hivecommons/hive#17620). +type WorkflowConfig struct { + Enabled bool `yaml:"enabled" json:"enabled"` +} + // UnmarshalYAML accepts the operator-facing agents.github_api_hourly_cap // scalar alongside the existing agents. map entries. The Agents field // remains a plain map everywhere else in the codebase. @@ -1096,6 +1106,9 @@ type ProjectConfig struct { // repo name. project.repos remains the watched-repo identity list; this // sidecar list lets existing string-list configs keep round-tripping. RepoPolicies []RepoPolicy `yaml:"repo_policies,omitempty" json:"repo_policies,omitempty"` + // WorkflowBindings assigns stored workflow models to repositories + // (hivecommons/hive#17620). Unbound repositories use hive-default. + WorkflowBindings []WorkflowBinding `yaml:"workflow_bindings,omitempty" json:"workflow_bindings,omitempty"` // WritingGuide is the hive owner's instruction for how the issues, PRs and // reviews its agents write should READ — length, structure, register — as // free text (hivecommons/hive#7667). It is rendered into every default @@ -5837,6 +5850,10 @@ type DataConfig struct { CopilotSessionsDir string `yaml:"copilot_sessions_dir"` BobSessionsDir string `yaml:"bob_sessions_dir"` AgentsDir string `yaml:"agents_dir"` + // WorkflowsDir is the durable root for stored workflow models + // (hivecommons/hive#17620); models live under /workflows. Empty + // means the hive data dir (/data); HIVE_WORKFLOWS_DATA_DIR overrides. + WorkflowsDir string `yaml:"workflows_dir,omitempty"` // SessionRetentionDays is how many days of CLI session-state directories to // keep. Session directories are created per CLI invocation and were never diff --git a/src/pkg/config/testdata/agent_runtime_golden.json b/src/pkg/config/testdata/agent_runtime_golden.json index ac730577ef..b5c0896436 100644 --- a/src/pkg/config/testdata/agent_runtime_golden.json +++ b/src/pkg/config/testdata/agent_runtime_golden.json @@ -440,8 +440,8 @@ "BaseURL": "", "AppAuthoredPRs": null, "AppSignedCommits": null, - "attention": {}, "notifications": {}, + "attention": {}, "OAuthBaseURLOverride": "", "OAuthAPIURLOverride": "" }, @@ -463,6 +463,7 @@ "Port": 8080, "AuthToken": "", "AgentPollIntervalS": 10, + "matrix": {}, "AuthorizedUsers": null, "AuthorizedUserNames": null, "HubProxied": false, @@ -473,7 +474,6 @@ } }, "issue_bands": {}, - "matrix": {}, "stage_tracker": {} }, "Data": { @@ -483,6 +483,7 @@ "CopilotSessionsDir": "/data/home/.copilot/session-state", "BobSessionsDir": "/data/home/.bob", "AgentsDir": "/data/agent-configs", + "WorkflowsDir": "", "SessionRetentionDays": null }, "deployment": {}, @@ -668,6 +669,9 @@ }, "dedupe": {}, "contribute": {}, + "workflow": { + "enabled": false + }, "HiveID": "", "acmm_level": null, "Variables": { diff --git a/src/pkg/config/validate.go b/src/pkg/config/validate.go index 35277f96e6..8aab8ad642 100644 --- a/src/pkg/config/validate.go +++ b/src/pkg/config/validate.go @@ -96,6 +96,9 @@ func (c *Config) ValidateWithOptions(opts ValidateOptions) error { if err := ValidateWritingGuide(c.Project.WritingGuide); err != nil { return err } + if err := validateWorkflowBindings(c.Project.WorkflowBindings); err != nil { + return err + } if _, err := c.Ioscan.Policy(); err != nil { return err } diff --git a/src/pkg/config/workflow_binding.go b/src/pkg/config/workflow_binding.go new file mode 100644 index 0000000000..f6dcb51c31 --- /dev/null +++ b/src/pkg/config/workflow_binding.go @@ -0,0 +1,95 @@ +package config + +import ( + "fmt" + "strings" +) + +// WorkflowBinding assigns a stored workflow model (name@version) to a +// repository (hivecommons/hive#17620). Repositories without a binding keep +// the built-in hive-default behaviour unchanged. +type WorkflowBinding struct { + Repo string `yaml:"repo" json:"repo"` + Name string `yaml:"name" json:"name"` + Version string `yaml:"version" json:"version"` +} + +// Key returns the name@version identity of the bound model. +func (b WorkflowBinding) Key() string { return b.Name + "@" + b.Version } + +// WorkflowBindingFor returns the binding covering repo, if any. repo may be +// bare or org-qualified. +func (c *Config) WorkflowBindingFor(repo string) (WorkflowBinding, bool) { + if c == nil { + return WorkflowBinding{}, false + } + key := repoPauseKey(c.Project.Org, repo) + if key == "" { + return WorkflowBinding{}, false + } + repoPauseMu.RLock() + defer repoPauseMu.RUnlock() + for _, b := range c.Project.WorkflowBindings { + if repoPauseKey(c.Project.Org, b.Repo) == key { + return b, true + } + } + return WorkflowBinding{}, false +} + +// SetWorkflowBinding binds repo to name@version, replacing any existing +// binding for the repo. An empty name removes the binding. It reports whether +// the config changed. +func (c *Config) SetWorkflowBinding(repo, name, version string) (bool, error) { + if c == nil { + return false, fmt.Errorf("nil config") + } + repo = strings.TrimSpace(repo) + key := repoPauseKey(c.Project.Org, repo) + if key == "" { + return false, fmt.Errorf("workflow binding: empty repo") + } + name, version = strings.TrimSpace(name), strings.TrimSpace(version) + if name != "" && version == "" { + return false, fmt.Errorf("workflow binding: %s has no version", name) + } + repoPauseMu.Lock() + defer repoPauseMu.Unlock() + kept := c.Project.WorkflowBindings[:0:0] + changed := false + for _, b := range c.Project.WorkflowBindings { + if repoPauseKey(c.Project.Org, b.Repo) == key { + changed = changed || name == "" || b.Name != name || b.Version != version + continue + } + kept = append(kept, b) + } + if name != "" { + if len(kept) == len(c.Project.WorkflowBindings) { + changed = true + } + kept = append(kept, WorkflowBinding{Repo: repo, Name: name, Version: version}) + } + if changed { + c.Project.WorkflowBindings = kept + } + return changed, nil +} + +func validateWorkflowBindings(bs []WorkflowBinding) error { + seen := map[string]bool{} + for i, b := range bs { + if strings.TrimSpace(b.Repo) == "" { + return fmt.Errorf("project.workflow_bindings[%d]: repo is required", i) + } + if strings.TrimSpace(b.Name) == "" || strings.TrimSpace(b.Version) == "" { + return fmt.Errorf("project.workflow_bindings[%d] (%s): name and version are required", i, b.Repo) + } + k := strings.ToLower(b.Repo) + if seen[k] { + return fmt.Errorf("project.workflow_bindings: %s bound twice", b.Repo) + } + seen[k] = true + } + return nil +} diff --git a/src/pkg/config/workflow_binding_test.go b/src/pkg/config/workflow_binding_test.go new file mode 100644 index 0000000000..55410374e6 --- /dev/null +++ b/src/pkg/config/workflow_binding_test.go @@ -0,0 +1,68 @@ +package config + +import ( + "strings" + "testing" +) + +func TestWorkflowBindings(t *testing.T) { + var nilCfg *Config + if _, ok := nilCfg.WorkflowBindingFor("x"); ok { + t.Fatal("nil config bound") + } + if _, err := nilCfg.SetWorkflowBinding("x", "a", "1"); err == nil { + t.Fatal("nil config set") + } + c := &Config{Project: ProjectConfig{Org: "hivecommons"}} + if _, ok := c.WorkflowBindingFor(""); ok { + t.Fatal("empty repo bound") + } + if _, err := c.SetWorkflowBinding(" ", "a", "1"); err == nil { + t.Fatal("empty repo accepted") + } + if _, err := c.SetWorkflowBinding("docs", "a", ""); err == nil { + t.Fatal("missing version accepted") + } + changed, err := c.SetWorkflowBinding("docs", "cncf-techdocs", "1") + if err != nil || !changed { + t.Fatalf("%v %v", changed, err) + } + b, ok := c.WorkflowBindingFor("HiveCommons/Docs") + if !ok || b.Key() != "cncf-techdocs@1" { + t.Fatalf("%+v %v", b, ok) + } + if changed, _ := c.SetWorkflowBinding("hivecommons/docs", "cncf-techdocs", "1"); changed { + t.Fatal("same binding reported changed") + } + if changed, _ := c.SetWorkflowBinding("docs", "cncf-techdocs", "2"); !changed || len(c.Project.WorkflowBindings) != 1 { + t.Fatalf("%+v", c.Project.WorkflowBindings) + } + if changed, _ := c.SetWorkflowBinding("other", "", ""); changed { + t.Fatal("unbinding an unbound repo changed") + } + if changed, _ := c.SetWorkflowBinding("docs", "", ""); !changed || len(c.Project.WorkflowBindings) != 0 { + t.Fatalf("%+v", c.Project.WorkflowBindings) + } +} + +func TestValidateWorkflowBindings(t *testing.T) { + cases := []struct { + bs []WorkflowBinding + want string + }{ + {nil, ""}, + {[]WorkflowBinding{{Repo: "a", Name: "m", Version: "1"}}, ""}, + {[]WorkflowBinding{{Name: "m", Version: "1"}}, "repo is required"}, + {[]WorkflowBinding{{Repo: "a", Name: "m"}}, "name and version"}, + {[]WorkflowBinding{{Repo: "a", Name: "m", Version: "1"}, {Repo: "A", Name: "n", Version: "1"}}, "bound twice"}, + } + for _, tc := range cases { + err := validateWorkflowBindings(tc.bs) + if tc.want == "" && err != nil { + t.Fatalf("%+v: %v", tc.bs, err) + } + if tc.want != "" && (err == nil || !strings.Contains(err.Error(), tc.want)) { + t.Fatalf("%+v: %v", tc.bs, err) + } + } +} diff --git a/src/pkg/dashboard/api.go b/src/pkg/dashboard/api.go index ddaf166a01..3f540af75c 100644 --- a/src/pkg/dashboard/api.go +++ b/src/pkg/dashboard/api.go @@ -35,6 +35,7 @@ func (s *Server) RegisterAPI(deps *Dependencies) { // Issue claims (hivecommons/hive#8380). Routes register unconditionally; // handlers answer "not enabled" when deps.IssueClaims is nil. s.registerClaimsRoutes() + s.registerWorkflowRoutes() // Approval desk (RFC #4000). Routes register unconditionally; the handlers // report "not enabled" when deps.ApprovalInbox is nil, so a disabled desk is // an honest 200 the panel can render rather than a 404 that looks broken. diff --git a/src/pkg/dashboard/api_workflows.go b/src/pkg/dashboard/api_workflows.go new file mode 100644 index 0000000000..c1065c0952 --- /dev/null +++ b/src/pkg/dashboard/api_workflows.go @@ -0,0 +1,223 @@ +package dashboard + +import ( + "errors" + "fmt" + "net/http" + "time" + + "github.com/hivecommons/hive/pkg/config" + "github.com/hivecommons/hive/pkg/workflowmodel" + wfstore "github.com/hivecommons/hive/pkg/workflowmodel/store" +) + +// Workflow-model library and bindings (hivecommons/hive#17620, RFC phase 2). +// +// Reads are open to any dashboard user; writes are owner-gated and audited. +// The store is optional: without one every endpoint answers 503 so the UI can +// say "workflows not enabled" instead of showing an empty library. +func (s *Server) registerWorkflowRoutes() { + s.mux.HandleFunc("GET /api/workflows", s.handleWorkflowList) + s.mux.HandleFunc("POST /api/workflows", s.handleWorkflowSave) + s.mux.HandleFunc("POST /api/workflows/compile", s.handleWorkflowCompile) + s.mux.HandleFunc("POST /api/workflows/ingest", s.handleWorkflowIngest) + s.mux.HandleFunc("GET /api/workflows/bindings", s.handleWorkflowBindingsGet) + s.mux.HandleFunc("PUT /api/workflows/bindings", s.handleWorkflowBindingPut) + s.mux.HandleFunc("GET /api/workflows/{name}/{version}", s.handleWorkflowGet) + s.mux.HandleFunc("PUT /api/workflows/{name}/{version}", s.handleWorkflowSave) + s.mux.HandleFunc("DELETE /api/workflows/{name}/{version}", s.handleWorkflowDelete) +} + +func (s *Server) workflowStore(w http.ResponseWriter) *wfstore.Store { + if s.deps == nil || s.deps.WorkflowStore == nil { + jsonError(w, "workflow store not enabled", http.StatusServiceUnavailable) + return nil + } + return s.deps.WorkflowStore +} + +// workflowProfileLookup is the store's Profile method, or nil without a store. +func (s *Server) workflowProfileLookup() workflowProfileLookup { + if s.deps == nil || s.deps.WorkflowStore == nil { + return nil + } + return s.deps.WorkflowStore.Profile +} + +func workflowStoreError(w http.ResponseWriter, err error) { + switch { + case errors.Is(err, wfstore.ErrNotFound): + jsonError(w, err.Error(), http.StatusNotFound) + case errors.Is(err, wfstore.ErrReadOnly): + jsonError(w, err.Error(), http.StatusForbidden) + case errors.Is(err, wfstore.ErrBadName), errors.Is(err, wfstore.ErrBadVersion), errors.Is(err, wfstore.ErrNotCompiled): + jsonError(w, err.Error(), http.StatusBadRequest) + default: + jsonError(w, err.Error(), http.StatusInternalServerError) + } +} + +func (s *Server) handleWorkflowList(w http.ResponseWriter, r *http.Request) { + st := s.workflowStore(w) + if st == nil { + return + } + entries, err := st.List() + if err != nil { + workflowStoreError(w, err) + return + } + jsonResponse(w, map[string]interface{}{ + "workflows": entries, + "bindings": s.workflowBindings(), + "enabled": s.deps.Config != nil && s.deps.Config.Workflow.Enabled, + }) +} + +func (s *Server) handleWorkflowGet(w http.ResponseWriter, r *http.Request) { + st := s.workflowStore(w) + if st == nil { + return + } + m, e, err := st.Get(r.PathValue("name"), r.PathValue("version")) + if err != nil { + workflowStoreError(w, err) + return + } + resp := map[string]interface{}{"entry": e, "model": m} + if p, _, cerr := workflowmodel.Compile(m); cerr == nil { + resp["profile"] = p + } + jsonResponse(w, resp) +} + +// handleWorkflowCompile compiles a model without saving it so the editor can +// show diagnostics as the operator types. +func (s *Server) handleWorkflowCompile(w http.ResponseWriter, r *http.Request) { + var m workflowmodel.Model + if err := decodeBody(r, &m); err != nil { + jsonError(w, err.Error(), http.StatusBadRequest) + return + } + p, diags, err := workflowmodel.Compile(&m) + resp := map[string]interface{}{"compiled": err == nil, "diagnostics": diags} + if diags == nil { + resp["diagnostics"] = []workflowmodel.Diagnostic{} + } + if p != nil { + resp["profile"] = p + } + jsonResponse(w, resp) +} + +func (s *Server) handleWorkflowSave(w http.ResponseWriter, r *http.Request) { + if !requireOwnerRole(w, r) { + return + } + st := s.workflowStore(w) + if st == nil { + return + } + var m workflowmodel.Model + if err := decodeBody(r, &m); err != nil { + jsonError(w, err.Error(), http.StatusBadRequest) + return + } + if name := r.PathValue("name"); name != "" && (name != m.Metadata.Name || r.PathValue("version") != m.Spec.Meta.Version) { + jsonError(w, fmt.Sprintf("body is %s@%s, path is %s@%s", m.Metadata.Name, m.Spec.Meta.Version, name, r.PathValue("version")), http.StatusBadRequest) + return + } + e, err := st.Save(&m, requestUser(r), time.Now().UTC()) + if err != nil { + workflowStoreError(w, err) + return + } + s.auditFromRequest(r, "workflow_save", fmt.Sprintf("%s compiled=%v", e.Key(), e.Compiled), "") + jsonStatusResponse(w, http.StatusCreated, map[string]interface{}{"entry": e}) +} + +func (s *Server) handleWorkflowDelete(w http.ResponseWriter, r *http.Request) { + if !requireOwnerRole(w, r) { + return + } + st := s.workflowStore(w) + if st == nil { + return + } + name, version := r.PathValue("name"), r.PathValue("version") + if s.deps.Config != nil { + for _, b := range s.deps.Config.Project.WorkflowBindings { + if b.Name == name && b.Version == version { + jsonError(w, fmt.Sprintf("%s@%s is bound to %s; unbind first", name, version, b.Repo), http.StatusConflict) + return + } + } + } + if err := st.Delete(name, version); err != nil { + workflowStoreError(w, err) + return + } + s.auditFromRequest(r, "workflow_delete", name+"@"+version, "") + okResponse(w, nil) +} + +func (s *Server) workflowBindings() []config.WorkflowBinding { + if s.deps == nil || s.deps.Config == nil || s.deps.Config.Project.WorkflowBindings == nil { + return []config.WorkflowBinding{} + } + return s.deps.Config.Project.WorkflowBindings +} + +func (s *Server) handleWorkflowBindingsGet(w http.ResponseWriter, r *http.Request) { + if s.deps == nil || s.deps.Config == nil { + jsonError(w, "config unavailable", http.StatusServiceUnavailable) + return + } + jsonResponse(w, map[string]interface{}{ + "bindings": s.workflowBindings(), + "enabled": s.deps.Config.Workflow.Enabled, + }) +} + +// handleWorkflowBindingPut binds (name+version set) or unbinds (name empty) a +// repository. A bound model must exist and compile so a stale binding can +// never leave the tracker without columns. +func (s *Server) handleWorkflowBindingPut(w http.ResponseWriter, r *http.Request) { + if !requireOwnerRole(w, r) { + return + } + if s.deps == nil || s.deps.Config == nil { + jsonError(w, "config unavailable", http.StatusServiceUnavailable) + return + } + var body config.WorkflowBinding + if err := decodeBody(r, &body); err != nil { + jsonError(w, err.Error(), http.StatusBadRequest) + return + } + action := "workflow_unbind" + if body.Name != "" { + st := s.workflowStore(w) + if st == nil { + return + } + if _, err := st.Profile(body.Name, body.Version); err != nil { + workflowStoreError(w, err) + return + } + action = "workflow_bind" + } + changed, err := s.deps.Config.SetWorkflowBinding(body.Repo, body.Name, body.Version) + if err != nil { + jsonError(w, err.Error(), http.StatusBadRequest) + return + } + if changed { + if err := s.saveConfig(); err != nil { + jsonError(w, "save config: "+err.Error(), http.StatusInternalServerError) + return + } + s.auditFromRequest(r, action, fmt.Sprintf("%s -> %s@%s", body.Repo, body.Name, body.Version), "") + } + jsonResponse(w, map[string]interface{}{"changed": changed, "bindings": s.workflowBindings()}) +} diff --git a/src/pkg/dashboard/api_workflows_ingest.go b/src/pkg/dashboard/api_workflows_ingest.go new file mode 100644 index 0000000000..155f08a24f --- /dev/null +++ b/src/pkg/dashboard/api_workflows_ingest.go @@ -0,0 +1,134 @@ +package dashboard + +import ( + "context" + "errors" + "fmt" + "net/http" + "strconv" + "strings" + "time" + + "github.com/hivecommons/hive/pkg/workflowmodel" + "github.com/hivecommons/hive/pkg/workflowmodel/ingest" + wfstore "github.com/hivecommons/hive/pkg/workflowmodel/store" +) + +// workflowIngestTimeout bounds one repository fetch + ingest. +const workflowIngestTimeout = 2 * time.Minute + +// workflowIngestRequest is the body of POST /api/workflows/ingest. Exactly +// one of Repo (fetch the default branch through the forge) or Files +// (pasted/uploaded CI definitions, repo-relative path -> contents) is set. +type workflowIngestRequest struct { + Repo string `json:"repo"` + Files map[string]string `json:"files,omitempty"` + // Save stores the result as a draft under Name (default: the ingest's + // slug of the repository) at the next free integer version. + Save bool `json:"save,omitempty"` + Name string `json:"name,omitempty"` +} + +// handleWorkflowIngest turns a repository's CI definitions into a lifecycle +// model (RFC #17620 P3): hive-default's stages plus the repository's +// pipelines, provenance ingested-unreviewed. Owner-gated because it reads +// the forge on the hive's credential and may write the store. +func (s *Server) handleWorkflowIngest(w http.ResponseWriter, r *http.Request) { + if !requireOwnerRole(w, r) { + return + } + var req workflowIngestRequest + if err := decodeBody(r, &req); err != nil { + jsonError(w, err.Error(), http.StatusBadRequest) + return + } + req.Repo = strings.TrimSpace(req.Repo) + if (req.Repo == "") == (len(req.Files) == 0) { + jsonError(w, "set exactly one of repo or files", http.StatusBadRequest) + return + } + var files map[string][]byte + ref := "" + repo := req.Repo + switch { + case len(req.Files) > 0: + if repo == "" { + repo = "local/pasted" + } + files = make(map[string][]byte, len(req.Files)) + for p, body := range req.Files { + files[strings.TrimPrefix(strings.TrimSpace(p), "/")] = []byte(body) + } + default: + if !strings.Contains(repo, "/") { + jsonError(w, "repo must be owner/name", http.StatusBadRequest) + return + } + if s.deps == nil || s.deps.WorkflowRepoFiles == nil { + jsonError(w, "repository ingest not available: no forge client", http.StatusServiceUnavailable) + return + } + ctx, cancel := context.WithTimeout(r.Context(), workflowIngestTimeout) + defer cancel() + var err error + files, ref, err = s.deps.WorkflowRepoFiles(ctx, repo) + if err != nil { + jsonError(w, fmt.Sprintf("fetch %s: %v", repo, err), http.StatusBadGateway) + return + } + } + + m := ingest.Lifecycle(workflowmodel.Scope{Forge: "github.com", Repository: repo}, files) + if req.Name != "" { + m.Metadata.Name = req.Name + } + p, diags, cerr := workflowmodel.Compile(m) + if diags == nil { + diags = []workflowmodel.Diagnostic{} + } + resp := map[string]interface{}{ + "model": m, "compiled": cerr == nil, "diagnostics": diags, + "ref": ref, "files": len(files), "pipelines": len(m.Spec.Pipelines), + } + if p != nil { + resp["profile"] = p + } + if !req.Save { + jsonResponse(w, resp) + return + } + st := s.workflowStore(w) + if st == nil { + return + } + version, err := nextWorkflowVersion(st, m.Metadata.Name) + if err != nil { + workflowStoreError(w, err) + return + } + m.Spec.Meta.Version = version + e, err := st.Save(m, requestUser(r), time.Now().UTC()) + if err != nil { + workflowStoreError(w, err) + return + } + s.auditFromRequest(r, "workflow_ingest", fmt.Sprintf("%s from %s@%s files=%d compiled=%v", e.Key(), repo, ref, len(files), e.Compiled), "") + resp["entry"] = e + jsonStatusResponse(w, http.StatusCreated, resp) +} + +// nextWorkflowVersion is one past the highest integer version stored under +// name ("1" for a new name); non-integer versions are ignored. +func nextWorkflowVersion(st *wfstore.Store, name string) (string, error) { + entries, err := st.Versions(name) + if err != nil && !errors.Is(err, wfstore.ErrNotFound) { + return "", err + } + max := 0 + for _, e := range entries { + if n, err := strconv.Atoi(e.Version); err == nil && n > max { + max = n + } + } + return strconv.Itoa(max + 1), nil +} diff --git a/src/pkg/dashboard/api_workflows_ingest_test.go b/src/pkg/dashboard/api_workflows_ingest_test.go new file mode 100644 index 0000000000..ec96d72e6a --- /dev/null +++ b/src/pkg/dashboard/api_workflows_ingest_test.go @@ -0,0 +1,105 @@ +package dashboard + +import ( + "context" + "encoding/json" + "errors" + "net/http" + "testing" + + "github.com/hivecommons/hive/pkg/workflowmodel" + wfstore "github.com/hivecommons/hive/pkg/workflowmodel/store" +) + +const ingestCIYAML = "name: CI\non: [pull_request, push]\njobs:\n test:\n runs-on: ubuntu-latest\n steps: [{run: make test}]\n" + +func TestWorkflowIngestFromFilesCompilesAndSaves(t *testing.T) { + s := workflowTestServer(t, true) + body := map[string]any{"files": map[string]string{".github/workflows/ci.yml": ingestCIYAML}, "name": "pasted", "save": true} + rec := wfReq(s, http.MethodPost, "/api/workflows/ingest", body, true) + if rec.Code != http.StatusCreated { + t.Fatalf("%d %s", rec.Code, rec.Body) + } + var resp struct { + Compiled bool `json:"compiled"` + Pipelines int `json:"pipelines"` + Entry wfstore.Entry `json:"entry"` + Model workflowmodel.Model `json:"model"` + Diags []workflowmodel.Diagnostic `json:"diagnostics"` + } + if err := json.Unmarshal(rec.Body.Bytes(), &resp); err != nil { + t.Fatal(err) + } + if !resp.Compiled || resp.Pipelines != 1 || resp.Entry.Key() != "pasted@1" || !resp.Entry.Compiled { + t.Fatalf("%+v", resp) + } + if resp.Model.Spec.Meta.Provenance != workflowmodel.ProvenanceIngestedUnreviewed { + t.Fatalf("provenance %q", resp.Model.Spec.Meta.Provenance) + } + // A second ingest of the same name lands at the next version. + rec = wfReq(s, http.MethodPost, "/api/workflows/ingest", body, true) + if rec.Code != http.StatusCreated { + t.Fatalf("%d %s", rec.Code, rec.Body) + } + if err := json.Unmarshal(rec.Body.Bytes(), &resp); err != nil { + t.Fatal(err) + } + if resp.Entry.Key() != "pasted@2" { + t.Fatalf("second version %s", resp.Entry.Key()) + } + // Dry run (no save) needs no store and returns the model. + s2 := workflowTestServer(t, false) + rec = wfReq(s2, http.MethodPost, "/api/workflows/ingest", map[string]any{"files": map[string]string{".github/workflows/ci.yml": ingestCIYAML}}, true) + if rec.Code != http.StatusOK { + t.Fatalf("%d %s", rec.Code, rec.Body) + } +} + +func TestWorkflowIngestFromRepoUsesForgeFetcher(t *testing.T) { + s := workflowTestServer(t, true) + var asked string + s.deps.WorkflowRepoFiles = func(_ context.Context, repo string) (map[string][]byte, string, error) { + asked = repo + if repo == "acme/missing" { + return nil, "", errors.New("404") + } + return map[string][]byte{".github/workflows/ci.yml": []byte(ingestCIYAML)}, "main", nil + } + rec := wfReq(s, http.MethodPost, "/api/workflows/ingest", map[string]any{"repo": "acme/widgets"}, true) + if rec.Code != http.StatusOK || asked != "acme/widgets" { + t.Fatalf("%d %s asked=%q", rec.Code, rec.Body, asked) + } + var resp struct { + Ref string `json:"ref"` + Model workflowmodel.Model `json:"model"` + } + if err := json.Unmarshal(rec.Body.Bytes(), &resp); err != nil { + t.Fatal(err) + } + if resp.Ref != "main" || resp.Model.Metadata.Name != "acme-widgets" || resp.Model.Spec.Scope.Repository != "acme/widgets" { + t.Fatalf("%+v", resp) + } + if rec := wfReq(s, http.MethodPost, "/api/workflows/ingest", map[string]any{"repo": "acme/missing"}, true); rec.Code != http.StatusBadGateway { + t.Fatalf("fetch failure %d", rec.Code) + } +} + +func TestWorkflowIngestGuards(t *testing.T) { + s := workflowTestServer(t, true) + cases := []struct { + body map[string]any + owner bool + want int + }{ + {map[string]any{"repo": "acme/widgets"}, false, http.StatusForbidden}, + {map[string]any{}, true, http.StatusBadRequest}, + {map[string]any{"repo": "acme/widgets", "files": map[string]string{"a.yml": "x"}}, true, http.StatusBadRequest}, + {map[string]any{"repo": "nojslash"}, true, http.StatusBadRequest}, + {map[string]any{"repo": "acme/widgets"}, true, http.StatusServiceUnavailable}, // no fetcher + } + for i, c := range cases { + if rec := wfReq(s, http.MethodPost, "/api/workflows/ingest", c.body, c.owner); rec.Code != c.want { + t.Fatalf("case %d: %d %s", i, rec.Code, rec.Body) + } + } +} diff --git a/src/pkg/dashboard/api_workflows_test.go b/src/pkg/dashboard/api_workflows_test.go new file mode 100644 index 0000000000..bb9e077957 --- /dev/null +++ b/src/pkg/dashboard/api_workflows_test.go @@ -0,0 +1,278 @@ +package dashboard + +import ( + "bytes" + "encoding/json" + "io" + "log/slog" + "net/http" + "net/http/httptest" + "os" + "testing" + "time" + + "github.com/hivecommons/hive/pkg/workflowmodel" + "github.com/hivecommons/hive/pkg/workflowmodel/hivedefault" + wfstore "github.com/hivecommons/hive/pkg/workflowmodel/store" +) + +func workflowTestServer(t *testing.T, withStore bool) *Server { + t.Helper() + logger := slog.New(slog.NewTextHandler(io.Discard, &slog.HandlerOptions{Level: slog.LevelError})) + s := NewServer(0, logger) + deps := testDeps(t) + if withStore { + deps.WorkflowStore = wfstore.Open(t.TempDir()) + } + s.RegisterAPI(deps) + return s +} + +func techdocsModel(t *testing.T) *workflowmodel.Model { + t.Helper() + b, err := os.ReadFile("../workflowmodel/testdata/models/techdocs.json") + if err != nil { + t.Fatal(err) + } + var m workflowmodel.Model + if err := json.Unmarshal(b, &m); err != nil { + t.Fatal(err) + } + return &m +} + +func wfReq(s *Server, method, path string, body any, owner bool) *httptest.ResponseRecorder { + var buf bytes.Buffer + if body != nil { + if raw, ok := body.(string); ok { + buf.WriteString(raw) + } else { + _ = json.NewEncoder(&buf).Encode(body) + } + } + req := httptest.NewRequest(method, path, &buf) + req.Header.Set("Content-Type", "application/json") + req.Header.Set("X-Hive-User", "andy") + if owner { + req.Header.Set("X-Hive-Role", "owner") + req.Header.Set(ownerRoleVerifiedHeader, "true") + } + rec := httptest.NewRecorder() + s.mux.ServeHTTP(rec, req) + return rec +} + +func TestWorkflowAPIDisabledWithoutStore(t *testing.T) { + s := workflowTestServer(t, false) + for _, c := range []struct{ method, path string }{ + {http.MethodGet, "/api/workflows"}, + {http.MethodGet, "/api/workflows/x/1"}, + {http.MethodPost, "/api/workflows"}, + {http.MethodDelete, "/api/workflows/x/1"}, + } { + if rec := wfReq(s, c.method, c.path, nil, true); rec.Code != http.StatusServiceUnavailable { + t.Fatalf("%s %s: %d %s", c.method, c.path, rec.Code, rec.Body) + } + } + // compile needs no store + rec := wfReq(s, http.MethodPost, "/api/workflows/compile", techdocsModel(t), false) + var out struct{ Compiled bool } + if rec.Code != 200 || json.Unmarshal(rec.Body.Bytes(), &out) != nil || !out.Compiled { + t.Fatalf("%d %s", rec.Code, rec.Body) + } + // binding a model requires the store + rec = wfReq(s, http.MethodPut, "/api/workflows/bindings", map[string]string{"repo": "repo1", "name": "x", "version": "1"}, true) + if rec.Code != http.StatusServiceUnavailable { + t.Fatalf("%d %s", rec.Code, rec.Body) + } +} + +func TestWorkflowAPIOwnerGate(t *testing.T) { + s := workflowTestServer(t, true) + for _, c := range []struct{ method, path string }{ + {http.MethodPost, "/api/workflows"}, + {http.MethodPut, "/api/workflows/x/1"}, + {http.MethodDelete, "/api/workflows/x/1"}, + {http.MethodPut, "/api/workflows/bindings"}, + } { + if rec := wfReq(s, c.method, c.path, techdocsModel(t), false); rec.Code != http.StatusForbidden { + t.Fatalf("%s %s: %d", c.method, c.path, rec.Code) + } + } +} + +func TestWorkflowAPILifecycle(t *testing.T) { + s := workflowTestServer(t, true) + m := techdocsModel(t) + + rec := wfReq(s, http.MethodGet, "/api/workflows", nil, false) + var list struct { + Workflows []wfstore.Entry + Bindings []json.RawMessage + Enabled bool + } + if rec.Code != 200 || json.Unmarshal(rec.Body.Bytes(), &list) != nil || len(list.Workflows) != 1 || list.Bindings == nil || list.Enabled { + t.Fatalf("%d %s", rec.Code, rec.Body) + } + + if rec := wfReq(s, http.MethodPost, "/api/workflows", "{", true); rec.Code != http.StatusBadRequest { + t.Fatalf("bad json: %d", rec.Code) + } + if rec := wfReq(s, http.MethodPost, "/api/workflows", m, true); rec.Code != http.StatusCreated { + t.Fatalf("save: %d %s", rec.Code, rec.Body) + } + if rec := wfReq(s, http.MethodPut, "/api/workflows/cncf-techdocs/9", m, true); rec.Code != http.StatusBadRequest { + t.Fatalf("path mismatch: %d %s", rec.Code, rec.Body) + } + if rec := wfReq(s, http.MethodPut, "/api/workflows/cncf-techdocs/1", m, true); rec.Code != http.StatusCreated { + t.Fatalf("put: %d %s", rec.Code, rec.Body) + } + if rec := wfReq(s, http.MethodPost, "/api/workflows", hivedefault.Generate(), true); rec.Code != http.StatusForbidden { + t.Fatalf("builtin save: %d", rec.Code) + } + + rec = wfReq(s, http.MethodGet, "/api/workflows/cncf-techdocs/1", nil, false) + var got struct { + Entry wfstore.Entry + Profile *workflowmodel.Profile + } + if rec.Code != 200 || json.Unmarshal(rec.Body.Bytes(), &got) != nil || got.Profile == nil || got.Entry.SavedBy != "andy" { + t.Fatalf("%d %s", rec.Code, rec.Body) + } + if rec := wfReq(s, http.MethodGet, "/api/workflows/cncf-techdocs/2", nil, false); rec.Code != http.StatusNotFound { + t.Fatalf("missing: %d", rec.Code) + } + if rec := wfReq(s, http.MethodGet, "/api/workflows/Bad%20Name/2", nil, false); rec.Code != http.StatusBadRequest { + t.Fatalf("bad name: %d", rec.Code) + } + + // drafts save but cannot be bound + draft := techdocsModel(t) + draft.Spec.Meta.Version = "draft" + draft.Spec.Gates[0].ReleasedBy = "" + if rec := wfReq(s, http.MethodPost, "/api/workflows", draft, true); rec.Code != http.StatusCreated { + t.Fatalf("draft save: %d %s", rec.Code, rec.Body) + } + if rec := wfReq(s, http.MethodPut, "/api/workflows/bindings", map[string]string{"repo": "repo1", "name": "cncf-techdocs", "version": "draft"}, true); rec.Code != http.StatusBadRequest { + t.Fatalf("bind draft: %d %s", rec.Code, rec.Body) + } + if rec := wfReq(s, http.MethodPut, "/api/workflows/bindings", "{", true); rec.Code != http.StatusBadRequest { + t.Fatalf("bind bad json: %d", rec.Code) + } + if rec := wfReq(s, http.MethodPut, "/api/workflows/bindings", map[string]string{"repo": "", "name": "cncf-techdocs", "version": "1"}, true); rec.Code != http.StatusBadRequest { + t.Fatalf("bind empty repo: %d", rec.Code) + } + + rec = wfReq(s, http.MethodPut, "/api/workflows/bindings", map[string]string{"repo": "repo1", "name": "cncf-techdocs", "version": "1"}, true) + var bind struct { + Changed bool + Bindings []struct{ Repo, Name, Version string } + } + if rec.Code != 200 || json.Unmarshal(rec.Body.Bytes(), &bind) != nil || !bind.Changed || len(bind.Bindings) != 1 { + t.Fatalf("bind: %d %s", rec.Code, rec.Body) + } + if b, ok := s.deps.Config.WorkflowBindingFor("myorg/repo1"); !ok || b.Version != "1" { + t.Fatalf("%+v %v", b, ok) + } + rec = wfReq(s, http.MethodGet, "/api/workflows/bindings", nil, false) + if rec.Code != 200 || !bytes.Contains(rec.Body.Bytes(), []byte(`"cncf-techdocs"`)) { + t.Fatalf("%d %s", rec.Code, rec.Body) + } + + // bound models cannot be deleted + if rec := wfReq(s, http.MethodDelete, "/api/workflows/cncf-techdocs/1", nil, true); rec.Code != http.StatusConflict { + t.Fatalf("delete bound: %d %s", rec.Code, rec.Body) + } + rec = wfReq(s, http.MethodPut, "/api/workflows/bindings", map[string]string{"repo": "repo1"}, true) + if rec.Code != 200 || json.Unmarshal(rec.Body.Bytes(), &bind) != nil || !bind.Changed || len(bind.Bindings) != 0 { + t.Fatalf("unbind: %d %s", rec.Code, rec.Body) + } + rec = wfReq(s, http.MethodPut, "/api/workflows/bindings", map[string]string{"repo": "repo1"}, true) + if rec.Code != 200 || json.Unmarshal(rec.Body.Bytes(), &bind) != nil || bind.Changed { + t.Fatalf("unbind again: %d %s", rec.Code, rec.Body) + } + if rec := wfReq(s, http.MethodDelete, "/api/workflows/cncf-techdocs/1", nil, true); rec.Code != 200 { + t.Fatalf("delete: %d %s", rec.Code, rec.Body) + } + if rec := wfReq(s, http.MethodDelete, "/api/workflows/cncf-techdocs/1", nil, true); rec.Code != http.StatusNotFound { + t.Fatalf("delete twice: %d", rec.Code) + } + if rec := wfReq(s, http.MethodDelete, "/api/workflows/hive-default/1", nil, true); rec.Code != http.StatusForbidden { + t.Fatalf("delete builtin: %d", rec.Code) + } + + // compile reports diagnostics for drafts and invalid shapes + rec = wfReq(s, http.MethodPost, "/api/workflows/compile", draft, false) + var comp struct { + Compiled bool + Diagnostics []workflowmodel.Diagnostic + } + if rec.Code != 200 || json.Unmarshal(rec.Body.Bytes(), &comp) != nil || comp.Compiled || len(comp.Diagnostics) == 0 { + t.Fatalf("%d %s", rec.Code, rec.Body) + } + if rec := wfReq(s, http.MethodPost, "/api/workflows/compile", "{", false); rec.Code != http.StatusBadRequest { + t.Fatalf("compile bad json: %d", rec.Code) + } +} + +func TestWorkflowBindingsWithoutConfig(t *testing.T) { + s := workflowTestServer(t, true) + s.deps.Config = nil + if rec := wfReq(s, http.MethodGet, "/api/workflows/bindings", nil, false); rec.Code != http.StatusServiceUnavailable { + t.Fatalf("%d", rec.Code) + } + if rec := wfReq(s, http.MethodPut, "/api/workflows/bindings", map[string]string{"repo": "r"}, true); rec.Code != http.StatusServiceUnavailable { + t.Fatalf("%d", rec.Code) + } + rec := wfReq(s, http.MethodGet, "/api/workflows", nil, false) + if rec.Code != 200 || !bytes.Contains(rec.Body.Bytes(), []byte(`"bindings":[]`)) { + t.Fatalf("%d %s", rec.Code, rec.Body) + } +} + +func TestStageTrackerWorkflowsBlock(t *testing.T) { + qualify := func(r string) string { return "myorg/" + r } + if stageTrackerWorkflows(nil, nil, qualify) != nil { + t.Fatal("nil config produced a block") + } + s := workflowTestServer(t, true) + if stageTrackerWorkflows(s.deps.Config, s.workflowProfileLookup(), qualify) != nil { + t.Fatal("unbound hive produced a block") + } + if rec := wfReq(s, http.MethodPost, "/api/workflows", techdocsModel(t), true); rec.Code != http.StatusCreated { + t.Fatal(rec.Body.String()) + } + if _, err := s.deps.Config.SetWorkflowBinding("repo1", "cncf-techdocs", "1"); err != nil { + t.Fatal(err) + } + if _, err := s.deps.Config.SetWorkflowBinding("repo2", "ghost", "1"); err != nil { + t.Fatal(err) + } + blk := stageTrackerWorkflows(s.deps.Config, s.workflowProfileLookup(), qualify) + if blk == nil || blk.Default.Name != hivedefault.Name || len(blk.Default.Columns) != 7 { + t.Fatalf("%+v", blk) + } + td := blk.Repos["myorg/repo1"] + if td.Name != "cncf-techdocs" || td.Error != "" || len(td.Columns) != 5 { + t.Fatalf("%+v", td) + } + var nums int + for _, c := range td.Columns { + nums += len(c.Numbers) + } + if nums == 0 { + t.Fatalf("no stage numbers anchored: %+v", td.Columns) + } + if g := blk.Repos["myorg/repo2"]; g.Error == "" || g.Name != "ghost" { + t.Fatalf("%+v", g) + } + noStore := stageTrackerWorkflows(s.deps.Config, nil, qualify) + if noStore.Repos["myorg/repo1"].Error == "" { + t.Fatalf("%+v", noStore.Repos) + } + // the tracker response carries it + resp := s.buildStageTracker(&StatusPayload{HiveID: "h"}, nil, time.Now()) + if resp.Workflows == nil || len(resp.Workflows.Repos) != 2 { + t.Fatalf("%+v", resp.Workflows) + } +} diff --git a/src/pkg/dashboard/deps.go b/src/pkg/dashboard/deps.go index 04f5167c82..1a52a3fd65 100644 --- a/src/pkg/dashboard/deps.go +++ b/src/pkg/dashboard/deps.go @@ -2,6 +2,7 @@ package dashboard import ( "context" + wfstore "github.com/hivecommons/hive/pkg/workflowmodel/store" "log/slog" "os" "sync" @@ -36,6 +37,13 @@ import ( type Dependencies struct { Config *config.Config + // WorkflowStore serves the workflow-model library (hivecommons/hive#17620). + // Nil disables the /api/workflows endpoints. + WorkflowStore *wfstore.Store + // WorkflowRepoFiles fetches a repository's default-branch CI definition + // files (repo-relative slash path -> contents) and the ref read, for + // POST /api/workflows/ingest. Nil disables ingest from a repository. + WorkflowRepoFiles func(ctx context.Context, repo string) (files map[string][]byte, ref string, err error) // AgentMgr is the agent runtime (hivecommons/hive#13536). The dashboard // programs against the crux.Runtime surface, not *agent.Manager; tests that // need manager-only seams unwrap it with agentManager. diff --git a/src/pkg/dashboard/import_count_test.go b/src/pkg/dashboard/import_count_test.go index 14574bccf0..d2e43b1707 100644 --- a/src/pkg/dashboard/import_count_test.go +++ b/src/pkg/dashboard/import_count_test.go @@ -63,8 +63,11 @@ func TestDashboardInternalImportCountRatchet(t *testing.T) { // surfaces render one markup (hivecommons/hive#16236), plus // pkg/maintainernotes, the append-only store behind the Activity // Monitor's private per-item maintainer notes and the /api/notes routes, - // shared with the scheduler's kick injection (hivecommons/hive#17244). - const maxDashboardInternalImports = 59 + // shared with the scheduler's kick injection (hivecommons/hive#17244), + // plus pkg/workflowmodel/store, the spoke-local library of workflow + // models behind the /api/workflows routes and the tracker's per-repo + // column block (RFC hivecommons/hive#17620, phase 2). + const maxDashboardInternalImports = 60 entries, err := os.ReadDir(".") if err != nil { diff --git a/src/pkg/dashboard/settings_parity_allowlist.txt b/src/pkg/dashboard/settings_parity_allowlist.txt index aec5c040e5..d0844ab284 100644 --- a/src/pkg/dashboard/settings_parity_allowlist.txt +++ b/src/pkg/dashboard/settings_parity_allowlist.txt @@ -613,3 +613,7 @@ governor.hotspot_min_lines # merge-hotspot advisor; yaml-only governor.hotspot_refile_days # merge-hotspot advisor; yaml-only governor.hotspot_max_open # merge-hotspot advisor; yaml-only governor.hotspot_max_per_pass # merge-hotspot advisor; yaml-only +data.workflows_dir # workflow-model store root; deployment knob, no Settings control planned (RFC #17620) +project.workflow_bindings[].name # set through the per-repo workflow select on Settings → Repos via PUT /api/workflows/bindings, not a leaf-path control (RFC #17620) +project.workflow_bindings[].repo # see project.workflow_bindings[].name +project.workflow_bindings[].version # see project.workflow_bindings[].name diff --git a/src/pkg/dashboard/stage_tracker.go b/src/pkg/dashboard/stage_tracker.go index 9d911b8197..3d2d1418e8 100644 --- a/src/pkg/dashboard/stage_tracker.go +++ b/src/pkg/dashboard/stage_tracker.go @@ -77,6 +77,10 @@ type StageTrackerResponse struct { // Engine is the ADR-0025 central stage engine's mode and counters for // the tracker's status strip; absent when the hive has no engine. Engine *stageengine.Status `json:"engine,omitempty"` + // Workflows names the workflow model each bound repository follows and + // its column layout (hivecommons/hive#17620); absent when no repository + // is bound. + Workflows *StageTrackerWorkflows `json:"workflows,omitempty"` // RecentlyMerged lists the pull requests GitHub reports merged within // stageTrackerMergedWindow, whoever merged them: the audit log's // pr_merged entries (the hive's own sweep, queue, relay and forge diff --git a/src/pkg/dashboard/stage_tracker_build.go b/src/pkg/dashboard/stage_tracker_build.go index ddde26791b..2a99c9406e 100644 --- a/src/pkg/dashboard/stage_tracker_build.go +++ b/src/pkg/dashboard/stage_tracker_build.go @@ -56,6 +56,7 @@ func (s *Server) buildStageTracker(status *StatusPayload, store *stagelabels.Sto cfg = s.deps.Config } resp.NeedsHumanLabels = needsHumanPresetLabels(cfg, status.HiveID) + resp.Workflows = stageTrackerWorkflows(cfg, s.workflowProfileLookup(), qualifyRepo) expectedPerspectives := stageTrackerExpectedPerspectives(cfg) verifier, verifierErr := s.reviewEvidenceVerifier() var evidenceVerifier *github.ReviewEvidenceVerifier diff --git a/src/pkg/dashboard/stage_tracker_workflows.go b/src/pkg/dashboard/stage_tracker_workflows.go new file mode 100644 index 0000000000..3e0d593361 --- /dev/null +++ b/src/pkg/dashboard/stage_tracker_workflows.go @@ -0,0 +1,89 @@ +package dashboard + +import ( + "sort" + + "github.com/hivecommons/hive/pkg/config" + "github.com/hivecommons/hive/pkg/workflowmodel" + "github.com/hivecommons/hive/pkg/workflowmodel/hivedefault" +) + +// StageTrackerWorkflows is the tracker's `workflows` block +// (hivecommons/hive#17620, RFC phase 2): which model each bound repository +// follows and the column/state layout the SPA renders for it. Absent when no +// repository is bound, so the v5/v6 payload is unchanged for every hive that +// has not opted in. +type StageTrackerWorkflows struct { + // Default is the hive-default model every unbound repository follows. + Default StageTrackerWorkflow `json:"default"` + // Repos maps an org-qualified repository to its bound model. Repos + // bound to a model that is missing or no longer compiles are listed + // with Error set and fall back to Default in the SPA. + Repos map[string]StageTrackerWorkflow `json:"repos"` +} + +// StageTrackerWorkflow is one model's identity and column layout. +type StageTrackerWorkflow struct { + Name string `json:"name"` + Version string `json:"version"` + Hash string `json:"hash,omitempty"` + Columns []StageTrackerWorkflowColumn `json:"columns,omitempty"` + Error string `json:"error,omitempty"` +} + +// StageTrackerWorkflowColumn is a column with the stage numbers it anchors, +// so the SPA can place an item by its stage/NNN number without knowing the +// model's state ids. +type StageTrackerWorkflowColumn struct { + ID string `json:"id"` + Label string `json:"label"` + Hint string `json:"hint,omitempty"` + Terminal bool `json:"terminal,omitempty"` + States []string `json:"states"` + Numbers []int `json:"numbers"` +} + +// workflowProfileLookup resolves a bound name@version to its compiled +// profile; the store's Profile method satisfies it. +type workflowProfileLookup func(name, version string) (*workflowmodel.Profile, error) + +func stageTrackerWorkflowBlock(p *workflowmodel.Profile) StageTrackerWorkflow { + out := StageTrackerWorkflow{Name: p.Name, Version: p.Version, Hash: p.Hash} + for _, c := range p.Columns { + col := StageTrackerWorkflowColumn{ID: c.ID, Label: c.Label, Hint: c.Hint, Terminal: c.Terminal, States: append([]string{}, c.States...), Numbers: []int{}} + for _, id := range c.States { + if st := p.States[id]; st != nil && st.Lifecycle.Number > 0 { + col.Numbers = append(col.Numbers, st.Lifecycle.Number) + } + } + sort.Ints(col.Numbers) + out.Columns = append(out.Columns, col) + } + return out +} + +// stageTrackerWorkflows builds the block from the config's bindings; nil when +// there are none. +func stageTrackerWorkflows(cfg *config.Config, lookup workflowProfileLookup, qualify func(string) string) *StageTrackerWorkflows { + if cfg == nil || len(cfg.Project.WorkflowBindings) == 0 { + return nil + } + out := &StageTrackerWorkflows{ + Default: stageTrackerWorkflowBlock(hivedefault.Profile()), + Repos: map[string]StageTrackerWorkflow{}, + } + for _, b := range cfg.Project.WorkflowBindings { + repo := qualify(b.Repo) + if lookup == nil { + out.Repos[repo] = StageTrackerWorkflow{Name: b.Name, Version: b.Version, Error: "workflow store not enabled"} + continue + } + p, err := lookup(b.Name, b.Version) + if err != nil { + out.Repos[repo] = StageTrackerWorkflow{Name: b.Name, Version: b.Version, Error: err.Error()} + continue + } + out.Repos[repo] = stageTrackerWorkflowBlock(p) + } + return out +} diff --git a/src/pkg/dashboard/stage_tracker_workflows_ui_test.go b/src/pkg/dashboard/stage_tracker_workflows_ui_test.go new file mode 100644 index 0000000000..1e2d2a1ce9 --- /dev/null +++ b/src/pkg/dashboard/stage_tracker_workflows_ui_test.go @@ -0,0 +1,101 @@ +package dashboard + +import ( + "os/exec" + "strings" + "testing" +) + +// TestStageTrackerWorkflowColumnsWiring pins the RFC phase-2 surface +// (hivecommons/hive#17620): the tracker reads `workflows` from the payload, +// a bound repository's group header carries the model badge and its own +// column header, and the row strip switches to the model's columns only +// when stageTrackerWorkflowOf resolves one. +func TestStageTrackerWorkflowColumnsWiring(t *testing.T) { + html := indexHTML(t) + for _, want := range []string{ + `ctx.workflows = d.workflows || null;`, + `function stageTrackerWorkflowOf(ctx, repo)`, + `function stageTrackerWorkflowColumnIndex(wf, stage)`, + `function stageTrackerWorkflowStripItems(item, wf, stages)`, + `function stageTrackerWorkflowHeaderHtml(wf)`, + `const rowWorkflow = typeof stageTrackerWorkflowOf === 'function' ? stageTrackerWorkflowOf(ctx, item.repo) : null;`, + `${rowWorkflow ? stageTrackerWorkflowStripItems(shown, rowWorkflow, stages) : stageTrackerStripItems(`, + `const groupWorkflow = typeof stageTrackerWorkflowOf === 'function' ? stageTrackerWorkflowOf(ctx, g.repo) : null;`, + `${groupWorkflow && !g.collapsed ? stageTrackerWorkflowHeaderHtml(groupWorkflow) : ''}`, + `.ham-wf-badge {`, + `.ham-seg.ham-wf-uncovered {`, + // Settings → Repos binding select. + `function renderRepoWorkflowRow(repo)`, + `async function setRepoWorkflow(repo, value)`, + `fetch('/api/workflows/bindings', {`, + `+ ldSub + renderRepoWorkflowRow(r);`, + } { + if !strings.Contains(html, want) { + t.Errorf("index.html missing workflow-columns wiring %q", want) + } + } +} + +// The pure helpers under node: an unbound hive resolves no model (so the +// built-in strip is untouched), hive-default itself resolves none, a bound +// repository places rows by stage number into the model's columns, a +// terminal stage lights the terminal column, and an uncovered number is +// drawn as "Not covered" rather than dropped. +func TestStageTrackerWorkflowColumnsNode(t *testing.T) { + node, err := exec.LookPath("node") + if err != nil { + t.Skip("node not installed") + } + html := indexHTML(t) + script := `const assert = require('assert'); +const STAGE_TRACKER_TERMINAL = 110; +function esc(s) { return String(s).replace(/&/g, '&').replace(/ html.match(/
  • m.slice(11, -1)); +const words = html => [...html.matchAll(/([^<]*)<\/span>/g)].map(m => m[1]); +let s = stageTrackerWorkflowStripItems({ stage: 45 }, td, stages); +assert.deepStrictEqual(li(s), ['ham-seg ham-wf-seg done', 'ham-seg ham-wf-seg active', 'ham-seg ham-wf-seg todo', 'ham-seg ham-wf-seg todo cap']); +assert.deepStrictEqual(words(s), ['Requested', 'Decision', 'Writing', 'Published'], 'active column wears the stage word'); +s = stageTrackerWorkflowStripItems({ stage: 110 }, td, stages); +assert.deepStrictEqual(li(s), ['ham-seg ham-wf-seg done', 'ham-seg ham-wf-seg done', 'ham-seg ham-wf-seg done', 'ham-seg ham-wf-seg active cap']); +s = stageTrackerWorkflowStripItems({ stage: 85 }, td, stages); +assert.ok(li(s)[0].includes('ham-wf-uncovered') && words(s)[0] === 'Not covered', s); +assert.ok(s.includes('stage/085 is not a state of cncf-techdocs@1')); +s = stageTrackerWorkflowStripItems({ stage: 0 }, td, stages); +assert.deepStrictEqual(li(s)[0], 'ham-seg ham-wf-seg active'); + +const head = stageTrackerWorkflowHeaderHtml(td); +assert.ok(head.includes('data-ham-workflow="cncf-techdocs@1"') && head.includes('title="Asked for"') && li(head).length === 4, head); +assert.ok(stageTrackerWorkflowBadgeHtml(td).includes('>cncf-techdocs@1<')); +assert.strictEqual(stageTrackerWorkflowBadgeHtml(null), ''); +` + if out, err := runNodeStdin(node, script); err != nil { + t.Fatalf("node workflow columns fixture failed: %v\n%s", err, out) + } +} diff --git a/src/pkg/dashboard/static-src/css/tracker.css b/src/pkg/dashboard/static-src/css/tracker.css index 079f2b3049..53a8eadfc8 100644 --- a/src/pkg/dashboard/static-src/css/tracker.css +++ b/src/pkg/dashboard/static-src/css/tracker.css @@ -266,6 +266,13 @@ .ham-group-n, .ham-group-attn { flex: 0 0 auto; font-size: var(--fs-xs); color: var(--text-faint); white-space: nowrap; } .ham-group-attn { color: var(--amber); } .ham-group-collapsed .ham-group-name { color: var(--muted); } + /* Workflow-model columns (RFC 17620): the model badge on a + bound repository's group header, its own column header, and the + "Not covered" chevron for a stage the model has no state for. */ + .ham-wf-badge { flex: 0 0 auto; font-family: var(--font-mono); font-size: var(--fs-2xs); font-weight: 700; letter-spacing: 0.02em; padding: var(--sp-1) var(--sp-3); border-radius: var(--r-pill); white-space: nowrap; color: var(--indigo); background: color-mix(in srgb, var(--indigo) 12%, transparent); border: 1px solid color-mix(in srgb, var(--indigo) 40%, transparent); } + .ham-wf-cols { position: static; } + .ham-wf-cols .ham-wf-col { color: var(--indigo); } + .ham-seg.ham-wf-uncovered { background: color-mix(in srgb, var(--amber) 18%, transparent); color: var(--amber); outline: 1px dashed var(--amber); } .ham-group-dist { flex: 0 0 auto; margin-left: auto; margin-right: calc(4em + var(--sp-3)); display: inline-flex; align-items: flex-end; gap: var(--sp-1); height: 0.9em; } .ham-gd { display: inline-block; width: 0.45em; height: 15%; border-radius: var(--sp-1); background: color-mix(in srgb, var(--text-faint) 35%, transparent); } .ham-gd-1 { height: 35%; background: color-mix(in srgb, var(--amber) 55%, transparent); } diff --git a/src/pkg/dashboard/static-src/js/config-governor.js b/src/pkg/dashboard/static-src/js/config-governor.js index 0cc16442a8..85e0a0216d 100644 --- a/src/pkg/dashboard/static-src/js/config-governor.js +++ b/src/pkg/dashboard/static-src/js/config-governor.js @@ -1104,7 +1104,7 @@ `; // Fully forge-qualified so the row is unambiguous: github.ibm.com/org/repo. const displayName = ghHost + '/' + r; - return `
    ${isDefault ? '★' : '☆'}${esc(displayName)}${inheritButton}${rtRow}${cbRow}${pillLabel}
    ` + ldSub; + return `
    ${isDefault ? '★' : '☆'}${esc(displayName)}${inheritButton}${rtRow}${cbRow}${pillLabel}
    ` + ldSub + renderRepoWorkflowRow(r); }).join(''); const hostNote = isGHE ? ` on ${esc(ghHost)} (GitHub Enterprise)` diff --git a/src/pkg/dashboard/static-src/js/config-workflows.js b/src/pkg/dashboard/static-src/js/config-workflows.js new file mode 100644 index 0000000000..407bed42df --- /dev/null +++ b/src/pkg/dashboard/static-src/js/config-workflows.js @@ -0,0 +1,96 @@ + // Workflow-model bindings (hivecommons/hive#17620, RFC phase 2). + // + // Each repo row on Settings → Repos carries a "Workflow" select: hive-default + // (the built-in lifecycle every unbound repo follows) or any compiled model + // from the library. Binding persists immediately through + // PUT /api/workflows/bindings (owner-only, audited) — it is not part of the + // tab's Save button, because a binding is validated server-side against the + // store (the model must exist and compile) and the tracker picks the new + // columns up on its next poll. + const WORKFLOW_DEFAULT_KEY = 'hive-default@1'; + const _workflowState = { loaded: false, loading: false, enabled: false, workflows: [], bindings: [] }; + + function workflowKey(name, version) { return (name || '') + '@' + (version || ''); } + + async function loadWorkflowLibrary(onDone) { + if (_workflowState.loading) return; + _workflowState.loading = true; + try { + const res = await fetch('/api/workflows', { headers: _inceptionAuthHeaders() }); + if (res.status === 503) { _workflowState.loaded = true; _workflowState.workflows = []; return; } + if (!res.ok) throw new Error(await saveErrorMessage(res)); + const data = await res.json(); + _workflowState.workflows = Array.isArray(data.workflows) ? data.workflows : []; + _workflowState.bindings = Array.isArray(data.bindings) ? data.bindings : []; + _workflowState.enabled = !!data.enabled; + _workflowState.loaded = true; + } catch (err) { + _workflowState.loaded = true; + showToast('Could not load the workflow library: ' + err.message, 'error'); + } finally { + _workflowState.loading = false; + if (typeof onDone === 'function') onDone(); + } + } + + function workflowBindingFor(repo) { + const want = String(repo || '').toLowerCase(); + const org = String((_configState.data && _configState.data.org) || '').toLowerCase(); + return _workflowState.bindings.find(b => { + const r = String(b.repo || '').toLowerCase(); + return r === want || (org && (r === org + '/' + want || want === org + '/' + r)); + }) || null; + } + + // renderRepoWorkflowRow returns the per-repo select, or '' when the hive + // has no workflow store (the row simply does not appear; nothing else on + // the tab changes). + function renderRepoWorkflowRow(repo) { + if (!_workflowState.loaded) { + loadWorkflowLibrary(() => { if (_configState.tab === 'Repos') renderConfigTab('Repos'); }); + return ''; + } + if (!_workflowState.workflows.length) return ''; + const bound = workflowBindingFor(repo); + const current = bound ? workflowKey(bound.name, bound.version) : WORKFLOW_DEFAULT_KEY; + const locked = !dashboardRoleAtLeast(window._hiveRole || 'read', 'owner'); + const options = _workflowState.workflows.map(w => { + const key = workflowKey(w.name, w.version); + const compiled = w.compiled !== false; + const label = key + (w.description ? ' — ' + w.description : '') + (compiled ? '' : ' (draft, does not compile)'); + return ``; + }).join(''); + const flag = _workflowState.enabled + ? '' + : 'derivation off'; + return `
    ${flag}${bound ? `bound` : 'default'}
    `; + } + + async function setRepoWorkflow(repo, value) { + if (!dashboardRoleAtLeast(window._hiveRole || 'read', 'owner')) { + showToast('Owner role is required to bind a workflow.', 'error'); + renderConfigTab('Repos'); + return; + } + const key = String(value || ''); + const body = { repo: repo }; + if (key && key !== WORKFLOW_DEFAULT_KEY) { + const at = key.lastIndexOf('@'); + body.name = key.slice(0, at); + body.version = key.slice(at + 1); + } + try { + const res = await fetch('/api/workflows/bindings', { + method: 'PUT', + headers: _inceptionAuthHeaders(), + body: JSON.stringify(body), + }); + if (!res.ok) throw new Error(await saveErrorMessage(res)); + const data = await res.json(); + _workflowState.bindings = Array.isArray(data.bindings) ? data.bindings : []; + showToast(body.name ? `${repo} now follows ${key}` : `${repo} back on hive-default`); + } catch (err) { + showToast('Failed to bind workflow: ' + err.message, 'error'); + } + renderConfigTab('Repos'); + } diff --git a/src/pkg/dashboard/static-src/js/tracker-filter-render.js b/src/pkg/dashboard/static-src/js/tracker-filter-render.js index fee4a2adfc..d90bdceaa9 100644 --- a/src/pkg/dashboard/static-src/js/tracker-filter-render.js +++ b/src/pkg/dashboard/static-src/js/tracker-filter-render.js @@ -87,6 +87,8 @@ } stageTrackerSyncToolbar(f); const ctx = { nowMs, stages, moves: _stageTrackerMoves }; + // Workflow-model columns for bound repositories (hivecommons/hive#17620). + ctx.workflows = d.workflows || null; ctx.workers = stageTrackerWorkers(window._lastStatus); ctx.events = _stageTrackerEvents; ctx.noteTuck = stageTrackerNoteTuckSetting(); diff --git a/src/pkg/dashboard/static-src/js/tracker-row-render.js b/src/pkg/dashboard/static-src/js/tracker-row-render.js index 63c8f4f4b3..98c6f556f9 100644 --- a/src/pkg/dashboard/static-src/js/tracker-row-render.js +++ b/src/pkg/dashboard/static-src/js/tracker-row-render.js @@ -63,6 +63,7 @@ // needs-attention count and a mini per-column distribution. Keyed // `group:owner/repo` by the render so the reconcile keeps it stable. function stageTrackerGroupHeaderHtml(g, stages, ctx) { + const groupWorkflow = typeof stageTrackerWorkflowOf === 'function' ? stageTrackerWorkflowOf(ctx, g.repo) : null; const max = Math.max(1, ...g.cols); const dist = g.cols.map((c, i) => ``).join(''); const attn = g.attention ? `${g.attention} need${g.attention === 1 ? 's' : ''} attention` : ''; @@ -70,10 +71,10 @@ const chipCtx = { repoChips: Object.assign({}, (ctx && ctx.repoChips) || {}) }; return `
    - ${stageTrackerRepoChipHtml(g.repo, chipCtx)}${esc(g.repo)} + ${stageTrackerRepoChipHtml(g.repo, chipCtx)}${esc(g.repo)}${groupWorkflow ? stageTrackerWorkflowBadgeHtml(groupWorkflow) : ''} ${g.count} ${g.count === 1 ? 'row' : 'rows'}${attn} - ${dist} -
    `; + ${groupWorkflow ? '' : `${dist}`} + ${groupWorkflow && !g.collapsed ? stageTrackerWorkflowHeaderHtml(groupWorkflow) : ''}`; } // Row = title line (number, title, chips, since) + thin strip. Non-active @@ -194,6 +195,9 @@ // One note line per row: the human line takes the stage note (folded // or dropped), so the event slot below the caption renders nothing. const rowKey = stageTrackerItemKey(item); + // A repository bound to a workflow model draws the model's columns + // (tracker-workflows.js); null keeps the built-in strip byte for byte. + const rowWorkflow = typeof stageTrackerWorkflowOf === 'function' ? stageTrackerWorkflowOf(ctx, item.repo) : null; const humanNote = typeof stageTrackerHumanContextHtml === 'function' ? stageTrackerHumanContextHtml(item, stages, nowMs, subRows, ctx, rowKey) : ''; const noteCtx = humanNote && typeof stageTrackerNoteTakenCtx === 'function' ? stageTrackerNoteTakenCtx(ctx, rowKey) : ctx; // One human-gate pill per gated row (hivecommons/hive#17470): it @@ -208,7 +212,7 @@
    ${typeof stageTrackerPreparationNodeHtml === 'function' ? stageTrackerPreparationNodeHtml(item, stages, ctx) : ''} -
      ${stageTrackerStripItems(shown, stages, range, nowMs, sharedUnder ? Object.assign({}, ctx, { gateElsewhere: true }) : ctx)}
    +
      ${rowWorkflow ? stageTrackerWorkflowStripItems(shown, rowWorkflow, stages) : stageTrackerStripItems(shown, stages, range, nowMs, sharedUnder ? Object.assign({}, ctx, { gateElsewhere: true }) : ctx)}
    ${stageTrackerSinceHtml(shown, nowMs)}${typeof stageTrackerPillHtml === 'function' ? stageTrackerPillHtml(shown, stages, range, nowMs, ctx) : ''}
    ${humanNote}${stageTrackerCurrentStageHtml(shown, stages, range, nowMs)}${typeof stageTrackerPreparationCommentHtml === 'function' ? stageTrackerPreparationCommentHtml(shown, noteCtx, rowKey) : typeof stageTrackerEventHtml === 'function' ? stageTrackerEventHtml(rowKey, noteCtx) : ''}${typeof stageTrackerExecutionHtml === 'function' ? stageTrackerExecutionHtml(shown) : ''}${typeof stageTrackerReviewNotesHtml === 'function' ? stageTrackerReviewNotesHtml(item, ctx) : ''}${typeof stageTrackerNotesPanelHtml === 'function' ? stageTrackerNotesPanelHtml(item, hamNotesCtx(item)) : ''}${subs} `; diff --git a/src/pkg/dashboard/static-src/js/tracker-workflows.js b/src/pkg/dashboard/static-src/js/tracker-workflows.js new file mode 100644 index 0000000000..df16e47b27 --- /dev/null +++ b/src/pkg/dashboard/static-src/js/tracker-workflows.js @@ -0,0 +1,77 @@ + /* ---- Workflow-model columns (hivecommons/hive#17620, RFC phase 2) ----- + GET /api/stage-tracker carries `workflows` only when at least one + repository is bound to a model other than hive-default. For such a + repository the group header names the model and draws the model's own + column header, and each row's strip is one chevron per model column, + placed by the stage/NNN number the row wears. Everything else on the + row (lead cell, chips, human gate, notes) is unchanged. Hives with no + binding never reach this code: stageTrackerWorkflowOf returns null and + the built-in strip renders exactly as before. Pure helpers so the node + harness (stage_tracker_workflows_ui_test.go) can run them. */ + const STAGE_TRACKER_WORKFLOW_UNCOVERED_WORD = 'Not covered'; + + // The model block for repo, or null when the repo follows hive-default + // (unbound, bound to hive-default itself, or bound to a model the + // server reported as unusable — the row then falls back to the + // built-in strip, which is what the engine does too). + function stageTrackerWorkflowOf(ctx, repo) { + const w = ctx && ctx.workflows; + if (!w || !w.repos) return null; + const wf = w.repos[String(repo || '')]; + if (!wf || wf.error || !Array.isArray(wf.columns) || !wf.columns.length) return null; + if (w.default && wf.name === w.default.name && wf.version === w.default.version) return null; + return wf; + } + + // Index of the column holding stage number `stage`; 0 for an unstaged + // item (the first column is the model's entry), the terminal column for + // a terminal stage, -1 when no column anchors the number (the model + // does not cover it: the engine abstained with "workflow: not covered"). + function stageTrackerWorkflowColumnIndex(wf, stage) { + const n = Number(stage) || 0; + const cols = wf.columns; + if (n === 0) return 0; + if (n >= (typeof STAGE_TRACKER_TERMINAL === 'number' ? STAGE_TRACKER_TERMINAL : 110)) { + const t = cols.findIndex(c => c.terminal); + return t >= 0 ? t : cols.length - 1; + } + return cols.findIndex(c => Array.isArray(c.numbers) && c.numbers.indexOf(n) >= 0); + } + + function stageTrackerWorkflowBadgeHtml(wf) { + if (!wf) return ''; + const key = `${wf.name}@${wf.version}`; + return `${esc(key)}`; + } + + // The model's column header, drawn under a bound repository's group + // header in place of the shared hive-default header. + function stageTrackerWorkflowHeaderHtml(wf) { + const cols = wf.columns.map(c => `
  • ${esc(c.label)}
  • `).join(''); + return ``; + } + + // One row's strip under its model: done before the anchored column, + // active at it (the stage's own word from `stages`, or the column + // label), todo after. An uncovered stage lights the first column amber + // with the "Not covered" word so the gap is visible, not silent. + function stageTrackerWorkflowStripItems(item, wf, stages) { + const cur = Number(item.stage) || 0; + const terminal = cur >= (typeof STAGE_TRACKER_TERMINAL === 'number' ? STAGE_TRACKER_TERMINAL : 110); + let at = stageTrackerWorkflowColumnIndex(wf, cur); + const uncovered = at < 0; + if (uncovered) at = 0; + const st = (stages || []).find(s => Number(s.number) === cur) || {}; + return wf.columns.map((c, i) => { + let state = 'todo'; + if (terminal || i < at) state = 'done'; + else if (i === at) state = 'active'; + if (terminal && c.terminal) state = 'active'; + const word = state !== 'active' ? c.label : (uncovered ? STAGE_TRACKER_WORKFLOW_UNCOVERED_WORD : (st.short || st.label || c.label)); + const tip = state === 'active' && uncovered + ? `stage/${String(cur).padStart(3, '0')} is not a state of ${wf.name}@${wf.version}; the engine leaves this item where it is.` + : (c.hint || c.label) + (state === 'active' && cur ? ` — stage/${String(cur).padStart(3, '0')}` : ''); + const cls = ['ham-seg', 'ham-wf-seg', state, c.terminal ? 'cap' : '', uncovered && state === 'active' ? 'ham-wf-uncovered' : ''].filter(Boolean).join(' '); + return `
  • ${esc(word)}
  • `; + }).join(''); + } diff --git a/src/pkg/dashboard/static-src/manifest.txt b/src/pkg/dashboard/static-src/manifest.txt index 2ba70649aa..2d894fd79b 100644 --- a/src/pkg/dashboard/static-src/manifest.txt +++ b/src/pkg/dashboard/static-src/manifest.txt @@ -75,6 +75,7 @@ js/tracker-words.js js/tracker-row-anchor.js js/tracker-row.js js/tracker-row-strip.js +js/tracker-workflows.js js/tracker-row-acted.js js/tracker-human-attention.js js/tracker-row-layout.js @@ -133,6 +134,7 @@ js/config-tabs.js js/config-hub.js js/config-agent.js js/config-governor.js +js/config-workflows.js js/config-watchdog-security.js js/config-features.js js/integration-execution-settings.js diff --git a/src/pkg/dashboard/static/index.html b/src/pkg/dashboard/static/index.html index 99c5edaa84..6df9a5d84e 100644 --- a/src/pkg/dashboard/static/index.html +++ b/src/pkg/dashboard/static/index.html @@ -1609,6 +1609,13 @@ .ham-group-n, .ham-group-attn { flex: 0 0 auto; font-size: var(--fs-xs); color: var(--text-faint); white-space: nowrap; } .ham-group-attn { color: var(--amber); } .ham-group-collapsed .ham-group-name { color: var(--muted); } + /* Workflow-model columns (RFC 17620): the model badge on a + bound repository's group header, its own column header, and the + "Not covered" chevron for a stage the model has no state for. */ + .ham-wf-badge { flex: 0 0 auto; font-family: var(--font-mono); font-size: var(--fs-2xs); font-weight: 700; letter-spacing: 0.02em; padding: var(--sp-1) var(--sp-3); border-radius: var(--r-pill); white-space: nowrap; color: var(--indigo); background: color-mix(in srgb, var(--indigo) 12%, transparent); border: 1px solid color-mix(in srgb, var(--indigo) 40%, transparent); } + .ham-wf-cols { position: static; } + .ham-wf-cols .ham-wf-col { color: var(--indigo); } + .ham-seg.ham-wf-uncovered { background: color-mix(in srgb, var(--amber) 18%, transparent); color: var(--amber); outline: 1px dashed var(--amber); } .ham-group-dist { flex: 0 0 auto; margin-left: auto; margin-right: calc(4em + var(--sp-3)); display: inline-flex; align-items: flex-end; gap: var(--sp-1); height: 0.9em; } .ham-gd { display: inline-block; width: 0.45em; height: 15%; border-radius: var(--sp-1); background: color-mix(in srgb, var(--text-faint) 35%, transparent); } .ham-gd-1 { height: 35%; background: color-mix(in srgb, var(--amber) 55%, transparent); } @@ -19540,6 +19547,83 @@

    Where to get help

    const rm = item && item.rebase_merge; return !!(rm && (rm.state === 'approved' || rm.state === 'rebasing' || rm.state === 'ci-pending')); } + /* ---- Workflow-model columns (hivecommons/hive#17620, RFC phase 2) ----- + GET /api/stage-tracker carries `workflows` only when at least one + repository is bound to a model other than hive-default. For such a + repository the group header names the model and draws the model's own + column header, and each row's strip is one chevron per model column, + placed by the stage/NNN number the row wears. Everything else on the + row (lead cell, chips, human gate, notes) is unchanged. Hives with no + binding never reach this code: stageTrackerWorkflowOf returns null and + the built-in strip renders exactly as before. Pure helpers so the node + harness (stage_tracker_workflows_ui_test.go) can run them. */ + const STAGE_TRACKER_WORKFLOW_UNCOVERED_WORD = 'Not covered'; + + // The model block for repo, or null when the repo follows hive-default + // (unbound, bound to hive-default itself, or bound to a model the + // server reported as unusable — the row then falls back to the + // built-in strip, which is what the engine does too). + function stageTrackerWorkflowOf(ctx, repo) { + const w = ctx && ctx.workflows; + if (!w || !w.repos) return null; + const wf = w.repos[String(repo || '')]; + if (!wf || wf.error || !Array.isArray(wf.columns) || !wf.columns.length) return null; + if (w.default && wf.name === w.default.name && wf.version === w.default.version) return null; + return wf; + } + + // Index of the column holding stage number `stage`; 0 for an unstaged + // item (the first column is the model's entry), the terminal column for + // a terminal stage, -1 when no column anchors the number (the model + // does not cover it: the engine abstained with "workflow: not covered"). + function stageTrackerWorkflowColumnIndex(wf, stage) { + const n = Number(stage) || 0; + const cols = wf.columns; + if (n === 0) return 0; + if (n >= (typeof STAGE_TRACKER_TERMINAL === 'number' ? STAGE_TRACKER_TERMINAL : 110)) { + const t = cols.findIndex(c => c.terminal); + return t >= 0 ? t : cols.length - 1; + } + return cols.findIndex(c => Array.isArray(c.numbers) && c.numbers.indexOf(n) >= 0); + } + + function stageTrackerWorkflowBadgeHtml(wf) { + if (!wf) return ''; + const key = `${wf.name}@${wf.version}`; + return `${esc(key)}`; + } + + // The model's column header, drawn under a bound repository's group + // header in place of the shared hive-default header. + function stageTrackerWorkflowHeaderHtml(wf) { + const cols = wf.columns.map(c => `
  • ${esc(c.label)}
  • `).join(''); + return ``; + } + + // One row's strip under its model: done before the anchored column, + // active at it (the stage's own word from `stages`, or the column + // label), todo after. An uncovered stage lights the first column amber + // with the "Not covered" word so the gap is visible, not silent. + function stageTrackerWorkflowStripItems(item, wf, stages) { + const cur = Number(item.stage) || 0; + const terminal = cur >= (typeof STAGE_TRACKER_TERMINAL === 'number' ? STAGE_TRACKER_TERMINAL : 110); + let at = stageTrackerWorkflowColumnIndex(wf, cur); + const uncovered = at < 0; + if (uncovered) at = 0; + const st = (stages || []).find(s => Number(s.number) === cur) || {}; + return wf.columns.map((c, i) => { + let state = 'todo'; + if (terminal || i < at) state = 'done'; + else if (i === at) state = 'active'; + if (terminal && c.terminal) state = 'active'; + const word = state !== 'active' ? c.label : (uncovered ? STAGE_TRACKER_WORKFLOW_UNCOVERED_WORD : (st.short || st.label || c.label)); + const tip = state === 'active' && uncovered + ? `stage/${String(cur).padStart(3, '0')} is not a state of ${wf.name}@${wf.version}; the engine leaves this item where it is.` + : (c.hint || c.label) + (state === 'active' && cur ? ` — stage/${String(cur).padStart(3, '0')}` : ''); + const cls = ['ham-seg', 'ham-wf-seg', state, c.terminal ? 'cap' : '', uncovered && state === 'active' ? 'ham-wf-uncovered' : ''].filter(Boolean).join(' '); + return `
  • ${esc(word)}
  • `; + }).join(''); + } const STAGE_TRACKER_ACTED_STATES = ['waiting', 'addressed', 'stuck', 'pending', 'failed', 'executing', 'blocked']; const STAGE_TRACKER_ACTED_FILTERS = ['any', 'awaiting', 'acted', 'stuck']; const STAGE_TRACKER_ACTED_STUCK_DEFAULT_S = 30 * 60; @@ -20485,6 +20569,7 @@

    Where to get help

    // needs-attention count and a mini per-column distribution. Keyed // `group:owner/repo` by the render so the reconcile keeps it stable. function stageTrackerGroupHeaderHtml(g, stages, ctx) { + const groupWorkflow = typeof stageTrackerWorkflowOf === 'function' ? stageTrackerWorkflowOf(ctx, g.repo) : null; const max = Math.max(1, ...g.cols); const dist = g.cols.map((c, i) => ``).join(''); const attn = g.attention ? `${g.attention} need${g.attention === 1 ? 's' : ''} attention` : ''; @@ -20492,10 +20577,10 @@

    Where to get help

    const chipCtx = { repoChips: Object.assign({}, (ctx && ctx.repoChips) || {}) }; return `
    - ${stageTrackerRepoChipHtml(g.repo, chipCtx)}${esc(g.repo)} + ${stageTrackerRepoChipHtml(g.repo, chipCtx)}${esc(g.repo)}${groupWorkflow ? stageTrackerWorkflowBadgeHtml(groupWorkflow) : ''} ${g.count} ${g.count === 1 ? 'row' : 'rows'}${attn} - ${dist} -
    `; + ${groupWorkflow ? '' : `${dist}`} + ${groupWorkflow && !g.collapsed ? stageTrackerWorkflowHeaderHtml(groupWorkflow) : ''}`; } // Row = title line (number, title, chips, since) + thin strip. Non-active @@ -20616,6 +20701,9 @@

    Where to get help

    // One note line per row: the human line takes the stage note (folded // or dropped), so the event slot below the caption renders nothing. const rowKey = stageTrackerItemKey(item); + // A repository bound to a workflow model draws the model's columns + // (tracker-workflows.js); null keeps the built-in strip byte for byte. + const rowWorkflow = typeof stageTrackerWorkflowOf === 'function' ? stageTrackerWorkflowOf(ctx, item.repo) : null; const humanNote = typeof stageTrackerHumanContextHtml === 'function' ? stageTrackerHumanContextHtml(item, stages, nowMs, subRows, ctx, rowKey) : ''; const noteCtx = humanNote && typeof stageTrackerNoteTakenCtx === 'function' ? stageTrackerNoteTakenCtx(ctx, rowKey) : ctx; // One human-gate pill per gated row (hivecommons/hive#17470): it @@ -20630,7 +20718,7 @@

    Where to get help

    ${typeof stageTrackerPreparationNodeHtml === 'function' ? stageTrackerPreparationNodeHtml(item, stages, ctx) : ''} -
      ${stageTrackerStripItems(shown, stages, range, nowMs, sharedUnder ? Object.assign({}, ctx, { gateElsewhere: true }) : ctx)}
    +
      ${rowWorkflow ? stageTrackerWorkflowStripItems(shown, rowWorkflow, stages) : stageTrackerStripItems(shown, stages, range, nowMs, sharedUnder ? Object.assign({}, ctx, { gateElsewhere: true }) : ctx)}
    ${stageTrackerSinceHtml(shown, nowMs)}${typeof stageTrackerPillHtml === 'function' ? stageTrackerPillHtml(shown, stages, range, nowMs, ctx) : ''}
    ${humanNote}${stageTrackerCurrentStageHtml(shown, stages, range, nowMs)}${typeof stageTrackerPreparationCommentHtml === 'function' ? stageTrackerPreparationCommentHtml(shown, noteCtx, rowKey) : typeof stageTrackerEventHtml === 'function' ? stageTrackerEventHtml(rowKey, noteCtx) : ''}${typeof stageTrackerExecutionHtml === 'function' ? stageTrackerExecutionHtml(shown) : ''}${typeof stageTrackerReviewNotesHtml === 'function' ? stageTrackerReviewNotesHtml(item, ctx) : ''}${typeof stageTrackerNotesPanelHtml === 'function' ? stageTrackerNotesPanelHtml(item, hamNotesCtx(item)) : ''}${subs} `; @@ -23095,6 +23183,8 @@

    All labels

    } stageTrackerSyncToolbar(f); const ctx = { nowMs, stages, moves: _stageTrackerMoves }; + // Workflow-model columns for bound repositories (hivecommons/hive#17620). + ctx.workflows = d.workflows || null; ctx.workers = stageTrackerWorkers(window._lastStatus); ctx.events = _stageTrackerEvents; ctx.noteTuck = stageTrackerNoteTuckSetting(); @@ -61614,7 +61704,7 @@

    Clanker-requested PRs

    `; // Fully forge-qualified so the row is unambiguous: github.ibm.com/org/repo. const displayName = ghHost + '/' + r; - return `
    ${isDefault ? '★' : '☆'}${esc(displayName)}${inheritButton}${rtRow}${cbRow}${pillLabel}
    ` + ldSub; + return `
    ${isDefault ? '★' : '☆'}${esc(displayName)}${inheritButton}${rtRow}${cbRow}${pillLabel}
    ` + ldSub + renderRepoWorkflowRow(r); }).join(''); const hostNote = isGHE ? ` on ${esc(ghHost)} (GitHub Enterprise)` @@ -62328,6 +62418,102 @@

    Agent Watchdog

    — fixed progression; the last step is the cap. `; } + // Workflow-model bindings (hivecommons/hive#17620, RFC phase 2). + // + // Each repo row on Settings → Repos carries a "Workflow" select: hive-default + // (the built-in lifecycle every unbound repo follows) or any compiled model + // from the library. Binding persists immediately through + // PUT /api/workflows/bindings (owner-only, audited) — it is not part of the + // tab's Save button, because a binding is validated server-side against the + // store (the model must exist and compile) and the tracker picks the new + // columns up on its next poll. + const WORKFLOW_DEFAULT_KEY = 'hive-default@1'; + const _workflowState = { loaded: false, loading: false, enabled: false, workflows: [], bindings: [] }; + + function workflowKey(name, version) { return (name || '') + '@' + (version || ''); } + + async function loadWorkflowLibrary(onDone) { + if (_workflowState.loading) return; + _workflowState.loading = true; + try { + const res = await fetch('/api/workflows', { headers: _inceptionAuthHeaders() }); + if (res.status === 503) { _workflowState.loaded = true; _workflowState.workflows = []; return; } + if (!res.ok) throw new Error(await saveErrorMessage(res)); + const data = await res.json(); + _workflowState.workflows = Array.isArray(data.workflows) ? data.workflows : []; + _workflowState.bindings = Array.isArray(data.bindings) ? data.bindings : []; + _workflowState.enabled = !!data.enabled; + _workflowState.loaded = true; + } catch (err) { + _workflowState.loaded = true; + showToast('Could not load the workflow library: ' + err.message, 'error'); + } finally { + _workflowState.loading = false; + if (typeof onDone === 'function') onDone(); + } + } + + function workflowBindingFor(repo) { + const want = String(repo || '').toLowerCase(); + const org = String((_configState.data && _configState.data.org) || '').toLowerCase(); + return _workflowState.bindings.find(b => { + const r = String(b.repo || '').toLowerCase(); + return r === want || (org && (r === org + '/' + want || want === org + '/' + r)); + }) || null; + } + + // renderRepoWorkflowRow returns the per-repo select, or '' when the hive + // has no workflow store (the row simply does not appear; nothing else on + // the tab changes). + function renderRepoWorkflowRow(repo) { + if (!_workflowState.loaded) { + loadWorkflowLibrary(() => { if (_configState.tab === 'Repos') renderConfigTab('Repos'); }); + return ''; + } + if (!_workflowState.workflows.length) return ''; + const bound = workflowBindingFor(repo); + const current = bound ? workflowKey(bound.name, bound.version) : WORKFLOW_DEFAULT_KEY; + const locked = !dashboardRoleAtLeast(window._hiveRole || 'read', 'owner'); + const options = _workflowState.workflows.map(w => { + const key = workflowKey(w.name, w.version); + const compiled = w.compiled !== false; + const label = key + (w.description ? ' — ' + w.description : '') + (compiled ? '' : ' (draft, does not compile)'); + return ``; + }).join(''); + const flag = _workflowState.enabled + ? '' + : 'derivation off'; + return `
    ${flag}${bound ? `bound` : 'default'}
    `; + } + + async function setRepoWorkflow(repo, value) { + if (!dashboardRoleAtLeast(window._hiveRole || 'read', 'owner')) { + showToast('Owner role is required to bind a workflow.', 'error'); + renderConfigTab('Repos'); + return; + } + const key = String(value || ''); + const body = { repo: repo }; + if (key && key !== WORKFLOW_DEFAULT_KEY) { + const at = key.lastIndexOf('@'); + body.name = key.slice(0, at); + body.version = key.slice(at + 1); + } + try { + const res = await fetch('/api/workflows/bindings', { + method: 'PUT', + headers: _inceptionAuthHeaders(), + body: JSON.stringify(body), + }); + if (!res.ok) throw new Error(await saveErrorMessage(res)); + const data = await res.json(); + _workflowState.bindings = Array.isArray(data.bindings) ? data.bindings : []; + showToast(body.name ? `${repo} now follows ${key}` : `${repo} back on hive-default`); + } catch (err) { + showToast('Failed to bind workflow: ' + err.message, 'error'); + } + renderConfigTab('Repos'); + } // ── Watchdog activity strip (#7254) ────────────────────────────────────── // "Review the watchdog-*-observed entries in the Audit Log, then switch to // Heal" is only a data-driven decision if the data is on the page. The diff --git a/src/pkg/stageengine/export_parity_test.go b/src/pkg/stageengine/export_parity_test.go new file mode 100644 index 0000000000..c00297d449 --- /dev/null +++ b/src/pkg/stageengine/export_parity_test.go @@ -0,0 +1,44 @@ +package stageengine + +import ( + "github.com/hivecommons/hive/pkg/claims" + "github.com/hivecommons/hive/pkg/stagelabels" +) + +// ParityFacts hands the external parity test (workflow_parity_test.go, +// package stageengine_test — it imports hivedefault, which imports this +// package, so it cannot live here) the Facts corpus the built-in derivers +// are pinned by: every generated PR stage golden plus a sweep of issue +// facts reaching each issue stage. hivecommons/hive#17620. +func ParityFacts() []Facts { + var out []Facts + for _, cases := range generateStageGoldens() { + for _, c := range cases { + _, f := runStageGoldenWithFacts(c.Input) + out = append(out, f) + } + } + pr99 := &Ref{Owner: "acme", Repo: "widgets", Number: 99, Kind: stagelabels.PRItem} + for _, mut := range []func(f *Facts){ + nil, + func(f *Facts) { f.Issue.Bucket = BucketActionable; f.Park.NeedsHuman = true }, + func(f *Facts) { f.Issue.TriageAwaiting = "carries needs-triage" }, + func(f *Facts) { + f.Issue.Bucket = BucketClaimed + f.Issue.Claim = &ClaimFact{Holder: "bob", Kind: "human"} + }, + func(f *Facts) { + f.Issue.Bucket = BucketClaimed + f.Issue.Claim = &ClaimFact{Kind: claims.KindAgent, PRStarted: true} + }, + func(f *Facts) { f.Issue.Bucket = BucketCoveredByPR; f.Issue.CoveringPR = pr99 }, + func(f *Facts) { f.Park.NeedsHuman = true }, + func(f *Facts) { f.Park.PossiblyExists = true }, + func(f *Facts) { f.Issue.DiscussionLabel = "proposal" }, + func(f *Facts) { f.Open = false }, + func(f *Facts) { f.Issue.Bucket = BucketFiltered }, + } { + out = append(out, derivedIssueFacts(mut)) + } + return out +} diff --git a/src/pkg/stageengine/reconciler.go b/src/pkg/stageengine/reconciler.go index dd5c0dde0d..9acbf0b32e 100644 --- a/src/pkg/stageengine/reconciler.go +++ b/src/pkg/stageengine/reconciler.go @@ -10,6 +10,7 @@ import ( "time" "github.com/hivecommons/hive/pkg/stagelabels" + "github.com/hivecommons/hive/pkg/workflowmodel" ) // Transition is what Reconcile reports to its caller and the audit log. @@ -38,14 +39,23 @@ type Reconciler interface { // Options configures New. type Options struct { - Mode Mode - Store *stagelabels.Store // stagelabels.Shared() by default; API unchanged - Writer StageWriter - Issue Deriver // defaults to DeriveIssue - PR Deriver // defaults to DerivePR - Now func() time.Time - Logger *slog.Logger - Metrics Metrics + Mode Mode + Store *stagelabels.Store // stagelabels.Shared() by default; API unchanged + Writer StageWriter + Issue Deriver // defaults to DeriveIssue + PR Deriver // defaults to DerivePR + // Workflow, when set, derives through the compiled workflow model + // (DeriveFromModel) and overrides Issue and PR. hivecommons/hive#17620. + Workflow *workflowmodel.Profile + // WorkflowFor, when set, resolves the compiled model bound to a + // repository (owner/repo) at derivation time; nil means the repository + // is unbound and the Issue/PR derivers run unchanged. It composes with + // Workflow: Workflow is the engine-wide model, WorkflowFor the + // per-repository override. hivecommons/hive#17620 (RFC phase 2). + WorkflowFor func(owner, repo string) *workflowmodel.Profile + Now func() time.Time + Logger *slog.Logger + Metrics Metrics } // ErrNoWriter is returned by an enforce-mode Reconcile with no Writer. @@ -73,12 +83,20 @@ func New(o Options) Reconciler { if o.Store == nil { o.Store = stagelabels.Shared() } + if o.Workflow != nil { + o.Issue = DeriveFromModel(o.Workflow, stagelabels.IssueItem) + o.PR = DeriveFromModel(o.Workflow, stagelabels.PRItem) + } if o.Issue == nil { o.Issue = IssueDeriver() } if o.PR == nil { o.PR = PRDeriver() } + if o.WorkflowFor != nil { + o.Issue = DeriveForRepo(o.WorkflowFor, o.Issue) + o.PR = DeriveForRepo(o.WorkflowFor, o.PR) + } if o.Now == nil { o.Now = time.Now } diff --git a/src/pkg/stageengine/workflow.go b/src/pkg/stageengine/workflow.go new file mode 100644 index 0000000000..5807d69ebd --- /dev/null +++ b/src/pkg/stageengine/workflow.go @@ -0,0 +1,337 @@ +package stageengine + +import ( + "fmt" + "strings" + "sync" + + "github.com/hivecommons/hive/pkg/authorkind" + "github.com/hivecommons/hive/pkg/stagelabels" + "github.com/hivecommons/hive/pkg/workflowmodel" +) + +// DeriveForRepo returns a Deriver of fallback's kind that derives through +// the model lookup returns for the item's repository, or through fallback +// when lookup returns nil. Profiles are compiled per lookup result and +// cached by hash, so a rebinding takes effect on the next derivation. +func DeriveForRepo(lookup func(owner, repo string) *workflowmodel.Profile, fallback Deriver) Deriver { + if lookup == nil || fallback == nil { + return fallback + } + kind := fallback.Kind() + var mu sync.Mutex + cache := map[string]Deriver{} + return DeriverFunc(kind, func(f Facts) Decision { + p := lookup(f.Ref.Owner, f.Ref.Repo) + if p == nil { + return fallback.Derive(f) + } + mu.Lock() + d, ok := cache[p.Hash] + if !ok { + d = DeriveFromModel(p, kind) + cache[p.Hash] = d + } + mu.Unlock() + return d.Derive(f) + }) +} + +// DeriveFromModel is the third Deriver (hivecommons/hive#17620 "One +// engine, one writer"): it derives an item's stage under a compiled +// workflow model and produces the same Decision the built-in derivers do, +// so the Reconciler stays the only label writer. +// +// Under derivation mode builtin the built-in deriver for kind decides and +// the model only has to cover the stage it produced; a stage the model has +// no state for is "not covered" and the deriver abstains (fail closed, +// never a silent fall-through). Under mode rules the model's ordered rules +// are evaluated over Facts; the first rule whose predicates all hold names +// the state, and an item no rule matches is not covered. +func DeriveFromModel(p *workflowmodel.Profile, kind stagelabels.Item) Deriver { + if p == nil { + return builtinDeriver(kind) + } + return modelDeriver{p: p, kind: kind, inner: builtinDeriver(kind)} +} + +func builtinDeriver(kind stagelabels.Item) Deriver { + if kind == stagelabels.PRItem { + return prDeriverF + } + return issueDeriverF +} + +type modelDeriver struct { + p *workflowmodel.Profile + kind stagelabels.Item + inner Deriver +} + +func (d modelDeriver) Kind() stagelabels.Item { return d.kind } + +// ReasonNotCovered prefixes every abstention the model forces. +const ReasonNotCovered = "workflow: not covered" + +func (d modelDeriver) Derive(f Facts) Decision { + if f.Ref.Kind != d.kind { + return abstain(reasonKindMismatch) + } + switch d.p.Mode { + case workflowmodel.DerivationBuiltin: + dec := d.inner.Derive(f) + if dec.Abstain || dec.Stage == stagelabels.None { + return dec + } + if _, ok := d.p.StateForNumber(dec.Stage.Number()); !ok { + return abstain(fmt.Sprintf("%s: %s@%s has no state for stage %s", ReasonNotCovered, d.p.Name, d.p.Version, dec.Stage)) + } + return dec + case workflowmodel.DerivationRules: + return d.rules(f) + } + return abstain(fmt.Sprintf("%s: %s@%s has derivation mode %q", ReasonNotCovered, d.p.Name, d.p.Version, d.p.Mode)) +} + +// rules evaluates the profile's rules in order. +func (d modelDeriver) rules(f Facts) Decision { + want := workflowmodel.KindIssue + if d.kind == stagelabels.PRItem { + want = workflowmodel.KindPR + } + for i, r := range d.p.Rules { + if r.Kind != workflowmodel.KindAny && r.Kind != want { + continue + } + if !holdsAll(f, r.When) { + continue + } + id := r.ID + if id == "" { + id = fmt.Sprintf("rules[%d]", i) + } + reason := r.Reason + if reason == "" { + reason = fmt.Sprintf("workflow %s@%s: rule %s", d.p.Name, d.p.Version, id) + } + return d.decisionFor(r.State, reason, f) + } + return abstain(fmt.Sprintf("%s: %s@%s has no rule for this %s", ReasonNotCovered, d.p.Name, d.p.Version, want)) +} + +// decisionFor turns a model state into a Decision: its stage number maps +// to the stage label, a terminal state with no number is Stage None. +func (d modelDeriver) decisionFor(state, reason string, f Facts) Decision { + st := d.p.States[state] + if st == nil { + return abstain(fmt.Sprintf("%s: state %q is not in %s@%s", ReasonNotCovered, state, d.p.Name, d.p.Version)) + } + dec := Decision{Reason: reason, Kind: f.Author.Kind, Role: f.Author.Role} + if st.Lifecycle.Number == 0 { + if d.p.Terminal[state] { + dec.Terminal = true + return dec + } + return abstain(fmt.Sprintf("%s: state %q carries no stage number and is not terminal", ReasonNotCovered, state)) + } + stage, ok := stageByNumber(st.Lifecycle.Number) + if !ok { + return abstain(fmt.Sprintf("%s: state %q stage number %d is not in the stage table", ReasonNotCovered, state, st.Lifecycle.Number)) + } + if !stage.Applies(d.kind) { + return abstain(fmt.Sprintf("%s: state %q stage number %d is not a %s stage", ReasonNotCovered, state, st.Lifecycle.Number, d.kind)) + } + dec.Stage = stage + if f.PR != nil { + dec.DCO = f.PR.DCO + } + return dec +} + +func stageByNumber(n int) (stagelabels.Stage, bool) { + for _, def := range stagelabels.Table { + if def.Number == n { + return def.Stage, true + } + } + return stagelabels.None, false +} + +func holdsAll(f Facts, preds []workflowmodel.Predicate) bool { + for _, p := range preds { + if !holds(f, p) { + return false + } + } + return true +} + +// holds evaluates one predicate from the workflowmodel.Facts vocabulary. +// An unknown fact or a mismatched operator is false: the compiler has +// already refused such a model, so this is defence in depth. +func holds(f Facts, p workflowmodel.Predicate) bool { + switch workflowmodel.Facts[p.Fact] { + case workflowmodel.FactBool: + v, ok := boolFact(f, p.Fact) + if !ok { + return false + } + return (p.Op == workflowmodel.OpTrue) == v + case workflowmodel.FactScalar: + v, ok := scalarFact(f, p.Fact) + if !ok { + return false + } + eq := strings.EqualFold(v, p.Value) + return (p.Op == workflowmodel.OpIs) == eq + case workflowmodel.FactSet: + set, ok := setFact(f, p.Fact) + if !ok { + return false + } + has := false + for _, s := range set { + if matchGlob(p.Value, s) { + has = true + break + } + } + return (p.Op == workflowmodel.OpHas) == has + } + return false +} + +// matchGlob matches value against pattern, where a trailing `*` matches +// any suffix. Comparison is case-insensitive, as labels are. +func matchGlob(pattern, value string) bool { + if strings.HasSuffix(pattern, "*") { + return strings.HasPrefix(strings.ToLower(value), strings.ToLower(strings.TrimSuffix(pattern, "*"))) + } + return strings.EqualFold(pattern, value) +} + +func boolFact(f Facts, name string) (bool, bool) { + switch name { + case "open": + return f.Open, true + case "merged": + return f.Merged, true + case "draft": + return f.Draft, true + case "hold": + return len(f.Park.Holds) > 0, true + case "park": + return f.Park.NeedsHuman || f.Park.DecisionLabel != "" || f.Park.PossiblyExists, true + case "issue.claimed": + return f.Issue != nil && f.Issue.Claim != nil, true + case "issue.covered": + return f.Issue != nil && f.Issue.CoveringPR != nil, true + case "issue.stalled": + return f.Issue != nil && f.Issue.Stalled, true + case "issue.open_children": + return f.Issue != nil && f.Issue.OpenChildren > 0, true + case "pr.changes_requested": + return f.PR != nil && f.PR.ChangesRequested, true + case "pr.conflicting": + return f.PR != nil && f.PR.Conflicting, true + case "pr.head_fresh": + return f.PR != nil && f.PR.HeadFresh, true + } + return false, false +} + +// Scalar fact values. pr.checks: green | red | pending | missing | unknown. +// pr.review: pending | approved | changes_requested | unavailable. +// pr.dco: yes | no | unknown. author.kind: human | hive | clanker | unknown. +func scalarFact(f Facts, name string) (string, bool) { + switch name { + case "origin": + return f.Origin, true + case "author.kind": + return authorKindName(f.Author.Kind), true + case "author.login": + return f.Author.Login, true + case "issue.bucket": + if f.Issue == nil { + return "", true + } + return string(f.Issue.Bucket), true + case "pr.checks": + if f.PR == nil { + return "", true + } + return checksName(f.PR.Checks), true + case "pr.review": + if f.PR == nil { + return "", true + } + return reviewName(f.PR.Review.State), true + case "pr.dco": + if f.PR == nil { + return "", true + } + return dcoName(f.PR.DCO), true + case "pr.base": + if f.PR == nil { + return "", true + } + return f.PR.BaseRef, true + case "pipeline.status": + // Post-merge runs are not observed yet (see workflowmodel.Facts). + return "unknown", true + } + return "", false +} + +func setFact(f Facts, name string) ([]string, bool) { + if name == "labels" { + return f.Labels, true + } + return nil, false +} + +func authorKindName(k authorkind.Kind) string { + switch k { + case authorkind.Human: + return "human" + case authorkind.Hive: + return "hive" + case authorkind.Clanker: + return "clanker" + } + return "unknown" +} + +func checksName(c stagelabels.CheckSummary) string { + switch { + case !c.Known: + return "unknown" + case len(c.Red) > 0: + return "red" + case c.Pending > 0: + return "pending" + } + return "green" +} + +func reviewName(s stagelabels.VerdictState) string { + switch s { + case stagelabels.VerdictApproved: + return "approved" + case stagelabels.VerdictChangesRequested: + return "changes_requested" + case stagelabels.VerdictUnavailable: + return "unavailable" + } + return "pending" +} + +func dcoName(d stagelabels.DCOState) string { + switch d { + case stagelabels.DCOYes: + return "yes" + case stagelabels.DCONo: + return "no" + } + return "unknown" +} diff --git a/src/pkg/stageengine/workflow_parity_test.go b/src/pkg/stageengine/workflow_parity_test.go new file mode 100644 index 0000000000..665e6f16c0 --- /dev/null +++ b/src/pkg/stageengine/workflow_parity_test.go @@ -0,0 +1,147 @@ +package stageengine_test + +import ( + "reflect" + "strings" + "testing" + + "github.com/hivecommons/hive/pkg/stageengine" + "github.com/hivecommons/hive/pkg/stagelabels" + "github.com/hivecommons/hive/pkg/workflowmodel" + "github.com/hivecommons/hive/pkg/workflowmodel/hivedefault" +) + +// TestDeriveFromModelHiveDefaultParity is the RFC's P1 bar +// (hivecommons/hive#17620): under the generated hive-default model every +// Decision is identical to the built-in derivers', across the whole PR +// golden corpus and the issue stage sweep. +func TestDeriveFromModelHiveDefaultParity(t *testing.T) { + p := hivedefault.Profile() + derivers := map[stagelabels.Item][2]stageengine.Deriver{ + stagelabels.IssueItem: {stageengine.IssueDeriver(), stageengine.DeriveFromModel(p, stagelabels.IssueItem)}, + stagelabels.PRItem: {stageengine.PRDeriver(), stageengine.DeriveFromModel(p, stagelabels.PRItem)}, + } + facts := stageengine.ParityFacts() + if len(facts) < 100 { + t.Fatalf("parity corpus has only %d facts", len(facts)) + } + stages := map[stagelabels.Stage]bool{} + for i, f := range facts { + d := derivers[f.Ref.Kind] + want, got := d[0].Derive(f), d[1].Derive(f) + if !reflect.DeepEqual(want, got) { + t.Errorf("facts[%d] %s: built-in %+v, hive-default %+v", i, f.Ref.Key(), want, got) + } + if !want.Abstain && !want.Terminal { + stages[want.Stage] = true + } + } + if len(stages) < 15 { + t.Errorf("corpus reached only %d stages: %v", len(stages), stages) + } +} + +// A nil profile is the built-in deriver, and a model that is missing a +// stage the built-in deriver produced abstains "not covered" rather than +// writing a label the model does not know. +func TestDeriveFromModelCoverage(t *testing.T) { + f := stageengine.ParityFacts()[0] + if f.Ref.Kind != stagelabels.PRItem { + t.Fatalf("expected a PR fixture first, got %s", f.Ref.Kind) + } + nilD := stageengine.DeriveFromModel(nil, stagelabels.PRItem) + if got, want := nilD.Derive(f), stageengine.PRDeriver().Derive(f); !reflect.DeepEqual(got, want) { + t.Fatalf("nil profile: %+v != %+v", got, want) + } + if stageengine.DeriveFromModel(nil, stagelabels.IssueItem).Derive(f).Abstain != true { + t.Fatal("kind mismatch must abstain") + } + + base := stageengine.PRDeriver().Derive(f) + if base.Abstain || base.Terminal { + t.Skipf("first fixture is %+v; coverage case needs a staged PR", base) + } + m := hivedefault.Generate() + drop := hivedefault.StateID(base.Stage) + var kept []workflowmodel.State + for _, s := range m.Spec.States { + if s.ID != drop { + kept = append(kept, s) + } + } + m.Spec.States = kept + var tr []workflowmodel.Transition + for _, x := range m.Spec.Transitions { + if x.From != drop && x.To != drop { + tr = append(tr, x) + } + } + m.Spec.Transitions = tr + for i := range m.Spec.Columns { + var st []string + for _, s := range m.Spec.Columns[i].States { + if s != drop { + st = append(st, s) + } + } + m.Spec.Columns[i].States = st + } + for i := range m.Spec.Agents { + var st []string + for _, s := range m.Spec.Agents[i].States { + if s != drop { + st = append(st, s) + } + } + m.Spec.Agents[i].States = st + } + p, _, err := workflowmodel.Compile(m) + if err != nil { + t.Fatalf("trimmed model: %v", err) + } + got := stageengine.DeriveFromModel(p, stagelabels.PRItem).Derive(f) + if !got.Abstain || !strings.HasPrefix(got.Reason, stageengine.ReasonNotCovered) { + t.Fatalf("missing state %s: got %+v", drop, got) + } +} + +// DeriveForRepo routes bound repositories through their model and leaves +// unbound repositories on the built-in deriver, decision for decision. +func TestDeriveForRepo(t *testing.T) { + p := hivedefault.Profile() + calls := 0 + lookup := func(owner, repo string) *workflowmodel.Profile { + calls++ + if repo == "bound" { + return p + } + return nil + } + if stageengine.DeriveForRepo(nil, stageengine.PRDeriver()) == nil || stageengine.DeriveForRepo(lookup, nil) != nil { + t.Fatal("nil handling") + } + derivers := map[stagelabels.Item][2]stageengine.Deriver{ + stagelabels.IssueItem: {stageengine.IssueDeriver(), stageengine.DeriveForRepo(lookup, stageengine.IssueDeriver())}, + stagelabels.PRItem: {stageengine.PRDeriver(), stageengine.DeriveForRepo(lookup, stageengine.PRDeriver())}, + } + if derivers[stagelabels.PRItem][1].Kind() != stagelabels.PRItem { + t.Fatal("kind") + } + for i, f := range stageengine.ParityFacts() { + for _, repo := range []string{"bound", "free"} { + f.Ref.Repo = repo + d := derivers[f.Ref.Kind] + if want, got := d[0].Derive(f), d[1].Derive(f); !reflect.DeepEqual(want, got) { + t.Errorf("facts[%d] %s/%s: built-in %+v, routed %+v", i, repo, f.Ref.Key(), want, got) + } + } + } + if calls == 0 { + t.Fatal("lookup never consulted") + } + // the engine composes it when Options.WorkflowFor is set + r := stageengine.New(stageengine.Options{WorkflowFor: lookup, Store: stagelabels.NewStore(t.TempDir()+"/stages.json", nil)}) + if r == nil { + t.Fatal("engine") + } +} diff --git a/src/pkg/stageengine/workflow_rules_test.go b/src/pkg/stageengine/workflow_rules_test.go new file mode 100644 index 0000000000..418a661983 --- /dev/null +++ b/src/pkg/stageengine/workflow_rules_test.go @@ -0,0 +1,211 @@ +package stageengine_test + +import ( + "encoding/json" + "os" + "strings" + "testing" + + "github.com/hivecommons/hive/pkg/authorkind" + "github.com/hivecommons/hive/pkg/stageengine" + "github.com/hivecommons/hive/pkg/stagelabels" + "github.com/hivecommons/hive/pkg/workflowmodel" +) + +func techdocsProfile(t *testing.T) *workflowmodel.Profile { + t.Helper() + b, err := os.ReadFile("../workflowmodel/testdata/models/techdocs.json") + if err != nil { + t.Fatal(err) + } + var m workflowmodel.Model + if err := json.Unmarshal(b, &m); err != nil { + t.Fatal(err) + } + p, diags, err := workflowmodel.Compile(&m) + if err != nil { + t.Fatalf("%v %+v", err, diags) + } + return p +} + +func issueFacts(labels ...string) stageengine.Facts { + return stageengine.Facts{ + Ref: stageengine.Ref{Owner: "cncf", Repo: "techdocs", Number: 1, Kind: stagelabels.IssueItem}, + Open: true, + Labels: labels, + Author: stageengine.AuthorFacts{Kind: authorkind.Human, Login: "nate"}, + Issue: &stageengine.IssueFacts{}, + } +} + +func prFacts(labels ...string) stageengine.Facts { + return stageengine.Facts{ + Ref: stageengine.Ref{Owner: "cncf", Repo: "techdocs", Number: 2, Kind: stagelabels.PRItem}, + Open: true, + Labels: labels, + Author: stageengine.AuthorFacts{Kind: authorkind.Hive}, + PR: &stageengine.PRFacts{Checks: stagelabels.CheckSummary{Known: true, Total: 2, Green: 2}, DCO: stagelabels.DCOYes}, + } +} + +// TestDeriveFromModelRules walks a TechDocs engagement through the rules +// deriver: ordered first match, kind filtering, glob labels, scalar +// facts, terminal states and the not-covered abstention. +func TestDeriveFromModelRules(t *testing.T) { + p := techdocsProfile(t) + issues := stageengine.DeriveFromModel(p, stagelabels.IssueItem) + prs := stageengine.DeriveFromModel(p, stagelabels.PRItem) + + cases := []struct { + name string + d stageengine.Deriver + f func() stageengine.Facts + stage stagelabels.Stage + term bool + reason string + }{ + {"request filed", issues, func() stageengine.Facts { return issueFacts("techdocs/request") }, stagelabels.Actionable, false, "workflow cncf-techdocs@1: rule requested"}, + {"accepted", issues, func() stageengine.Facts { return issueFacts("techdocs/request", "techdocs/accepted") }, stagelabels.Claimed, false, ""}, + {"analysis posted", issues, func() stageengine.Facts { return issueFacts("techdocs/accepted", "techdocs/analysis") }, stagelabels.IssueNeedsDecision, false, ""}, + {"covered by a draft PR", issues, func() stageengine.Facts { + f := issueFacts("techdocs/analysis") + f.Issue.CoveringPR = &stageengine.Ref{Owner: "cncf", Repo: "techdocs", Number: 2, Kind: stagelabels.PRItem} + return f + }, stagelabels.CoveredByPR, false, ""}, + {"closed unpublished", issues, func() stageengine.Facts { f := issueFacts("techdocs/request"); f.Open = false; return f }, stagelabels.None, true, "closed without publishing"}, + {"draft PR", prs, func() stageengine.Facts { f := prFacts("techdocs/project-x"); f.Draft = true; return f }, stagelabels.Fixing, false, ""}, + {"ready PR awaits review", prs, func() stageengine.Facts { return prFacts("TechDocs/project-x") }, stagelabels.ReviewPending, false, ""}, + {"approved green PR", prs, func() stageengine.Facts { + f := prFacts("techdocs/project-x") + f.PR.Review.State = stagelabels.VerdictApproved + return f + }, stagelabels.ReadyForHumanMerge, false, ""}, + {"approved but red stays in review", prs, func() stageengine.Facts { + f := prFacts("techdocs/project-x") + f.PR.Review.State = stagelabels.VerdictApproved + f.PR.Checks.Red = []string{"lint"} + return f + }, stagelabels.ReviewPending, false, ""}, + {"merged", prs, func() stageengine.Facts { f := prFacts(); f.Open, f.Merged = false, true; return f }, stagelabels.None, true, ""}, + } + for _, tc := range cases { + t.Run(tc.name, func(t *testing.T) { + got := tc.d.Derive(tc.f()) + if got.Abstain { + t.Fatalf("abstained: %s", got.Reason) + } + if got.Terminal != tc.term || got.Stage != tc.stage { + t.Fatalf("got %+v; want stage %s terminal %v", got, tc.stage, tc.term) + } + if tc.reason != "" && got.Reason != tc.reason { + t.Fatalf("reason %q, want %q", got.Reason, tc.reason) + } + if tc.f().Ref.Kind == stagelabels.PRItem && !tc.term && got.DCO != stagelabels.DCOYes { + t.Fatalf("DCO not carried: %+v", got) + } + }) + } + + t.Run("no rule matches", func(t *testing.T) { + got := issues.Derive(issueFacts("bug")) + if !got.Abstain || !strings.HasPrefix(got.Reason, stageengine.ReasonNotCovered) { + t.Fatalf("%+v", got) + } + }) + t.Run("kind mismatch", func(t *testing.T) { + if got := prs.Derive(issueFacts("techdocs/request")); !got.Abstain { + t.Fatalf("%+v", got) + } + }) + t.Run("stage numbers follow the model", func(t *testing.T) { + if got := issues.Derive(issueFacts("techdocs/request")); got.Stage.Number() != 10 { + t.Fatalf("%+v", got) + } + }) + t.Run("issue-side number on a PR state is not covered", func(t *testing.T) { + q := *p + q.Rules = []workflowmodel.Rule{{State: "requested", When: []workflowmodel.Predicate{{Fact: "open", Op: workflowmodel.OpTrue}}}} + got := stageengine.DeriveFromModel(&q, stagelabels.PRItem).Derive(prFacts()) + if !got.Abstain || !strings.Contains(got.Reason, "not a pr stage") { + t.Fatalf("%+v", got) + } + }) +} + +// Every fact in the vocabulary evaluates; none is silently false because +// the evaluator forgot it. +func TestRulesFactVocabularyIsComplete(t *testing.T) { + p := techdocsProfile(t) + f := prFacts("x/y") + f.PR.Review.State = stagelabels.VerdictChangesRequested + f.PR.BaseRef = "main" + f.PR.ChangesRequested, f.PR.Conflicting, f.PR.HeadFresh = true, true, true + f.Draft, f.Merged = true, true + f.Park.Holds = []string{"hold"} + f.Park.NeedsHuman = true + f.Origin = "origin/scanner" + f.Author.Login = "someone" + fi := issueFacts("x/y") + fi.Issue.Bucket = stageengine.BucketClaimed + fi.Issue.Claim = &stageengine.ClaimFact{} + fi.Issue.CoveringPR = &stageengine.Ref{} + fi.Issue.Stalled = true + fi.Issue.OpenChildren = 1 + + for name, ft := range workflowmodel.Facts { + var pos, neg workflowmodel.Op + value := "" + switch ft { + case workflowmodel.FactBool: + pos, neg = workflowmodel.OpTrue, workflowmodel.OpFalse + case workflowmodel.FactScalar: + pos, neg = workflowmodel.OpIs, workflowmodel.OpNot + value = scalarValue(name) + case workflowmodel.FactSet: + pos, neg = workflowmodel.OpHas, workflowmodel.OpLacks + value = "x/*" + } + facts, state := f, "drafting" + if strings.HasPrefix(name, "issue.") { + facts, state = fi, "requested" + } + rp := *p + rp.Rules = []workflowmodel.Rule{{ID: "t", State: state, When: []workflowmodel.Predicate{{Fact: name, Op: pos, Value: value}}}} + rn := *p + rn.Rules = []workflowmodel.Rule{{ID: "t", State: state, When: []workflowmodel.Predicate{{Fact: name, Op: neg, Value: value}}}} + gp := stageengine.DeriveFromModel(&rp, facts.Ref.Kind).Derive(facts) + gn := stageengine.DeriveFromModel(&rn, facts.Ref.Kind).Derive(facts) + if gp.Abstain == gn.Abstain { + t.Errorf("fact %q: %s → %v, %s → %v (one must match)", name, pos, gp.Abstain, neg, gn.Abstain) + } + if gp.Abstain { + t.Errorf("fact %q: positive predicate %s %q did not hold on %+v", name, pos, value, facts) + } + } +} + +func scalarValue(fact string) string { + switch fact { + case "origin": + return "origin/scanner" + case "author.kind": + return "hive" + case "author.login": + return "someone" + case "issue.bucket": + return string(stageengine.BucketClaimed) + case "pr.checks": + return "green" + case "pr.review": + return "changes_requested" + case "pr.dco": + return "yes" + case "pipeline.status": + // Post-merge runs are not observed yet; the engine reports unknown. + return "unknown" + case "pr.base": + return "main" + } + return "?" +} diff --git a/src/pkg/workflowmodel/compile.go b/src/pkg/workflowmodel/compile.go new file mode 100644 index 0000000000..ce30c1ac6f --- /dev/null +++ b/src/pkg/workflowmodel/compile.go @@ -0,0 +1,469 @@ +package workflowmodel + +import ( + "errors" + "fmt" + "sort" + "strings" +) + +// Diagnostic is one compiler finding. Errors refuse the model; warnings +// are shown in the editor and on the monitor but compile. +type Diagnostic struct { + Code string `json:"code"` + Severity string `json:"severity"` + Subject string `json:"subject,omitempty"` + Detail string `json:"detail"` +} + +// Diagnostic severities. +const ( + DiagError = "error" + DiagWarning = "warning" +) + +// Diagnostic codes. Each is one of the refusals in hivecommons/hive#17620 +// "Deterministic", or a structural check the engine needs. +const ( + CodeNoExit = "state-no-exit" + CodeGateNoOwner = "gate-no-owner" + CodeAgentNoHumanGate = "agent-no-human-gate" + CodeDuplicateEvent = "duplicate-event-fact" + CodeNoColumn = "state-no-column" + CodeUnknownColumn = "unknown-column" + CodeUnknownState = "unknown-state" + CodeColumnEmpty = "column-empty" + CodeNoTerminalColumn = "no-terminal-column" + CodeDuplicateNumber = "duplicate-stage-number" + CodeUnknownFact = "unknown-fact" + CodeBadOp = "bad-op" + CodeNoRules = "no-rules" + CodeBadMode = "bad-derivation-mode" + CodeUnknownRole = "unknown-role" + CodeUnknownGate = "unknown-gate" + CodeUnknownEvent = "unknown-event" + CodeNoVersion = "no-version" + CodeInferred = "inferred-element" +) + +// ErrCompile wraps a compile refused by error diagnostics. +var ErrCompile = errors.New("workflow model does not compile") + +// Profile is a compiled model: what the engine and the monitor consume. +// It is derived, never edited; recompile from the Model instead. +type Profile struct { + Name string `json:"name"` + Version string `json:"version"` + Hash string `json:"hash"` + // Mode is the derivation mode. + Mode DerivationMode `json:"mode"` + // Columns in display order with their states resolved. + Columns []Column `json:"columns"` + // States by id, flattened (regions included). + States map[string]*State `json:"-"` + // ColumnOf maps a state id to its column id. + ColumnOf map[string]string `json:"columnOf"` + // StateOfNumber maps a stage number to the state id that carries it. + StateOfNumber map[int]string `json:"stateOfNumber"` + // Terminal is the set of states in a terminal column or of final type. + Terminal map[string]bool `json:"terminal"` + // Rules are the derivation rules, in evaluation order (rules mode). + Rules []Rule `json:"rules,omitempty"` + // Replies maps a state id to the human-gate verbs it offers. + Replies map[string][]string `json:"replies,omitempty"` + // Labels is the label set to reconcile. + Labels LabelSet `json:"labels,omitzero"` + // Diagnostics are the warnings left after a successful compile. + Diagnostics []Diagnostic `json:"diagnostics,omitempty"` +} + +// Compile validates m, then checks the lifecycle half and builds the +// Profile. Error diagnostics refuse the compile: the returned Profile is +// nil and err wraps ErrCompile; diagnostics are always returned so an +// editor can show them all at once. +func Compile(m *Model) (*Profile, []Diagnostic, error) { + if err := m.Validate(); err != nil { + return nil, []Diagnostic{{Code: "invalid", Severity: DiagError, Detail: err.Error()}}, err + } + c := &compiler{m: m} + c.run() + sort.SliceStable(c.diags, func(i, j int) bool { + if c.diags[i].Severity != c.diags[j].Severity { + return c.diags[i].Severity == DiagError + } + return false + }) + for _, d := range c.diags { + if d.Severity == DiagError { + return nil, c.diags, fmt.Errorf("%w: %s", ErrCompile, summarize(c.diags)) + } + } + c.p.Diagnostics = c.diags + return c.p, c.diags, nil +} + +func summarize(diags []Diagnostic) string { + var parts []string + for _, d := range diags { + if d.Severity != DiagError { + continue + } + if d.Subject != "" { + parts = append(parts, d.Code+" "+d.Subject) + } else { + parts = append(parts, d.Code) + } + } + return strings.Join(parts, "; ") +} + +type compiler struct { + m *Model + p *Profile + diags []Diagnostic +} + +func (c *compiler) errorf(code, subject, format string, a ...any) { + c.diags = append(c.diags, Diagnostic{Code: code, Severity: DiagError, Subject: subject, Detail: fmt.Sprintf(format, a...)}) +} + +func (c *compiler) warnf(code, subject, format string, a ...any) { + c.diags = append(c.diags, Diagnostic{Code: code, Severity: DiagWarning, Subject: subject, Detail: fmt.Sprintf(format, a...)}) +} + +func (c *compiler) run() { + m := c.m + c.p = &Profile{Name: m.Metadata.Name, Version: m.Spec.Meta.Version, Hash: m.Hash(), Mode: m.Spec.Derivation.Mode, States: map[string]*State{}, ColumnOf: map[string]string{}, StateOfNumber: map[int]string{}, Terminal: map[string]bool{}, Replies: map[string][]string{}, Labels: m.Spec.Labels} + if c.p.Version == "" { + c.errorf(CodeNoVersion, "", "spec.meta.version is empty") + } + var flatten func(states []State) + flatten = func(states []State) { + for i := range states { + s := &states[i] + c.p.States[s.ID] = s + flatten(s.Regions) + } + } + flatten(m.Spec.States) + + c.columns() + c.numbers() + c.exits() + c.roles() + c.events() + c.derivation() + c.inferred() +} + +// columns checks every top-level state sits in exactly one column and +// every column names real states; a terminal column must exist. +func (c *compiler) columns() { + m := c.m + cols := map[string]bool{} + terminal := false + for _, col := range m.Spec.Columns { + cols[col.ID] = true + if col.Terminal { + terminal = true + } + if len(col.States) == 0 { + c.errorf(CodeColumnEmpty, col.ID, "column %q collects no state", col.ID) + } + for _, sid := range col.States { + s, ok := c.p.States[sid] + if !ok { + c.errorf(CodeUnknownState, col.ID, "column %q collects unknown state %q", col.ID, sid) + continue + } + if prev, dup := c.p.ColumnOf[sid]; dup && prev != col.ID { + c.errorf(CodeUnknownColumn, sid, "state %q is in columns %q and %q", sid, prev, col.ID) + } + c.p.ColumnOf[sid] = col.ID + if col.Terminal || s.Type == StateFinal { + c.p.Terminal[sid] = true + } + } + } + for _, s := range m.Spec.States { + if s.Lifecycle.Column != "" && !cols[s.Lifecycle.Column] { + c.errorf(CodeUnknownColumn, s.ID, "state %q names column %q, which does not exist", s.ID, s.Lifecycle.Column) + } + if s.Lifecycle.Column != "" && c.p.ColumnOf[s.ID] != "" && c.p.ColumnOf[s.ID] != s.Lifecycle.Column { + c.errorf(CodeUnknownColumn, s.ID, "state %q says column %q but column %q collects it", s.ID, s.Lifecycle.Column, c.p.ColumnOf[s.ID]) + } + if _, ok := c.p.ColumnOf[s.ID]; !ok { + if s.Lifecycle.Column != "" && cols[s.Lifecycle.Column] { + c.p.ColumnOf[s.ID] = s.Lifecycle.Column + continue + } + c.errorf(CodeNoColumn, s.ID, "state %q is not in any column", s.ID) + } + if s.Type == StateFinal { + c.p.Terminal[s.ID] = true + } + } + if len(m.Spec.Columns) > 0 && !terminal { + c.errorf(CodeNoTerminalColumn, "", "no column is terminal") + } + c.p.Columns = append([]Column(nil), m.Spec.Columns...) +} + +// numbers checks stage numbers are unique per item kind. +func (c *compiler) numbers() { + seen := map[string]string{} + for _, s := range c.m.Spec.States { + n := s.Lifecycle.Number + if n == 0 { + continue + } + kinds := s.Lifecycle.Kinds + if len(kinds) == 0 { + kinds = []ItemKind{KindIssue, KindPR} + } + for _, k := range kinds { + key := fmt.Sprintf("%s/%d", k, n) + if prev, dup := seen[key]; dup { + c.errorf(CodeDuplicateNumber, s.ID, "states %q and %q both carry stage number %d for %s", prev, s.ID, n, k) + } + seen[key] = s.ID + } + if _, ok := c.p.StateOfNumber[n]; !ok { + c.p.StateOfNumber[n] = s.ID + } + if len(s.Lifecycle.Replies) > 0 { + c.p.Replies[s.ID] = append([]string(nil), s.Lifecycle.Replies...) + } + } +} + +// exits checks every non-terminal state has a way out. +func (c *compiler) exits() { + out := map[string]bool{} + for _, t := range c.m.Spec.Transitions { + out[t.From] = true + } + for _, s := range c.m.Spec.States { + if c.p.Terminal[s.ID] || out[s.ID] || s.Type == StateParallel { + continue + } + c.errorf(CodeNoExit, s.ID, "state %q is not terminal and has no transition out", s.ID) + } +} + +// roles checks gates name a releasing role and that an agent-granted +// state is followed by a human-released gate on every path out. +func (c *compiler) roles() { + m := c.m + roles := map[string]RoleDef{} + for _, r := range m.Spec.Roles { + roles[r.ID] = r + } + gates := map[string]Gate{} + for _, g := range m.Spec.Gates { + gates[g.ID] = g + if g.ReleasedBy == "" { + c.errorf(CodeGateNoOwner, g.ID, "gate %q has no releasing role", g.ID) + continue + } + if _, ok := roles[g.ReleasedBy]; !ok { + c.errorf(CodeUnknownRole, g.ID, "gate %q is released by unknown role %q", g.ID, g.ReleasedBy) + } + } + for _, t := range m.Spec.Transitions { + if t.GateRef != "" { + if _, ok := gates[t.GateRef]; !ok { + c.errorf(CodeUnknownGate, t.From+"->"+t.To, "transition %s->%s needs unknown gate %q", t.From, t.To, t.GateRef) + } + } + } + for _, e := range m.Spec.Events { + if e.Actor != "" { + if _, ok := roles[e.Actor]; !ok { + c.errorf(CodeUnknownRole, e.ID, "event %q actor %q is not a role", e.ID, e.Actor) + } + } + } + if len(m.Spec.Agents) == 0 { + return + } + humanGate := func(t Transition) bool { + g, ok := gates[t.GateRef] + return ok && roles[g.ReleasedBy].Human + } + // A closure edge (item closed unmerged) is an observation, not an + // outcome an agent can bring about, so it is not a path to inspect. + events := map[string]Event{} + for _, e := range m.Spec.Events { + events[e.ID] = e + } + closure := func(t Transition) bool { + e, ok := events[t.On] + return ok && c.p.Terminal[t.To] && e.Fact.Fact == "open" && e.Fact.Op == OpFalse + } + // adjacency over transitions + next := map[string][]Transition{} + for _, t := range m.Spec.Transitions { + next[t.From] = append(next[t.From], t) + } + for _, a := range m.Spec.Agents { + for _, sid := range a.States { + if _, ok := c.p.States[sid]; !ok { + c.errorf(CodeUnknownState, a.Profile, "agent %q acts in unknown state %q", a.Profile, sid) + continue + } + if !c.humanGateReachable(sid, next, humanGate, closure) { + c.errorf(CodeAgentNoHumanGate, sid, "agent %q acts in %q but no human-released gate follows it before a terminal state", a.Profile, sid) + } + } + } +} + +// humanGateReachable is true when every path from start to a terminal +// state crosses a human-released gate (a path with no such gate is a +// refusal). Cycles without a gate do not count as a path; closure edges +// are skipped. +func (c *compiler) humanGateReachable(start string, next map[string][]Transition, human, skip func(Transition) bool) bool { + var visit func(state string, seen map[string]bool) bool + visit = func(state string, seen map[string]bool) bool { + if c.p.Terminal[state] { + return false // reached terminal without a gate + } + if seen[state] { + return true // cycle: no gate-free path to terminal through here + } + seen[state] = true + defer delete(seen, state) + for _, t := range next[state] { + if human(t) || skip(t) { + continue + } + if !visit(t.To, seen) { + return false + } + } + return true + } + return visit(start, map[string]bool{}) +} + +// events checks no two events bind the same predicate and transitions +// name declared events. +func (c *compiler) events() { + m := c.m + ids := map[string]bool{} + byFact := map[string]string{} + for _, e := range m.Spec.Events { + ids[e.ID] = true + c.predicate(e.ID, e.Fact) + k := e.Fact.Fact + "|" + string(e.Fact.Op) + "|" + e.Fact.Value + if prev, dup := byFact[k]; dup { + c.errorf(CodeDuplicateEvent, e.ID, "events %q and %q both fire on %s %s %s", prev, e.ID, e.Fact.Fact, e.Fact.Op, e.Fact.Value) + } + byFact[k] = e.ID + } + if len(m.Spec.Events) == 0 { + return + } + for _, t := range m.Spec.Transitions { + if t.On != "" && !ids[t.On] { + c.errorf(CodeUnknownEvent, t.From+"->"+t.To, "transition %s->%s fires on undeclared event %q", t.From, t.To, t.On) + } + } + for _, b := range m.Spec.Bookkeeping { + if !ids[b.On] { + c.errorf(CodeUnknownEvent, b.Action, "bookkeeping %q fires on undeclared event %q", b.Action, b.On) + } + } +} + +// predicate checks a predicate against the fact vocabulary. +func (c *compiler) predicate(subject string, p Predicate) { + ft, ok := Facts[p.Fact] + if !ok { + c.errorf(CodeUnknownFact, subject, "fact %q is not in the vocabulary", p.Fact) + return + } + valid := map[FactType][]Op{ + FactBool: {OpTrue, OpFalse}, + FactScalar: {OpIs, OpNot}, + FactSet: {OpHas, OpLacks}, + }[ft] + for _, op := range valid { + if op == p.Op { + if ft != FactBool && p.Value == "" { + c.errorf(CodeBadOp, subject, "fact %q %s needs a value", p.Fact, p.Op) + } + return + } + } + c.errorf(CodeBadOp, subject, "fact %q is %s; operator %q does not apply", p.Fact, ft, p.Op) +} + +// derivation checks the mode and compiles the rules. +func (c *compiler) derivation() { + d := c.m.Spec.Derivation + switch d.Mode { + case DerivationBuiltin: + if len(d.Rules) > 0 { + c.warnf(CodeBadMode, "", "derivation is builtin; %d rules are ignored", len(d.Rules)) + } + case DerivationRules: + if len(d.Rules) == 0 { + c.errorf(CodeNoRules, "", "derivation is rules but spec.derivation.rules is empty") + } + for i, r := range d.Rules { + subject := r.ID + if subject == "" { + subject = fmt.Sprintf("rules[%d]", i) + } + if _, ok := c.p.States[r.State]; !ok { + c.errorf(CodeUnknownState, subject, "rule %s puts items in unknown state %q", subject, r.State) + } + if len(r.When) == 0 { + c.warnf(CodeNoRules, subject, "rule %s has no predicates and matches everything after it is reached", subject) + } + for _, p := range r.When { + c.predicate(subject, p) + } + } + c.p.Rules = append([]Rule(nil), d.Rules...) + default: + c.errorf(CodeBadMode, "", "spec.derivation.mode %q is not builtin or rules", d.Mode) + } +} + +// inferred warns about every element a human has not confirmed. +func (c *compiler) inferred() { + for _, s := range c.m.Spec.States { + if s.Lifecycle.Confidence == ConfidenceInferred { + c.warnf(CodeInferred, s.ID, "state %q is inferred and unconfirmed", s.ID) + } + } + for _, e := range c.m.Spec.Events { + if e.Confidence == ConfidenceInferred { + c.warnf(CodeInferred, e.ID, "event %q is inferred and unconfirmed", e.ID) + } + } + for _, g := range c.m.Spec.Gates { + if g.Confidence == ConfidenceInferred { + c.warnf(CodeInferred, g.ID, "gate %q is inferred and unconfirmed", g.ID) + } + } +} + +// StateForNumber returns the state carrying stage number n, if any. +func (p *Profile) StateForNumber(n int) (string, bool) { + s, ok := p.StateOfNumber[n] + return s, ok +} + +// Column returns the column with id, or nil. +func (p *Profile) Column(id string) *Column { + for i := range p.Columns { + if p.Columns[i].ID == id { + return &p.Columns[i] + } + } + return nil +} diff --git a/src/pkg/workflowmodel/compile_test.go b/src/pkg/workflowmodel/compile_test.go new file mode 100644 index 0000000000..38ee918c6c --- /dev/null +++ b/src/pkg/workflowmodel/compile_test.go @@ -0,0 +1,175 @@ +package workflowmodel + +import ( + "encoding/json" + "os" + "strings" + "testing" +) + +func loadModel(t *testing.T, path string) *Model { + t.Helper() + b, err := os.ReadFile(path) + if err != nil { + t.Fatal(err) + } + var m Model + if err := json.Unmarshal(b, &m); err != nil { + t.Fatal(err) + } + return &m +} + +func techdocs(t *testing.T) *Model { return loadModel(t, "testdata/models/techdocs.json") } + +func TestCompileTechDocs(t *testing.T) { + p, diags, err := Compile(techdocs(t)) + if err != nil { + t.Fatalf("%v\n%+v", err, diags) + } + if p.Mode != DerivationRules || len(p.Rules) != 9 { + t.Fatalf("mode %s rules %d", p.Mode, len(p.Rules)) + } + if p.ColumnOf["draft-review"] != "writing" || !p.Terminal["published"] || p.Terminal["approved"] { + t.Fatalf("columns %v terminal %v", p.ColumnOf, p.Terminal) + } + if id, ok := p.StateForNumber(45); !ok || id != "analysis-review" { + t.Fatalf("StateForNumber(45) = %q %v", id, ok) + } + if got := p.Replies["draft-review"]; len(got) != 2 { + t.Fatalf("replies %v", got) + } + if p.Hash == "" || p.Version != "1" { + t.Fatalf("hash %q version %q", p.Hash, p.Version) + } + for _, d := range diags { + if d.Severity == DiagError { + t.Errorf("unexpected error diagnostic %+v", d) + } + } +} + +// Each refusal from the RFC's "Deterministic" list, and the structural +// checks around them, produces its code and no Profile. +func TestCompileRefusals(t *testing.T) { + cases := []struct { + name string + mut func(m *Model) + code string + }{ + {"no version", func(m *Model) { m.Spec.Meta.Version = "" }, CodeNoVersion}, + {"state without exit", func(m *Model) { + var tr []Transition + for _, x := range m.Spec.Transitions { + if x.From != "approved" { + tr = append(tr, x) + } + } + m.Spec.Transitions = tr + }, CodeNoExit}, + {"gate without owner", func(m *Model) { m.Spec.Gates[0].ReleasedBy = "" }, CodeGateNoOwner}, + {"gate with unknown role", func(m *Model) { m.Spec.Gates[0].ReleasedBy = "nobody" }, CodeUnknownRole}, + {"agent with no human gate after it", func(m *Model) { + for i := range m.Spec.Roles { + m.Spec.Roles[i].Human = false + } + }, CodeAgentNoHumanGate}, + {"agent in unknown state", func(m *Model) { m.Spec.Agents[0].States = []string{"ghost"} }, CodeUnknownState}, + {"two events on one fact", func(m *Model) { m.Spec.Events[1].Fact = m.Spec.Events[0].Fact }, CodeDuplicateEvent}, + {"state in no column", func(m *Model) { m.Spec.Columns[0].States = []string{"analysing"} }, CodeNoColumn}, + {"state in two columns", func(m *Model) { m.Spec.Columns[0].States = []string{"requested", "analysing"} }, CodeUnknownColumn}, + {"state names a column that does not collect it", func(m *Model) { m.Spec.States[0].Lifecycle.Column = "done" }, CodeUnknownColumn}, + {"column collects unknown state", func(m *Model) { m.Spec.Columns[0].States = append(m.Spec.Columns[0].States, "ghost") }, CodeUnknownState}, + {"empty column", func(m *Model) { m.Spec.Columns = append(m.Spec.Columns, Column{ID: "x", Label: "x"}) }, CodeColumnEmpty}, + {"no terminal column", func(m *Model) { m.Spec.Columns[4].Terminal = false }, CodeNoTerminalColumn}, + {"duplicate stage number", func(m *Model) { m.Spec.States[1].Lifecycle.Number = 10 }, CodeDuplicateNumber}, + {"unknown fact", func(m *Model) { m.Spec.Derivation.Rules[0].When[0].Fact = "weather" }, CodeUnknownFact}, + {"bad operator", func(m *Model) { m.Spec.Derivation.Rules[0].When[0].Op = OpHas }, CodeBadOp}, + {"scalar op without value", func(m *Model) { m.Spec.Derivation.Rules[2].When[0].Value = "" }, CodeBadOp}, + {"rules mode with no rules", func(m *Model) { m.Spec.Derivation.Rules = nil }, CodeNoRules}, + {"rule names unknown state", func(m *Model) { m.Spec.Derivation.Rules[0].State = "ghost" }, CodeUnknownState}, + {"bad derivation mode", func(m *Model) { m.Spec.Derivation.Mode = "oracle" }, CodeBadMode}, + {"transition on undeclared event", func(m *Model) { m.Spec.Transitions[0].On = "thunder" }, CodeUnknownEvent}, + {"transition needs unknown gate", func(m *Model) { m.Spec.Transitions[0].GateRef = "ghost" }, CodeUnknownGate}, + {"event actor is not a role", func(m *Model) { m.Spec.Events[0].Actor = "ghost" }, CodeUnknownRole}, + {"bookkeeping on undeclared event", func(m *Model) { + m.Spec.Bookkeeping = []Effect{{On: "thunder", Action: "comment"}} + }, CodeUnknownEvent}, + } + for _, tc := range cases { + t.Run(tc.name, func(t *testing.T) { + m := techdocs(t) + tc.mut(m) + p, diags, err := Compile(m) + if err == nil || p != nil { + t.Fatalf("compiled; diags %+v", diags) + } + if !strings.Contains(err.Error(), ErrCompile.Error()) { + t.Fatalf("err %v does not wrap ErrCompile", err) + } + found := false + for _, d := range diags { + if d.Code == tc.code && d.Severity == DiagError { + found = true + } + } + if !found { + t.Fatalf("want %s in %+v", tc.code, diags) + } + }) + } +} + +func TestCompileWarnings(t *testing.T) { + m := techdocs(t) + m.Spec.Gates[0].Confidence = ConfidenceInferred + m.Spec.Events[0].Confidence = ConfidenceInferred + p, diags, err := Compile(m) + if err != nil { + t.Fatal(err) + } + n := 0 + for _, d := range diags { + if d.Code == CodeInferred && d.Severity == DiagWarning { + n++ + } + } + if n != 2 || len(p.Diagnostics) != len(diags) { + t.Fatalf("inferred warnings %d; profile carries %d of %d", n, len(p.Diagnostics), len(diags)) + } + + m = techdocs(t) + m.Spec.Derivation.Mode = DerivationBuiltin + if _, diags, err = Compile(m); err != nil { + t.Fatal(err) + } + if len(diags) != 1 || diags[0].Code != CodeBadMode || diags[0].Severity != DiagWarning { + t.Fatalf("builtin with rules: %+v", diags) + } +} + +func TestCompileInvalidModel(t *testing.T) { + m := techdocs(t) + m.Spec.Initial = "ghost" + p, diags, err := Compile(m) + if err == nil || p != nil || len(diags) != 1 || diags[0].Code != "invalid" { + t.Fatalf("%v %+v", err, diags) + } +} + +// A closure edge out of an agent state is an observation, not a way for +// the agent to reach an outcome unsupervised; a merge edge is. +func TestCompileClosureEdgesAreNotAgentPaths(t *testing.T) { + m := techdocs(t) + if _, _, err := Compile(m); err != nil { + t.Fatal(err) + } + m.Spec.Transitions = append(m.Spec.Transitions, Transition{From: "drafting", To: "published", On: "merged"}) + _, diags, err := Compile(m) + if err == nil { + t.Fatal("agent merging its own draft must be refused") + } + if !strings.Contains(err.Error(), CodeAgentNoHumanGate+" drafting") { + t.Fatalf("%v %+v", err, diags) + } +} diff --git a/src/pkg/workflowmodel/defaults.go b/src/pkg/workflowmodel/defaults.go new file mode 100644 index 0000000000..3462a45431 --- /dev/null +++ b/src/pkg/workflowmodel/defaults.go @@ -0,0 +1,301 @@ +package workflowmodel + +import ( + "fmt" + "sort" + "strings" +) + +// State ids of the hive-default process graph. These are the columns the +// Activity Monitor draws today (agent-now.js STAGE_TRACKER_PATH = +// [0,10,20,30,50,80] + the merged cap); the ids are stable so a bound +// model keeps meaning the same thing to the dashboard. +const ( + StateOpen = "open" + StateActionable = "actionable" + StateClaimed = "claimed" + StatePRInProgress = "pr-in-progress" + StateVerify = "verify" + StateReview = "review" + StateCI = "ci" + StateMergeQueue = "merge-queue" + StateMerged = "merged" + StateMainCI = "main-ci" + StateRelease = "release" + StateDone = "done" +) + +// Guard ids of the hive-default process graph. +const ( + GuardIDRequiredChecks = "required-checks-satisfied" + GuardIDReviewPolicy = "review-policy-satisfied" +) + +// DeployStateID is the id of the deploy state for an environment. +func DeployStateID(env string) string { return "deploy-" + slug(env) } + +// HiveDefault is the process graph every hive runs today, as a model. +// Binding no pipelines into it changes nothing; that is the compatibility +// promise of #17620. +func HiveDefault(scope Scope) *Model { + m := New("hive-default", scope) + m.Spec.Initial = StateOpen + m.Spec.States = []State{ + {ID: StateOpen, Label: "Open", Owner: OwnerHuman, Origin: OriginHiveDefault}, + {ID: StateActionable, Label: "Actionable", Owner: OwnerHuman, Origin: OriginHiveDefault}, + {ID: StateClaimed, Label: "Claimed", Owner: OwnerRegisteredAgent, Origin: OriginHiveDefault}, + {ID: StatePRInProgress, Label: "PR in progress", Owner: OwnerRegisteredAgent, Origin: OriginHiveDefault}, + {ID: StateVerify, Label: "CI / Review", Owner: OwnerExternalService, Type: StateParallel, Origin: OriginHiveDefault, Regions: []State{ + {ID: StateReview, Label: "Review", Owner: OwnerHuman, GuardRef: GuardIDReviewPolicy, Origin: OriginHiveDefault}, + {ID: StateCI, Label: "CI", Owner: OwnerExternalService, GuardRef: GuardIDRequiredChecks, Origin: OriginHiveDefault}, + }}, + {ID: StateMergeQueue, Label: "Merge queue", Owner: OwnerExternalService, Origin: OriginHiveDefault}, + {ID: StateMerged, Label: "Merged", Owner: OwnerExternalObservation, Type: StateFinal, Origin: OriginHiveDefault}, + } + m.Spec.Transitions = []Transition{ + {From: StateOpen, To: StateActionable, On: "triaged"}, + {From: StateActionable, To: StateClaimed, On: "claimed"}, + {From: StateClaimed, To: StatePRInProgress, On: "pr-opened"}, + {From: StatePRInProgress, To: StateVerify, On: "ready-for-review"}, + {From: StateVerify, To: StateMergeQueue, On: "verified", GuardRef: GuardIDRequiredChecks}, + {From: StateMergeQueue, To: StateMerged, On: "merged"}, + } + m.Spec.Guards = []Guard{ + {ID: GuardIDReviewPolicy, Kind: GuardReviewPolicy}, + {ID: GuardIDRequiredChecks, Kind: GuardRequiredChecks}, + } + m.Spec.Lanes = []Lane{ + {ID: "implement", Label: "Implement", StateRef: StateClaimed, Actor: OwnerRegisteredAgent, Effects: []string{"branch", "pr"}}, + {ID: "review", Label: "Review", StateRef: StateReview, Actor: OwnerRegisteredAgent, Effects: []string{"review"}}, + {ID: "repair", Label: "Repair", StateRef: StateCI, Actor: OwnerRegisteredAgent, Effects: []string{"push"}}, + } + return m +} + +// Bind composes ingested pipelines into the hive-default process graph: +// PR-verify pipelines fill the CI region; push-verify, release and deploy +// pipelines add post-merge states, in that order, each only when present. +// Scheduled and manual pipelines are kept in Spec.Pipelines but bind to no +// state. Sources and coverage are recorded as given. +func Bind(scope Scope, pipelines []Pipeline, sources []Source, cov Coverage) *Model { + m := HiveDefault(scope) + m.Metadata.Name = slug(scope.Repository) + pipelines = DedupeIDs(pipelines) + m.Spec.Pipelines = pipelines + m.Spec.Sources = append([]Source(nil), sources...) + m.Spec.Coverage = cov + if len(pipelines) == 0 { + return m + } + m.Metadata.Annotations = map[string]string{"workflow.hivecommons.dev/bound": "true"} + + byRole := map[Role][]string{} + envs := map[string][]string{} + for _, p := range pipelines { + for _, r := range BindingRoles(p) { + byRole[r] = append(byRole[r], p.ID) + } + if p.Role == RoleDeploy { + for _, n := range p.Nodes { + switch { + case n.Environment == "": + case IsExpression(n.Environment): + // A computed environment name is one deploy state, not a + // column per possible value; the finding keeps it visible. + m.Spec.Coverage.Findings = append(m.Spec.Coverage.Findings, Finding{Path: p.Path, Node: n.ID, Code: CodeEnvironmentDynamic, Detail: n.Environment, Severity: SeverityWarning}) + default: + envs[n.Environment] = appendUnique(envs[n.Environment], p.ID) + } + } + } + } + if ci := m.FindState(StateCI); ci != nil && len(byRole[RolePRVerify]) > 0 { + ci.Pipelines = byRole[RolePRVerify] + ci.Origin = OriginIngested + } + + var post []State + if ids := byRole[RolePushVerify]; len(ids) > 0 { + post = append(post, State{ID: StateMainCI, Label: "Main CI", Owner: OwnerExternalService, Pipelines: ids, Origin: OriginIngested}) + } + if ids := byRole[RoleRelease]; len(ids) > 0 { + post = append(post, State{ID: StateRelease, Label: "Release", Owner: OwnerExternalService, Pipelines: ids, Origin: OriginIngested}) + } + envNames := make([]string, 0, len(envs)) + for e := range envs { + envNames = append(envNames, e) + } + sort.Strings(envNames) + for _, e := range envNames { + post = append(post, State{ID: DeployStateID(e), Label: "Deploy: " + e, Owner: OwnerExternalService, GuardRef: envGuardID(e), Pipelines: envs[e], Origin: OriginIngested}) + m.Spec.Guards = append(m.Spec.Guards, Guard{ID: envGuardID(e), Kind: GuardEnvironmentApproval, Params: map[string]string{"environment": e}, Source: string(SystemGitHubActions)}) + } + if deployNoEnv := withoutEnv(pipelines, byRole[RoleDeploy]); len(deployNoEnv) > 0 { + post = append(post, State{ID: "deploy", Label: "Deploy", Owner: OwnerExternalService, Pipelines: deployNoEnv, Origin: OriginIngested}) + } + if len(post) == 0 { + return m + } + // Merged stops being final; the ingested post-merge chain follows it. + merged := m.FindState(StateMerged) + merged.Type = StateSequential + prev := StateMerged + for _, s := range post { + m.Spec.States = append(m.Spec.States, s) + m.Spec.Transitions = append(m.Spec.Transitions, Transition{From: prev, To: s.ID, On: "pipeline-succeeded", GuardRef: s.GuardRef}) + prev = s.ID + } + m.Spec.States = append(m.Spec.States, State{ID: StateDone, Label: "Done", Owner: OwnerExternalObservation, Type: StateFinal, Origin: OriginIngested}) + m.Spec.Transitions = append(m.Spec.Transitions, Transition{From: prev, To: StateDone, On: "pipeline-succeeded"}) + return m +} + +// BindingRoles is every state role a pipeline binds to: its primary Role +// plus the secondary ones its triggers imply, because one workflow often +// serves several (`on: [pull_request, push]` verifies PRs and the default +// branch; a PaC pipeline shared by pull-request, push and tag runs does +// all three). Order: primary first, then push-verify, then release. +func BindingRoles(p Pipeline) []Role { + roles := []Role{p.Role} + add := func(r Role) { + for _, x := range roles { + if x == r { + return + } + } + roles = append(roles, r) + } + for _, t := range p.Triggers { + switch t.Event { + case "push": + if len(positive(t.Tags)) > 0 { + add(RoleRelease) + } + if len(positive(t.Tags)) == 0 || len(positive(t.Branches)) > 0 { + add(RolePushVerify) + } + case "release": + add(RoleRelease) + } + } + return roles +} + +// positive drops "!"-prefixed (ignore) filters. +func positive(xs []string) []string { + var out []string + for _, x := range xs { + if !strings.HasPrefix(x, "!") { + out = append(out, x) + } + } + return out +} + +func withoutEnv(all []Pipeline, ids []string) []string { + var out []string + for _, id := range ids { + for _, p := range all { + if p.ID != id { + continue + } + hasEnv := false + for _, n := range p.Nodes { + if n.Environment != "" && !IsExpression(n.Environment) { + hasEnv = true + break + } + } + if !hasEnv { + out = append(out, id) + } + } + } + return out +} + +// CodeEnvironmentDynamic is the finding for a deploy environment whose +// name is an expression. +const CodeEnvironmentDynamic = "environment-dynamic" + +// IsExpression reports whether s is a GitHub Actions `${{ }}` or Tekton +// `$( )` expression rather than a literal. +func IsExpression(s string) bool { + return strings.Contains(s, "${{") || strings.Contains(s, "$(") +} + +// DedupeIDs makes pipeline ids unique in order: a second pipeline whose id +// collides (for example `_stale.yml` and `stale.yml` both slug to the same +// id) gets a "-2", "-3" suffix. Deterministic for a given input order. +func DedupeIDs(pipelines []Pipeline) []Pipeline { + out := append([]Pipeline(nil), pipelines...) + seen := map[string]int{} + for i := range out { + base := out[i].ID + seen[base]++ + if seen[base] == 1 { + continue + } + for { + candidate := fmt.Sprintf("%s-%d", base, seen[base]) + if seen[candidate] == 0 { + seen[candidate] = 1 + out[i].ID = candidate + break + } + seen[base]++ + } + } + return out +} + +func envGuardID(env string) string { return "environment-approved-" + slug(env) } + +func appendUnique(xs []string, x string) []string { + for _, y := range xs { + if y == x { + return xs + } + } + return append(xs, x) +} + +// slug lowercases s and keeps [a-z0-9-], collapsing runs of other +// characters to one hyphen; empty input yields "x". +func slug(s string) string { + var b strings.Builder + dash := false + for _, r := range strings.ToLower(s) { + switch { + case r >= 'a' && r <= 'z', r >= '0' && r <= '9': + b.WriteRune(r) + dash = false + case !dash && b.Len() > 0: + b.WriteByte('-') + dash = true + } + } + out := strings.TrimSuffix(b.String(), "-") + if out == "" { + return "x" + } + return out +} + +// Slug is slug, exported for ingesters that need stable ids from names. +func Slug(s string) string { return slug(s) } + +// PipelineID is the stable id of a pipeline from its system and path. +func PipelineID(system System, path string) string { + return fmt.Sprintf("%s-%s", shortSystem(system), slug(strings.TrimSuffix(strings.TrimSuffix(path, ".yml"), ".yaml"))) +} + +func shortSystem(s System) string { + switch s { + case SystemGitHubActions: + return "gha" + case SystemTekton: + return "tkn" + } + return slug(string(s)) +} diff --git a/src/pkg/workflowmodel/doc.go b/src/pkg/workflowmodel/doc.go new file mode 100644 index 0000000000..110c6fa70b --- /dev/null +++ b/src/pkg/workflowmodel/doc.go @@ -0,0 +1,13 @@ +// Package workflowmodel is the declared workflow model behind hivecommons/hive#17620. +// +// A Model is a typed, versioned description of how work moves through a +// repository: states (sequential, parallel regions, final), the transitions +// and guards between them, who owns each state, and the lanes agents may +// work in. Ingesters under this package (githubactions, tekton) read a +// repository's CI definitions deterministically and fill the model's +// execution graph; render derives diagrams and Activity Monitor columns +// from the model. +// +// Nothing here is wired into the stage engine yet: a hive with no bound +// model behaves exactly as before (RFC phase 1 is ingest, store and view). +package workflowmodel diff --git a/src/pkg/workflowmodel/githubactions/ingest.go b/src/pkg/workflowmodel/githubactions/ingest.go new file mode 100644 index 0000000000..c69656fec3 --- /dev/null +++ b/src/pkg/workflowmodel/githubactions/ingest.go @@ -0,0 +1,634 @@ +// Package githubactions ingests GitHub Actions workflow files into +// workflowmodel pipelines. It reads the YAML as a node tree so job and +// trigger order follow the file, never a map iteration, and it never +// rejects a repository because one file is malformed: bad files are counted +// in Coverage and reported as findings. +package githubactions + +import ( + "errors" + "fmt" + "path" + "sort" + "strings" + + "github.com/hivecommons/hive/pkg/workflowmodel" + "gopkg.in/yaml.v3" +) + +// Finding codes this ingester records. +const ( + // CodeYAMLParse is a file that is not valid YAML. + CodeYAMLParse = "yaml-parse" + // CodeNotAWorkflow is valid YAML that has neither `on` nor `jobs`. + CodeNotAWorkflow = "not-a-workflow" + // CodeReusableOnly is a workflow triggered only by workflow_call. + CodeReusableOnly = "reusable-only" + // CodeEnvironmentOnPR is a PR-gating job that targets an environment. + CodeEnvironmentOnPR = "environment-on-pr" + // CodeNeedsUnresolved is a need naming no job; the need is dropped. + CodeNeedsUnresolved = "needs-unresolved" + // CodeNeedsCycle is a cycle in needs; all levels are reset to zero. + CodeNeedsCycle = "needs-cycle" + // CodeConcurrencyExpression is a cancel-in-progress that is an expression. + CodeConcurrencyExpression = "concurrency-expression" + // CodeJobContainer is a job with container or services, which are not modelled. + CodeJobContainer = "job-container" + // CodeSecretsInherit is a reusable job passing `secrets: inherit`. + CodeSecretsInherit = "secrets-inherit" + // CodeJobEmpty is a job with no steps and no uses. + CodeJobEmpty = "job-empty" + // CodeNoJobs is a workflow whose jobs map is missing or empty. + CodeNoJobs = "no-jobs" +) + +// Event names the role derivation looks at. +const ( + EventPush = "push" + EventPullRequest = "pull_request" + EventPullRequestTarget = "pull_request_target" + EventMergeGroup = "merge_group" + EventRelease = "release" + EventSchedule = "schedule" + EventWorkflowDispatch = "workflow_dispatch" + EventWorkflowCall = "workflow_call" + EventWorkflowRun = "workflow_run" + EventDeployment = "deployment" + EventDeploymentStatus = "deployment_status" +) + +// Workflow file keys. +const ( + keyOn = "on" + keyName = "name" + keyJobs = "jobs" + keyConcurrency = "concurrency" + keyGroup = "group" + keyCancelInProgress = "cancel-in-progress" + keyTypes = "types" + keyBranches = "branches" + keyBranchesIgnore = "branches-ignore" + keyTags = "tags" + keyTagsIgnore = "tags-ignore" + keyPaths = "paths" + keyPathsIgnore = "paths-ignore" + keyCron = "cron" + keyWorkflows = "workflows" + keyUses = "uses" + keyNeeds = "needs" + keyEnvironment = "environment" + keyIf = "if" + keyRunsOn = "runs-on" + keyLabels = "labels" + keyStrategy = "strategy" + keyMatrix = "matrix" + keyInclude = "include" + keyExclude = "exclude" + keyContainer = "container" + keyServices = "services" + keySecrets = "secrets" + keySteps = "steps" + secretsInherit = "inherit" +) + +// Workflow path layout. +const ( + workflowsDir = ".github/workflows/" + extYML = ".yml" + extYAML = ".yaml" +) + +// negatePrefix marks an ignore-list entry when folded into its positive list. +const negatePrefix = "!" + +// YAML tags and literals the parser distinguishes. +const ( + tagBool = "!!bool" + tagNull = "!!null" + literalTrue = "true" + literalFalse = "false" +) + +// expressionOpen starts a GitHub Actions expression. +const expressionOpen = "${{" + +// Sentinel errors ParseWorkflow wraps. +var ( + // ErrYAML is returned when the file is not valid YAML. + ErrYAML = errors.New("yaml parse error") + // ErrNotAWorkflow is returned when the document is not a workflow. + ErrNotAWorkflow = errors.New("not a workflow") +) + +// File is one workflow file read from a repository. +type File struct { + Path string + Data []byte +} + +// Result of ingesting a repository's workflows. +type Result struct { + Pipelines []workflowmodel.Pipeline + Sources []workflowmodel.Source + Coverage workflowmodel.Coverage +} + +// Ingest parses every file as a GitHub Actions workflow. It never returns +// an error for a bad file: the file is counted in Coverage.Files but not +// Coverage.Parsed and a Finding (code "yaml-parse" or "not-a-workflow", +// severity unknown) is recorded. Output is deterministic: pipelines sorted +// by Path, nodes in job-declaration order, findings sorted by (Path, Node, +// Code). +func Ingest(files []File) Result { + sorted := make([]File, len(files)) + copy(sorted, files) + sort.SliceStable(sorted, func(i, j int) bool { return sorted[i].Path < sorted[j].Path }) + + var res Result + for _, f := range sorted { + res.Sources = append(res.Sources, workflowmodel.Source{ + System: workflowmodel.SystemGitHubActions, + Path: f.Path, + SHA256: workflowmodel.SHA256Hex(f.Data), + }) + res.Coverage.Files++ + p, findings, err := ParseWorkflow(f.Path, f.Data) + if err != nil { + code := CodeNotAWorkflow + if errors.Is(err, ErrYAML) { + code = CodeYAMLParse + } + res.Coverage.Findings = append(res.Coverage.Findings, workflowmodel.Finding{ + Path: f.Path, Code: code, Detail: err.Error(), Severity: workflowmodel.SeverityUnknown, + }) + continue + } + res.Coverage.Parsed++ + res.Pipelines = append(res.Pipelines, p) + res.Coverage.Findings = append(res.Coverage.Findings, findings...) + } + sort.SliceStable(res.Pipelines, func(i, j int) bool { return res.Pipelines[i].Path < res.Pipelines[j].Path }) + sortFindings(res.Coverage.Findings) + return res +} + +// IsWorkflowPath reports whether path is `.github/workflows/.yml|.yaml` +// (top level only). +func IsWorkflowPath(p string) bool { + rest, ok := strings.CutPrefix(p, workflowsDir) + if !ok || rest == "" || strings.Contains(rest, "/") { + return false + } + ext := path.Ext(rest) + return (ext == extYML || ext == extYAML) && len(rest) > len(ext) +} + +// ParseWorkflow parses one workflow file into a Pipeline; returns findings +// for constructs it recognised but did not model. +func ParseWorkflow(p string, data []byte) (workflowmodel.Pipeline, []workflowmodel.Finding, error) { + var doc yaml.Node + if err := yaml.Unmarshal(data, &doc); err != nil { + return workflowmodel.Pipeline{}, nil, fmt.Errorf("%w: %v", ErrYAML, err) + } + root := documentRoot(&doc) + if root == nil || root.Kind != yaml.MappingNode { + return workflowmodel.Pipeline{}, nil, fmt.Errorf("%w: document is not a mapping", ErrNotAWorkflow) + } + top := entries(root) + onNode, hasOn := top[keyOn] + jobsNode, hasJobs := top[keyJobs] + if !hasOn && !hasJobs { + return workflowmodel.Pipeline{}, nil, fmt.Errorf("%w: neither on nor jobs present", ErrNotAWorkflow) + } + + pl := workflowmodel.Pipeline{ + ID: workflowmodel.PipelineID(workflowmodel.SystemGitHubActions, p), + Name: baseName(p), + System: workflowmodel.SystemGitHubActions, + Path: p, + } + if n := scalar(top[keyName]); n != "" { + pl.Name = n + } + + w := &walker{path: p} + pl.Triggers = w.triggers(onNode) + pl.Nodes = w.jobs(jobsNode) + pl.Levels = w.levels(pl.Nodes) + pl.Role = w.role(pl.Triggers, pl.Nodes) + pl.Concurrency = w.concurrency(top[keyConcurrency]) + sortFindings(w.findings) + return pl, w.findings, nil +} + +// walker accumulates findings while mapping one workflow file. +type walker struct { + path string + findings []workflowmodel.Finding +} + +func (w *walker) find(node, code, detail string, sev workflowmodel.Severity) { + w.findings = append(w.findings, workflowmodel.Finding{Path: w.path, Node: node, Code: code, Detail: detail, Severity: sev}) +} + +// triggers maps the `on` value (string, list or map) to Triggers. +func (w *walker) triggers(on *yaml.Node) []workflowmodel.Trigger { + on = deref(on) + if on == nil { + return nil + } + switch on.Kind { + case yaml.ScalarNode: + if on.Tag == tagNull { + return nil + } + return []workflowmodel.Trigger{{Event: on.Value}} + case yaml.SequenceNode: + var out []workflowmodel.Trigger + for _, e := range strs(on) { + out = append(out, workflowmodel.Trigger{Event: e}) + } + return out + case yaml.MappingNode: + var out []workflowmodel.Trigger + for _, kv := range pairs(on) { + out = append(out, triggerFor(kv.key, kv.val)) + } + return out + } + return nil +} + +func triggerFor(event string, cfg *yaml.Node) workflowmodel.Trigger { + t := workflowmodel.Trigger{Event: event} + cfg = deref(cfg) + if event == EventSchedule { + if cfg != nil && cfg.Kind == yaml.SequenceNode { + for _, item := range cfg.Content { + if c := scalar(entries(deref(item))[keyCron]); c != "" { + t.Schedule = append(t.Schedule, c) + } + } + } + return t + } + if cfg == nil || cfg.Kind != yaml.MappingNode { + return t + } + m := entries(cfg) + if event == EventWorkflowRun { + t.Types = append(t.Types, strs(m[keyWorkflows])...) + } + t.Types = append(t.Types, strs(m[keyTypes])...) + t.Branches = append(strs(m[keyBranches]), negate(strs(m[keyBranchesIgnore]))...) + t.Tags = append(strs(m[keyTags]), negate(strs(m[keyTagsIgnore]))...) + t.Paths = append(strs(m[keyPaths]), negate(strs(m[keyPathsIgnore]))...) + return t +} + +func negate(in []string) []string { + out := make([]string, 0, len(in)) + for _, s := range in { + out = append(out, negatePrefix+s) + } + return out +} + +// jobs maps the `jobs` mapping to Nodes in declaration order. +func (w *walker) jobs(jobs *yaml.Node) []workflowmodel.Node { + jobs = deref(jobs) + nodes := []workflowmodel.Node{} + if jobs == nil || jobs.Kind != yaml.MappingNode || len(jobs.Content) == 0 { + w.find("", CodeNoJobs, "jobs is missing or empty", workflowmodel.SeverityUnknown) + return nodes + } + for _, kv := range pairs(jobs) { + nodes = append(nodes, w.job(kv.key, kv.val)) + } + return nodes +} + +func (w *walker) job(id string, job *yaml.Node) workflowmodel.Node { + n := workflowmodel.Node{ID: id, Kind: workflowmodel.NodeJob} + job = deref(job) + if job == nil || job.Kind != yaml.MappingNode { + w.find(id, CodeJobEmpty, "job is not a mapping", workflowmodel.SeverityUnknown) + return n + } + m := entries(job) + n.Name = scalar(m[keyName]) + if uses := scalar(m[keyUses]); uses != "" { + n.Kind = workflowmodel.NodeReusableWorkflow + n.Ref = uses + } + n.Needs = strs(m[keyNeeds]) + n.Environment = environmentName(m[keyEnvironment]) + n.If = scalar(m[keyIf]) + n.RunsOn = runsOn(m[keyRunsOn]) + n.Matrix = matrix(entries(deref(m[keyStrategy]))[keyMatrix]) + + if _, ok := m[keyContainer]; ok { + w.find(id, CodeJobContainer, "container is not modelled", workflowmodel.SeverityUnsupported) + } else if _, ok := m[keyServices]; ok { + w.find(id, CodeJobContainer, "services are not modelled", workflowmodel.SeverityUnsupported) + } + if scalar(m[keySecrets]) == secretsInherit { + w.find(id, CodeSecretsInherit, "reusable job inherits all secrets", workflowmodel.SeverityWarning) + } + steps := deref(m[keySteps]) + if n.Ref == "" && (steps == nil || steps.Kind != yaml.SequenceNode || len(steps.Content) == 0) { + w.find(id, CodeJobEmpty, "job has no steps and no uses", workflowmodel.SeverityUnknown) + } + return n +} + +func environmentName(n *yaml.Node) string { + n = deref(n) + if n == nil { + return "" + } + if n.Kind == yaml.MappingNode { + return scalar(entries(n)[keyName]) + } + return scalar(n) +} + +func runsOn(n *yaml.Node) []string { + n = deref(n) + if n == nil { + return nil + } + if n.Kind == yaml.MappingNode { + m := entries(n) + return append(strs(m[keyGroup]), strs(m[keyLabels])...) + } + return strs(n) +} + +func matrix(n *yaml.Node) *workflowmodel.Matrix { + n = deref(n) + if n == nil { + return nil + } + if n.Kind == yaml.ScalarNode { + return &workflowmodel.Matrix{Dynamic: isExpression(n.Value)} + } + if n.Kind != yaml.MappingNode { + return nil + } + mx := &workflowmodel.Matrix{} + for _, kv := range pairs(n) { + v := deref(kv.val) + if v != nil && v.Kind == yaml.ScalarNode && isExpression(v.Value) { + mx.Dynamic = true + } + switch kv.key { + case keyInclude: + mx.Include = seqLen(v) + case keyExclude: + mx.Exclude = seqLen(v) + default: + mx.Keys = append(mx.Keys, kv.key) + } + } + sort.Strings(mx.Keys) + return mx +} + +func seqLen(n *yaml.Node) int { + if n == nil || n.Kind != yaml.SequenceNode { + return 0 + } + return len(n.Content) +} + +func isExpression(s string) bool { return strings.Contains(s, expressionOpen) } + +// levels assigns topological levels, repairing unknown needs and cycles so +// the pipeline is always well-formed. +func (w *walker) levels(nodes []workflowmodel.Node) int { + count, err := workflowmodel.Levels(nodes) + if err == nil { + return count + } + known := make(map[string]bool, len(nodes)) + for _, n := range nodes { + known[n.ID] = true + } + for i := range nodes { + kept := nodes[i].Needs[:0] + for _, need := range nodes[i].Needs { + if known[need] { + kept = append(kept, need) + continue + } + w.find(nodes[i].ID, CodeNeedsUnresolved, fmt.Sprintf("needs unknown job %q", need), workflowmodel.SeverityUnknown) + } + if len(kept) == 0 { + kept = nil + } + nodes[i].Needs = kept + } + count, err = workflowmodel.Levels(nodes) + if err == nil { + return count + } + w.find("", CodeNeedsCycle, err.Error(), workflowmodel.SeverityUnknown) + for i := range nodes { + nodes[i].Level = 0 + } + if len(nodes) == 0 { + return 0 + } + return 1 +} + +// role derives the pipeline role from its triggers; first rule wins. +func (w *walker) role(triggers []workflowmodel.Trigger, nodes []workflowmodel.Node) workflowmodel.Role { + events := map[string]bool{} + pushTags := false + for _, t := range triggers { + events[t.Event] = true + if t.Event == EventPush { + for _, tag := range t.Tags { + if !strings.HasPrefix(tag, negatePrefix) { + pushTags = true + } + } + } + } + var envJobs []string + for _, n := range nodes { + if n.Environment != "" { + envJobs = append(envJobs, n.ID) + } + } + // onlyWithin reports whether every event is e or one of the manual + // companions (workflow_dispatch, workflow_call), and e is present: a + // nightly with a manual button is still scheduled. + onlyWithin := func(e string) bool { + if !events[e] { + return false + } + for ev := range events { + if ev != e && ev != EventWorkflowDispatch && ev != EventWorkflowCall { + return false + } + } + return true + } + only := func(e string) bool { return len(events) == 1 && events[e] } + + switch { + case events[EventPullRequest] || events[EventPullRequestTarget] || events[EventMergeGroup]: + for _, id := range envJobs { + w.find(id, CodeEnvironmentOnPR, "PR-gating job targets an environment", workflowmodel.SeverityWarning) + } + return workflowmodel.RolePRVerify + case (events[EventPush] && pushTags) || events[EventRelease]: + return workflowmodel.RoleRelease + case events[EventPush]: + return workflowmodel.RolePushVerify + case events[EventWorkflowDispatch] && len(envJobs) > 0: + return workflowmodel.RoleDeploy + case events[EventDeployment] || events[EventDeploymentStatus]: + return workflowmodel.RoleDeploy + case onlyWithin(EventSchedule): + return workflowmodel.RoleScheduled + case onlyWithin(EventWorkflowDispatch): + return workflowmodel.RoleManual + case only(EventWorkflowCall): + w.find("", CodeReusableOnly, "called by other workflows", workflowmodel.SeverityWarning) + return workflowmodel.RoleOther + } + return workflowmodel.RoleOther +} + +// concurrency maps the top-level concurrency string or map. +func (w *walker) concurrency(n *yaml.Node) *workflowmodel.Concurrency { + n = deref(n) + if n == nil { + return nil + } + if n.Kind != yaml.MappingNode { + return &workflowmodel.Concurrency{Group: scalar(n)} + } + m := entries(n) + c := &workflowmodel.Concurrency{Group: scalar(m[keyGroup])} + cancel := deref(m[keyCancelInProgress]) + if cancel == nil || cancel.Kind != yaml.ScalarNode { + return c + } + switch { + case strings.EqualFold(cancel.Value, literalTrue): + c.CancelInProgress = true + case strings.EqualFold(cancel.Value, literalFalse) || cancel.Value == "": + c.CancelInProgress = false + default: + w.find("", CodeConcurrencyExpression, fmt.Sprintf("cancel-in-progress is %q", cancel.Value), workflowmodel.SeverityWarning) + } + return c +} + +// YAML node helpers. All of them resolve aliases and tolerate nil. + +func documentRoot(doc *yaml.Node) *yaml.Node { + if doc.Kind != yaml.DocumentNode || len(doc.Content) == 0 { + return nil + } + return deref(doc.Content[0]) +} + +func deref(n *yaml.Node) *yaml.Node { + for n != nil && n.Kind == yaml.AliasNode { + n = n.Alias + } + return n +} + +type pair struct { + key string + val *yaml.Node +} + +// pairs lists a mapping's entries in declaration order. A key YAML 1.1 +// would read as the boolean `on` is normalised back to "on". +func pairs(n *yaml.Node) []pair { + n = deref(n) + if n == nil || n.Kind != yaml.MappingNode { + return nil + } + out := make([]pair, 0, len(n.Content)/2) + for i := 0; i+1 < len(n.Content); i += 2 { + k := deref(n.Content[i]) + key := k.Value + if k.Tag == tagBool && strings.EqualFold(key, keyOn) { + key = keyOn + } + out = append(out, pair{key: key, val: n.Content[i+1]}) + } + return out +} + +func entries(n *yaml.Node) map[string]*yaml.Node { + ps := pairs(n) + if ps == nil { + return nil + } + m := make(map[string]*yaml.Node, len(ps)) + for _, p := range ps { + if _, dup := m[p.key]; !dup { + m[p.key] = p.val + } + } + return m +} + +func scalar(n *yaml.Node) string { + n = deref(n) + if n == nil || n.Kind != yaml.ScalarNode || n.Tag == tagNull { + return "" + } + return n.Value +} + +// strs reads a scalar as a one-element list or a sequence of scalars. +func strs(n *yaml.Node) []string { + n = deref(n) + if n == nil { + return nil + } + switch n.Kind { + case yaml.ScalarNode: + if s := scalar(n); s != "" { + return []string{s} + } + case yaml.SequenceNode: + var out []string + for _, c := range n.Content { + if s := scalar(c); s != "" { + out = append(out, s) + } + } + return out + } + return nil +} + +func baseName(p string) string { + b := path.Base(p) + return strings.TrimSuffix(b, path.Ext(b)) +} + +func sortFindings(fs []workflowmodel.Finding) { + sort.SliceStable(fs, func(i, j int) bool { + a, b := fs[i], fs[j] + if a.Path != b.Path { + return a.Path < b.Path + } + if a.Node != b.Node { + return a.Node < b.Node + } + return a.Code < b.Code + }) +} diff --git a/src/pkg/workflowmodel/githubactions/ingest_test.go b/src/pkg/workflowmodel/githubactions/ingest_test.go new file mode 100644 index 0000000000..65c4649f52 --- /dev/null +++ b/src/pkg/workflowmodel/githubactions/ingest_test.go @@ -0,0 +1,513 @@ +package githubactions + +import ( + "encoding/json" + "errors" + "os" + "path/filepath" + "reflect" + "strings" + "testing" + + "github.com/hivecommons/hive/pkg/workflowmodel" +) + +const jobsOneStep = ` +jobs: + a: + runs-on: ubuntu-latest + steps: [{run: echo}] +` + +func mustParse(t *testing.T, path, src string) (workflowmodel.Pipeline, []workflowmodel.Finding) { + t.Helper() + p, fs, err := ParseWorkflow(path, []byte(src)) + if err != nil { + t.Fatalf("ParseWorkflow(%s): %v", path, err) + } + return p, fs +} + +func hasFinding(fs []workflowmodel.Finding, code, node string) bool { + for _, f := range fs { + if f.Code == code && f.Node == node { + return true + } + } + return false +} + +func TestIsWorkflowPath(t *testing.T) { + cases := map[string]bool{ + ".github/workflows/ci.yml": true, + ".github/workflows/ci.yaml": true, + ".github/workflows/.yml": false, + ".github/workflows/sub/ci.yml": false, + ".github/workflows/ci.json": false, + ".github/workflows/README.md": false, + "workflows/ci.yml": false, + ".github/actions/setup/action.yml": false, + } + for p, want := range cases { + if got := IsWorkflowPath(p); got != want { + t.Errorf("IsWorkflowPath(%q) = %v, want %v", p, got, want) + } + } +} + +func TestTriggers(t *testing.T) { + cases := []struct { + name string + src string + want []workflowmodel.Trigger + }{ + {"string", "on: push\n" + jobsOneStep, []workflowmodel.Trigger{{Event: "push"}}}, + {"list", "on: [push, pull_request]\n" + jobsOneStep, []workflowmodel.Trigger{{Event: "push"}, {Event: "pull_request"}}}, + {"map", ` +on: + push: + branches: [main] + branches-ignore: [wip/*] + tags: ["v*"] + tags-ignore: ["v0*"] + paths: [src/**] + paths-ignore: ["**.md"] + pull_request: + types: [opened, synchronize] + schedule: + - cron: "0 3 * * *" + - cron: "0 15 * * 1" + workflow_dispatch: + inputs: + level: {type: string} + workflow_call: + workflow_run: + workflows: [CI, Lint] + types: [completed] +` + jobsOneStep, []workflowmodel.Trigger{ + {Event: "push", Branches: []string{"main", "!wip/*"}, Tags: []string{"v*", "!v0*"}, Paths: []string{"src/**", "!**.md"}}, + {Event: "pull_request", Types: []string{"opened", "synchronize"}}, + {Event: "schedule", Schedule: []string{"0 3 * * *", "0 15 * * 1"}}, + {Event: "workflow_dispatch"}, + {Event: "workflow_call"}, + {Event: "workflow_run", Types: []string{"CI", "Lint", "completed"}}, + }}, + {"bare-on-as-yaml11-bool", "on:\n push:\n branches: [main]\n" + jobsOneStep, + []workflowmodel.Trigger{{Event: "push", Branches: []string{"main"}}}}, + {"quoted-on", "\"on\": [push]\n" + jobsOneStep, []workflowmodel.Trigger{{Event: "push"}}}, + } + for _, tc := range cases { + t.Run(tc.name, func(t *testing.T) { + p, _ := mustParse(t, ".github/workflows/x.yml", tc.src) + if !reflect.DeepEqual(p.Triggers, tc.want) { + t.Errorf("triggers = %+v, want %+v", p.Triggers, tc.want) + } + }) + } +} + +func TestOnKeyNotDecodedAsBoolFalse(t *testing.T) { + // If the YAML library ever reads the bare `on` key as a boolean, the + // workflow must still be recognised. + p, _ := mustParse(t, ".github/workflows/x.yml", "name: X\non: push\njobs: {a: {steps: [{run: x}]}}\n") + if len(p.Triggers) != 1 || p.Triggers[0].Event != "push" { + t.Fatalf("triggers = %+v", p.Triggers) + } + if _, _, err := ParseWorkflow(".github/workflows/x.yml", []byte("name: X\n")); !errors.Is(err, ErrNotAWorkflow) { + t.Fatalf("name-only doc: err = %v, want ErrNotAWorkflow", err) + } +} + +func TestNameDefaultsToBasename(t *testing.T) { + p, _ := mustParse(t, ".github/workflows/nightly-build.yaml", "on: push\n"+jobsOneStep) + if p.Name != "nightly-build" { + t.Errorf("Name = %q", p.Name) + } + if p.ID != workflowmodel.PipelineID(workflowmodel.SystemGitHubActions, p.Path) { + t.Errorf("ID = %q", p.ID) + } + if p.System != workflowmodel.SystemGitHubActions { + t.Errorf("System = %q", p.System) + } +} + +func TestJobs(t *testing.T) { + src := ` +on: push +jobs: + a: + name: A job + runs-on: ubuntu-latest + if: github.actor != 'bot' + environment: staging + steps: [{run: echo}] + b: + needs: a + runs-on: [self-hosted, linux] + environment: + name: production + url: https://example.com + steps: [{run: echo}] + c: + needs: [a, b] + runs-on: + group: big-runners + labels: [gpu] + steps: [{run: echo}] + d: + needs: [c] + uses: org/repo/.github/workflows/reusable.yml@main + secrets: inherit + e: + runs-on: ${{ matrix.os }} + environment: ${{ inputs.env }} + container: {image: golang:1.23} + services: + db: {image: postgres} + steps: [{run: echo}] + f: + runs-on: ubuntu-latest +` + p, fs := mustParse(t, ".github/workflows/x.yml", src) + want := []workflowmodel.Node{ + {ID: "a", Name: "A job", Kind: workflowmodel.NodeJob, If: "github.actor != 'bot'", Environment: "staging", RunsOn: []string{"ubuntu-latest"}}, + {ID: "b", Kind: workflowmodel.NodeJob, Needs: []string{"a"}, Environment: "production", RunsOn: []string{"self-hosted", "linux"}, Level: 1}, + {ID: "c", Kind: workflowmodel.NodeJob, Needs: []string{"a", "b"}, RunsOn: []string{"big-runners", "gpu"}, Level: 2}, + {ID: "d", Kind: workflowmodel.NodeReusableWorkflow, Needs: []string{"c"}, Ref: "org/repo/.github/workflows/reusable.yml@main", Level: 3}, + {ID: "e", Kind: workflowmodel.NodeJob, Environment: "${{ inputs.env }}", RunsOn: []string{"${{ matrix.os }}"}}, + {ID: "f", Kind: workflowmodel.NodeJob, RunsOn: []string{"ubuntu-latest"}}, + } + if !reflect.DeepEqual(p.Nodes, want) { + got, _ := json.MarshalIndent(p.Nodes, "", " ") + t.Errorf("nodes:\n%s", got) + } + if p.Levels != 4 { + t.Errorf("Levels = %d, want 4", p.Levels) + } + for _, f := range []struct{ code, node string }{ + {CodeSecretsInherit, "d"}, {CodeJobContainer, "e"}, {CodeJobEmpty, "f"}, + } { + if !hasFinding(fs, f.code, f.node) { + t.Errorf("missing finding %s on %s in %+v", f.code, f.node, fs) + } + } + if hasFinding(fs, CodeJobEmpty, "d") { + t.Errorf("reusable job must not be flagged job-empty") + } +} + +func TestMatrix(t *testing.T) { + cases := []struct { + name string + src string + want *workflowmodel.Matrix + }{ + {"static", ` +on: push +jobs: + a: + strategy: + fail-fast: false + matrix: + os: [ubuntu, macos] + go: ["1.22", "1.23"] + include: + - {os: windows, go: "1.23"} + exclude: + - {os: macos, go: "1.22"} + - {os: ubuntu, go: "1.22"} + steps: [{run: echo}] +`, &workflowmodel.Matrix{Keys: []string{"go", "os"}, Include: 1, Exclude: 2}}, + {"dynamic-whole", ` +on: push +jobs: + a: + strategy: + matrix: ${{ fromJSON(needs.plan.outputs.matrix) }} + steps: [{run: echo}] +`, &workflowmodel.Matrix{Dynamic: true}}, + {"dynamic-value", ` +on: push +jobs: + a: + strategy: + matrix: + os: [ubuntu] + version: ${{ fromJSON(needs.plan.outputs.versions) }} + steps: [{run: echo}] +`, &workflowmodel.Matrix{Keys: []string{"os", "version"}, Dynamic: true}}, + {"none", "on: push\n" + jobsOneStep, nil}, + } + for _, tc := range cases { + t.Run(tc.name, func(t *testing.T) { + p, _ := mustParse(t, ".github/workflows/x.yml", tc.src) + if !reflect.DeepEqual(p.Nodes[0].Matrix, tc.want) { + t.Errorf("matrix = %+v, want %+v", p.Nodes[0].Matrix, tc.want) + } + }) + } +} + +func TestRole(t *testing.T) { + envJob := "\njobs:\n deploy:\n environment: prod\n steps: [{run: echo}]\n" + cases := []struct { + name string + src string + want workflowmodel.Role + finding string + }{ + {"pull_request", "on: pull_request\n" + jobsOneStep, workflowmodel.RolePRVerify, ""}, + {"pull_request_target", "on: [pull_request_target]\n" + jobsOneStep, workflowmodel.RolePRVerify, ""}, + {"merge_group", "on:\n merge_group:\n" + jobsOneStep, workflowmodel.RolePRVerify, ""}, + {"pr+push", "on: [push, pull_request]\n" + jobsOneStep, workflowmodel.RolePRVerify, ""}, + {"pr+environment", "on: pull_request" + envJob, workflowmodel.RolePRVerify, CodeEnvironmentOnPR}, + {"push-tags", "on:\n push:\n tags: ['v*']\n" + jobsOneStep, workflowmodel.RoleRelease, ""}, + {"push-tags-ignore-only", "on:\n push:\n tags-ignore: ['v0*']\n" + jobsOneStep, workflowmodel.RolePushVerify, ""}, + {"release", "on:\n release:\n types: [published]\n" + jobsOneStep, workflowmodel.RoleRelease, ""}, + {"push-branches", "on:\n push:\n branches: [main]\n" + jobsOneStep, workflowmodel.RolePushVerify, ""}, + {"push-unrestricted", "on: push\n" + jobsOneStep, workflowmodel.RolePushVerify, ""}, + {"push+dispatch", "on: [push, workflow_dispatch]\n" + jobsOneStep, workflowmodel.RolePushVerify, ""}, + {"dispatch+environment", "on: workflow_dispatch" + envJob, workflowmodel.RoleDeploy, ""}, + {"deployment", "on: deployment\n" + jobsOneStep, workflowmodel.RoleDeploy, ""}, + {"deployment_status", "on: [deployment_status]\n" + jobsOneStep, workflowmodel.RoleDeploy, ""}, + {"schedule-only", "on:\n schedule:\n - cron: '0 0 * * *'\n" + jobsOneStep, workflowmodel.RoleScheduled, ""}, + {"dispatch-only", "on: workflow_dispatch\n" + jobsOneStep, workflowmodel.RoleManual, ""}, + {"call-only", "on:\n workflow_call:\n inputs:\n x: {type: string}\n" + jobsOneStep, workflowmodel.RoleOther, CodeReusableOnly}, + {"schedule+dispatch", "on: [schedule, workflow_dispatch]\n" + jobsOneStep, workflowmodel.RoleScheduled, ""}, + {"schedule+dispatch+call", "on: [schedule, workflow_dispatch, workflow_call]\n" + jobsOneStep, workflowmodel.RoleScheduled, ""}, + {"schedule+issue_comment", "on: [schedule, issue_comment]\n" + jobsOneStep, workflowmodel.RoleOther, ""}, + {"dispatch+call", "on: [workflow_dispatch, workflow_call]\n" + jobsOneStep, workflowmodel.RoleManual, ""}, + {"dispatch+call-no-reusable-finding", "on: [workflow_call, workflow_dispatch]\n" + jobsOneStep, workflowmodel.RoleManual, ""}, + {"issue_comment", "on: issue_comment\n" + jobsOneStep, workflowmodel.RoleOther, ""}, + } + for _, tc := range cases { + t.Run(tc.name, func(t *testing.T) { + p, fs := mustParse(t, ".github/workflows/x.yml", tc.src) + if p.Role != tc.want { + t.Errorf("role = %q, want %q", p.Role, tc.want) + } + if tc.finding != "" && !hasFinding(fs, tc.finding, "") && !hasFinding(fs, tc.finding, "deploy") { + t.Errorf("missing finding %s in %+v", tc.finding, fs) + } + if tc.finding == "" && hasFinding(fs, CodeReusableOnly, "") { + t.Errorf("unexpected reusable-only finding in %+v", fs) + } + }) + } +} + +func TestNeedsRepair(t *testing.T) { + t.Run("unknown", func(t *testing.T) { + p, fs := mustParse(t, ".github/workflows/x.yml", ` +on: push +jobs: + a: + steps: [{run: echo}] + b: + needs: [a, ghost] + steps: [{run: echo}] +`) + if !hasFinding(fs, CodeNeedsUnresolved, "b") { + t.Fatalf("findings = %+v", fs) + } + if got := p.Nodes[1].Needs; !reflect.DeepEqual(got, []string{"a"}) { + t.Errorf("needs = %v, want [a]", got) + } + if p.Levels != 2 || p.Nodes[1].Level != 1 { + t.Errorf("Levels = %d, b.Level = %d", p.Levels, p.Nodes[1].Level) + } + }) + t.Run("cycle", func(t *testing.T) { + p, fs := mustParse(t, ".github/workflows/x.yml", ` +on: push +jobs: + a: + needs: b + steps: [{run: echo}] + b: + needs: a + steps: [{run: echo}] +`) + if !hasFinding(fs, CodeNeedsCycle, "") { + t.Fatalf("findings = %+v", fs) + } + if p.Levels != 1 { + t.Errorf("Levels = %d, want 1", p.Levels) + } + for _, n := range p.Nodes { + if n.Level != 0 { + t.Errorf("node %s level = %d, want 0", n.ID, n.Level) + } + } + }) +} + +func TestConcurrency(t *testing.T) { + cases := []struct { + name string + src string + want *workflowmodel.Concurrency + finding bool + }{ + {"none", "on: push\n" + jobsOneStep, nil, false}, + {"string", "on: push\nconcurrency: release\n" + jobsOneStep, &workflowmodel.Concurrency{Group: "release"}, false}, + {"map-true", "on: push\nconcurrency:\n group: ci-${{ github.ref }}\n cancel-in-progress: true\n" + jobsOneStep, + &workflowmodel.Concurrency{Group: "ci-${{ github.ref }}", CancelInProgress: true}, false}, + {"map-false", "on: push\nconcurrency:\n group: g\n cancel-in-progress: false\n" + jobsOneStep, + &workflowmodel.Concurrency{Group: "g"}, false}, + {"map-expression", "on: push\nconcurrency:\n group: g\n cancel-in-progress: ${{ github.event_name == 'pull_request' }}\n" + jobsOneStep, + &workflowmodel.Concurrency{Group: "g"}, true}, + } + for _, tc := range cases { + t.Run(tc.name, func(t *testing.T) { + p, fs := mustParse(t, ".github/workflows/x.yml", tc.src) + if !reflect.DeepEqual(p.Concurrency, tc.want) { + t.Errorf("concurrency = %+v, want %+v", p.Concurrency, tc.want) + } + if got := hasFinding(fs, CodeConcurrencyExpression, ""); got != tc.finding { + t.Errorf("concurrency-expression finding = %v, want %v", got, tc.finding) + } + }) + } +} + +func TestNoJobs(t *testing.T) { + for _, src := range []string{"on: push\n", "on: push\njobs: {}\n", "on: push\njobs:\n"} { + p, fs := mustParse(t, ".github/workflows/x.yml", src) + if len(p.Nodes) != 0 || p.Levels != 0 || !hasFinding(fs, CodeNoJobs, "") { + t.Errorf("%q: nodes=%d levels=%d findings=%+v", src, len(p.Nodes), p.Levels, fs) + } + } +} + +func TestParseErrors(t *testing.T) { + cases := []struct { + name string + src string + want error + }{ + {"yaml", "on: [push\njobs: {", ErrYAML}, + {"scalar-doc", "just a string\n", ErrNotAWorkflow}, + {"list-doc", "- a\n- b\n", ErrNotAWorkflow}, + {"empty-doc", "", ErrNotAWorkflow}, + {"action-metadata", "name: Setup\nruns:\n using: composite\n", ErrNotAWorkflow}, + } + for _, tc := range cases { + t.Run(tc.name, func(t *testing.T) { + _, _, err := ParseWorkflow(".github/workflows/x.yml", []byte(tc.src)) + if !errors.Is(err, tc.want) { + t.Errorf("err = %v, want %v", err, tc.want) + } + }) + } +} + +func TestIngest(t *testing.T) { + files := []File{ + {Path: ".github/workflows/zeta.yml", Data: []byte("on: push\n" + jobsOneStep)}, + {Path: ".github/workflows/broken.yml", Data: []byte("on: [\n")}, + {Path: ".github/workflows/alpha.yml", Data: []byte("on: pull_request\njobs:\n b:\n uses: o/r/.github/workflows/w.yml@v1\n secrets: inherit\n a:\n runs-on: x\n")}, + {Path: ".github/workflows/action.yml", Data: []byte("name: not a workflow\nruns: {using: node20}\n")}, + } + r1 := Ingest(files) + r2 := Ingest(files) + j1, _ := json.Marshal(r1) + j2, _ := json.Marshal(r2) + if string(j1) != string(j2) { + t.Fatalf("Ingest is not deterministic:\n%s\n%s", j1, j2) + } + + if r1.Coverage.Files != 4 || r1.Coverage.Parsed != 2 { + t.Errorf("coverage = files %d parsed %d", r1.Coverage.Files, r1.Coverage.Parsed) + } + if len(r1.Sources) != 4 { + t.Errorf("sources = %d", len(r1.Sources)) + } + for i, s := range r1.Sources { + if s.System != workflowmodel.SystemGitHubActions || s.SHA256 == "" { + t.Errorf("source %d = %+v", i, s) + } + } + var paths []string + for _, p := range r1.Pipelines { + paths = append(paths, p.Path) + } + if !reflect.DeepEqual(paths, []string{".github/workflows/alpha.yml", ".github/workflows/zeta.yml"}) { + t.Errorf("pipeline order = %v", paths) + } + if ids := []string{r1.Pipelines[0].Nodes[0].ID, r1.Pipelines[0].Nodes[1].ID}; !reflect.DeepEqual(ids, []string{"b", "a"}) { + t.Errorf("node order = %v, want declaration order [b a]", ids) + } + + var keys []string + for _, f := range r1.Coverage.Findings { + keys = append(keys, f.Path+"|"+f.Node+"|"+f.Code) + } + wantKeys := []string{ + ".github/workflows/action.yml||" + CodeNotAWorkflow, + ".github/workflows/alpha.yml|a|" + CodeJobEmpty, + ".github/workflows/alpha.yml|b|" + CodeSecretsInherit, + ".github/workflows/broken.yml||" + CodeYAMLParse, + } + if !reflect.DeepEqual(keys, wantKeys) { + t.Errorf("findings = %v\nwant %v", keys, wantKeys) + } + for _, f := range r1.Coverage.Findings { + if (f.Code == CodeYAMLParse || f.Code == CodeNotAWorkflow) && f.Severity != workflowmodel.SeverityUnknown { + t.Errorf("%s severity = %s", f.Code, f.Severity) + } + } +} + +func TestFixtures(t *testing.T) { + read := func(name string) File { + t.Helper() + b, err := os.ReadFile(filepath.Join("testdata", name)) + if err != nil { + t.Fatal(err) + } + return File{Path: ".github/workflows/" + name, Data: b} + } + r := Ingest([]File{read("release.yml"), read("kubernetes-ci.yml")}) + if r.Coverage.Files != 2 || r.Coverage.Parsed != 2 || len(r.Coverage.Findings) != 0 { + t.Fatalf("coverage = %+v", r.Coverage) + } + + ci := r.Pipelines[0] + if ci.Name != "CI" || ci.Role != workflowmodel.RolePRVerify || ci.Levels != 4 { + t.Errorf("ci: name=%q role=%q levels=%d", ci.Name, ci.Role, ci.Levels) + } + wantLevels := map[string]int{"lint": 0, "build": 1, "unit": 2, "integration": 2, "e2e": 3} + if len(ci.Nodes) != len(wantLevels) { + t.Fatalf("ci nodes = %d", len(ci.Nodes)) + } + for _, n := range ci.Nodes { + if n.Level != wantLevels[n.ID] { + t.Errorf("ci %s level = %d, want %d", n.ID, n.Level, wantLevels[n.ID]) + } + } + if got := workflowmodel.ByLevel(ci.Nodes); !reflect.DeepEqual(got, [][]string{{"lint"}, {"build"}, {"integration", "unit"}, {"e2e"}}) { + t.Errorf("ByLevel = %v", got) + } + integ := ci.Nodes[3] + if integ.Matrix == nil || !reflect.DeepEqual(integ.Matrix.Keys, []string{"go", "os"}) || integ.Matrix.Exclude != 1 || integ.Matrix.Dynamic { + t.Errorf("integration matrix = %+v", integ.Matrix) + } + if !strings.HasPrefix(integ.Name, "Integration (") { + t.Errorf("integration name = %q", integ.Name) + } + if ci.Concurrency == nil || !ci.Concurrency.CancelInProgress { + t.Errorf("ci concurrency = %+v", ci.Concurrency) + } + if len(ci.Triggers) != 2 || ci.Triggers[0].Event != "pull_request" || !reflect.DeepEqual(ci.Triggers[0].Paths, []string{"!docs/**", "!**.md"}) { + t.Errorf("ci triggers = %+v", ci.Triggers) + } + + rel := r.Pipelines[1] + if rel.Name != "Release" || rel.Role != workflowmodel.RoleRelease || rel.Levels != 2 { + t.Errorf("release: name=%q role=%q levels=%d", rel.Name, rel.Role, rel.Levels) + } + if rel.Nodes[1].Environment != "production" || rel.Nodes[1].Level != 1 { + t.Errorf("release publish-image = %+v", rel.Nodes[1]) + } +} diff --git a/src/pkg/workflowmodel/githubactions/testdata/kubernetes-ci.yml b/src/pkg/workflowmodel/githubactions/testdata/kubernetes-ci.yml new file mode 100644 index 0000000000..349fed8264 --- /dev/null +++ b/src/pkg/workflowmodel/githubactions/testdata/kubernetes-ci.yml @@ -0,0 +1,71 @@ +# Hand-written fixture modelled on a typical multi-job Go project CI. +name: CI + +on: + pull_request: + branches: [main, "release-*"] + paths-ignore: + - "docs/**" + - "**.md" + push: + branches: [main] + +concurrency: + group: ci-${{ github.workflow }}-${{ github.ref }} + cancel-in-progress: true + +permissions: + contents: read + +env: + GO_VERSION: "1.23" + +jobs: + lint: + name: Lint + runs-on: ubuntu-latest + timeout-minutes: 15 + steps: + - uses: actions/checkout@v4 + - uses: golangci/golangci-lint-action@v6 + + build: + name: Build + needs: lint + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + - run: make build + + unit: + name: Unit tests + needs: build + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + - run: make test + + integration: + name: Integration (${{ matrix.os }}, go ${{ matrix.go }}) + needs: build + runs-on: ${{ matrix.os }} + strategy: + fail-fast: false + matrix: + go: ["1.22", "1.23"] + os: [ubuntu-latest, macos-latest] + exclude: + - go: "1.22" + os: macos-latest + steps: + - uses: actions/checkout@v4 + - run: make integration + + e2e: + name: E2E + needs: [unit, integration] + runs-on: ubuntu-latest + if: github.event_name == 'push' + steps: + - uses: actions/checkout@v4 + - run: make e2e diff --git a/src/pkg/workflowmodel/githubactions/testdata/release.yml b/src/pkg/workflowmodel/githubactions/testdata/release.yml new file mode 100644 index 0000000000..636c295c0d --- /dev/null +++ b/src/pkg/workflowmodel/githubactions/testdata/release.yml @@ -0,0 +1,22 @@ +name: Release + +on: + push: + tags: + - "v*" + +permissions: + contents: write + +jobs: + goreleaser: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + - uses: goreleaser/goreleaser-action@v6 + publish-image: + needs: goreleaser + runs-on: ubuntu-latest + environment: production + steps: + - run: make image-push diff --git a/src/pkg/workflowmodel/hivedefault/hivedefault.go b/src/pkg/workflowmodel/hivedefault/hivedefault.go new file mode 100644 index 0000000000..80f488b950 --- /dev/null +++ b/src/pkg/workflowmodel/hivedefault/hivedefault.go @@ -0,0 +1,254 @@ +// Package hivedefault generates the `hive-default` workflow model: Hive's +// own lifecycle expressed in the workflow-model schema +// (hivecommons/hive#17620). Nothing here is hand-maintained: states come +// from stagelabels.Table, transitions from stageengine.DiagramEdges and +// the columns from the Activity Monitor's layout constants, so the model +// is the schema's conformance test — if it cannot express Hive's graph, +// the schema is not done. +// +// The model declares derivation mode builtin: an item's state is still +// decided by the Go derivers, and DeriveFromModel only checks that what +// they produce is a state the model knows. That is what keeps the stage +// goldens byte-for-byte and the v5/v6 monitor pixel-identical for +// repositories that bind nothing. +package hivedefault + +import ( + "fmt" + "sort" + "strings" + + "github.com/hivecommons/hive/pkg/stageengine" + "github.com/hivecommons/hive/pkg/stagelabels" + "github.com/hivecommons/hive/pkg/workflowmodel" +) + +// Name is the model's metadata.name; Version its spec.meta.version. The +// version is bumped by hand when the generator's output changes shape. +const ( + Name = "hive-default" + Version = "1" +) + +// Column ids. These are the Activity Monitor's columns in order; the +// anchor numbers mirror STAGE_TRACKER_PATH = [0, 10, 20, 30, 50, 80] plus +// the terminal Merged cap (agent-now.js, tracker-motion.js). +const ( + ColumnOpen = "open" + ColumnActionable = "actionable" + ColumnClaimed = "claimed" + ColumnPRInProgress = "pr-in-progress" + ColumnCIReview = "ci-review" + ColumnMergeQueue = "merge-queue" + ColumnMerged = "merged" +) + +// Terminal state ids (no stage label; observed, not written). +const ( + StateOpen = "open" + StateMerged = "merged" + StateClosed = "closed" +) + +// Role ids. +const ( + RoleMaintainer = "maintainer" + RoleAuthor = "author" + RoleReviewSwarm = "review-swarm" + RoleHiveAgent = "hive-agent" + RoleCI = "ci" +) + +// Gate and event ids. +const ( + GateMaintainerDecision = "maintainer-decision" + GateMaintainerApprove = "maintainer-approve" + GateAuthorPush = "author-push" + EventOpened = "opened" + EventMerged = "merged" + EventClosed = "closed" + EventStageChange = "stage-change" +) + +// columnAnchors maps a stage number to the column that lights for it, +// mirroring stageTrackerColumn / tracker-row-anchor.js. Every Table +// number must be here; Generate panics otherwise. +var columnAnchors = map[int]string{ + 10: ColumnActionable, 15: ColumnActionable, 17: ColumnActionable, 18: ColumnActionable, 45: ColumnActionable, + 20: ColumnClaimed, + 30: ColumnPRInProgress, 40: ColumnPRInProgress, + 50: ColumnCIReview, 60: ColumnCIReview, 70: ColumnCIReview, 71: ColumnCIReview, 72: ColumnCIReview, 74: ColumnCIReview, + 80: ColumnMergeQueue, 85: ColumnMergeQueue, 90: ColumnMergeQueue, 95: ColumnMergeQueue, 100: ColumnMergeQueue, +} + +// replies are the human-gate verbs the monitor offers in a stage today +// (tracker-actions.js); declared here so a model can carry its own. +var replies = map[int][]string{ + 15: {"accept-triage", "decide:close-issue"}, + 17: {"decide:approve", "decide:close-superseded"}, + 45: {"decide:delegate", "decide:approve", "decide:direction", "decide:close-superseded", "decide:close-wontfix", "hold"}, + 85: {"decide:delegate", "decide:approve", "decide:direction", "decide:merge-now", "decide:queue-merge", "decide:adopt", "decide:close-superseded", "hold"}, + 90: {"decide:approve", "decide:merge-now"}, +} + +// owner says who moves work out of a stage. +func owner(d stagelabels.Definition) workflowmodel.Owner { + switch d.Stage { + case stagelabels.TriageNeeded, stagelabels.PossiblyExisting, stagelabels.OpenForDiscussion, stagelabels.IssueNeedsDecision, + stagelabels.NeedsDecision, stagelabels.ReadyForHumanMerge, stagelabels.NeedsAuthor, stagelabels.ReturnedToContributor: + return workflowmodel.OwnerHuman + case stagelabels.ReviewPending, stagelabels.Reviewed, stagelabels.AwaitingMergeAdmission, stagelabels.PreMergeSoak, stagelabels.MergeQueue: + return workflowmodel.OwnerExternalService + case stagelabels.CoveredByPR: + return workflowmodel.OwnerExternalObservation + } + return workflowmodel.OwnerRegisteredAgent +} + +// StateID is the model state id for a stage: its slug, suffixed with the +// item kind where two stages share a slug (045 and 085 needs-decision). +func StateID(s stagelabels.Stage) string { + switch s { + case stagelabels.None: + return "" + case stagelabels.IssueNeedsDecision: + return "needs-decision-issue" + case stagelabels.NeedsDecision: + return "needs-decision-pr" + } + return s.Slug() +} + +// Generate builds the hive-default model from the live stage table. +func Generate() *workflowmodel.Model { + m := workflowmodel.New(Name, workflowmodel.Scope{Forge: "github.com", Repository: "*"}) + m.Spec.Meta = workflowmodel.Meta{Version: Version, Author: "hivecommons", Description: "Hive's built-in issue and pull-request lifecycle, generated from the stage table", License: "Apache-2.0", Source: workflowmodel.SourceBuiltIn, Provenance: workflowmodel.ProvenanceBuiltIn} + m.Spec.Initial = StateOpen + m.Spec.Derivation = workflowmodel.Derivation{Mode: workflowmodel.DerivationBuiltin} + m.Spec.Labels.Prefix = stagelabels.Prefix + + order := []workflowmodel.Column{ + {ID: ColumnOpen, Label: "Open", Hint: "Filed; not yet staged", States: []string{StateOpen}}, + {ID: ColumnActionable, Label: "Actionable", Hint: "Triaged or parked on a person"}, + {ID: ColumnClaimed, Label: "Claimed", Hint: "An agent or contributor holds it"}, + {ID: ColumnPRInProgress, Label: "PR in progress", Hint: "A PR is under way or open"}, + {ID: ColumnCIReview, Label: "CI / Review", Hint: "Verdict and checks on the current head"}, + {ID: ColumnMergeQueue, Label: "Merge queue", Hint: "Admitted; waiting on soak, tide or a human merge"}, + {ID: ColumnMerged, Label: "Merged", Hint: "Done", Terminal: true, States: []string{StateMerged, StateClosed}}, + } + cols := map[string]*workflowmodel.Column{} + for i := range order { + cols[order[i].ID] = &order[i] + } + + m.Spec.States = append(m.Spec.States, workflowmodel.State{ID: StateOpen, Label: "Open", Owner: workflowmodel.OwnerExternalObservation, Origin: workflowmodel.OriginHiveDefault, Lifecycle: workflowmodel.StateLifecycle{Column: ColumnOpen}}) + + table := append([]stagelabels.Definition(nil), stagelabels.Table...) + sort.SliceStable(table, func(i, j int) bool { return table[i].Number < table[j].Number }) + for _, d := range table { + id := StateID(d.Stage) + col, ok := columnAnchors[d.Number] + if !ok { + panic(fmt.Sprintf("hivedefault: stage %d has no column anchor", d.Number)) + } + var kinds []workflowmodel.ItemKind + if d.Issue { + kinds = append(kinds, workflowmodel.KindIssue) + } + if d.PR { + kinds = append(kinds, workflowmodel.KindPR) + } + st := workflowmodel.State{ID: id, Label: label(d), Owner: owner(d), Origin: workflowmodel.OriginHiveDefault, Lifecycle: workflowmodel.StateLifecycle{Column: col, Number: d.Number, Kinds: kinds, Replies: replies[d.Number]}} + if d.Dwell > 0 { + st.Lifecycle.Waiting = []string{"stalled after " + d.Dwell.String()} + } + m.Spec.States = append(m.Spec.States, st) + cols[col].States = append(cols[col].States, id) + m.Spec.Labels.Entries = append(m.Spec.Labels.Entries, workflowmodel.LabelEntry{Name: d.Stage.Label(), Description: d.Description, State: id}) + } + m.Spec.States = append(m.Spec.States, + workflowmodel.State{ID: StateMerged, Label: "Merged", Owner: workflowmodel.OwnerExternalObservation, Type: workflowmodel.StateFinal, Origin: workflowmodel.OriginHiveDefault, Lifecycle: workflowmodel.StateLifecycle{Column: ColumnMerged, Kinds: []workflowmodel.ItemKind{workflowmodel.KindPR}}}, + workflowmodel.State{ID: StateClosed, Label: "Closed", Owner: workflowmodel.OwnerExternalObservation, Type: workflowmodel.StateFinal, Origin: workflowmodel.OriginHiveDefault, Lifecycle: workflowmodel.StateLifecycle{Column: ColumnMerged}}, + ) + m.Spec.Columns = order + + m.Spec.Roles = []workflowmodel.RoleDef{ + {ID: RoleMaintainer, Label: "Maintainer", Kind: workflowmodel.RolePermission, Params: map[string]string{"permission": "write"}, Human: true}, + {ID: RoleAuthor, Label: "Author", Kind: workflowmodel.RoleAuthorKind, Params: map[string]string{"kind": "human"}, Human: true}, + {ID: RoleReviewSwarm, Label: "Review swarm", Kind: workflowmodel.RoleBot, Params: map[string]string{"login": "hive"}}, + {ID: RoleHiveAgent, Label: "Hive agent", Kind: workflowmodel.RoleBot, Params: map[string]string{"login": "hive"}}, + {ID: RoleCI, Label: "CI", Kind: workflowmodel.RoleBot}, + } + m.Spec.Gates = []workflowmodel.Gate{ + {ID: GateMaintainerDecision, ReleasedBy: RoleMaintainer, Reply: "/hive decision"}, + {ID: GateMaintainerApprove, ReleasedBy: RoleMaintainer, Reply: "/lgtm /approve", Conditions: []workflowmodel.Predicate{{Fact: "pr.checks", Op: workflowmodel.OpIs, Value: "green"}}}, + {ID: GateAuthorPush, ReleasedBy: RoleAuthor}, + } + m.Spec.Events = []workflowmodel.Event{ + {ID: EventOpened, Label: "Item opened", Fact: workflowmodel.Predicate{Fact: "open", Op: workflowmodel.OpTrue}}, + {ID: EventMerged, Label: "PR merged", Fact: workflowmodel.Predicate{Fact: "merged", Op: workflowmodel.OpTrue}}, + {ID: EventClosed, Label: "Item closed unmerged", Fact: workflowmodel.Predicate{Fact: "open", Op: workflowmodel.OpFalse}}, + {ID: "claimed", Label: "Claim taken", Fact: workflowmodel.Predicate{Fact: "issue.claimed", Op: workflowmodel.OpTrue}}, + {ID: "covered", Label: "PR covers the issue", Fact: workflowmodel.Predicate{Fact: "issue.covered", Op: workflowmodel.OpTrue}}, + {ID: "checks-green", Label: "Checks green", Fact: workflowmodel.Predicate{Fact: "pr.checks", Op: workflowmodel.OpIs, Value: "green"}}, + {ID: "checks-red", Label: "Checks red", Fact: workflowmodel.Predicate{Fact: "pr.checks", Op: workflowmodel.OpIs, Value: "red"}}, + {ID: "review-approved", Label: "Verdict approve", Fact: workflowmodel.Predicate{Fact: "pr.review", Op: workflowmodel.OpIs, Value: "approved"}}, + {ID: "changes-requested", Label: "Changes requested", Fact: workflowmodel.Predicate{Fact: "pr.changes_requested", Op: workflowmodel.OpTrue}}, + {ID: "held", Label: "Hold label applied", Fact: workflowmodel.Predicate{Fact: "hold", Op: workflowmodel.OpTrue}}, + {ID: "parked", Label: "Park label applied", Fact: workflowmodel.Predicate{Fact: "park", Op: workflowmodel.OpTrue}}, + {ID: EventStageChange, Label: "Stage re-derived", Fact: workflowmodel.Predicate{Fact: "labels", Op: workflowmodel.OpHas, Value: stagelabels.Prefix + "*"}}, + } + m.Spec.Agents = []workflowmodel.AgentGrant{ + {Profile: "scanner", States: []string{StateID(stagelabels.Actionable), StateID(stagelabels.TriageNeeded), StateID(stagelabels.OpenForDiscussion)}, Produces: []string{"claim"}}, + {Profile: "worker", States: []string{StateID(stagelabels.Claimed), StateID(stagelabels.PRInProgress), StateID(stagelabels.Fixing), StateID(stagelabels.AdoptedByHive)}, Produces: []string{"pull-request", "push"}}, + {Profile: "review-swarm", States: []string{StateID(stagelabels.ReviewPending)}, Produces: []string{"review"}}, + } + + // Transitions from the expected-edge table; gates on the edges a human + // releases. + for _, e := range stageengine.DiagramEdges() { + t := workflowmodel.Transition{From: StateID(e.From), To: StateID(e.To), On: EventStageChange, Note: fmt.Sprintf("%s → %s", e.From, e.To)} + switch { + case e.From == stagelabels.IssueNeedsDecision || e.From == stagelabels.NeedsDecision || e.From == stagelabels.PossiblyExisting: + t.GateRef = GateMaintainerDecision + case e.From == stagelabels.ReadyForHumanMerge && e.To == stagelabels.MergeQueue: + t.GateRef = GateMaintainerApprove + case e.From == stagelabels.NeedsAuthor || e.From == stagelabels.ReturnedToContributor: + t.GateRef = GateAuthorPush + } + m.Spec.Transitions = append(m.Spec.Transitions, t) + } + // Intake and terminal edges, which the diagram draws as [*]. + m.Spec.Transitions = append(m.Spec.Transitions, + workflowmodel.Transition{From: StateOpen, To: StateID(stagelabels.Actionable), On: EventOpened, Note: "triaged intake"}, + workflowmodel.Transition{From: StateOpen, To: StateID(stagelabels.TriageNeeded), On: EventOpened, Note: "reporter-trust gate"}, + workflowmodel.Transition{From: StateOpen, To: StateID(stagelabels.ReviewPending), On: EventOpened, Note: "PR opened"}, + ) + for _, d := range table { + switch d.Stage { + case stagelabels.MergeQueue: + m.Spec.Transitions = append(m.Spec.Transitions, workflowmodel.Transition{From: StateID(d.Stage), To: StateMerged, On: EventMerged, GateRef: GateMaintainerApprove, Note: "tide / lane merges"}) + case stagelabels.ReadyForHumanMerge: + m.Spec.Transitions = append(m.Spec.Transitions, workflowmodel.Transition{From: StateID(d.Stage), To: StateMerged, On: EventMerged, GateRef: GateMaintainerApprove, Note: "maintainer merges"}) + } + m.Spec.Transitions = append(m.Spec.Transitions, workflowmodel.Transition{From: StateID(d.Stage), To: StateClosed, On: EventClosed}) + } + return m +} + +// label is the display label for a stage: its slug, humanised. +func label(d stagelabels.Definition) string { + s := strings.ReplaceAll(d.Slug, "-", " ") + return strings.ToUpper(s[:1]) + s[1:] +} + +// Profile generates and compiles the model. It panics on a compile +// error: the generator and the schema ship together, so a refusal is a +// programming error, not an operator one. +func Profile() *workflowmodel.Profile { + p, diags, err := workflowmodel.Compile(Generate()) + if err != nil { + panic(fmt.Sprintf("hivedefault: %v (%d diagnostics)", err, len(diags))) + } + return p +} diff --git a/src/pkg/workflowmodel/hivedefault/hivedefault_test.go b/src/pkg/workflowmodel/hivedefault/hivedefault_test.go new file mode 100644 index 0000000000..585029e136 --- /dev/null +++ b/src/pkg/workflowmodel/hivedefault/hivedefault_test.go @@ -0,0 +1,151 @@ +package hivedefault + +import ( + "encoding/json" + "testing" + + "github.com/hivecommons/hive/pkg/stageengine" + "github.com/hivecommons/hive/pkg/stagelabels" + "github.com/hivecommons/hive/pkg/workflowmodel" +) + +func TestGenerateCompiles(t *testing.T) { + m := Generate() + if err := m.Validate(); err != nil { + t.Fatal(err) + } + p, diags, err := workflowmodel.Compile(m) + if err != nil { + t.Fatalf("%v\n%+v", err, diags) + } + if len(diags) != 0 { + t.Errorf("hive-default must compile clean, got %+v", diags) + } + if p.Name != Name || p.Version != Version || p.Mode != workflowmodel.DerivationBuiltin { + t.Fatalf("profile %s@%s mode %s", p.Name, p.Version, p.Mode) + } + if Profile().Hash != p.Hash { + t.Fatal("Profile() is not Generate() compiled") + } +} + +func TestEveryStageIsAState(t *testing.T) { + p := Profile() + ids := map[string]bool{} + for _, d := range stagelabels.Table { + id := StateID(d.Stage) + if ids[id] { + t.Errorf("StateID(%s) = %q collides", d.Stage, id) + } + ids[id] = true + s, ok := p.States[id] + if !ok { + t.Errorf("stage %s has no state %q", d.Stage, id) + continue + } + if s.Lifecycle.Number != d.Number { + t.Errorf("state %q number %d, stage %d", id, s.Lifecycle.Number, d.Number) + } + if got, ok := p.StateForNumber(d.Number); !ok || got != id { + t.Errorf("StateForNumber(%d) = %q %v, want %q", d.Number, got, ok, id) + } + if p.ColumnOf[id] == "" || p.ColumnOf[id] == ColumnOpen || p.ColumnOf[id] == ColumnMerged { + t.Errorf("state %q sits in column %q", id, p.ColumnOf[id]) + } + wantKind := workflowmodel.KindIssue + if d.PR { + wantKind = workflowmodel.KindPR + } + if len(s.Lifecycle.Kinds) != 1 || s.Lifecycle.Kinds[0] != wantKind { + t.Errorf("state %q kinds %v, want [%s]", id, s.Lifecycle.Kinds, wantKind) + } + } + if StateID(stagelabels.IssueNeedsDecision) == StateID(stagelabels.NeedsDecision) { + t.Error("045 and 085 must have distinct state ids") + } +} + +func TestColumnsMatchTheMonitor(t *testing.T) { + p := Profile() + want := []string{ColumnOpen, ColumnActionable, ColumnClaimed, ColumnPRInProgress, ColumnCIReview, ColumnMergeQueue, ColumnMerged} + if len(p.Columns) != len(want) { + t.Fatalf("columns %+v", p.Columns) + } + for i, c := range p.Columns { + if c.ID != want[i] { + t.Errorf("column %d = %q, want %q", i, c.ID, want[i]) + } + if c.Terminal != (c.ID == ColumnMerged) { + t.Errorf("column %q terminal %v", c.ID, c.Terminal) + } + } + // tracker-row-anchor.js: 15/17/18/45 light Actionable, 40 lights PR in + // progress, 6x/7x light CI / Review, 85+ light Merge queue. + for n, col := range map[int]string{10: ColumnActionable, 15: ColumnActionable, 45: ColumnActionable, 20: ColumnClaimed, 30: ColumnPRInProgress, 40: ColumnPRInProgress, 50: ColumnCIReview, 60: ColumnCIReview, 74: ColumnCIReview, 80: ColumnMergeQueue, 85: ColumnMergeQueue, 100: ColumnMergeQueue} { + id, _ := p.StateForNumber(n) + if p.ColumnOf[id] != col { + t.Errorf("stage %d (%s) in %q, want %q", n, id, p.ColumnOf[id], col) + } + } + if !p.Terminal[StateMerged] || !p.Terminal[StateClosed] || p.Terminal[StateOpen] { + t.Errorf("terminal %v", p.Terminal) + } +} + +// The transition graph is the stage engine's expected-edge table plus +// intake and closure: no edge the engine would log as an anomaly. +func TestTransitionsMirrorTheEngine(t *testing.T) { + m := Generate() + have := map[[2]string]bool{} + for _, tr := range m.Spec.Transitions { + have[[2]string{tr.From, tr.To}] = true + } + for _, e := range stageengine.DiagramEdges() { + k := [2]string{StateID(e.From), StateID(e.To)} + if !have[k] { + t.Errorf("engine edge %s -> %s missing from the model", e.From, e.To) + } + } + for _, d := range stagelabels.Table { + id := StateID(d.Stage) + if !have[[2]string{id, StateClosed}] { + t.Errorf("%s has no closure edge", id) + } + } + if !have[[2]string{StateOpen, StateID(stagelabels.Actionable)}] || !have[[2]string{StateOpen, StateID(stagelabels.ReviewPending)}] { + t.Error("intake edges missing") + } +} + +func TestModelRoundTripsJSON(t *testing.T) { + m := Generate() + b, err := json.Marshal(m) + if err != nil { + t.Fatal(err) + } + var back workflowmodel.Model + if err := json.Unmarshal(b, &back); err != nil { + t.Fatal(err) + } + p1, _, err := workflowmodel.Compile(&back) + if err != nil { + t.Fatal(err) + } + if p1.Hash != Profile().Hash { + t.Fatal("hash changed across a JSON round trip") + } +} + +func TestOwnerAndStateIDHelpers(t *testing.T) { + if StateID(stagelabels.None) != "" { + t.Error("None is not a state") + } + for _, d := range stagelabels.Table { + if owner(d) == "" { + t.Errorf("stage %s has no owner", d.Stage) + } + if label(d) == "" { + t.Errorf("stage %s has no label", d.Stage) + } + } +} diff --git a/src/pkg/workflowmodel/ingest/corpus_test.go b/src/pkg/workflowmodel/ingest/corpus_test.go new file mode 100644 index 0000000000..f3f4c66e4c --- /dev/null +++ b/src/pkg/workflowmodel/ingest/corpus_test.go @@ -0,0 +1,209 @@ +package ingest + +import ( + "encoding/json" + "flag" + "os" + "path/filepath" + "sort" + "strings" + "testing" + + "github.com/hivecommons/hive/pkg/workflowmodel" + "github.com/hivecommons/hive/pkg/workflowmodel/render" +) + +var update = flag.Bool("update", false, "rewrite corpus goldens") + +// corpusDir holds real CI definitions copied from public CNCF-landscape +// repositories (the proving corpus for hivecommons/hive#17620), one +// directory per repository named owner__repo. +const corpusDir = "../testdata/corpus" + +// golden is the stable summary a repository's model must keep producing. +type golden struct { + Columns []string `json:"columns"` + Roles map[workflowmodel.Role]int `json:"roles"` + Pipelines []goldenPipeline `json:"pipelines"` + Substages []string `json:"ciSubstages"` + Findings map[string]int `json:"findings"` + Diff []string `json:"columnDiff"` +} + +type goldenPipeline struct { + ID string `json:"id"` + Role workflowmodel.Role `json:"role"` + Nodes int `json:"nodes"` + Levels int `json:"levels"` + Events []string `json:"events"` +} + +func readRepo(t *testing.T, dir string) map[string][]byte { + t.Helper() + files := map[string][]byte{} + err := filepath.WalkDir(dir, func(p string, d os.DirEntry, err error) error { + if err != nil || d.IsDir() || strings.HasSuffix(p, ".golden.json") { + return err + } + b, err := os.ReadFile(p) + if err != nil { + return err + } + rel, _ := filepath.Rel(dir, p) + files[filepath.ToSlash(rel)] = b + return nil + }) + if err != nil { + t.Fatal(err) + } + return files +} + +func summarize(m *workflowmodel.Model) golden { + g := golden{Roles: map[workflowmodel.Role]int{}, Findings: map[string]int{}, Diff: render.Diff(m)} + for _, c := range render.Columns(m) { + g.Columns = append(g.Columns, c.Label) + for _, s := range c.Substages { + g.Substages = append(g.Substages, s.Label) + } + } + for _, p := range m.Spec.Pipelines { + g.Roles[p.Role]++ + gp := goldenPipeline{ID: p.ID, Role: p.Role, Nodes: len(p.Nodes), Levels: p.Levels} + for _, t := range p.Triggers { + gp.Events = append(gp.Events, t.Event) + } + g.Pipelines = append(g.Pipelines, gp) + } + for _, f := range m.Spec.Coverage.Findings { + g.Findings[f.Code]++ + } + return g +} + +// Every corpus repository must ingest into a valid model whose summary +// matches its golden; run with -update after an intended change. +func TestCorpusGoldens(t *testing.T) { + entries, err := os.ReadDir(corpusDir) + if err != nil { + t.Fatal(err) + } + for _, e := range entries { + if !e.IsDir() { + continue + } + t.Run(e.Name(), func(t *testing.T) { + repo := strings.Replace(e.Name(), "__", "/", 1) + m := Repository(workflowmodel.Scope{Forge: "github.com", Repository: repo}, readRepo(t, filepath.Join(corpusDir, e.Name()))) + if err := m.Validate(); err != nil { + t.Fatal(err) + } + // Deterministic: a second ingest hashes the same. + if again := Repository(m.Spec.Scope, readRepo(t, filepath.Join(corpusDir, e.Name()))); again.Hash() != m.Hash() { + t.Fatal("ingest is not deterministic") + } + got, _ := json.MarshalIndent(summarize(m), "", " ") + got = append(got, '\n') + goldenPath := filepath.Join(corpusDir, e.Name()+".golden.json") + if *update { + if err := os.WriteFile(goldenPath, got, 0o644); err != nil { + t.Fatal(err) + } + return + } + want, err := os.ReadFile(goldenPath) + if err != nil { + t.Fatalf("%v (run with -update)", err) + } + if string(want) != string(got) { + t.Errorf("golden mismatch for %s (run with -update if intended)\n--- want\n%s\n--- got\n%s", repo, want, got) + } + }) + } +} + +// Specific claims the corpus must keep proving, independent of goldens. +func TestCorpusClaims(t *testing.T) { + load := func(name string) *workflowmodel.Model { + return Repository(workflowmodel.Scope{Forge: "github.com", Repository: strings.Replace(name, "__", "/", 1)}, readRepo(t, filepath.Join(corpusDir, name))) + } + t.Run("envoy underscore workflows keep distinct ids", func(t *testing.T) { + m := load("envoyproxy__envoy") + ids := map[string]bool{} + for _, p := range m.Spec.Pipelines { + if ids[p.ID] { + t.Fatalf("duplicate id %s", p.ID) + } + ids[p.ID] = true + } + if !ids["gha-github-workflows-request"] || !ids["gha-github-workflows-request-2"] { + t.Fatalf("ids = %v", keys(ids)) + } + }) + t.Run("konflux PaC runs classify by event", func(t *testing.T) { + m := load("konflux-ci__konflux-ci") + // pull-request and push runs both reference build-pipeline: one + // pipeline, bound to both CI and Main CI. + if n := len(m.Spec.Pipelines); n != 1 { + t.Fatalf("pipelines = %d, want 1 shared", n) + } + for _, id := range []string{workflowmodel.StateCI, workflowmodel.StateMainCI} { + s := m.FindState(id) + if s == nil || len(s.Pipelines) != 1 { + t.Fatalf("state %s = %+v", id, s) + } + } + }) + t.Run("serverless-operator runs share pipelines with deduped triggers", func(t *testing.T) { + m := load("openshift-knative__serverless-operator") + var tk int + for _, p := range m.Spec.Pipelines { + if p.System != workflowmodel.SystemTekton { + continue + } + tk++ + seen := map[string]bool{} + for _, tr := range p.Triggers { + k := tr.Event + strings.Join(tr.Branches, ",") + strings.Join(tr.Types, ",") + if seen[k] { + t.Fatalf("%s: duplicate trigger %q", p.ID, k) + } + seen[k] = true + } + } + if tk != 1 { + t.Fatalf("expected one shared tekton pipeline, got %d", tk) + } + }) + t.Run("cilium environments become deploy columns and merged stops being the cap", func(t *testing.T) { + m := load("cilium__cilium") + cols := render.Columns(m) + labels := make([]string, 0, len(cols)) + for _, c := range cols { + labels = append(labels, c.Label) + } + joined := strings.Join(labels, "|") + if !strings.Contains(joined, "Release|") || !strings.Contains(joined, "Deploy: ") || !strings.HasSuffix(joined, "Done") { + t.Fatalf("columns = %s", joined) + } + if m.FindState(workflowmodel.StateMerged).Type == workflowmodel.StateFinal { + t.Fatal("merged must not be final") + } + }) + t.Run("hive default is untouched by an empty repository", func(t *testing.T) { + m := Repository(workflowmodel.Scope{Forge: "github.com", Repository: "x/y"}, nil) + cols := render.Columns(m) + if len(cols) != len(render.HiveDefaultColumns()) || len(render.Diff(m)) != 0 { + t.Fatalf("columns = %+v", cols) + } + }) +} + +func keys(m map[string]bool) []string { + out := make([]string, 0, len(m)) + for k := range m { + out = append(out, k) + } + sort.Strings(out) + return out +} diff --git a/src/pkg/workflowmodel/ingest/ingest.go b/src/pkg/workflowmodel/ingest/ingest.go new file mode 100644 index 0000000000..222e10a723 --- /dev/null +++ b/src/pkg/workflowmodel/ingest/ingest.go @@ -0,0 +1,63 @@ +// Package ingest turns a repository's CI definition files into one bound +// workflow model: GitHub Actions workflows and Tekton resources are read by +// their ingesters, then composed into the hive-default process graph. +package ingest + +import ( + "path" + "sort" + "strings" + + "github.com/hivecommons/hive/pkg/workflowmodel" + "github.com/hivecommons/hive/pkg/workflowmodel/githubactions" + "github.com/hivecommons/hive/pkg/workflowmodel/tekton" +) + +// MaxFileBytes is the largest file the ingest reads; bigger ones are +// generated manifests, not process definitions, and are reported. +const MaxFileBytes = 2 << 20 + +// CodeFileTooLarge is the finding for a skipped oversized file. +const CodeFileTooLarge = "file-too-large" + +// Repository ingests files (repo-relative slash paths to contents) for +// scope. Files that are neither GitHub Actions workflows nor Tekton +// documents are ignored. Output is deterministic for a given input. +func Repository(scope workflowmodel.Scope, files map[string][]byte) *workflowmodel.Model { + paths := make([]string, 0, len(files)) + for p := range files { + paths = append(paths, p) + } + sort.Strings(paths) + var gha []githubactions.File + var tkn []tekton.File + var findings []workflowmodel.Finding + for _, p := range paths { + data := files[p] + if len(data) > MaxFileBytes { + findings = append(findings, workflowmodel.Finding{Path: p, Code: CodeFileTooLarge, Severity: workflowmodel.SeverityWarning}) + continue + } + switch { + case githubactions.IsWorkflowPath(p): + gha = append(gha, githubactions.File{Path: p, Data: data}) + case isYAMLPath(p) && tekton.LooksTekton(data): + tkn = append(tkn, tekton.File{Path: p, Data: data}) + } + } + g := githubactions.Ingest(gha) + t := tekton.Ingest(tkn) + pipelines := append(append([]workflowmodel.Pipeline(nil), g.Pipelines...), t.Pipelines...) + sources := append(append([]workflowmodel.Source(nil), g.Sources...), t.Sources...) + cov := workflowmodel.Coverage{ + Files: g.Coverage.Files + t.Coverage.Files + len(findings), + Parsed: g.Coverage.Parsed + t.Coverage.Parsed, + Findings: append(append(append([]workflowmodel.Finding(nil), g.Coverage.Findings...), t.Coverage.Findings...), findings...), + } + return workflowmodel.Bind(scope, pipelines, sources, cov) +} + +func isYAMLPath(p string) bool { + ext := strings.ToLower(path.Ext(p)) + return ext == ".yml" || ext == ".yaml" +} diff --git a/src/pkg/workflowmodel/ingest/lifecycle.go b/src/pkg/workflowmodel/ingest/lifecycle.go new file mode 100644 index 0000000000..b231f66f18 --- /dev/null +++ b/src/pkg/workflowmodel/ingest/lifecycle.go @@ -0,0 +1,139 @@ +package ingest + +import ( + "fmt" + "sort" + "strings" + + "github.com/hivecommons/hive/pkg/workflowmodel" + "github.com/hivecommons/hive/pkg/workflowmodel/hivedefault" +) + +// LifecycleVersion is the spec.meta.version every ingested lifecycle +// model starts at; the store and marketplace assign later versions. +const LifecycleVersion = "1" + +// EventPipelineSucceeded moves work along the ingested post-merge chain. +const EventPipelineSucceeded = "pipeline-succeeded" + +// Lifecycle ingests files for scope and returns a model with the RFC's +// lifecycle half filled in so that it compiles: the hive-default lifecycle +// (states, columns, builtin derivation, roles, gates, events) plus the +// repository's pipelines bound where they run — PR-verify pipelines on the +// CI / Review states, and the post-merge chain (main CI, release, deploys, +// done) as new columns after Merged. Everything the ingest added is +// origin ingested, confidence inferred (the compiler warns per element) +// and the model's provenance is ingested-unreviewed until a human edits it. +// +// The process half (Repository) is unchanged; this composes on top of it. +func Lifecycle(scope workflowmodel.Scope, files map[string][]byte) *workflowmodel.Model { + base := Repository(scope, files) + m := hivedefault.Generate() + m.Metadata.Name = base.Metadata.Name + m.Metadata.Annotations = base.Metadata.Annotations + m.Spec.Scope = scope + m.Spec.Meta.Version = LifecycleVersion + m.Spec.Meta.Author = "" + m.Spec.Meta.Description = fmt.Sprintf("Lifecycle ingested from %s: hive-default stages with the repository's pipelines bound", scope.Repository) + m.Spec.Meta.Source = ingestSource(base) + m.Spec.Meta.Provenance = workflowmodel.ProvenanceIngestedUnreviewed + m.Spec.Pipelines = base.Spec.Pipelines + m.Spec.Sources = base.Spec.Sources + m.Spec.Coverage = base.Spec.Coverage + if len(base.Spec.Pipelines) == 0 { + return m + } + + cite := citations(base.Spec.Pipelines) + + // PR-verify pipelines run on every head in the CI / Review column. + if ci := base.FindState(workflowmodel.StateCI); ci != nil && len(ci.Pipelines) > 0 { + for i := range m.Spec.States { + s := &m.Spec.States[i] + if s.Lifecycle.Column == hivedefault.ColumnCIReview { + s.Pipelines = append([]string(nil), ci.Pipelines...) + s.Lifecycle.Citations = cite(ci.Pipelines) + } + } + } + + // The post-merge chain: every ingested state after Merged, in order. + var post []workflowmodel.State + for _, s := range base.Spec.States { + if s.Origin == workflowmodel.OriginIngested && s.ID != workflowmodel.StateCI { + post = append(post, s) + } + } + if len(post) == 0 { + return m + } + for _, g := range base.Spec.Guards { + if g.Kind == workflowmodel.GuardEnvironmentApproval { + m.Spec.Guards = append(m.Spec.Guards, g) + } + } + m.Spec.Events = append(m.Spec.Events, workflowmodel.Event{ + ID: EventPipelineSucceeded, Label: "Post-merge pipeline succeeded", + Fact: workflowmodel.Predicate{Fact: "pipeline.status", Op: workflowmodel.OpIs, Value: "green"}, + Confidence: workflowmodel.ConfidenceInferred, + }) + prev := hivedefault.StateMerged + for _, s := range post { + st := s + st.Lifecycle = workflowmodel.StateLifecycle{Column: st.ID, Kinds: []workflowmodel.ItemKind{workflowmodel.KindPR}, Confidence: workflowmodel.ConfidenceInferred, Citations: cite(st.Pipelines)} + col := workflowmodel.Column{ID: st.ID, Label: st.Label, States: []string{st.ID}, Terminal: st.Type == workflowmodel.StateFinal} + if st.ID == workflowmodel.StateDone { + col.Hint = "Every post-merge pipeline has run" + } else { + col.Hint = "Runs after merge: " + strings.Join(st.Pipelines, ", ") + } + m.Spec.States = append(m.Spec.States, st) + m.Spec.Columns = append(m.Spec.Columns, col) + m.Spec.Transitions = append(m.Spec.Transitions, workflowmodel.Transition{From: prev, To: st.ID, On: EventPipelineSucceeded, GuardRef: st.GuardRef, Note: "ingested post-merge chain"}) + prev = st.ID + } + return m +} + +// ingestSource is the meta source for the systems the pipelines came from. +func ingestSource(m *workflowmodel.Model) workflowmodel.MetaSource { + var gha, tkn bool + for _, p := range m.Spec.Pipelines { + if p.System == workflowmodel.SystemTekton { + tkn = true + } else { + gha = true + } + } + switch { + case gha && !tkn: + return workflowmodel.SourceGitHubActions + case tkn && !gha: + return workflowmodel.SourceTekton + case gha && tkn: + return workflowmodel.SourceGitHubActions + "+" + workflowmodel.SourceTekton + } + return "" +} + +// citations returns a lookup from pipeline ids to the files they came from. +func citations(pipelines []workflowmodel.Pipeline) func(ids []string) []workflowmodel.Citation { + paths := map[string]string{} + for _, p := range pipelines { + paths[p.ID] = p.Path + } + return func(ids []string) []workflowmodel.Citation { + seen := map[string]bool{} + var out []workflowmodel.Citation + for _, id := range ids { + p, ok := paths[id] + if !ok || seen[p] { + continue + } + seen[p] = true + out = append(out, workflowmodel.Citation{Path: p, Text: id}) + } + sort.Slice(out, func(i, j int) bool { return out[i].Path < out[j].Path }) + return out + } +} diff --git a/src/pkg/workflowmodel/ingest/lifecycle_test.go b/src/pkg/workflowmodel/ingest/lifecycle_test.go new file mode 100644 index 0000000000..f37f884001 --- /dev/null +++ b/src/pkg/workflowmodel/ingest/lifecycle_test.go @@ -0,0 +1,144 @@ +package ingest + +import ( + "os" + "path/filepath" + "strings" + "testing" + + "github.com/hivecommons/hive/pkg/workflowmodel" + "github.com/hivecommons/hive/pkg/workflowmodel/hivedefault" +) + +// Every corpus repository must yield a lifecycle model that compiles; +// refusals are the ingest's bug, warnings are the inferred elements. +func TestLifecycleCorpusCompiles(t *testing.T) { + entries, err := os.ReadDir(corpusDir) + if err != nil { + t.Fatal(err) + } + seen := 0 + for _, e := range entries { + if !e.IsDir() { + continue + } + seen++ + t.Run(e.Name(), func(t *testing.T) { + repo := strings.ReplaceAll(e.Name(), "__", "/") + m := Lifecycle(workflowmodel.Scope{Forge: "github.com", Repository: repo}, readRepo(t, filepath.Join(corpusDir, e.Name()))) + p, diags, err := workflowmodel.Compile(m) + if err != nil { + t.Fatalf("compile: %v\n%v", err, diags) + } + if p.Mode != workflowmodel.DerivationBuiltin { + t.Fatalf("mode %q", p.Mode) + } + if m.Spec.Meta.Provenance != workflowmodel.ProvenanceIngestedUnreviewed || m.Spec.Meta.Version == "" { + t.Fatalf("meta %+v", m.Spec.Meta) + } + if len(m.Spec.Pipelines) == 0 { + t.Fatal("corpus repo ingested no pipelines") + } + // The hive-default lifecycle is intact underneath. + for _, s := range hivedefault.Generate().Spec.States { + if _, ok := p.States[s.ID]; !ok { + t.Fatalf("hive-default state %q missing", s.ID) + } + } + // Every ingested element is inferred and warned about. + inferred := 0 + for _, d := range diags { + if d.Code == workflowmodel.CodeInferred { + inferred++ + } + } + ingested := 0 + for _, s := range m.Spec.States { + if s.Origin == workflowmodel.OriginIngested { + ingested++ + if s.Lifecycle.Confidence != workflowmodel.ConfidenceInferred || s.Lifecycle.Column != s.ID { + t.Fatalf("ingested state %+v", s.Lifecycle) + } + if s.ID != workflowmodel.StateDone && len(s.Lifecycle.Citations) == 0 { + t.Fatalf("ingested state %q has no citations", s.ID) + } + } + } + if ingested > 0 && inferred == 0 { + t.Fatalf("%d ingested states but no inferred warnings", ingested) + } + }) + } + if seen == 0 { + t.Fatal("empty corpus") + } +} + +func TestLifecyclePostMergeChain(t *testing.T) { + files := map[string][]byte{ + ".github/workflows/ci.yml": []byte("name: CI\non: [pull_request, push]\njobs:\n test:\n runs-on: ubuntu-latest\n steps: [{run: make test}]\n"), + ".github/workflows/release.yml": []byte("name: Release\non:\n push:\n tags: ['v*']\njobs:\n publish:\n runs-on: ubuntu-latest\n environment: production\n steps: [{run: make publish}]\n"), + } + m := Lifecycle(workflowmodel.Scope{Forge: "github.com", Repository: "acme/widgets"}, files) + p, diags, err := workflowmodel.Compile(m) + if err != nil { + t.Fatalf("%v %v", err, diags) + } + if m.Spec.Meta.Source != workflowmodel.SourceGitHubActions { + t.Fatalf("source %q", m.Spec.Meta.Source) + } + // Columns: hive-default's seven, then main CI, release, done. + var cols []string + for _, c := range m.Spec.Columns { + cols = append(cols, c.ID) + } + got := strings.Join(cols, " ") + if !strings.HasSuffix(got, hivedefault.ColumnMerged+" "+workflowmodel.StateMainCI+" "+workflowmodel.StateRelease+" "+workflowmodel.StateDone) { + t.Fatalf("columns %s", got) + } + if !p.Terminal[workflowmodel.StateDone] || !p.Terminal[hivedefault.StateMerged] { + t.Fatalf("terminal %v", p.Terminal) + } + // PR-verify pipelines sit on every CI / Review state with a citation. + bound := 0 + for _, s := range m.Spec.States { + if s.Lifecycle.Column == hivedefault.ColumnCIReview { + if len(s.Pipelines) != 1 || len(s.Lifecycle.Citations) != 1 || s.Lifecycle.Citations[0].Path != ".github/workflows/ci.yml" { + t.Fatalf("ci state %q: %v %v", s.ID, s.Pipelines, s.Lifecycle.Citations) + } + bound++ + } + } + if bound == 0 { + t.Fatal("no CI / Review state bound") + } + // The chain fires on the declared pipeline event, merged -> main-ci -> release -> done. + var chain []string + for _, tr := range m.Spec.Transitions { + if tr.On == EventPipelineSucceeded { + chain = append(chain, tr.From+">"+tr.To) + } + } + if want := "merged>main-ci main-ci>release release>done"; strings.Join(chain, " ") != want { + t.Fatalf("chain %v", chain) + } +} + +func TestLifecycleNoPipelinesIsHiveDefaultLifecycle(t *testing.T) { + m := Lifecycle(workflowmodel.Scope{Forge: "github.com", Repository: "acme/empty"}, map[string][]byte{"README.md": []byte("hi")}) + p, _, err := workflowmodel.Compile(m) + if err != nil { + t.Fatal(err) + } + ref, _, _ := workflowmodel.Compile(hivedefault.Generate()) + if len(p.States) != len(ref.States) || len(m.Spec.Columns) != len(hivedefault.Generate().Spec.Columns) { + t.Fatalf("states %d/%d", len(p.States), len(ref.States)) + } + if m.Spec.Meta.Source != "" || m.Spec.Meta.Provenance != workflowmodel.ProvenanceIngestedUnreviewed { + t.Fatalf("meta %+v", m.Spec.Meta) + } +} + +func scopeFor(repo string) workflowmodel.Scope { + return workflowmodel.Scope{Forge: "github.com", Repository: repo} +} diff --git a/src/pkg/workflowmodel/ingest/tree.go b/src/pkg/workflowmodel/ingest/tree.go new file mode 100644 index 0000000000..570674ef2f --- /dev/null +++ b/src/pkg/workflowmodel/ingest/tree.go @@ -0,0 +1,90 @@ +package ingest + +import ( + "io/fs" + "os" + "path/filepath" + "strings" + + "github.com/hivecommons/hive/pkg/workflowmodel/githubactions" + "github.com/hivecommons/hive/pkg/workflowmodel/tekton" +) + +// MaxTreeFiles bounds how many CI files ReadTree returns from one +// checkout; a monorepo past this is reported, not read to exhaustion. +const MaxTreeFiles = 2000 + +// tektonDirs are the directories Tekton resources conventionally live in +// (the same list the survey collector used, so a spoke ingest reads what +// the corpus read). +var tektonDirs = []string{".tekton/", "tekton/", "pipelines/", ".pipelines/", "openshift/ci/", "ci/tekton/"} + +// skipDirs are never descended into. +var skipDirs = map[string]bool{".git": true, "node_modules": true, "vendor": true, "testdata": true, "third_party": true} + +// ReadTree walks a checkout and returns the CI definition files the ingest +// reads, keyed by repo-relative slash path: every GitHub Actions workflow, +// every YAML under a Tekton directory, and any other YAML whose name hints +// at a pipeline and whose content looks like a Tekton document. Files over +// MaxFileBytes are returned too (Repository reports them). The result is +// what Repository and Lifecycle take. +func ReadTree(root string) (map[string][]byte, error) { + files := map[string][]byte{} + err := filepath.WalkDir(root, func(p string, d fs.DirEntry, err error) error { + if err != nil { + return err + } + rel, rerr := filepath.Rel(root, p) + if rerr != nil { + return rerr + } + rel = filepath.ToSlash(rel) + if d.IsDir() { + if rel != "." && skipDirs[d.Name()] { + return filepath.SkipDir + } + return nil + } + if !d.Type().IsRegular() || !isYAMLPath(rel) || len(files) >= MaxTreeFiles { + return nil + } + switch { + case githubactions.IsWorkflowPath(rel), underTektonDir(rel): + case hintsPipeline(rel): + data, err := os.ReadFile(p) + if err != nil { + return err + } + if tekton.LooksTekton(data) { + files[rel] = data + } + return nil + default: + return nil + } + data, err := os.ReadFile(p) + if err != nil { + return err + } + files[rel] = data + return nil + }) + if err != nil { + return nil, err + } + return files, nil +} + +func underTektonDir(rel string) bool { + for _, d := range tektonDirs { + if strings.HasPrefix(rel, d) { + return true + } + } + return false +} + +func hintsPipeline(rel string) bool { + base := strings.ToLower(filepath.Base(rel)) + return strings.Contains(base, "pipeline") || strings.Contains(base, "tekton") +} diff --git a/src/pkg/workflowmodel/ingest/tree_test.go b/src/pkg/workflowmodel/ingest/tree_test.go new file mode 100644 index 0000000000..6c25e1b936 --- /dev/null +++ b/src/pkg/workflowmodel/ingest/tree_test.go @@ -0,0 +1,66 @@ +package ingest + +import ( + "os" + "path/filepath" + "sort" + "strings" + "testing" +) + +func TestReadTreeSelectsCIFiles(t *testing.T) { + root := t.TempDir() + write := func(rel, body string) { + t.Helper() + p := filepath.Join(root, filepath.FromSlash(rel)) + if err := os.MkdirAll(filepath.Dir(p), 0o755); err != nil { + t.Fatal(err) + } + if err := os.WriteFile(p, []byte(body), 0o644); err != nil { + t.Fatal(err) + } + } + write(".github/workflows/ci.yml", "on: push\njobs: {}\n") + write(".github/workflows/nested/no.yml", "on: push\n") + write(".github/dependabot.yml", "version: 2\n") + write(".tekton/pr.yaml", "apiVersion: tekton.dev/v1\nkind: PipelineRun\n") + write("ci/tekton/build.yaml", "kind: Task\n") + write("deploy/release-pipeline.yaml", "apiVersion: tekton.dev/v1\nkind: Pipeline\n") + write("deploy/other-pipeline.yaml", "apiVersion: apps/v1\nkind: Deployment\n") + write("charts/values.yaml", "replicas: 1\n") + write("vendor/x/.github/workflows/v.yml", "on: push\n") + write("testdata/pipeline.yaml", "apiVersion: tekton.dev/v1\n") + write("README.md", "hi\n") + + files, err := ReadTree(root) + if err != nil { + t.Fatal(err) + } + var got []string + for p := range files { + got = append(got, p) + } + sort.Strings(got) + want := ".github/workflows/ci.yml .tekton/pr.yaml ci/tekton/build.yaml deploy/release-pipeline.yaml" + if strings.Join(got, " ") != want { + t.Fatalf("got %v", got) + } +} + +func TestReadTreeFeedsLifecycle(t *testing.T) { + root := t.TempDir() + if err := os.MkdirAll(filepath.Join(root, ".github", "workflows"), 0o755); err != nil { + t.Fatal(err) + } + if err := os.WriteFile(filepath.Join(root, ".github", "workflows", "ci.yml"), []byte("name: CI\non: pull_request\njobs:\n t:\n runs-on: ubuntu-latest\n steps: [{run: true}]\n"), 0o644); err != nil { + t.Fatal(err) + } + files, err := ReadTree(root) + if err != nil { + t.Fatal(err) + } + m := Lifecycle(scopeFor("acme/x"), files) + if len(m.Spec.Pipelines) != 1 { + t.Fatalf("pipelines %d", len(m.Spec.Pipelines)) + } +} diff --git a/src/pkg/workflowmodel/lifecycle.go b/src/pkg/workflowmodel/lifecycle.go new file mode 100644 index 0000000000..5860bdac49 --- /dev/null +++ b/src/pkg/workflowmodel/lifecycle.go @@ -0,0 +1,281 @@ +package workflowmodel + +// This file holds the lifecycle half of the schema (hivecommons/hive#17620 +// "The model"): columns, stage numbers, events, gates, roles, agents, +// checks, bookkeeping and labels. The ingest half (pipelines, coverage) is +// in model.go. A model that only carries the ingest half still validates; +// Compile needs the lifecycle half too. + +// Meta describes a model for the library and, later, the marketplace. +type Meta struct { + Version string `json:"version"` + Author string `json:"author,omitempty"` + Description string `json:"description,omitempty"` + License string `json:"license,omitempty"` + Source MetaSource `json:"source,omitempty"` + // Provenance is how much a human has vouched for the model. + Provenance Provenance `json:"provenance,omitempty"` +} + +// MetaSource is where the model came from. +type MetaSource string + +// Sources a model can be ingested from or written by. +const ( + SourceGitHubActions MetaSource = "github-actions" + SourceTekton MetaSource = "tekton" + SourceIssueTemplates MetaSource = "issue-templates" + SourceSpec MetaSource = "spec" + SourceHand MetaSource = "hand" + SourceBuiltIn MetaSource = "built-in" +) + +// Provenance tiers. Ratings and adoption never change a tier; only a +// human review does. +type Provenance string + +// Provenance values. +const ( + ProvenanceBuiltIn Provenance = "built-in" + ProvenanceCurated Provenance = "curated" + ProvenanceIngestedUnreviewed Provenance = "ingested-unreviewed" + // ProvenanceLocal is a model authored or edited on this hive and not + // (yet) published anywhere. + ProvenanceLocal Provenance = "local" +) + +// Confidence marks an element a human has not confirmed yet. +type Confidence string + +// Confidence values. The empty value means confirmed (hand-written or +// edited by a person). +const ( + ConfidenceConfirmed Confidence = "" + ConfidenceInferred Confidence = "inferred" +) + +// Citation points at the source paragraph or file an inferred element +// came from. +type Citation struct { + Path string `json:"path"` + Line int `json:"line,omitempty"` + Text string `json:"text,omitempty"` +} + +// Column is one Activity Monitor column; several states may share it. +type Column struct { + ID string `json:"id"` + Label string `json:"label"` + Hint string `json:"hint,omitempty"` + States []string `json:"states"` + Terminal bool `json:"terminal,omitempty"` +} + +// ItemKind is the kind of forge item a state or rule applies to. +type ItemKind string + +// Item kinds. The empty kind means both. +const ( + KindAny ItemKind = "" + KindIssue ItemKind = "issue" + KindPR ItemKind = "pr" +) + +// StateLifecycle is the lifecycle data on a State: where it sits on the +// board, which stage label it maps to and what a row in it may say. +type StateLifecycle struct { + // Column is the Column.ID the state is displayed in. + Column string `json:"column,omitempty"` + // Number is the stage/NNN label number; zero for states that carry + // no label (terminal observation states). + Number int `json:"number,omitempty"` + // Kinds says which item kinds may be in this state. + Kinds []ItemKind `json:"kinds,omitempty"` + // Waiting are the waiting reasons the monitor may show in this state. + Waiting []string `json:"waiting,omitempty"` + // Replies are the human-gate verbs offered in this state + // (decide:approve, release, …). + Replies []string `json:"replies,omitempty"` + Confidence Confidence `json:"confidence,omitempty"` + Citations []Citation `json:"citations,omitempty"` +} + +// DerivationMode says how an item's state is computed from facts. +type DerivationMode string + +// Derivation modes. Builtin delegates to the hive's Go derivers (the +// hive-default model); Rules evaluates the model's own ordered rules. +const ( + DerivationBuiltin DerivationMode = "builtin" + DerivationRules DerivationMode = "rules" +) + +// Derivation is how the engine maps facts to a state. +type Derivation struct { + Mode DerivationMode `json:"mode"` + Rules []Rule `json:"rules,omitempty"` +} + +// Rule puts an item in State when every predicate holds. Rules are +// evaluated in order, first match wins; an item no rule matches is "not +// covered" and the engine abstains. +type Rule struct { + ID string `json:"id,omitempty"` + Kind ItemKind `json:"kind,omitempty"` + State string `json:"state"` + When []Predicate `json:"when"` + // Reason is the text stored on the record and shown on the monitor. + Reason string `json:"reason,omitempty"` +} + +// Predicate is one test over a fact the engine already has. +type Predicate struct { + Fact string `json:"fact"` + Op Op `json:"op"` + Value string `json:"value,omitempty"` +} + +// Op is a predicate operator. +type Op string + +// Operators. Is/Not compare a scalar fact; Has/Lacks test membership in a +// set fact (labels, checks); True/False test a boolean fact. +const ( + OpIs Op = "is" + OpNot Op = "not" + OpHas Op = "has" + OpLacks Op = "lacks" + OpTrue Op = "true" + OpFalse Op = "false" +) + +// FactType is the shape of a fact in the vocabulary. +type FactType string + +// Fact types. +const ( + FactBool FactType = "bool" + FactScalar FactType = "scalar" + FactSet FactType = "set" +) + +// Facts is the vocabulary a Rule may test. The engine implements every +// entry (pkg/stageengine); the compiler refuses a fact outside it. +var Facts = map[string]FactType{ + "open": FactBool, + "merged": FactBool, + "draft": FactBool, + "labels": FactSet, + "origin": FactScalar, + "author.kind": FactScalar, + "author.login": FactScalar, + "hold": FactBool, + "park": FactBool, + "issue.bucket": FactScalar, + "issue.claimed": FactBool, + "issue.covered": FactBool, + "issue.stalled": FactBool, + "issue.open_children": FactBool, + "pr.checks": FactScalar, + "pr.review": FactScalar, + "pr.changes_requested": FactBool, + "pr.conflicting": FactBool, + "pr.dco": FactScalar, + "pr.head_fresh": FactBool, + "pr.base": FactScalar, + // pipeline.status is the latest post-merge run of the pipelines bound + // to the item's state: green | red | running | unknown. The engine does + // not observe post-merge runs yet and always reports unknown, so a + // transition on it never fires; it exists so ingested post-merge chains + // compile honestly instead of inventing a fact. + "pipeline.status": FactScalar, +} + +// Event is something that moves work: a fact the engine observes, named +// so transitions and bookkeeping can refer to it. +type Event struct { + ID string `json:"id"` + Label string `json:"label,omitempty"` + // Fact is the predicate that fires the event. Two events may not bind + // the same predicate. + Fact Predicate `json:"fact"` + // Actor is the Role that may cause the event; empty for observations. + Actor string `json:"actor,omitempty"` + Confidence Confidence `json:"confidence,omitempty"` + Citations []Citation `json:"citations,omitempty"` +} + +// Gate is a condition on a transition released by a role. +type Gate struct { + ID string `json:"id"` + Conditions []Predicate `json:"conditions,omitempty"` + // ReleasedBy is the Role that may release the gate. Required. + ReleasedBy string `json:"releasedBy"` + // Reply is the text posted when the gate is released. + Reply string `json:"reply,omitempty"` + Confidence Confidence `json:"confidence,omitempty"` + Citations []Citation `json:"citations,omitempty"` +} + +// RoleKind is how a role's members are determined. +type RoleKind string + +// Role kinds. Permission is a repository permission (admin, maintain, +// write); FrozenSet is a set named at an earlier event (the reviewers at +// request time); AuthorKind is authorkind (human, clanker, hive); Bot is a +// named bot login. +const ( + RolePermission RoleKind = "permission" + RoleFrozenSet RoleKind = "frozen-set" + RoleAuthorKind RoleKind = "author-kind" + RoleBot RoleKind = "bot" +) + +// RoleDef is who may fire which events and release which gates. +type RoleDef struct { + ID string `json:"id"` + Label string `json:"label,omitempty"` + Kind RoleKind `json:"kind"` + Params map[string]string `json:"params,omitempty"` + // Human is true for roles a person fills; the compiler needs at least + // one human-released gate after any agent action. + Human bool `json:"human,omitempty"` +} + +// AgentGrant says which agent profile may act in which states. +type AgentGrant struct { + Profile string `json:"profile"` + States []string `json:"states"` + Backend string `json:"backend,omitempty"` + Tools []string `json:"tools,omitempty"` + Produces []string `json:"produces,omitempty"` +} + +// Check is a CI check a gate may require. +type Check struct { + ID string `json:"id"` + Name string `json:"name"` + Required bool `json:"required,omitempty"` + Source string `json:"source,omitempty"` +} + +// Effect is a deterministic bookkeeping action on an event. +type Effect struct { + On string `json:"on"` + Action string `json:"action"` + Params map[string]string `json:"params,omitempty"` +} + +// LabelSet is the label vocabulary reconciled against the repository. +type LabelSet struct { + Prefix string `json:"prefix,omitempty"` + Entries []LabelEntry `json:"entries,omitempty"` +} + +// LabelEntry is one label; State is the state it marks, if any. +type LabelEntry struct { + Name string `json:"name"` + Color string `json:"color,omitempty"` + Description string `json:"description,omitempty"` + State string `json:"state,omitempty"` +} diff --git a/src/pkg/workflowmodel/model.go b/src/pkg/workflowmodel/model.go new file mode 100644 index 0000000000..7ec6c90386 --- /dev/null +++ b/src/pkg/workflowmodel/model.go @@ -0,0 +1,497 @@ +package workflowmodel + +import ( + "crypto/sha256" + "encoding/hex" + "encoding/json" + "errors" + "fmt" + "sort" + "strings" +) + +// APIVersion is the schema version every Model carries. +const APIVersion = "workflow.hivecommons.dev/v1alpha1" + +// KindWorkflowModel is the Kind of a Model document. +const KindWorkflowModel = "WorkflowModel" + +// Owner names who moves work out of a state. +type Owner string + +// Owners. ExternalObservation is a state Hive only watches (merged, +// released); ExternalService is CI or another system that acts on its own. +const ( + OwnerHuman Owner = "human" + OwnerRegisteredAgent Owner = "registered-agent" + OwnerExternalService Owner = "external-service" + OwnerExternalObservation Owner = "external-observation" +) + +// StateType distinguishes plain states from parallel and final ones. +type StateType string + +// State types. The empty type is an ordinary sequential state. +const ( + StateSequential StateType = "" + StateParallel StateType = "parallel" + StateFinal StateType = "final" +) + +// Origin records where a state came from, so re-import can keep human +// edits and replace ingested parts (design §3 "Re-import without erasing +// people's work"). +type Origin string + +// Origins. +const ( + OriginHiveDefault Origin = "hive-default" + OriginIngested Origin = "ingested" + OriginHuman Origin = "human" +) + +// System names a CI definition syntax an ingester understands. +type System string + +// Systems. +const ( + SystemGitHubActions System = "github-actions" + SystemTekton System = "tekton" +) + +// Role is what a pipeline is for, derived from its triggers. +type Role string + +// Roles. PRVerify gates pull requests; PushVerify runs on the default +// branch; Release and Deploy are post-merge; Scheduled and Manual never +// gate anything by themselves. +const ( + RolePRVerify Role = "pr-verify" + RolePushVerify Role = "push-verify" + RoleRelease Role = "release" + RoleDeploy Role = "deploy" + RoleScheduled Role = "scheduled" + RoleManual Role = "manual" + RoleOther Role = "other" +) + +// Model is one declared workflow model document. +type Model struct { + APIVersion string `json:"apiVersion"` + Kind string `json:"kind"` + Metadata Metadata `json:"metadata"` + Spec Spec `json:"spec"` +} + +// Metadata names the model. +type Metadata struct { + Name string `json:"name"` + Labels map[string]string `json:"labels,omitempty"` + Annotations map[string]string `json:"annotations,omitempty"` +} + +// Spec is the model body: the process graph (states, transitions, guards, +// lanes) and the execution graphs (pipelines) bound into it. +type Spec struct { + Meta Meta `json:"meta,omitzero"` + Scope Scope `json:"scope"` + Sources []Source `json:"sources,omitempty"` + Initial string `json:"initial"` + Columns []Column `json:"columns,omitempty"` + States []State `json:"states"` + Transitions []Transition `json:"transitions,omitempty"` + Guards []Guard `json:"guards,omitempty"` + Lanes []Lane `json:"lanes,omitempty"` + Derivation Derivation `json:"derivation,omitzero"` + Events []Event `json:"events,omitempty"` + Gates []Gate `json:"gates,omitempty"` + Roles []RoleDef `json:"roles,omitempty"` + Agents []AgentGrant `json:"agents,omitempty"` + Checks []Check `json:"checks,omitempty"` + Bookkeeping []Effect `json:"bookkeeping,omitempty"` + Labels LabelSet `json:"labels,omitzero"` + Pipelines []Pipeline `json:"pipelines,omitempty"` + Coverage Coverage `json:"coverage"` +} + +// Scope is the repository identity the model applies to. Forge is the +// forge instance (github.com, a GHES host, a GitLab host), never implied +// from the owner/name alone. +type Scope struct { + Forge string `json:"forge"` + Repository string `json:"repository"` + WorkTypes []string `json:"workTypes,omitempty"` +} + +// Source is one file the ingest read, with its content hash. +type Source struct { + System System `json:"system"` + Path string `json:"path"` + SHA256 string `json:"sha256"` +} + +// State is a node of the process graph. +type State struct { + ID string `json:"id"` + Label string `json:"label"` + Owner Owner `json:"owner"` + Type StateType `json:"type,omitempty"` + Regions []State `json:"regions,omitempty"` + GuardRef string `json:"guardRef,omitempty"` + // Pipelines bound into this state (by Pipeline.ID). + Pipelines []string `json:"pipelines,omitempty"` + Origin Origin `json:"origin,omitempty"` + // Lifecycle is the board/label half of the state (lifecycle.go). + Lifecycle StateLifecycle `json:"lifecycle,omitzero"` +} + +// Transition moves work between states on an event, if its guard holds. +type Transition struct { + From string `json:"from"` + To string `json:"to"` + On string `json:"on"` + GuardRef string `json:"guardRef,omitempty"` + // GateRef is the lifecycle Gate (lifecycle.go) the move needs. + GateRef string `json:"gateRef,omitempty"` + // Note is the human-readable reason for the edge. + Note string `json:"note,omitempty"` +} + +// GuardKind is the family a guard belongs to. +type GuardKind string + +// Guard kinds. +const ( + GuardRequiredChecks GuardKind = "required-checks" + GuardReviewPolicy GuardKind = "review-policy" + GuardEnvironmentApproval GuardKind = "environment-approval" + GuardManual GuardKind = "manual" + GuardExternal GuardKind = "external" +) + +// Guard is a named condition a transition or region completion needs. +type Guard struct { + ID string `json:"id"` + Kind GuardKind `json:"kind"` + Params map[string]string `json:"params,omitempty"` + Source string `json:"source,omitempty"` +} + +// Lane is a bounded kind of work an actor may do while a state holds. +type Lane struct { + ID string `json:"id"` + Label string `json:"label"` + StateRef string `json:"stateRef"` + Actor Owner `json:"actor"` + Effects []string `json:"effects,omitempty"` +} + +// Pipeline is one execution graph: a GitHub Actions workflow file or a +// Tekton Pipeline. +type Pipeline struct { + ID string `json:"id"` + Name string `json:"name"` + System System `json:"system"` + Path string `json:"path"` + Role Role `json:"role"` + Triggers []Trigger `json:"triggers,omitempty"` + Nodes []Node `json:"nodes"` + Concurrency *Concurrency `json:"concurrency,omitempty"` + // Levels is the number of topological levels in Nodes (0 when empty). + Levels int `json:"levels"` +} + +// Trigger is one event that starts a pipeline. +type Trigger struct { + Event string `json:"event"` + Types []string `json:"types,omitempty"` + Branches []string `json:"branches,omitempty"` + Tags []string `json:"tags,omitempty"` + Paths []string `json:"paths,omitempty"` + Schedule []string `json:"schedule,omitempty"` +} + +// NodeKind is the kind of execution node. +type NodeKind string + +// Node kinds. +const ( + NodeJob NodeKind = "job" + NodeReusableWorkflow NodeKind = "reusable-workflow" + NodeTask NodeKind = "task" + NodeFinally NodeKind = "finally" +) + +// Node is a job or task in a pipeline. +type Node struct { + ID string `json:"id"` + Name string `json:"name,omitempty"` + Kind NodeKind `json:"kind"` + Needs []string `json:"needs,omitempty"` + // Ref is the reusable workflow or task reference, when Kind says so. + Ref string `json:"ref,omitempty"` + Matrix *Matrix `json:"matrix,omitempty"` + Environment string `json:"environment,omitempty"` + If string `json:"if,omitempty"` + When []string `json:"when,omitempty"` + RunsOn []string `json:"runsOn,omitempty"` + // Level is the node's topological level (0 = no needs). + Level int `json:"level"` +} + +// Matrix summarises a job's strategy.matrix without expanding it. +type Matrix struct { + Keys []string `json:"keys,omitempty"` + Include int `json:"include,omitempty"` + Exclude int `json:"exclude,omitempty"` + // Dynamic is set when the matrix is an expression (fromJSON etc). + Dynamic bool `json:"dynamic,omitempty"` +} + +// Concurrency is a pipeline's concurrency group. +type Concurrency struct { + Group string `json:"group,omitempty"` + CancelInProgress bool `json:"cancelInProgress,omitempty"` +} + +// Coverage says how completely the sources were understood. +type Coverage struct { + Files int `json:"files"` + Parsed int `json:"parsed"` + Findings []Finding `json:"findings,omitempty"` +} + +// Severity of a finding. +type Severity string + +// Severities. Unsupported means the construct is recognised and skipped; +// Unknown means the ingest could not tell what it was; Warning is advice. +const ( + SeverityUnsupported Severity = "unsupported" + SeverityUnknown Severity = "unknown" + SeverityWarning Severity = "warning" +) + +// Finding is one thing the ingest could not fully model. +type Finding struct { + Path string `json:"path"` + Node string `json:"node,omitempty"` + Code string `json:"code"` + Detail string `json:"detail,omitempty"` + Severity Severity `json:"severity"` +} + +// New returns an empty model for scope. +func New(name string, scope Scope) *Model { + return &Model{APIVersion: APIVersion, Kind: KindWorkflowModel, Metadata: Metadata{Name: name}, Spec: Spec{Scope: scope}} +} + +// Hash is the content hash of the spec: the same spec always hashes the +// same (JSON field order is fixed by the struct), independent of metadata. +func (m *Model) Hash() string { + b, _ := json.Marshal(m.Spec) + sum := sha256.Sum256(b) + return hex.EncodeToString(sum[:]) +} + +// FindState returns the state with id at any depth. +func (m *Model) FindState(id string) *State { + return findState(m.Spec.States, id) +} + +func findState(states []State, id string) *State { + for i := range states { + if states[i].ID == id { + return &states[i] + } + if s := findState(states[i].Regions, id); s != nil { + return s + } + } + return nil +} + +// Pipeline returns the pipeline with id, or nil. +func (m *Model) Pipeline(id string) *Pipeline { + for i := range m.Spec.Pipelines { + if m.Spec.Pipelines[i].ID == id { + return &m.Spec.Pipelines[i] + } + } + return nil +} + +// ErrInvalid wraps every validation failure. +var ErrInvalid = errors.New("workflow model invalid") + +// Validate checks the model is self-consistent: version and kind, unique +// state ids, initial and every reference resolving, and acyclic pipelines. +func (m *Model) Validate() error { + var errs []string + fail := func(format string, a ...any) { errs = append(errs, fmt.Sprintf(format, a...)) } + if m.APIVersion != APIVersion { + fail("apiVersion %q, want %q", m.APIVersion, APIVersion) + } + if m.Kind != KindWorkflowModel { + fail("kind %q, want %q", m.Kind, KindWorkflowModel) + } + if m.Metadata.Name == "" { + fail("metadata.name is empty") + } + if m.Spec.Scope.Repository == "" { + fail("spec.scope.repository is empty") + } + ids := map[string]bool{} + var walk func(states []State, parallelParent bool) + walk = func(states []State, parallelParent bool) { + for _, s := range states { + switch { + case s.ID == "": + fail("state with empty id (label %q)", s.Label) + case ids[s.ID]: + fail("duplicate state id %q", s.ID) + } + ids[s.ID] = true + if s.Owner == "" { + fail("state %q has no owner", s.ID) + } + if s.Type == StateParallel && len(s.Regions) == 0 { + fail("parallel state %q has no regions", s.ID) + } + if s.Type != StateParallel && len(s.Regions) > 0 { + fail("state %q has regions but is not parallel", s.ID) + } + walk(s.Regions, s.Type == StateParallel) + } + } + walk(m.Spec.States, false) + if len(m.Spec.States) == 0 { + fail("spec.states is empty") + } + if m.Spec.Initial == "" { + fail("spec.initial is empty") + } else if !ids[m.Spec.Initial] { + fail("spec.initial %q is not a state", m.Spec.Initial) + } + guards := map[string]bool{} + for _, g := range m.Spec.Guards { + if guards[g.ID] { + fail("duplicate guard id %q", g.ID) + } + guards[g.ID] = true + } + pipelines := map[string]bool{} + for _, p := range m.Spec.Pipelines { + if pipelines[p.ID] { + fail("duplicate pipeline id %q", p.ID) + } + pipelines[p.ID] = true + if _, err := Levels(p.Nodes); err != nil { + fail("pipeline %q: %v", p.ID, err) + } + } + var checkRefs func(states []State) + checkRefs = func(states []State) { + for _, s := range states { + if s.GuardRef != "" && !guards[s.GuardRef] { + fail("state %q guardRef %q is not a guard", s.ID, s.GuardRef) + } + for _, p := range s.Pipelines { + if !pipelines[p] { + fail("state %q binds unknown pipeline %q", s.ID, p) + } + } + checkRefs(s.Regions) + } + } + checkRefs(m.Spec.States) + for _, t := range m.Spec.Transitions { + if !ids[t.From] { + fail("transition from unknown state %q", t.From) + } + if !ids[t.To] { + fail("transition to unknown state %q", t.To) + } + if t.GuardRef != "" && !guards[t.GuardRef] { + fail("transition %s->%s guardRef %q is not a guard", t.From, t.To, t.GuardRef) + } + } + for _, l := range m.Spec.Lanes { + if !ids[l.StateRef] { + fail("lane %q stateRef %q is not a state", l.ID, l.StateRef) + } + } + if len(errs) == 0 { + return nil + } + return fmt.Errorf("%w: %s", ErrInvalid, strings.Join(errs, "; ")) +} + +// Levels assigns each node its topological level (longest path from a +// root) in place and returns the number of levels. It fails on a cycle or +// a need that names no node. +func Levels(nodes []Node) (int, error) { + idx := make(map[string]int, len(nodes)) + for i, n := range nodes { + idx[n.ID] = i + } + const unvisited, visiting, done = 0, 1, 2 + state := make([]int, len(nodes)) + var visit func(i int) error + visit = func(i int) error { + switch state[i] { + case done: + return nil + case visiting: + return fmt.Errorf("cycle through node %q", nodes[i].ID) + } + state[i] = visiting + level := 0 + for _, need := range nodes[i].Needs { + j, ok := idx[need] + if !ok { + return fmt.Errorf("node %q needs unknown node %q", nodes[i].ID, need) + } + if err := visit(j); err != nil { + return err + } + if nodes[j].Level+1 > level { + level = nodes[j].Level + 1 + } + } + nodes[i].Level = level + state[i] = done + return nil + } + max := -1 + for i := range nodes { + if err := visit(i); err != nil { + return 0, err + } + if nodes[i].Level > max { + max = nodes[i].Level + } + } + return max + 1, nil +} + +// ByLevel groups node ids by level, each level sorted by id. +func ByLevel(nodes []Node) [][]string { + var out [][]string + for _, n := range nodes { + for len(out) <= n.Level { + out = append(out, nil) + } + out[n.Level] = append(out[n.Level], n.ID) + } + for _, l := range out { + sort.Strings(l) + } + return out +} + +// SHA256Hex is the hex digest ingesters record in Source. +func SHA256Hex(b []byte) string { + sum := sha256.Sum256(b) + return hex.EncodeToString(sum[:]) +} diff --git a/src/pkg/workflowmodel/model_test.go b/src/pkg/workflowmodel/model_test.go new file mode 100644 index 0000000000..be3ed32d5c --- /dev/null +++ b/src/pkg/workflowmodel/model_test.go @@ -0,0 +1,213 @@ +package workflowmodel + +import ( + "encoding/json" + "errors" + "fmt" + "strings" + "testing" +) + +func testScope() Scope { return Scope{Forge: "github.com", Repository: "example/repo"} } + +func TestHiveDefaultValidatesAndIsStable(t *testing.T) { + m := HiveDefault(testScope()) + if err := m.Validate(); err != nil { + t.Fatal(err) + } + if m.Hash() != HiveDefault(testScope()).Hash() { + t.Fatal("hash differs between two identical defaults") + } + var ids []string + for _, s := range m.Spec.States { + ids = append(ids, s.ID) + } + want := []string{StateOpen, StateActionable, StateClaimed, StatePRInProgress, StateVerify, StateMergeQueue, StateMerged} + if strings.Join(ids, ",") != strings.Join(want, ",") { + t.Fatalf("default states = %v, want %v", ids, want) + } + if got := m.FindState(StateCI); got == nil || got.GuardRef != GuardIDRequiredChecks { + t.Fatalf("ci region = %+v", got) + } +} + +func TestBindNoPipelinesIsHiveDefault(t *testing.T) { + d := HiveDefault(testScope()) + b := Bind(testScope(), nil, nil, Coverage{}) + d.Metadata.Name = b.Metadata.Name + if d.Hash() != b.Hash() { + t.Fatal("Bind with no pipelines must not change the default spec") + } +} + +func pipeline(id string, role Role, nodes ...Node) Pipeline { + p := Pipeline{ID: id, Name: id, System: SystemGitHubActions, Path: id + ".yml", Role: role, Nodes: nodes} + p.Levels, _ = Levels(p.Nodes) + return p +} + +func TestBindFillsCIAndAddsPostMergeChain(t *testing.T) { + ps := []Pipeline{ + pipeline("ci", RolePRVerify, Node{ID: "lint", Kind: NodeJob}, Node{ID: "test", Kind: NodeJob, Needs: []string{"lint"}}), + pipeline("nightly", RoleScheduled, Node{ID: "scan", Kind: NodeJob}), + pipeline("main", RolePushVerify, Node{ID: "build", Kind: NodeJob}), + pipeline("release", RoleRelease, Node{ID: "publish", Kind: NodeJob}), + pipeline("deploy", RoleDeploy, Node{ID: "prod", Kind: NodeJob, Environment: "production"}, Node{ID: "stg", Kind: NodeJob, Environment: "staging"}), + } + m := Bind(testScope(), ps, nil, Coverage{Files: 5, Parsed: 5}) + if err := m.Validate(); err != nil { + t.Fatal(err) + } + if ci := m.FindState(StateCI); len(ci.Pipelines) != 1 || ci.Pipelines[0] != "ci" || ci.Origin != OriginIngested { + t.Fatalf("ci region = %+v", ci) + } + var ids []string + for _, s := range m.Spec.States { + ids = append(ids, s.ID) + } + want := "open,actionable,claimed,pr-in-progress,verify,merge-queue,merged,main-ci,release,deploy-production,deploy-staging,done" + if got := strings.Join(ids, ","); got != want { + t.Fatalf("states = %s\nwant %s", got, want) + } + if m.FindState(StateMerged).Type == StateFinal { + t.Fatal("merged must stop being final once a post-merge chain exists") + } + if m.FindState(StateDone).Type != StateFinal { + t.Fatal("done must be final") + } + if g := m.FindState(DeployStateID("production")).GuardRef; g != "environment-approved-production" { + t.Fatalf("deploy guard = %q", g) + } + for _, s := range m.Spec.States { + for _, id := range s.Pipelines { + if id == "nightly" { + t.Fatal("scheduled pipeline must bind to no state") + } + } + } +} + +func TestLevelsAndByLevel(t *testing.T) { + nodes := []Node{ + {ID: "e2e", Needs: []string{"unit", "integration"}}, + {ID: "unit", Needs: []string{"build"}}, + {ID: "integration", Needs: []string{"build"}}, + {ID: "build", Needs: []string{"lint"}}, + {ID: "lint"}, + {ID: "docs"}, + } + n, err := Levels(nodes) + if err != nil || n != 4 { + t.Fatalf("Levels = %d, %v", n, err) + } + got := ByLevel(nodes) + want := [][]string{{"docs", "lint"}, {"build"}, {"integration", "unit"}, {"e2e"}} + for i := range want { + if strings.Join(got[i], ",") != strings.Join(want[i], ",") { + t.Fatalf("level %d = %v, want %v", i, got[i], want[i]) + } + } + if _, err := Levels([]Node{{ID: "a", Needs: []string{"b"}}, {ID: "b", Needs: []string{"a"}}}); err == nil || !strings.Contains(err.Error(), "cycle") { + t.Fatalf("cycle err = %v", err) + } + if _, err := Levels([]Node{{ID: "a", Needs: []string{"nope"}}}); err == nil || !strings.Contains(err.Error(), "unknown") { + t.Fatalf("unknown err = %v", err) + } +} + +func TestValidateReportsEveryProblem(t *testing.T) { + m := HiveDefault(testScope()) + m.Spec.Initial = "nowhere" + m.Spec.States = append(m.Spec.States, State{ID: StateOpen, Label: "dup", Owner: OwnerHuman}) + m.Spec.Transitions = append(m.Spec.Transitions, Transition{From: "x", To: "y", GuardRef: "g"}) + m.Spec.Lanes = append(m.Spec.Lanes, Lane{ID: "l", StateRef: "zzz", Actor: OwnerHuman}) + m.Spec.Pipelines = []Pipeline{{ID: "p", Nodes: []Node{{ID: "a", Needs: []string{"a"}}}}} + err := m.Validate() + if !errors.Is(err, ErrInvalid) { + t.Fatalf("err = %v", err) + } + for _, want := range []string{`initial "nowhere"`, `duplicate state id "open"`, `from unknown state "x"`, `guardRef "g"`, `stateRef "zzz"`, `pipeline "p"`} { + if !strings.Contains(err.Error(), want) { + t.Errorf("missing %q in %v", want, err) + } + } +} + +func TestJSONRoundTrip(t *testing.T) { + m := Bind(testScope(), []Pipeline{pipeline("ci", RolePRVerify, Node{ID: "a", Kind: NodeJob, Matrix: &Matrix{Keys: []string{"go", "os"}}})}, []Source{{System: SystemGitHubActions, Path: "ci.yml", SHA256: "x"}}, Coverage{Files: 1, Parsed: 1}) + b, err := json.Marshal(m) + if err != nil { + t.Fatal(err) + } + var back Model + if err := json.Unmarshal(b, &back); err != nil { + t.Fatal(err) + } + if back.Hash() != m.Hash() || back.APIVersion != APIVersion || back.Kind != KindWorkflowModel { + t.Fatal("round trip changed the model") + } +} + +func TestSlugAndPipelineID(t *testing.T) { + cases := map[string]string{"Hello World!": "hello-world", "---": "x", "kubernetes/kubernetes": "kubernetes-kubernetes"} + for in, want := range cases { + if got := Slug(in); got != want { + t.Errorf("Slug(%q) = %q, want %q", in, got, want) + } + } + if got := PipelineID(SystemGitHubActions, ".github/workflows/CI Build.yaml"); got != "gha-github-workflows-ci-build" { + t.Errorf("PipelineID = %q", got) + } + if got := PipelineID(SystemTekton, ".tekton/pull-request.yaml"); got != "tkn-tekton-pull-request" { + t.Errorf("PipelineID = %q", got) + } +} + +func TestDedupeIDsAndDynamicEnvironment(t *testing.T) { + ps := []Pipeline{ + pipeline("gha-stale", RoleScheduled, Node{ID: "a", Kind: NodeJob}), + pipeline("gha-stale", RoleScheduled, Node{ID: "a", Kind: NodeJob}), + pipeline("gha-stale", RoleScheduled, Node{ID: "a", Kind: NodeJob}), + pipeline("deploy", RoleDeploy, Node{ID: "d", Kind: NodeJob, Environment: "${{ matrix.env }}"}), + } + m := Bind(testScope(), ps, nil, Coverage{}) + if err := m.Validate(); err != nil { + t.Fatal(err) + } + if got := m.Spec.Pipelines[1].ID + "," + m.Spec.Pipelines[2].ID; got != "gha-stale-2,gha-stale-3" { + t.Fatalf("ids = %s", got) + } + if s := m.FindState("deploy"); s == nil || len(s.Pipelines) != 1 { + t.Fatalf("expected one plain deploy state: %+v", s) + } + for _, s := range m.Spec.States { + if strings.Contains(s.Label, "${{") { + t.Fatalf("expression leaked into a state label: %q", s.Label) + } + } + if len(m.Spec.Coverage.Findings) != 1 || m.Spec.Coverage.Findings[0].Code != CodeEnvironmentDynamic { + t.Fatalf("findings = %+v", m.Spec.Coverage.Findings) + } +} + +func TestBindingRolesFollowTriggers(t *testing.T) { + p := pipeline("ci", RolePRVerify, Node{ID: "a", Kind: NodeJob}) + p.Triggers = []Trigger{{Event: "pull_request"}, {Event: "push", Branches: []string{"main"}, Tags: []string{"v*"}}} + if got := fmt.Sprint(BindingRoles(p)); got != "[pr-verify release push-verify]" { + t.Fatalf("roles = %s", got) + } + p.Triggers = []Trigger{{Event: "push", Tags: []string{"v*"}}} + p.Role = RoleRelease + if got := fmt.Sprint(BindingRoles(p)); got != "[release]" { + t.Fatalf("tags-only push = %s", got) + } + p.Triggers = []Trigger{{Event: "push", Tags: []string{"!v*"}}} + p.Role = RolePushVerify + if got := fmt.Sprint(BindingRoles(p)); got != "[push-verify]" { + t.Fatalf("tags-ignore push = %s", got) + } + m := Bind(testScope(), []Pipeline{{ID: "both", Role: RolePRVerify, Triggers: []Trigger{{Event: "pull_request"}, {Event: "push", Branches: []string{"main"}}}, Nodes: []Node{{ID: "a", Kind: NodeJob}}}}, nil, Coverage{}) + if ci, main := m.FindState(StateCI), m.FindState(StateMainCI); len(ci.Pipelines) != 1 || main == nil || len(main.Pipelines) != 1 { + t.Fatalf("ci=%+v main=%+v", ci, main) + } +} diff --git a/src/pkg/workflowmodel/render/columns.go b/src/pkg/workflowmodel/render/columns.go new file mode 100644 index 0000000000..01bbd73cdb --- /dev/null +++ b/src/pkg/workflowmodel/render/columns.go @@ -0,0 +1,178 @@ +// Package render derives views from a workflow model: a Mermaid diagram of +// the process and execution graphs, and the Activity Monitor column set +// the dashboard would draw for a repository bound to the model. +package render + +import ( + "fmt" + "sort" + "strings" + + "github.com/hivecommons/hive/pkg/workflowmodel" +) + +// Column is one Activity Monitor strip column. +type Column struct { + // ID is the process state the column lights. + ID string `json:"id"` + Label string `json:"label"` + // Regions are the parallel regions shown inside the column, if any. + Regions []string `json:"regions,omitempty"` + // Substages are the CI sub-stages (one per topological level of the + // bound PR-verify pipelines), shown under the column. + Substages []Substage `json:"substages,omitempty"` + // Pipelines bound into the column (process state and its regions). + Pipelines []string `json:"pipelines,omitempty"` + // Terminal marks the final cap column. + Terminal bool `json:"terminal,omitempty"` + // Origin says whether the column exists on an unbound hive. + Origin workflowmodel.Origin `json:"origin,omitempty"` +} + +// Substage is one level of a CI DAG, named after its nodes. +type Substage struct { + Level int `json:"level"` + Label string `json:"label"` + Nodes []string `json:"nodes"` +} + +// MaxSubstageNames caps how many node names a substage label lists before +// summarising the rest as "+N". +const MaxSubstageNames = 3 + +// MaxSubstages caps how many CI levels become substages; deeper DAGs are +// folded into the last one so the strip stays readable. +const MaxSubstages = 6 + +// Columns derives the strip columns from the model: one per top-level +// state in declaration order, parallel states as one column listing their +// regions, and the CI region's bound pipelines expanded into substages by +// topological level. HiveDefault yields exactly today's seven columns. +func Columns(m *workflowmodel.Model) []Column { + var out []Column + for _, s := range m.Spec.States { + c := Column{ID: s.ID, Label: s.Label, Origin: s.Origin, Terminal: s.Type == workflowmodel.StateFinal} + c.Pipelines = append(c.Pipelines, s.Pipelines...) + for _, r := range s.Regions { + c.Regions = append(c.Regions, r.Label) + c.Pipelines = append(c.Pipelines, r.Pipelines...) + if r.ID == workflowmodel.StateCI { + c.Substages = substages(m, r.Pipelines) + } + } + if s.ID == workflowmodel.StateCI { + c.Substages = substages(m, s.Pipelines) + } + if len(c.Pipelines) == 0 { + c.Pipelines = nil + } + out = append(out, c) + } + return out +} + +// HiveDefaultColumns is the column set every unbound hive draws today. +func HiveDefaultColumns() []Column { + return Columns(workflowmodel.HiveDefault(workflowmodel.Scope{Forge: "github.com", Repository: "example/repo"})) +} + +// substages merges the levels of every bound pipeline: level k of the +// result holds the level-k nodes of each pipeline. +func substages(m *workflowmodel.Model, ids []string) []Substage { + var levels [][]string + for _, id := range ids { + p := m.Pipeline(id) + if p == nil { + continue + } + for lvl, nodes := range workflowmodel.ByLevel(p.Nodes) { + for len(levels) <= lvl { + levels = append(levels, nil) + } + for _, n := range nodes { + levels[lvl] = append(levels[lvl], nodeDisplayName(p, n)) + } + } + } + if len(levels) > MaxSubstages { + tail := levels[MaxSubstages-1:] + var folded []string + for _, l := range tail { + folded = append(folded, l...) + } + levels = append(levels[:MaxSubstages-1], folded) + } + out := make([]Substage, 0, len(levels)) + for i, nodes := range levels { + nodes = dedupeSorted(nodes) + out = append(out, Substage{Level: i, Label: substageLabel(nodes), Nodes: nodes}) + } + return out +} + +func nodeDisplayName(p *workflowmodel.Pipeline, id string) string { + for _, n := range p.Nodes { + if n.ID == id { + if n.Name != "" && !strings.Contains(n.Name, "${{") { + return n.Name + } + return n.ID + } + } + return id +} + +func dedupeSorted(xs []string) []string { + seen := map[string]bool{} + out := xs[:0] + for _, x := range xs { + if !seen[x] { + seen[x] = true + out = append(out, x) + } + } + sort.Strings(out) + return out +} + +func substageLabel(nodes []string) string { + if len(nodes) <= MaxSubstageNames { + return strings.Join(nodes, " · ") + } + return fmt.Sprintf("%s +%d", strings.Join(nodes[:MaxSubstageNames], " · "), len(nodes)-MaxSubstageNames) +} + +// Diff lists how the model's columns differ from the hive default, as +// short human sentences; empty when they are identical. +func Diff(m *workflowmodel.Model) []string { + base := HiveDefaultColumns() + cols := Columns(m) + var out []string + byID := map[string]Column{} + for _, c := range cols { + byID[c.ID] = c + } + for _, b := range base { + c, ok := byID[b.ID] + if !ok { + out = append(out, fmt.Sprintf("column %q removed", b.Label)) + continue + } + if len(c.Substages) > 0 && len(b.Substages) == 0 { + out = append(out, fmt.Sprintf("column %q gains %d CI substage(s) from %d pipeline(s)", b.Label, len(c.Substages), len(c.Pipelines))) + } + if b.Terminal && !c.Terminal { + out = append(out, fmt.Sprintf("column %q is no longer the cap", b.Label)) + } + } + baseIDs := map[string]bool{} + for _, b := range base { + baseIDs[b.ID] = true + } + for _, c := range cols { + if !baseIDs[c.ID] { + out = append(out, fmt.Sprintf("column %q added after merge", c.Label)) + } + } + return out +} diff --git a/src/pkg/workflowmodel/render/mermaid.go b/src/pkg/workflowmodel/render/mermaid.go new file mode 100644 index 0000000000..a046963e88 --- /dev/null +++ b/src/pkg/workflowmodel/render/mermaid.go @@ -0,0 +1,160 @@ +package render + +import ( + "fmt" + "sort" + "strings" + + "github.com/hivecommons/hive/pkg/workflowmodel" +) + +// MaxDiagramNodes caps how many execution nodes one pipeline subgraph +// draws before the rest are summarised, so a 120-job workflow stays +// legible in the editor. +const MaxDiagramNodes = 40 + +// Mermaid renders the process graph as a Mermaid stateDiagram-v2, with +// parallel states as concurrent regions, and the pipelines bound to each +// state as notes naming their levels. Deterministic for a given model. +func Mermaid(m *workflowmodel.Model) string { + var b strings.Builder + b.WriteString("stateDiagram-v2\n") + fmt.Fprintf(&b, " [*] --> %s\n", mid(m.Spec.Initial)) + for _, s := range m.Spec.States { + writeState(&b, m, s, 1) + if s.Type == workflowmodel.StateFinal { + fmt.Fprintf(&b, " %s --> [*]\n", mid(s.ID)) + } + } + for _, t := range m.Spec.Transitions { + label := t.On + if t.GuardRef != "" { + label += " [" + t.GuardRef + "]" + } + fmt.Fprintf(&b, " %s --> %s: %s\n", mid(t.From), mid(t.To), label) + } + return b.String() +} + +func writeState(b *strings.Builder, m *workflowmodel.Model, s workflowmodel.State, depth int) { + ind := strings.Repeat(" ", depth) + fmt.Fprintf(b, "%sstate \"%s\" as %s\n", ind, escape(s.Label+ownerSuffix(s.Owner)), mid(s.ID)) + if len(s.Regions) > 0 { + fmt.Fprintf(b, "%sstate %s {\n", ind, mid(s.ID)) + for i, r := range s.Regions { + if i > 0 { + fmt.Fprintf(b, "%s --\n", ind) + } + writeState(b, m, r, depth+1) + } + fmt.Fprintf(b, "%s}\n", ind) + } + if len(s.Pipelines) > 0 { + fmt.Fprintf(b, "%snote right of %s\n", ind, mid(s.ID)) + for _, id := range s.Pipelines { + if p := m.Pipeline(id); p != nil { + fmt.Fprintf(b, "%s %s: %s\n", ind, escape(p.Name), levelsSummary(p)) + } + } + fmt.Fprintf(b, "%send note\n", ind) + } +} + +func ownerSuffix(o workflowmodel.Owner) string { + switch o { + case workflowmodel.OwnerHuman: + return " (human)" + case workflowmodel.OwnerRegisteredAgent: + return " (agent)" + case workflowmodel.OwnerExternalService: + return " (ci)" + } + return "" +} + +func levelsSummary(p *workflowmodel.Pipeline) string { + levels := workflowmodel.ByLevel(p.Nodes) + parts := make([]string, 0, len(levels)) + for _, l := range levels { + parts = append(parts, substageLabel(l)) + } + return strings.Join(parts, " → ") +} + +// PipelineMermaid renders one pipeline's execution DAG as a Mermaid +// flowchart (left to right), matrix jobs marked with their keys and +// reusable workflows with their ref. +func PipelineMermaid(p *workflowmodel.Pipeline) string { + var b strings.Builder + b.WriteString("flowchart LR\n") + fmt.Fprintf(&b, " subgraph %s[\"%s (%s)\"]\n", mid(p.ID), escape(p.Name), p.Role) + nodes := p.Nodes + truncated := 0 + if len(nodes) > MaxDiagramNodes { + truncated = len(nodes) - MaxDiagramNodes + nodes = nodes[:MaxDiagramNodes] + } + shown := map[string]bool{} + for _, n := range nodes { + shown[n.ID] = true + fmt.Fprintf(&b, " %s[\"%s\"]\n", mid(p.ID+"-"+n.ID), escape(nodeLabel(n))) + } + if truncated > 0 { + fmt.Fprintf(&b, " %s[\"… %d more\"]\n", mid(p.ID+"-more"), truncated) + } + for _, n := range nodes { + needs := append([]string(nil), n.Needs...) + sort.Strings(needs) + for _, need := range needs { + if shown[need] { + fmt.Fprintf(&b, " %s --> %s\n", mid(p.ID+"-"+need), mid(p.ID+"-"+n.ID)) + } + } + } + b.WriteString(" end\n") + return b.String() +} + +func nodeLabel(n workflowmodel.Node) string { + label := n.ID + if n.Name != "" && !strings.Contains(n.Name, "${{") { + label = n.Name + } + switch { + case n.Matrix != nil && n.Matrix.Dynamic: + label += " ⊗ dynamic" + case n.Matrix != nil && len(n.Matrix.Keys) > 0: + label += " ⊗ " + strings.Join(n.Matrix.Keys, ",") + } + if n.Kind == workflowmodel.NodeReusableWorkflow { + label += " ↗ " + n.Ref + } + if n.Kind == workflowmodel.NodeFinally { + label += " (finally)" + } + if n.Environment != "" { + label += " @" + n.Environment + } + return label +} + +// mid makes a Mermaid-safe identifier. +func mid(s string) string { + var b strings.Builder + for _, r := range s { + switch { + case r >= 'a' && r <= 'z', r >= 'A' && r <= 'Z', r >= '0' && r <= '9', r == '_': + b.WriteRune(r) + default: + b.WriteByte('_') + } + } + if b.Len() == 0 || (s[0] >= '0' && s[0] <= '9') { + return "n_" + b.String() + } + return b.String() +} + +func escape(s string) string { + return strings.NewReplacer(`"`, "#quot;", "<", "#lt;", ">", "#gt;").Replace(s) +} diff --git a/src/pkg/workflowmodel/render/render_test.go b/src/pkg/workflowmodel/render/render_test.go new file mode 100644 index 0000000000..cd5570040c --- /dev/null +++ b/src/pkg/workflowmodel/render/render_test.go @@ -0,0 +1,158 @@ +package render + +import ( + "strings" + "testing" + + "github.com/hivecommons/hive/pkg/workflowmodel" +) + +func scope() workflowmodel.Scope { + return workflowmodel.Scope{Forge: "github.com", Repository: "example/repo"} +} + +func pipeline(id string, role workflowmodel.Role, nodes ...workflowmodel.Node) workflowmodel.Pipeline { + p := workflowmodel.Pipeline{ID: id, Name: id, System: workflowmodel.SystemGitHubActions, Path: id + ".yml", Role: role, Nodes: nodes} + p.Levels, _ = workflowmodel.Levels(p.Nodes) + return p +} + +// The unbound default must draw exactly the seven columns the strip draws +// today (agent-now.js STAGE_TRACKER_PATH + the merged cap). +func TestHiveDefaultColumnsMatchToday(t *testing.T) { + cols := HiveDefaultColumns() + want := []string{"Open", "Actionable", "Claimed", "PR in progress", "CI / Review", "Merge queue", "Merged"} + if len(cols) != len(want) { + t.Fatalf("got %d columns, want %d", len(cols), len(want)) + } + for i, c := range cols { + if c.Label != want[i] { + t.Errorf("column %d = %q, want %q", i, c.Label, want[i]) + } + if len(c.Substages) != 0 || len(c.Pipelines) != 0 { + t.Errorf("column %q must be bare on the default: %+v", c.Label, c) + } + } + if !cols[len(cols)-1].Terminal { + t.Error("Merged must be the terminal cap") + } + if got := strings.Join(cols[4].Regions, "|"); got != "Review|CI" { + t.Errorf("verify regions = %q", got) + } + if d := Diff(workflowmodel.HiveDefault(scope())); len(d) != 0 { + t.Errorf("Diff(default) = %v, want none", d) + } +} + +func TestColumnsExpandCIAndAddPostMerge(t *testing.T) { + ps := []workflowmodel.Pipeline{ + pipeline("ci", workflowmodel.RolePRVerify, + workflowmodel.Node{ID: "lint", Name: "Lint", Kind: workflowmodel.NodeJob}, + workflowmodel.Node{ID: "build", Kind: workflowmodel.NodeJob, Needs: []string{"lint"}}, + workflowmodel.Node{ID: "unit", Kind: workflowmodel.NodeJob, Needs: []string{"build"}}, + workflowmodel.Node{ID: "integration", Kind: workflowmodel.NodeJob, Needs: []string{"build"}, Matrix: &workflowmodel.Matrix{Keys: []string{"os"}}}, + workflowmodel.Node{ID: "e2e", Kind: workflowmodel.NodeJob, Needs: []string{"unit", "integration"}}), + pipeline("verify", workflowmodel.RolePRVerify, workflowmodel.Node{ID: "dco", Kind: workflowmodel.NodeJob}), + pipeline("release", workflowmodel.RoleRelease, workflowmodel.Node{ID: "goreleaser", Kind: workflowmodel.NodeJob}), + } + m := workflowmodel.Bind(scope(), ps, nil, workflowmodel.Coverage{}) + cols := Columns(m) + labels := make([]string, 0, len(cols)) + for _, c := range cols { + labels = append(labels, c.Label) + } + if got := strings.Join(labels, ","); got != "Open,Actionable,Claimed,PR in progress,CI / Review,Merge queue,Merged,Release,Done" { + t.Fatalf("labels = %s", got) + } + ci := cols[4] + if len(ci.Substages) != 4 { + t.Fatalf("substages = %+v", ci.Substages) + } + // Level 0 merges both PR-verify pipelines; names prefer Node.Name. + if ci.Substages[0].Label != "Lint · dco" { + t.Errorf("level 0 = %q", ci.Substages[0].Label) + } + if ci.Substages[2].Label != "integration · unit" || ci.Substages[3].Label != "e2e" { + t.Errorf("levels = %+v", ci.Substages) + } + if strings.Join(ci.Pipelines, ",") != "ci,verify" { + t.Errorf("ci pipelines = %v", ci.Pipelines) + } + if cols[6].Terminal || !cols[8].Terminal { + t.Error("Done must be the cap once Release exists") + } + d := strings.Join(Diff(m), "\n") + for _, want := range []string{`"CI / Review" gains 4 CI substage(s) from 2 pipeline(s)`, `"Merged" is no longer the cap`, `"Release" added after merge`, `"Done" added after merge`} { + if !strings.Contains(d, want) { + t.Errorf("Diff missing %q:\n%s", want, d) + } + } +} + +func TestSubstageLabelFoldsLongLevels(t *testing.T) { + var nodes []workflowmodel.Node + for _, id := range []string{"a", "b", "c", "d", "e"} { + nodes = append(nodes, workflowmodel.Node{ID: id, Kind: workflowmodel.NodeJob}) + } + m := workflowmodel.Bind(scope(), []workflowmodel.Pipeline{pipeline("ci", workflowmodel.RolePRVerify, nodes...)}, nil, workflowmodel.Coverage{}) + got := Columns(m)[4].Substages[0].Label + if got != "a · b · c +2" { + t.Fatalf("label = %q", got) + } +} + +func TestDeepDAGFoldsIntoMaxSubstages(t *testing.T) { + var nodes []workflowmodel.Node + prev := "" + for i := 0; i < MaxSubstages+3; i++ { + n := workflowmodel.Node{ID: string(rune('a' + i)), Kind: workflowmodel.NodeJob} + if prev != "" { + n.Needs = []string{prev} + } + prev = n.ID + nodes = append(nodes, n) + } + m := workflowmodel.Bind(scope(), []workflowmodel.Pipeline{pipeline("ci", workflowmodel.RolePRVerify, nodes...)}, nil, workflowmodel.Coverage{}) + sub := Columns(m)[4].Substages + if len(sub) != MaxSubstages { + t.Fatalf("got %d substages, want %d", len(sub), MaxSubstages) + } + if len(sub[MaxSubstages-1].Nodes) != 4 { + t.Fatalf("last substage should fold 4 nodes: %+v", sub[MaxSubstages-1]) + } +} + +func TestMermaidIsDeterministicAndWellFormed(t *testing.T) { + ps := []workflowmodel.Pipeline{pipeline("ci", workflowmodel.RolePRVerify, + workflowmodel.Node{ID: "build", Kind: workflowmodel.NodeJob}, + workflowmodel.Node{ID: "test", Kind: workflowmodel.NodeJob, Needs: []string{"build"}, Matrix: &workflowmodel.Matrix{Dynamic: true}}, + workflowmodel.Node{ID: "call", Kind: workflowmodel.NodeReusableWorkflow, Ref: "org/repo/.github/workflows/x.yml@main", Needs: []string{"build"}})} + m := workflowmodel.Bind(scope(), ps, nil, workflowmodel.Coverage{}) + a, b := Mermaid(m), Mermaid(m) + if a != b { + t.Fatal("Mermaid not deterministic") + } + for _, want := range []string{"stateDiagram-v2", "[*] --> open", `state "CI / Review (ci)" as verify`, "state verify {", "--", "note right of ci", "ci: build → call · test", "merged --> [*]", "verify --> merge_queue: verified [required-checks-satisfied]"} { + if !strings.Contains(a, want) { + t.Errorf("missing %q in:\n%s", want, a) + } + } + pm := PipelineMermaid(&m.Spec.Pipelines[0]) + for _, want := range []string{"flowchart LR", `subgraph ci["ci (pr-verify)"]`, `ci_test["test ⊗ dynamic"]`, `ci_call["call ↗ org/repo/.github/workflows/x.yml@main"]`, "ci_build --> ci_test"} { + if !strings.Contains(pm, want) { + t.Errorf("missing %q in:\n%s", want, pm) + } + } +} + +func TestPipelineMermaidTruncatesHugeGraphs(t *testing.T) { + var nodes []workflowmodel.Node + for i := 0; i < MaxDiagramNodes+5; i++ { + nodes = append(nodes, workflowmodel.Node{ID: "j" + strings.Repeat("x", i%3) + string(rune('a'+i%26)) + string(rune('a'+i/26)), Kind: workflowmodel.NodeJob}) + } + p := pipeline("big", workflowmodel.RolePRVerify, nodes...) + out := PipelineMermaid(&p) + if !strings.Contains(out, "… 5 more") { + t.Fatalf("expected truncation marker:\n%s", out) + } +} diff --git a/src/pkg/workflowmodel/store/store.go b/src/pkg/workflowmodel/store/store.go new file mode 100644 index 0000000000..b81d46dba7 --- /dev/null +++ b/src/pkg/workflowmodel/store/store.go @@ -0,0 +1,415 @@ +// Package store persists workflow models on the hive's data volume +// (hivecommons/hive#17620, phase 2). +// +// Layout under Root: +// +// workflows//.json the model, as saved +// workflows/index.json catalogue: one entry per saved version +// +// hive-default is virtual: it is generated, listed read-only, and never +// written. Every save recompiles; a model that does not compile is still +// saved (so an editor can keep a draft) but its entry carries the +// diagnostics and Compiled=false, and Profile refuses it. +package store + +import ( + "encoding/json" + "errors" + "fmt" + "os" + "path/filepath" + "regexp" + "sort" + "strings" + "sync" + "time" + + "github.com/hivecommons/hive/pkg/workflowmodel" + "github.com/hivecommons/hive/pkg/workflowmodel/hivedefault" +) + +// Layout constants. +const ( + // DefaultRoot is the spoke data volume. + DefaultRoot = "/data" + // EnvRoot overrides the root (test seam and operator relocation). + EnvRoot = "HIVE_WORKFLOWS_DATA_DIR" + // Subdir is the directory under the root. + Subdir = "workflows" + // IndexFile is the catalogue inside Subdir. + IndexFile = "index.json" +) + +// Errors. +var ( + ErrNotFound = errors.New("workflow model not found") + ErrReadOnly = errors.New("workflow model is read-only") + ErrBadName = errors.New("workflow model name must be lower-case letters, digits and hyphens") + ErrBadVersion = errors.New("workflow model version must be letters, digits, dots and hyphens") + ErrNotCompiled = errors.New("workflow model does not compile") +) + +var ( + nameRe = regexp.MustCompile(`^[a-z0-9][a-z0-9-]{0,63}$`) + versionRe = regexp.MustCompile(`^[A-Za-z0-9][A-Za-z0-9.-]{0,31}$`) +) + +// Entry is one catalogue line. +type Entry struct { + Name string `json:"name"` + Version string `json:"version"` + Hash string `json:"hash"` + Description string `json:"description,omitempty"` + Author string `json:"author,omitempty"` + Source workflowmodel.MetaSource `json:"source,omitempty"` + Provenance workflowmodel.Provenance `json:"provenance,omitempty"` + Compiled bool `json:"compiled"` + Diagnostics []workflowmodel.Diagnostic `json:"diagnostics,omitempty"` + States int `json:"states"` + Columns int `json:"columns"` + SavedAt time.Time `json:"savedAt,omitzero"` + SavedBy string `json:"savedBy,omitempty"` + ReadOnly bool `json:"readOnly,omitempty"` + // Reserved for the marketplace and the benchmark / rating work (P7+): + // never written by the spoke store, carried verbatim across saves. + Adoption json.RawMessage `json:"adoption,omitempty"` + Ratings json.RawMessage `json:"ratings,omitempty"` + Benchmarks json.RawMessage `json:"benchmarks,omitempty"` +} + +// Key is name@version. +func (e Entry) Key() string { return e.Name + "@" + e.Version } + +// Store is the on-disk model library. +type Store struct { + root string + mu sync.Mutex +} + +// Root resolves the data root: EnvRoot, else configured, else DefaultRoot. +func Root(configured string) string { + if v := strings.TrimSpace(os.Getenv(EnvRoot)); v != "" { + return v + } + if strings.TrimSpace(configured) != "" { + return configured + } + return DefaultRoot +} + +// Open returns a Store rooted at root (the data dir, not the workflows +// subdirectory). Nothing is created until the first save. +func Open(root string) *Store { return &Store{root: root} } + +func (s *Store) dir() string { return filepath.Join(s.root, Subdir) } + +func (s *Store) modelPath(name, version string) string { + return filepath.Join(s.dir(), name, version+".json") +} + +func checkKey(name, version string) error { + if !nameRe.MatchString(name) { + return fmt.Errorf("%w: %q", ErrBadName, name) + } + if !versionRe.MatchString(version) { + return fmt.Errorf("%w: %q", ErrBadVersion, version) + } + return nil +} + +// IsBuiltin reports whether name is the generated, read-only model. +func IsBuiltin(name string) bool { return name == hivedefault.Name } + +func builtinEntry() Entry { + m := hivedefault.Generate() + p := hivedefault.Profile() + return Entry{ + Name: p.Name, Version: p.Version, Hash: p.Hash, + Description: m.Spec.Meta.Description, Author: m.Spec.Meta.Author, + Source: m.Spec.Meta.Source, Provenance: workflowmodel.ProvenanceBuiltIn, + Compiled: true, States: len(p.States), Columns: len(p.Columns), ReadOnly: true, + } +} + +func entryFor(m *workflowmodel.Model, by string, now time.Time) Entry { + e := Entry{ + Name: m.Metadata.Name, Version: m.Spec.Meta.Version, Hash: m.Hash(), + Description: m.Spec.Meta.Description, Author: m.Spec.Meta.Author, + Source: m.Spec.Meta.Source, Provenance: m.Spec.Meta.Provenance, + States: len(m.Spec.States), Columns: len(m.Spec.Columns), + SavedBy: by, + } + if !now.IsZero() { + e.SavedAt = now.UTC().Truncate(time.Second) + } + if e.Provenance == "" { + e.Provenance = workflowmodel.ProvenanceLocal + } + p, diags, err := workflowmodel.Compile(m) + e.Diagnostics = diags + e.Compiled = err == nil && p != nil + return e +} + +// List returns the catalogue: hive-default first, then saved entries by +// name then version. +func (s *Store) List() ([]Entry, error) { + s.mu.Lock() + defer s.mu.Unlock() + idx, err := s.readIndex() + if err != nil { + return nil, err + } + return append([]Entry{builtinEntry()}, idx...), nil +} + +// Get returns a saved model (or the generated hive-default) with its entry. +func (s *Store) Get(name, version string) (*workflowmodel.Model, Entry, error) { + if IsBuiltin(name) { + if version != "" && version != hivedefault.Version { + return nil, Entry{}, fmt.Errorf("%w: %s@%s", ErrNotFound, name, version) + } + return hivedefault.Generate(), builtinEntry(), nil + } + if err := checkKey(name, version); err != nil { + return nil, Entry{}, err + } + s.mu.Lock() + defer s.mu.Unlock() + b, err := os.ReadFile(s.modelPath(name, version)) + if errors.Is(err, os.ErrNotExist) { + return nil, Entry{}, fmt.Errorf("%w: %s@%s", ErrNotFound, name, version) + } + if err != nil { + return nil, Entry{}, err + } + var m workflowmodel.Model + if err := json.Unmarshal(b, &m); err != nil { + return nil, Entry{}, fmt.Errorf("%s@%s: %w", name, version, err) + } + idx, err := s.readIndex() + if err != nil { + return nil, Entry{}, err + } + for _, e := range idx { + if e.Name == name && e.Version == version { + return &m, e, nil + } + } + return &m, entryFor(&m, "", time.Time{}), nil +} + +// Profile returns the compiled profile for name@version; a saved draft +// that does not compile is ErrNotCompiled. +func (s *Store) Profile(name, version string) (*workflowmodel.Profile, error) { + if IsBuiltin(name) && (version == "" || version == hivedefault.Version) { + return hivedefault.Profile(), nil + } + m, _, err := s.Get(name, version) + if err != nil { + return nil, err + } + p, diags, err := workflowmodel.Compile(m) + if err != nil { + return nil, fmt.Errorf("%w: %s@%s: %s", ErrNotCompiled, name, version, summarize(diags)) + } + return p, nil +} + +func summarize(diags []workflowmodel.Diagnostic) string { + var parts []string + for _, d := range diags { + if d.Severity == workflowmodel.DiagError { + parts = append(parts, d.Code) + } + } + return strings.Join(parts, ", ") +} + +// Save writes m as metadata.name@spec.meta.version and refreshes the +// index. The model must validate (shape); a compile failure is recorded +// on the entry, not refused: drafts live in the library, never in a +// binding. +func (s *Store) Save(m *workflowmodel.Model, by string, now time.Time) (Entry, error) { + if m == nil { + return Entry{}, errors.New("nil model") + } + name, version := m.Metadata.Name, m.Spec.Meta.Version + if IsBuiltin(name) { + return Entry{}, fmt.Errorf("%w: %s", ErrReadOnly, name) + } + if err := checkKey(name, version); err != nil { + return Entry{}, err + } + if err := m.Validate(); err != nil { + return Entry{}, err + } + b, err := json.MarshalIndent(m, "", " ") + if err != nil { + return Entry{}, err + } + s.mu.Lock() + defer s.mu.Unlock() + if err := os.MkdirAll(filepath.Dir(s.modelPath(name, version)), 0o755); err != nil { + return Entry{}, err + } + if err := writeAtomic(s.modelPath(name, version), b); err != nil { + return Entry{}, err + } + idx, err := s.readIndex() + if err != nil { + return Entry{}, err + } + e := entryFor(m, by, now) + replaced := false + for i := range idx { + if idx[i].Name == name && idx[i].Version == version { + e.Adoption, e.Ratings, e.Benchmarks = idx[i].Adoption, idx[i].Ratings, idx[i].Benchmarks + idx[i] = e + replaced = true + } + } + if !replaced { + idx = append(idx, e) + } + return e, s.writeIndex(idx) +} + +// Delete removes one version. Whether a repository is bound to it is the +// caller's check: the config owns bindings. +func (s *Store) Delete(name, version string) error { + if IsBuiltin(name) { + return fmt.Errorf("%w: %s", ErrReadOnly, name) + } + if err := checkKey(name, version); err != nil { + return err + } + s.mu.Lock() + defer s.mu.Unlock() + if err := os.Remove(s.modelPath(name, version)); err != nil { + if errors.Is(err, os.ErrNotExist) { + return fmt.Errorf("%w: %s@%s", ErrNotFound, name, version) + } + return err + } + _ = os.Remove(filepath.Join(s.dir(), name)) // succeeds only when empty + idx, err := s.readIndex() + if err != nil { + return err + } + out := idx[:0] + for _, e := range idx { + if e.Name != name || e.Version != version { + out = append(out, e) + } + } + return s.writeIndex(out) +} + +// Versions lists the entries of name, newest save first. +func (s *Store) Versions(name string) ([]Entry, error) { + all, err := s.List() + if err != nil { + return nil, err + } + var out []Entry + for _, e := range all { + if e.Name == name { + out = append(out, e) + } + } + if len(out) == 0 { + return nil, fmt.Errorf("%w: %s", ErrNotFound, name) + } + sort.SliceStable(out, func(i, j int) bool { return out[i].SavedAt.After(out[j].SavedAt) }) + return out, nil +} + +// Rebuild regenerates the index from the model files on disk (recovery +// after a partial write or a hand-copied directory). +func (s *Store) Rebuild(now time.Time) ([]Entry, error) { + s.mu.Lock() + defer s.mu.Unlock() + names, err := os.ReadDir(s.dir()) + if errors.Is(err, os.ErrNotExist) { + return nil, s.writeIndex(nil) + } + if err != nil { + return nil, err + } + var idx []Entry + for _, d := range names { + if !d.IsDir() { + continue + } + files, err := os.ReadDir(filepath.Join(s.dir(), d.Name())) + if err != nil { + return nil, err + } + for _, f := range files { + if f.IsDir() || !strings.HasSuffix(f.Name(), ".json") { + continue + } + b, err := os.ReadFile(filepath.Join(s.dir(), d.Name(), f.Name())) + if err != nil { + return nil, err + } + var m workflowmodel.Model + if err := json.Unmarshal(b, &m); err != nil { + continue // not a model; leave it alone + } + if m.Metadata.Name != d.Name() || m.Spec.Meta.Version+".json" != f.Name() { + continue // misfiled + } + idx = append(idx, entryFor(&m, "", now)) + } + } + return idx, s.writeIndex(idx) +} + +func sortIndex(idx []Entry) { + sort.SliceStable(idx, func(i, j int) bool { + if idx[i].Name != idx[j].Name { + return idx[i].Name < idx[j].Name + } + return idx[i].Version < idx[j].Version + }) +} + +func (s *Store) readIndex() ([]Entry, error) { + b, err := os.ReadFile(filepath.Join(s.dir(), IndexFile)) + if errors.Is(err, os.ErrNotExist) { + return nil, nil + } + if err != nil { + return nil, err + } + var idx []Entry + if err := json.Unmarshal(b, &idx); err != nil { + return nil, fmt.Errorf("%s: %w", IndexFile, err) + } + return idx, nil +} + +func (s *Store) writeIndex(idx []Entry) error { + sortIndex(idx) + if idx == nil { + idx = []Entry{} + } + b, err := json.MarshalIndent(idx, "", " ") + if err != nil { + return err + } + if err := os.MkdirAll(s.dir(), 0o755); err != nil { + return err + } + return writeAtomic(filepath.Join(s.dir(), IndexFile), b) +} + +func writeAtomic(path string, b []byte) error { + tmp := path + ".tmp" + if err := os.WriteFile(tmp, b, 0o644); err != nil { + return err + } + return os.Rename(tmp, path) +} diff --git a/src/pkg/workflowmodel/store/store_test.go b/src/pkg/workflowmodel/store/store_test.go new file mode 100644 index 0000000000..01d178341d --- /dev/null +++ b/src/pkg/workflowmodel/store/store_test.go @@ -0,0 +1,259 @@ +package store + +import ( + "encoding/json" + "errors" + "os" + "path/filepath" + "testing" + "time" + + "github.com/hivecommons/hive/pkg/workflowmodel" + "github.com/hivecommons/hive/pkg/workflowmodel/hivedefault" +) + +func techdocs(t *testing.T) *workflowmodel.Model { + t.Helper() + b, err := os.ReadFile("../testdata/models/techdocs.json") + if err != nil { + t.Fatal(err) + } + var m workflowmodel.Model + if err := json.Unmarshal(b, &m); err != nil { + t.Fatal(err) + } + return &m +} + +var t0 = time.Date(2026, 11, 10, 12, 0, 0, 0, time.UTC) + +func TestEmptyStoreListsBuiltin(t *testing.T) { + s := Open(t.TempDir()) + got, err := s.List() + if err != nil || len(got) != 1 || got[0].Name != hivedefault.Name || !got[0].ReadOnly || !got[0].Compiled || got[0].Provenance != workflowmodel.ProvenanceBuiltIn { + t.Fatalf("%v %+v", err, got) + } + m, e, err := s.Get(hivedefault.Name, "") + if err != nil || m == nil || e.Hash != hivedefault.Profile().Hash { + t.Fatalf("%v %+v", err, e) + } + if _, _, err := s.Get(hivedefault.Name, "9"); !errors.Is(err, ErrNotFound) { + t.Fatal(err) + } + p, err := s.Profile(hivedefault.Name, hivedefault.Version) + if err != nil || p.Name != hivedefault.Name { + t.Fatal(err) + } + if _, err := s.Save(hivedefault.Generate(), "andy", t0); !errors.Is(err, ErrReadOnly) { + t.Fatal(err) + } + if err := s.Delete(hivedefault.Name, "1"); !errors.Is(err, ErrReadOnly) { + t.Fatal(err) + } +} + +func TestSaveGetListDelete(t *testing.T) { + root := t.TempDir() + s := Open(root) + m := techdocs(t) + e, err := s.Save(m, "andy", t0) + if err != nil { + t.Fatal(err) + } + if !e.Compiled || e.Provenance != workflowmodel.ProvenanceLocal || e.SavedBy != "andy" || !e.SavedAt.Equal(t0) || e.States != 9 || e.Columns != 5 || e.Key() != "cncf-techdocs@1" { + t.Fatalf("%+v", e) + } + if _, err := os.Stat(filepath.Join(root, Subdir, "cncf-techdocs", "1.json")); err != nil { + t.Fatal(err) + } + list, err := s.List() + if err != nil || len(list) != 2 || list[1].Key() != "cncf-techdocs@1" { + t.Fatalf("%v %+v", err, list) + } + got, ge, err := s.Get("cncf-techdocs", "1") + if err != nil || got.Hash() != m.Hash() || ge.SavedBy != "andy" { + t.Fatalf("%v %+v", err, ge) + } + if p, err := s.Profile("cncf-techdocs", "1"); err != nil || p.Mode != workflowmodel.DerivationRules { + t.Fatal(err) + } + + // second version, then Versions is newest first + m2 := techdocs(t) + m2.Spec.Meta.Version = "2" + if _, err := s.Save(m2, "nate", t0.Add(time.Hour)); err != nil { + t.Fatal(err) + } + vs, err := s.Versions("cncf-techdocs") + if err != nil || len(vs) != 2 || vs[0].Version != "2" { + t.Fatalf("%v %+v", err, vs) + } + if _, err := s.Versions("nobody"); !errors.Is(err, ErrNotFound) { + t.Fatal(err) + } + + if err := s.Delete("cncf-techdocs", "1"); err != nil { + t.Fatal(err) + } + if err := s.Delete("cncf-techdocs", "1"); !errors.Is(err, ErrNotFound) { + t.Fatal(err) + } + if _, _, err := s.Get("cncf-techdocs", "1"); !errors.Is(err, ErrNotFound) { + t.Fatal(err) + } + list, _ = s.List() + if len(list) != 2 || list[1].Version != "2" { + t.Fatalf("%+v", list) + } + if err := s.Delete("cncf-techdocs", "2"); err != nil { + t.Fatal(err) + } + if _, err := os.Stat(filepath.Join(root, Subdir, "cncf-techdocs")); !errors.Is(err, os.ErrNotExist) { + t.Fatalf("empty model dir should be removed: %v", err) + } +} + +func TestDraftsSaveButDoNotProfile(t *testing.T) { + s := Open(t.TempDir()) + m := techdocs(t) + m.Spec.Gates[0].ReleasedBy = "" + e, err := s.Save(m, "", t0) + if err != nil { + t.Fatal(err) + } + if e.Compiled || len(e.Diagnostics) == 0 { + t.Fatalf("%+v", e) + } + if _, err := s.Profile("cncf-techdocs", "1"); !errors.Is(err, ErrNotCompiled) { + t.Fatal(err) + } + // an invalid shape is refused outright + m.Spec.Initial = "ghost" + if _, err := s.Save(m, "", t0); err == nil { + t.Fatal("invalid model saved") + } +} + +func TestKeysAreChecked(t *testing.T) { + s := Open(t.TempDir()) + m := techdocs(t) + m.Metadata.Name = "../etc" + if _, err := s.Save(m, "", t0); !errors.Is(err, ErrBadName) { + t.Fatal(err) + } + m = techdocs(t) + m.Spec.Meta.Version = "1/../x" + if _, err := s.Save(m, "", t0); !errors.Is(err, ErrBadVersion) { + t.Fatal(err) + } + if _, _, err := s.Get("Bad Name", "1"); !errors.Is(err, ErrBadName) { + t.Fatal(err) + } + if err := s.Delete("ok", "bad version"); !errors.Is(err, ErrBadVersion) { + t.Fatal(err) + } + if _, err := s.Save(nil, "", t0); err == nil { + t.Fatal("nil saved") + } +} + +func TestResaveKeepsReservedFields(t *testing.T) { + root := t.TempDir() + s := Open(root) + m := techdocs(t) + if _, err := s.Save(m, "", t0); err != nil { + t.Fatal(err) + } + idx, _ := s.readIndex() + idx[0].Ratings = json.RawMessage(`{"stars":5}`) + if err := s.writeIndex(idx); err != nil { + t.Fatal(err) + } + m.Spec.Meta.Description = "edited" + e, err := s.Save(m, "", t0.Add(time.Minute)) + if err != nil || e.Description != "edited" { + t.Fatalf("%v %+v", err, e) + } + var r struct{ Stars int } + if err := json.Unmarshal(e.Ratings, &r); err != nil || r.Stars != 5 { + t.Fatalf("ratings %s: %v", e.Ratings, err) + } +} + +func TestRebuild(t *testing.T) { + root := t.TempDir() + s := Open(root) + if _, err := s.Save(techdocs(t), "", t0); err != nil { + t.Fatal(err) + } + // corrupt the index, drop a stray file and a misfiled model + if err := os.WriteFile(filepath.Join(root, Subdir, IndexFile), []byte("nope"), 0o644); err != nil { + t.Fatal(err) + } + if _, err := s.List(); err == nil { + t.Fatal("corrupt index read") + } + _ = os.WriteFile(filepath.Join(root, Subdir, "cncf-techdocs", "notes.txt"), []byte("x"), 0o644) + _ = os.WriteFile(filepath.Join(root, Subdir, "cncf-techdocs", "7.json"), []byte(`{"metadata":{"name":"other"}}`), 0o644) + _ = os.WriteFile(filepath.Join(root, Subdir, "cncf-techdocs", "8.json"), []byte(`not json`), 0o644) + idx, err := s.Rebuild(t0) + if err != nil || len(idx) != 1 || idx[0].Key() != "cncf-techdocs@1" { + t.Fatalf("%v %+v", err, idx) + } + // an index entry missing for a model on disk still resolves on Get + if err := os.WriteFile(filepath.Join(root, Subdir, IndexFile), []byte("[]"), 0o644); err != nil { + t.Fatal(err) + } + _, e, err := s.Get("cncf-techdocs", "1") + if err != nil || !e.Compiled || e.SavedBy != "" { + t.Fatalf("%v %+v", err, e) + } + + empty := Open(t.TempDir()) + if idx, err := empty.Rebuild(t0); err != nil || len(idx) != 0 { + t.Fatalf("%v %+v", err, idx) + } +} + +func TestUnreadableModelAndIndex(t *testing.T) { + root := t.TempDir() + s := Open(root) + if _, err := s.Save(techdocs(t), "", t0); err != nil { + t.Fatal(err) + } + if err := os.WriteFile(s.modelPath("cncf-techdocs", "1"), []byte("{"), 0o644); err != nil { + t.Fatal(err) + } + if _, _, err := s.Get("cncf-techdocs", "1"); err == nil { + t.Fatal("truncated model read") + } + if _, err := s.Profile("cncf-techdocs", "1"); err == nil { + t.Fatal("truncated model profiled") + } + if err := os.WriteFile(filepath.Join(root, Subdir, IndexFile), []byte("["), 0o644); err != nil { + t.Fatal(err) + } + if _, err := s.Versions("cncf-techdocs"); err == nil { + t.Fatal("corrupt index listed") + } + if _, err := s.Save(techdocs(t), "", t0); err == nil { + t.Fatal("saved over a corrupt index") + } + if err := s.Delete("cncf-techdocs", "1"); err == nil { + t.Fatal("deleted over a corrupt index") + } + if vs, err := Open(t.TempDir()).Versions(hivedefault.Name); err != nil || len(vs) != 1 { + t.Fatalf("%v %+v", err, vs) + } +} + +func TestRoot(t *testing.T) { + t.Setenv(EnvRoot, "") + if Root("") != DefaultRoot || Root("/x") != "/x" { + t.Fatal("root resolution") + } + t.Setenv(EnvRoot, "/env") + if Root("/x") != "/env" { + t.Fatal("env must win") + } +} diff --git a/src/pkg/workflowmodel/tekton/doc.go b/src/pkg/workflowmodel/tekton/doc.go new file mode 100644 index 0000000000..3a29399043 --- /dev/null +++ b/src/pkg/workflowmodel/tekton/doc.go @@ -0,0 +1,11 @@ +// Package tekton ingests Tekton Pipeline and PipelineRun definitions into +// workflowmodel pipelines. +// +// Files may hold several YAML documents or a Kubernetes List; every +// document whose apiVersion is in the tekton.dev family is considered. +// Pipeline resources and PipelineRuns with an embedded pipelineSpec become +// one Pipeline each; PipelineRuns that reference a Pipeline by name +// contribute their Pipelines-as-Code triggers to it; Tekton Triggers +// documents contribute a webhook trigger. Everything the ingest recognises +// but cannot model is recorded as a Finding rather than an error. +package tekton diff --git a/src/pkg/workflowmodel/tekton/documents.go b/src/pkg/workflowmodel/tekton/documents.go new file mode 100644 index 0000000000..f2071d1a47 --- /dev/null +++ b/src/pkg/workflowmodel/tekton/documents.go @@ -0,0 +1,168 @@ +package tekton + +import ( + "bytes" + "errors" + "io" + "regexp" + "strings" + + "gopkg.in/yaml.v3" +) + +// Tekton API groups whose documents the ingest considers. +var tektonAPIGroups = []string{"tekton.dev/", "triggers.tekton.dev/", "pipelinesascode.tekton.dev/"} + +// Resource kinds, as they appear in the kind field. +const ( + kindPipeline = "Pipeline" + kindPipelineRun = "PipelineRun" + kindTask = "Task" + kindTaskRun = "TaskRun" + kindClusterTask = "ClusterTask" + kindStepAction = "StepAction" + kindEventListener = "EventListener" + kindTriggerTemplate = "TriggerTemplate" + kindTriggerBinding = "TriggerBinding" + kindTrigger = "Trigger" + kindList = "List" +) + +var kindLinePattern = regexp.MustCompile(`(?m)^kind:\s*(Pipeline|PipelineRun|Task|TaskRun|ClusterTask|StepAction|EventListener|TriggerTemplate|TriggerBinding|Trigger)\s*$`) + +// LooksTekton is a cheap pre-filter on raw bytes: the data mentions a +// tekton.dev API group or has a kind line naming a Tekton resource. +func LooksTekton(data []byte) bool { + return bytes.Contains(data, []byte("tekton.dev/")) || kindLinePattern.Match(data) +} + +// IsTektonDocument reports whether a parsed YAML mapping has an apiVersion +// in the tekton.dev, triggers.tekton.dev or pipelinesascode.tekton.dev +// groups. +func IsTektonDocument(doc map[string]any) bool { + apiVersion := stringAt(doc, "apiVersion") + for _, g := range tektonAPIGroups { + if strings.HasPrefix(apiVersion, g) { + return true + } + } + return false +} + +// SplitDocuments decodes every YAML document in data, skipping empty ones +// and expanding kind: List into its items. Documents that are not mappings +// are dropped. It stops at the first malformed document and returns the +// error. +func SplitDocuments(data []byte) ([]map[string]any, error) { + dec := yaml.NewDecoder(bytes.NewReader(data)) + var out []map[string]any + for { + var doc any + err := dec.Decode(&doc) + if errors.Is(err, io.EOF) { + return out, nil + } + if err != nil { + return out, err + } + m, ok := doc.(map[string]any) + if !ok { + continue + } + if stringAt(m, "kind") == kindList { + for _, item := range sliceAt(m, "items") { + if im, ok := item.(map[string]any); ok { + out = append(out, im) + } + } + continue + } + out = append(out, m) + } +} + +// stringAt returns m[key] when it is a string, else "". +func stringAt(m map[string]any, key string) string { + if m == nil { + return "" + } + s, _ := m[key].(string) + return s +} + +// mapAt returns m[key] when it is a mapping, else nil. +func mapAt(m map[string]any, key string) map[string]any { + if m == nil { + return nil + } + sub, _ := m[key].(map[string]any) + return sub +} + +// sliceAt returns m[key] when it is a sequence, else nil. +func sliceAt(m map[string]any, key string) []any { + if m == nil { + return nil + } + s, _ := m[key].([]any) + return s +} + +// mapsAt returns the mapping elements of the sequence at m[key]. +func mapsAt(m map[string]any, key string) []map[string]any { + items := sliceAt(m, key) + out := make([]map[string]any, 0, len(items)) + for _, it := range items { + if im, ok := it.(map[string]any); ok { + out = append(out, im) + } + } + return out +} + +// stringsAt returns m[key] as strings: a scalar becomes a one-element +// slice, a sequence keeps its scalar elements. +func stringsAt(m map[string]any, key string) []string { + if m == nil { + return nil + } + switch v := m[key].(type) { + case string: + return []string{v} + case []any: + out := make([]string, 0, len(v)) + for _, it := range v { + if s, ok := it.(string); ok { + out = append(out, s) + } + } + return out + } + return nil +} + +// metadataName returns metadata.name, falling back to generateName with +// its trailing hyphen removed. +func metadataName(doc map[string]any) string { + md := mapAt(doc, "metadata") + if name := stringAt(md, "name"); name != "" { + return name + } + return strings.TrimSuffix(stringAt(md, "generateName"), "-") +} + +// walkStrings calls fn with every string scalar reachable from v. +func walkStrings(v any, fn func(string)) { + switch t := v.(type) { + case string: + fn(t) + case []any: + for _, it := range t { + walkStrings(it, fn) + } + case map[string]any: + for _, it := range t { + walkStrings(it, fn) + } + } +} diff --git a/src/pkg/workflowmodel/tekton/ingest.go b/src/pkg/workflowmodel/tekton/ingest.go new file mode 100644 index 0000000000..ef7f0063e7 --- /dev/null +++ b/src/pkg/workflowmodel/tekton/ingest.go @@ -0,0 +1,315 @@ +package tekton + +import ( + "fmt" + "sort" + "strings" + + "github.com/hivecommons/hive/pkg/workflowmodel" +) + +// File is one source file handed to Ingest. +type File struct { + Path string + Data []byte +} + +// Result is what Ingest produces: pipelines, the sources they came from, +// and how completely the sources were understood. +type Result struct { + Pipelines []workflowmodel.Pipeline + Sources []workflowmodel.Source + Coverage workflowmodel.Coverage +} + +// document is one parsed YAML mapping with the path it came from. +type document struct { + path string + doc map[string]any +} + +// pipelineRun is a PipelineRun that references a Pipeline by name, held +// until every Pipeline has been collected. +type pipelineRun struct { + path string + doc map[string]any + ref string +} + +// ingest holds the state of one Ingest call. +type ingest struct { + result Result + findings []workflowmodel.Finding + pipelines []*workflowmodel.Pipeline + // byName indexes Pipeline resources (not embedded runs) by + // metadata.name; the first declaration wins. + byName map[string]*workflowmodel.Pipeline + runs []pipelineRun + // webhooks are Pipeline names a TriggerTemplate instantiates. + webhooks []string +} + +// Ingest reads every file, collects every Tekton document and emits one +// Pipeline per Pipeline resource and per PipelineRun with an embedded +// pipelineSpec. PipelineRuns that reference a Pipeline by name add their +// Pipelines-as-Code triggers to it; TriggerTemplates that instantiate a +// Pipeline add a webhook trigger. It never fails on a bad file: problems +// become Findings. Output is deterministic: pipelines sorted by (Path, +// Name), findings by (Path, Node, Code, Detail). +func Ingest(files []File) Result { + in := &ingest{byName: map[string]*workflowmodel.Pipeline{}} + in.result.Coverage.Files = len(files) + for _, f := range files { + in.file(f) + } + in.linkRuns() + in.linkWebhooks() + in.finish() + return in.result +} + +func (in *ingest) find(path, node, code, detail string, sev workflowmodel.Severity) { + in.findings = append(in.findings, workflowmodel.Finding{Path: path, Node: node, Code: code, Detail: detail, Severity: sev}) +} + +// file parses one file and dispatches its Tekton documents. +func (in *ingest) file(f File) { + in.result.Sources = append(in.result.Sources, workflowmodel.Source{System: workflowmodel.SystemTekton, Path: f.Path, SHA256: workflowmodel.SHA256Hex(f.Data)}) + docs, err := SplitDocuments(f.Data) + if err != nil { + in.find(f.Path, "", CodeYAML, err.Error(), workflowmodel.SeverityUnknown) + return + } + var tekton []document + for _, d := range docs { + if IsTektonDocument(d) { + tekton = append(tekton, document{path: f.Path, doc: d}) + } + } + if len(tekton) == 0 { + in.find(f.Path, "", CodeNotTekton, "no tekton.dev documents", workflowmodel.SeverityUnknown) + return + } + in.result.Coverage.Parsed++ + + // Pipeline ids only carry the resource name when the file emits more + // than one pipeline, so count first. + emitted := 0 + for _, d := range tekton { + if emitsPipeline(d.doc) { + emitted++ + } + } + taskDefs := 0 + for _, d := range tekton { + kind := stringAt(d.doc, "kind") + switch kind { + case kindPipeline: + in.pipeline(d, emitted > 1, true) + case kindPipelineRun: + in.pipelineRun(d, emitted > 1) + case kindTask, kindClusterTask, kindStepAction, kindTaskRun: + taskDefs++ + case kindEventListener, kindTriggerTemplate, kindTriggerBinding, kindTrigger: + in.find(f.Path, "", CodeTektonTriggers, kind+" "+metadataName(d.doc), workflowmodel.SeverityUnsupported) + if kind == kindTriggerTemplate { + in.webhooks = append(in.webhooks, templatePipelineRefs(d.doc)...) + } + default: + in.find(f.Path, "", CodeUnknownKind, kind+" "+metadataName(d.doc), workflowmodel.SeverityUnknown) + } + } + if taskDefs > 0 { + in.find(f.Path, "", CodeTaskDefinitions, fmt.Sprintf("%d task/stepaction documents", taskDefs), workflowmodel.SeverityWarning) + } +} + +// emitsPipeline reports whether doc becomes a Pipeline of its own. +func emitsPipeline(doc map[string]any) bool { + switch stringAt(doc, "kind") { + case kindPipeline: + return true + case kindPipelineRun: + return mapAt(mapAt(doc, "spec"), "pipelineSpec") != nil + } + return false +} + +// pipeline emits a Pipeline from a Pipeline resource or an embedded +// pipelineSpec. register says whether PipelineRuns may reference it by +// name. +func (in *ingest) pipeline(d document, qualify, register bool) *workflowmodel.Pipeline { + name := metadataName(d.doc) + id := workflowmodel.PipelineID(workflowmodel.SystemTekton, d.path) + if qualify { + id += "-" + workflowmodel.Slug(name) + } + spec := mapAt(d.doc, "spec") + if stringAt(d.doc, "kind") == kindPipelineRun { + spec = mapAt(spec, "pipelineSpec") + } + p, findings := buildPipeline(d.path, id, name, spec) + in.findings = append(in.findings, findings...) + in.pipelines = append(in.pipelines, &p) + if register { + if _, dup := in.byName[name]; !dup { + in.byName[name] = &p + } + } + return &p +} + +// pipelineRun handles a PipelineRun: embedded specs become pipelines +// carrying the run's own triggers; references are linked after every +// Pipeline is known. +func (in *ingest) pipelineRun(d document, qualify bool) { + spec := mapAt(d.doc, "spec") + if mapAt(spec, "pipelineSpec") != nil { + p := in.pipeline(d, qualify, false) + addTriggers(p, pacTriggers(d.doc)) + return + } + ref := mapAt(spec, "pipelineRef") + in.runs = append(in.runs, pipelineRun{path: d.path, doc: d.doc, ref: pipelineRefName(ref)}) +} + +// pipelineRefName renders a pipelineRef: the name, or resolver plus its +// identifying params when resolved remotely. +func pipelineRefName(ref map[string]any) string { + if name := stringAt(ref, "name"); name != "" { + return name + } + resolver := stringAt(ref, "resolver") + if resolver == "" { + return "" + } + var parts []string + for _, p := range mapsAt(ref, "params") { + if contains(resolverParamNames, stringAt(p, "name")) { + if v := stringAt(p, "value"); v != "" { + parts = append(parts, v) + } + } + } + if len(parts) == 0 { + return resolver + } + return resolver + ":" + strings.Join(parts, "/") +} + +// linkRuns attaches each referencing PipelineRun's triggers to its +// Pipeline, or records that the Pipeline lives elsewhere. +func (in *ingest) linkRuns() { + for _, r := range in.runs { + p, ok := in.byName[r.ref] + if !ok { + detail := "pipelineRef " + r.ref + " not in ingested files" + if r.ref == "" { + detail = "PipelineRun has neither pipelineRef nor pipelineSpec" + } + in.find(r.path, "", CodePipelineRefExternal, detail, workflowmodel.SeverityWarning) + continue + } + addTriggers(p, pacTriggers(r.doc)) + } +} + +// linkWebhooks adds a webhook trigger to every Pipeline a TriggerTemplate +// instantiates. +func (in *ingest) linkWebhooks() { + for _, name := range in.webhooks { + p, ok := in.byName[name] + if !ok { + continue + } + addTriggers(p, []workflowmodel.Trigger{{Event: eventWebhook}}) + } +} + +// templatePipelineRefs lists the Pipeline names a TriggerTemplate's +// resourcetemplates instantiate through pipelineRef.name. +func templatePipelineRefs(doc map[string]any) []string { + var out []string + for _, rt := range mapsAt(mapAt(doc, "spec"), "resourcetemplates") { + if stringAt(rt, "kind") != kindPipelineRun { + continue + } + if name := stringAt(mapAt(mapAt(rt, "spec"), "pipelineRef"), "name"); name != "" { + out = append(out, name) + } + } + return out +} + +// addTriggers appends ts to p.Triggers, skipping any trigger whose full +// value (event and every list, order-insensitive) is already present, so +// many PipelineRuns of one Pipeline do not repeat the same trigger. +// First-seen order is preserved. +func addTriggers(p *workflowmodel.Pipeline, ts []workflowmodel.Trigger) { + seen := make(map[string]bool, len(p.Triggers)+len(ts)) + for _, t := range p.Triggers { + seen[triggerKey(t)] = true + } + for _, t := range ts { + k := triggerKey(t) + if seen[k] { + continue + } + seen[k] = true + p.Triggers = append(p.Triggers, t) + } +} + +// triggerKey is a canonical string for a trigger: the event followed by +// each list sorted, so list order does not make two equal triggers differ. +func triggerKey(t workflowmodel.Trigger) string { + var b strings.Builder + b.WriteString(t.Event) + for _, list := range [][]string{t.Types, t.Branches, t.Tags, t.Paths, t.Schedule} { + sorted := append([]string(nil), list...) + sort.Strings(sorted) + b.WriteString("\x1f") + b.WriteString(strings.Join(sorted, "\x1e")) + } + return b.String() +} + +func hasEvent(ts []workflowmodel.Trigger, event string) bool { + for _, t := range ts { + if t.Event == event { + return true + } + } + return false +} + +// finish derives roles, flags trigger-less pipelines and sorts output. +func (in *ingest) finish() { + for _, p := range in.pipelines { + p.Role = roleFor(p.Triggers) + if len(p.Triggers) == 0 { + in.find(p.Path, "", CodeNoTrigger, "pipeline "+p.Name+" has no PipelineRun or trigger in the ingested files", workflowmodel.SeverityWarning) + } + in.result.Pipelines = append(in.result.Pipelines, *p) + } + sort.SliceStable(in.result.Pipelines, func(i, j int) bool { + a, b := in.result.Pipelines[i], in.result.Pipelines[j] + if a.Path != b.Path { + return a.Path < b.Path + } + return a.Name < b.Name + }) + sort.SliceStable(in.findings, func(i, j int) bool { + a, b := in.findings[i], in.findings[j] + switch { + case a.Path != b.Path: + return a.Path < b.Path + case a.Node != b.Node: + return a.Node < b.Node + case a.Code != b.Code: + return a.Code < b.Code + } + return a.Detail < b.Detail + }) + in.result.Coverage.Findings = in.findings +} diff --git a/src/pkg/workflowmodel/tekton/ingest_test.go b/src/pkg/workflowmodel/tekton/ingest_test.go new file mode 100644 index 0000000000..0316676f4b --- /dev/null +++ b/src/pkg/workflowmodel/tekton/ingest_test.go @@ -0,0 +1,671 @@ +package tekton + +import ( + "os" + "path/filepath" + "reflect" + "strings" + "testing" + + "github.com/hivecommons/hive/pkg/workflowmodel" +) + +func file(path, data string) File { return File{Path: path, Data: []byte(data)} } + +func findPipeline(t *testing.T, r Result, name string) workflowmodel.Pipeline { + t.Helper() + for _, p := range r.Pipelines { + if p.Name == name { + return p + } + } + t.Fatalf("pipeline %q not found; have %v", name, pipelineNames(r)) + return workflowmodel.Pipeline{} +} + +func pipelineNames(r Result) []string { + var out []string + for _, p := range r.Pipelines { + out = append(out, p.Name) + } + return out +} + +func findNode(t *testing.T, p workflowmodel.Pipeline, id string) workflowmodel.Node { + t.Helper() + for _, n := range p.Nodes { + if n.ID == id { + return n + } + } + t.Fatalf("node %q not in pipeline %q", id, p.Name) + return workflowmodel.Node{} +} + +func findingCodes(r Result) []string { + var out []string + for _, f := range r.Coverage.Findings { + out = append(out, f.Code) + } + return out +} + +func hasFinding(r Result, code string) bool { + for _, f := range r.Coverage.Findings { + if f.Code == code { + return true + } + } + return false +} + +func findingByCode(t *testing.T, r Result, code string) workflowmodel.Finding { + t.Helper() + for _, f := range r.Coverage.Findings { + if f.Code == code { + return f + } + } + t.Fatalf("finding %q not present; have %v", code, findingCodes(r)) + return workflowmodel.Finding{} +} + +const pipelineWithTask = `apiVersion: tekton.dev/v1 +kind: Task +metadata: + name: unit +spec: + steps: + - name: go + image: golang + script: go test ./... +--- +apiVersion: tekton.dev/v1 +kind: Pipeline +metadata: + name: verify +spec: + tasks: + - name: unit + taskRef: + name: unit + - name: lint + taskSpec: + steps: + - image: golangci/golangci-lint +` + +const pacRun = `apiVersion: tekton.dev/v1 +kind: PipelineRun +metadata: + generateName: build-on-pr- + annotations: + pipelinesascode.tekton.dev/on-event: "[pull_request]" + pipelinesascode.tekton.dev/on-target-branch: "[main, 'release-*']" + pipelinesascode.tekton.dev/on-path-change: "[src/***]" +spec: + pipelineSpec: + tasks: + - name: clone + taskRef: + resolver: git + params: + - name: url + value: https://github.com/example/catalog.git + - name: pathInRepo + value: tasks/git-clone.yaml + - name: build + runAfter: [clone] + taskRef: + name: buildah + params: + - name: context + value: $(tasks.clone.results.path) + - name: test + params: + - name: image + value: $(tasks.build.results.IMAGE_URL) + when: + - input: $(params.skip-tests) + operator: notin + values: ["true"] + taskRef: + name: go-test + finally: + - name: notify + taskRef: + name: slack +` + +func TestIngestMultiDocPipelineAndTask(t *testing.T) { + r := Ingest([]File{file(".tekton/verify.yaml", pipelineWithTask)}) + if len(r.Pipelines) != 1 { + t.Fatalf("pipelines = %d, want 1", len(r.Pipelines)) + } + p := r.Pipelines[0] + if p.ID != "tkn-tekton-verify" || p.Name != "verify" || p.System != workflowmodel.SystemTekton || p.Path != ".tekton/verify.yaml" { + t.Errorf("pipeline identity = %+v", p) + } + if p.Role != workflowmodel.RoleOther { + t.Errorf("role = %q, want other", p.Role) + } + if got := findNode(t, p, "unit").Ref; got != "unit" { + t.Errorf("unit ref = %q", got) + } + if got := findNode(t, p, "lint").Ref; got != refInline { + t.Errorf("lint ref = %q, want inline", got) + } + if p.Levels != 1 { + t.Errorf("levels = %d, want 1", p.Levels) + } + tasks := findingByCode(t, r, CodeTaskDefinitions) + if tasks.Detail != "1 task/stepaction documents" || tasks.Severity != workflowmodel.SeverityWarning { + t.Errorf("task-definitions finding = %+v", tasks) + } + if !hasFinding(r, CodeNoTrigger) { + t.Errorf("want no-trigger finding; have %v", findingCodes(r)) + } + if r.Coverage.Files != 1 || r.Coverage.Parsed != 1 { + t.Errorf("coverage = %+v", r.Coverage) + } + if len(r.Sources) != 1 || r.Sources[0].SHA256 != workflowmodel.SHA256Hex([]byte(pipelineWithTask)) { + t.Errorf("sources = %+v", r.Sources) + } +} + +func TestIngestEmbeddedSpecWithPaC(t *testing.T) { + r := Ingest([]File{file(".tekton/build.yaml", pacRun)}) + if len(r.Pipelines) != 1 { + t.Fatalf("pipelines = %d, want 1 (%v)", len(r.Pipelines), pipelineNames(r)) + } + p := r.Pipelines[0] + if p.Name != "build-on-pr" { + t.Errorf("name = %q, want generateName trimmed", p.Name) + } + if p.Role != workflowmodel.RolePRVerify { + t.Errorf("role = %q, want pr-verify", p.Role) + } + wantTrigger := workflowmodel.Trigger{Event: "pull_request", Branches: []string{"main", "release-*"}, Paths: []string{"src/***"}} + if len(p.Triggers) != 1 || !reflect.DeepEqual(p.Triggers[0], wantTrigger) { + t.Errorf("triggers = %+v, want [%+v]", p.Triggers, wantTrigger) + } + if got := findNode(t, p, "clone").Ref; got != "git:https://github.com/example/catalog.git/tasks/git-clone.yaml" { + t.Errorf("clone ref = %q", got) + } + if got := findNode(t, p, "build").Needs; !reflect.DeepEqual(got, []string{"clone"}) { + t.Errorf("build needs = %v", got) + } + test := findNode(t, p, "test") + if !reflect.DeepEqual(test.Needs, []string{"build"}) { + t.Errorf("test implicit needs = %v, want [build]", test.Needs) + } + if !reflect.DeepEqual(test.When, []string{"$(params.skip-tests) notin [true]"}) { + t.Errorf("test when = %v", test.When) + } + notify := findNode(t, p, "notify") + if notify.Kind != workflowmodel.NodeFinally { + t.Errorf("notify kind = %q", notify.Kind) + } + if !reflect.DeepEqual(notify.Needs, []string{"clone", "build", "test"}) { + t.Errorf("notify needs = %v", notify.Needs) + } + if p.Levels != 4 || notify.Level != 3 { + t.Errorf("levels = %d, notify level = %d; want 4 and 3", p.Levels, notify.Level) + } + f := findingByCode(t, r, CodeFinally) + if f.Severity != workflowmodel.SeverityWarning { + t.Errorf("finally finding = %+v", f) + } + if hasFinding(r, CodeNoTrigger) { + t.Errorf("unexpected no-trigger finding") + } +} + +func TestIngestPipelineRunReferences(t *testing.T) { + const pipeline = `apiVersion: tekton.dev/v1beta1 +kind: Pipeline +metadata: + name: release +spec: + tasks: + - name: publish + taskRef: + name: publish +` + const runs = `apiVersion: tekton.dev/v1beta1 +kind: PipelineRun +metadata: + name: release-on-tag + annotations: + pipelinesascode.tekton.dev/on-event: "[push]" + pipelinesascode.tekton.dev/on-target-branch: "[refs/tags/*]" +spec: + pipelineRef: + name: release +--- +apiVersion: tekton.dev/v1beta1 +kind: PipelineRun +metadata: + name: orphan +spec: + pipelineRef: + name: does-not-exist +` + r := Ingest([]File{file("tekton/release.yaml", pipeline), file(".tekton/runs.yaml", runs)}) + if len(r.Pipelines) != 1 { + t.Fatalf("pipelines = %v, want only release", pipelineNames(r)) + } + p := r.Pipelines[0] + if p.Role != workflowmodel.RoleRelease { + t.Errorf("role = %q, want release", p.Role) + } + if len(p.Triggers) != 1 || p.Triggers[0].Event != "push" || !reflect.DeepEqual(p.Triggers[0].Branches, []string{"refs/tags/*"}) { + t.Errorf("triggers = %+v", p.Triggers) + } + ext := findingByCode(t, r, CodePipelineRefExternal) + if ext.Path != ".tekton/runs.yaml" || !strings.Contains(ext.Detail, "does-not-exist") || ext.Severity != workflowmodel.SeverityWarning { + t.Errorf("pipelineref-external = %+v", ext) + } + if hasFinding(r, CodeNoTrigger) { + t.Errorf("release has a run; no-trigger must not fire") + } +} + +func TestIngestKubernetesList(t *testing.T) { + const list = `apiVersion: v1 +kind: List +items: + - apiVersion: tekton.dev/v1 + kind: Pipeline + metadata: + name: alpha + spec: + tasks: + - name: a + taskRef: + name: a + - apiVersion: tekton.dev/v1 + kind: Pipeline + metadata: + name: beta + spec: + tasks: + - name: b + taskRef: + name: b +` + r := Ingest([]File{file("all.yaml", list)}) + if got := pipelineNames(r); !reflect.DeepEqual(got, []string{"alpha", "beta"}) { + t.Fatalf("pipelines = %v", got) + } + if r.Pipelines[0].ID != "tkn-all-alpha" || r.Pipelines[1].ID != "tkn-all-beta" { + t.Errorf("ids = %q %q, want name-qualified", r.Pipelines[0].ID, r.Pipelines[1].ID) + } +} + +func TestIngestTriggerTemplateWebhook(t *testing.T) { + const docs = `apiVersion: tekton.dev/v1 +kind: Pipeline +metadata: + name: deploy +spec: + tasks: + - name: apply + taskRef: + name: kubectl +--- +apiVersion: triggers.tekton.dev/v1beta1 +kind: TriggerTemplate +metadata: + name: deploy-template +spec: + resourcetemplates: + - apiVersion: tekton.dev/v1 + kind: PipelineRun + metadata: + generateName: deploy- + spec: + pipelineRef: + name: deploy +--- +apiVersion: triggers.tekton.dev/v1beta1 +kind: EventListener +metadata: + name: listener +spec: + triggers: [] +` + r := Ingest([]File{file("triggers.yaml", docs)}) + p := findPipeline(t, r, "deploy") + if len(p.Triggers) != 1 || p.Triggers[0].Event != "webhook" { + t.Errorf("triggers = %+v, want webhook", p.Triggers) + } + if p.Role != workflowmodel.RoleOther { + t.Errorf("role = %q, want other", p.Role) + } + var details []string + for _, f := range r.Coverage.Findings { + if f.Code == CodeTektonTriggers { + if f.Severity != workflowmodel.SeverityUnsupported { + t.Errorf("tekton-triggers severity = %q", f.Severity) + } + details = append(details, f.Detail) + } + } + if !reflect.DeepEqual(details, []string{"EventListener listener", "TriggerTemplate deploy-template"}) { + t.Errorf("tekton-triggers details = %v", details) + } +} + +func TestIngestCustomTaskAndBundleFindings(t *testing.T) { + const doc = `apiVersion: tekton.dev/v1beta1 +kind: Pipeline +metadata: + name: mixed +spec: + resources: + - name: src + type: git + tasks: + - name: legacy + taskRef: + name: git-clone + bundle: gcr.io/tekton-releases/catalog/upstream/git-clone:0.9 + - name: approve + onError: continue + taskRef: + apiVersion: openshift-pipelines.org/v1alpha1 + kind: ApprovalTask + name: approve +` + r := Ingest([]File{file("mixed.yaml", doc)}) + cases := []struct { + code string + node string + sev workflowmodel.Severity + want string + }{ + {CodeTaskRefBundle, "legacy", workflowmodel.SeverityWarning, "gcr.io/tekton-releases"}, + {CodeCustomTask, "approve", workflowmodel.SeverityUnsupported, "ApprovalTask"}, + {CodeOnError, "approve", workflowmodel.SeverityWarning, "onError"}, + {CodePipelineResources, "", workflowmodel.SeverityUnsupported, "PipelineResources"}, + } + for _, c := range cases { + f := findingByCode(t, r, c.code) + if f.Node != c.node || f.Severity != c.sev || !strings.Contains(f.Detail, c.want) { + t.Errorf("%s = %+v", c.code, f) + } + } +} + +func TestIngestRunAfterCycleAndUnknown(t *testing.T) { + const doc = `apiVersion: tekton.dev/v1 +kind: Pipeline +metadata: + name: broken +spec: + tasks: + - name: a + runAfter: [b] + taskRef: {name: t} + - name: b + runAfter: [a, ghost] + taskRef: {name: t} +` + r := Ingest([]File{file("broken.yaml", doc)}) + p := findPipeline(t, r, "broken") + if p.Levels != 2 { + t.Errorf("levels = %d, want 2 after breaking the cycle", p.Levels) + } + if _, err := workflowmodel.Levels(p.Nodes); err != nil { + t.Errorf("Levels on emitted nodes: %v", err) + } + unresolved := findingByCode(t, r, CodeRunAfterUnresolved) + if unresolved.Node != "b" || !strings.Contains(unresolved.Detail, "ghost") || unresolved.Severity != workflowmodel.SeverityUnknown { + t.Errorf("runafter-unresolved = %+v", unresolved) + } + cycle := findingByCode(t, r, CodeRunAfterCycle) + if cycle.Severity != workflowmodel.SeverityUnknown { + t.Errorf("runafter-cycle = %+v", cycle) + } +} + +func TestIngestMatrix(t *testing.T) { + const doc = `apiVersion: tekton.dev/v1 +kind: Pipeline +metadata: + name: matrix +spec: + tasks: + - name: static + taskRef: {name: t} + matrix: + params: + - name: platform + value: [linux, darwin] + - name: arch + value: [amd64, arm64] + include: + - name: extra + params: + - name: platform + value: windows + - name: dynamic + taskRef: {name: t} + matrix: + params: + - name: platform + value: ["$(params.platforms[*])"] +` + r := Ingest([]File{file("matrix.yaml", doc)}) + p := findPipeline(t, r, "matrix") + static := findNode(t, p, "static").Matrix + want := &workflowmodel.Matrix{Keys: []string{"platform", "arch"}, Include: 1} + if !reflect.DeepEqual(static, want) { + t.Errorf("static matrix = %+v, want %+v", static, want) + } + dynamic := findNode(t, p, "dynamic").Matrix + if dynamic == nil || !dynamic.Dynamic || !reflect.DeepEqual(dynamic.Keys, []string{"platform"}) { + t.Errorf("dynamic matrix = %+v", dynamic) + } +} + +func TestIngestNotTektonAndParseError(t *testing.T) { + cases := []struct { + name string + data string + code string + }{ + {"deployment", "apiVersion: apps/v1\nkind: Deployment\nmetadata:\n name: web\n", CodeNotTekton}, + {"bad yaml", "apiVersion: tekton.dev/v1\nkind: Pipeline\nspec: [unclosed\n", CodeYAML}, + {"empty", "", CodeNotTekton}, + } + for _, c := range cases { + t.Run(c.name, func(t *testing.T) { + r := Ingest([]File{file("x.yaml", c.data)}) + if len(r.Pipelines) != 0 { + t.Errorf("pipelines = %v, want none", pipelineNames(r)) + } + f := findingByCode(t, r, c.code) + if f.Severity != workflowmodel.SeverityUnknown { + t.Errorf("severity = %q", f.Severity) + } + if r.Coverage.Files != 1 || r.Coverage.Parsed != 0 { + t.Errorf("coverage = %+v", r.Coverage) + } + if len(r.Sources) != 1 { + t.Errorf("sources = %d, want 1 even for bad files", len(r.Sources)) + } + }) + } +} + +func TestIngestDeterministic(t *testing.T) { + files := []File{ + file("z/pipeline.yaml", pipelineWithTask), + file("a/build.yaml", pacRun), + file("m/list.yaml", "apiVersion: v1\nkind: List\nitems:\n- apiVersion: tekton.dev/v1\n kind: Pipeline\n metadata: {name: zeta}\n spec: {tasks: [{name: t, taskRef: {name: t}}]}\n- apiVersion: tekton.dev/v1\n kind: Pipeline\n metadata: {name: alpha}\n spec: {tasks: [{name: t, taskRef: {name: t}}]}\n"), + } + first := Ingest(files) + if got := pipelineNames(first); !reflect.DeepEqual(got, []string{"build-on-pr", "alpha", "zeta", "verify"}) { + t.Errorf("pipeline order = %v", got) + } + for i := 0; i < 5; i++ { + again := Ingest(files) + if !reflect.DeepEqual(first, again) { + t.Fatalf("run %d differs:\n%+v\n%+v", i, first, again) + } + } + for i := 1; i < len(first.Coverage.Findings); i++ { + a, b := first.Coverage.Findings[i-1], first.Coverage.Findings[i] + if a.Path > b.Path || (a.Path == b.Path && a.Node > b.Node) || (a.Path == b.Path && a.Node == b.Node && a.Code > b.Code) { + t.Errorf("findings not sorted at %d: %+v > %+v", i, a, b) + } + } +} + +func TestIngestDedupesTriggersAcrossRuns(t *testing.T) { + const pipeline = `apiVersion: tekton.dev/v1 +kind: Pipeline +metadata: + name: shared +spec: + tasks: + - name: t + taskRef: {name: t} +` + run := func(name, branches string) string { + return `apiVersion: tekton.dev/v1 +kind: PipelineRun +metadata: + name: ` + name + ` + annotations: + pipelinesascode.tekton.dev/on-cel-expression: event == "pull_request" && target_branch == "main" + pipelinesascode.tekton.dev/on-event: "[pull_request, push]" + pipelinesascode.tekton.dev/on-target-branch: "` + branches + `" +spec: + pipelineRef: + name: shared +` + } + files := []File{file("pipeline.yaml", pipeline)} + // Five runs with identical annotations, one with the same branches in + // a different order, one with a genuinely different branch set. + for i, b := range []string{"[main, release-1.35]", "[main, release-1.35]", "[main, release-1.35]", "[main, release-1.35]", "[main, release-1.35]", "[release-1.35, main]", "[release-1.36]"} { + files = append(files, file("run-"+string(rune('a'+i))+".yaml", run("r"+string(rune('a'+i)), b))) + } + p := findPipeline(t, Ingest(files), "shared") + want := []workflowmodel.Trigger{ + {Event: "pull_request", Branches: []string{"main", "release-1.35"}}, + {Event: "push", Branches: []string{"main", "release-1.35"}}, + {Event: "cel", Types: []string{`event == "pull_request" && target_branch == "main"`}}, + {Event: "pull_request", Branches: []string{"release-1.36"}}, + {Event: "push", Branches: []string{"release-1.36"}}, + } + if !reflect.DeepEqual(p.Triggers, want) { + t.Errorf("triggers =\n%+v\nwant\n%+v", p.Triggers, want) + } + if p.Role != workflowmodel.RolePRVerify { + t.Errorf("role = %q", p.Role) + } +} + +func TestIngestCELExpression(t *testing.T) { + const doc = `apiVersion: tekton.dev/v1 +kind: PipelineRun +metadata: + name: nightly + annotations: + pipelinesascode.tekton.dev/on-cel-expression: event == "push" && target_branch == "main" +spec: + pipelineSpec: + tasks: + - name: t + taskRef: {name: t} +` + r := Ingest([]File{file("cel.yaml", doc)}) + p := findPipeline(t, r, "nightly") + if p.Role != workflowmodel.RolePushVerify { + t.Errorf("role = %q, want push-verify", p.Role) + } + if len(p.Triggers) != 2 || p.Triggers[0].Event != "cel" || p.Triggers[1].Event != "push" { + t.Errorf("triggers = %+v", p.Triggers) + } +} + +func TestIngestHelpers(t *testing.T) { + if !LooksTekton([]byte("kind: Pipeline\n")) || !LooksTekton([]byte("apiVersion: tekton.dev/v1\n")) || LooksTekton([]byte("kind: Deployment\n")) { + t.Error("LooksTekton misclassified") + } + if !IsTektonDocument(map[string]any{"apiVersion": "triggers.tekton.dev/v1beta1"}) || IsTektonDocument(map[string]any{"apiVersion": "v1"}) || IsTektonDocument(nil) { + t.Error("IsTektonDocument misclassified") + } + got := parsePaCList(`["pull_request", 'push' ,incoming]`) + if !reflect.DeepEqual(got, []string{"pull_request", "push", "incoming"}) { + t.Errorf("parsePaCList = %v", got) + } + docs, err := SplitDocuments([]byte("---\n---\na: 1\n---\n- x\n---\nkind: List\nitems: [{kind: Task}, 3]\n")) + if err != nil || len(docs) != 2 { + t.Errorf("SplitDocuments = %d docs, %v; want 2 mappings", len(docs), err) + } +} + +func TestIngestKonfluxFixture(t *testing.T) { + data, err := os.ReadFile(filepath.Join("testdata", "konflux-build.yaml")) + if err != nil { + t.Fatal(err) + } + r := Ingest([]File{{Path: ".tekton/hive-pull-request.yaml", Data: data}}) + if len(r.Pipelines) != 1 { + t.Fatalf("pipelines = %v", pipelineNames(r)) + } + p := r.Pipelines[0] + if p.Name != "hive-on-pull-request" || p.Role != workflowmodel.RolePRVerify { + t.Errorf("name/role = %q %q", p.Name, p.Role) + } + if !hasEvent(p.Triggers, "pull_request") || !hasEvent(p.Triggers, "cel") { + t.Errorf("triggers = %+v", p.Triggers) + } + wantLevels := map[string]int{ + "init": 0, "clone-repository": 1, "prefetch-dependencies": 2, "build-container": 3, "build-image-index": 4, + "clair-scan": 5, "sast-snyk-check": 5, "clamav-scan": 5, "deprecated-base-image-check": 5, + "show-sbom": 6, "show-summary": 6, + } + if len(p.Nodes) != len(wantLevels) { + t.Errorf("nodes = %d, want %d", len(p.Nodes), len(wantLevels)) + } + for id, lvl := range wantLevels { + if got := findNode(t, p, id).Level; got != lvl { + t.Errorf("%s level = %d, want %d", id, got, lvl) + } + } + // Six task levels plus the finally level. + if p.Levels != 7 { + t.Errorf("levels = %d, want 7", p.Levels) + } + bc := findNode(t, p, "build-container") + if bc.Ref != "bundles:buildah-remote-oci-ta" || bc.Matrix == nil || !bc.Matrix.Dynamic || !reflect.DeepEqual(bc.Matrix.Keys, []string{"PLATFORM"}) { + t.Errorf("build-container = %+v matrix %+v", bc, bc.Matrix) + } + if !reflect.DeepEqual(bc.Needs, []string{"init", "prefetch-dependencies"}) { + t.Errorf("build-container needs = %v (when on init.results plus runAfter)", bc.Needs) + } + if !reflect.DeepEqual(findNode(t, p, "clone-repository").When, []string{"$(tasks.init.results.build) in [true]"}) { + t.Errorf("clone when = %v", findNode(t, p, "clone-repository").When) + } + for _, n := range p.Nodes { + if n.Kind == workflowmodel.NodeFinally && n.Level != p.Levels-1 { + t.Errorf("finally %s at level %d, want last", n.ID, n.Level) + } + } + for _, code := range []string{CodeFinally} { + if !hasFinding(r, code) { + t.Errorf("missing finding %s; have %v", code, findingCodes(r)) + } + } + for _, code := range []string{CodeNoTrigger, CodeRunAfterUnresolved, CodeRunAfterCycle, CodeNotTekton} { + if hasFinding(r, code) { + t.Errorf("unexpected finding %s", code) + } + } +} diff --git a/src/pkg/workflowmodel/tekton/pipeline.go b/src/pkg/workflowmodel/tekton/pipeline.go new file mode 100644 index 0000000000..6e4ec2cbcf --- /dev/null +++ b/src/pkg/workflowmodel/tekton/pipeline.go @@ -0,0 +1,274 @@ +package tekton + +import ( + "fmt" + "regexp" + "sort" + "strings" + + "github.com/hivecommons/hive/pkg/workflowmodel" +) + +// Finding codes the ingest emits. +const ( + CodeYAML = "yaml-parse" + CodeNotTekton = "not-tekton" + CodeUnknownKind = "tekton-kind" + CodeTaskDefinitions = "task-definitions" + CodeTektonTriggers = "tekton-triggers" + CodePipelineRefExternal = "pipelineref-external" + CodeNoTrigger = "no-trigger" + CodeFinally = "finally" + CodeRunAfterUnresolved = "runafter-unresolved" + CodeRunAfterCycle = "runafter-cycle" + CodeDuplicateTask = "duplicate-task" + CodeTaskRefBundle = "taskref-bundle" + CodePipelineResources = "pipeline-resources" + CodeOnError = "on-error" + CodeCustomTask = "custom-task" +) + +// refInline is the Ref of a task whose taskSpec is embedded. +const refInline = "inline" + +// resultRefPattern matches $(tasks..results.) and the other +// $(tasks..…) forms that imply an ordering dependency. +var resultRefPattern = regexp.MustCompile(`\$\(tasks\.([A-Za-z0-9_-]+)\.`) + +// resolverParamNames are the resolver params, in order, that identify what +// a resolver-based taskRef points at. +var resolverParamNames = []string{"name", "url", "pathInRepo"} + +// builder accumulates one pipeline's nodes and findings. +type builder struct { + path string + findings []workflowmodel.Finding +} + +func (b *builder) find(node, code, detail string, sev workflowmodel.Severity) { + b.findings = append(b.findings, workflowmodel.Finding{Path: b.path, Node: node, Code: code, Detail: detail, Severity: sev}) +} + +// buildPipeline turns a pipelineSpec mapping into a Pipeline with nodes +// levelled. Findings about the spec are returned alongside. +func buildPipeline(path, id, name string, spec map[string]any) (workflowmodel.Pipeline, []workflowmodel.Finding) { + b := &builder{path: path} + p := workflowmodel.Pipeline{ID: id, Name: name, System: workflowmodel.SystemTekton, Path: path, Nodes: []workflowmodel.Node{}} + + if _, ok := spec["resources"]; ok { + b.find("", CodePipelineResources, "spec.resources (PipelineResources) are not modelled", workflowmodel.SeverityUnsupported) + } + + seen := map[string]bool{} + var taskNames []string + for _, t := range mapsAt(spec, "tasks") { + n := b.node(t, workflowmodel.NodeTask) + if n.ID == "" { + continue + } + if seen[n.ID] { + b.find(n.ID, CodeDuplicateTask, "task name declared more than once", workflowmodel.SeverityUnknown) + continue + } + seen[n.ID] = true + taskNames = append(taskNames, n.ID) + p.Nodes = append(p.Nodes, n) + } + finally := mapsAt(spec, "finally") + if len(finally) > 0 { + b.find("", CodeFinally, "finally tasks run regardless of outcome", workflowmodel.SeverityWarning) + } + for _, t := range finally { + n := b.node(t, workflowmodel.NodeFinally) + if n.ID == "" { + continue + } + if seen[n.ID] { + b.find(n.ID, CodeDuplicateTask, "task name declared more than once", workflowmodel.SeverityUnknown) + continue + } + seen[n.ID] = true + n.Needs = append([]string(nil), taskNames...) + p.Nodes = append(p.Nodes, n) + } + + b.resolveNeeds(p.Nodes) + levels, err := workflowmodel.Levels(p.Nodes) + if err != nil { + // resolveNeeds removed every unknown need and every back edge, so + // Levels cannot fail; record it rather than drop the pipeline if + // it somehow does. + b.find("", CodeRunAfterCycle, err.Error(), workflowmodel.SeverityUnknown) + } + p.Levels = levels + return p, b.findings +} + +// node maps one spec.tasks or spec.finally entry. +func (b *builder) node(t map[string]any, kind workflowmodel.NodeKind) workflowmodel.Node { + name := stringAt(t, "name") + n := workflowmodel.Node{ID: name, Name: name, Kind: kind} + n.Ref = b.taskRef(name, t) + if _, ok := t["onError"]; ok { + b.find(name, CodeOnError, "onError is not modelled", workflowmodel.SeverityWarning) + } + n.Needs = implicitNeeds(name, stringsAt(t, "runAfter"), t) + n.When = whenClauses(t) + n.Matrix = matrixSummary(mapAt(t, "matrix")) + return n +} + +// taskRef renders the task reference: a name, a resolver with its +// identifying params, or "inline" for an embedded taskSpec. +func (b *builder) taskRef(node string, t map[string]any) string { + ref := mapAt(t, "taskRef") + if ref == nil { + if _, ok := t["taskSpec"]; ok { + return refInline + } + return "" + } + if bundle := stringAt(ref, "bundle"); bundle != "" { + b.find(node, CodeTaskRefBundle, "taskRef.bundle "+bundle+" is deprecated", workflowmodel.SeverityWarning) + } + if apiVersion := stringAt(ref, "apiVersion"); apiVersion != "" { + if k := stringAt(ref, "kind"); k != kindTask && k != kindClusterTask { + b.find(node, CodeCustomTask, fmt.Sprintf("custom task %s %s", apiVersion, k), workflowmodel.SeverityUnsupported) + } + } + if name := stringAt(ref, "name"); name != "" { + return name + } + resolver := stringAt(ref, "resolver") + if resolver == "" { + return "" + } + params := map[string]string{} + for _, p := range mapsAt(ref, "params") { + params[stringAt(p, "name")] = stringAt(p, "value") + } + parts := []string{} + for _, k := range resolverParamNames { + if v := params[k]; v != "" { + parts = append(parts, v) + } + } + if len(parts) == 0 { + return resolver + } + return resolver + ":" + strings.Join(parts, "/") +} + +// implicitNeeds merges runAfter with every task named by a +// $(tasks..…) reference in params, when, workspaces or matrix. +func implicitNeeds(self string, runAfter []string, t map[string]any) []string { + set := map[string]bool{} + for _, r := range runAfter { + if r != "" && r != self { + set[r] = true + } + } + for _, key := range []string{"params", "when", "workspaces", "matrix"} { + walkStrings(t[key], func(s string) { + for _, m := range resultRefPattern.FindAllStringSubmatch(s, -1) { + if m[1] != self { + set[m[1]] = true + } + } + }) + } + if len(set) == 0 { + return nil + } + out := make([]string, 0, len(set)) + for k := range set { + out = append(out, k) + } + sort.Strings(out) + return out +} + +// whenClauses renders each when[] entry as " [values]", +// or "cel: " for CEL entries. +func whenClauses(t map[string]any) []string { + var out []string + for _, w := range mapsAt(t, "when") { + if cel := stringAt(w, "cel"); cel != "" { + out = append(out, "cel: "+cel) + continue + } + out = append(out, fmt.Sprintf("%s %s [%s]", stringAt(w, "input"), stringAt(w, "operator"), strings.Join(stringsAt(w, "values"), ", "))) + } + return out +} + +// matrixSummary summarises matrix.params and matrix.include. +func matrixSummary(m map[string]any) *workflowmodel.Matrix { + if m == nil { + return nil + } + out := &workflowmodel.Matrix{Include: len(sliceAt(m, "include"))} + for _, p := range mapsAt(m, "params") { + if name := stringAt(p, "name"); name != "" { + out.Keys = append(out.Keys, name) + } + } + walkStrings(m, func(s string) { + if strings.Contains(s, "$(") { + out.Dynamic = true + } + }) + return out +} + +// resolveNeeds drops needs that name no node and back edges that would +// close a cycle, recording a finding for each, so Levels succeeds. +func (b *builder) resolveNeeds(nodes []workflowmodel.Node) { + idx := make(map[string]int, len(nodes)) + for i, n := range nodes { + idx[n.ID] = i + } + for i := range nodes { + kept := nodes[i].Needs[:0] + for _, need := range nodes[i].Needs { + if _, ok := idx[need]; ok { + kept = append(kept, need) + continue + } + b.find(nodes[i].ID, CodeRunAfterUnresolved, "depends on unknown task "+need, workflowmodel.SeverityUnknown) + } + nodes[i].Needs = nilIfEmpty(kept) + } + const unvisited, visiting, done = 0, 1, 2 + state := make([]int, len(nodes)) + var visit func(i int) + visit = func(i int) { + state[i] = visiting + kept := nodes[i].Needs[:0] + for _, need := range nodes[i].Needs { + j := idx[need] + switch state[j] { + case visiting: + b.find(nodes[i].ID, CodeRunAfterCycle, "dependency on "+need+" closes a cycle; dropped", workflowmodel.SeverityUnknown) + continue + case unvisited: + visit(j) + } + kept = append(kept, need) + } + nodes[i].Needs = nilIfEmpty(kept) + state[i] = done + } + for i := range nodes { + if state[i] == unvisited { + visit(i) + } + } +} + +func nilIfEmpty(s []string) []string { + if len(s) == 0 { + return nil + } + return s +} diff --git a/src/pkg/workflowmodel/tekton/testdata/konflux-build.yaml b/src/pkg/workflowmodel/tekton/testdata/konflux-build.yaml new file mode 100644 index 0000000000..b8ddc4f583 --- /dev/null +++ b/src/pkg/workflowmodel/tekton/testdata/konflux-build.yaml @@ -0,0 +1,343 @@ +# Modelled on a Konflux / AppStudio build PipelineRun as Pipelines-as-Code +# places it under .tekton/ in a component repository. +apiVersion: tekton.dev/v1 +kind: PipelineRun +metadata: + name: hive-on-pull-request + namespace: hivecommons-tenant + annotations: + build.appstudio.openshift.io/repo: https://github.com/hivecommons/hive?rev={{revision}} + build.appstudio.redhat.com/commit_sha: '{{revision}}' + build.appstudio.redhat.com/pull_request_number: '{{pull_request_number}}' + build.appstudio.redhat.com/target_branch: '{{target_branch}}' + pipelinesascode.tekton.dev/max-keep-runs: "3" + pipelinesascode.tekton.dev/on-cel-expression: event == "pull_request" && target_branch == "main" + pipelinesascode.tekton.dev/on-event: "[pull_request]" + pipelinesascode.tekton.dev/on-target-branch: "[main]" + labels: + appstudio.openshift.io/application: hive + appstudio.openshift.io/component: hive + pipelines.appstudio.openshift.io/type: build +spec: + params: + - name: git-url + value: '{{source_url}}' + - name: revision + value: '{{revision}}' + - name: output-image + value: quay.io/redhat-user-workloads/hivecommons-tenant/hive:on-pr-{{revision}} + - name: image-expires-after + value: 5d + - name: dockerfile + value: Dockerfile + - name: build-platforms + value: + - linux/x86_64 + - linux/arm64 + pipelineSpec: + description: | + Multi-arch container build with SBOM, vulnerability and antivirus scans. + params: + - name: git-url + type: string + - name: revision + type: string + default: "" + - name: output-image + type: string + - name: dockerfile + type: string + default: Dockerfile + - name: image-expires-after + type: string + default: "" + - name: build-platforms + type: array + default: + - linux/x86_64 + - name: prefetch-input + type: string + default: "" + - name: skip-checks + type: string + default: "false" + results: + - name: IMAGE_URL + value: $(tasks.build-image-index.results.IMAGE_URL) + - name: IMAGE_DIGEST + value: $(tasks.build-image-index.results.IMAGE_DIGEST) + - name: CHAINS-GIT_URL + value: $(tasks.clone-repository.results.url) + - name: CHAINS-GIT_COMMIT + value: $(tasks.clone-repository.results.commit) + workspaces: + - name: git-auth + optional: true + - name: netrc + optional: true + tasks: + - name: init + params: + - name: image-url + value: $(params.output-image) + - name: rebuild + value: "false" + - name: skip-checks + value: $(params.skip-checks) + taskRef: + resolver: bundles + params: + - name: name + value: init + - name: bundle + value: quay.io/konflux-ci/tekton-catalog/task-init:0.2@sha256:0000000000000000000000000000000000000000000000000000000000000000 + - name: kind + value: task + - name: clone-repository + params: + - name: url + value: $(params.git-url) + - name: revision + value: $(params.revision) + - name: ociStorage + value: $(params.output-image).git + - name: ociArtifactExpiresAfter + value: $(params.image-expires-after) + runAfter: + - init + taskRef: + resolver: bundles + params: + - name: name + value: git-clone-oci-ta + - name: bundle + value: quay.io/konflux-ci/tekton-catalog/task-git-clone-oci-ta:0.1@sha256:0000000000000000000000000000000000000000000000000000000000000000 + - name: kind + value: task + when: + - input: $(tasks.init.results.build) + operator: in + values: + - "true" + workspaces: + - name: basic-auth + workspace: git-auth + - name: prefetch-dependencies + params: + - name: input + value: $(params.prefetch-input) + - name: SOURCE_ARTIFACT + value: $(tasks.clone-repository.results.SOURCE_ARTIFACT) + - name: ociStorage + value: $(params.output-image).prefetch + - name: ociArtifactExpiresAfter + value: $(params.image-expires-after) + runAfter: + - clone-repository + taskRef: + resolver: bundles + params: + - name: name + value: prefetch-dependencies-oci-ta + - name: bundle + value: quay.io/konflux-ci/tekton-catalog/task-prefetch-dependencies-oci-ta:0.1@sha256:0000000000000000000000000000000000000000000000000000000000000000 + - name: kind + value: task + workspaces: + - name: git-basic-auth + workspace: git-auth + - name: netrc + workspace: netrc + - name: build-container + matrix: + params: + - name: PLATFORM + value: + - $(params.build-platforms) + params: + - name: IMAGE + value: $(params.output-image) + - name: DOCKERFILE + value: $(params.dockerfile) + - name: SOURCE_ARTIFACT + value: $(tasks.prefetch-dependencies.results.SOURCE_ARTIFACT) + - name: CACHI2_ARTIFACT + value: $(tasks.prefetch-dependencies.results.CACHI2_ARTIFACT) + - name: IMAGE_APPEND_PLATFORM + value: "true" + runAfter: + - prefetch-dependencies + taskRef: + resolver: bundles + params: + - name: name + value: buildah-remote-oci-ta + - name: bundle + value: quay.io/konflux-ci/tekton-catalog/task-buildah-remote-oci-ta:0.4@sha256:0000000000000000000000000000000000000000000000000000000000000000 + - name: kind + value: task + when: + - input: $(tasks.init.results.build) + operator: in + values: + - "true" + - name: build-image-index + params: + - name: IMAGE + value: $(params.output-image) + - name: COMMIT_SHA + value: $(tasks.clone-repository.results.commit) + - name: IMAGE_EXPIRES_AFTER + value: $(params.image-expires-after) + - name: ALWAYS_BUILD_INDEX + value: "true" + - name: IMAGES + value: + - $(tasks.build-container.results.IMAGE_REF[*]) + runAfter: + - build-container + taskRef: + resolver: bundles + params: + - name: name + value: build-image-index + - name: bundle + value: quay.io/konflux-ci/tekton-catalog/task-build-image-index:0.1@sha256:0000000000000000000000000000000000000000000000000000000000000000 + - name: kind + value: task + when: + - input: $(tasks.init.results.build) + operator: in + values: + - "true" + - name: clair-scan + params: + - name: image-digest + value: $(tasks.build-image-index.results.IMAGE_DIGEST) + - name: image-url + value: $(tasks.build-image-index.results.IMAGE_URL) + runAfter: + - build-image-index + taskRef: + resolver: bundles + params: + - name: name + value: clair-scan + - name: bundle + value: quay.io/konflux-ci/tekton-catalog/task-clair-scan:0.2@sha256:0000000000000000000000000000000000000000000000000000000000000000 + - name: kind + value: task + when: + - input: $(params.skip-checks) + operator: in + values: + - "false" + - name: sast-snyk-check + params: + - name: image-digest + value: $(tasks.build-image-index.results.IMAGE_DIGEST) + - name: image-url + value: $(tasks.build-image-index.results.IMAGE_URL) + - name: SOURCE_ARTIFACT + value: $(tasks.prefetch-dependencies.results.SOURCE_ARTIFACT) + - name: CACHI2_ARTIFACT + value: $(tasks.prefetch-dependencies.results.CACHI2_ARTIFACT) + runAfter: + - build-image-index + taskRef: + resolver: bundles + params: + - name: name + value: sast-snyk-check-oci-ta + - name: bundle + value: quay.io/konflux-ci/tekton-catalog/task-sast-snyk-check-oci-ta:0.4@sha256:0000000000000000000000000000000000000000000000000000000000000000 + - name: kind + value: task + when: + - input: $(params.skip-checks) + operator: in + values: + - "false" + - name: clamav-scan + params: + - name: image-digest + value: $(tasks.build-image-index.results.IMAGE_DIGEST) + - name: image-url + value: $(tasks.build-image-index.results.IMAGE_URL) + runAfter: + - build-image-index + taskRef: + resolver: bundles + params: + - name: name + value: clamav-scan + - name: bundle + value: quay.io/konflux-ci/tekton-catalog/task-clamav-scan:0.2@sha256:0000000000000000000000000000000000000000000000000000000000000000 + - name: kind + value: task + when: + - input: $(params.skip-checks) + operator: in + values: + - "false" + - name: deprecated-base-image-check + params: + - name: IMAGE_URL + value: $(tasks.build-image-index.results.IMAGE_URL) + - name: IMAGE_DIGEST + value: $(tasks.build-image-index.results.IMAGE_DIGEST) + runAfter: + - build-image-index + taskRef: + resolver: bundles + params: + - name: name + value: deprecated-image-check + - name: bundle + value: quay.io/konflux-ci/tekton-catalog/task-deprecated-image-check:0.5@sha256:0000000000000000000000000000000000000000000000000000000000000000 + - name: kind + value: task + when: + - input: $(params.skip-checks) + operator: in + values: + - "false" + finally: + - name: show-sbom + params: + - name: IMAGE_URL + value: $(tasks.build-image-index.results.IMAGE_URL) + taskRef: + resolver: bundles + params: + - name: name + value: show-sbom + - name: bundle + value: quay.io/konflux-ci/tekton-catalog/task-show-sbom:0.1@sha256:0000000000000000000000000000000000000000000000000000000000000000 + - name: kind + value: task + - name: show-summary + params: + - name: pipelinerun-name + value: $(context.pipelineRun.name) + - name: git-url + value: $(tasks.clone-repository.results.url)?rev=$(tasks.clone-repository.results.commit) + - name: image-url + value: $(params.output-image) + - name: build-task-status + value: $(tasks.build-image-index.status) + taskRef: + resolver: bundles + params: + - name: name + value: summary + - name: bundle + value: quay.io/konflux-ci/tekton-catalog/task-summary:0.2@sha256:0000000000000000000000000000000000000000000000000000000000000000 + - name: kind + value: task + taskRunTemplate: + serviceAccountName: build-pipeline-hive + workspaces: + - name: git-auth + secret: + secretName: '{{ git_auth_secret }}' diff --git a/src/pkg/workflowmodel/tekton/triggers.go b/src/pkg/workflowmodel/tekton/triggers.go new file mode 100644 index 0000000000..8982df4fbc --- /dev/null +++ b/src/pkg/workflowmodel/tekton/triggers.go @@ -0,0 +1,137 @@ +package tekton + +import ( + "strings" + + "github.com/hivecommons/hive/pkg/workflowmodel" +) + +// Pipelines-as-Code annotation and label keys read from PipelineRun +// metadata. +const ( + pacPrefix = "pipelinesascode.tekton.dev/" + pacOnEvent = pacPrefix + "on-event" + pacOnTargetBranch = pacPrefix + "on-target-branch" + pacOnCELExpression = pacPrefix + "on-cel-expression" + pacOnPathChange = pacPrefix + "on-path-change" + pacEventTypeLabel = pacPrefix + "event-type" + tagRefPrefix = "refs/tags/" + tagGlobMarker = "v*" + eventPullRequest = "pull_request" + eventPush = "push" + eventIncoming = "incoming" + eventCEL = "cel" + eventWebhook = "webhook" + celPullRequestFragment = `event == "pull_request"` + celPushFragment = `event == "push"` +) + +// pacTriggers derives triggers from a PipelineRun's Pipelines-as-Code +// annotations and the event-type label PaC stamps on runs. +func pacTriggers(doc map[string]any) []workflowmodel.Trigger { + md := mapAt(doc, "metadata") + ann := stringMap(mapAt(md, "annotations")) + labels := stringMap(mapAt(md, "labels")) + + events := parsePaCList(ann[pacOnEvent]) + if et := labels[pacEventTypeLabel]; et != "" && !contains(events, et) { + events = append(events, et) + } + branches := parsePaCList(ann[pacOnTargetBranch]) + paths := parsePaCList(ann[pacOnPathChange]) + + var out []workflowmodel.Trigger + for _, e := range events { + out = append(out, workflowmodel.Trigger{Event: e, Branches: branches, Paths: paths}) + } + if cel := strings.TrimSpace(ann[pacOnCELExpression]); cel != "" { + out = append(out, workflowmodel.Trigger{Event: eventCEL, Types: []string{cel}}) + for _, f := range celFragments { + if strings.Contains(cel, f.fragment) && !contains(events, f.event) { + events = append(events, f.event) + out = append(out, workflowmodel.Trigger{Event: f.event}) + } + } + } + return out +} + +// celFragments are the CEL sub-expressions that pin an on-cel-expression +// to a concrete event, checked in this order. +var celFragments = []struct{ fragment, event string }{ + {celPullRequestFragment, eventPullRequest}, + {celPushFragment, eventPush}, +} + +// parsePaCList parses the PaC bracketed list syntax: "[a, b]" or "a". +// Brackets and quotes are stripped, entries are trimmed, empties dropped. +func parsePaCList(s string) []string { + s = strings.TrimSpace(s) + s = strings.TrimPrefix(s, "[") + s = strings.TrimSuffix(s, "]") + var out []string + for _, part := range strings.Split(s, ",") { + part = strings.Trim(strings.TrimSpace(part), `"'`) + if part != "" { + out = append(out, part) + } + } + return out +} + +// roleFor derives the role from triggers: pull_request gates PRs; a push +// on a tag-looking branch is a release; a plain push verifies the branch; +// incoming is manual; anything else (webhook only, none) is other. +func roleFor(triggers []workflowmodel.Trigger) workflowmodel.Role { + var push *workflowmodel.Trigger + hasIncoming := false + for i := range triggers { + switch triggers[i].Event { + case eventPullRequest: + return workflowmodel.RolePRVerify + case eventPush: + if push == nil { + push = &triggers[i] + } + case eventIncoming: + hasIncoming = true + } + } + switch { + case push != nil && anyTagBranch(push.Branches): + return workflowmodel.RoleRelease + case push != nil: + return workflowmodel.RolePushVerify + case hasIncoming: + return workflowmodel.RoleManual + } + return workflowmodel.RoleOther +} + +func anyTagBranch(branches []string) bool { + for _, b := range branches { + if strings.HasPrefix(b, tagRefPrefix) || strings.Contains(b, tagGlobMarker) { + return true + } + } + return false +} + +func stringMap(m map[string]any) map[string]string { + out := make(map[string]string, len(m)) + for k, v := range m { + if s, ok := v.(string); ok { + out[k] = s + } + } + return out +} + +func contains(xs []string, x string) bool { + for _, v := range xs { + if v == x { + return true + } + } + return false +} diff --git a/src/pkg/workflowmodel/testdata/corpus/argoproj__argo-cd.golden.json b/src/pkg/workflowmodel/testdata/corpus/argoproj__argo-cd.golden.json new file mode 100644 index 0000000000..6b943e73d3 --- /dev/null +++ b/src/pkg/workflowmodel/testdata/corpus/argoproj__argo-cd.golden.json @@ -0,0 +1,65 @@ +{ + "columns": [ + "Open", + "Actionable", + "Claimed", + "PR in progress", + "CI / Review", + "Merge queue", + "Merged", + "Main CI", + "Release", + "Done" + ], + "roles": { + "pr-verify": 2, + "release": 1 + }, + "pipelines": [ + { + "id": "gha-github-workflows-ci-build", + "role": "pr-verify", + "nodes": 14, + "levels": 4, + "events": [ + "push", + "pull_request" + ] + }, + { + "id": "gha-github-workflows-image", + "role": "pr-verify", + "nodes": 5, + "levels": 3, + "events": [ + "push", + "pull_request" + ] + }, + { + "id": "gha-github-workflows-release", + "role": "release", + "nodes": 8, + "levels": 6, + "events": [ + "push" + ] + } + ], + "ciSubstages": [ + "Detect changed files · Lint shell scripts · Set build variables", + "Build \u0026 cache Go code · Build, test \u0026 lint UI code · Check changes to generated code +6", + "Deploy to staging · Run unit tests for Go packages · Run unit tests with -race for Go packages +2", + "E2E Tests - Composite result · Process \u0026 analyze test artifacts" + ], + "findings": { + "concurrency-expression": 1 + }, + "columnDiff": [ + "column \"CI / Review\" gains 4 CI substage(s) from 2 pipeline(s)", + "column \"Merged\" is no longer the cap", + "column \"Main CI\" added after merge", + "column \"Release\" added after merge", + "column \"Done\" added after merge" + ] +} diff --git a/src/pkg/workflowmodel/testdata/corpus/argoproj__argo-cd/.github/workflows/ci-build.yaml b/src/pkg/workflowmodel/testdata/corpus/argoproj__argo-cd/.github/workflows/ci-build.yaml new file mode 100644 index 0000000000..b2634c6e7c --- /dev/null +++ b/src/pkg/workflowmodel/testdata/corpus/argoproj__argo-cd/.github/workflows/ci-build.yaml @@ -0,0 +1,921 @@ +name: Integration tests +on: + push: + branches: + - 'master' + - 'release-*' + - '!release-1.4' + - '!release-1.5' + pull_request: + branches: + - 'master' + - 'release-*' + +env: + # Golang version to use across CI steps + # renovate: datasource=golang-version packageName=golang + GOLANG_VERSION: '1.27.3' + # a workaround to disable harden runner + STEP_SECURITY_HARDEN_RUNNER: ${{ vars.disable_harden_runner }} + +concurrency: + group: ${{ github.workflow }}-${{ github.ref }} + cancel-in-progress: ${{ github.ref != 'refs/heads/master' }} + +permissions: + contents: read + +jobs: + changes: + name: Detect changed files + runs-on: ubuntu-24.04 + outputs: + backend: ${{ steps.filter.outputs.backend_any_changed }} + frontend: ${{ steps.filter.outputs.frontend_any_changed }} + docs: ${{ steps.filter.outputs.docs_any_changed }} + e2e: ${{ steps.filter.outputs.e2e_any_changed }} + e2e_shards: ${{ steps.shards.outputs.list }} + e2e_shard_count: ${{ steps.shards.outputs.count }} + steps: + - name: Harden the runner (Audit all outbound calls) + if: ${{ vars.disable_harden_runner != 'true' }} + uses: step-security/harden-runner@ccd8616d44fd3846e67624a50d5aad6d37bf2d25 # v2.22.1 + with: + egress-policy: audit + - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 + with: + persist-credentials: false + - uses: tj-actions/changed-files@9426d40962ed5378910ee2e21d5f8c6fcbf2dd96 # v47.0.6 + id: filter + with: + # Any file which is not under docs/, ui/, manifests/, is not VERSION, or is not a markdown + # file is counted as a backend file. + # Version bumps (VERSION + manifests only) skip backend CI (tests, lint, e2e, etc.). + # ui/embed.go and ui/dist/app placeholder files are Go embed inputs; changes must run make build-local. + files_yaml: | + backend: + - '**' + - '!ui/**' + - '!**.md' + - '!**/*.md' + - '!docs/**' + - '!VERSION' + - '!manifests/**' + - 'ui/embed.go' + - 'ui/dist/app/gitkeep' + - 'ui/dist/app/assets/images/resources/.gitkeep' + # The e2e suite is the backend set minus paths that provably cannot change its + # outcome. Keep this list conservative: a false negative here means a regression + # ships untested, while a false positive only costs CI time. + e2e: + - '**' + - '!ui/**' + - '!**.md' + - '!**/*.md' + - '!docs/**' + # Workflows other than this one never execute during the e2e job. This file is + # re-included below because it defines the job itself. + - '!.github/**' + - '.github/workflows/ci-build.yaml' + # Consumed by docs generation and the notifications engine's own unit tests. + - '!notifications_catalog/**' + # Sample manifests for users; never applied by the e2e fixtures. + - '!examples/**' + # Renovate/Sonar/ownership metadata, not build or runtime inputs. + - '!renovate-presets/**' + - '!renovate.json' + - '!sonar-project.properties' + - '!CODEOWNERS' + - '!OWNERS' + - 'ui/embed.go' + - 'ui/dist/app/gitkeep' + - 'ui/dist/app/assets/images/resources/.gitkeep' + frontend: + - 'ui/**' + - Dockerfile + docs: + - 'docs/**' + - name: Compute the e2e shard split + id: shards + env: + SHARD_COUNT: '4' + run: | + set -eu + # awk rather than `seq -s,`: BSD seq appends a trailing separator and GNU does + # not, which would emit invalid JSON on a non-GNU runner. + list=$(awk -v n="${SHARD_COUNT}" 'BEGIN { printf "["; for (i = 1; i <= n; i++) printf "%s%d", (i > 1 ? "," : ""), i; printf "]" }') + echo "count=${SHARD_COUNT}" >> "$GITHUB_OUTPUT" + echo "list=${list}" >> "$GITHUB_OUTPUT" + check-go: + name: Ensure Go modules synchronicity + if: ${{ needs.changes.outputs.backend == 'true' }} + runs-on: ubuntu-24.04 + needs: + - changes + steps: + - name: Harden the runner (Audit all outbound calls) + if: ${{ vars.disable_harden_runner != 'true' }} + uses: step-security/harden-runner@ccd8616d44fd3846e67624a50d5aad6d37bf2d25 # v2.22.1 + with: + egress-policy: audit + - name: Checkout code + uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 + with: + persist-credentials: false + - name: Setup Golang + uses: actions/setup-go@b7ad1dad31e06c5925ef5d2fc7ad053ef454303e # v7.0.0 + with: + go-version: ${{ env.GOLANG_VERSION }} + cache: false + - name: Restore go build and module cache + uses: actions/cache/restore@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0 + with: + path: | + ~/.cache/go-build + ~/go/pkg/mod + key: ${{ runner.os }}-go-build-v2-${{ env.GOLANG_VERSION }}-${{ hashFiles('go.sum', 'gitops-engine/go.sum') }} + restore-keys: | + ${{ runner.os }}-go-build-v2-${{ env.GOLANG_VERSION }}- + - name: Download all Go modules + run: | + go mod download + - name: Check for tidiness of go.mod and go.sum + run: | + go mod tidy + git diff --exit-code -- . + build-go: + name: Build & cache Go code + if: ${{ needs.changes.outputs.backend == 'true' }} + runs-on: ubuntu-24.04 + needs: + - changes + steps: + - name: Harden the runner (Audit all outbound calls) + if: ${{ vars.disable_harden_runner != 'true' }} + uses: step-security/harden-runner@ccd8616d44fd3846e67624a50d5aad6d37bf2d25 # v2.22.1 + with: + egress-policy: audit + - name: Checkout code + uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 + with: + persist-credentials: false + - name: Setup Golang + uses: actions/setup-go@b7ad1dad31e06c5925ef5d2fc7ad053ef454303e # v7.0.0 + with: + go-version: ${{ env.GOLANG_VERSION }} + cache: false + - name: Restore go build and module cache + uses: actions/cache@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0 + with: + path: | + ~/.cache/go-build + ~/go/pkg/mod + key: ${{ runner.os }}-go-build-v2-${{ env.GOLANG_VERSION }}-${{ hashFiles('go.sum', 'gitops-engine/go.sum') }} + restore-keys: | + ${{ runner.os }}-go-build-v2-${{ env.GOLANG_VERSION }}- + - name: Download Go modules + run: | + go mod download + - name: Compile all packages + run: make build-local + + lint-go: + permissions: + contents: read # for actions/checkout to fetch code + pull-requests: read # for golangci/golangci-lint-action to fetch pull requests + name: Lint Go code + if: ${{ needs.changes.outputs.backend == 'true' }} + runs-on: ubuntu-24.04 + needs: + - changes + steps: + - name: Harden the runner (Audit all outbound calls) + if: ${{ vars.disable_harden_runner != 'true' }} + uses: step-security/harden-runner@ccd8616d44fd3846e67624a50d5aad6d37bf2d25 # v2.22.1 + with: + egress-policy: audit + - name: Checkout code + uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 + with: + persist-credentials: false + - name: Setup Golang + uses: actions/setup-go@b7ad1dad31e06c5925ef5d2fc7ad053ef454303e # v7.0.0 + with: + go-version: ${{ env.GOLANG_VERSION }} + cache: false + - name: Restore go build and module cache + uses: actions/cache/restore@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0 + with: + path: | + ~/.cache/go-build + ~/go/pkg/mod + key: ${{ runner.os }}-go-build-v2-${{ env.GOLANG_VERSION }}-${{ hashFiles('go.sum', 'gitops-engine/go.sum') }} + restore-keys: | + ${{ runner.os }}-go-build-v2-${{ env.GOLANG_VERSION }}- + - name: Run golangci-lint + uses: golangci/golangci-lint-action@ba0d7d2ec06a0ea1cb5fa41b2e4a3ab91d21278a # v9.3.0 + with: + # renovate: datasource=go packageName=github.com/golangci/golangci-lint/v2 versioning=regex:^v(?\d+)\.(?\d+)\.(?\d+)?$ + version: v2.15.0 + args: --verbose + + lint-actions: + permissions: + contents: read # for actions/checkout to fetch code + name: Lint GitHub Actions workflows + if: ${{ needs.changes.outputs.backend == 'true' }} + runs-on: ubuntu-24.04 + needs: + - changes + steps: + - name: Harden the runner (Audit all outbound calls) + if: ${{ vars.disable_harden_runner != 'true' }} + uses: step-security/harden-runner@ccd8616d44fd3846e67624a50d5aad6d37bf2d25 # v2.22.1 + with: + egress-policy: audit + - name: Checkout code + uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 + with: + persist-credentials: false + - name: Setup Golang + uses: actions/setup-go@b7ad1dad31e06c5925ef5d2fc7ad053ef454303e # v7.0.0 + with: + go-version: ${{ env.GOLANG_VERSION }} + - name: Install actionlint + run: | + ./hack/install.sh actionlint + echo "$(go env GOPATH)/bin" >> "$GITHUB_PATH" + - name: Run actionlint + run: make actionlint-local + + lint-manifests: + permissions: + contents: read # for actions/checkout to fetch code + name: Validate install manifests + if: ${{ needs.changes.outputs.backend == 'true' }} + runs-on: ubuntu-24.04 + needs: + - changes + steps: + - name: Harden the runner (Audit all outbound calls) + if: ${{ vars.disable_harden_runner != 'true' }} + uses: step-security/harden-runner@ccd8616d44fd3846e67624a50d5aad6d37bf2d25 # v2.22.1 + with: + egress-policy: audit + - name: Checkout code + uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 + with: + persist-credentials: false + - name: Setup Golang + uses: actions/setup-go@b7ad1dad31e06c5925ef5d2fc7ad053ef454303e # v7.0.0 + with: + go-version: ${{ env.GOLANG_VERSION }} + - name: Install kubeconform + run: | + ./hack/install.sh kubeconform + echo "$(go env GOPATH)/bin" >> "$GITHUB_PATH" + - name: Run kubeconform + run: make kubeconform-local + + test-go: + name: Run unit tests for Go packages + if: ${{ needs.changes.outputs.backend == 'true' }} + runs-on: ubuntu-24.04 + needs: + - build-go + - changes + env: + GITHUB_TOKEN: ${{ secrets.E2E_TEST_GITHUB_TOKEN || secrets.GITHUB_TOKEN }} + GITLAB_TOKEN: ${{ secrets.E2E_TEST_GITLAB_TOKEN }} + ARGOCD_TEST_PARALLEL: '2' + steps: + - name: Harden the runner (Audit all outbound calls) + if: ${{ vars.disable_harden_runner != 'true' }} + uses: step-security/harden-runner@ccd8616d44fd3846e67624a50d5aad6d37bf2d25 # v2.22.1 + with: + egress-policy: audit + - name: Create checkout directory + run: mkdir -p ~/go/src/github.com/argoproj + - name: Checkout code + uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 + with: + persist-credentials: false + - name: Create symlink in GOPATH + run: ln -s "$(pwd)" ~/go/src/github.com/argoproj/argo-cd + - name: Setup Golang + uses: actions/setup-go@b7ad1dad31e06c5925ef5d2fc7ad053ef454303e # v7.0.0 + with: + go-version: ${{ env.GOLANG_VERSION }} + cache: false + - name: Install required packages + run: | + sudo apt-get install git -y + - name: Switch to temporal branch so we re-attach head + run: | + git switch -c temporal-pr-branch + git status + - name: Fetch complete history for blame information + run: | + git fetch --prune --no-tags --depth=1 origin +refs/heads/*:refs/remotes/origin/* + - name: Add ~/go/bin to PATH + run: | + echo "/home/runner/go/bin" >> $GITHUB_PATH + - name: Add /usr/local/bin to PATH + run: | + echo "/usr/local/bin" >> $GITHUB_PATH + - name: Restore go build and module cache + uses: actions/cache@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0 + with: + path: | + ~/.cache/go-build + ~/go/pkg/mod + key: ${{ runner.os }}-go-test-v2-${{ env.GOLANG_VERSION }}-${{ hashFiles('go.sum', 'gitops-engine/go.sum') }} + restore-keys: | + ${{ runner.os }}-go-test-v2-${{ env.GOLANG_VERSION }}- + ${{ runner.os }}-go-build-v2-${{ env.GOLANG_VERSION }}- + - name: Install all tools required for building & testing + run: | + make install-test-tools-local + # We install kustomize in the dist directory + - name: Add dist to PATH + run: | + echo "/home/runner/work/argo-cd/argo-cd/dist" >> $GITHUB_PATH + - name: Setup git username and email + run: | + git config --global user.name "John Doe" + git config --global user.email "john.doe@example.com" + - name: Download Go modules + run: | + go mod download + - name: Run all unit tests + run: make test-local + - name: Generate test results artifacts + uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7.0.2 + with: + name: test-results + path: test-results + + test-go-race: + name: Run unit tests with -race for Go packages + if: ${{ needs.changes.outputs.backend == 'true' }} + runs-on: ubuntu-24.04 + needs: + - build-go + - changes + env: + GITHUB_TOKEN: ${{ secrets.E2E_TEST_GITHUB_TOKEN || secrets.GITHUB_TOKEN }} + ARGOCD_TEST_PARALLEL: '2' + GITLAB_TOKEN: ${{ secrets.E2E_TEST_GITLAB_TOKEN }} + steps: + - name: Harden the runner (Audit all outbound calls) + if: ${{ vars.disable_harden_runner != 'true' }} + uses: step-security/harden-runner@ccd8616d44fd3846e67624a50d5aad6d37bf2d25 # v2.22.1 + with: + egress-policy: audit + - name: Create checkout directory + run: mkdir -p ~/go/src/github.com/argoproj + - name: Checkout code + uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 + with: + persist-credentials: false + - name: Create symlink in GOPATH + run: ln -s "$(pwd)" ~/go/src/github.com/argoproj/argo-cd + - name: Setup Golang + uses: actions/setup-go@b7ad1dad31e06c5925ef5d2fc7ad053ef454303e # v7.0.0 + with: + go-version: ${{ env.GOLANG_VERSION }} + cache: false + - name: Install required packages + run: | + sudo apt-get install git -y + - name: Switch to temporal branch so we re-attach head + run: | + git switch -c temporal-pr-branch + git status + - name: Fetch complete history for blame information + run: | + git fetch --prune --no-tags --depth=1 origin +refs/heads/*:refs/remotes/origin/* + - name: Add ~/go/bin to PATH + run: | + echo "/home/runner/go/bin" >> $GITHUB_PATH + - name: Add /usr/local/bin to PATH + run: | + echo "/usr/local/bin" >> $GITHUB_PATH + - name: Restore go build and module cache + uses: actions/cache@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0 + with: + path: | + ~/.cache/go-build + ~/go/pkg/mod + key: ${{ runner.os }}-go-race-v2-${{ env.GOLANG_VERSION }}-${{ hashFiles('go.sum', 'gitops-engine/go.sum') }} + restore-keys: | + ${{ runner.os }}-go-race-v2-${{ env.GOLANG_VERSION }}- + ${{ runner.os }}-go-build-v2-${{ env.GOLANG_VERSION }}- + - name: Install all tools required for building & testing + run: | + make install-test-tools-local + # We install kustomize in the dist directory + - name: Add dist to PATH + run: | + echo "/home/runner/work/argo-cd/argo-cd/dist" >> $GITHUB_PATH + - name: Setup git username and email + run: | + git config --global user.name "John Doe" + git config --global user.email "john.doe@example.com" + - name: Download Go modules + run: | + go mod download + - name: Run all unit tests + run: make test-race-local + - name: Generate test results artifacts + uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7.0.2 + with: + name: race-results + path: test-results/ + + codegen: + name: Check changes to generated code + if: ${{ needs.changes.outputs.backend == 'true' || needs.changes.outputs.docs == 'true'}} + runs-on: ubuntu-24.04 + needs: + - changes + steps: + - name: Harden the runner (Audit all outbound calls) + if: ${{ vars.disable_harden_runner != 'true' }} + uses: step-security/harden-runner@ccd8616d44fd3846e67624a50d5aad6d37bf2d25 # v2.22.1 + with: + egress-policy: audit + - name: Checkout code + uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 + with: + persist-credentials: false + - name: Setup Golang + uses: actions/setup-go@b7ad1dad31e06c5925ef5d2fc7ad053ef454303e # v7.0.0 + with: + go-version: ${{ env.GOLANG_VERSION }} + cache: false + - name: Restore go build and module cache + uses: actions/cache/restore@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0 + with: + path: | + ~/.cache/go-build + ~/go/pkg/mod + key: ${{ runner.os }}-go-build-v2-${{ env.GOLANG_VERSION }}-${{ hashFiles('go.sum', 'gitops-engine/go.sum') }} + restore-keys: | + ${{ runner.os }}-go-build-v2-${{ env.GOLANG_VERSION }}- + - name: Create symlink in GOPATH + # generalizing repo name for forks: ${{ github.event.repository.name }} + run: | + mkdir -p ~/go/src/github.com/argoproj + cp -a ../${GITHUB_EVENT_REPOSITORY_NAME} ~/go/src/github.com/argoproj + env: + GITHUB_EVENT_REPOSITORY_NAME: ${{ github.event.repository.name }} + - name: Add ~/go/bin to PATH + run: | + echo "/home/runner/go/bin" >> $GITHUB_PATH + - name: Add /usr/local/bin to PATH + run: | + echo "/usr/local/bin" >> $GITHUB_PATH + - name: Download dependencies + run: | + go mod download + # generalizing repo name for forks: ${{ github.event.repository.name }} + working-directory: /home/runner/go/src/github.com/argoproj/${{ github.event.repository.name }} + - name: Install toolchain for codegen + run: | + make install-codegen-tools-local + make install-go-tools-local + # generalizing repo name for forks: ${{ github.event.repository.name }} + working-directory: /home/runner/go/src/github.com/argoproj/${{ github.event.repository.name }} + # We install kustomize in the dist directory + - name: Add dist to PATH + run: | + echo "/home/runner/work/argo-cd/argo-cd/dist" >> $GITHUB_PATH + - name: Run codegen + run: | + set -x + GOPATH=$(go env GOPATH) + export GOPATH + git checkout -- go.mod go.sum + make codegen-local + # generalizing repo name for forks: ${{ github.event.repository.name }} + working-directory: /home/runner/go/src/github.com/argoproj/${{ github.event.repository.name }} + - name: Check nothing has changed + run: | + set -xo pipefail + git diff --exit-code -- . ':!go.sum' ':!go.mod' ':!assets/swagger.json' | tee codegen.patch + # generalizing repo name for forks: ${{ github.event.repository.name }} + working-directory: /home/runner/go/src/github.com/argoproj/${{ github.event.repository.name }} + + build-ui: + name: Build, test & lint UI code + # We run UI logic for backend changes so that we have a complete set of coverage documents to send to codecov. + if: ${{ needs.changes.outputs.backend == 'true' || needs.changes.outputs.frontend == 'true' }} + runs-on: ubuntu-24.04 + needs: + - changes + steps: + - name: Harden the runner (Audit all outbound calls) + if: ${{ vars.disable_harden_runner != 'true' }} + uses: step-security/harden-runner@ccd8616d44fd3846e67624a50d5aad6d37bf2d25 # v2.22.1 + with: + egress-policy: audit + - name: Checkout code + uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 + with: + persist-credentials: false + - name: Install pnpm + uses: pnpm/action-setup@ea17c68df8912ef543352723c149a84f56e3d413 # v6.1.0 + with: + package_json_file: ui/package.json + - name: Setup NodeJS + uses: actions/setup-node@949feb2413d6458794dcd2491c4babbbce0c15c1 # v7.1.0 + with: + # renovate: datasource=node-version packageName=node versioning=node + node-version: '26.13.0' + cache: 'pnpm' + cache-dependency-path: '**/pnpm-lock.yaml' + - name: Install node dependencies + run: | + cd ui && pnpm i --frozen-lockfile + - name: Build UI code + run: | + NODE_ENV='test' pnpm test + pnpm build + env: + NODE_ENV: production + NODE_ONLINE_ENV: online + # If we're on the master branch, set the codecov token so that we upload bundle analysis + CODECOV_TOKEN: ${{ github.ref == 'refs/heads/master' && secrets.CODECOV_TOKEN || '' }} + working-directory: ui/ + - name: Run ESLint + run: pnpm lint + working-directory: ui/ + + shellcheck: + name: Lint shell scripts + runs-on: ubuntu-24.04 + steps: + - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 + with: + persist-credentials: false + - run: | + sudo apt-get install shellcheck + # shellcheck disable=SC2046 + shellcheck -e SC2059 -e SC2154 -e SC2034 -e SC2016 -e SC1091 $(find . -type f -name '*.sh' | grep -v './ui/node_modules') | tee sc.log + test ! -s sc.log + + analyze: + name: Process & analyze test artifacts + if: >- + ${{ !cancelled() + && needs.test-go.result == 'success' + && needs.build-ui.result == 'success' + && (needs.test-e2e.result == 'success' || needs.test-e2e.result == 'skipped') }} + runs-on: ubuntu-24.04 + needs: + - test-go + - build-ui + - changes + - test-e2e + env: + sonar_secret: ${{ secrets.SONAR_TOKEN }} + codecov_secret: ${{ secrets.CODECOV_TOKEN }} + steps: + - name: Harden the runner (Audit all outbound calls) + if: ${{ vars.disable_harden_runner != 'true' }} + uses: step-security/harden-runner@ccd8616d44fd3846e67624a50d5aad6d37bf2d25 # v2.22.1 + with: + egress-policy: audit + - name: Checkout code + uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 + with: + fetch-depth: 0 + persist-credentials: false + - name: Get e2e code coverage + if: ${{ needs.test-e2e.result == 'success' }} + uses: actions/download-artifact@9000827ccba6bdab643e8b6fd33ac0654aef8333 # v8.0.2 + with: + pattern: e2e-code-coverage-shard* + path: e2e-code-coverage + - name: Get unit test code coverage + uses: actions/download-artifact@9000827ccba6bdab643e8b6fd33ac0654aef8333 # v8.0.2 + with: + name: test-results + path: test-results + - name: Generate unit test coverage profile + # -pcombine allows merging coverage from different binaries whose counter + # modes differ (main uses set, gitops-engine uses atomic due to -race). + run: | + mkdir -p test-results/merged + go tool covdata merge -pcombine -i=test-results,test-results/gitops-engine -o test-results/merged + go tool covdata textfmt -i=test-results/merged -o test-results/unit-coverage.out + - name: Upload unit test coverage to codecov.io + if: github.repository == 'argoproj/argo-cd' + uses: codecov/codecov-action@303a32d7a59b442fa8d48b6a1cc6825c09c847a5 # v7.1.1 + with: + version: v11.2.8 # https://github.com/getsentry/prevent-cli/releases/tag/v11.2.8 + files: test-results/unit-coverage.out + flags: unit-tests + disable_search: true + fail_ci_if_error: true + env: + CODECOV_TOKEN: ${{ secrets.CODECOV_TOKEN }} + - name: Upload test results to Codecov + # Codecov uploads test results to Codecov.io on upstream master branch. + if: github.repository == 'argoproj/argo-cd' && github.ref == 'refs/heads/master' && github.event_name == 'push' + uses: codecov/codecov-action@303a32d7a59b442fa8d48b6a1cc6825c09c847a5 # v7.1.1 + with: + version: v11.2.8 # https://github.com/getsentry/prevent-cli/releases/tag/v11.2.8 + files: test-results/junit.xml + fail_ci_if_error: true + token: ${{ secrets.CODECOV_TOKEN }} + report_type: test_results + - name: Perform static code analysis using SonarCloud + env: + GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} + SONAR_TOKEN: ${{ secrets.SONAR_TOKEN }} + uses: SonarSource/sonarqube-scan-action@d209202bc7d53ff1cc128f7f907dac145c9d6ae9 # v8.3.0 + if: env.sonar_secret != '' + - name: Generate e2e test coverage profile + if: ${{ needs.test-e2e.result == 'success' }} + # Runs after the unit coverage upload and the SonarCloud scan, so an incomplete e2e + # profile fails the job without skipping them. + # We generate coverage reports for all Argo CD components, but only the applicationset-controller, + # app-controller, repo-server, and commit-server report contain coverage data. The other components currently + # don't shut down gracefully, so no coverage data is produced. Once those components are fixed, we can add + # references to their coverage output directories. + # Each e2e shard uploads its own artifact, so the per-component directories appear + # once per shard and have to be merged before they can be converted to text. + # A shard that uploaded nothing only warns on its upload step, and the pattern + # download then skips it, so check that every shard contributed. + env: + E2E_SHARD_COUNT: ${{ needs.changes.outputs.e2e_shard_count }} + run: | + set -o pipefail + dirs=$(find e2e-code-coverage -mindepth 2 -maxdepth 2 -type d \ + \( -name applicationset-controller -o -name repo-server -o -name app-controller -o -name commit-server \) \ + | sort | paste -sd, -) + if [ -z "$dirs" ]; then + echo "no e2e coverage directories found under e2e-code-coverage/" >&2 + exit 1 + fi + shards=$(echo "$dirs" | tr ',' '\n' | xargs -n1 dirname | sort -u | wc -l | tr -d ' ') + if [ "$shards" -ne "$E2E_SHARD_COUNT" ]; then + echo "e2e coverage came from $shards of $E2E_SHARD_COUNT shards; refusing to upload a partial profile" >&2 + exit 1 + fi + echo "merging: $dirs" + mkdir -p e2e-code-coverage/merged + go tool covdata merge -pcombine -i="$dirs" -o e2e-code-coverage/merged + go tool covdata textfmt -i=e2e-code-coverage/merged -o test-results/e2e-coverage.out + - name: Upload e2e test coverage to codecov.io + if: ${{ needs.test-e2e.result == 'success' && github.repository == 'argoproj/argo-cd' }} + uses: codecov/codecov-action@303a32d7a59b442fa8d48b6a1cc6825c09c847a5 # v7.1.1 + with: + version: v11.2.8 # https://github.com/getsentry/prevent-cli/releases/tag/v11.2.8 + files: test-results/e2e-coverage.out + flags: e2e + disable_search: true + fail_ci_if_error: true + env: + CODECOV_TOKEN: ${{ secrets.CODECOV_TOKEN }} + test-e2e: + # Without an explicit name GitHub builds one from every matrix value, which leaks + # the meaningless `latest` boolean into all 16 check names, e.g. "(v1.34.11, false, 1)". + name: e2e (k8s ${{ matrix.k3s.version }}, shard ${{ matrix.shard }}) + if: ${{ needs.changes.outputs.e2e == 'true' }} + runs-on: ${{ github.repository == 'argoproj/argo-cd' && 'oracle-vm-16cpu-64gb-x86-64' || 'ubuntu-24.04' }} + strategy: + fail-fast: false + matrix: + # latest: true means that this version must upload the coverage report to codecov.io + # We designate the latest version because we only collect code coverage for that version. + k3s: + - version: v1.37.0 + latest: true + - version: v1.36.4 + latest: false + - version: v1.35.8 + latest: false + - version: v1.34.11 + latest: false + # The e2e suite is one serial package; splitting it across jobs is the only way to + # cut its wall-clock time. Legs and count both come from the changes job. + shard: ${{ fromJSON(needs.changes.outputs.e2e_shards) }} + needs: + - build-go + - changes + env: + ARGOCD_FAKE_IN_CLUSTER: 'true' + ARGOCD_E2E_K3S: 'true' + ARGOCD_IN_CI: 'true' + ARGOCD_E2E_APISERVER_PORT: '8088' + ARGOCD_E2E_SHARD: ${{ matrix.shard }} + ARGOCD_E2E_SHARD_COUNT: ${{ needs.changes.outputs.e2e_shard_count }} + ARGOCD_APPLICATION_NAMESPACES: 'argocd-e2e-external,argocd-e2e-external-2' + ARGOCD_SERVER: '127.0.0.1:8088' + GITHUB_TOKEN: ${{ secrets.E2E_TEST_GITHUB_TOKEN || secrets.GITHUB_TOKEN }} + GITLAB_TOKEN: ${{ secrets.E2E_TEST_GITLAB_TOKEN }} + steps: + - name: Harden the runner (Audit all outbound calls) + if: ${{ vars.disable_harden_runner != 'true' }} + uses: step-security/harden-runner@ccd8616d44fd3846e67624a50d5aad6d37bf2d25 # v2.22.1 + with: + egress-policy: audit + - name: Free Disk Space (Ubuntu) + # The upstream repository runs this job on the Oracle runners selected above, which + # start with well over 200 GB free. Removing ~20 GB of unused toolchains there costs + # close to four minutes per job for nothing. GitHub-hosted runners, which forks use, + # have far less headroom, so the cleanup still runs for them. + if: ${{ github.repository != 'argoproj/argo-cd' }} + uses: jlumbroso/free-disk-space@ceedf095f4ec1a097402bc6bd80831f2e1a6fde6 # v2.0.0 + with: + large-packages: false + docker-images: false + swap-storage: false + tool-cache: false + - name: Checkout code + uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 + with: + persist-credentials: false + - name: Setup Golang + uses: actions/setup-go@b7ad1dad31e06c5925ef5d2fc7ad053ef454303e # v7.0.0 + with: + go-version: ${{ env.GOLANG_VERSION }} + cache: false + - name: Set GOPATH + run: | + echo "GOPATH=$HOME/go" >> $GITHUB_ENV + - name: Setup NodeJS + uses: actions/setup-node@949feb2413d6458794dcd2491c4babbbce0c15c1 # v7.1.0 + with: + # renovate: datasource=node-version packageName=node versioning=node + node-version: '26.13.0' + - name: Install pnpm + uses: pnpm/action-setup@ea17c68df8912ef543352723c149a84f56e3d413 # v6.1.0 + with: + package_json_file: ui/package.json + - name: GH actions workaround - Kill XSP4 process + run: | + sudo pkill mono || true + - name: Install K3S + env: + INSTALL_K3S_VERSION: ${{ matrix.k3s.version }}+k3s1 + run: | + set -x + curl -sfL https://get.k3s.io | sh - + sudo chmod -R a+rw /etc/rancher/k3s + sudo mkdir -p $HOME/.kube && sudo chown -R "$(whoami)" $HOME/.kube + # shellcheck disable=SC2024 + sudo k3s kubectl config view --raw > $HOME/.kube/config + sudo chown "$(whoami)" $HOME/.kube/config + sudo chmod go-r $HOME/.kube/config + kubectl version + - name: Restore go build and module cache + uses: actions/cache/restore@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0 + with: + path: | + ~/.cache/go-build + ~/go/pkg/mod + key: ${{ runner.os }}-go-build-v2-${{ env.GOLANG_VERSION }}-${{ hashFiles('go.sum', 'gitops-engine/go.sum') }} + restore-keys: | + ${{ runner.os }}-go-build-v2-${{ env.GOLANG_VERSION }}- + - name: Add ~/go/bin to PATH + run: | + echo "$HOME/go/bin" >> $GITHUB_PATH + - name: Add /usr/local/bin to PATH + run: | + echo "/usr/local/bin" >> $GITHUB_PATH + - name: Add ./dist to PATH + run: | + echo "$(pwd)/dist" >> $GITHUB_PATH + - name: Download Go modules + run: | + go mod download + - name: Install goreman + run: | + go install github.com/mattn/goreman@v0.3.17 + - name: Install all tools required for building & testing + run: | + make install-test-tools-local + - name: Setup git username and email + run: | + git config --global user.name "John Doe" + git config --global user.email "john.doe@example.com" + - name: Pull Docker image required for tests + run: | + # renovate: datasource=github-releases depName=dexidp/dex + docker pull ghcr.io/dexidp/dex:v2.46.1 + docker pull argoproj/argo-cd-ci-builder:v1.0.0 + # renovate: datasource=docker packageName=docker.io/library/redis + docker pull redis:8.10.3-alpine + - name: Create target directory for binaries in the build-process + run: | + mkdir -p dist + chown "$(whoami)" dist + - name: Run E2E server and wait for it being available + timeout-minutes: 30 + run: | + set -x + # Something is weird in GH runners -- there's a phantom listener for + # port 8080 which is not visible in netstat -tulpen, but still there + # with a HTTP listener. We have API server listening on port 8088 + # instead. + make start-e2e-local COVERAGE_ENABLED=true 2>&1 | sed -r "s/[[:cntrl:]]\[[0-9]{1,3}m//g" > /tmp/e2e-server.log & + count=1 + until curl -f http://127.0.0.1:8088/healthz; do + sleep 10; + if test $count -ge 180; then + echo "Timeout" + exit 1 + fi + count=$((count+1)) + done + - name: Run E2E testsuite + id: e2e + run: | + set -x + make test-e2e-local + # gotestsum retries failures (--rerun-fails), so a green job can still hide a + # flake. Flag it so the server logs below are kept for diagnosis. + if [ -s rerunreport.txt ]; then + echo "reran=true" >> "$GITHUB_OUTPUT" + echo "::warning::e2e tests were retried; see the e2e-server log artifact" + cat rerunreport.txt + fi + goreman run stop-all || echo "goreman trouble" + sleep 30 + - name: Upload e2e coverage report + uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7.0.2 + with: + name: e2e-code-coverage-shard${{ matrix.shard }} + path: /tmp/coverage + if: ${{ matrix.k3s.latest }} + - name: Upload e2e test timings + # Feeds the shard coverage check in the composite job below and the e2e-shard-weights.yaml + # workflow; keep the artifact name in sync with both and hack/e2e-shard-weights-download.sh. + # Only the "latest" k3s version is needed: one full pass over the suite. + uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7.0.2 + with: + name: e2e-test-timings-shard${{ matrix.shard }} + path: test-results/junit.xml + if: ${{ matrix.k3s.latest }} + - name: Upload e2e-server logs + # Also kept when a test only failed on its first attempt: without this the logs + # for a retried-then-passed flake are discarded and it cannot be diagnosed. + uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7.0.2 + with: + name: e2e-server-k8s${{ matrix.k3s.version }}-shard${{ matrix.shard }}.log + path: | + /tmp/e2e-server.log + rerunreport.txt + if: ${{ failure() || steps.e2e.outputs.reran == 'true' }} + + # workaround for status checks -- check this one job instead of each individual E2E job in the matrix + # this allows us to skip the entire matrix when it doesn't need to run while still having accurate status checks + # see: + # https://github.com/argoproj/argo-workflows/pull/12006 + # https://github.com/orgs/community/discussions/9141#discussioncomment-2296809 + # https://github.com/orgs/community/discussions/26822#discussioncomment-3305794 + test-e2e-composite-result: + name: E2E Tests - Composite result + if: ${{ always() }} + needs: + - test-e2e + - changes + runs-on: ubuntu-24.04 + steps: + - name: Harden the runner (Audit all outbound calls) + if: ${{ vars.disable_harden_runner != 'true' }} + uses: step-security/harden-runner@ccd8616d44fd3846e67624a50d5aad6d37bf2d25 # v2.22.1 + with: + egress-policy: audit + - run: | + result="${NEEDS_TEST_E2E_RESULT}" + # mark as successful even if skipped + if [[ $result == "success" || $result == "skipped" ]]; then + exit 0 + else + exit 1 + fi + env: + NEEDS_TEST_E2E_RESULT: ${{ needs.test-e2e.result }} + # Runs in this job because it is the required one: a test that no shard selected has + # to block the merge, not only turn a non-required job red. + - name: Checkout code + if: ${{ needs.test-e2e.result == 'success' }} + uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 + with: + persist-credentials: false + - name: Get e2e test reports + if: ${{ needs.test-e2e.result == 'success' }} + uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1 + with: + pattern: e2e-test-timings-shard* + path: e2e-test-reports + - name: Check that every e2e test ran in a shard + if: ${{ needs.test-e2e.result == 'success' }} + env: + E2E_SHARD_COUNT: ${{ needs.changes.outputs.e2e_shard_count }} + run: ./hack/e2e-check-shard-coverage.sh "$E2E_SHARD_COUNT" e2e-test-reports/*/junit.xml diff --git a/src/pkg/workflowmodel/testdata/corpus/argoproj__argo-cd/.github/workflows/image.yaml b/src/pkg/workflowmodel/testdata/corpus/argoproj__argo-cd/.github/workflows/image.yaml new file mode 100644 index 0000000000..f69e0abfdb --- /dev/null +++ b/src/pkg/workflowmodel/testdata/corpus/argoproj__argo-cd/.github/workflows/image.yaml @@ -0,0 +1,177 @@ +name: Image + +on: + push: + branches: + - master + pull_request: + branches: + - master + types: [labeled, unlabeled, opened, synchronize, reopened] + +concurrency: + group: ${{ github.workflow }}-${{ github.ref }} + cancel-in-progress: true + +permissions: {} + +env: + # a workaround to disable harden runner + STEP_SECURITY_HARDEN_RUNNER: ${{ vars.disable_harden_runner }} + +jobs: + set-vars: + name: Set build variables + permissions: + contents: read + # Always run to calculate variables - other jobs check outputs + runs-on: ubuntu-24.04 + outputs: + image-tag: ${{ steps.image.outputs.tag}} + platforms: ${{ steps.platforms.outputs.platforms }} + image_namespace: ${{ steps.image.outputs.image_namespace }} + image_repository: ${{ steps.image.outputs.image_repository }} + quay_image_name: ${{ steps.image.outputs.quay_image_name }} + ghcr_image_name: ${{ steps.image.outputs.ghcr_image_name }} + ghcr_provenance_image: ${{ steps.image.outputs.ghcr_provenance_image }} + allow_ghcr_publish: ${{ steps.image.outputs.allow_ghcr_publish }} + steps: + - name: Harden the runner (Audit all outbound calls) + if: ${{ vars.disable_harden_runner != 'true' }} + uses: step-security/harden-runner@ccd8616d44fd3846e67624a50d5aad6d37bf2d25 # v2.22.1 + with: + egress-policy: audit + + - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 + with: + persist-credentials: false + + - name: Set image tag and names + run: | + # Calculate image tag + TAG="$(cat ./VERSION)-${GITHUB_SHA::8}" + echo "tag=$TAG" >> $GITHUB_OUTPUT + + # Calculate image names with defaults + IMAGE_NAMESPACE="${VARS_IMAGE_NAMESPACE:-argoproj}" + IMAGE_REPOSITORY="${VARS_IMAGE_REPOSITORY:-argocd}" + GHCR_NAMESPACE="${VARS_GHCR_NAMESPACE:-$GITHUB_REPOSITORY}" + GHCR_REPOSITORY="${VARS_GHCR_REPOSITORY:-argocd}" + + echo "image_namespace=$IMAGE_NAMESPACE" >> $GITHUB_OUTPUT + echo "image_repository=$IMAGE_REPOSITORY" >> $GITHUB_OUTPUT + + # Construct image name + echo "quay_image_name=quay.io/$IMAGE_NAMESPACE/$IMAGE_REPOSITORY:latest" >> $GITHUB_OUTPUT + + ALLOW_GHCR_PUBLISH=false + if [[ "$GITHUB_REPOSITORY" == "argoproj/argo-cd" || "$GHCR_NAMESPACE" != argoproj/* ]]; then + ALLOW_GHCR_PUBLISH=true + echo "ghcr_image_name=ghcr.io/$GHCR_NAMESPACE/$GHCR_REPOSITORY:$TAG" >> $GITHUB_OUTPUT + echo "ghcr_provenance_image=ghcr.io/$GHCR_NAMESPACE/$GHCR_REPOSITORY" >> $GITHUB_OUTPUT + else + echo "GhCR publish skipped: refusing to push to namespace '$GHCR_NAMESPACE'. Please override GHCR_* for forks." >&2 + echo "ghcr_image_name=" >> $GITHUB_OUTPUT + echo "ghcr_provenance_image=" >> $GITHUB_OUTPUT + fi + echo "allow_ghcr_publish=$ALLOW_GHCR_PUBLISH" >> $GITHUB_OUTPUT + id: image + env: + VARS_IMAGE_NAMESPACE: ${{ vars.IMAGE_NAMESPACE }} + VARS_IMAGE_REPOSITORY: ${{ vars.IMAGE_REPOSITORY }} + VARS_GHCR_NAMESPACE: ${{ vars.GHCR_NAMESPACE }} + VARS_GHCR_REPOSITORY: ${{ vars.GHCR_REPOSITORY }} + + - name: Determine image platforms to use + id: platforms + run: | + IMAGE_PLATFORMS=linux/amd64 + if [[ "${GITHUB_EVENT_NAME}" == "push" || "${HAS_MULTI_IMAGE_LABEL}" == "true" ]] + then + IMAGE_PLATFORMS=linux/amd64,linux/arm64,linux/s390x,linux/ppc64le + fi + echo "Building image for platforms: $IMAGE_PLATFORMS" + echo "platforms=$IMAGE_PLATFORMS" >> $GITHUB_OUTPUT + env: + HAS_MULTI_IMAGE_LABEL: ${{ contains(github.event.pull_request.labels.*.name, 'test-multi-image') }} + + build-only: + needs: [set-vars] + permissions: + contents: read + packages: write # for pushing packages to GHCR, which is used by cd.apps.argoproj.io to avoid polluting Quay with tags + id-token: write # for creating OIDC tokens for signing. + if: ${{ (github.repository == 'argoproj/argo-cd' || needs.set-vars.outputs.image_namespace != 'argoproj') && github.event_name != 'push' }} + uses: ./.github/workflows/image-reuse.yaml + with: + # Note: cannot use env variables to set go-version (https://docs.github.com/en/actions/using-workflows/reusing-workflows#limitations) + # renovate: datasource=golang-version packageName=golang + go-version: 1.27.3 + platforms: ${{ needs.set-vars.outputs.platforms }} + push: false + + build-and-publish: + needs: [set-vars] + permissions: + contents: read + packages: write # for pushing packages to GHCR, which is used by cd.apps.argoproj.io to avoid polluting Quay with tags + id-token: write # for creating OIDC tokens for signing. + if: ${{ (github.repository == 'argoproj/argo-cd' || needs.set-vars.outputs.image_namespace != 'argoproj') && github.event_name == 'push' }} + uses: ./.github/workflows/image-reuse.yaml + with: + quay_image_name: ${{ needs.set-vars.outputs.quay_image_name }} + ghcr_image_name: ${{ needs.set-vars.outputs.ghcr_image_name }} + # Note: cannot use env variables to set go-version (https://docs.github.com/en/actions/using-workflows/reusing-workflows#limitations) + # renovate: datasource=golang-version packageName=golang + go-version: 1.27.3 + platforms: ${{ needs.set-vars.outputs.platforms }} + push: true + secrets: + quay_username: ${{ secrets.RELEASE_QUAY_USERNAME }} + quay_password: ${{ secrets.RELEASE_QUAY_TOKEN }} + ghcr_username: ${{ github.actor }} + ghcr_password: ${{ secrets.GITHUB_TOKEN }} + + build-and-publish-provenance: # Push attestations to GHCR, latest image is polluting quay.io + needs: + - set-vars + - build-and-publish + permissions: + actions: read # for detecting the Github Actions environment. + id-token: write # for creating OIDC tokens for signing. + packages: write # for uploading attestations. (https://github.com/slsa-framework/slsa-github-generator/blob/main/internal/builders/container/README.md#known-issues) + if: ${{ (github.repository == 'argoproj/argo-cd' || needs.set-vars.outputs.image_namespace != 'argoproj') && github.event_name == 'push' && needs.set-vars.outputs.allow_ghcr_publish == 'true'}} + # Must be referenced by a tag. https://github.com/slsa-framework/slsa-github-generator/blob/main/internal/builders/container/README.md#referencing-the-slsa-generator + uses: slsa-framework/slsa-github-generator/.github/workflows/generator_container_slsa3.yml@v2.1.0 + with: + image: ${{ needs.set-vars.outputs.ghcr_provenance_image }} + digest: ${{ needs.build-and-publish.outputs.image-digest }} + registry-username: ${{ github.actor }} + secrets: + registry-password: ${{ secrets.GITHUB_TOKEN }} + + Deploy: + name: Deploy to staging + needs: + - build-and-publish + - set-vars + permissions: + contents: write # for git to push upgrade commit if not already deployed + packages: write # for pushing packages to GHCR, which is used by cd.apps.argoproj.io to avoid polluting Quay with tags + if: ${{ github.repository == 'argoproj/argo-cd' && github.event_name == 'push' }} + runs-on: ubuntu-24.04 + steps: + - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 + with: + persist-credentials: false + - run: git clone "https://$TOKEN@github.com/argoproj/argoproj-deployments" + env: + TOKEN: ${{ secrets.TOKEN }} + - run: | + docker run -u "$(id -u):$(id -g)" -v "$(pwd):/src" -w /src --rm -t ghcr.io/argoproj/argo-cd/argocd:${NEEDS_SET_VARS_OUTPUTS_IMAGE_TAG} kustomize edit set image quay.io/argoproj/argocd=ghcr.io/argoproj/argo-cd/argocd:${NEEDS_SET_VARS_OUTPUTS_IMAGE_TAG} + git config --global user.email 'ci@argoproj.com' + git config --global user.name 'CI' + git diff --exit-code && echo 'Already deployed' || (git commit -am "Upgrade argocd to ${NEEDS_SET_VARS_OUTPUTS_IMAGE_TAG}" && git push) + working-directory: argoproj-deployments/argocd + env: + NEEDS_SET_VARS_OUTPUTS_IMAGE_TAG: ${{ needs.set-vars.outputs.image-tag }} diff --git a/src/pkg/workflowmodel/testdata/corpus/argoproj__argo-cd/.github/workflows/release.yaml b/src/pkg/workflowmodel/testdata/corpus/argoproj__argo-cd/.github/workflows/release.yaml new file mode 100644 index 0000000000..3090bf94ac --- /dev/null +++ b/src/pkg/workflowmodel/testdata/corpus/argoproj__argo-cd/.github/workflows/release.yaml @@ -0,0 +1,424 @@ +name: Publish ArgoCD Release +on: + push: + tags: + - 'v*' + - '!v2.4*' + - '!v2.5*' + - '!v2.6*' + +permissions: {} + +env: + # renovate: datasource=golang-version packageName=golang + GOLANG_VERSION: '1.27.3' # Note: go-version must also be set in job argocd-image.with.go-version + # a workaround to disable harden runner + STEP_SECURITY_HARDEN_RUNNER: ${{ vars.disable_harden_runner }} + +jobs: + argocd-image: + needs: [setup-variables] + permissions: + contents: read + id-token: write # for creating OIDC tokens for signing. + packages: write # used to push images to `ghcr.io` if used. + if: github.repository == 'argoproj/argo-cd' || needs.setup-variables.outputs.allow_fork_release == 'true' + uses: ./.github/workflows/image-reuse.yaml + with: + quay_image_name: ${{ needs.setup-variables.outputs.quay_image_name }} + # Note: cannot use env variables to set go-version (https://docs.github.com/en/actions/using-workflows/reusing-workflows#limitations) + # renovate: datasource=golang-version packageName=golang + go-version: 1.27.3 + platforms: linux/amd64,linux/arm64,linux/s390x,linux/ppc64le + push: true + secrets: + quay_username: ${{ secrets.RELEASE_QUAY_USERNAME }} + quay_password: ${{ secrets.RELEASE_QUAY_TOKEN }} + + setup-variables: + name: Setup Release Variables + if: github.repository == 'argoproj/argo-cd' || (github.repository_owner != 'argoproj' && vars.ENABLE_FORK_RELEASES == 'true' && vars.IMAGE_NAMESPACE && vars.IMAGE_NAMESPACE != 'argoproj') + runs-on: ubuntu-24.04 + outputs: + is_pre_release: ${{ steps.var.outputs.is_pre_release }} + is_latest_release: ${{ steps.var.outputs.is_latest_release }} + enable_fork_releases: ${{ steps.var.outputs.enable_fork_releases }} + image_namespace: ${{ steps.var.outputs.image_namespace }} + image_repository: ${{ steps.var.outputs.image_repository }} + quay_image_name: ${{ steps.var.outputs.quay_image_name }} + provenance_image: ${{ steps.var.outputs.provenance_image }} + allow_fork_release: ${{ steps.var.outputs.allow_fork_release }} + steps: + - name: Harden the runner (Audit all outbound calls) + if: ${{ vars.disable_harden_runner != 'true' }} + uses: step-security/harden-runner@ccd8616d44fd3846e67624a50d5aad6d37bf2d25 # v2.22.1 + with: + egress-policy: audit + - name: Checkout code + uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 + with: + fetch-depth: 0 + token: ${{ secrets.GITHUB_TOKEN }} + persist-credentials: false + - name: Setup variables + id: var + run: | + set -xue + # Fetch all tag information + git fetch --prune --tags --force + + LATEST_RELEASE_TAG=$(git -c 'versionsort.suffix=-rc' tag --list --sort=version:refname | grep -v '-' | tail -n1) + + PRE_RELEASE=false + # Check if latest tag is a pre-release + if echo ${GITHUB_REF_NAME} | grep -E -- '-rc[0-9]+$';then + PRE_RELEASE=true + fi + + IS_LATEST=false + # Ensure latest release tag matches github.ref_name + if [[ $LATEST_RELEASE_TAG == "${GITHUB_REF_NAME}" ]];then + IS_LATEST=true + fi + echo "is_pre_release=$PRE_RELEASE" >> $GITHUB_OUTPUT + echo "is_latest_release=$IS_LATEST" >> $GITHUB_OUTPUT + + # Calculate configuration with defaults + ENABLE_FORK_RELEASES="${VARS_ENABLE_FORK_RELEASES:-false}" + IMAGE_NAMESPACE="${VARS_IMAGE_NAMESPACE:-argoproj}" + IMAGE_REPOSITORY="${VARS_IMAGE_REPOSITORY:-argocd}" + + echo "enable_fork_releases=$ENABLE_FORK_RELEASES" >> $GITHUB_OUTPUT + + echo "image_namespace=$IMAGE_NAMESPACE" >> $GITHUB_OUTPUT + echo "image_repository=$IMAGE_REPOSITORY" >> $GITHUB_OUTPUT + echo "quay_image_name=quay.io/$IMAGE_NAMESPACE/$IMAGE_REPOSITORY:${GITHUB_REF_NAME}" >> $GITHUB_OUTPUT + echo "provenance_image=quay.io/$IMAGE_NAMESPACE/$IMAGE_REPOSITORY" >> $GITHUB_OUTPUT + + ALLOW_FORK_RELEASE=false + if [[ "$GITHUB_REPOSITORY_OWNER" != "argoproj" && "$ENABLE_FORK_RELEASES" == "true" && "$IMAGE_NAMESPACE" != "argoproj" && "${GITHUB_REF}" == refs/tags/* ]]; then + ALLOW_FORK_RELEASE=true + fi + echo "allow_fork_release=$ALLOW_FORK_RELEASE" >> $GITHUB_OUTPUT + env: + VARS_ENABLE_FORK_RELEASES: ${{ vars.ENABLE_FORK_RELEASES }} + VARS_IMAGE_NAMESPACE: ${{ vars.IMAGE_NAMESPACE }} + VARS_IMAGE_REPOSITORY: ${{ vars.IMAGE_REPOSITORY }} + + argocd-image-provenance: + needs: [setup-variables, argocd-image] + permissions: + actions: read # for detecting the Github Actions environment. + id-token: write # for creating OIDC tokens for signing. + packages: write # for uploading attestations. (https://github.com/slsa-framework/slsa-github-generator/blob/main/internal/builders/container/README.md#known-issues) + # Must be referenced by a tag. https://github.com/slsa-framework/slsa-github-generator/blob/main/internal/builders/container/README.md#referencing-the-slsa-generator + if: github.repository == 'argoproj/argo-cd' || needs.setup-variables.outputs.allow_fork_release == 'true' + uses: slsa-framework/slsa-github-generator/.github/workflows/generator_container_slsa3.yml@v2.1.0 + with: + image: ${{ needs.setup-variables.outputs.provenance_image }} + digest: ${{ needs.argocd-image.outputs.image-digest }} + secrets: + registry-username: ${{ secrets.RELEASE_QUAY_USERNAME }} + registry-password: ${{ secrets.RELEASE_QUAY_TOKEN }} + + goreleaser: + name: Run GoReleaser + needs: + - setup-variables + - argocd-image + - argocd-image-provenance + permissions: + contents: write # used for uploading assets + if: github.repository == 'argoproj/argo-cd' || needs.setup-variables.outputs.allow_fork_release == 'true' + runs-on: ubuntu-24.04 + env: + GORELEASER_MAKE_LATEST: ${{ needs.setup-variables.outputs.is_latest_release }} + outputs: + hashes: ${{ steps.hash.outputs.hashes }} + steps: + - name: Checkout code + uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 + with: + fetch-depth: 0 + token: ${{ secrets.GITHUB_TOKEN }} + persist-credentials: false + + - name: Fetch all tags + run: git fetch --force --tags + + - name: Setup Golang + uses: actions/setup-go@b7ad1dad31e06c5925ef5d2fc7ad053ef454303e # v7.0.0 + with: + go-version: ${{ env.GOLANG_VERSION }} + cache: false + + - name: Set GORELEASER_PREVIOUS_TAG # Workaround, GoReleaser uses 'git-describe' to determine a previous tag. Our tags are created in release branches. + run: | + set -xue + GORELEASER_PREVIOUS_TAG=$(go run hack/get-previous-release/get-previous-version-for-release-notes.go ${GITHUB_REF_NAME}) || exit 1 + echo "GORELEASER_PREVIOUS_TAG=$GORELEASER_PREVIOUS_TAG" >> $GITHUB_ENV + + - name: Set environment variables for ldflags + id: set_ldflag + run: | + echo "KUBECTL_VERSION=$(go list -m k8s.io/client-go | head -n 1 | rev | cut -d' ' -f1 | rev)" >> $GITHUB_ENV + echo "GIT_TREE_STATE=$(if [ -z "`git status --porcelain`" ]; then echo "clean" ; else echo "dirty"; fi)" >> $GITHUB_ENV + + - name: Free Disk Space (Ubuntu) + uses: jlumbroso/free-disk-space@ceedf095f4ec1a097402bc6bd80831f2e1a6fde6 # v2.0.0 + with: + large-packages: false + docker-images: false + swap-storage: false + tool-cache: false + + - name: Run GoReleaser + uses: goreleaser/goreleaser-action@f06c13b6b1a9625abc9e6e439d9c05a8f2190e94 # v7.2.3 + id: run-goreleaser + with: + version: v2.16.0 + args: release --clean --timeout 55m + env: + GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} + KUBECTL_VERSION: ${{ env.KUBECTL_VERSION }} + GIT_TREE_STATE: ${{ env.GIT_TREE_STATE }} + # Used to determine the current repository in the goreleaser config to display correct manifest links + GORELEASER_CURRENT_REPOSITORY: ${{ github.repository }} + + - name: Generate subject for provenance + id: hash + env: + ARTIFACTS: '${{ steps.run-goreleaser.outputs.artifacts }}' + run: | + set -euo pipefail + + hashes=$(echo $ARTIFACTS | jq --raw-output '.[] | {name, "digest": (.extra.Digest // .extra.Checksum)} | select(.digest) | {digest} + {name} | join(" ") | sub("^sha256:";"")' | base64 -w0) + if test "$hashes" = ""; then # goreleaser < v1.13.0 + checksum_file=$(echo "$ARTIFACTS" | jq -r '.[] | select (.type=="Checksum") | .path') + hashes=$(cat $checksum_file | base64 -w0) + fi + echo "hashes=$hashes" >> $GITHUB_OUTPUT + + goreleaser-provenance: + needs: [goreleaser, setup-variables] + permissions: + actions: read # for detecting the Github Actions environment + id-token: write # Needed for provenance signing and ID + contents: write # Needed for release uploads + if: github.repository == 'argoproj/argo-cd' || needs.setup-variables.outputs.allow_fork_release == 'true' + # Must be refernced by a tag. https://github.com/slsa-framework/slsa-github-generator/blob/main/internal/builders/container/README.md#referencing-the-slsa-generator + uses: slsa-framework/slsa-github-generator/.github/workflows/generator_generic_slsa3.yml@v2.1.0 + with: + base64-subjects: '${{ needs.goreleaser.outputs.hashes }}' + provenance-name: 'argocd-cli.intoto.jsonl' + upload-assets: true + + generate-sbom: + name: Create SBOM and generate hash + needs: + - argocd-image + - goreleaser + - setup-variables + permissions: + contents: write # Needed for release uploads + outputs: + hashes: ${{ steps.sbom-hash.outputs.hashes }} + if: github.repository == 'argoproj/argo-cd' || needs.setup-variables.outputs.allow_fork_release == 'true' + runs-on: ubuntu-24.04 + steps: + - name: Checkout code + uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 + with: + fetch-depth: 0 + token: ${{ secrets.GITHUB_TOKEN }} + persist-credentials: false + + - name: Install pnpm + uses: pnpm/action-setup@ea17c68df8912ef543352723c149a84f56e3d413 # v6.1.0 + with: + package_json_file: ui/package.json + + - name: Setup Golang + uses: actions/setup-go@b7ad1dad31e06c5925ef5d2fc7ad053ef454303e # v7.0.0 + with: + go-version: ${{ env.GOLANG_VERSION }} + cache: false + + - name: Generate SBOM (spdx) + id: spdx-builder + env: + # defines the spdx/spdx-sbom-generator version to use. + SPDX_GEN_VERSION: v0.0.13 + # defines the sigs.k8s.io/bom version to use. + SIGS_BOM_VERSION: v0.2.1 + # full qualified name of the docker image to be inspected + DOCKER_IMAGE: ${{ needs.setup-variables.outputs.quay_image_name }} + run: | + set -euo pipefail + pnpm install --dir ./ui --frozen-lockfile + go install github.com/spdx/spdx-sbom-generator/cmd/generator@$SPDX_GEN_VERSION + go install sigs.k8s.io/bom/cmd/bom@$SIGS_BOM_VERSION + + generator -p . -o /tmp + + pnpm --dir ./ui sbom --sbom-format spdx --prod > /tmp/bom-ui-pnpm.spdx.json + + if [[ -n "${DOCKER_IMAGE:-}" ]]; then + bom generate -o /tmp/bom-docker-image.spdx -i "${DOCKER_IMAGE}" + fi + + cd /tmp + shopt -s nullglob + spdx_files=( *.spdx ) + shopt -u nullglob + if [[ ${#spdx_files[@]} -eq 0 ]]; then + echo "No .spdx files produced under /tmp" + exit 1 + fi + tar -zcf sbom.tar.gz "${spdx_files[@]}" bom-ui-pnpm.spdx.json + + - name: Generate SBOM hash + shell: bash + id: sbom-hash + run: | + # sha256sum generates sha256 hash for sbom. + # base64 -w0 encodes to base64 and outputs on a single line. + # sha256sum /tmp/sbom.tar.gz ... | base64 -w0 + echo "hashes=$(sha256sum /tmp/sbom.tar.gz | base64 -w0)" >> "$GITHUB_OUTPUT" + + - name: Upload SBOM + run: gh release upload "${GITHUB_REF_NAME}" /tmp/sbom.tar.gz --clobber + env: + GH_TOKEN: ${{ secrets.GITHUB_TOKEN }} + + sbom-provenance: + needs: [generate-sbom, setup-variables] + permissions: + actions: read # for detecting the Github Actions environment + id-token: write # Needed for provenance signing and ID + contents: write # Needed for release uploads + if: github.repository == 'argoproj/argo-cd' || needs.setup-variables.outputs.allow_fork_release == 'true' + # Must be referenced by a tag. https://github.com/slsa-framework/slsa-github-generator/blob/main/internal/builders/container/README.md#referencing-the-slsa-generator + uses: slsa-framework/slsa-github-generator/.github/workflows/generator_generic_slsa3.yml@v2.1.0 + with: + base64-subjects: '${{ needs.generate-sbom.outputs.hashes }}' + provenance-name: 'argocd-sbom.intoto.jsonl' + upload-assets: true + + post-release: + name: Post-release tasks + needs: + - setup-variables + - argocd-image + - goreleaser + - generate-sbom + permissions: + contents: write # Needed to push commit to update stable and gitops-engine tags + pull-requests: write # Needed to create PR for VERSION update. + if: github.repository == 'argoproj/argo-cd' || needs.setup-variables.outputs.allow_fork_release == 'true' + runs-on: ubuntu-24.04 + env: + TAG_STABLE: ${{ needs.setup-variables.outputs.is_latest_release }} + steps: + - name: Checkout code + uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 + with: + fetch-depth: 0 + token: ${{ secrets.GITHUB_TOKEN }} + persist-credentials: false + + - name: Setup Git author information + run: | + set -ue + git config --global user.email 'ci@argoproj.com' + git config --global user.name 'CI' + + - name: Update stable tag to latest version + run: | + git tag -f stable ${GITHUB_REF_NAME} + git push "https://x-access-token:${GH_TOKEN}@github.com/${GITHUB_REPOSITORY}.git" -f stable + if: ${{ env.TAG_STABLE == 'true' }} + env: + GH_TOKEN: ${{ secrets.GITHUB_TOKEN }} + + - name: Tag gitops-engine submodule + run: | + set -ue + # Create a path-prefixed tag for gitops-engine + # so downstream consumers can resolve github.com/argoproj/argo-cd/gitops-engine/v3 + # https://go.dev/doc/modules/managing-source#multiple-module-source + GE_TAG="gitops-engine/${GITHUB_REF_NAME}" + # checkout uses persist-credentials: false, so push with an explicit token URL + REMOTE="https://x-access-token:${GH_TOKEN}@github.com/${GITHUB_REPOSITORY}.git" + # checkout with fetch-depth: 0 fetches all tags, so a local lookup is enough + if git rev-parse -q --verify "refs/tags/${GE_TAG}" >/dev/null; then + echo "Tag ${GE_TAG} already exists; skipping" + else + git tag "${GE_TAG}" "${GITHUB_REF_NAME}^{}" + git push "${REMOTE}" "refs/tags/${GE_TAG}:refs/tags/${GE_TAG}" + echo "Pushed tag ${GE_TAG}" + fi + env: + GH_TOKEN: ${{ secrets.GITHUB_TOKEN }} + + - name: Check to see if VERSION should be updated on master branch + run: | + set -xue + SOURCE_TAG=${GITHUB_REF_NAME} + VERSION_REF="${SOURCE_TAG#*v}" + COMMIT_HASH=$(git rev-parse HEAD) + if echo "$VERSION_REF" | grep -E -- '^[0-9]+\.[0-9]+\.0-rc1';then + VERSION=$(awk 'BEGIN {FS=OFS="."} {$2++; print}' <<< "${VERSION_REF%-rc1}") + echo "Updating VERSION to: $VERSION" + echo "UPDATE_VERSION=true" >> $GITHUB_ENV + echo "NEW_VERSION=$VERSION" >> $GITHUB_ENV + echo "COMMIT_HASH=$COMMIT_HASH" >> $GITHUB_ENV + else + echo "Not updating VERSION" + echo "UPDATE_VERSION=false" >> $GITHUB_ENV + fi + + - name: Create PR to update VERSION on master branch + if: ${{ env.UPDATE_VERSION == 'true' }} + run: | + set -xue + git config user.name "github-actions[bot]" + git config user.email "41898282+github-actions[bot]@users.noreply.github.com" + BRANCH="update-version-$(date +%s)" + # Make new branch explicitly from master since we're on the release branch + git fetch origin master + git checkout -b "$BRANCH" -f origin/master + # Ensure there is no junk left from the previous steps + git clean -fd + # Update the files + echo ${NEW_VERSION} > VERSION + # Replace the 'project-release: vX.X.X-rcX' line in SECURITY-INSIGHTS.yml + sed -i "s/project-release: v.*$/project-release: v${NEW_VERSION}/" SECURITY-INSIGHTS.yml + # Update the 'commit-hash: XXXXXXX' line in SECURITY-INSIGHTS.yml + sed -i "s/commit-hash: .*/commit-hash: ${COMMIT_HASH}/" SECURITY-INSIGHTS.yml + # Refresh 'last-updated', because the 'project-release' edit above is a + # change this field is required to track. Per the Security Insights v1.0.0 + # spec, last-updated is "the date of the last update to SECURITY-INSIGHTS.yml, + # excluding the properties commit-hash and last-reviewed" -- project-release + # is not excluded, so bumping it without bumping last-updated leaves the + # manifest reporting an update date older than its own contents. + # 'last-reviewed' and 'expiration-date' are deliberately NOT touched here: + # they are maintainer attestations ("reviewed by a project maintainer to + # confirm holistic accuracy"), and automating them would assert a review + # that nobody performed. + sed -i "s/last-updated: .*/last-updated: '$(date -u +%Y-%m-%d)'/" SECURITY-INSIGHTS.yml + # making commit and PR + git add -A + if git diff --cached --quiet; then + echo "No changes to commit" + exit 0 + fi + git commit --signoff -m "Bump version in master" + git push "https://x-access-token:${GH_TOKEN}@github.com/${GITHUB_REPOSITORY}.git" "$BRANCH" + gh pr create \ + --title "chore: Bump version in master" \ + --body "All images built from master should indicate which version we are on track for." \ + --head "$BRANCH" \ + --base master + env: + GH_TOKEN: ${{ secrets.GITHUB_TOKEN }} diff --git a/src/pkg/workflowmodel/testdata/corpus/cilium__cilium.golden.json b/src/pkg/workflowmodel/testdata/corpus/cilium__cilium.golden.json new file mode 100644 index 0000000000..8bb5e8b4fc --- /dev/null +++ b/src/pkg/workflowmodel/testdata/corpus/cilium__cilium.golden.json @@ -0,0 +1,82 @@ +{ + "columns": [ + "Open", + "Actionable", + "Claimed", + "PR in progress", + "CI / Review", + "Merge queue", + "Merged", + "Main CI", + "Release", + "Deploy: release-helm", + "Deploy: release-tool", + "Done" + ], + "roles": { + "deploy": 1, + "pr-verify": 1, + "push-verify": 1, + "release": 1 + }, + "pipelines": [ + { + "id": "gha-github-workflows-build-images-ci", + "role": "pr-verify", + "nodes": 5, + "levels": 3, + "events": [ + "pull_request_target", + "push", + "merge_group" + ] + }, + { + "id": "gha-github-workflows-build-images-releases", + "role": "release", + "nodes": 4, + "levels": 3, + "events": [ + "push" + ] + }, + { + "id": "gha-github-workflows-conformance-kind-proxy-embedded", + "role": "push-verify", + "nodes": 4, + "levels": 2, + "events": [ + "workflow_dispatch", + "push" + ] + }, + { + "id": "gha-github-workflows-release", + "role": "deploy", + "nodes": 2, + "levels": 1, + "events": [ + "workflow_dispatch", + "workflow_call" + ] + } + ], + "ciSubstages": [ + "Build and Push Images", + "Display Digests · Pre-Comment · Push dev chart", + "Post test comment for Renovate PRs after images built" + ], + "findings": { + "environment-on-pr": 2, + "secrets-inherit": 1 + }, + "columnDiff": [ + "column \"CI / Review\" gains 3 CI substage(s) from 1 pipeline(s)", + "column \"Merged\" is no longer the cap", + "column \"Main CI\" added after merge", + "column \"Release\" added after merge", + "column \"Deploy: release-helm\" added after merge", + "column \"Deploy: release-tool\" added after merge", + "column \"Done\" added after merge" + ] +} diff --git a/src/pkg/workflowmodel/testdata/corpus/cilium__cilium/.github/workflows/build-images-ci.yaml b/src/pkg/workflowmodel/testdata/corpus/cilium__cilium/.github/workflows/build-images-ci.yaml new file mode 100644 index 0000000000..f77ba770d8 --- /dev/null +++ b/src/pkg/workflowmodel/testdata/corpus/cilium__cilium/.github/workflows/build-images-ci.yaml @@ -0,0 +1,591 @@ +name: Image CI Build + +# Any change in triggers needs to be reflected in the concurrency group. +on: + pull_request_target: + types: + - opened + - synchronize + - reopened + branches-ignore: + - v1.20 + - v1.19 + - v1.18 + push: + branches: + - main + - ft/main/** + merge_group: + types: [checks_requested] + +permissions: + # To be able to access the repository with `actions/checkout` + contents: read + # To be able to check if base images were built + actions: read + +concurrency: + group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.event.after || (github.event_name == 'merge_group' && github.run_id) }} + cancel-in-progress: true + +jobs: + # GH-46482: Lint checks action is broken, skip dependency for now. + #wait-for-base-images: + # name: Wait for lint checks + # uses: cilium/actions/.github/workflows/wait-for-status-check.yaml@342436e0898719fabc2337b145b91490f33cec2a + # with: + # # Only run this job if the event is pull_request_target and if the PR + # # is not opened from a fork. + # # This is to avoid waiting for base images on push to main or merge_group + # # events as the lint-images-base does not run on those events. + # if: ${{ github.event_name == 'pull_request_target' && github.event.pull_request.head.repo.full_name == github.event.pull_request.base.repo.full_name }} + # pr-number: ${{ github.event.pull_request.number }} + # lint-names: 'Lint image build logic' + # timeout-minutes: 2 + # poll-interval: 15 + + build-and-push-prs: + timeout-minutes: 45 + name: Build and Push Images + runs-on: ${{ vars.UBUNTU_2604_4CPU_16GB || 'ubuntu-26.04' }} + permissions: + contents: read + actions: read + # Required by the Quay OIDC token exchange and keyless cosign signing below. + id-token: write + # Pins the OIDC token subject to + # repo:cilium/cilium:environment:publish-ci-images, which is the identity + # federated with the Quay robot account. + environment: + name: publish-ci-images + deployment: false + # GH-46482: Lint checks action is broken, skip dependency for now. + #needs: wait-for-base-images + outputs: + sha: ${{ steps.tag.outputs.sha }} + strategy: + # Do not cancel the other image builds if one of them fails. + fail-fast: false + matrix: + include: + - name: cilium + dockerfile: ./images/cilium/Dockerfile + platforms: linux/amd64,linux/arm64 + + - name: cilium-cli + dockerfile: ./cilium-cli/Dockerfile + platforms: linux/amd64 + require-dir: cilium-cli + + - name: operator-aws + dockerfile: ./images/operator/Dockerfile + platforms: linux/amd64,linux/arm64 + + - name: operator-azure + dockerfile: ./images/operator/Dockerfile + platforms: linux/amd64,linux/arm64 + + - name: operator-alibabacloud + dockerfile: ./images/operator/Dockerfile + platforms: linux/amd64,linux/arm64 + + - name: operator-generic + dockerfile: ./images/operator/Dockerfile + platforms: linux/amd64,linux/arm64 + + - name: hubble-relay + dockerfile: ./images/hubble-relay/Dockerfile + platforms: linux/amd64,linux/arm64 + + - name: clustermesh-apiserver + dockerfile: ./images/clustermesh-apiserver/Dockerfile + platforms: linux/amd64,linux/arm64 + + steps: + - name: Checkout base or default branch (trusted) + uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 + with: + # We first check if base_ref exist, meaning we're in pull_request_target context, and if not we just use default_branch + ref: ${{ github.base_ref || github.event.repository.default_branch }} + persist-credentials: false + + - name: Set Environment Variables + uses: ./.github/actions/set-env-variables + + - name: Copy scripts to trusted directory + run: | + mkdir -p ../cilium-base-branch/.github/actions + cp -r .github/actions/set-runtime-image ../cilium-base-branch/.github/actions/ + cp -r .github/actions/cosign ../cilium-base-branch/.github/actions/ + + - name: Check for disk usage and cleanup /mnt + shell: bash + run: | + echo "# Disk usage" + df -h + echo "# Usage for /mnt" + sudo du --human-readable \ + -- /mnt + if compgen -G "/mnt/.*" > /dev/null; then + echo "# Hidden files in /mnt:" + sudo du --human-readable -- /mnt/.* 2>/dev/null + fi + echo "# Removing all contents of /mnt except /mnt/swapfile" + sudo find /mnt -mindepth 1 ! -path /mnt/swapfile -exec rm -rf {} + || true + + - name: Free up disk space on runner root filesystem + # The golang-cache import needs at least 40G free to not run out of space. + uses: ./.github/actions/disk-cleanup + with: + min-free-gb: 40 + + - name: Setup docker volumes into /mnt + # This allows us to make use of all available disk. + shell: bash + run: | + sudo systemctl stop docker + sudo mv /var/lib/docker/volumes /mnt/docker-volumes + sudo ln -s /mnt/docker-volumes /var/lib/docker/volumes + sudo systemctl start docker + + - name: Set up Docker Buildx + uses: docker/setup-buildx-action@f87e5991a6d7451dcb8d9637bfbc97413f497069 # v4.4.1 + # Disable GC entirely to avoid buildkit from GC caches. + with: + buildkitd-config-inline: | + [worker.oci] + gc=false + + # Quay's registry endpoint only accepts credentials or a JWT signed by + # Quay itself, so the GitHub OIDC token cannot be used as the registry + # password directly. Trade it for a short lived robot token first. + # + # Because the job references an environment, the subject of the token is + # repo:cilium@21054566/cilium@48109239:environment:publish-ci-images, the + # immutable form that carries the owner and repository ids. That is the + # identity federated with the robot account on the Quay side, so renaming + # the environment breaks the login. + - name: Get a quay.io robot token via OIDC + id: registry-token + env: + ROBOT: ${{ vars.QUAY_ROBOT_CI }} + run: | + oidc_token="$(curl -sSf --retry 3 --retry-all-errors \ + -H "Authorization: bearer ${ACTIONS_ID_TOKEN_REQUEST_TOKEN}" \ + "${ACTIONS_ID_TOKEN_REQUEST_URL}&audience=quay.io" \ + | jq -er '.value')" + echo "::add-mask::${oidc_token}" + + # Pass the credentials on stdin so that the OIDC token is not visible + # in the process list of the runner. + robot_token="$(printf 'user = "%s:%s"\n' "${ROBOT}" "${oidc_token}" \ + | curl -sSf --retry 3 --retry-all-errors -K - \ + "https://quay.io/oauth2/federation/robot/token" \ + | jq -er '.token')" + echo "::add-mask::${robot_token}" + + echo "token=${robot_token}" >> "$GITHUB_OUTPUT" + + - name: Login to ${{ env.REGISTRY_DEV }} for CI + uses: docker/login-action@81c6f93d4bc991e510b342a00b419b3bb4efc5c6 # v4.7.0 + with: + registry: ${{ env.REGISTRY_DEV }} + username: ${{ vars.QUAY_ROBOT_CI }} + password: ${{ steps.registry-token.outputs.token }} + + - name: Getting image tag + id: tag + run: | + if [ "${{ github.event.pull_request.head.sha }}" != "" ]; then + sha=${{ github.event.pull_request.head.sha }} + else + sha=${{ github.sha }} + fi + echo sha=${sha} >> $GITHUB_OUTPUT + + tag=${sha} + echo tag=${tag} >> $GITHUB_OUTPUT + + if [[ "${{ github.event_name == 'push' }}" == "true" ]]; then + if [[ "${{ github.ref_name }}" == "${{ github.event.repository.default_branch }}" ]]; then + floating_tag=latest + else + floating_tag="${{ github.ref_name }}" + # Remove slashes from branch names + floating_tag=${floating_tag##*/} + fi + echo floating_tag=${floating_tag} >> $GITHUB_OUTPUT + fi + + normal_tag="${{ env.REGISTRY_DEV }}/${{ env.ORGANIZATION_DEV }}/${{ matrix.name }}-ci:${tag}" + race_tag="${normal_tag}-race" + unstripped_tag="${normal_tag}-unstripped" + + if [ -n "${floating_tag}" ]; then + floating_normal_tag="${{ env.REGISTRY_DEV }}/${{ env.ORGANIZATION_DEV }}/${{ matrix.name }}-ci:${floating_tag}" + normal_tag="${normal_tag},${floating_normal_tag}" + fi + + if [[ "${{ github.event_name }}" == 'merge_group' ]]; then + # Don't build race images for merge_group events. + race_tag="" + fi + + if [[ "${{ github.event_name }}" == 'merge_group' || "${{ github.event_name }}" == 'push' ]]; then + # Don't build unstripped images for merge_group or push events. + unstripped_tag="" + fi + + echo normal_tag=${normal_tag} >> $GITHUB_OUTPUT + echo race_tag=${race_tag} >> $GITHUB_OUTPUT + echo unstripped_tag=${unstripped_tag} >> $GITHUB_OUTPUT + echo parent_tag="$(cat VERSION)" >> $GITHUB_OUTPUT + + # Warning: since this is a privileged workflow, subsequent workflow job + # steps must take care not to execute untrusted code. + - name: Checkout pull request branch (NOT TRUSTED) + uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 + with: + persist-credentials: false + ref: ${{ steps.tag.outputs.sha }} + # Required by checkout v7+ to fetch fork PR code; see the NOT TRUSTED warning above. + allow-unsafe-pr-checkout: true + + - name: Check for disk usage + shell: bash + run: | + df -h + + - name: Copy runtime image tag from untrusted branch + run: | + cp -r .github/actions/set-runtime-image/runtime-image*.txt ../cilium-base-branch/.github/actions/set-runtime-image/ + + - name: Set runtime image environment variable + uses: ./../cilium-base-branch/.github/actions/set-runtime-image + with: + repository: ${{ env.CILIUM_RUNTIME_IMAGE_PREFIX }} + + # Load Golang cache build from GitHub + - name: Restore Golang cache build from GitHub + uses: actions/cache/restore@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0 + id: go-cache + with: + path: ${{ runner.temp }}/.cache/go + key: ${{ runner.os }}-go-${{ matrix.name }}-cache-${{ hashFiles('**/go.sum') }} + restore-keys: | + ${{ runner.os }}-go-${{ matrix.name }}-cache- + + - name: Check for disk usage + shell: bash + run: | + df -h + docker buildx du + + - name: Create cache directories if they don't exist + if: ${{ steps.go-cache.outputs.cache-hit != 'true' }} + shell: bash + run: | + mkdir -p "${RUNNER_TEMP}/.cache/go" + + # Import GitHub's cache build to docker cache + - name: Copy ${{ matrix.name }} Golang cache to docker cache + uses: docker/build-push-action@c3c9e263c25d99ce0380d002d59b67737d91b0dc # v7.4.0 + with: + provenance: false + context: ${{ runner.temp }}/.cache/go + file: ./images/cache/Dockerfile + push: false + platforms: linux/amd64 + target: import-cache + + - name: Check for disk usage + shell: bash + run: | + df -h + docker buildx du + + - name: Check build constraints + id: check + run: | + if [[ -z "${{ matrix.require-dir }}" ]] || + [[ -d "${{ matrix.require-dir }}" ]]; then + echo build="true" >> $GITHUB_OUTPUT + fi + + # If the images were already built on the merge_group, then we don't need to build them again. + - name: Check if the ${{ matrix.name }} image was already built + id: check-published + if: ${{ steps.check.outputs.build != '' && github.event_name == 'push' }} + shell: bash + env: + IMAGE: ${{ env.REGISTRY_DEV }}/${{ env.ORGANIZATION_DEV }}/${{ matrix.name }}-ci:${{ steps.tag.outputs.tag }} + run: | + exists="false" + digest="" + err="${RUNNER_TEMP}/imagetools-inspect.err" + # Only a "not found" is conclusive; any other failure falls back to building the image. + for attempt in 1 2 3; do + if out="$(docker buildx imagetools inspect --format '{{.Manifest.Digest}}' "${IMAGE}" 2>"${err}")"; then + if [[ "${out}" =~ ^sha256:[0-9a-f]{64}$ ]]; then + echo "${IMAGE} was already built as ${out}" + exists="true" + digest="${out}" + else + echo "::warning::${IMAGE} reported an unexpected digest, building it again" + fi + break + fi + cat "${err}" + if grep --quiet ': not found' "${err}"; then + echo "${IMAGE} has not been built yet" + break + fi + echo "::warning::attempt ${attempt}: cannot tell whether ${IMAGE} exists" + sleep 5 + done + echo exists="${exists}" >> $GITHUB_OUTPUT + echo digest="${digest}" >> $GITHUB_OUTPUT + + - name: Tag local module version + id: create-local-tag + env: + PARENT: ${{ steps.tag.outputs.parent_tag }} + run: | + git config --global user.name "GitHub Actions" + git config --global user.email "github-actions@users.noreply.github.com" + REV="$(git show -s --date=format:'%Y%m%d%H%M%S' --format='%cd-%h')" + LOCAL_TAG="v$PARENT+$REV" + git tag -m "$LOCAL_TAG" --no-sign "$LOCAL_TAG" HEAD + git log -1 "$LOCAL_TAG" + + - name: CI Build ${{ matrix.name }} + uses: docker/build-push-action@c3c9e263c25d99ce0380d002d59b67737d91b0dc # v7.4.0 + id: docker_build_ci + if: ${{ steps.check.outputs.build != '' && steps.check-published.outputs.exists != 'true' }} + with: + provenance: false + context: . + file: ${{ matrix.dockerfile }} + push: true + platforms: ${{ matrix.platforms }} + tags: ${{ steps.tag.outputs.normal_tag }} + target: release + build-args: | + CILIUM_RUNTIME_IMAGE=${{ env.CILIUM_RUNTIME_IMAGE }} + OPERATOR_VARIANT=${{ matrix.name }} + + - name: Add the floating tag to the ${{ matrix.name }} image + id: float-tag + if: ${{ steps.check-published.outputs.exists == 'true' && steps.tag.outputs.floating_tag != '' }} + shell: bash + env: + IMAGE: ${{ env.REGISTRY_DEV }}/${{ env.ORGANIZATION_DEV }}/${{ matrix.name }}-ci + DIGEST: ${{ steps.check-published.outputs.digest }} + FLOATING_TAG: ${{ steps.tag.outputs.floating_tag }} + run: | + # --prefer-index=false makes this a carbon copy, so the digest is preserved. + for attempt in 1 2 3; do + if docker buildx imagetools create --prefer-index=false \ + --tag "${IMAGE}:${FLOATING_TAG}" "${IMAGE}@${DIGEST}"; then + break + fi + if [[ "${attempt}" == "3" ]]; then + echo "::error::cannot tag ${IMAGE}@${DIGEST} as ${IMAGE}:${FLOATING_TAG}" + exit 1 + fi + sleep 5 + done + created="$(docker buildx imagetools inspect --format '{{.Manifest.Digest}}' "${IMAGE}:${FLOATING_TAG}" || true)" + if [[ "${created}" != "${DIGEST}" ]]; then + echo "::warning::${IMAGE}:${FLOATING_TAG} is ${created} and not ${DIGEST}" + fi + echo digest="${created}" >> $GITHUB_OUTPUT + + - name: CI race detection Build ${{ matrix.name }} + uses: docker/build-push-action@c3c9e263c25d99ce0380d002d59b67737d91b0dc # v7.4.0 + id: docker_build_ci_detect_race_condition + if: ${{ steps.check.outputs.build != '' && steps.tag.outputs.race_tag != ''}} + with: + provenance: false + context: . + file: ${{ matrix.dockerfile }} + push: true + platforms: linux/amd64 + tags: ${{ steps.tag.outputs.race_tag }} + target: release + build-args: | + BASE_IMAGE=${{ env.CILIUM_RUNTIME_IMAGE }} + CILIUM_RUNTIME_IMAGE=${{ env.CILIUM_RUNTIME_IMAGE }} + MODIFIERS="LOCKDEBUG=1 RACE=1" + OPERATOR_VARIANT=${{ matrix.name }} + + - name: CI Unstripped Binaries Build ${{ matrix.name }} + uses: docker/build-push-action@c3c9e263c25d99ce0380d002d59b67737d91b0dc # v7.4.0 + id: docker_build_ci_unstripped + if: ${{ steps.check.outputs.build != '' && steps.tag.outputs.unstripped_tag != ''}} + with: + provenance: false + context: . + file: ${{ matrix.dockerfile }} + push: true + platforms: linux/amd64 + tags: ${{ steps.tag.outputs.unstripped_tag }} + target: release + build-args: | + CILIUM_RUNTIME_IMAGE=${{ env.CILIUM_RUNTIME_IMAGE }} + MODIFIERS="NOSTRIP=1" + OPERATOR_VARIANT=${{ matrix.name }} + + - name: Generate SBOM, Sign, and Attest Main CI Image + id: cosign_ci + if: ${{ matrix.name != 'cilium-cli' && steps.check.outputs.build != '' }} + # Don't redden a pull request over a syft, quay or sigstore flake; push and merge_group still fail. + continue-on-error: ${{ startsWith(github.event_name, 'pull_request') }} + uses: ./../cilium-base-branch/.github/actions/cosign + with: + image: "${{ env.REGISTRY_DEV }}/${{ env.ORGANIZATION_DEV }}/${{ matrix.name }}-ci@${{ steps.docker_build_ci.outputs.digest || steps.check-published.outputs.digest }}" + image_tag: "${{ env.REGISTRY_DEV }}/${{ env.ORGANIZATION_DEV }}/${{ matrix.name }}-ci:${{ steps.tag.outputs.tag }}" + sbom_name: "ci_${{ matrix.name }}_${{ steps.tag.outputs.tag }}" + + - name: Generate SBOM, Sign, and Attest Race Detection Image + id: cosign_race + if: ${{ matrix.name != 'cilium-cli' && steps.cosign_ci.outcome != 'failure' && steps.docker_build_ci_detect_race_condition.outcome != 'skipped' }} + continue-on-error: ${{ startsWith(github.event_name, 'pull_request') }} + uses: ./../cilium-base-branch/.github/actions/cosign + with: + image: "${{ env.REGISTRY_DEV }}/${{ env.ORGANIZATION_DEV }}/${{ matrix.name }}-ci@${{ steps.docker_build_ci_detect_race_condition.outputs.digest }}" + image_tag: "${{ env.REGISTRY_DEV }}/${{ env.ORGANIZATION_DEV }}/${{ matrix.name }}-ci:${{ steps.tag.outputs.tag }}-race" + sbom_name: "ci_race_${{ matrix.name }}_${{ steps.tag.outputs.tag }}" + install_cosign: 'false' + + - name: Generate SBOM, Sign, and Attest Unstripped Image + id: cosign_unstripped + if: ${{ matrix.name != 'cilium-cli' && steps.cosign_ci.outcome != 'failure' && steps.docker_build_ci_unstripped.outcome != 'skipped' }} + continue-on-error: ${{ startsWith(github.event_name, 'pull_request') }} + uses: ./../cilium-base-branch/.github/actions/cosign + with: + image: "${{ env.REGISTRY_DEV }}/${{ env.ORGANIZATION_DEV }}/${{ matrix.name }}-ci@${{ steps.docker_build_ci_unstripped.outputs.digest }}" + image_tag: "${{ env.REGISTRY_DEV }}/${{ env.ORGANIZATION_DEV }}/${{ matrix.name }}-ci:${{ steps.tag.outputs.tag }}-unstripped" + sbom_name: "ci_unstripped_${{ matrix.name }}_${{ steps.tag.outputs.tag }}" + install_cosign: 'false' + + # A tolerated failure reports success in the jobs API, so say it out loud here. + - name: Warn about unsigned CI images + if: ${{ steps.cosign_ci.outcome == 'failure' || steps.cosign_race.outcome == 'failure' || steps.cosign_unstripped.outcome == 'failure' }} + shell: bash + run: | + echo "::warning title=Unsigned CI image::${{ matrix.name }} was pushed without a cosign signature or an SBOM attestation." + + - name: CI Image Releases digests + shell: bash + if: ${{ steps.check.outputs.build != '' }} + env: + OUTPUT_FOLDER: ${{ runner.temp }}/image-digest + run: | + mkdir -p "$OUTPUT_FOLDER" + if [ -n "${{ steps.tag.outputs.floating_tag }}" ]; then + echo "${{ env.REGISTRY_DEV }}/${{ env.ORGANIZATION_DEV }}/${{ matrix.name }}-ci:${{ steps.tag.outputs.floating_tag }}@${{ steps.docker_build_ci.outputs.digest || steps.float-tag.outputs.digest || steps.check-published.outputs.digest }}" > "$OUTPUT_FOLDER/${{ matrix.name }}.txt" + fi + echo "${{ env.REGISTRY_DEV }}/${{ env.ORGANIZATION_DEV }}/${{ matrix.name }}-ci:${{ steps.tag.outputs.tag }}@${{ steps.docker_build_ci.outputs.digest || steps.check-published.outputs.digest }}" >> "$OUTPUT_FOLDER/${{ matrix.name }}.txt" + if [[ "${{ steps.docker_build_ci_detect_race_condition.outcome }}" != 'skipped' ]]; then + echo "${{ env.REGISTRY_DEV }}/${{ env.ORGANIZATION_DEV }}/${{ matrix.name }}-ci:${{ steps.tag.outputs.tag }}-race@${{ steps.docker_build_ci_detect_race_condition.outputs.digest }}" >> "$OUTPUT_FOLDER/${{ matrix.name }}.txt" + fi + if [[ "${{ steps.docker_build_ci_unstripped.outcome }}" != 'skipped' ]]; then + echo "${{ env.REGISTRY_DEV }}/${{ env.ORGANIZATION_DEV }}/${{ matrix.name }}-ci:${{ steps.tag.outputs.tag }}-unstripped@${{ steps.docker_build_ci_unstripped.outputs.digest }}" >> "$OUTPUT_FOLDER/${{ matrix.name }}.txt" + fi + + # Upload artifact digests + - name: Upload artifact digests + # Nothing reads this artifact; the digests job only cats it into the log. + continue-on-error: ${{ startsWith(github.event_name, 'pull_request') }} + uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7.0.2 + if: ${{ steps.check.outputs.build != '' }} + with: + name: image-digest ${{ matrix.name }} + path: ${{ runner.temp }}/image-digest + retention-days: 1 + + - name: Check for disk usage + if: ${{ always() }} + shell: bash + run: | + df -h + + image-digests: + if: ${{ always() }} + name: Display Digests + runs-on: ubuntu-26.04 + needs: build-and-push-prs + steps: + - name: Downloading Image Digests + shell: bash + run: | + mkdir -p image-digest/ + + - name: Download digests of all images built + uses: actions/download-artifact@9000827ccba6bdab643e8b6fd33ac0654aef8333 # v8.0.2 + with: + path: image-digest/ + pattern: "*image-digest *" + + - name: Image Digests Output + shell: bash + run: | + cd image-digest/ + find -type f | sort | xargs -d '\n' cat + + push-chart: + name: Push dev chart + needs: build-and-push-prs + permissions: + contents: read + pull-requests: read + statuses: write + # Required by the Quay OIDC token exchange in the called workflow. + id-token: write + uses: ./.github/workflows/push-chart-ci.yaml + with: + checkout_ref: ${{ needs.build-and-push-prs.outputs.sha }} + image_tag: ${{ needs.build-and-push-prs.outputs.sha }} + + pre-comment: + # Avoid running the "Trigger CI from renovate PRs" environment if we don't need to. + name: Pre-Comment + needs: build-and-push-prs + runs-on: ubuntu-26.04 + if: ${{ + github.event_name == 'pull_request_target' && + github.event.pull_request.user.login == vars.RENOVATE_BOT_USERNAME + }} + steps: + - name: Debug + run: | + echo ${{ github.event.pull_request.user.login }} + echo ${{ github.event.event_name }} + + comment: + name: Post test comment for Renovate PRs after images built + runs-on: ubuntu-26.04 + needs: pre-comment + environment: + name: "Trigger CI from renovate PRs" + deployment: false + if: ${{ + github.event_name == 'pull_request_target' && + github.event.pull_request.user.login == vars.RENOVATE_BOT_USERNAME + }} + steps: + - name: Generate Github App Token + uses: actions/create-github-app-token@64ab729a6294000f5aa4a4bc0b73d1f32bc45510 # v3.3.0 + id: generate-token + with: + client-id: ${{ secrets.AUTO_COMMENT_BOT_CLIENT_ID }} + private-key: ${{ secrets.AUTO_COMMENT_BOT_PEM }} + - name: Post /test comment + env: + TOKEN: ${{ steps.generate-token.outputs.token }} + GITHUB_REPOSITORY: ${{ github.repository }} + PULL_REQUEST_NUMBER: ${{ github.event.pull_request.number }} + run: | + echo ${TOKEN} | gh auth login --with-token + gh pr --repo ${GITHUB_REPOSITORY} comment ${PULL_REQUEST_NUMBER} --body "/test" diff --git a/src/pkg/workflowmodel/testdata/corpus/cilium__cilium/.github/workflows/build-images-releases.yaml b/src/pkg/workflowmodel/testdata/corpus/cilium__cilium/.github/workflows/build-images-releases.yaml new file mode 100644 index 0000000000..524d88a3f8 --- /dev/null +++ b/src/pkg/workflowmodel/testdata/corpus/cilium__cilium/.github/workflows/build-images-releases.yaml @@ -0,0 +1,304 @@ +name: Image Release Build + +on: + push: + tags: + - v[0-9]+.[0-9]+.[0-9]+ + - v[0-9]+.[0-9]+.[0-9]+-* + +permissions: + # To be able to access the repository with `actions/checkout` + contents: read + +jobs: + build-and-push: + timeout-minutes: 45 + name: Build and Push Images + permissions: + contents: read + # Required to generate OIDC tokens for `sigstore/cosign-installer` authentication + id-token: write + # Also decides the subject of the OIDC token, see the login steps below. + environment: + name: release + deployment: false + runs-on: ubuntu-26.04 + strategy: + matrix: + include: + - name: cilium + dockerfile: ./images/cilium/Dockerfile + + - name: operator + dockerfile: ./images/operator/Dockerfile + + - name: operator-aws + dockerfile: ./images/operator/Dockerfile + + - name: operator-azure + dockerfile: ./images/operator/Dockerfile + + - name: operator-alibabacloud + dockerfile: ./images/operator/Dockerfile + + - name: operator-generic + dockerfile: ./images/operator/Dockerfile + + - name: hubble-relay + dockerfile: ./images/hubble-relay/Dockerfile + + - name: clustermesh-apiserver + dockerfile: ./images/clustermesh-apiserver/Dockerfile + + steps: + - name: Checkout main branch to access local actions + uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 + with: + ref: ${{ github.event.repository.default_branch }} + persist-credentials: false + + - name: Cleanup Disk space in runner + uses: ./.github/actions/disk-cleanup + + - name: Set Environment Variables + uses: ./.github/actions/set-env-variables + + - name: Copy runtime image action to base branch directory + run: | + mkdir -p ../cilium-base-branch + cp -r .github/actions/set-runtime-image ../cilium-base-branch + + - name: Set up Docker Buildx + uses: docker/setup-buildx-action@f87e5991a6d7451dcb8d9637bfbc97413f497069 # v4.4.1 + + # Docker Hub trades the OIDC token for an access token on its own + # identity endpoint, which the action does for us as long as no password + # is given. The connection ID only names the trust configured on the + # organization, so it lives in a variable of the release environment + # rather than in a secret. + - name: Login to DockerHub + uses: docker/login-action@81c6f93d4bc991e510b342a00b419b3bb4efc5c6 # v4.7.0 + if: ${{ env.PUSH_TO_DOCKER_HUB == 'true' }} + env: + DOCKERHUB_OIDC_CONNECTIONID: ${{ vars.DOCKERHUB_OIDC_CONNECTIONID }} + DOCKERHUB_OIDC_EXPIREIN: 2700 + with: + username: ${{ vars.DOCKERHUB_ORGANIZATION }} + + # Quay's registry endpoint only accepts credentials or a JWT signed by + # Quay itself, so the GitHub OIDC token cannot be used as the registry + # password directly. Trade it for a short lived robot token first. + # + # Because the job references an environment, the subject of the token is + # repo:cilium@21054566/cilium@48109239:environment:release, the immutable + # form that carries the owner and repository ids. That is the identity + # federated with the robot account on the Quay side, so renaming the + # environment breaks the login. + - name: Get a ${{ env.REGISTRY }} robot token via OIDC + id: registry-token + env: + ROBOT: ${{ vars.QUAY_ROBOT_RELEASE }} + run: | + oidc_token="$(curl -sSf --retry 3 --retry-all-errors \ + -H "Authorization: bearer ${ACTIONS_ID_TOKEN_REQUEST_TOKEN}" \ + "${ACTIONS_ID_TOKEN_REQUEST_URL}&audience=quay.io" \ + | jq -er '.value')" + echo "::add-mask::${oidc_token}" + + # Pass the credentials on stdin so that the OIDC token is not visible + # in the process list of the runner. + robot_token="$(printf 'user = "%s:%s"\n' "${ROBOT}" "${oidc_token}" \ + | curl -sSf --retry 3 --retry-all-errors -K - \ + "https://quay.io/oauth2/federation/robot/token" \ + | jq -er '.token')" + echo "::add-mask::${robot_token}" + + echo "token=${robot_token}" >> "$GITHUB_OUTPUT" + + - name: Login to ${{ env.REGISTRY }} + uses: docker/login-action@81c6f93d4bc991e510b342a00b419b3bb4efc5c6 # v4.7.0 + with: + registry: ${{ env.REGISTRY }} + username: ${{ vars.QUAY_ROBOT_RELEASE }} + password: ${{ steps.registry-token.outputs.token }} + + - name: Getting image tag + id: tag + run: | + tag=${GITHUB_REF##*/} + echo tag=$tag >> $GITHUB_OUTPUT + if [ "${{ env.PUSH_TO_DOCKER_HUB }}" == "true" ]; then + echo image_tags=${{ github.repository_owner }}/${{ matrix.name }}:$tag,${{ env.REGISTRY }}/${{ env.ORGANIZATION }}/${{ matrix.name }}:$tag >> $GITHUB_OUTPUT + else + echo image_tags=${{ env.REGISTRY }}/${{ env.ORGANIZATION }}/${{ matrix.name }}:$tag >> $GITHUB_OUTPUT + fi + + - name: Checkout Source Code + uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 + with: + persist-credentials: false + + - name: Copy runtime image tag from stable branch + run: | + cp -r .github/actions/set-runtime-image/runtime-image*.txt ../cilium-base-branch/set-runtime-image/ + + - name: Set runtime image environment variable + uses: ./../cilium-base-branch/set-runtime-image + with: + repository: ${{ env.CILIUM_RUNTIME_IMAGE_PREFIX }} + + - name: Release Build ${{ matrix.name }} + uses: docker/build-push-action@c3c9e263c25d99ce0380d002d59b67737d91b0dc # v7.4.0 + id: docker_build_release + with: + provenance: false + context: . + file: ${{ matrix.dockerfile }} + push: true + platforms: linux/amd64,linux/arm64 + tags: ${{ steps.tag.outputs.image_tags }} + target: release + build-args: | + CILIUM_RUNTIME_IMAGE=${{ env.CILIUM_RUNTIME_IMAGE }} + OPERATOR_VARIANT=${{ matrix.name }} + + - name: Generate SBOM, Sign Images, and Attach Attestations + uses: ./.github/actions/cosign + with: + image: "${{ env.REGISTRY }}/${{ env.ORGANIZATION }}/${{ matrix.name }}@${{ steps.docker_build_release.outputs.digest }}" + image_tag: "${{ env.REGISTRY }}/${{ env.ORGANIZATION }}/${{ matrix.name }}:${{ steps.tag.outputs.tag }}" + sbom_name: "${{ matrix.name }}_${{ steps.tag.outputs.tag }}" + upload_sbom_release_assets: 'false' + + - name: Generate SBOM, Sign Images, and Attach Attestations + if: ${{ env.PUSH_TO_DOCKER_HUB == 'true' }} + uses: ./.github/actions/cosign + with: + image: "${{ env.BACKUP_REGISTRY }}/${{ github.repository_owner }}/${{ matrix.name }}@${{ steps.docker_build_release.outputs.digest }}" + sbom_name: "${{ matrix.name }}_${{ steps.tag.outputs.tag }}" + generate_sbom: 'false' + install_cosign: 'false' + upload_sbom_release_assets: 'false' + + - name: Image Release Digest + shell: bash + run: | + mkdir -p image-digest/ + job_name=${{ matrix.name }} + job_name_capital=${job_name^^} + job_name_underscored=${job_name_capital//-/_} + echo "${job_name_underscored}_DIGEST := \"${{ steps.docker_build_release.outputs.digest }}\"" > image-digest/makefile-digest.txt + + echo "### ${{ matrix.name }}" > image-digest/${{ matrix.name }}.txt + echo "" >> image-digest/${{ matrix.name }}.txt + if [ "${{ env.PUSH_TO_DOCKERHUB }}" == "true" ]; then + echo "\`${{ env.BACKUP_REGISTRY }}/${{ github.repository_owner }}/${{ matrix.name }}:${{ steps.tag.outputs.tag }}@${{ steps.docker_build_release.outputs.digest }}\`" >> image-digest/${{ matrix.name }}.txt + fi + echo "\`${{ env.REGISTRY }}/${{ env.ORGANIZATION }}/${{ matrix.name }}:${{ steps.tag.outputs.tag }}@${{ steps.docker_build_release.outputs.digest }}\`" >> image-digest/${{ matrix.name }}.txt + echo "" >> image-digest/${{ matrix.name }}.txt + + # Upload artifact digests + - name: Upload artifact digests + uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7.0.2 + with: + name: image-digest ${{ matrix.name }} + path: image-digest + + image-digests: + name: Display Digests + runs-on: ubuntu-26.04 + needs: build-and-push + steps: + - name: Getting image tag + id: tag + run: | + echo tag=${GITHUB_REF##*/} >> $GITHUB_OUTPUT + - name: Downloading Image Digests + shell: bash + run: | + mkdir -p image-digest/ + + - name: Download digests of all images built + uses: actions/download-artifact@9000827ccba6bdab643e8b6fd33ac0654aef8333 # v8.0.2 + with: + path: image-digest/ + pattern: "*image-digest *" + + - name: Image Digests Output + shell: bash + run: | + cd image-digest/ + echo "## Docker Manifests" > ../image-digest-output.txt + echo "" >> ../image-digest-output.txt + find -type f -regex ".*image-digest .*" -not -name "makefile-digest.txt" | sort | xargs -d '\n' cat >> ../image-digest-output.txt + + - name: Image Makefile Digests + shell: bash + run: | + cd image-digest/ + echo "# File generated by .github/workflows/build-images-releases.yaml; DO NOT EDIT." > ../Makefile.digests + echo "# Copyright $(date +'%Y') Authors of Cilium" >> ../Makefile.digests + echo "# SPDX-License-Identifier: Apache-2.0" >> ../Makefile.digests + echo "" >> ../Makefile.digests + # shellcheck disable=SC2016 + find -type f -name "makefile-digest.txt" | sort | xargs -d '\n' awk '{print "export " $0}' >> ../Makefile.digests + + # Upload artifact digests + - name: Upload artifact digests + uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7.0.2 + with: + name: image-digest-output.txt-${{ steps.tag.outputs.tag }} + path: image-digest-output.txt + + # Upload artifact digests + - name: Upload artifact digests + uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7.0.2 + with: + name: Makefile.digests-${{ steps.tag.outputs.tag }} + path: Makefile.digests + + call-post-release: + if: github.repository_owner == 'cilium' + name: Call Post-Release Tool + permissions: + contents: read + # release.yaml's release-helm job requests it, even when skipped + id-token: write + uses: cilium/cilium/.github/workflows/release.yaml@main # main + needs: image-digests + with: + step: "4-post-release" + version: ${{ github.ref_name }} + # This is the intended behavior of GitHub Actions. Declaring entries under + # "secrets:" grants the called workflow permission to read + # CILIUM_RELEASE_BOT_PEM and CILIUM_RELEASE_BOT_APP_ID. It does not pass + # literal values here; the values are resolved only in the called + # workflow's scope. If the called workflow sets an environment + # (e.g., environment: TestEnvironment), environment-scoped secrets are + # populated there and become available to its steps. + secrets: + CILIUM_RELEASE_BOT_PEM: ${{ secrets.CILIUM_RELEASE_BOT_PEM }} + CILIUM_RELEASE_BOT_APP_ID: ${{ secrets.CILIUM_RELEASE_BOT_APP_ID }} + + call-publish-helm: + if: github.repository_owner == 'cilium' + name: Publish Helm Chart + permissions: + contents: read + # Required to generate OIDC tokens for `sigstore/cosign-installer` authentication + id-token: write + uses: cilium/cilium/.github/workflows/release.yaml@main # main + needs: image-digests + with: + step: "5-publish-helm" + version: ${{ github.ref_name }} + # This is the intended behavior of GitHub Actions. Declaring entries under + # "secrets:" grants the called workflow permission to read + # CILIUM_RELEASE_BOT_PEM and CILIUM_RELEASE_BOT_APP_ID. It does not pass + # literal values here; the values are resolved only in the called + # workflow's scope. If the called workflow sets an environment + # (e.g., environment: TestEnvironment), environment-scoped secrets are + # populated there and become available to its steps. + secrets: + CILIUM_RELEASE_BOT_PEM: ${{ secrets.CILIUM_RELEASE_BOT_PEM }} + CILIUM_RELEASE_BOT_APP_ID: ${{ secrets.CILIUM_RELEASE_BOT_APP_ID }} diff --git a/src/pkg/workflowmodel/testdata/corpus/cilium__cilium/.github/workflows/conformance-kind-proxy-embedded.yaml b/src/pkg/workflowmodel/testdata/corpus/cilium__cilium/.github/workflows/conformance-kind-proxy-embedded.yaml new file mode 100644 index 0000000000..b9c3363ae3 --- /dev/null +++ b/src/pkg/workflowmodel/testdata/corpus/cilium__cilium/.github/workflows/conformance-kind-proxy-embedded.yaml @@ -0,0 +1,308 @@ +name: Conformance Kind Envoy Embedded + +# Any change in triggers needs to be reflected in the concurrency group. +on: + workflow_dispatch: + inputs: + PR-number: + description: "Pull request number." + required: true + UID: + description: "A second number, to differentiate multiple runs of the same workflow by the same PR" + required: false + default: 1 + context-ref: + description: "Context in which the workflow runs. If PR is from a fork, will be the PR target branch (general case). If PR is NOT from a fork, will be the PR branch itself (this allows committers to test changes to workflows directly from PRs)." + required: true + SHA: + description: "SHA under test (head of the PR branch)." + required: true + base-SHA: + description: "SHA of the base branch (target branch of the PR)." + required: false + extra-args: + description: "[JSON object] Arbitrary arguments passed from the trigger comment via regex capture group. Parse with 'fromJson(inputs.extra-args).argName' in workflow." + required: false + default: '{}' + + push: + branches: + - main + - ft/main/** + paths-ignore: + - 'Documentation/**' + - 'test/**' + +# By specifying the access of one of the scopes, all of those that are not +# specified are set to 'none'. +permissions: + # To read actions state with cilium/workflow-telemetry-action + actions: read + # To be able to access the repository with actions/checkout + contents: read + # To allow retrieving information from the PR API + pull-requests: read + # To be able to set commit status + statuses: write + # To be able to request the JWT from GitHub's OIDC provider + id-token: write + +concurrency: + group: ${{ github.workflow }}-${{ inputs.PR-number || github.event.after }} + cancel-in-progress: true + +env: + kind_config: .github/kind-config-dual.yaml + +jobs: + echo-inputs: + if: ${{ github.event_name == 'workflow_dispatch' }} + name: Echo Workflow Dispatch Inputs + runs-on: ubuntu-26.04 + steps: + - name: Echo Workflow Dispatch Inputs + run: | + echo '${{ tojson(inputs) }}' + + commit-status-start: + name: Commit Status Start + runs-on: ubuntu-26.04 + steps: + - name: Set initial commit status + uses: cilium/actions/set-commit-status@687b48a0547e534dd7270d5069a5ab7967c0e907 # main + with: + sha: ${{ inputs.SHA || github.sha }} + + installation-and-connectivity: + name: "Installation and Connectivity Test (${{ matrix.name }})" + runs-on: ${{ vars.UBUNTU_2604_4CPU_16GB || 'ubuntu-26.04' }} + timeout-minutes: 60 + strategy: + fail-fast: false + matrix: + include: + - name: ads + extra-helm-set: "--helm-set=envoy.xdsMode=ads" + - name: split + extra-helm-set: "--helm-set=envoy.xdsMode=split" + - name: delta-split + extra-helm-set: "--helm-set=envoy.xdsMode=delta-split" + env: + job_name: "Installation and Connectivity Test (${{ matrix.name }})" + ENABLE_CLUSTERNETWORKPOLICY: "true" + steps: + - name: Collect Workflow Telemetry + uses: cilium/workflow-telemetry-action@b1a318f80a08acede58027afe1455d037379966f # v2.2.0 + with: + comment_on_pr: false + + - name: Checkout code (TRUSTED) + uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 + with: + persist-credentials: false + ref: ${{ inputs.context-ref || github.sha}} + + - name: Set Environment Variables + uses: ./.github/actions/set-env-variables + + - name: Get Cilium's default values + id: default_vars + uses: ./.github/actions/helm-default + with: + image-tag: ${{ inputs.SHA || github.sha }} + registry_host: ${{ vars.DOCKER_READ_HOST }} + registry_organization: ${{ vars.DOCKER_READ_ORG }} + image-pull-secret: "cilium-registry" + chart-dir: 'untrusted/install/kubernetes/cilium' + + - name: Add external docker network + uses: ./.github/actions/kind-external-network + id: external_network + + - name: Create kind cluster + uses: helm/kind-action@1544676c07bb3570fcb70166beaa07c4528fd4aa # v1.15.1 + with: + version: ${{ env.KIND_VERSION }} + node_image: ${{ env.KIND_K8S_IMAGE }} + kubectl_version: ${{ env.KIND_K8S_VERSION }} + config: ${{ env.kind_config }} + wait: 0 # The control-plane never becomes ready, since no CNI is present + + - name: Add external targets to kind cluster + uses: ./.github/actions/kind-external-targets + id: external_targets + with: + kind_network: ${{ steps.external_network.outputs.kind_network }} + ipv4_external_target: ${{ steps.external_network.outputs.ipv4_external_target }} + ipv4_other_external_target: ${{ steps.external_network.outputs.ipv4_other_external_target }} + ipv6_external_target: ${{ steps.external_network.outputs.ipv6_external_target }} + ipv6_other_external_target: ${{ steps.external_network.outputs.ipv6_other_external_target }} + + - name: Get connectivity test flags + id: e2e_config + uses: ./.github/actions/cli-test-config + with: + hubble: false + include-unsafe-tests: true + test-concurrency: 5 + external-cidr: ${{ steps.external_network.outputs.ipv4_external_cidr }} + external-cidrv6: ${{ steps.external_network.outputs.ipv6_external_cidr }} + external-ip: ${{ steps.external_network.outputs.ipv4_external_target }} + external-ipv6: ${{ steps.external_network.outputs.ipv6_external_target }} + external-other-ip: ${{ steps.external_network.outputs.ipv4_other_external_target }} + external-other-ipv6: ${{ steps.external_network.outputs.ipv6_other_external_target }} + external-other-target: ${{ steps.external_targets.outputs.other_external_target_name }} + external-target: ${{ steps.external_targets.outputs.external_target_name }} + external-target-ca-namespace: external-target-secrets + external-target-ca-name: custom-ca + external-target-ipv6-capable: true + + - name: Set up job variables + id: vars + run: | + # Note: On Kind, we install Cilium with HostPort (portmap CNI chaining) enabled, + # to ensure coverage of that feature in cilium connectivity test + # This test intentionally sends traffic with and without L7 LB in succession. + # Bound attempts on a stale pooled connection and close that socket immediately + # so Envoy can release a colliding 5-tuple and retry when the client reuses its + # source port for a direct connection. + CILIUM_INSTALL_DEFAULTS="${{ steps.default_vars.outputs.cilium_install_defaults }} \ + --helm-set=hubble.relay.enabled=true + --helm-set=cni.chainingMode=portmap \ + --helm-set-string=kubeProxyReplacement=true \ + --helm-set=k8sClusterNetworkPolicy.enabled=${ENABLE_CLUSTERNETWORKPOLICY} \ + --helm-set=loadBalancer.l7.backend=envoy \ + --helm-set=envoy.httpRetryTimeout=1 \ + --helm-set=envoy.httpUpstreamLingerTimeout=0 \ + --helm-set=tls.readSecretsOnlyFromSecretsNamespace=true \ + --helm-set=tls.secretSync.enabled=true \ + --helm-set=envoy.enabled=false \ + --helm-set=ipv6.enabled=true \ + ${{ matrix.extra-helm-set }} \ + --wait=false" + CONNECTIVITY_TEST_DEFAULTS="${{ steps.e2e_config.outputs.test_flags }}" + echo cilium_install_defaults=${CILIUM_INSTALL_DEFAULTS} >> $GITHUB_OUTPUT + echo "test default: ${CONNECTIVITY_TEST_DEFAULTS}" + echo connectivity_test_defaults=${CONNECTIVITY_TEST_DEFAULTS} >> $GITHUB_OUTPUT + echo sha=${{ steps.default_vars.outputs.sha }} >> $GITHUB_OUTPUT + + - name: Install Cilium CLI + uses: cilium/cilium-cli@1ac7fa66909e3730ab3869c8ba7aa7d68f0197fd # v0.20.2 + with: + skip-build: ${{ env.CILIUM_CLI_SKIP_BUILD }} + image-repo: ${{ env.CILIUM_CLI_IMAGE_REPO }} + image-tag: ${{ steps.vars.outputs.sha }} + repository: ${{ env.CILIUM_CLI_RELEASE_REPO }} + release-version: ${{ env.CILIUM_CLI_VERSION }} + + - name: Create imagePullSecret + if: ${{ vars.DOCKER_AUTH_REQUIRED == 'true' }} + run: | + kubectl create secret docker-registry cilium-registry \ + --namespace kube-system \ + --docker-server=${{ vars.DOCKER_READ_HOST }} \ + --docker-username="${{ vars.DOCKER_READ_USERNAME }}" \ + --docker-password="${{ secrets.DOCKER_READ_PASSWORD }}" + + - name: Wait for images to be available + uses: ./.github/actions/wait-for-images + with: + SHA: ${{ steps.vars.outputs.sha }} + images: cilium-ci operator-generic-ci hubble-relay-ci + login-host: ${{ vars.DOCKER_READ_HOST }} + login-username: ${{ vars.DOCKER_READ_USERNAME }} + login-password: ${{ secrets.DOCKER_READ_PASSWORD }} + auth-required: ${{ vars.DOCKER_AUTH_REQUIRED }} + + # Warning: since this is a privileged workflow, subsequent workflow job + # steps must take care not to execute untrusted code. + - name: Checkout pull request branch (NOT TRUSTED) + uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 + with: + ref: ${{ steps.vars.outputs.sha }} + persist-credentials: false + # Required by checkout v7+ to fetch fork PR code; see the NOT TRUSTED warning above. + allow-unsafe-pr-checkout: true + path: untrusted + sparse-checkout: | + install/kubernetes/cilium + + - name: Install KCNP CRD + run: | + make kind-install-kcnp-crd ENABLE_CLUSTERNETWORKPOLICY=${ENABLE_CLUSTERNETWORKPOLICY} + + - name: Install Cilium + id: install-cilium + run: | + cilium install ${{ steps.vars.outputs.cilium_install_defaults }} + + - name: Wait for Cilium status to be ready + run: | + cilium status --wait --interactive=false + kubectl -n kube-system get pods + + - name: Make JUnit report directory + run: | + mkdir -p cilium-junits + + - name: Run connectivity test + id: run-tests + run: | + cilium connectivity test ${{ steps.vars.outputs.connectivity_test_defaults }} \ + --curl-parallel 3 \ + --junit-file "cilium-junits/${{ env.job_name }}.xml" --junit-property github_job_step="Run connectivity test" + + - name: Features tested + if: ${{ always() && steps.install-cilium.outcome == 'success' }} + uses: ./.github/actions/feature-status + with: + title: "Summary of all features tested" + json-filename: "${{ env.job_name }}" + + - name: Install Cilium with secret sync disabled + id: install-cilium-secret-sync-disabled + run: | + helm upgrade cilium untrusted/install/kubernetes/cilium \ + --namespace kube-system \ + --reuse-values \ + --set=k8sClusterNetworkPolicy.enabled=${ENABLE_CLUSTERNETWORKPOLICY} \ + --set=tls.readSecretsOnlyFromSecretsNamespace=false \ + --set=tls.secretSync.enabled=false + + kubectl -n kube-system rollout restart ds/cilium deployment/cilium-operator + cilium status --wait --interactive=false --wait-duration=10m + kubectl -n kube-system get pods + + - name: Run L7 related connectivity test + run: | + cilium connectivity test ${{ steps.vars.outputs.connectivity_test_defaults }} \ + --curl-parallel 3 \ + --junit-file "cilium-junits/${{ env.job_name }}-without-secret-sync.xml" \ + --junit-property github_job_step="Run connectivity test with secret sync disabled" \ + --test="l7|sni|check-log-errors" + + - name: Features tested + if: ${{ always() && steps.install-cilium-secret-sync-disabled.outcome == 'success' }} + uses: ./.github/actions/feature-status + with: + title: "Summary of all features tested" + json-filename: "${{ env.job_name }}-without-secret-sync" + + - name: Run common post steps + if: ${{ always() && steps.install-cilium.outcome != 'skipped' }} + uses: ./.github/actions/post-logic + with: + artifacts_suffix: "${{ env.job_name }}" + job_status: "${{ job.status }}" + capture_features_tested: false + + merge-upload-and-status: + name: Merge Upload and Status + if: ${{ always() }} + needs: installation-and-connectivity + uses: ./.github/workflows/common-post-jobs.yaml + secrets: inherit + with: + context-ref: ${{ inputs.context-ref || github.sha }} + sha: ${{ inputs.SHA || github.sha }} + success: ${{ needs.installation-and-connectivity.result == 'success' }} diff --git a/src/pkg/workflowmodel/testdata/corpus/cilium__cilium/.github/workflows/release.yaml b/src/pkg/workflowmodel/testdata/corpus/cilium__cilium/.github/workflows/release.yaml new file mode 100644 index 0000000000..568d56eaae --- /dev/null +++ b/src/pkg/workflowmodel/testdata/corpus/cilium__cilium/.github/workflows/release.yaml @@ -0,0 +1,440 @@ +name: Release Tool + +on: + workflow_dispatch: + inputs: + step: + description: 'Which step do you want to (re-)run?' + required: true + type: choice + options: + - 2-prepare-release + - 4-post-release + - 5-publish-helm + version: + description: 'Which version are you releasing? (e.g. vX.Y.Z[-(pre|rc).W])' + required: true + type: string + default: vX.Y.Z + + workflow_call: + inputs: + step: + description: 'Which step do you want to (re-)run?' + required: true + type: string + version: + description: 'Which version are you releasing? (e.g. vX.Y.Z[-(pre|rc).W])' + required: true + type: string + secrets: + CILIUM_RELEASE_BOT_PEM: + required: true + CILIUM_RELEASE_BOT_APP_ID: + required: true + +permissions: + # To be able to access the repository with `actions/checkout` + contents: read + +concurrency: + group: ${{ github.workflow }}-${{ inputs.version }}-${{ inputs.step }} + cancel-in-progress: true + +jobs: + release: + name: Release + if: ${{ inputs.step != '5-publish-helm' }} + environment: + name: release-tool + deployment: false + timeout-minutes: 40 + runs-on: ubuntu-26.04 + steps: + - name: Install Go + uses: actions/setup-go@b7ad1dad31e06c5925ef5d2fc7ad053ef454303e # v7.0.0 + with: + # renovate: datasource=golang-version depName=go + go-version: 1.27.3 + + - name: Get token + id: get_token + uses: cilium/actions-app-token@61a6271ce92ba02f49bf81c755685d59fb25a59a # v0.21.1 + with: + APP_PEM: ${{ secrets.CILIUM_RELEASE_BOT_PEM }} + APP_ID: ${{ secrets.CILIUM_RELEASE_BOT_APP_ID }} + + - name: Authenticate with GH CLI + run: | + gh auth login --with-token <<< "${{ steps.get_token.outputs.app_token }}" + + - name: Checkout release tool + uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 + with: + persist-credentials: false + repository: cilium/release + path: "./release" + + - name: Move release source code to upper directory + run: mv release ../ + + - name: Checkout cilium source code + uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 + with: + ref: ${{ github.event.repository.default_branch }} + persist-credentials: false + + - name: Build release tool + shell: bash + run: | + cd ../release + make + + - name: Set-up git + run: | + git config user.name "Cilium Release Bot" + git config user.email "noreply@cilium.io" + git remote set-url origin https://x-access-token:${{ steps.get_token.outputs.app_token }}@github.com/${{ github.repository }}.git + + - name: Run release tool + shell: bash + env: + GITHUB_TOKEN: "${{ steps.get_token.outputs.app_token }}" + ORG: "${{ github.repository_owner }}" + run: | + cd ../release + ./release start \ + --force \ + --release-tool-dir "$(pwd)" \ + --charts-repo-dir "$(pwd)/../charts" \ + --repo-dir "$(pwd)/../cilium" \ + --repo ${{ github.repository }} \ + --target-version ${{ inputs.version }} \ + --steps ${{ inputs.step }} \ + --exclude-labels "cilium-cli-exclusive" + + release-helm: + name: Release Helm + if: ${{ inputs.step == '5-publish-helm' }} + permissions: + contents: read + # Required to generate OIDC tokens for `sigstore/cosign-installer` authentication + id-token: write + environment: + name: release-helm + deployment: false + timeout-minutes: 55 + runs-on: ubuntu-26.04 + steps: + - name: Install Go + uses: actions/setup-go@b7ad1dad31e06c5925ef5d2fc7ad053ef454303e # v7.0.0 + with: + # renovate: datasource=golang-version depName=go + go-version: 1.27.3 + + - name: Install helm + shell: bash + run: | + # We don't want renovate to update this version, because we want to + # use a specific version of helm for the release tool to avoid + # breaking the release process. + HELM_VERSION=v3.18.5 + wget "https://get.helm.sh/helm-${HELM_VERSION}-linux-amd64.tar.gz" + tar -xf "helm-${HELM_VERSION}-linux-amd64.tar.gz" + sudo mv ./linux-amd64/helm /usr/local/bin + + - name: Get token + id: get_token + uses: cilium/actions-app-token@61a6271ce92ba02f49bf81c755685d59fb25a59a # v0.21.1 + with: + APP_PEM: ${{ secrets.CILIUM_RELEASE_BOT_PEM }} + APP_ID: ${{ secrets.CILIUM_RELEASE_BOT_APP_ID }} + + - name: Authenticate with GH CLI + run: | + gh auth login --with-token <<< "${{ steps.get_token.outputs.app_token }}" + + - name: Checkout release tool + uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 + with: + persist-credentials: false + repository: cilium/release + path: "./release" + + - name: Move release source code to upper directory + run: mv release ../ + + - name: Checkout helm chart + uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 + with: + persist-credentials: false + repository: cilium/charts + path: "./cilium-charts" + + - name: Set up charts repository git configuration + shell: bash + run: | + mv cilium-charts ../charts + cd ../charts + git config user.name "Cilium Release Bot" + git config user.email "noreply@cilium.io" + git remote set-url origin https://x-access-token:${{ steps.get_token.outputs.app_token }}@github.com/cilium/charts.git + git remote set-head origin --auto + + - name: Checkout cilium source code + uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 + with: + ref: ${{ github.event.repository.default_branch }} + persist-credentials: false + + - name: Build release tool + shell: bash + run: | + cd ../release + make + + - name: Set-up git + run: | + git config user.name "Cilium Release Bot" + git config user.email "noreply@cilium.io" + git remote set-url origin https://x-access-token:${{ steps.get_token.outputs.app_token }}@github.com/${{ github.repository }}.git + + # Quay's registry endpoint only accepts credentials or a JWT signed by + # Quay itself, so the GitHub OIDC token cannot be used as the registry + # password directly. Trade it for a short lived robot token first. + # + # Because the job references an environment, the subject of the token is + # repo:cilium@21054566/cilium@48109239:environment:release-helm, the + # immutable form that carries the owner and repository ids. That is the + # identity federated with the robot account on the Quay side, so renaming + # the environment breaks the login. + - name: Get a quay.io robot token via OIDC + id: registry-token + env: + ROBOT: ${{ vars.QUAY_ROBOT_HELM }} + run: | + oidc_token="$(curl -sSf --retry 3 --retry-all-errors \ + -H "Authorization: bearer ${ACTIONS_ID_TOKEN_REQUEST_TOKEN}" \ + "${ACTIONS_ID_TOKEN_REQUEST_URL}&audience=quay.io" \ + | jq -er '.value')" + echo "::add-mask::${oidc_token}" + + # Pass the credentials on stdin so that the OIDC token is not visible + # in the process list of the runner. + robot_token="$(printf 'user = "%s:%s"\n' "${ROBOT}" "${oidc_token}" \ + | curl -sSf --retry 3 --retry-all-errors -K - \ + "https://quay.io/oauth2/federation/robot/token" \ + | jq -er '.token')" + echo "::add-mask::${robot_token}" + + echo "token=${robot_token}" >> "$GITHUB_OUTPUT" + + - name: Login to Helm registries + shell: bash + env: + ROBOT: ${{ vars.QUAY_ROBOT_HELM }} + ROBOT_TOKEN: ${{ steps.registry-token.outputs.token }} + run: | + echo "Logging into Quay.io..." + echo "${ROBOT_TOKEN}" | helm registry login quay.io --username "${ROBOT}" --password-stdin + + - name: Run release tool + shell: bash + env: + GITHUB_TOKEN: "${{ steps.get_token.outputs.app_token }}" + ORG: "${{ github.repository_owner }}" + run: | + cd ../release + ./release start \ + --force \ + --release-tool-dir "$(pwd)" \ + --charts-repo-dir "$(pwd)/../charts" \ + --repo-dir "$(pwd)/../cilium" \ + --repo ${{ github.repository }} \ + --target-version ${{ inputs.version }} \ + --steps ${{ inputs.step }} \ + --exclude-labels "cilium-cli-exclusive" + + - name: Wait for helm chart PR checks and merge + shell: bash + env: + GITHUB_TOKEN: "${{ steps.get_token.outputs.app_token }}" + VERSION: ${{ inputs.version }} + run: | + echo "🔄 Waiting for helm chart PR checks and merging for version $VERSION" + + # Find the PR for this release + BRANCH="pr/prepare-$VERSION" + echo "Looking for PR from branch: $BRANCH" + + # Wait for PR to exist (in case it was just created) + MAX_WAIT=300 # 5 minutes + WAIT_TIME=0 + PR_NUMBER="" + + while [ $WAIT_TIME -lt $MAX_WAIT ]; do + PR_NUMBER=$(gh pr list --repo cilium/charts --head "$BRANCH" --state open --json number,headRepositoryOwner \ + --jq '[.[] | select(.headRepositoryOwner.login == "cilium")][0].number // empty') + if [ -n "$PR_NUMBER" ]; then + echo "Found PR #$PR_NUMBER" + break + fi + echo "Waiting for PR to be created..." + sleep 10 + WAIT_TIME=$((WAIT_TIME + 10)) + done + + if [ -z "$PR_NUMBER" ]; then + echo "❌ No PR found for branch $BRANCH after waiting $MAX_WAIT seconds" + echo "Available PRs:" + gh pr list --repo cilium/charts --state open || echo "Failed to list PRs" + exit 1 + fi + + # Only the Cilium chart workflow validates the chart that this job releases. + wait_for_chart_checks() { + local head_sha=$1 run_id="" wait_time=0 + echo "⏳ Waiting for the Cilium chart checks of $head_sha..." + while [ $wait_time -lt $MAX_WAIT ]; do + run_id=$(gh api -X GET repos/cilium/charts/actions/workflows/validate-cilium-chart.yaml/runs \ + -f head_sha="$head_sha" -f event=pull_request -f branch="$BRANCH" \ + --jq '[.workflow_runs[] | select(.head_repository.full_name == "cilium/charts")][0].id // empty') + if [ -n "$run_id" ]; then + break + fi + sleep 10 + wait_time=$((wait_time + 10)) + done + if [ -z "$run_id" ]; then + echo "❌ No Cilium chart checks found for $head_sha after waiting $MAX_WAIT seconds" + return 1 + fi + until gh run watch "$run_id" --repo cilium/charts --exit-status --compact --interval 10; do + case "$(gh run view "$run_id" --repo cilium/charts --json status,conclusion --jq '.status + "/" + .conclusion')" in + completed/success) return 0 ;; + completed/*) + echo "❌ Cilium chart checks failed: https://github.com/cilium/charts/actions/runs/$run_id" + return 1 ;; + esac + sleep 10 + done + } + + repush_chart() { + echo "🔄 Re-running release tool to update PR..." + + # Reset the charts directory by removing any files + # This is a workaround due to left overs made by the generate_helm_release.sh script + echo "🧹 Resetting ../charts directory..." + cd ../charts + git reset --hard "$(git symbolic-ref refs/remotes/origin/HEAD)" + git clean -fdx + cd - + + # Re-run the release tool + cd ../release + ./release start \ + --force \ + --release-tool-dir "$(pwd)" \ + --charts-repo-dir "$(pwd)/../charts" \ + --repo-dir "$(pwd)/../cilium" \ + --repo ${{ github.repository }} \ + --target-version $VERSION \ + --steps ${{ inputs.step }} \ + --exclude-labels "cilium-cli-exclusive" + cd - + } + + echo "Merging PR #$PR_NUMBER..." + CHECK_FAILURES=0 + MERGE_FAILURES=0 + MAX_FAILURES=3 + MERGE_SUCCESS=false + + while [ $CHECK_FAILURES -lt $MAX_FAILURES ] && [ $MERGE_FAILURES -lt $MAX_FAILURES ]; do + if [ $((CHECK_FAILURES + MERGE_FAILURES)) -gt 0 ]; then + repush_chart + fi + + HEAD_SHA=$(git -C ../charts rev-parse HEAD) + if ! wait_for_chart_checks "$HEAD_SHA"; then + CHECK_FAILURES=$((CHECK_FAILURES + 1)) + echo "❌ PR #$PR_NUMBER failed the Cilium chart checks ($CHECK_FAILURES of $MAX_FAILURES)" + continue + fi + + echo "✅ Cilium chart checks passed for PR #$PR_NUMBER" + + # Attempt to merge + echo "Merging PR #$PR_NUMBER..." + if gh pr merge $PR_NUMBER --repo cilium/charts --rebase --delete-branch --match-head-commit "$HEAD_SHA"; then + echo "✅ Successfully merged PR #$PR_NUMBER" + MERGE_SUCCESS=true + break + fi + MERGE_FAILURES=$((MERGE_FAILURES + 1)) + echo "⚠️ Merge of PR #$PR_NUMBER failed ($MERGE_FAILURES of $MAX_FAILURES)" + done + + if [ "$MERGE_SUCCESS" = false ]; then + echo "❌ Failed to merge PR #$PR_NUMBER after $CHECK_FAILURES failed chart runs and $MERGE_FAILURES failed merges" + exit 1 + fi + + - name: Get Helm Chart Digests + id: helm_digests + shell: bash + run: | + # Extract version without 'v' prefix for Helm chart tag + VERSION="${{ inputs.version }}" + CHART_VERSION="${VERSION#v}" + + # Read digests from the file created by release tool + DIGEST_FILE="../release/helm-digests-${CHART_VERSION}.txt" + + if [ ! -f "$DIGEST_FILE" ]; then + echo "❌ Digest file not found: $DIGEST_FILE" + exit 1 + fi + + echo "⏳ Reading digests from $DIGEST_FILE" + cat "$DIGEST_FILE" + + # Extract digest for quay.io + QUAY_DIGEST=$(grep "quay.io/cilium" "$DIGEST_FILE" | awk '{print $2}') + echo "quay_digest=${QUAY_DIGEST}" >> $GITHUB_OUTPUT + echo "✅ Quay.io digest: ${QUAY_DIGEST}" + + # Waiting for the chart pull request to merge above takes tens of minutes, + # which outlives the token minted for the helm login, so cosign gets one + # of its own. The subject is the same as above. + - name: Get a fresh quay.io robot token via OIDC + id: signing-token + env: + ROBOT: ${{ vars.QUAY_ROBOT_HELM }} + run: | + oidc_token="$(curl -sSf --retry 3 --retry-all-errors \ + -H "Authorization: bearer ${ACTIONS_ID_TOKEN_REQUEST_TOKEN}" \ + "${ACTIONS_ID_TOKEN_REQUEST_URL}&audience=quay.io" \ + | jq -er '.value')" + echo "::add-mask::${oidc_token}" + + # Pass the credentials on stdin so that the OIDC token is not visible + # in the process list of the runner. + robot_token="$(printf 'user = "%s:%s"\n' "${ROBOT}" "${oidc_token}" \ + | curl -sSf --retry 3 --retry-all-errors -K - \ + "https://quay.io/oauth2/federation/robot/token" \ + | jq -er '.token')" + echo "::add-mask::${robot_token}" + + echo "token=${robot_token}" >> "$GITHUB_OUTPUT" + + - name: Sign quay.io OCI helm chart + uses: ./.github/actions/cosign + with: + image: "quay.io/${{ github.repository_owner }}/charts/cilium@${{ steps.helm_digests.outputs.quay_digest }}" + generate_sbom: 'false' + attach_sbom: 'false' + registry_username: ${{ vars.QUAY_ROBOT_HELM }} + registry_password: ${{ steps.signing-token.outputs.token }} + registry_url: 'quay.io' diff --git a/src/pkg/workflowmodel/testdata/corpus/envoyproxy__envoy.golden.json b/src/pkg/workflowmodel/testdata/corpus/envoyproxy__envoy.golden.json new file mode 100644 index 0000000000..198dca6aae --- /dev/null +++ b/src/pkg/workflowmodel/testdata/corpus/envoyproxy__envoy.golden.json @@ -0,0 +1,94 @@ +{ + "columns": [ + "Open", + "Actionable", + "Claimed", + "PR in progress", + "CI / Review", + "Merge queue", + "Merged", + "Main CI", + "Release", + "Done" + ], + "roles": { + "other": 3, + "pr-verify": 2, + "release": 1 + }, + "pipelines": [ + { + "id": "gha-github-workflows-codeql", + "role": "other", + "nodes": 1, + "levels": 1, + "events": [ + "workflow_call" + ] + }, + { + "id": "gha-github-workflows-request", + "role": "other", + "nodes": 2, + "levels": 2, + "events": [ + "workflow_call" + ] + }, + { + "id": "gha-github-workflows-request-checks", + "role": "other", + "nodes": 1, + "levels": 1, + "events": [ + "workflow_call" + ] + }, + { + "id": "gha-github-workflows-codeql-push", + "role": "pr-verify", + "nodes": 3, + "levels": 2, + "events": [ + "push", + "pull_request" + ] + }, + { + "id": "gha-github-workflows-envoy-release", + "role": "release", + "nodes": 6, + "levels": 2, + "events": [ + "release", + "workflow_dispatch" + ] + }, + { + "id": "gha-github-workflows-request-2", + "role": "pr-verify", + "nodes": 2, + "levels": 2, + "events": [ + "pull_request_target", + "push", + "schedule" + ] + } + ], + "ciSubstages": [ + "authorize · targets", + "CodeQL-Build · CodeQL-Noop · request" + ], + "findings": { + "environment-on-pr": 1, + "reusable-only": 3 + }, + "columnDiff": [ + "column \"CI / Review\" gains 2 CI substage(s) from 2 pipeline(s)", + "column \"Merged\" is no longer the cap", + "column \"Main CI\" added after merge", + "column \"Release\" added after merge", + "column \"Done\" added after merge" + ] +} diff --git a/src/pkg/workflowmodel/testdata/corpus/envoyproxy__envoy/.github/workflows/_codeql.yml b/src/pkg/workflowmodel/testdata/corpus/envoyproxy__envoy/.github/workflows/_codeql.yml new file mode 100644 index 0000000000..5d2f39b694 --- /dev/null +++ b/src/pkg/workflowmodel/testdata/corpus/envoyproxy__envoy/.github/workflows/_codeql.yml @@ -0,0 +1,94 @@ +name: CodeQL + +on: + workflow_call: + inputs: + targets: + description: Newline-separated, @envoy-prefixed Bazel targets to build + required: true + type: string + llvm-version: + description: Host LLVM major version + default: '22' + required: false + type: string + +jobs: + CodeQL-Build: + if: github.repository == 'envoyproxy/envoy' + permissions: + actions: read + contents: read + security-events: write + pull-requests: read + runs-on: ubuntu-22.04 + steps: + - uses: envoyproxy/toolshed/actions/bind-mounts@9bd3b300cf19f67cd2ef8b6e4aaa920f79894c24 # actions-v0.4.29 + if: | + ! github.event.repository.private + with: + mounts: | + - src: /mnt/workspace + target: GITHUB_WORKSPACE + chown: "runner:runner" + - src: /mnt/runner-home + target: /home/runner/.cache + chown: "runner:runner" + + - name: Free disk space + if: github.event.repository.private + uses: envoyproxy/toolshed/actions/diskspace@9bd3b300cf19f67cd2ef8b6e4aaa920f79894c24 # actions-v0.4.29 + with: + to_remove: | + /usr/local/.ghcup + /usr/local/lib/android + + - name: Checkout repository + uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 + + - name: Initialize CodeQL + uses: github/codeql-action/init@87557b9c84dde89fdd9b10e88954ac2f4248e463 # codeql-bundle-v4.36.1 + with: + languages: cpp + trap-caching: false + + - name: Install host LLVM + env: + LLVM_VERSION: ${{ inputs.llvm-version }} + run: | + sudo apt-get -qq update --error-on=any + sudo apt-get -qq install --yes libtool libtinfo5 automake autoconf curl unzip + wget -q https://apt.llvm.org/llvm.sh + chmod +x llvm.sh + sudo ./llvm.sh "${LLVM_VERSION}" + sudo apt-get -qq install --yes \ + "libc++-${LLVM_VERSION}-dev" \ + "libc++abi-${LLVM_VERSION}-dev" \ + "libclang-${LLVM_VERSION}-dev" \ + "libclang-cpp${LLVM_VERSION}-dev" \ + "lld-${LLVM_VERSION}" + # IMPORTANT: LLVM must be installed outside $GITHUB_WORKSPACE. The CodeQL C++ extractor + # treats in-tree headers as project code, producing false positives such as alert #1881. + # apt.llvm.org installs it under /usr/lib/llvm-N, which CodeQL treats as a system location. + + - name: Build + env: + TARGETS: ${{ inputs.targets }} + working-directory: bazel/tests/codeql + run: | + readarray -t targets <<< "${TARGETS}" + bazel build \ + -c fastbuild \ + --spawn_strategy=local \ + --discard_analysis_cache \ + --nouse_action_cache \ + --features=-layering_check \ + --noshow_progress \ + --noshow_loading_progress \ + "${targets[@]}" + + - name: Clean Artifacts + run: git clean -xdf + + - name: Perform CodeQL Analysis + uses: github/codeql-action/analyze@87557b9c84dde89fdd9b10e88954ac2f4248e463 # codeql-bundle-v4.36.1 diff --git a/src/pkg/workflowmodel/testdata/corpus/envoyproxy__envoy/.github/workflows/_request.yml b/src/pkg/workflowmodel/testdata/corpus/envoyproxy__envoy/.github/workflows/_request.yml new file mode 100644 index 0000000000..c495761152 --- /dev/null +++ b/src/pkg/workflowmodel/testdata/corpus/envoyproxy__envoy/.github/workflows/_request.yml @@ -0,0 +1,133 @@ +name: Request/incoming + +permissions: + contents: read + +on: + workflow_call: + secrets: + app-id: + required: true + app-key: + required: true + lock-app-id: + required: true + lock-app-key: + required: true + + # Defaults are set .github/config.yml on the `main` branch. + inputs: + config-file: + type: string + default: ./.github/config.yml + +concurrency: + group: | + ${{ github.actor != 'trigger-release-envoy[bot]' + && github.head_ref + || github.run_id + }}-${{ github.workflow }}-env-prime + cancel-in-progress: true + +env: + CI_DEBUG: ${{ (vars.CI_DEBUG || vars.RUNNER_DEBUG) && true || false }} + + +jobs: + incoming: + if: ${{ github.repository == 'envoyproxy/envoy' || vars.ENVOY_CI }} + runs-on: ubuntu-24.04 + permissions: + actions: read + contents: read + pull-requests: read + outputs: + env: ${{ steps.data.outputs.value }} + config: ${{ steps.config.outputs.config }} + steps: + - uses: envoyproxy/toolshed/actions/jq@9bd3b300cf19f67cd2ef8b6e4aaa920f79894c24 # actions-v0.4.29 + id: started + name: Create timestamp + with: + options: -r + filter: | + now + - uses: envoyproxy/toolshed/actions/github/checkout@9bd3b300cf19f67cd2ef8b6e4aaa920f79894c24 # actions-v0.4.29 + id: checkout + name: Checkout Envoy repository (requested) + with: + pr: ${{ github.event.number }} + branch: ${{ github.base_ref || github.ref_name }} + config: | + fetch-depth: ${{ startsWith(github.event_name, 'pull_request') && 1 || 2 }} + path: requested + # actions/checkout >= 7 refuses to fetch a fork PR's merge commit from a + # `pull_request_target` workflow unless told to. This job fetches it on purpose, and + # only to read it: see below, nothing from the checkout is executed. + allow-unsafe-pr-checkout: true + # This step *LOOKS AT* the repo at the point requested + # Its essential that this _job_ *MUST NOT EXECUTE ANY CODE FROM THE CHECKED OUT REPO* + # *ALL* variables collected should be treated as untrusted and should be sanitized before + # use + - name: Generate environment variables from commit + uses: envoyproxy/toolshed/actions/envoy/ci/request@9bd3b300cf19f67cd2ef8b6e4aaa920f79894c24 # actions-v0.4.29 + id: env + with: + branch-name: ${{ steps.checkout.outputs.branch-name }} + config-file: ${{ inputs.config-file }} + merge-commit: ${{ steps.checkout.outputs.merge-commit }} + started: ${{ steps.started.outputs.value }} + token: ${{ secrets.GITHUB_TOKEN }} + vars: ${{ toJSON(vars) }} + working-directory: requested + + - name: Request summary + id: summary + uses: envoyproxy/toolshed/actions/github/env/summary@9bd3b300cf19f67cd2ef8b6e4aaa920f79894c24 # actions-v0.4.29 + with: + actor: ${{ toJSON(fromJSON(steps.env.outputs.data).request.actor) }} + base-sha: ${{ fromJSON(steps.env.outputs.data).request.base-sha }} + link: ${{ format('https://github.com/{0}/actions/runs/{1}', github.repository, github.run_id) }} + output-path: GITHUB_STEP_SUMMARY + pr: ${{ github.event.number }} + data: ${{ steps.env.outputs.data }} + tables: ${{ toJSON(fromJSON(steps.env.outputs.data).config.tables) }} + icon: ${{ fromJSON(steps.env.outputs.data).config.envoy.icon }} + message: ${{ fromJSON(steps.env.outputs.data).request.message }} + ref: ${{ fromJSON(steps.env.outputs.data).request.ref }} + sha: ${{ fromJSON(steps.env.outputs.data).request.sha }} + target-branch: ${{ fromJSON(steps.env.outputs.data).request.target-branch }} + + - name: Environment data + uses: envoyproxy/toolshed/actions/jq@9bd3b300cf19f67cd2ef8b6e4aaa920f79894c24 # actions-v0.4.29 + id: data + with: + input: | + env: ${{ steps.env.outputs.data }} + title: ${{ steps.summary.outputs.title }} + link: ${{ format('https://github.com/{0}/actions/runs/{1}', github.repository, github.run_id) }} + summary: ${{ steps.summary.outputs.summary }} + input-format: yaml + filter: | + .title as $title + | .env.config.envoy.icon as $icon + | .link as $link + | "\($icon) Request ([\($title)](\($link)))" as $linkedTitle + | .summary as $summary + | .env + | .summary = { + $summary, + $title, + $link, + "linked-title": $linkedTitle} + | del(.config.tables) + + checks: + if: ${{ github.repository == 'envoyproxy/envoy' || vars.ENVOY_CI }} + needs: incoming + uses: ./.github/workflows/_request_checks.yml + secrets: + app-id: ${{ secrets.app-id }} + app-key: ${{ secrets.app-key }} + with: + env: ${{ needs.incoming.outputs.env }} diff --git a/src/pkg/workflowmodel/testdata/corpus/envoyproxy__envoy/.github/workflows/_request_checks.yml b/src/pkg/workflowmodel/testdata/corpus/envoyproxy__envoy/.github/workflows/_request_checks.yml new file mode 100644 index 0000000000..fed9740f61 --- /dev/null +++ b/src/pkg/workflowmodel/testdata/corpus/envoyproxy__envoy/.github/workflows/_request_checks.yml @@ -0,0 +1,133 @@ +name: Workflow start +# This workflow is only required for externally triggered jobs that need to manually +# set the check status for a commit/PR + +permissions: + contents: read + +on: + workflow_call: + secrets: + app-id: + required: true + app-key: + required: true + inputs: + details-url: + type: string + default: >- + https://github.com/envoyproxy/envoy/tree/main/.github/workflows + env: + type: string + required: true + run-summary: + type: string + default: >- + The check will start once any required jobs have completed and a VM becomes available + run-title: + type: string + default: >- + Waiting for check ... + skipped-summary: + type: string + default: >- + This check was not triggered in this CI run + skipped-title: + type: string + default: >- + Check was skipped + template-run-text: + type: string + default: | + ## \($icon) Check run pending + + ## Details of the check run will be provided here once it has started. + + ### Check started by + + +env: + CI_DEBUG: ${{ (vars.CI_DEBUG || vars.RUNNER_DEBUG) && true || false }} + + +jobs: + start: + runs-on: ${{ fromJSON(inputs.env).config.ci.agent-ubuntu }} + name: Start checks + steps: + - uses: envoyproxy/toolshed/actions/jq@9bd3b300cf19f67cd2ef8b6e4aaa920f79894c24 # actions-v0.4.29 + id: check-config + name: Prepare check data + with: + print-result: ${{ fromJSON(env.CI_DEBUG || 'false') && true || false }} + input: ${{ inputs.env }} + filter: | + . as $env + | .config.envoy.icon as $icon + | {} + | .["head_sha"] = $env.request.sha + | .details_url = "${{ inputs.details-url }}" + | {run: ., skipped: ., request: $env.summary.summary} + | .run.output.title = "${{ inputs.run-title }}" + | .run.output.summary = "${{ inputs.run-summary }}" + | .run.output.text = "${{ inputs.template-run-text }}" + | .run.status = "queued" + | .skipped.status = "completed" + | .skipped.conclusion = "skipped" + | .skipped.output.title = "${{ inputs.skipped-title }}" + | .skipped.output.summary = "${{ inputs.skipped-summary }}" + | .skipped.output.text = "" + + - uses: envoyproxy/toolshed/actions/appauth@9bd3b300cf19f67cd2ef8b6e4aaa920f79894c24 # actions-v0.4.29 + name: Appauth + id: appauth + with: + app_id: ${{ secrets.app-id }} + key: ${{ secrets.app-key }} + - uses: envoyproxy/toolshed/actions/github/checks@9bd3b300cf19f67cd2ef8b6e4aaa920f79894c24 # actions-v0.4.29 + name: Start checks + id: checks + with: + checks: ${{ toJSON(fromJSON(inputs.env).checks) }} + config: ${{ steps.check-config.outputs.value }} + text-extra: | + ## ${{ fromJSON(inputs.env).summary.linked-title }} + + ${{ fromJSON(inputs.env).summary.summary }} + token: ${{ steps.appauth.outputs.token }} + - uses: envoyproxy/toolshed/actions/json/table@9bd3b300cf19f67cd2ef8b6e4aaa920f79894c24 # actions-v0.4.29 + name: Summary + with: + collapse-open: true + json: | + {"checks": ${{ steps.checks.outputs.checks }}, + "config": ${{ toJSON(fromJSON(inputs.env).checks) }}} + filter: | + .checks + heading: >- + ${{ fromJSON(inputs.env).config.envoy.icon }} Checks + mutate-cells: | + .cell as $cell + | .row as $row + | .table as $table + | $cell + | if ($row | index($cell) == 0) then + $table.data.config[$cell].name + elif ($table.data.config[$row[0]].action != "SKIP") then + "[started](http://github.com/${{ github.repository }}/runs/\($cell))" + else "skipped" end + output-path: GITHUB_STEP_SUMMARY + title: Checks started/skipped + + - uses: envoyproxy/toolshed/actions/github/env/save@9bd3b300cf19f67cd2ef8b6e4aaa920f79894c24 # actions-v0.4.29 + name: Save env + id: data + with: + env: ${{ inputs.env }} + env-filter: | + ${{ steps.checks.outputs.checks }} as $checksStarted + | .checks + |= with_entries( + if $checksStarted[.key] != "skipped" then + .value["check-id"] = $checksStarted[.key] + else . end) diff --git a/src/pkg/workflowmodel/testdata/corpus/envoyproxy__envoy/.github/workflows/codeql-push.yml b/src/pkg/workflowmodel/testdata/corpus/envoyproxy__envoy/.github/workflows/codeql-push.yml new file mode 100644 index 0000000000..09680b566d --- /dev/null +++ b/src/pkg/workflowmodel/testdata/corpus/envoyproxy__envoy/.github/workflows/codeql-push.yml @@ -0,0 +1,108 @@ +name: CodeQL/push + +permissions: + contents: read + +on: + push: + branches: + - main + pull_request: + branches: + - main + +concurrency: + group: ${{ github.head_ref || github.run_id }}-${{ github.workflow }} + cancel-in-progress: true + +env: + SEARCH_FOLDER: //source/common/... + +jobs: + targets: + if: github.repository == 'envoyproxy/envoy' + permissions: + contents: read + runs-on: ubuntu-22.04 + outputs: + build_targets: ${{ steps.build-targets.outputs.targets }} + cpp_changed: ${{ steps.detect.outputs.cpp_changed }} + steps: + - name: Checkout repository + uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 + with: + fetch-depth: 2 + + - name: Detect C++ changes + id: detect + run: | + if [[ "${{ github.event_name }}" == "pull_request" ]]; then + git fetch --depth=1 "${GITHUB_SERVER_URL}/${GITHUB_REPOSITORY}" main 2>/dev/null + TO_OTHER=FETCH_HEAD + else + TO_OTHER=HEAD^1 + fi + echo "to_other=${TO_OTHER}" >> "$GITHUB_OUTPUT" + if git diff --name-only HEAD "${TO_OTHER}" -- source/common/ envoy/ | grep -q .; then + echo "cpp_changed=true" >> "$GITHUB_OUTPUT" + else + echo "cpp_changed=false" >> "$GITHUB_OUTPUT" + fi + + - name: Get build targets + id: build-targets + if: steps.detect.outputs.cpp_changed == 'true' + env: + TO_OTHER: ${{ steps.detect.outputs.to_other }} + run: | + # TODO(phlax): Shift this to an action + compare_head () { + while IFS= read -r line; do + if [[ -n "$line" ]]; then + bazel query "rdeps($SEARCH_FOLDER, $line, 1)" 2>/dev/null + fi + done < <(git diff --name-only HEAD "${TO_OTHER}" -- source/common/* envoy/*) + } + BUILD_TARGETS="$(compare_head | grep -v '\.cc\|\.h' | sort -u | head -n 3)" + if [[ -z "$BUILD_TARGETS" ]]; then + BUILD_TARGETS="//source/common/common:assert_lib" + fi + { + echo 'targets<> "$GITHUB_OUTPUT" + + CodeQL-Build: + if: needs.targets.outputs.cpp_changed == 'true' + needs: targets + permissions: + actions: read + contents: read + security-events: write + pull-requests: read + uses: ./.github/workflows/_codeql.yml + with: + targets: ${{ needs.targets.outputs.build_targets }} + + CodeQL-Noop: + if: needs.targets.outputs.cpp_changed != 'true' + needs: targets + permissions: + actions: read + contents: read + security-events: write + runs-on: ubuntu-22.04 + steps: + - name: Checkout repository + uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 + + # No-op analysis satisfies Scorecard SAST when no C++ changes are present. + - name: Initialize CodeQL (noop) + uses: github/codeql-action/init@87557b9c84dde89fdd9b10e88954ac2f4248e463 # codeql-bundle-v4.36.1 + with: + languages: javascript + trap-caching: false + + - name: Perform CodeQL Analysis + uses: github/codeql-action/analyze@87557b9c84dde89fdd9b10e88954ac2f4248e463 # codeql-bundle-v4.36.1 diff --git a/src/pkg/workflowmodel/testdata/corpus/envoyproxy__envoy/.github/workflows/envoy-release.yml b/src/pkg/workflowmodel/testdata/corpus/envoyproxy__envoy/.github/workflows/envoy-release.yml new file mode 100644 index 0000000000..9d093378bd --- /dev/null +++ b/src/pkg/workflowmodel/testdata/corpus/envoyproxy__envoy/.github/workflows/envoy-release.yml @@ -0,0 +1,207 @@ +name: Envoy/release + +permissions: + contents: read + +on: + # Post-publish chaining lives in _publish_release.yml -> _release_chain.yml. + release: + types: + - published + workflow_dispatch: + inputs: + task: + description: Select a task + required: true + default: create-release + type: choice + options: + - create-release + - reopen-branch + - sync-version-histories + - deprecate-guards + branch: + description: >- + Target branch, eg `release/v1.35` + (default: the branch the workflow is run from; not used by `deprecate-guards`) + type: string + dry-run: + type: boolean + default: false + pr: + type: boolean + default: true + description: Create a PR + pr-message: + description: Additional message for PR, eg to fix an issue or additional signoff (optional) + wip: + type: boolean + default: false + description: WIP + author: + description: >- + Author: User/email, eg 'Myname ' + (used by create-release, default: `changelogs/summary.md` last committer) + summary: + type: boolean + default: true + description: Use changelog summary (required to publish release) + recreate: + type: boolean + default: false + description: >- + Delete existing PR/branch + (closes any open bot PR and deletes the bot branch before recreating it) + +env: + COMMITTER_NAME: publish-envoy[bot] + COMMITTER_EMAIL: 140627008+publish-envoy[bot]@users.noreply.github.com + + +jobs: + validate: + if: github.event_name == 'workflow_dispatch' && inputs.task != 'deprecate-guards' + runs-on: ubuntu-24.04 + name: Validate dispatch + outputs: + branch: ${{ steps.validate.outputs.branch }} + steps: + - id: validate + env: + REF: ${{ github.ref_name }} + BRANCH: ${{ inputs.branch }} + run: | + if [[ "$REF" != "main" && ! "$REF" =~ ^release/v[0-9]+\.[0-9]+$ ]]; then + echo "::error::Release tasks must be dispatched from main or a release/vX.Y branch (got ${REF})" + exit 1 + fi + if [[ -n "$BRANCH" && "$REF" != "main" && "$BRANCH" != "$REF" ]]; then + echo "::error::Dispatches from ${REF} cannot target a different branch (${BRANCH}); dispatch from main instead" + exit 1 + fi + echo "branch=${BRANCH:-$REF}" >> "$GITHUB_OUTPUT" + + ## Triggerable actions + + # Create a release commit, when landed this will publish. + create_release: + needs: validate + if: inputs.task == 'create-release' + name: Create release + uses: ./.github/workflows/_release_task.yml + secrets: + ENVOY_CI_PUBLISH_APP_ID: ${{ secrets.ENVOY_CI_PUBLISH_APP_ID }} + ENVOY_CI_PUBLISH_APP_KEY: ${{ secrets.ENVOY_CI_PUBLISH_APP_KEY }} + with: + task: create-release + branch: ${{ needs.validate.outputs.branch }} + recreate: ${{ inputs.recreate }} + pr: ${{ inputs.pr }} + pr-message: ${{ inputs.pr-message }} + wip: ${{ inputs.wip }} + author: ${{ inputs.author }} + summary: ${{ inputs.summary }} + + # Re-open a branch. + reopen-branch: + needs: validate + if: inputs.task == 'reopen-branch' + name: Re-open branch + uses: ./.github/workflows/_release_task.yml + secrets: + ENVOY_CI_PUBLISH_APP_ID: ${{ secrets.ENVOY_CI_PUBLISH_APP_ID }} + ENVOY_CI_PUBLISH_APP_KEY: ${{ secrets.ENVOY_CI_PUBLISH_APP_KEY }} + with: + task: reopen-branch + branch: ${{ needs.validate.outputs.branch }} + recreate: ${{ inputs.recreate }} + pr: ${{ inputs.pr }} + pr-message: ${{ inputs.pr-message }} + wip: ${{ inputs.wip }} + summary: ${{ inputs.summary }} + + sync_version_histories: + needs: validate + if: inputs.task == 'sync-version-histories' + name: Sync version histories + uses: ./.github/workflows/_release_task.yml + secrets: + ENVOY_CI_PUBLISH_APP_ID: ${{ secrets.ENVOY_CI_PUBLISH_APP_ID }} + ENVOY_CI_PUBLISH_APP_KEY: ${{ secrets.ENVOY_CI_PUBLISH_APP_KEY }} + with: + task: sync-version-histories + branch: ${{ needs.validate.outputs.branch }} + recreate: ${{ inputs.recreate }} + pr: ${{ inputs.pr }} + pr-message: ${{ inputs.pr-message }} + + deprecate_guards: + runs-on: ubuntu-24.04 + if: >- + ${{ (github.event_name == 'workflow_dispatch' + && inputs.task == 'deprecate-guards') + || (github.event_name == 'release' + && endsWith(github.ref, '.0')) }} + name: Deprecate guards + steps: + - id: appauth + name: App auth + uses: envoyproxy/toolshed/actions/appauth@9bd3b300cf19f67cd2ef8b6e4aaa920f79894c24 # actions-v0.4.29 + with: + app_id: ${{ secrets.ENVOY_CI_PUBLISH_APP_ID }} + key: ${{ secrets.ENVOY_CI_PUBLISH_APP_KEY }} + - id: checkout + name: Checkout Envoy repository + uses: envoyproxy/toolshed/actions/github/checkout@9bd3b300cf19f67cd2ef8b6e4aaa920f79894c24 # actions-v0.4.29 + with: + config: | + fetch-depth: 0 + - name: Run deprecation tool + run: | + bazel run --config=ci \ + //tools/deprecate_guards \ + -- \ + ${{ ! inputs.dry-run && ' --create-issues' || '' }} \ + ${{ github.repository != 'envoyproxy/envoy' + && format('--staging-repo {0}', github.repository) + || '' }} + env: + GITHUB_TOKEN: ${{ steps.appauth.outputs.token }} + + ## Triggered actions + + # On release to `main`: + # - fork the branch to a release branch + # - add an initial dev commit + # - remove anything unwanted + # - push branch + create_release_branch: + runs-on: ubuntu-24.04 + if: github.event_name == 'release' && endsWith(github.ref, '.0') + name: Create release branch + steps: + - id: appauth + name: App auth + uses: envoyproxy/toolshed/actions/appauth@9bd3b300cf19f67cd2ef8b6e4aaa920f79894c24 # actions-v0.4.29 + with: + app_id: ${{ secrets.ENVOY_CI_PUBLISH_APP_ID }} + key: ${{ secrets.ENVOY_CI_PUBLISH_APP_KEY }} + + - name: Checkout repository + uses: envoyproxy/toolshed/actions/github/checkout@9bd3b300cf19f67cd2ef8b6e4aaa920f79894c24 # actions-v0.4.29 + with: + committer-name: ${{ env.COMMITTER_NAME }} + committer-email: ${{ env.COMMITTER_EMAIL }} + token: ${{ steps.appauth.outputs.token }} + - name: Create release branch + run: | + version="$(cut -d- -f1 < VERSION.txt | cut -d. -f-2)" + release_branch="release/v${version}" + commit_sha="$(git rev-parse HEAD)" + echo "Creating ${release_branch} from ${commit_sha}" + git checkout -b "$release_branch" + bazel run @envoy_repo//:dev -- --patch + git rm -rf .github/workflows/mobile*yml + git commit . -m "repo: Remove mobile ci for release branch" + git log + git push origin "$release_branch" diff --git a/src/pkg/workflowmodel/testdata/corpus/envoyproxy__envoy/.github/workflows/request.yml b/src/pkg/workflowmodel/testdata/corpus/envoyproxy__envoy/.github/workflows/request.yml new file mode 100644 index 0000000000..92253d32f4 --- /dev/null +++ b/src/pkg/workflowmodel/testdata/corpus/envoyproxy__envoy/.github/workflows/request.yml @@ -0,0 +1,74 @@ +# This file must live on every branch and pass necessary secrets and permissions +# to initiate the request +name: Request + +permissions: + contents: read + +on: + pull_request_target: + branches: + - main + - release/v* + - ci/testing + push: + branches: + - main + - release/v* + - ci/testing + schedule: + - cron: '30 6 * * *' + +concurrency: + group: | + ${{ github.head_ref + || github.run_id + }}-${{ github.workflow }}-request + cancel-in-progress: true + + +jobs: + # Envoy (and mirror repos) have an environment setup that requires maintainer approval + # to use it. This CI checks if the request is from a first-time contributor, and in that + # case it uses the environment and requires the permission to proceed. + authorize: + if: >- + ${{ github.repository == 'envoyproxy/envoy' + || (vars.ENVOY_CI && github.event_name != 'schedule') + || (vars.ENVOY_SCHEDULED_CI && github.event_name == 'schedule') }} + runs-on: ubuntu-24.04 + environment: >- + ${{ github.event_name == 'pull_request_target' + && github.event.pull_request.author_association != 'MEMBER' + && github.event.pull_request.author_association != 'COLLABORATOR' + && github.event.pull_request.author_association != 'CONTRIBUTOR' + && github.event.pull_request.author_association != 'OWNER' + && 'external-contributors' + || '' }} + steps: + - run: | + echo "Authorized" + echo " Event: ${{ github.event_name }}" + echo " Author association: ${{ github.event.pull_request.author_association }}" + + request: + needs: authorize + permissions: + actions: write + contents: read + packages: read + # required to fetch merge commit + pull-requests: read + secrets: + # these are required to start checks + app-key: ${{ secrets.ENVOY_CI_APP_KEY }} + app-id: ${{ secrets.ENVOY_CI_APP_ID }} + lock-app-key: ${{ secrets.ENVOY_CI_MUTEX_APP_KEY }} + lock-app-id: ${{ secrets.ENVOY_CI_MUTEX_APP_ID }} + # For branches this can be pinned to a specific version if required + # NB: `uses` cannot be dynamic so it _must_ be hardcoded anywhere it is read + uses: envoyproxy/envoy/.github/workflows/_request.yml@main + if: >- + ${{ github.repository == 'envoyproxy/envoy' + || (vars.ENVOY_CI && github.event_name != 'schedule') + || (vars.ENVOY_SCHEDULED_CI && github.event_name == 'schedule') }} diff --git a/src/pkg/workflowmodel/testdata/corpus/konflux-ci__konflux-ci.golden.json b/src/pkg/workflowmodel/testdata/corpus/konflux-ci__konflux-ci.golden.json new file mode 100644 index 0000000000..ea30911079 --- /dev/null +++ b/src/pkg/workflowmodel/testdata/corpus/konflux-ci__konflux-ci.golden.json @@ -0,0 +1,46 @@ +{ + "columns": [ + "Open", + "Actionable", + "Claimed", + "PR in progress", + "CI / Review", + "Merge queue", + "Merged", + "Main CI", + "Done" + ], + "roles": { + "pr-verify": 1 + }, + "pipelines": [ + { + "id": "tkn-tekton-build-pipeline", + "role": "pr-verify", + "nodes": 18, + "levels": 7, + "events": [ + "cel", + "pull_request", + "push", + "cel", + "cel" + ] + } + ], + "ciSubstages": [ + "init", + "clone-repository", + "prefetch-dependencies", + "build-images", + "build-image-index", + "apply-tags · build-source-image · clamav-scan +10" + ], + "findings": {}, + "columnDiff": [ + "column \"CI / Review\" gains 6 CI substage(s) from 1 pipeline(s)", + "column \"Merged\" is no longer the cap", + "column \"Main CI\" added after merge", + "column \"Done\" added after merge" + ] +} diff --git a/src/pkg/workflowmodel/testdata/corpus/konflux-ci__konflux-ci/.tekton/build-pipeline.yaml b/src/pkg/workflowmodel/testdata/corpus/konflux-ci__konflux-ci/.tekton/build-pipeline.yaml new file mode 100644 index 0000000000..d8cfbe6fa4 --- /dev/null +++ b/src/pkg/workflowmodel/testdata/corpus/konflux-ci__konflux-ci/.tekton/build-pipeline.yaml @@ -0,0 +1,596 @@ +apiVersion: tekton.dev/v1 +kind: Pipeline +metadata: + name: build-pipeline +spec: + description: | + This pipeline is ideal for building multi-arch container images from a Containerfile while maintaining trust after pipeline customization. + + _Uses `buildah` to create a multi-platform container image leveraging [trusted artifacts](https://konflux-ci.dev/architecture/ADR/0036-trusted-artifacts.html). It also optionally creates a source image and runs some build-time tests. This pipeline requires that the [multi platform controller](https://github.com/konflux-ci/multi-platform-controller) is deployed and configured on your Konflux instance. Information is shared between tasks using OCI artifacts instead of PVCs. EC will pass the [`trusted_task.trusted`](https://enterprisecontract.dev/docs/ec-policies/release_policy.html#trusted_task__trusted) policy as long as all data used to build the artifact is generated from trusted tasks. + This pipeline is pushed as a Tekton bundle to [quay.io](https://quay.io/repository/konflux-ci/tekton-catalog/pipeline-docker-build-multi-platform-oci-ta?tab=tags)_ + params: + - description: Source Repository URL + name: git-url + type: string + - default: "" + description: Revision of the Source Repository + name: revision + type: string + - description: Fully Qualified Output Image + name: output-image + type: string + - default: . + description: Path to the source code of an application's component from where to build image. + name: path-context + type: string + - default: Dockerfile + description: Path to the Dockerfile inside the context specified by parameter path-context + name: dockerfile + type: string + - default: "false" + description: Skip checks against built image + name: skip-checks + type: string + - default: "true" + description: Execute the build with network isolation + name: hermetic + type: string + - default: '[{"type": "gomod", "path": "operator"}]' + description: Build dependencies to be prefetched by Cachi2 + name: prefetch-input + type: string + - default: "" + description: Image tag expiration time, time values could be something like 1h, 2d, 3w for hours, days, and weeks, respectively. + name: image-expires-after + type: string + - default: "false" + description: Build a source image. + name: build-source-image + type: string + - default: "true" + description: Add built image into an OCI image index + name: build-image-index + type: string + - default: docker + description: The format for the resulting image's mediaType. Valid values are oci or docker. + name: buildah-format + type: string + - default: "false" + description: Enable cache proxy configuration + name: enable-cache-proxy + - default: [] + description: Array of --build-arg values ("arg=value" strings) for buildah + name: build-args + type: array + - default: "" + description: Path to a file with build arguments for buildah, see https://www.mankier.com/1/buildah-build#--build-arg-file + name: build-args-file + type: string + - default: "false" + description: Whether to enable privileged mode, should be used only with remote VMs + name: privileged-nested + type: string + - default: + - "linux/x86_64" + - "linux/arm64" + description: List of platforms to build the container images on. The available set of values is determined by the configuration of the multi-platform-controller. + name: build-platforms + type: array + - name: enable-package-registry-proxy + default: 'true' + description: Use the package registry proxy when prefetching dependencies + type: string + - name: sast-target-dirs + type: string + default: . + description: Target directories to scan with SAST tools. Multiple values should be separated with commas. + - name: source-date-epoch + type: string + default: '' + description: Sets the image created time and the SOURCE_DATE_EPOCH build argument. On its own, it does not change file timestamps inside the layers (set rewrite-timestamp to "true" for that). Leave empty to keep the actual build time. + - name: rewrite-timestamp + type: string + default: 'false' + description: When "true", clamp file modification times in the image layers to at most source-date-epoch. Does nothing unless source-date-epoch is set. + - name: omit-history + type: string + default: 'false' + description: When "true", omit the build history (history timestamps, layer metadata, etc.) from the resulting image. + results: + - description: "" + name: IMAGE_URL + value: $(tasks.build-image-index.results.IMAGE_URL) + - description: "" + name: IMAGE_DIGEST + value: $(tasks.build-image-index.results.IMAGE_DIGEST) + - description: "" + name: CHAINS-GIT_URL + value: $(tasks.clone-repository.results.url) + - description: "" + name: CHAINS-GIT_COMMIT + value: $(tasks.clone-repository.results.commit) + tasks: + - name: init + params: + - name: enable-cache-proxy + value: $(params.enable-cache-proxy) + taskRef: + params: + - name: name + value: init + - name: bundle + value: quay.io/konflux-ci/tekton-catalog/task-init:0.4.3@sha256:aa7ede4ed5470bc128f62701f3d7ba1fce87ba996eed9f59dd65eb62b8de437e + - name: kind + value: task + resolver: bundles + - name: clone-repository + params: + - name: url + value: $(params.git-url) + - name: revision + value: $(params.revision) + - name: ociStorage + value: $(params.output-image).git + - name: ociArtifactExpiresAfter + value: $(params.image-expires-after) + runAfter: + - init + taskRef: + params: + - name: name + value: git-clone-oci-ta + - name: bundle + value: quay.io/konflux-ci/tekton-catalog/task-git-clone-oci-ta:0.2.6@sha256:26fc26a0bde8081ed4244024f56d4fe79dedca6a9ab37fbb09c24e34c02456fc + - name: kind + value: task + resolver: bundles + workspaces: + - name: basic-auth + workspace: git-auth + - name: prefetch-dependencies + params: + - name: input + value: $(params.prefetch-input) + - name: SOURCE_ARTIFACT + value: $(tasks.clone-repository.results.SOURCE_ARTIFACT) + - name: ociStorage + value: $(params.output-image).prefetch + - name: ociArtifactExpiresAfter + value: $(params.image-expires-after) + - name: enable-package-registry-proxy + value: $(params.enable-package-registry-proxy) + runAfter: + - clone-repository + taskRef: + params: + - name: name + value: prefetch-dependencies-oci-ta + - name: bundle + value: quay.io/konflux-ci/tekton-catalog/task-prefetch-dependencies-oci-ta:0.11.0@sha256:3238f1eed5a012de9d99845b02717c512c89f2b7eb14c19ce61c709205ef50cc + - name: kind + value: task + resolver: bundles + workspaces: + - name: git-basic-auth + workspace: git-auth + - name: netrc + workspace: netrc + - name: build-images + matrix: + params: + - name: PLATFORM + value: + - $(params.build-platforms) + params: + - name: IMAGE + value: $(params.output-image) + - name: DOCKERFILE + value: $(params.dockerfile) + - name: CONTEXT + value: $(params.path-context) + - name: HERMETIC + value: $(params.hermetic) + - name: PREFETCH_INPUT + value: $(params.prefetch-input) + - name: IMAGE_EXPIRES_AFTER + value: $(params.image-expires-after) + - name: COMMIT_SHA + value: $(tasks.clone-repository.results.commit) + - name: BUILD_ARGS + value: + - $(params.build-args[*]) + - name: PRIVILEGED_NESTED + value: $(params.privileged-nested) + - name: SOURCE_URL + value: $(tasks.clone-repository.results.url) + - name: BUILDAH_FORMAT + value: $(params.buildah-format) + - name: HTTP_PROXY + value: $(tasks.init.results.http-proxy) + - name: NO_PROXY + value: $(tasks.init.results.no-proxy) + - name: SOURCE_ARTIFACT + value: $(tasks.prefetch-dependencies.results.SOURCE_ARTIFACT) + - name: INHERIT_BASE_IMAGE_LABELS + value: "false" + - name: IMAGE_APPEND_PLATFORM + value: "true" + - name: SOURCE_DATE_EPOCH + value: $(params.source-date-epoch) + - name: REWRITE_TIMESTAMP + value: $(params.rewrite-timestamp) + - name: OMIT_HISTORY + value: $(params.omit-history) + - name: BUILD_ARGS_FILES + value: + - $(params.build-args-file) + - name: PREFETCH_ARTIFACT + value: $(tasks.prefetch-dependencies.results.PREFETCH_ARTIFACT) + runAfter: + - prefetch-dependencies + taskRef: + params: + - name: name + value: buildah-remote-oci-ta + - name: bundle + value: quay.io/konflux-ci/tekton-catalog/task-buildah-remote-oci-ta:0.14.1@sha256:3419466b403188efc156532f85509882f1462813c12d9fdd0c3c9c02bedc392f + - name: kind + value: task + resolver: bundles + - name: build-image-index + params: + - name: IMAGE + value: $(params.output-image) + - name: ALWAYS_BUILD_INDEX + value: $(params.build-image-index) + - name: IMAGES + value: + - $(tasks.build-images.results.IMAGE_REF[*]) + - name: BUILDAH_FORMAT + value: $(params.buildah-format) + runAfter: + - build-images + taskRef: + params: + - name: name + value: build-image-index + - name: bundle + value: quay.io/konflux-ci/tekton-catalog/task-build-image-index:0.4@sha256:52ea41a78adb3307e69d156169c37912772918ca77d5dcb91d49f2313b4ad663 + - name: kind + value: task + resolver: bundles + - name: build-source-image + params: + - name: BINARY_IMAGE + value: $(tasks.build-image-index.results.IMAGE_URL) + - name: BINARY_IMAGE_DIGEST + value: $(tasks.build-image-index.results.IMAGE_DIGEST) + - name: SOURCE_ARTIFACT + value: $(tasks.prefetch-dependencies.results.SOURCE_ARTIFACT) + - name: PREFETCH_ARTIFACT + value: $(tasks.prefetch-dependencies.results.PREFETCH_ARTIFACT) + runAfter: + - build-image-index + taskRef: + params: + - name: name + value: source-build-oci-ta + - name: bundle + value: quay.io/konflux-ci/tekton-catalog/task-source-build-oci-ta:0.4.0@sha256:e5c2269eb4bb4a44f8c2c67dd6f224566baf284347844080fa9f9aace0513dea + - name: kind + value: task + resolver: bundles + when: + - input: $(params.build-source-image) + operator: in + values: + - "true" + - name: deprecated-base-image-check + params: + - name: IMAGE_URL + value: $(tasks.build-image-index.results.IMAGE_URL) + - name: IMAGE_DIGEST + value: $(tasks.build-image-index.results.IMAGE_DIGEST) + runAfter: + - build-image-index + taskRef: + params: + - name: name + value: deprecated-image-check + - name: bundle + value: quay.io/konflux-ci/tekton-catalog/task-deprecated-image-check:0.5@sha256:f86e420c8af92a8ed9ca8f82dee3cd14f8ba03cc592583595ea78642513f00c2 + - name: kind + value: task + resolver: bundles + when: + - input: $(params.skip-checks) + operator: in + values: + - "false" + - name: roxctl-scan + params: + - name: image-digest + value: $(tasks.build-image-index.results.IMAGE_DIGEST) + - name: image-url + value: $(tasks.build-image-index.results.IMAGE_URL) + runAfter: + - build-image-index + taskRef: + params: + - name: name + value: roxctl-scan + - name: bundle + value: quay.io/konflux-ci/tekton-catalog/task-roxctl-scan:0.2@sha256:d5e7fb9798e4d5e342a249d6e2972900a15c61f81eda30b1369772622171a31d + - name: kind + value: task + resolver: bundles + when: + - input: $(params.skip-checks) + operator: in + values: + - "false" + - name: ecosystem-cert-preflight-checks + matrix: + params: + - name: platform + value: + - $(params.build-platforms) + params: + - name: image-url + value: $(tasks.build-image-index.results.IMAGE_URL) + runAfter: + - build-image-index + taskRef: + params: + - name: name + value: ecosystem-cert-preflight-checks + - name: bundle + value: quay.io/konflux-ci/tekton-catalog/task-ecosystem-cert-preflight-checks:0.2@sha256:0b1783a11d9066cd4a26fb20251785cf7283258ebe7a1f4e9c2247f6d57b7064 + - name: kind + value: task + resolver: bundles + when: + - input: $(params.skip-checks) + operator: in + values: + - "false" + - name: sast-snyk-check + params: + - name: image-digest + value: $(tasks.build-image-index.results.IMAGE_DIGEST) + - name: image-url + value: $(tasks.build-image-index.results.IMAGE_URL) + - name: SOURCE_ARTIFACT + value: $(tasks.prefetch-dependencies.results.SOURCE_ARTIFACT) + - name: TARGET_DIRS + value: $(params.sast-target-dirs) + runAfter: + - build-image-index + taskRef: + params: + - name: name + value: sast-snyk-check-oci-ta + - name: bundle + value: quay.io/konflux-ci/tekton-catalog/task-sast-snyk-check-oci-ta:0.5.1@sha256:53443febc3a87ea6d4baacffc70d6f7dbdd1b4e69e0852337661097c73433519 + - name: kind + value: task + resolver: bundles + when: + - input: $(params.skip-checks) + operator: in + values: + - "false" + - name: clamav-scan + matrix: + params: + - name: image-arch + value: + - $(params.build-platforms) + params: + - name: image-digest + value: $(tasks.build-image-index.results.IMAGE_DIGEST) + - name: image-url + value: $(tasks.build-image-index.results.IMAGE_URL) + runAfter: + - build-image-index + taskRef: + params: + - name: name + value: clamav-scan + - name: bundle + value: quay.io/konflux-ci/tekton-catalog/task-clamav-scan:0.3.6@sha256:ecd3e67c0384219d70bcdf378aa1b4cc83af1ca98c68ac3211263ad2edaefcbe + - name: kind + value: task + resolver: bundles + when: + - input: $(params.skip-checks) + operator: in + values: + - "false" + - name: sast-coverity-check + params: + - name: image-digest + value: $(tasks.build-image-index.results.IMAGE_DIGEST) + - name: image-url + value: $(tasks.build-image-index.results.IMAGE_URL) + - name: IMAGE + value: $(params.output-image) + - name: DOCKERFILE + value: $(params.dockerfile) + - name: CONTEXT + value: $(params.path-context) + - name: HERMETIC + value: $(params.hermetic) + - name: PREFETCH_INPUT + value: $(params.prefetch-input) + - name: IMAGE_EXPIRES_AFTER + value: $(params.image-expires-after) + - name: COMMIT_SHA + value: $(tasks.clone-repository.results.commit) + - name: BUILD_ARGS + value: + - $(params.build-args[*]) + - name: BUILD_ARGS_FILE + value: $(params.build-args-file) + - name: SOURCE_ARTIFACT + value: $(tasks.prefetch-dependencies.results.SOURCE_ARTIFACT) + - name: CACHI2_ARTIFACT + value: $(tasks.prefetch-dependencies.results.PREFETCH_ARTIFACT) + - name: TARGET_DIRS + value: $(params.sast-target-dirs) + runAfter: + - coverity-availability-check + taskRef: + params: + - name: name + value: sast-coverity-check-oci-ta + - name: bundle + value: quay.io/konflux-ci/tekton-catalog/task-sast-coverity-check-oci-ta:0.3@sha256:e92d00ed858233d0096627861192d3e4fc013cf1559c0d0b0ea0657d3377ce75 + - name: kind + value: task + resolver: bundles + when: + - input: $(params.skip-checks) + operator: in + values: + - "false" + - input: $(tasks.coverity-availability-check.results.STATUS) + operator: in + values: + - success + - name: coverity-availability-check + runAfter: + - build-image-index + taskRef: + params: + - name: name + value: coverity-availability-check + - name: bundle + value: quay.io/konflux-ci/tekton-catalog/task-coverity-availability-check:0.2@sha256:8b501440a960aec446db2ebc6625a49d0317a9fc7bf0f7bd9b18cb63052db7de + - name: kind + value: task + resolver: bundles + when: + - input: $(params.skip-checks) + operator: in + values: + - "false" + - name: sast-shell-check + params: + - name: image-digest + value: $(tasks.build-image-index.results.IMAGE_DIGEST) + - name: image-url + value: $(tasks.build-image-index.results.IMAGE_URL) + - name: SOURCE_ARTIFACT + value: $(tasks.prefetch-dependencies.results.SOURCE_ARTIFACT) + - name: TARGET_DIRS + value: $(params.sast-target-dirs) + runAfter: + - build-image-index + taskRef: + params: + - name: name + value: sast-shell-check-oci-ta + - name: bundle + value: quay.io/konflux-ci/tekton-catalog/task-sast-shell-check-oci-ta:0.1.1@sha256:83722286ec3297e3f66b0c220cec8895e0f745337e464c1d1c038ed737cf69fd + - name: kind + value: task + resolver: bundles + when: + - input: $(params.skip-checks) + operator: in + values: + - "false" + - name: sast-unicode-check + params: + - name: image-digest + value: $(tasks.build-image-index.results.IMAGE_DIGEST) + - name: image-url + value: $(tasks.build-image-index.results.IMAGE_URL) + - name: SOURCE_ARTIFACT + value: $(tasks.prefetch-dependencies.results.SOURCE_ARTIFACT) + - name: TARGET_DIRS + value: $(params.sast-target-dirs) + runAfter: + - build-image-index + taskRef: + params: + - name: name + value: sast-unicode-check-oci-ta + - name: bundle + value: quay.io/konflux-ci/tekton-catalog/task-sast-unicode-check-oci-ta:0.4.1@sha256:2ee89730856f542d851d99b5d945963583a840c295354dc4257a0052841e35ab + - name: kind + value: task + resolver: bundles + when: + - input: $(params.skip-checks) + operator: in + values: + - "false" + - name: apply-tags + params: + - name: IMAGE_URL + value: $(tasks.build-image-index.results.IMAGE_URL) + - name: IMAGE_DIGEST + value: $(tasks.build-image-index.results.IMAGE_DIGEST) + runAfter: + - build-image-index + taskRef: + params: + - name: name + value: apply-tags + - name: bundle + value: quay.io/konflux-ci/tekton-catalog/task-apply-tags:0.3.2@sha256:a80e8ee065e552a29c1e1473630b50378564d52c883fffdd8c164103e68b0e24 + - name: kind + value: task + resolver: bundles + - name: push-dockerfile + params: + - name: IMAGE + value: $(tasks.build-image-index.results.IMAGE_URL) + - name: IMAGE_DIGEST + value: $(tasks.build-image-index.results.IMAGE_DIGEST) + - name: DOCKERFILE + value: $(params.dockerfile) + - name: CONTEXT + value: $(params.path-context) + - name: SOURCE_ARTIFACT + value: $(tasks.prefetch-dependencies.results.SOURCE_ARTIFACT) + runAfter: + - build-image-index + taskRef: + params: + - name: name + value: push-dockerfile-oci-ta + - name: bundle + value: quay.io/konflux-ci/tekton-catalog/task-push-dockerfile-oci-ta:0.3.1@sha256:ab7b259a2d61639e3f2d130434b7cbb0ef56e7fb52b32dc4904eaf7da1020de9 + - name: kind + value: task + resolver: bundles + - name: rpms-signature-scan + params: + - name: image-url + value: $(tasks.build-image-index.results.IMAGE_URL) + - name: image-digest + value: $(tasks.build-image-index.results.IMAGE_DIGEST) + runAfter: + - build-image-index + taskRef: + params: + - name: name + value: rpms-signature-scan + - name: bundle + value: quay.io/konflux-ci/tekton-catalog/task-rpms-signature-scan:0.2.2@sha256:a09426471bb6300723ccca8f212cd71621db60ce1acf22bc0eef4e3ba17034de + - name: kind + value: task + resolver: bundles + when: + - input: $(params.skip-checks) + operator: in + values: + - "false" + workspaces: + - name: git-auth + optional: true + - name: netrc + optional: true diff --git a/src/pkg/workflowmodel/testdata/corpus/konflux-ci__konflux-ci/.tekton/konflux-operator-pull-request.yaml b/src/pkg/workflowmodel/testdata/corpus/konflux-ci__konflux-ci/.tekton/konflux-operator-pull-request.yaml new file mode 100644 index 0000000000..d8ba21f1f5 --- /dev/null +++ b/src/pkg/workflowmodel/testdata/corpus/konflux-ci__konflux-ci/.tekton/konflux-operator-pull-request.yaml @@ -0,0 +1,59 @@ +apiVersion: tekton.dev/v1 +kind: PipelineRun +metadata: + annotations: + build.appstudio.openshift.io/repo: https://github.com/konflux-ci/konflux-ci?rev={{revision}} + build.appstudio.redhat.com/commit_sha: '{{revision}}' + build.appstudio.redhat.com/pull_request_number: '{{pull_request_number}}' + build.appstudio.redhat.com/target_branch: '{{target_branch}}' + pipelinesascode.tekton.dev/cancel-in-progress: "true" + pipelinesascode.tekton.dev/max-keep-runs: "3" + pipelinesascode.tekton.dev/on-cel-expression: | + event == "pull_request" && target_branch == "main" && + (!has(body.pull_request) || !has(body.pull_request.labels) || ( + !body.pull_request.labels.exists(l, l.name == "superseded-by-companion") && + !body.pull_request.labels.exists(l, l.name == "pending-upstream-image") + )) && + (!has(body.issue) || !has(body.issue.labels) || ( + !body.issue.labels.exists(l, l.name == "superseded-by-companion") && + !body.issue.labels.exists(l, l.name == "pending-upstream-image") + )) || + event == "push" && target_branch.startsWith("gh-readonly-queue/main/") + creationTimestamp: null + labels: + appstudio.openshift.io/application: konflux-operator + appstudio.openshift.io/component: konflux-operator + pipelines.appstudio.openshift.io/type: build + name: konflux-operator-on-pull-request + namespace: konflux-vanguard-tenant +spec: + params: + - name: git-url + value: '{{source_url}}' + - name: revision + value: '{{revision}}' + - name: output-image + value: quay.io/redhat-user-workloads/konflux-vanguard-tenant/konflux-operator:on-pr-{{revision}} + - name: image-expires-after + value: 5d + - name: dockerfile + value: Containerfile + - name: path-context + value: operator + - name: build-platforms + value: + - linux/x86_64 + - linux/arm64 + - name: build-args + value: + - "OPERATOR_VERSION={{revision}}" + - "GIT_COMMIT={{revision}}" + pipelineRef: + name: build-pipeline + taskRunTemplate: + serviceAccountName: build-pipeline-konflux-operator + workspaces: + - name: git-auth + secret: + secretName: '{{ git_auth_secret }}' +status: {} diff --git a/src/pkg/workflowmodel/testdata/corpus/konflux-ci__konflux-ci/.tekton/konflux-operator-push.yaml b/src/pkg/workflowmodel/testdata/corpus/konflux-ci__konflux-ci/.tekton/konflux-operator-push.yaml new file mode 100644 index 0000000000..fed4ae6264 --- /dev/null +++ b/src/pkg/workflowmodel/testdata/corpus/konflux-ci__konflux-ci/.tekton/konflux-operator-push.yaml @@ -0,0 +1,46 @@ +apiVersion: tekton.dev/v1 +kind: PipelineRun +metadata: + annotations: + build.appstudio.openshift.io/repo: https://github.com/konflux-ci/konflux-ci?rev={{revision}} + build.appstudio.redhat.com/commit_sha: '{{revision}}' + build.appstudio.redhat.com/target_branch: '{{target_branch}}' + pipelinesascode.tekton.dev/cancel-in-progress: "false" + pipelinesascode.tekton.dev/max-keep-runs: "3" + pipelinesascode.tekton.dev/on-cel-expression: event == "push" && target_branch == "main" + creationTimestamp: null + labels: + appstudio.openshift.io/application: konflux-operator + appstudio.openshift.io/component: konflux-operator + pipelines.appstudio.openshift.io/type: build + name: konflux-operator-on-push + namespace: konflux-vanguard-tenant +spec: + params: + - name: git-url + value: '{{source_url}}' + - name: revision + value: '{{revision}}' + - name: output-image + value: quay.io/redhat-user-workloads/konflux-vanguard-tenant/konflux-operator:{{revision}} + - name: dockerfile + value: Containerfile + - name: path-context + value: operator + - name: build-platforms + value: + - linux/x86_64 + - linux/arm64 + - name: build-args + value: + - "OPERATOR_VERSION={{revision}}" + - "GIT_COMMIT={{revision}}" + pipelineRef: + name: build-pipeline + taskRunTemplate: + serviceAccountName: build-pipeline-konflux-operator + workspaces: + - name: git-auth + secret: + secretName: '{{ git_auth_secret }}' +status: {} diff --git a/src/pkg/workflowmodel/testdata/corpus/konflux-ci__konflux-ci/.tekton/konflux-operator-tag.yaml b/src/pkg/workflowmodel/testdata/corpus/konflux-ci__konflux-ci/.tekton/konflux-operator-tag.yaml new file mode 100644 index 0000000000..edd3274579 --- /dev/null +++ b/src/pkg/workflowmodel/testdata/corpus/konflux-ci__konflux-ci/.tekton/konflux-operator-tag.yaml @@ -0,0 +1,49 @@ +apiVersion: tekton.dev/v1 +kind: PipelineRun +metadata: + annotations: + build.appstudio.openshift.io/repo: https://github.com/konflux-ci/konflux-ci?rev={{revision}} + build.appstudio.redhat.com/commit_sha: '{{revision}}' + build.appstudio.redhat.com/target_branch: '{{target_branch}}' + pipelinesascode.tekton.dev/cancel-in-progress: "false" + pipelinesascode.tekton.dev/max-keep-runs: "3" + pipelinesascode.tekton.dev/on-cel-expression: |- + event == "push" && target_branch.matches("refs/tags/v0[.]2[.].*") + creationTimestamp: null + labels: + appstudio.openshift.io/application: konflux-operator + appstudio.openshift.io/component: konflux-operator + pipelines.appstudio.openshift.io/type: build + name: konflux-operator-on-tag + namespace: konflux-vanguard-tenant +spec: + params: + - name: git-url + value: '{{source_url}}' + - name: revision + value: '{{revision}}' + - name: output-image + value: quay.io/redhat-user-workloads/konflux-vanguard-tenant/konflux-operator:{{revision}}-{{git_tag}} + - name: dockerfile + value: Containerfile + - name: path-context + value: operator + - name: build-platforms + value: + - linux/x86_64 + - linux/arm64 + - name: rebuild + value: "true" + - name: build-args + value: + - "OPERATOR_VERSION={{git_tag}}" + - "GIT_COMMIT={{revision}}" + pipelineRef: + name: build-pipeline + taskRunTemplate: + serviceAccountName: build-pipeline-konflux-operator + workspaces: + - name: git-auth + secret: + secretName: '{{ git_auth_secret }}' +status: {} diff --git a/src/pkg/workflowmodel/testdata/corpus/openshift-knative__serverless-operator.golden.json b/src/pkg/workflowmodel/testdata/corpus/openshift-knative__serverless-operator.golden.json new file mode 100644 index 0000000000..076e767931 --- /dev/null +++ b/src/pkg/workflowmodel/testdata/corpus/openshift-knative__serverless-operator.golden.json @@ -0,0 +1,29 @@ +{ + "columns": [ + "Open", + "Actionable", + "Claimed", + "PR in progress", + "CI / Review", + "Merge queue", + "Merged" + ], + "roles": { + "other": 1 + }, + "pipelines": [ + { + "id": "tkn-tekton-docker-build", + "role": "other", + "nodes": 16, + "levels": 6, + "events": null + } + ], + "ciSubstages": null, + "findings": { + "no-trigger": 1, + "pipelineref-external": 2 + }, + "columnDiff": null +} diff --git a/src/pkg/workflowmodel/testdata/corpus/openshift-knative__serverless-operator/.tekton/docker-build.yaml b/src/pkg/workflowmodel/testdata/corpus/openshift-knative__serverless-operator/.tekton/docker-build.yaml new file mode 100644 index 0000000000..8b65220fff --- /dev/null +++ b/src/pkg/workflowmodel/testdata/corpus/openshift-knative__serverless-operator/.tekton/docker-build.yaml @@ -0,0 +1,548 @@ +apiVersion: tekton.dev/v1 +kind: Pipeline +metadata: + labels: + pipelines.openshift.io/runtime: generic + pipelines.openshift.io/strategy: docker + pipelines.openshift.io/used-by: build-cloud + name: docker-build +spec: + description: | + This pipeline is ideal for building multi-arch container images from a Containerfile while maintaining trust after pipeline customization. + + _Uses `buildah` to create a multi-platform container image leveraging [trusted artifacts](https://konflux-ci.dev/architecture/ADR/0036-trusted-artifacts.html). It also optionally creates a source image and runs some build-time tests. This pipeline requires that the [multi platform controller](https://github.com/konflux-ci/multi-platform-controller) is deployed and configured on your Konflux instance. Information is shared between tasks using OCI artifacts instead of PVCs. EC will pass the [`trusted_task.trusted`](https://conforma.dev/docs/policy/packages/release_trusted_task.html#trusted_task__trusted) policy as long as all data used to build the artifact is generated from trusted tasks. + This pipeline is pushed as a Tekton bundle to [quay.io](https://quay.io/repository/konflux-ci/tekton-catalog/pipeline-docker-build-multi-platform-oci-ta?tab=tags)_ + params: + - default: + - linux/x86_64 + - linux/arm64 + - linux/ppc64le + - linux/s390x + description: List of platforms to build the container images on. The available + set of values is determined by the configuration of the multi-platform-controller. + name: build-platforms + type: array + - default: --all-projects --org=3e1a4cca-ebfb-495f-b64c-3cc960d566b4 --exclude=test*,vendor,third_party + description: Append arguments to Snyk code command. + name: snyk-args + type: string + - default: "true" + description: Build a source image. + name: build-source-image + type: string + - default: "false" + description: 'Enable in-development package managers. WARNING: the behavior may + change at any time without notice. Use at your own risk.' + name: prefetch-input-dev-package-managers + - default: [] + description: Additional image tags + name: additional-tags + type: array + - description: Source Repository URL + name: git-url + type: string + - default: "" + description: Revision of the Source Repository + name: revision + type: string + - description: Fully Qualified Output Image + name: output-image + type: string + - default: . + description: Path to the source code of an application's component from where + to build image. + name: path-context + type: string + - default: Dockerfile + description: Path to the Dockerfile inside the context specified by parameter + path-context + name: dockerfile + type: string + - default: "false" + description: Skip checks against built image + name: skip-checks + type: string + - default: "false" + description: Execute the build with network isolation + name: hermetic + type: string + - default: "" + description: Build dependencies to be prefetched + name: prefetch-input + type: string + - default: "" + description: Image tag expiration time, time values could be something like 1h, + 2d, 3w for hours, days, and weeks, respectively. + name: image-expires-after + type: string + - default: "true" + description: Add built image into an OCI image index + name: build-image-index + type: string + - default: docker + description: The format for the resulting image's mediaType. Valid values are + oci or docker. + name: buildah-format + type: string + - default: "false" + description: Enable cache proxy configuration + name: enable-cache-proxy + - default: "true" + description: Use the package registry proxy when prefetching dependencies + name: enable-package-registry-proxy + - default: . + description: Target directories in component's source code to scan with SAST tools. + Multiple values should be separated with commas. + name: sast-target-dirs + type: string + - default: [] + description: Array of --build-arg values ("arg=value" strings) for buildah + name: build-args + type: array + - default: "" + description: Path to a file with build arguments for buildah, see https://www.mankier.com/1/buildah-build#--build-arg-file + name: build-args-file + type: string + - default: "false" + description: Whether to enable privileged mode, should be used only with remote + VMs + name: privileged-nested + type: string + results: + - description: "" + name: IMAGE_URL + value: $(tasks.build-image-index.results.IMAGE_URL) + - description: "" + name: IMAGE_DIGEST + value: $(tasks.build-image-index.results.IMAGE_DIGEST) + - description: "" + name: CHAINS-GIT_URL + value: $(tasks.clone-repository.results.url) + - description: "" + name: CHAINS-GIT_COMMIT + value: $(tasks.clone-repository.results.commit) + tasks: + - name: sast-snyk-check + params: + - name: ARGS + value: $(params.snyk-args) + - name: image-digest + value: $(tasks.build-image-index.results.IMAGE_DIGEST) + - name: image-url + value: $(tasks.build-image-index.results.IMAGE_URL) + - name: TARGET_DIRS + value: $(params.sast-target-dirs) + - name: SOURCE_ARTIFACT + value: $(tasks.prefetch-dependencies.results.SOURCE_ARTIFACT) + - name: CACHI2_ARTIFACT + value: $(tasks.prefetch-dependencies.results.CACHI2_ARTIFACT) + runAfter: + - build-image-index + taskRef: + params: + - name: name + value: sast-snyk-check-oci-ta + - name: bundle + value: quay.io/konflux-ci/tekton-catalog/task-sast-snyk-check-oci-ta:0.4@sha256:0ebf28a0abd5a167438d4628938a74ade6f00a44a4b7ed1cfa9cfc57a5b24748 + - name: kind + value: task + resolver: bundles + when: + - input: $(params.skip-checks) + operator: in + values: + - "false" + - name: prefetch-dependencies + params: + - name: dev-package-managers + value: $(params.prefetch-input-dev-package-managers) + - name: input + value: $(params.prefetch-input) + - name: enable-package-registry-proxy + value: $(params.enable-package-registry-proxy) + - name: SOURCE_ARTIFACT + value: $(tasks.clone-repository.results.SOURCE_ARTIFACT) + - name: ociStorage + value: $(params.output-image).prefetch + - name: ociArtifactExpiresAfter + value: $(params.image-expires-after) + runAfter: + - clone-repository + taskRef: + params: + - name: name + value: prefetch-dependencies-oci-ta + - name: bundle + value: quay.io/konflux-ci/tekton-catalog/task-prefetch-dependencies-oci-ta:0.3@sha256:3dc78afbf3a441e0280067433cb28ea3d2d0088ec214c73bf063f145b4f273ef + - name: kind + value: task + resolver: bundles + workspaces: + - name: git-basic-auth + workspace: git-auth + - name: netrc + workspace: netrc + - name: apply-tags + params: + - name: ADDITIONAL_TAGS + value: $(params.additional-tags[*]) + - name: IMAGE_URL + value: $(tasks.build-image-index.results.IMAGE_URL) + - name: IMAGE_DIGEST + value: $(tasks.build-image-index.results.IMAGE_DIGEST) + runAfter: + - build-image-index + taskRef: + params: + - name: name + value: apply-tags + - name: bundle + value: quay.io/konflux-ci/tekton-catalog/task-apply-tags:0.3@sha256:a291081de7fb27f832c6fc3c4b078acf7e6162ca4c085db38b118ca87e8b5b66 + - name: kind + value: task + resolver: bundles + - name: init + params: + - name: enable-cache-proxy + value: $(params.enable-cache-proxy) + taskRef: + params: + - name: name + value: init + - name: bundle + value: quay.io/konflux-ci/tekton-catalog/task-init:0.4@sha256:5a423246792ac501ea279229b42ee57da9927da441c04b5c9ff86817b0856b08 + - name: kind + value: task + resolver: bundles + - name: clone-repository + params: + - name: url + value: $(params.git-url) + - name: revision + value: $(params.revision) + - name: ociStorage + value: $(params.output-image).git + - name: ociArtifactExpiresAfter + value: $(params.image-expires-after) + runAfter: + - init + taskRef: + params: + - name: name + value: git-clone-oci-ta + - name: bundle + value: quay.io/konflux-ci/tekton-catalog/task-git-clone-oci-ta:0.1@sha256:d30f13dd15daf89dd6dc645243b3444d35570d13f7840c3fd65e366022515205 + - name: kind + value: task + resolver: bundles + workspaces: + - name: basic-auth + workspace: git-auth + - matrix: + params: + - name: PLATFORM + value: + - $(params.build-platforms) + name: build-images + params: + - name: IMAGE + value: $(params.output-image) + - name: DOCKERFILE + value: $(params.dockerfile) + - name: CONTEXT + value: $(params.path-context) + - name: HERMETIC + value: $(params.hermetic) + - name: PREFETCH_INPUT + value: $(params.prefetch-input) + - name: IMAGE_EXPIRES_AFTER + value: $(params.image-expires-after) + - name: COMMIT_SHA + value: $(tasks.clone-repository.results.commit) + - name: BUILD_ARGS + value: + - $(params.build-args[*]) + - name: BUILD_ARGS_FILE + value: $(params.build-args-file) + - name: PRIVILEGED_NESTED + value: $(params.privileged-nested) + - name: SOURCE_URL + value: $(tasks.clone-repository.results.url) + - name: BUILDAH_FORMAT + value: $(params.buildah-format) + - name: HTTP_PROXY + value: $(tasks.init.results.http-proxy) + - name: NO_PROXY + value: $(tasks.init.results.no-proxy) + - name: SOURCE_ARTIFACT + value: $(tasks.prefetch-dependencies.results.SOURCE_ARTIFACT) + - name: CACHI2_ARTIFACT + value: $(tasks.prefetch-dependencies.results.CACHI2_ARTIFACT) + - name: IMAGE_APPEND_PLATFORM + value: "true" + runAfter: + - prefetch-dependencies + taskRef: + params: + - name: name + value: buildah-remote-oci-ta + - name: bundle + value: quay.io/konflux-ci/tekton-catalog/task-buildah-remote-oci-ta:0.9@sha256:77007259cc87f32d63d2c201226aadaab98313cfd4e02b46abc243c4d2cc27bd + - name: kind + value: task + resolver: bundles + - name: build-image-index + params: + - name: IMAGE + value: $(params.output-image) + - name: ALWAYS_BUILD_INDEX + value: $(params.build-image-index) + - name: IMAGES + value: + - $(tasks.build-images.results.IMAGE_REF[*]) + - name: BUILDAH_FORMAT + value: $(params.buildah-format) + runAfter: + - build-images + taskRef: + params: + - name: name + value: build-image-index + - name: bundle + value: quay.io/konflux-ci/tekton-catalog/task-build-image-index:0.3@sha256:b33bfa8dc27dbf459f0779598ba45dcaa490bcc9f8efe1652bcf360ec8cb5582 + - name: kind + value: task + resolver: bundles + - name: build-source-image + params: + - name: BINARY_IMAGE + value: $(tasks.build-image-index.results.IMAGE_URL) + - name: BINARY_IMAGE_DIGEST + value: $(tasks.build-image-index.results.IMAGE_DIGEST) + - name: SOURCE_ARTIFACT + value: $(tasks.prefetch-dependencies.results.SOURCE_ARTIFACT) + - name: CACHI2_ARTIFACT + value: $(tasks.prefetch-dependencies.results.CACHI2_ARTIFACT) + runAfter: + - build-image-index + taskRef: + params: + - name: name + value: source-build-oci-ta + - name: bundle + value: quay.io/konflux-ci/tekton-catalog/task-source-build-oci-ta:0.3@sha256:8567bb7bf8fa9147c96b297533336fa7079ecf972cb86c09ccdd6bddedb25711 + - name: kind + value: task + resolver: bundles + when: + - input: $(params.build-source-image) + operator: in + values: + - "true" + - name: deprecated-base-image-check + params: + - name: IMAGE_URL + value: $(tasks.build-image-index.results.IMAGE_URL) + - name: IMAGE_DIGEST + value: $(tasks.build-image-index.results.IMAGE_DIGEST) + runAfter: + - build-image-index + taskRef: + params: + - name: name + value: deprecated-image-check + - name: bundle + value: quay.io/konflux-ci/tekton-catalog/task-deprecated-image-check:0.5@sha256:e78d0d3baf3c8cfc1a5ad278196b74032d9568b143a87c7a79ab780fedfb296e + - name: kind + value: task + resolver: bundles + when: + - input: $(params.skip-checks) + operator: in + values: + - "false" + - matrix: + params: + - name: image-platform + value: + - $(params.build-platforms) + name: clair-scan + params: + - name: image-digest + value: $(tasks.build-image-index.results.IMAGE_DIGEST) + - name: image-url + value: $(tasks.build-image-index.results.IMAGE_URL) + runAfter: + - build-image-index + taskRef: + params: + - name: name + value: clair-scan + - name: bundle + value: quay.io/konflux-ci/tekton-catalog/task-clair-scan:0.3@sha256:8fad4c2e2f470f82ee43d6b2ac72327b4d9c6e9cb514a678911c1c9359c29894 + - name: kind + value: task + resolver: bundles + when: + - input: $(params.skip-checks) + operator: in + values: + - "false" + - matrix: + params: + - name: platform + value: + - $(params.build-platforms) + name: ecosystem-cert-preflight-checks + params: + - name: image-url + value: $(tasks.build-image-index.results.IMAGE_URL) + runAfter: + - build-image-index + taskRef: + params: + - name: name + value: ecosystem-cert-preflight-checks + - name: bundle + value: quay.io/konflux-ci/tekton-catalog/task-ecosystem-cert-preflight-checks:0.2@sha256:88f4fd6d7812a3c46f120f3035974f5fb8cb06b5e3e927badf6e8370f1516a88 + - name: kind + value: task + resolver: bundles + when: + - input: $(params.skip-checks) + operator: in + values: + - "false" + - matrix: + params: + - name: image-arch + value: + - $(params.build-platforms) + name: clamav-scan + params: + - name: image-digest + value: $(tasks.build-image-index.results.IMAGE_DIGEST) + - name: image-url + value: $(tasks.build-image-index.results.IMAGE_URL) + runAfter: + - build-image-index + taskRef: + params: + - name: name + value: clamav-scan + - name: bundle + value: quay.io/konflux-ci/tekton-catalog/task-clamav-scan:0.3@sha256:567cb66bd2e1f4b58b9d4d756f3317fc62479e0b40aa0de66094b1f12d296cfc + - name: kind + value: task + resolver: bundles + when: + - input: $(params.skip-checks) + operator: in + values: + - "false" + - name: sast-shell-check + params: + - name: image-digest + value: $(tasks.build-image-index.results.IMAGE_DIGEST) + - name: image-url + value: $(tasks.build-image-index.results.IMAGE_URL) + - name: TARGET_DIRS + value: $(params.sast-target-dirs) + - name: SOURCE_ARTIFACT + value: $(tasks.prefetch-dependencies.results.SOURCE_ARTIFACT) + - name: CACHI2_ARTIFACT + value: $(tasks.prefetch-dependencies.results.CACHI2_ARTIFACT) + runAfter: + - build-image-index + taskRef: + params: + - name: name + value: sast-shell-check-oci-ta + - name: bundle + value: quay.io/konflux-ci/tekton-catalog/task-sast-shell-check-oci-ta:0.1@sha256:3cbb3535af6e7d4396858179a6427caaffb2e68775594795692fc01f28ae313f + - name: kind + value: task + resolver: bundles + when: + - input: $(params.skip-checks) + operator: in + values: + - "false" + - name: sast-unicode-check + params: + - name: image-digest + value: $(tasks.build-image-index.results.IMAGE_DIGEST) + - name: image-url + value: $(tasks.build-image-index.results.IMAGE_URL) + - name: TARGET_DIRS + value: $(params.sast-target-dirs) + - name: SOURCE_ARTIFACT + value: $(tasks.prefetch-dependencies.results.SOURCE_ARTIFACT) + - name: CACHI2_ARTIFACT + value: $(tasks.prefetch-dependencies.results.CACHI2_ARTIFACT) + runAfter: + - build-image-index + taskRef: + params: + - name: name + value: sast-unicode-check-oci-ta + - name: bundle + value: quay.io/konflux-ci/tekton-catalog/task-sast-unicode-check-oci-ta:0.4@sha256:223812001607b07f0e07d56bef7b7d619144e660c0c57f21ddd44ce0c8c4785b + - name: kind + value: task + resolver: bundles + when: + - input: $(params.skip-checks) + operator: in + values: + - "false" + - name: push-dockerfile + params: + - name: IMAGE + value: $(tasks.build-image-index.results.IMAGE_URL) + - name: IMAGE_DIGEST + value: $(tasks.build-image-index.results.IMAGE_DIGEST) + - name: DOCKERFILE + value: $(params.dockerfile) + - name: CONTEXT + value: $(params.path-context) + - name: SOURCE_ARTIFACT + value: $(tasks.prefetch-dependencies.results.SOURCE_ARTIFACT) + runAfter: + - build-image-index + taskRef: + params: + - name: name + value: push-dockerfile-oci-ta + - name: bundle + value: quay.io/konflux-ci/tekton-catalog/task-push-dockerfile-oci-ta:0.3@sha256:7855471abfe87de080b914f2f3ca27c59e64f6448a7c2435e51435b764494c71 + - name: kind + value: task + resolver: bundles + - name: rpms-signature-scan + params: + - name: image-url + value: $(tasks.build-image-index.results.IMAGE_URL) + - name: image-digest + value: $(tasks.build-image-index.results.IMAGE_DIGEST) + runAfter: + - build-image-index + taskRef: + params: + - name: name + value: rpms-signature-scan + - name: bundle + value: quay.io/konflux-ci/tekton-catalog/task-rpms-signature-scan:0.2@sha256:237c54b069d16c3785d1302f19be309aa6c0ae2313d446e30cb74671e07ca676 + - name: kind + value: task + resolver: bundles + when: + - input: $(params.skip-checks) + operator: in + values: + - "false" + workspaces: + - name: git-auth + optional: true + - name: netrc + optional: true diff --git a/src/pkg/workflowmodel/testdata/corpus/openshift-knative__serverless-operator/.tekton/serverless-bundle-138-pull-request.yaml b/src/pkg/workflowmodel/testdata/corpus/openshift-knative__serverless-operator/.tekton/serverless-bundle-138-pull-request.yaml new file mode 100644 index 0000000000..6ae1676ebf --- /dev/null +++ b/src/pkg/workflowmodel/testdata/corpus/openshift-knative__serverless-operator/.tekton/serverless-bundle-138-pull-request.yaml @@ -0,0 +1,66 @@ +apiVersion: tekton.dev/v1 +kind: PipelineRun +metadata: + annotations: + build.appstudio.openshift.io/repo: https://github.com/openshift-knative/serverless-operator?rev={{revision}} + build.appstudio.redhat.com/commit_sha: '{{revision}}' + build.appstudio.redhat.com/pull_request_number: '{{pull_request_number}}' + build.appstudio.redhat.com/target_branch: '{{target_branch}}' + pipelinesascode.tekton.dev/max-keep-runs: "3" + pipelinesascode.tekton.dev/on-cel-expression: event == "pull_request" && target_branch == "main" && ( files.all.exists(x, x.matches('^olm-catalog/serverless-operator/')) || files.all.exists(x, x.matches('^.tekton/')) ) + creationTimestamp: null + labels: + appstudio.openshift.io/application: serverless-operator-138 + appstudio.openshift.io/component: serverless-bundle-138 + pipelines.appstudio.openshift.io/type: build + name: serverless-bundle-138-on-pull-request + namespace: ocp-serverless-tenant +spec: + params: + - name: dockerfile + value: olm-catalog/serverless-operator/Dockerfile + - name: build-args + value: + - VERSION=1.38.0 + - TAG=v1.21.0 + - CLI_ARTIFACTS=registry.redhat.io/openshift4/ose-cli-artifacts-rhel9:v4.16 + - GO_BUILDER=brew.registry.redhat.io/rh-osbs/openshift-golang-builder:rhel_9_golang_1.25 + - GO_RUNTIME=registry.access.redhat.com/ubi9/ubi-minimal + - JAVA_BUILDER=registry.access.redhat.com/ubi9/openjdk-21 + - JAVA_RUNTIME=registry.access.redhat.com/ubi9/openjdk-21-runtime + - NODE_BUILDER=registry.access.redhat.com/ubi9/nodejs-20 + - NODE_RUNTIME=registry.access.redhat.com/ubi9/nodejs-20 + - name: git-url + value: '{{source_url}}' + - name: hermetic + value: "true" + - name: image-expires-after + value: 5d + - name: output-image + value: quay.io/redhat-user-workloads/ocp-serverless-tenant/serverless-operator-138/serverless-bundle:on-pr-{{revision}} + - name: build-platforms + value: + - linux/x86_64 + - name: revision + value: '{{revision}}' + - name: prefetch-input + value: '[{"type":"rpm"}]' + - name: prefetch-input-dev-package-managers + value: 'true' + taskRunSpecs: + - pipelineTaskName: sast-shell-check + stepSpecs: + - name: sast-shell-check + computeResources: + requests: + memory: 4Gi + limits: + memory: 4Gi + pipelineRef: + name: bundle-build + taskRunTemplate: + serviceAccountName: build-pipeline-serverless-bundle-138 + workspaces: + - name: git-auth + secret: + secretName: '{{ git_auth_secret }}' diff --git a/src/pkg/workflowmodel/testdata/corpus/openshift-knative__serverless-operator/.tekton/serverless-bundle-138-push.yaml b/src/pkg/workflowmodel/testdata/corpus/openshift-knative__serverless-operator/.tekton/serverless-bundle-138-push.yaml new file mode 100644 index 0000000000..c7cfd3aa1f --- /dev/null +++ b/src/pkg/workflowmodel/testdata/corpus/openshift-knative__serverless-operator/.tekton/serverless-bundle-138-push.yaml @@ -0,0 +1,65 @@ +apiVersion: tekton.dev/v1 +kind: PipelineRun +metadata: + annotations: + build.appstudio.openshift.io/repo: https://github.com/openshift-knative/serverless-operator?rev={{revision}} + build.appstudio.redhat.com/commit_sha: '{{revision}}' + build.appstudio.redhat.com/target_branch: '{{target_branch}}' + pipelinesascode.tekton.dev/max-keep-runs: "3" + pipelinesascode.tekton.dev/on-cel-expression: event == "push" && target_branch == "main" && ( files.all.exists(x, x.matches('^olm-catalog/serverless-operator/')) || files.all.exists(x, x.matches('^.tekton/')) ) + creationTimestamp: null + labels: + appstudio.openshift.io/application: serverless-operator-138 + appstudio.openshift.io/component: serverless-bundle-138 + pipelines.appstudio.openshift.io/type: build + name: serverless-bundle-138-on-push + namespace: ocp-serverless-tenant +spec: + params: + - name: dockerfile + value: olm-catalog/serverless-operator/Dockerfile + - name: build-args + value: + - VERSION=1.38.0 + - TAG=v1.21.0 + - CLI_ARTIFACTS=registry.redhat.io/openshift4/ose-cli-artifacts-rhel9:v4.16 + - GO_BUILDER=brew.registry.redhat.io/rh-osbs/openshift-golang-builder:rhel_9_golang_1.25 + - GO_RUNTIME=registry.access.redhat.com/ubi9/ubi-minimal + - JAVA_BUILDER=registry.access.redhat.com/ubi9/openjdk-21 + - JAVA_RUNTIME=registry.access.redhat.com/ubi9/openjdk-21-runtime + - NODE_BUILDER=registry.access.redhat.com/ubi9/nodejs-20 + - NODE_RUNTIME=registry.access.redhat.com/ubi9/nodejs-20 + - name: git-url + value: '{{source_url}}' + - name: hermetic + value: "true" + - name: output-image + value: quay.io/redhat-user-workloads/ocp-serverless-tenant/serverless-operator-138/serverless-bundle:{{revision}} + - name: revision + value: '{{revision}}' + - name: additional-tags + value: + - $(context.pipelineRun.uid)-{{revision}} + - 1.38.0 + - latest + - name: prefetch-input + value: '[{"type":"rpm"}]' + - name: prefetch-input-dev-package-managers + value: 'true' + taskRunSpecs: + - pipelineTaskName: sast-shell-check + stepSpecs: + - name: sast-shell-check + computeResources: + requests: + memory: 4Gi + limits: + memory: 4Gi + pipelineRef: + name: bundle-build + taskRunTemplate: + serviceAccountName: build-pipeline-serverless-bundle-138 + workspaces: + - name: git-auth + secret: + secretName: '{{ git_auth_secret }}' diff --git a/src/pkg/workflowmodel/testdata/corpus/tektoncd__pipeline.golden.json b/src/pkg/workflowmodel/testdata/corpus/tektoncd__pipeline.golden.json new file mode 100644 index 0000000000..b894c109fa --- /dev/null +++ b/src/pkg/workflowmodel/testdata/corpus/tektoncd__pipeline.golden.json @@ -0,0 +1,37 @@ +{ + "columns": [ + "Open", + "Actionable", + "Claimed", + "PR in progress", + "CI / Review", + "Merge queue", + "Merged" + ], + "roles": { + "pr-verify": 1 + }, + "pipelines": [ + { + "id": "gha-github-workflows-ci", + "role": "pr-verify", + "nodes": 9, + "levels": 4, + "events": [ + "pull_request" + ] + } + ], + "ciSubstages": [ + "categorize changes", + "build · buildFips · lint", + "Check generated code · Multi-arch build · e2e-tests +1", + "CI summary" + ], + "findings": { + "pipelineref-external": 1 + }, + "columnDiff": [ + "column \"CI / Review\" gains 4 CI substage(s) from 1 pipeline(s)" + ] +} diff --git a/src/pkg/workflowmodel/testdata/corpus/tektoncd__pipeline/.github/workflows/ci.yaml b/src/pkg/workflowmodel/testdata/corpus/tektoncd__pipeline/.github/workflows/ci.yaml new file mode 100644 index 0000000000..0381269dd4 --- /dev/null +++ b/src/pkg/workflowmodel/testdata/corpus/tektoncd__pipeline/.github/workflows/ci.yaml @@ -0,0 +1,248 @@ +name: ci + +on: + pull_request: + types: [opened, synchronize, reopened, ready_for_review] + +concurrency: + group: ${{ github.workflow }}-${{ github.event.pull-request.number || github.ref }} + cancel-in-progress: true + +defaults: + run: + shell: bash + +permissions: + contents: read + +jobs: + changes: + name: categorize changes + runs-on: ubuntu-latest + outputs: + non-docs: ${{ steps.detect.outputs.non-docs }} + yaml: ${{ steps.detect.outputs.yaml }} + steps: + - name: Get base depth + id: base-depth + run: echo "base-depth=$(expr ${{ github.event.pull_request.commits }} + 1)" >> $GITHUB_OUTPUT + - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 + with: + ref: ${{ github.event.pull_request.head.sha }} + fetch-depth: ${{ steps.base-depth.outputs.base-depth }} + persist-credentials: false + - name: detect + id: detect + run: | + git fetch origin ${GITHUB_BASE_REF} + CHANGED_FILES=$(git diff --name-only ${{ github.event.pull_request.base.sha }}...${{ github.event.pull_request.head.sha }} | tr ' ' '\n') + + echo -e "Changed files:\n${CHANGED_FILES}" + + # If there are no changed files, non-docs and yaml are not set and remain false by default. + # This ensures that jobs depending on these outputs do not run when there are no changes. + if [[ -n "${CHANGED_FILES}" ]]; then + NON_DOCS='false' + YAML='false' + + while read -r file; do + if [[ "$file" != *.md ]]; then + NON_DOCS='true' + break + fi + done <<< "${CHANGED_FILES}" + + while read -r file; do + if [[ "$file" == *.yaml || "$file" == *.yml ]]; then + YAML='true' + break + fi + done <<< "${CHANGED_FILES}" + + echo "non-docs=${NON_DOCS}" | tee -a $GITHUB_OUTPUT + echo "yaml=${YAML}" | tee -a $GITHUB_OUTPUT + fi + + build: + name: build + runs-on: ubuntu-latest-8-cores + needs: [changes] + if: ${{ needs.changes.outputs.non-docs == 'true' }} + steps: + - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 + with: + persist-credentials: false + - uses: actions/setup-go@b7ad1dad31e06c5925ef5d2fc7ad053ef454303e # v7.0.0 + with: + go-version-file: "go.mod" + - name: build + run: | + go build -v ./... + buildFips: + name: buildFips + runs-on: ubuntu-latest-8-cores + needs: [changes] + if: ${{ needs.changes.outputs.non-docs == 'true' }} + steps: + - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 + with: + persist-credentials: false + - uses: actions/setup-go@b7ad1dad31e06c5925ef5d2fc7ad053ef454303e # v7.0.0 + with: + go-version-file: "go.mod" + - name: build + run: | + go build -v -tags "disable_spire,disable_tls" ./cmd/entrypoint + echo "Build finished with exit code: $?" + linting: + name: lint + runs-on: ubuntu-latest-8-cores + permissions: + contents: read + checks: write # Used by golangci-lint to annotate code in the PR + needs: [changes] + steps: + - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 + with: + fetch-depth: 0 + persist-credentials: false + - uses: actions/setup-go@b7ad1dad31e06c5925ef5d2fc7ad053ef454303e # v7.0.0 + with: + go-version-file: "go.mod" + - name: gofmt + if: ${{ needs.changes.outputs.non-docs == 'true' }} + run: | + gofmt_out=$(gofmt -d $(find * -name '*.go' ! -path 'vendor/*' ! -path 'third_party/*')) + if [[ -n "$gofmt_out" ]]; then + failed=1 + fi + echo "$gofmt_out" + - name: golangci-lint + uses: golangci/golangci-lint-action@ba0d7d2ec06a0ea1cb5fa41b2e4a3ab91d21278a # v9.3.0 + if: ${{ needs.changes.outputs.non-docs == 'true' }} + with: + version: v2.15.0 + install-mode: goinstall + args: --new-from-merge-base=origin/${{ github.base_ref }} --timeout=10m + - name: yamllint + if: ${{ needs.changes.outputs.yaml == 'true' }} + run: | + apt-get update && apt-get install -y yamllint + make yamllint + - name: check-license + if: ${{ needs.changes.outputs.non-docs == 'true' }} + run: | + go install github.com/google/go-licenses@v1.0.0 + go-licenses check ./... + tests: + needs: [build] + name: test + runs-on: ubuntu-latest-8-cores + steps: + - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 + with: + persist-credentials: false + - uses: actions/setup-go@b7ad1dad31e06c5925ef5d2fc7ad053ef454303e # v7.0.0 + with: + go-version-file: "go.mod" + - name: unit-test + run: | + make test-unit-verbose-and-race + generated: + needs: [build] + name: Check generated code + runs-on: ubuntu-latest-8-cores + steps: + - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 + with: + persist-credentials: false + - uses: actions/setup-go@b7ad1dad31e06c5925ef5d2fc7ad053ef454303e # v7.0.0 + with: + go-version-file: "go.mod" + - name: generated + run: | + ./hack/verify-codegen.sh + multi-arch-build: + needs: [build] + name: Multi-arch build + runs-on: ubuntu-latest-8-cores + env: + KOCACHE: /tmp/ko-cache + steps: + - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 + with: + persist-credentials: false + - uses: actions/setup-go@b7ad1dad31e06c5925ef5d2fc7ad053ef454303e # v7.0.0 + with: + go-version-file: "go.mod" + - name: Cache ko build cache + uses: actions/cache@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0 + with: + path: /tmp/ko-cache + key: ${{ runner.os }}-${{ runner.arch }}-ko-${{ hashFiles('go.sum') }} + restore-keys: | + ${{ runner.os }}-${{ runner.arch }}-ko- + - uses: ko-build/setup-ko@61b4d1d396f5b2e7d6bb6fefdce3dc38d1a13445 # v0.10 + with: + version: v0.18.1 + - name: ko-resolve + run: | + # Use the repository's .ko.yaml for consistent base images + KO_DOCKER_REPO=example.com ko resolve -l 'app.kubernetes.io/component!=resolvers' --platform=all --push=false -R -f config 1>/dev/null + KO_DOCKER_REPO=example.com ko resolve --platform=all --push=false -f config/resolvers 1>/dev/null + e2e-tests: + needs: [build] + uses: ./.github/workflows/e2e-matrix.yml + with: + is-draft: ${{ github.event.pull_request.draft }} + + ci-summary: + name: CI summary + needs: [build, buildFips, linting, tests, generated, multi-arch-build, e2e-tests] + runs-on: ubuntu-latest + if: always() + steps: + - name: Block merge for draft PRs + if: ${{ github.event.pull_request.draft == true }} + run: | + echo "::warning::Draft PR — only minimal e2e tests were run (amd64, k8s-latest, stable)" + echo "Mark as 'Ready for Review' to run the full CI matrix." + exit 1 + env: + IS_DRAFT: ${{ github.event.pull_request.draft }} + - name: Check CI results + if: ${{ github.event.pull_request.draft == false }} + run: | + results=( + "build=${NEEDS_BUILD_RESULT}" + "buildFips=${NEEDS_BUILDFIPS_RESULT}" + "linting=${NEEDS_LINTING_RESULT}" + "tests=${NEEDS_TESTS_RESULT}" + "generated=${NEEDS_GENERATED_RESULT}" + "multi-arch-build=${NEEDS_MULTI_ARCH_BUILD_RESULT}" + "e2e-tests=${NEEDS_E2E_TESTS_RESULT}" + ) + failed=0 + for r in "${results[@]}"; do + name="${r%%=*}" + result="${r#*=}" + echo "${name}: ${result}" + if [ "$result" != "success" ] && [ "$result" != "skipped" ]; then + failed=1 + fi + done + if [ "$failed" -eq 1 ]; then + echo "" + echo "Some CI jobs failed or were cancelled" + exit 1 + fi + echo "" + echo "All CI checks passed" + env: + NEEDS_BUILD_RESULT: ${{ needs.build.result }} + NEEDS_BUILDFIPS_RESULT: ${{ needs.buildFips.result }} + NEEDS_LINTING_RESULT: ${{ needs.linting.result }} + NEEDS_TESTS_RESULT: ${{ needs.tests.result }} + NEEDS_GENERATED_RESULT: ${{ needs.generated.result }} + NEEDS_MULTI_ARCH_BUILD_RESULT: ${{ needs.multi-arch-build.result }} + NEEDS_E2E_TESTS_RESULT: ${{ needs.e2e-tests.result }} diff --git a/src/pkg/workflowmodel/testdata/corpus/tektoncd__pipeline/.tekton/release.yaml b/src/pkg/workflowmodel/testdata/corpus/tektoncd__pipeline/.tekton/release.yaml new file mode 100644 index 0000000000..5e26aeb1ba --- /dev/null +++ b/src/pkg/workflowmodel/testdata/corpus/tektoncd__pipeline/.tekton/release.yaml @@ -0,0 +1,95 @@ +# Copyright 2026 The Tekton Authors +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# This PipelineRun is triggered by Pipelines-as-Code when a release +# branch (release-v*) is first created. It runs the release pipeline +# defined in tekton/release-pipeline.yaml from this repository. +# +# The release pipeline builds, publishes images, uploads artifacts to +# the release bucket, and the draft GitHub release is created separately. +--- +apiVersion: tekton.dev/v1 +kind: PipelineRun +metadata: + name: release-pipeline + annotations: + # NOTE: on-cel-expression takes precedence over on-event/on-target-branch, + # so the branch filter MUST be included in the CEL expression. + pipelinesascode.tekton.dev/on-event: "[push]" + pipelinesascode.tekton.dev/on-target-branch: "[refs/heads/release-v*]" + pipelinesascode.tekton.dev/on-cel-expression: | + event == "push" && has(body.created) && body.created == true && + target_branch.startsWith("release-v") + pipelinesascode.tekton.dev/pipeline: "tekton/release-pipeline.yaml" + pipelinesascode.tekton.dev/max-keep-runs: "5" +spec: + taskRunTemplate: + serviceAccountName: release + pipelineRef: + name: pipeline-release + params: + - name: package + value: github.com/tektoncd/pipeline + - name: repoName + value: pipeline + - name: gitRevision + value: "{{ revision }}" + - name: imageRegistry + value: ghcr.io + - name: imageRegistryPath + value: tektoncd/pipeline + - name: imageRegistryRegions + value: "" + - name: imageRegistryUser + value: tekton-robot + # For initial releases, the version is derived from the branch name: + # release-v0.76.x → v0.76.0 + - name: versionTag + value: '{{ cel: pac.target_branch.replace("release-", "").replace(".x", ".0") }}' + - name: releaseBucket + value: tekton-releases + - name: releaseAsLatest + value: "true" + - name: koExtraArgs + value: "" + - name: serviceAccountImagesPath + value: credentials + - name: runTests + value: "true" + # Auto-detected from git tags when empty + - name: previousReleaseTag + value: "" + timeouts: + pipeline: 3h0m0s + workspaces: + - name: workarea + volumeClaimTemplate: + spec: + accessModes: + - ReadWriteOnce + resources: + requests: + storage: 5Gi + - name: release-secret + secret: + secretName: oci-release-secret + - name: release-images-secret + secret: + secretName: ghcr-creds + # Enables automatic draft GitHub release creation (PR #9420). + # When bound, the pipeline waits for Chains signing and creates + # a draft release with changelog. When not bound, skipped. + - name: github-secret + secret: + secretName: github-token diff --git a/src/pkg/workflowmodel/testdata/models/techdocs.json b/src/pkg/workflowmodel/testdata/models/techdocs.json new file mode 100644 index 0000000000..bd1a529d30 --- /dev/null +++ b/src/pkg/workflowmodel/testdata/models/techdocs.json @@ -0,0 +1,79 @@ +{ + "apiVersion": "workflow.hivecommons.dev/v1alpha1", + "kind": "WorkflowModel", + "metadata": {"name": "cncf-techdocs"}, + "spec": { + "meta": {"version": "1", "author": "cncf/techdocs", "description": "Documentation engagement: request, analyse, review, iterate, publish", "source": "written"}, + "scope": {"forge": "github.com", "repository": "cncf/techdocs"}, + "initial": "requested", + "columns": [ + {"id": "requested", "label": "Requested", "states": ["requested"]}, + {"id": "analysis", "label": "Analysis", "states": ["analysing", "analysis-review"]}, + {"id": "writing", "label": "Writing", "states": ["draft-under-way", "drafting", "draft-review"]}, + {"id": "publishing", "label": "Publishing", "states": ["approved"]}, + {"id": "done", "label": "Published", "states": ["published", "withdrawn"], "terminal": true} + ], + "states": [ + {"id": "requested", "label": "Requested", "owner": "human", "lifecycle": {"number": 10, "kinds": ["issue"]}}, + {"id": "analysing", "label": "Analysing", "owner": "registered-agent", "lifecycle": {"number": 20, "kinds": ["issue"], "replies": ["hand back"]}}, + {"id": "analysis-review", "label": "Analysis review", "owner": "human", "lifecycle": {"number": 45, "kinds": ["issue"], "replies": ["approve analysis", "revise"]}}, + {"id": "draft-under-way", "label": "Draft under way", "owner": "registered-agent", "lifecycle": {"number": 40, "kinds": ["issue"]}}, + {"id": "drafting", "label": "Drafting", "owner": "registered-agent", "lifecycle": {"number": 70, "kinds": ["pr"]}}, + {"id": "draft-review", "label": "Draft review", "owner": "human", "lifecycle": {"number": 50, "kinds": ["pr"], "replies": ["approve", "request changes"]}}, + {"id": "approved", "label": "Approved", "owner": "human", "lifecycle": {"number": 90, "kinds": ["pr"]}}, + {"id": "published", "label": "Published", "owner": "external-observation", "type": "final"}, + {"id": "withdrawn", "label": "Withdrawn", "owner": "external-observation", "type": "final"} + ], + "roles": [ + {"id": "techdocs-maintainer", "label": "TechDocs maintainer", "kind": "permission", "params": {"min": "maintain"}, "human": true}, + {"id": "project-maintainer", "label": "Project maintainer", "kind": "frozen-set", "human": true} + ], + "gates": [ + {"id": "analysis-approved", "releasedBy": "techdocs-maintainer", "reply": "approve analysis"}, + {"id": "draft-approved", "releasedBy": "project-maintainer", "reply": "approve"} + ], + "events": [ + {"id": "accepted", "fact": {"fact": "labels", "op": "has", "value": "techdocs/accepted"}, "actor": "techdocs-maintainer"}, + {"id": "analysis-posted", "fact": {"fact": "labels", "op": "has", "value": "techdocs/analysis"}}, + {"id": "draft-opened", "fact": {"fact": "issue.covered", "op": "true"}}, + {"id": "pr-approved", "fact": {"fact": "pr.review", "op": "is", "value": "approved"}}, + {"id": "merged", "fact": {"fact": "merged", "op": "true"}}, + {"id": "closed", "fact": {"fact": "open", "op": "false"}} + ], + "agents": [ + {"profile": "techdocs-analyst", "states": ["analysing"], "produces": ["analysis"]}, + {"profile": "techdocs-writer", "states": ["drafting"], "produces": ["pr"]} + ], + "transitions": [ + {"from": "requested", "to": "analysing", "on": "accepted"}, + {"from": "analysing", "to": "analysis-review", "on": "analysis-posted"}, + {"from": "analysis-review", "to": "analysing", "on": "analysis-posted", "note": "revise"}, + {"from": "analysis-review", "to": "draft-under-way", "on": "draft-opened", "gateRef": "analysis-approved"}, + {"from": "draft-under-way", "to": "published", "on": "merged"}, + {"from": "draft-under-way", "to": "withdrawn", "on": "closed"}, + {"from": "analysis-review", "to": "drafting", "on": "draft-opened", "gateRef": "analysis-approved"}, + {"from": "drafting", "to": "draft-review", "on": "analysis-posted"}, + {"from": "draft-review", "to": "drafting", "on": "analysis-posted"}, + {"from": "draft-review", "to": "approved", "on": "pr-approved", "gateRef": "draft-approved"}, + {"from": "approved", "to": "published", "on": "merged"}, + {"from": "requested", "to": "withdrawn", "on": "closed"}, + {"from": "analysing", "to": "withdrawn", "on": "closed"}, + {"from": "drafting", "to": "withdrawn", "on": "closed"} + ], + "derivation": { + "mode": "rules", + "rules": [ + {"id": "closed", "state": "withdrawn", "when": [{"fact": "open", "op": "false"}, {"fact": "merged", "op": "false"}], "reason": "closed without publishing"}, + {"id": "published", "state": "published", "when": [{"fact": "merged", "op": "true"}]}, + {"id": "approved", "kind": "pr", "state": "approved", "when": [{"fact": "pr.review", "op": "is", "value": "approved"}, {"fact": "pr.checks", "op": "is", "value": "green"}]}, + {"id": "draft-review", "kind": "pr", "state": "draft-review", "when": [{"fact": "draft", "op": "false"}, {"fact": "labels", "op": "has", "value": "techdocs/*"}]}, + {"id": "drafting", "kind": "pr", "state": "drafting", "when": [{"fact": "labels", "op": "has", "value": "techdocs/*"}]}, + {"id": "analysis-review", "kind": "issue", "state": "analysis-review", "when": [{"fact": "labels", "op": "has", "value": "techdocs/analysis"}, {"fact": "issue.covered", "op": "false"}]}, + {"id": "drafting-issue", "kind": "issue", "state": "draft-under-way", "when": [{"fact": "issue.covered", "op": "true"}]}, + {"id": "analysing", "kind": "issue", "state": "analysing", "when": [{"fact": "labels", "op": "has", "value": "techdocs/accepted"}]}, + {"id": "requested", "kind": "issue", "state": "requested", "when": [{"fact": "labels", "op": "has", "value": "techdocs/request"}]} + ] + }, + "coverage": {"files": 0, "parsed": 0} + } +}