Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions changelog.d/added-workflowmodel-ingest-lifecycle.md
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
- Workflow models, RFC phase 3a ([#17620](https://github.com/hivecommons/hive/issues/17620)): an ingested repository now yields a model that compiles. `ingest.Lifecycle` composes the hive-default lifecycle (states, columns, builtin derivation, roles, gates, events) with the repository's pipelines — PR-verify pipelines bound to the CI / Review states with file citations, the post-merge chain (main CI, release, per-environment deploys, done) as new columns after Merged — every ingested element marked inferred and the model's provenance `ingested-unreviewed`. New fact `pipeline.status` names the post-merge run the chain waits on (the engine reports `unknown` until it observes post-merge runs, so nothing moves on it yet). New owner-gated `POST /api/workflows/ingest` takes `{repo}` (default branch fetched read-only through the hive's GitHub client, CI files selected by `ingest.ReadTree`) or pasted `{files}`, returns the model, compile result and diagnostics, and with `save: true` stores it at the next integer version. `cmd/workflow-survey` compiles every repository's lifecycle model and reports pass/refusal counts by code; over the 317-repository CNCF corpus 317/317 compile with 1,117 inferred-element warnings and no refusals.
105 changes: 105 additions & 0 deletions dashboard/openapi.json
Original file line number Diff line number Diff line change
Expand Up @@ -39356,6 +39356,79 @@
}
}
}
},
"/api/workflows/ingest": {
"post": {
"tags": [
"Workflows"
],
"summary": "Ingest a repository's CI definitions into a lifecycle workflow model",
"description": "Reads the repository's default-branch GitHub Actions workflows and Tekton resources (or pasted files) and returns the hive-default lifecycle with the repository's pipelines bound: PR-verify pipelines on the CI / Review states, the post-merge chain as new columns. Provenance ingested-unreviewed. With save=true the model is stored as the next integer version under name. Owner role required.",
"requestBody": {
"required": true,
"content": {
"application/json": {
"schema": {
"type": "object",
"properties": {
"repo": {
"type": "string",
"description": "owner/name to fetch through the forge client; exactly one of repo or files"
},
"files": {
"type": "object",
"additionalProperties": {
"type": "string"
},
"description": "repo-relative path to file contents, for pasted or uploaded definitions"
},
"save": {
"type": "boolean"
},
"name": {
"type": "string",
"description": "store name (default: slug of the repository)"
}
}
}
}
}
},
"responses": {
"200": {
"description": "Ingested (not saved)",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/WorkflowIngestResult"
}
}
}
},
"201": {
"description": "Ingested and saved",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/WorkflowIngestResult"
}
}
}
},
"400": {
"description": "Bad request"
},
"403": {
"description": "Owner role required"
},
"502": {
"description": "Repository fetch failed"
},
"503": {
"description": "No forge client or no workflow store"
}
}
}
}
},
"components": {
Expand Down Expand Up @@ -39759,6 +39832,38 @@
"type": "object",
"description": "Compiled, derivation-ready form of a model (hash, states, columns, ordered rules). Never trusted from the wire: the spoke recompiles on save and on bind.",
"additionalProperties": true
},
"WorkflowIngestResult": {
"type": "object",
"properties": {
"model": {
"$ref": "#/components/schemas/WorkflowModel"
},
"compiled": {
"type": "boolean"
},
"diagnostics": {
"type": "array",
"items": {
"$ref": "#/components/schemas/WorkflowDiagnostic"
}
},
"profile": {
"type": "object"
},
"ref": {
"type": "string"
},
"files": {
"type": "integer"
},
"pipelines": {
"type": "integer"
},
"entry": {
"$ref": "#/components/schemas/WorkflowEntry"
}
}
}
}
}
Expand Down
1 change: 1 addition & 0 deletions src/cmd/hive/main.go
Original file line number Diff line number Diff line change
Expand Up @@ -1087,6 +1087,7 @@ func (b *boot) wireBootClosures() {
return &dashboard.Dependencies{
Config: b.cfg,
WorkflowStore: wfstore.Open(wfstore.Root(b.cfg.Data.WorkflowsDir)),
WorkflowRepoFiles: workflowRepoFiles(func() *github.Client { return b.ghClient }),
AgentMgr: b.agentMgr,
CruxRuntimeHealth: cruxRuntimeHealthFunc(b.cruxSupervision),
// Provider gateways (#5565 slice 3): concrete openrouter/watsonx/
Expand Down
37 changes: 37 additions & 0 deletions src/cmd/hive/workflow_ingest.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,37 @@
package main

import (
"context"
"errors"

"github.com/hivecommons/hive/pkg/github"
"github.com/hivecommons/hive/pkg/workflowmodel/ingest"
)

// errNoForgeClient is returned by the workflow ingest fetcher before a
// GitHub client exists.
var errNoForgeClient = errors.New("no GitHub client")

// workflowRepoFiles is the dashboard's WorkflowRepoFiles dependency
// (POST /api/workflows/ingest, RFC #17620 P3): it downloads the repo's
// default branch read-only, keeps only the CI definition files and
// removes the checkout. client is read per call so a credential rebuild is
// followed.
func workflowRepoFiles(client func() *github.Client) func(ctx context.Context, repo string) (map[string][]byte, string, error) {
return func(ctx context.Context, repo string) (map[string][]byte, string, error) {
c := client()
if c == nil {
return nil, "", errNoForgeClient
}
co, err := c.FetchDefaultBranchTree(ctx, repo)
if err != nil {
return nil, "", err
}
defer co.Cleanup()
files, err := ingest.ReadTree(co.Dir)
if err != nil {
return nil, "", err
}
return files, co.Ref, nil
}
}
16 changes: 16 additions & 0 deletions src/cmd/hive/workflow_ingest_test.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,16 @@
package main

import (
"context"
"errors"
"testing"

"github.com/hivecommons/hive/pkg/github"
)

func TestWorkflowRepoFilesWithoutClient(t *testing.T) {
fetch := workflowRepoFiles(func() *github.Client { return nil })
if _, _, err := fetch(context.Background(), "acme/widgets"); !errors.Is(err, errNoForgeClient) {
t.Fatalf("err = %v", err)
}
}
39 changes: 37 additions & 2 deletions src/cmd/workflow-survey/main.go
Original file line number Diff line number Diff line change
Expand Up @@ -68,6 +68,12 @@ type repoResult struct {
ValidateError string `json:"validateError,omitempty"`
Hash string `json:"hash"`
Severities map[workflowmodel.Severity]int `json:"severities,omitempty"`
// Compiled says whether the lifecycle model (ingest.Lifecycle) compiles;
// CompileCodes counts its diagnostics by code, errors and warnings.
Compiled bool `json:"compiled"`
CompileError string `json:"compileError,omitempty"`
CompileCodes map[string]int `json:"compileCodes,omitempty"`
LifecycleCols int `json:"lifecycleColumns"`
}

type summary struct {
Expand All @@ -78,6 +84,8 @@ type summary struct {
Parsed int `json:"parsed"`
Pipelines int `json:"pipelines"`
Valid int `json:"valid"`
Compiled int `json:"compiled"`
CompileCodes map[string]int `json:"compileCodes"`
Roles map[workflowmodel.Role]int `json:"roles"`
Constructs map[string]int `json:"constructs"`
FindingCodes map[string]int `json:"findingCodes"`
Expand Down Expand Up @@ -160,7 +168,7 @@ func run(corpus, out string, filter map[string]bool, stdout, stderr io.Writer) e
return err
}
maturity := loadProjects(corpus)
sum := &summary{Roles: map[workflowmodel.Role]int{}, Constructs: map[string]int{}, FindingCodes: map[string]int{}, Severities: map[workflowmodel.Severity]int{}, ColumnSets: map[string]int{}, ByProject: map[string]*projectStats{}}
sum := &summary{Roles: map[workflowmodel.Role]int{}, Constructs: map[string]int{}, FindingCodes: map[string]int{}, Severities: map[workflowmodel.Severity]int{}, CompileCodes: map[string]int{}, ColumnSets: map[string]int{}, ByProject: map[string]*projectStats{}}
for _, d := range dirs {
if !d.IsDir() {
continue
Expand Down Expand Up @@ -252,9 +260,25 @@ func surveyRepo(dir, repo, outDir string) (repoResult, error) {
}
res.ColumnDiff = render.Diff(m)

lm := ingest.Lifecycle(workflowmodel.Scope{Forge: defaultForge, Repository: repo}, files)
res.LifecycleCols = len(lm.Spec.Columns)
res.CompileCodes = map[string]int{}
_, diags, cerr := workflowmodel.Compile(lm)
for _, d := range diags {
res.CompileCodes[d.Code]++
}
if cerr != nil {
res.CompileError = cerr.Error()
} else {
res.Compiled = true
}

if err := os.MkdirAll(outDir, 0o755); err != nil {
return res, err
}
if err := writeJSON(filepath.Join(outDir, "lifecycle.json"), lm); err != nil {
return res, err
}
if err := writeJSON(filepath.Join(outDir, "model.json"), m); err != nil {
return res, err
}
Expand Down Expand Up @@ -324,6 +348,12 @@ func accumulate(s *summary, r repoResult) {
if r.Valid {
s.Valid++
}
if r.Compiled {
s.Compiled++
}
for k, v := range r.CompileCodes {
s.CompileCodes[k] += v
}
if len(r.ColumnDiff) > 0 {
s.ReposChangedCols++
}
Expand Down Expand Up @@ -375,7 +405,12 @@ func markdown(s *summary) string {
fmt.Fprintf(&b, "# Workflow model survey\n\n")
fmt.Fprintf(&b, "| | |\n|---|---|\n")
fmt.Fprintf(&b, "| Repositories | %d |\n| with GitHub Actions pipelines | %d |\n| with Tekton pipelines | %d |\n", s.Repos, s.ReposWithGHA, s.ReposWithTekton)
fmt.Fprintf(&b, "| Files read | %d |\n| Files parsed | %d (%.1f%%) |\n| Pipelines | %d |\n| Models valid | %d / %d |\n| Repos whose columns change | %d |\n\n", s.Files, s.Parsed, pct(s.Parsed, s.Files), s.Pipelines, s.Valid, s.Repos, s.ReposChangedCols)
fmt.Fprintf(&b, "| Files read | %d |\n| Files parsed | %d (%.1f%%) |\n| Pipelines | %d |\n| Models valid | %d / %d |\n| Lifecycle models compiled | %d / %d |\n| Repos whose columns change | %d |\n\n", s.Files, s.Parsed, pct(s.Parsed, s.Files), s.Pipelines, s.Valid, s.Repos, s.Compiled, s.Repos, s.ReposChangedCols)
b.WriteString("## Lifecycle compile diagnostics\n\n| Code | Count |\n|---|---|\n")
for _, k := range sortedKeys(s.CompileCodes) {
fmt.Fprintf(&b, "| %s | %d |\n", k, s.CompileCodes[k])
}
b.WriteString("\n")
b.WriteString("## Pipeline roles\n\n| Role | Pipelines |\n|---|---|\n")
for _, k := range sortedKeys(s.Roles) {
fmt.Fprintf(&b, "| %s | %d |\n", k, s.Roles[workflowmodel.Role(k)])
Expand Down
30 changes: 30 additions & 0 deletions src/docs/design/workflow-models.md
Original file line number Diff line number Diff line change
Expand Up @@ -125,3 +125,33 @@ kind is not covered. Models are whole (no inheritance) through phase 4.
columns, numbers, error}}}` (nil when nothing is bound). The JS labels the model column,
renders a per-row column strip by stage number, and shows an amber "Not covered" chevron for
abstentions. Settings → Repos has a per-repo Workflow select that PUTs the binding.

## Ingest → lifecycle (RFC phase 3a)

`ingest.Repository` (phase 0) is unchanged and still produces the process half. `ingest.Lifecycle`
composes on top of it:

1. Start from `hivedefault.Generate()` — the compiled hive-default lifecycle, `derivation: builtin`.
2. Copy the ingested pipelines, sources and coverage; `meta.source` is the system(s) read,
`meta.provenance` is `ingested-unreviewed`, `meta.version` is `1` (the store assigns later
versions).
3. PR-verify pipelines are bound to every state in the CI / Review column, each with a citation to
the file the pipeline came from.
4. Every post-merge state the process half produced (main CI, release, `deploy-<env>`, done)
becomes a state in its own column after Merged, `confidence: inferred`, with the environment
guards carried over. The chain's transitions fire on `pipeline-succeeded` =
`pipeline.status is green`. The engine does not observe post-merge runs yet and reports
`unknown` for that fact, so a bound ingested model derives exactly like hive-default today;
the extra columns render, nothing lands in them until post-merge observation exists.

The compiler warns once per inferred element; a human confirming them in the editor (phase 4)
is what moves provenance to `local`/`curated`.

`POST /api/workflows/ingest` is the spoke entry point (owner role): `{repo}` fetches the default
branch through `FetchDefaultBranchTree`, `ingest.ReadTree` keeps GitHub Actions workflows, YAML
under Tekton directories and pipeline-named YAML that looks like Tekton (the survey collector's
rules), and the checkout is removed. `{files}` accepts pasted definitions. `save: true` stores the
result at the next integer version under `name` (default: the repository slug).

`cmd/workflow-survey` now compiles every corpus repository's lifecycle model (`lifecycle.json`
alongside `model.json`) and reports pass/refusal counts by diagnostic code.
1 change: 1 addition & 0 deletions src/pkg/dashboard/api_workflows.go
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,7 @@ func (s *Server) registerWorkflowRoutes() {
s.mux.HandleFunc("GET /api/workflows", s.handleWorkflowList)
s.mux.HandleFunc("POST /api/workflows", s.handleWorkflowSave)
s.mux.HandleFunc("POST /api/workflows/compile", s.handleWorkflowCompile)
s.mux.HandleFunc("POST /api/workflows/ingest", s.handleWorkflowIngest)
s.mux.HandleFunc("GET /api/workflows/bindings", s.handleWorkflowBindingsGet)
s.mux.HandleFunc("PUT /api/workflows/bindings", s.handleWorkflowBindingPut)
s.mux.HandleFunc("GET /api/workflows/{name}/{version}", s.handleWorkflowGet)
Expand Down
Loading
Loading