Skip to content

feat: add comply pipeline workflow#40

Open
jpower432 wants to merge 16 commits into
complytime:mainfrom
jpower432:feat/comply-pipeline
Open

feat: add comply pipeline workflow#40
jpower432 wants to merge 16 commits into
complytime:mainfrom
jpower432:feat/comply-pipeline

Conversation

@jpower432

@jpower432 jpower432 commented Jun 15, 2026

Copy link
Copy Markdown
Member

Summary

This PR adds a user invocable command for running with multiple stages - scoping from a system profile + mapping determines imports and parameters harmonization + adherence to determine what evidence will be collection.

Related Issues

Blocked by #31
Closes #26
Demo'd using #39

Review Hints

 /comply:pipeline   # scoping → mapping → adherence
 /comply:pack       # generates Rego from the child policy
 ls .complytime/          # scoping.yaml, delta-report.yaml, child-policy.yaml
 ls policy/               # *.rego files

@jpower432 jpower432 changed the title feat: add comply pipelin workflow feat: add comply pipeline workflow Jun 15, 2026
@jpower432 jpower432 force-pushed the feat/comply-pipeline branch 2 times, most recently from c1f5db2 to df96f2f Compare June 16, 2026 19:43
@jpower432 jpower432 marked this pull request as ready for review June 16, 2026 21:00
@jpower432 jpower432 requested a review from a team as a code owner June 16, 2026 21:00
@jpower432 jpower432 requested review from em-redhat and removed request for a team June 16, 2026 21:00
jpower432 added 14 commits June 16, 2026 17:42
Add repeatable --source and --schema flags to the mcp serve command,
allowing direct configuration without a YAML file. When --source flags
are present, a ComplyPackConfig is built from flag values; otherwise
the existing --config file path is used.

- parseSourceFlags: handles oci:// (TLS) and oci+http:// (plain HTTP)
- parseSchemaFlags: handles bare platform names and platform=source syntax
- Refactor NewServer to accept ServerOptions.Config directly

Assisted-by: Claude (Anthropic, Claude Opus 4.6)
Signed-off-by: Jennifer Power <barnabei.jennifer@gmail.com>
Remove hardcoded version "1.0" from buildConfigFromFlags in mcp.go
since the MCP server does not use the version field (it's only needed
for pack/scan commands).

Add comprehensive test for buildConfigFromFlags to verify complete
flag-to-config transformation including source parsing, schema parsing,
and proper struct field population.

Assisted-by: Claude (Anthropic, Claude Opus 4.6)
Signed-off-by: Jennifer Power <barnabei.jennifer@gmail.com>
Add delta comparison engine for parameter harmonization across
framework layers with mismatch-only verdicts. Add analyze_parameter_delta
MCP tool. Extend get_assessment_requirements with scope filter (array
of applicability groups) so models can query by maturity level without
parsing catalog files. Include artifact kind (Policy, ControlCatalog,
etc.) in MCP resource listing. Add ImportedGuidanceIDs to ResolvedPolicy.

Assisted-by: Claude (Anthropic, Claude Opus 4.6)
Signed-off-by: Jennifer Power <barnabei.jennifer@gmail.com>
Add comply pipeline skills (scoping, mapping, adherence) with router
that dispatches sub-stages by filename from the skill base directory.
Add /comply:pack for Rego generation and /comply:setup for workspace
configuration. Skills enforce MCP-grounded control data access via
get_assessment_requirements with scope filter. Update plugin manifests
to register new commands.

Assisted-by: Claude (Anthropic, Claude Opus 4.6)
Signed-off-by: Jennifer Power <barnabei.jennifer@gmail.com>
…mply pipeline

Assisted-by: Claude (Anthropic, Claude Opus 4.6)
Signed-off-by: Jennifer Power <barnabei.jennifer@gmail.com>
Strip verdicts, specificity detection, and string-matching heuristics
from the delta engine. The tool now gathers structured L3 parameter
values alongside L1/L2 requirement text and returns them as pairs.
The model interprets the relationship — parsing prose for parameter
values is what AI does well and heuristics do poorly.

Assisted-by: Claude (Anthropic, Claude Opus 4.6)
Signed-off-by: Jennifer Power <barnabei.jennifer@gmail.com>
Signed-off-by: Jennifer Power <barnabei.jennifer@gmail.com>
Remove verdict types, specificity layers, and heuristic references.
Mapping skill now instructs the model to interpret parameter
comparisons using domain context rather than relying on engine
verdicts. Output schema uses comparisons with interpretation field.

Assisted-by: Claude (Anthropic, Claude Opus 4.6)
Signed-off-by: Jennifer Power <barnabei.jennifer@gmail.com>
The tool previously only looked up resolved policies by name. When a
catalog name was passed, it failed with "policy not found". Now falls
back to wrapping a bare catalog in a synthetic ResolvedPolicy so the
tool works with both policy and catalog names.

Assisted-by: Claude (Anthropic, Claude Opus 4.6)
Signed-off-by: Jennifer Power <barnabei.jennifer@gmail.com>
Signed-off-by: Jennifer Power <barnabei.jennifer@gmail.com>
Signed-off-by: Jennifer Power <barnabei.jennifer@gmail.com>
Use hyphenated field names (mapping-references, assessment-plans,
evaluation-methods, accepted-values, reference-id). Add required
fields: title (top-level), metadata.author, contacts, scope with
applicability groups. Fix evaluation-methods to use id/type/mode
structure. Add id fields to assessment plans and parameters.

Assisted-by: Claude (Anthropic, Claude Opus 4.6)
Signed-off-by: Jennifer Power <barnabei.jennifer@gmail.com>
Fixes golangci-lint unused finding.

Assisted-by: Claude (Anthropic, Claude Opus 4.6)
Signed-off-by: Jennifer Power <barnabei.jennifer@gmail.com>
The updated reusable_ci.yml in org-infra added pull-requests: read to
its megalinter job, causing a startup_failure when the caller didn't
grant that permission.

Assisted-by: Claude (Anthropic, Claude Opus 4.6)
Signed-off-by: Jennifer Power <barnabei.jennifer@gmail.com>
@jpower432 jpower432 force-pushed the feat/comply-pipeline branch from df96f2f to dc4e63d Compare June 16, 2026 21:44
Add missing newline at EOF in delta.go, add language specifiers to
fenced code blocks, and fix table pipe alignment in skill docs.

Assisted-by: Claude (Anthropic, Claude Opus 4.6)
Signed-off-by: Jennifer Power <barnabei.jennifer@gmail.com>

@em-redhat em-redhat left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PR Review: #40 — feat: add comply pipeline workflow

CI Status

Check Status Classification
Test (1.26) PASS N/A
Standardized CI / Run linters PASS N/A
Security Scan / OpenSSF Scorecards PASS N/A
Vulnerability Scan / OSV-Scanner / osv-scan PASS N/A
Vulnerability Scan / Trivy Source Scan SKIPPED N/A

Local Tool Results

Local tool CI check that covers it CI status Run locally?
golangci-lint Standardized CI / Run linters PASS No
go test Test (1.26) PASS No

All CI checks pass. No local tool execution needed.

Summary

This PR adds a multi-stage compliance pipeline workflow (scoping → mapping → adherence) exposed as plugin skills, backed by new MCP tools (analyze_parameter_delta, enhanced get_assessment_requirements with scope filtering). The delta engine design is sound — it gathers structured parameter pairs and defers interpretation to the model, which aligns with ADR 014. Code is well-tested and well-structured.

What Was Checked

  • CI: All checks pass — tests, linting, security scans, vulnerability scans
  • Alignment: PR changes match stated intent (issue #26, ADRs 012-015). Scope is consistent.
  • Security: Reviewed resolveFromCatalog synthetic policy construction, input handling in MCP tool handlers, no hardcoded secrets, no injection vectors, no file path construction from external input.
  • Architecture: Reviewed separation between delta engine (internal/requirement/delta.go), MCP tool layer (internal/mcp/tool_delta.go), and skill documentation. Clean package boundaries maintained.
  • Convention packs: Checked against default, go, and severity packs.

Findings

One MEDIUM finding on test fixture consistency — see inline comments.

Verdict

APPROVE

Well-structured PR with clean separation of concerns, comprehensive tests, and clear ADR rationale.

This review was generated by /review-pr (AI-assisted).

}

func TestAnalyzeDelta(t *testing.T) {
set := testDeltaArtifactSet()

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[MEDIUM] [TC-001] This ArtifactSet is missing the Mappings field introduced in this PR. While Go zero-values uninitialized map fields to nil (which doesn't cause panics in the current code path), this is inconsistent with NewArtifactSet() which initializes all fields including Mappings. If future code writes to Mappings on this test set, it will panic.

Suggested change
set := testDeltaArtifactSet()
return &ArtifactSet{
Catalogs: map[string]*gemara.ControlCatalog{"container-baseline": catalog},
Policies: map[string]*gemara.Policy{"org-parent-policy": policy},
Guidance: make(map[string]*gemara.GuidanceCatalog),
Mappings: make(map[string]*gemara.MappingDocument),
}

}

set := &requirement.ArtifactSet{
Catalogs: map[string]*gemara.ControlCatalog{"container-baseline": catalog},

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[MEDIUM] [TC-001] Same issue here — ArtifactSet missing Mappings field. Recommend adding Mappings: make(map[string]*gemara.MappingDocument) for consistency with NewArtifactSet().

Suggested change
Catalogs: map[string]*gemara.ControlCatalog{"container-baseline": catalog},
set := &requirement.ArtifactSet{
Catalogs: map[string]*gemara.ControlCatalog{"container-baseline": catalog},
Policies: map[string]*gemara.Policy{"org-policy": policy},
Guidance: make(map[string]*gemara.GuidanceCatalog),
Mappings: make(map[string]*gemara.MappingDocument),
}

@hbraswelrh hbraswelrh left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

APPROVE with 2 comments (novel findings not covered by prior review).

This review was generated by /review-pr (AI-assisted).

assert.Empty(t, params)
})
}

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[MEDIUM] [TC-001] ArtifactSet fixture missing Mappings field — same pattern @em-redhat flagged in delta_test.go and tool_delta_test.go, but this instance was not covered. Inconsistent with NewArtifactSet() which initializes all fields.

Suggested change
set := &ArtifactSet{
Catalogs: map[string]*gemara.ControlCatalog{"test-catalog": catalog},
Policies: map[string]*gemara.Policy{"test-policy": policy},
Guidance: map[string]*gemara.GuidanceCatalog{"guidance-1": guidanceCatalog},
Mappings: make(map[string]*gemara.MappingDocument),
}

// ParameterComparison pairs a structured L3 parameter with the
// L1/L2 requirement text it maps to. The caller interprets the
// relationship — the engine does not judge.
type ParameterComparison struct {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[MEDIUM] [DR-001] Exported struct fields lack GoDoc comments. The JSON tags document wire format but not semantics — e.g., what distinguishes PolicySource from CatalogSource, or what Label represents in the domain model. Consider adding field-level comments for downstream consumers.

Address PR complytime#40 review comments: add missing Mappings field to all
test ArtifactSet literals for consistency with NewArtifactSet(), and
add GoDoc comments to ParameterComparison exported fields.

Assisted-by: Claude (Anthropic, Claude Opus 4.6)
Signed-off-by: Jennifer Power <barnabei.jennifer@gmail.com>
@marcusburghardt

Copy link
Copy Markdown
Member

Value

Ran the full pipeline locally (scoping → mapping → adherence → pack). The separation
of concerns is clear and effective:

  • MCP tools handle data retrievalget_assessment_requirements with scope
    filtering and analyze_parameter_delta remove the manual cross-referencing work.
  • AI handles interpretation — reading prose requirements and deciding if a
    concrete parameter value satisfies them. This is where heuristics fail and LLMs
    actually add value.
  • Humans keep the decisions — maturity level, system classification, and
    parameter acceptance stay with the user.

The "gatherer, not judge" design (ADR 014) is the right call. Each stage produces
an auditable YAML artifact, not a chat transcript.

Commit history

There are places where a commit introduces something and the next one immediately
fixes or reworks it within the same session. A few examples:

  • ce66915 adds the delta engine with verdicts, then 572ca66 (31 min later)
    redesigns it as a gatherer. f26f877 and b315548 clean up leftovers. These
    four commits are one design iteration.
  • 8e8f3ca adds the skills, then ed67ab5, 4760978, and b9c33e9 are
    successive fixups to the adherence YAML snippet.
  • aef7b01 adds --source/--schema flags, then 1bdfb41 (15 min later) removes
    a hardcoded version it introduced.

Completely normal during development. The suggestion is to squash before merge so
git log on main reads as intentional steps. An interactive rebase could bring
16 commits down to ~7 clean units:

  1. feat: add --source and --schema flags to mcp serve
  2. feat: add parameter delta engine, scope filter, and resource listing
  3. feat: add /comply plugin with pipeline, pack, and setup skills
  4. docs: add ADRs 012-015
  5. fix: allow get_assessment_requirements to accept catalog names
  6. fix: CI permission and lint errors
  7. fix: initialize Mappings field in test ArtifactSet fixtures

Local testing with OpenCode

Testing this PR locally with OpenCode required some extra steps worth documenting:

  1. Skill discovery mismatch: The skills live in skills/pipeline/SKILL.md but
    OpenCode expects .opencode/skills/pipeline/SKILL.md. I had to create symlinks
    under .opencode/skills/. Claude Code discovers them via the plugin manifest
    instead. If the team uses both tools, consider adding .opencode/skills/ symlinks
    to the repo (.opencode/ is already gitignored) or documenting the setup.

  2. MCP server wiring: OpenCode uses .mcp.json while Claude Code uses its own
    plugin system. I created a temporary .mcp.json pointing to a locally built
    binary with test artifacts. This worked, but required building from source and
    creating sample Gemara YAML files since the repo has no standalone test fixtures —
    all test data is inline Go string constants.

  3. No test fixtures shipped: Having a testdata/ or examples/ directory with
    a minimal catalog + policy YAML would make local testing straightforward for any
    tool. Right now, the reviewer has to reverse-engineer the expected YAML format
    from Go test files.

A short "Testing the pipeline locally" section in the PR description or a
examples/ directory with sample artifacts would lower the barrier for reviewers
using any tool.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Generate scoped Gemara Policy from ControlCatalog and system profile

4 participants