feat: add comply pipeline workflow#40
Conversation
c1f5db2 to
df96f2f
Compare
Add repeatable --source and --schema flags to the mcp serve command, allowing direct configuration without a YAML file. When --source flags are present, a ComplyPackConfig is built from flag values; otherwise the existing --config file path is used. - parseSourceFlags: handles oci:// (TLS) and oci+http:// (plain HTTP) - parseSchemaFlags: handles bare platform names and platform=source syntax - Refactor NewServer to accept ServerOptions.Config directly Assisted-by: Claude (Anthropic, Claude Opus 4.6) Signed-off-by: Jennifer Power <barnabei.jennifer@gmail.com>
Remove hardcoded version "1.0" from buildConfigFromFlags in mcp.go since the MCP server does not use the version field (it's only needed for pack/scan commands). Add comprehensive test for buildConfigFromFlags to verify complete flag-to-config transformation including source parsing, schema parsing, and proper struct field population. Assisted-by: Claude (Anthropic, Claude Opus 4.6) Signed-off-by: Jennifer Power <barnabei.jennifer@gmail.com>
Add delta comparison engine for parameter harmonization across framework layers with mismatch-only verdicts. Add analyze_parameter_delta MCP tool. Extend get_assessment_requirements with scope filter (array of applicability groups) so models can query by maturity level without parsing catalog files. Include artifact kind (Policy, ControlCatalog, etc.) in MCP resource listing. Add ImportedGuidanceIDs to ResolvedPolicy. Assisted-by: Claude (Anthropic, Claude Opus 4.6) Signed-off-by: Jennifer Power <barnabei.jennifer@gmail.com>
Add comply pipeline skills (scoping, mapping, adherence) with router that dispatches sub-stages by filename from the skill base directory. Add /comply:pack for Rego generation and /comply:setup for workspace configuration. Skills enforce MCP-grounded control data access via get_assessment_requirements with scope filter. Update plugin manifests to register new commands. Assisted-by: Claude (Anthropic, Claude Opus 4.6) Signed-off-by: Jennifer Power <barnabei.jennifer@gmail.com>
…mply pipeline Assisted-by: Claude (Anthropic, Claude Opus 4.6) Signed-off-by: Jennifer Power <barnabei.jennifer@gmail.com>
Strip verdicts, specificity detection, and string-matching heuristics from the delta engine. The tool now gathers structured L3 parameter values alongside L1/L2 requirement text and returns them as pairs. The model interprets the relationship — parsing prose for parameter values is what AI does well and heuristics do poorly. Assisted-by: Claude (Anthropic, Claude Opus 4.6) Signed-off-by: Jennifer Power <barnabei.jennifer@gmail.com>
Signed-off-by: Jennifer Power <barnabei.jennifer@gmail.com>
Remove verdict types, specificity layers, and heuristic references. Mapping skill now instructs the model to interpret parameter comparisons using domain context rather than relying on engine verdicts. Output schema uses comparisons with interpretation field. Assisted-by: Claude (Anthropic, Claude Opus 4.6) Signed-off-by: Jennifer Power <barnabei.jennifer@gmail.com>
The tool previously only looked up resolved policies by name. When a catalog name was passed, it failed with "policy not found". Now falls back to wrapping a bare catalog in a synthetic ResolvedPolicy so the tool works with both policy and catalog names. Assisted-by: Claude (Anthropic, Claude Opus 4.6) Signed-off-by: Jennifer Power <barnabei.jennifer@gmail.com>
Signed-off-by: Jennifer Power <barnabei.jennifer@gmail.com>
Signed-off-by: Jennifer Power <barnabei.jennifer@gmail.com>
Use hyphenated field names (mapping-references, assessment-plans, evaluation-methods, accepted-values, reference-id). Add required fields: title (top-level), metadata.author, contacts, scope with applicability groups. Fix evaluation-methods to use id/type/mode structure. Add id fields to assessment plans and parameters. Assisted-by: Claude (Anthropic, Claude Opus 4.6) Signed-off-by: Jennifer Power <barnabei.jennifer@gmail.com>
Fixes golangci-lint unused finding. Assisted-by: Claude (Anthropic, Claude Opus 4.6) Signed-off-by: Jennifer Power <barnabei.jennifer@gmail.com>
The updated reusable_ci.yml in org-infra added pull-requests: read to its megalinter job, causing a startup_failure when the caller didn't grant that permission. Assisted-by: Claude (Anthropic, Claude Opus 4.6) Signed-off-by: Jennifer Power <barnabei.jennifer@gmail.com>
df96f2f to
dc4e63d
Compare
Add missing newline at EOF in delta.go, add language specifiers to fenced code blocks, and fix table pipe alignment in skill docs. Assisted-by: Claude (Anthropic, Claude Opus 4.6) Signed-off-by: Jennifer Power <barnabei.jennifer@gmail.com>
em-redhat
left a comment
There was a problem hiding this comment.
PR Review: #40 — feat: add comply pipeline workflow
CI Status
| Check | Status | Classification |
|---|---|---|
| Test (1.26) | PASS | N/A |
| Standardized CI / Run linters | PASS | N/A |
| Security Scan / OpenSSF Scorecards | PASS | N/A |
| Vulnerability Scan / OSV-Scanner / osv-scan | PASS | N/A |
| Vulnerability Scan / Trivy Source Scan | SKIPPED | N/A |
Local Tool Results
| Local tool | CI check that covers it | CI status | Run locally? |
|---|---|---|---|
golangci-lint |
Standardized CI / Run linters | PASS | No |
go test |
Test (1.26) | PASS | No |
All CI checks pass. No local tool execution needed.
Summary
This PR adds a multi-stage compliance pipeline workflow (scoping → mapping → adherence) exposed as plugin skills, backed by new MCP tools (analyze_parameter_delta, enhanced get_assessment_requirements with scope filtering). The delta engine design is sound — it gathers structured parameter pairs and defers interpretation to the model, which aligns with ADR 014. Code is well-tested and well-structured.
What Was Checked
- CI: All checks pass — tests, linting, security scans, vulnerability scans
- Alignment: PR changes match stated intent (issue #26, ADRs 012-015). Scope is consistent.
- Security: Reviewed
resolveFromCatalogsynthetic policy construction, input handling in MCP tool handlers, no hardcoded secrets, no injection vectors, no file path construction from external input. - Architecture: Reviewed separation between delta engine (
internal/requirement/delta.go), MCP tool layer (internal/mcp/tool_delta.go), and skill documentation. Clean package boundaries maintained. - Convention packs: Checked against default, go, and severity packs.
Findings
One MEDIUM finding on test fixture consistency — see inline comments.
Verdict
APPROVE
Well-structured PR with clean separation of concerns, comprehensive tests, and clear ADR rationale.
This review was generated by /review-pr (AI-assisted).
| } | ||
|
|
||
| func TestAnalyzeDelta(t *testing.T) { | ||
| set := testDeltaArtifactSet() |
There was a problem hiding this comment.
[MEDIUM] [TC-001] This ArtifactSet is missing the Mappings field introduced in this PR. While Go zero-values uninitialized map fields to nil (which doesn't cause panics in the current code path), this is inconsistent with NewArtifactSet() which initializes all fields including Mappings. If future code writes to Mappings on this test set, it will panic.
| set := testDeltaArtifactSet() | |
| return &ArtifactSet{ | |
| Catalogs: map[string]*gemara.ControlCatalog{"container-baseline": catalog}, | |
| Policies: map[string]*gemara.Policy{"org-parent-policy": policy}, | |
| Guidance: make(map[string]*gemara.GuidanceCatalog), | |
| Mappings: make(map[string]*gemara.MappingDocument), | |
| } |
| } | ||
|
|
||
| set := &requirement.ArtifactSet{ | ||
| Catalogs: map[string]*gemara.ControlCatalog{"container-baseline": catalog}, |
There was a problem hiding this comment.
[MEDIUM] [TC-001] Same issue here — ArtifactSet missing Mappings field. Recommend adding Mappings: make(map[string]*gemara.MappingDocument) for consistency with NewArtifactSet().
| Catalogs: map[string]*gemara.ControlCatalog{"container-baseline": catalog}, | |
| set := &requirement.ArtifactSet{ | |
| Catalogs: map[string]*gemara.ControlCatalog{"container-baseline": catalog}, | |
| Policies: map[string]*gemara.Policy{"org-policy": policy}, | |
| Guidance: make(map[string]*gemara.GuidanceCatalog), | |
| Mappings: make(map[string]*gemara.MappingDocument), | |
| } |
hbraswelrh
left a comment
There was a problem hiding this comment.
APPROVE with 2 comments (novel findings not covered by prior review).
This review was generated by /review-pr (AI-assisted).
| assert.Empty(t, params) | ||
| }) | ||
| } | ||
|
|
There was a problem hiding this comment.
[MEDIUM] [TC-001] ArtifactSet fixture missing Mappings field — same pattern @em-redhat flagged in delta_test.go and tool_delta_test.go, but this instance was not covered. Inconsistent with NewArtifactSet() which initializes all fields.
| set := &ArtifactSet{ | |
| Catalogs: map[string]*gemara.ControlCatalog{"test-catalog": catalog}, | |
| Policies: map[string]*gemara.Policy{"test-policy": policy}, | |
| Guidance: map[string]*gemara.GuidanceCatalog{"guidance-1": guidanceCatalog}, | |
| Mappings: make(map[string]*gemara.MappingDocument), | |
| } |
| // ParameterComparison pairs a structured L3 parameter with the | ||
| // L1/L2 requirement text it maps to. The caller interprets the | ||
| // relationship — the engine does not judge. | ||
| type ParameterComparison struct { |
There was a problem hiding this comment.
[MEDIUM] [DR-001] Exported struct fields lack GoDoc comments. The JSON tags document wire format but not semantics — e.g., what distinguishes PolicySource from CatalogSource, or what Label represents in the domain model. Consider adding field-level comments for downstream consumers.
Address PR complytime#40 review comments: add missing Mappings field to all test ArtifactSet literals for consistency with NewArtifactSet(), and add GoDoc comments to ParameterComparison exported fields. Assisted-by: Claude (Anthropic, Claude Opus 4.6) Signed-off-by: Jennifer Power <barnabei.jennifer@gmail.com>
ValueRan the full pipeline locally (scoping → mapping → adherence → pack). The separation
The "gatherer, not judge" design (ADR 014) is the right call. Each stage produces Commit historyThere are places where a commit introduces something and the next one immediately
Completely normal during development. The suggestion is to squash before merge so
Local testing with OpenCodeTesting this PR locally with OpenCode required some extra steps worth documenting:
A short "Testing the pipeline locally" section in the PR description or a |
Summary
This PR adds a user invocable command for running with multiple stages -
scopingfrom a system profile +mappingdetermines imports and parameters harmonization +adherenceto determine what evidence will be collection.Related Issues
Blocked by #31
Closes #26
Demo'd using #39
Review Hints