Skip to content

fix: track complypack digests in generation state (#583)#592

Open
jpower432 wants to merge 2 commits into
complytime:mainfrom
jpower432:fix/complypack-digest-tracking
Open

fix: track complypack digests in generation state (#583)#592
jpower432 wants to merge 2 commits into
complytime:mainfrom
jpower432:fix/complypack-digest-tracking

Conversation

@jpower432

@jpower432 jpower432 commented Jun 17, 2026

Copy link
Copy Markdown
Member

Summary

  • Add ComplypackDigests map[string]string to GenerationState so freshness detection accounts for complypack changes, not just policy digest changes
  • Update IsFresh(), checkGenerationFreshness(), needsRegeneration(), runGeneration(), and saveGenerationAndPrint() to thread complypack digests through the full generation pipeline
  • Backward compatible: existing state files without complypack_digests deserialize with nil (triggers one-time regeneration when complypacks are present)

Related Issues

Fixes #583

Review Hints

  • make test-unit — all tests pass
  • make lint — 0 issues
  • make vet — clean
  • New unit tests for IsFresh with complypack digest scenarios (changed, added, removed, nil-vs-empty)
  • New unit tests for complypackDigestsByEvaluator helper (empty, incomplete entries, valid extraction)
  • New unit test for needsRegeneration with changed complypack digest
  • Backward-compat test: JSON without complypack_digests round-trips correctly
  • Review council: 5/5 APPROVE (Adversary, Architect 9/10, Guard, Tester, SRE)

jpower432 added 2 commits June 16, 2026 20:56
@jpower432
fix: track complypack digests in generation state (complytime#583)
5ac32aa 
GenerationState only tracked PolicyDigest for freshness detection.
When a complypack was updated without a policy change, regeneration
was skipped and providers used stale artifacts.

Add ComplypackDigests (evaluatorID → digest) to GenerationState and
check them alongside the policy digest in IsFresh. Both the scan and
generate code paths now record and compare complypack digests.

Assisted-by: Claude (Anthropic, Claude Opus 4.6)
Signed-off-by: Jennifer Power <barnabei.jennifer@gmail.com>
@jpower432
fix: address review findings and update CRAP baseline
fa1c587 
- Add CHANGELOG entry for complypack digest tracking (complytime#583)
- Add TestIsFresh_ComplypackDigestRemoved test case
- Fix misleading "policy unchanged" reuse message to
  "unchanged since last generate"
- Regenerate CRAP baseline (IsFresh: 1 → 2, legitimate
  complexity increase from added branch)

Assisted-by: Claude (Anthropic, Claude Opus 4.6)
Signed-off-by: Jennifer Power <barnabei.jennifer@gmail.com>
@jpower432 jpower432 marked this pull request as ready for review June 17, 2026 19:39
@jpower432 jpower432 requested a review from a team as a code owner June 17, 2026 19:39
@jpower432 jpower432 requested review from gvauter, gxmiranda, hbraswelrh and yvonnedevlinrh and removed request for gxmiranda and yvonnedevlinrh June 17, 2026 19:39
hbraswelrh
hbraswelrh approved these changes Jun 17, 2026
View reviewed changes

@hbraswelrh hbraswelrh left a comment •
edited
Loading

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Left one comment with some context from Issue #complytime/org-infra#341

Compliance Scan Report

Compliance Scan Report: complytime/policies-ampel-branch-protection:latest

Generated: 2026-06-17T21:01:16Z

Control: force-push-restriction

  • Result: Passed
  • Message: 1 of 1 repositories passed

block-force-push

  • Confidence: High
  • Result: Passed
  • Message: 1 of 1 repositories passed
  • Steps Executed: 1

Control: approval-requirements

  • Result: Passed
  • Message: 3 of 3 repositories passed

minimum-approvals

  • Confidence: High
  • Result: Passed
  • Message: 3 of 3 repositories passed
  • Steps Executed: 3

Control: admin-bypass-prevention

  • Result: Passed
  • Message: 1 of 1 repositories passed

prevent-admin-bypass

  • Confidence: High
  • Result: Passed
  • Message: 1 of 1 repositories passed
  • Steps Executed: 1

Control: code-owner-enforcement

  • Result: Passed
  • Message: 1 of 1 repositories passed

require-code-owner-review

  • Confidence: High
  • Result: Passed
  • Message: 1 of 1 repositories passed
  • Steps Executed: 1

Control: pull-request-enforcement

  • Result: Passed
  • Message: 1 of 1 repositories passed

require-pull-request

  • Confidence: High
  • Result: Passed
  • Message: 1 of 1 repositories passed
  • Steps Executed: 1

Evaluation Log

metadata:
  id: complytime/policies-ampel-branch-protection:latest
  type: EvaluationLog
  gemara-version: v1.0.0
  description: Compliance scan evaluation log
  author:
    id: complytime
    name: complytime
    type: Software
    uri: https://github.com/complytime/complyctl
result: Passed
evaluations:
- name: force-push-restriction
  result: Passed
  message: 1 of 1 repositories passed
  control:
    reference-id: complytime/policies-ampel-branch-protection:latest
    entry-id: force-push-restriction
  assessment-logs:
  - requirement:
      reference-id: complytime/policies-ampel-branch-protection:latest
      entry-id: block-force-push
    plan:
      reference-id: complytime/policies-ampel-branch-protection:latest
      entry-id: block-force-push
    description: 1 of 1 repositories passed
    result: Passed
    message: 1 of 1 repositories passed
    applicability:
    - default
    steps:
    - "complytime/complypack-ampel-branch-protection@sha256:8856c34cc7bf7d0f7149afd3b0d73027ee704d33057be5dc1a70eaef236e8b8e#complytime/complyctl@main"
    steps-executed: 1
    start: "2026-06-17T21:01:16Z"
    confidence-level: High
- name: approval-requirements
  result: Passed
  message: 3 of 3 repositories passed
  control:
    reference-id: complytime/policies-ampel-branch-protection:latest
    entry-id: approval-requirements
  assessment-logs:
  - requirement:
      reference-id: complytime/policies-ampel-branch-protection:latest
      entry-id: minimum-approvals
    plan:
      reference-id: complytime/policies-ampel-branch-protection:latest
      entry-id: minimum-approvals
    description: 3 of 3 repositories passed
    result: Passed
    message: 3 of 3 repositories passed
    applicability:
    - default
    steps:
    - "complytime/complypack-ampel-branch-protection@sha256:8856c34cc7bf7d0f7149afd3b0d73027ee704d33057be5dc1a70eaef236e8b8e#complytime/complyctl@main"
    - "complytime/complypack-ampel-branch-protection@sha256:8856c34cc7bf7d0f7149afd3b0d73027ee704d33057be5dc1a70eaef236e8b8e#complytime/complyctl@main"
    - "complytime/complypack-ampel-branch-protection@sha256:8856c34cc7bf7d0f7149afd3b0d73027ee704d33057be5dc1a70eaef236e8b8e#complytime/complyctl@main"
    steps-executed: 3
    start: "2026-06-17T21:01:16Z"
    confidence-level: High
- name: admin-bypass-prevention
  result: Passed
  message: 1 of 1 repositories passed
  control:
    reference-id: complytime/policies-ampel-branch-protection:latest
    entry-id: admin-bypass-prevention
  assessment-logs:
  - requirement:
      reference-id: complytime/policies-ampel-branch-protection:latest
      entry-id: prevent-admin-bypass
    plan:
      reference-id: complytime/policies-ampel-branch-protection:latest
      entry-id: prevent-admin-bypass
    description: 1 of 1 repositories passed
    result: Passed
    message: 1 of 1 repositories passed
    applicability:
    - default
    steps:
    - "complytime/complypack-ampel-branch-protection@sha256:8856c34cc7bf7d0f7149afd3b0d73027ee704d33057be5dc1a70eaef236e8b8e#complytime/complyctl@main"
    steps-executed: 1
    start: "2026-06-17T21:01:16Z"
    confidence-level: High
- name: code-owner-enforcement
  result: Passed
  message: 1 of 1 repositories passed
  control:
    reference-id: complytime/policies-ampel-branch-protection:latest
    entry-id: code-owner-enforcement
  assessment-logs:
  - requirement:
      reference-id: complytime/policies-ampel-branch-protection:latest
      entry-id: require-code-owner-review
    plan:
      reference-id: complytime/policies-ampel-branch-protection:latest
      entry-id: require-code-owner-review
    description: 1 of 1 repositories passed
    result: Passed
    message: 1 of 1 repositories passed
    applicability:
    - default
    steps:
    - "complytime/complypack-ampel-branch-protection@sha256:8856c34cc7bf7d0f7149afd3b0d73027ee704d33057be5dc1a70eaef236e8b8e#complytime/complyctl@main"
    steps-executed: 1
    start: "2026-06-17T21:01:16Z"
    confidence-level: High
- name: pull-request-enforcement
  result: Passed
  message: 1 of 1 repositories passed
  control:
    reference-id: complytime/policies-ampel-branch-protection:latest
    entry-id: pull-request-enforcement
  assessment-logs:
  - requirement:
      reference-id: complytime/policies-ampel-branch-protection:latest
      entry-id: require-pull-request
    plan:
      reference-id: complytime/policies-ampel-branch-protection:latest
      entry-id: require-pull-request
    description: 1 of 1 repositories passed
    result: Passed
    message: 1 of 1 repositories passed
    applicability:
    - default
    steps:
    - "complytime/complypack-ampel-branch-protection@sha256:8856c34cc7bf7d0f7149afd3b0d73027ee704d33057be5dc1a70eaef236e8b8e#complytime/complyctl@main"
    steps-executed: 1
    start: "2026-06-17T21:01:16Z"
    confidence-level: High
target:
  id: complytime-complyctl
  name: complytime-complyctl
  type: Software

Comment thread internal/policy/generation_state_test.go
func TestIsFresh_ComplypackDigestChanged(t *testing.T) {
s := &GenerationState{
PolicyDigest: "sha256:abc123",
ComplypackDigests: map[string]string{"opa": "sha256:old"},

@hbraswelrh hbraswelrh Jun 17, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@jpower432 I ran into an issue earlier when testing the quay registry link to the complypack-ampel-branch-protection in the complytime.yaml workspace config. This may serve as an additional test or consideration for the TestIsFresh_ComplypackDigestChanged.

The issue was caused by the : oci format used to reference the complypack at quay.io/complytime/complypack-ampel-branch-protection:v0.4.0. I replaced the complypacks entry with quay.io/complytime/complypack-ampel-branch-protection@v0.4.0 using the @ which uses the tagged v0.4.0 from the org-infra GHCR release.

I believe that until the release is cut in quay.io the complypack cannot be referenced in the oci format :v0.4.0. However, once the release is cut, it should be possible to point the complytime.yaml workspace config policies and complypacks to the same :version/sha for executing the get, generate, and scan commands.

See comment on the quay release issue here.

@jpower432 jpower432 Jun 17, 2026

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@hbraswelrh Seems like it might be a bug in complyctl then. The policies use : still, right? : is usually associated with a tag a @ with a digest so that can get confusing fast.

@hbraswelrh hbraswelrh Jun 17, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah. That's what I was thinking. They wither will have to be released together and pinned together or complypack will need to support the oci format necessary to reference the registry image with : in the workspace config.

@hbraswelrh hbraswelrh Jun 18, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@jpower432 ill open an issue and work on this today.

@hbraswelrh hbraswelrh Jun 18, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Opened a 🐛 issue here

@hbraswelrh hbraswelrh mentioned this pull request Jun 18, 2026
Open
gvauter
gvauter approved these changes Jun 18, 2026
View reviewed changes

@gvauter gvauter left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@hbraswelrh hbraswelrh hbraswelrh approved these changes

@gvauter gvauter gvauter approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Generation state does not track complypack digests

3 participants

@jpower432 @gvauter @hbraswelrh