Skip to content

docs: add blog-artifact-check to the pre-publish gate - #486

Merged
williamzujkowski merged 1 commit into
mainfrom
pipeline-artifact-check
Aug 19, 2026
Merged

docs: add blog-artifact-check to the pre-publish gate#486
williamzujkowski merged 1 commit into
mainfrom
pipeline-artifact-check

Conversation

@williamzujkowski

Copy link
Copy Markdown
Owner

Adds a sixth Layer-1 audit and documents it in the pipeline diagram and the concern-ownership table.

The gap it fills. Auditing ninety agent-assisted posts, the highest-yield defect was never a wrong sentence. It was an artifact the post pointed at: a config file the tool silently ignores, a flag that does not exist, a gist created in a 21-second batch weeks after the post that presents it as working notes.

blog-factcheck verifies claims against cited sources. Nothing verified the config against the tool that has to accept it.

Why it needs to be Layer 1 rather than CI. Machine-generated config is plausible by construction, and every format forgives it:

tomllib.loads('[scanning]\nworkers = 4\n')   # parses clean
yaml.safe_load('expiration: 2025-12-31')      # parses clean

No parser errors. No test covers it, because a test would have to assert the tool read the key rather than that the file parsed. CI cannot see a key that is silently ignored. Catching it requires fetching the upstream schema and comparing key by key, which is judgement, not a grep.

What it checks:

  1. Provenance — gist creation timestamps against the post date, batch-creation spans, line counts against stated counts, and git log -S for a commit message that states the motive.
  2. Every key against upstream — one at a time, from the struct definition or CLI reference, including semantics. A flag can exist and do the opposite.
  3. Does the artifact match the prose — a job named as a gate that never exits non-zero, claimed parallelism with no matrix, "validates tests pass" with no test step, error paths that return empty/zero/false.
  4. Provenance of the prose — commit bodies that state an optimisation target, and the corollary that hedges written in the same pass as the numbers they disclaim are not evidence of care.

Calibrated on real findings from the audit: a grype expiration absent from IgnoreRule; an entire osv-scanner.toml of invented keys; trivy --policy, whose real counterpart filters findings out rather than denying.

The strongest single signal is a measurement attached to an invented key. "workers = 4 is 40% faster" cannot have been measured if the key has never existed, and it means the surrounding numbers should be treated as generated too.

Auditing ninety agent-assisted posts found the highest-yield defect was never a
wrong sentence. It was an artifact the post pointed at: a config file the tool
silently ignores, a flag that does not exist, a gist created in a 21-second
batch weeks after the post that presents it as working notes.

None of the existing Layer-1 audits look at those. blog-factcheck verifies
claims against cited sources; nothing verified the config against the tool that
has to accept it.

The gap is structural rather than an oversight. Machine-generated config is
plausible by construction, and every format forgives it:

  tomllib.loads('[scanning]\nworkers = 4\n')   # parses clean
  yaml.safe_load('expiration: 2025-12-31')      # parses clean

No parser errors, no test covers it, and CI cannot see a key the tool ignores.
It needs the upstream schema fetched and compared key by key, which is a
Layer-1 judgement call, not a grep.

Calibrated on real findings: a grype 'expiration' absent from IgnoreRule so the
suppression never lapsed; an entire osv-scanner.toml of invented keys; Trivy
--policy, whose real counterpart has inverted semantics. And the strongest
signal of all, a measurement attached to an invented key, which cannot have
been taken.
@williamzujkowski
williamzujkowski merged commit 67c1ef8 into main Aug 19, 2026
11 checks passed
@williamzujkowski
williamzujkowski deleted the pipeline-artifact-check branch August 19, 2026 06:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant