docs: add blog-artifact-check to the pre-publish gate - #486
Merged
Conversation
Auditing ninety agent-assisted posts found the highest-yield defect was never a
wrong sentence. It was an artifact the post pointed at: a config file the tool
silently ignores, a flag that does not exist, a gist created in a 21-second
batch weeks after the post that presents it as working notes.
None of the existing Layer-1 audits look at those. blog-factcheck verifies
claims against cited sources; nothing verified the config against the tool that
has to accept it.
The gap is structural rather than an oversight. Machine-generated config is
plausible by construction, and every format forgives it:
tomllib.loads('[scanning]\nworkers = 4\n') # parses clean
yaml.safe_load('expiration: 2025-12-31') # parses clean
No parser errors, no test covers it, and CI cannot see a key the tool ignores.
It needs the upstream schema fetched and compared key by key, which is a
Layer-1 judgement call, not a grep.
Calibrated on real findings: a grype 'expiration' absent from IgnoreRule so the
suppression never lapsed; an entire osv-scanner.toml of invented keys; Trivy
--policy, whose real counterpart has inverted semantics. And the strongest
signal of all, a measurement attached to an invented key, which cannot have
been taken.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds a sixth Layer-1 audit and documents it in the pipeline diagram and the concern-ownership table.
The gap it fills. Auditing ninety agent-assisted posts, the highest-yield defect was never a wrong sentence. It was an artifact the post pointed at: a config file the tool silently ignores, a flag that does not exist, a gist created in a 21-second batch weeks after the post that presents it as working notes.
blog-factcheckverifies claims against cited sources. Nothing verified the config against the tool that has to accept it.Why it needs to be Layer 1 rather than CI. Machine-generated config is plausible by construction, and every format forgives it:
No parser errors. No test covers it, because a test would have to assert the tool read the key rather than that the file parsed. CI cannot see a key that is silently ignored. Catching it requires fetching the upstream schema and comparing key by key, which is judgement, not a grep.
What it checks:
git log -Sfor a commit message that states the motive.Calibrated on real findings from the audit: a grype
expirationabsent fromIgnoreRule; an entireosv-scanner.tomlof invented keys;trivy --policy, whose real counterpart filters findings out rather than denying.The strongest single signal is a measurement attached to an invented key. "
workers = 4is 40% faster" cannot have been measured if the key has never existed, and it means the surrounding numbers should be treated as generated too.