diff --git a/src/posts/2026-08-18-checks-that-pass-for-the-wrong-reason.md b/src/posts/2026-08-18-checks-that-pass-for-the-wrong-reason.md index f26a92b8..8e2f6b35 100644 --- a/src/posts/2026-08-18-checks-that-pass-for-the-wrong-reason.md +++ b/src/posts/2026-08-18-checks-that-pass-for-the-wrong-reason.md @@ -2,204 +2,140 @@ author: William Zujkowski date: 2026-08-18 -description: "A gate that echoes a string and exits zero. A suppression field that does not exist. A scanner that reports clean because its error path returns an empty list. Notes from auditing every post I have written." -title: "Checks That Pass for the Wrong Reason" +description: "Ninety agent-assisted posts, audited against source. The defects weren't bad reasoning — they were artifacts nobody had ever executed. Why models produce that, why review can't see it, and what actually catches it." +title: "Nobody Ran It" tags: - security - - devops + - ai - automation --- -I spent the last stretch auditing every post on this site against the code, configs and papers they describe. Ninety posts. The single most common defect was not a wrong number, though there were plenty of those. It was a **control that looked enforced and was not** — and in almost every case the thing that made it invisible was that it *passed*. +I audited every post on this site against the code, configs and papers it describes. Ninety of them, most written with agent assistance. Almost all had defects. -A failing check gets fixed on the afternoon it fails. A check that passes for the wrong reason can sit there for a year while you build on top of it. +The interesting part is not that. It is that the defects had a **shape** — one specific enough to design against, and specific enough that I can tell you exactly why my review missed it. + +None of them were bad reasoning. Every one was an artifact that had never been executed.

hooked through, hanging open

-## The one that started it - -The oldest instance is mine, and it is a naming problem before it is a technical one. - -I built a convention for loading coding standards into a model's context. It looked like this: - -``` -@load [CS:python + TS:pytest + SEC:*] -``` - -I described it in writing as an "intelligent router." It is not a router. Nothing parses that line. It is a string in a file that a language model reads and usually honours, and "usually" is carrying the entire sentence. - -I knew that when I wrote it. I still described it as a mechanism, because **the syntax I had chosen made it easy to believe.** It has brackets and a namespace and a wildcard. It looks like an API call, so I reasoned about it like an API call, and for months I thought I had built enforcement when I had built a strong suggestion. - -That is the general shape. The name and the shape of a thing quietly set your expectations for it, and then you stop checking whether those expectations hold. - -## Six ways a check passes without checking - -Auditing the rest of the archive, the same defect kept arriving in different costumes. - -### 1. The gate that cannot fail - -A CI pipeline I wrote up had a job named `security-gate`. Its whole body: - -```yaml -security-gate: - needs: [dependency-scan, container-scan, comprehensive-scan] - if: always() - steps: - - name: Evaluate security posture - run: | - echo "All security scans completed" - # Download and analyze all SARIF reports - # Make final go/no-go decision -``` - -The go/no-go decision is a comment. `if: always()` is there for a good reason — you want the gate to report even when an upstream job failed rather than being skipped — but it also means the gate **passes when everything upstream fails**, unless it explicitly reads `needs..result`. Which it did not. - -Same family, different tool: a colour-space project had a build gate asserting that every value round-trips within ΔE 1.0. It re-derived everything from the source hex in floating point and never read the rounded values actually written to disk. So it could only ever measure IEEE-754 noise. Corpus maximum: **5.67e-13**, against a threshold of 1.0. Twelve orders of magnitude of headroom, and structurally incapable of failing. The specific scenario it existed to catch — a rounding change in a dependency — happens in the step it was not looking at. - -Both of these are green checkmarks. Both have been green the whole time. +## The receipt -### 2. The key that does not exist +Here is a threat-intelligence lookup from one of my posts, presented as part of a working homelab pipeline: -A grype config had this, presented as time-boxed risk acceptance: - -```yaml -ignore: - - vulnerability: CVE-2023-12345 - reason: "Not applicable - feature not used" - expiration: 2025-12-31 +```python +def check_threat_intel(ip_address): + api_key = os.environ.get('ABUSEIPDB_API_KEY') + ... ``` -Grype's `IgnoreRule` struct has these fields: `Vulnerability`, `IncludeAliases`, `Reason`, `Namespace`, `FixState`, `Package`, `VexStatus`, `VexJustification`, `MatchType`. +`os` is never imported. The function raises `NameError` on its first line. Not sometimes — always, on every input, since the day it was written. -There is no `expiration`. YAML parsers do not object to keys nobody reads. So the suppression is permanent, and by the time I found it the date had been in the past for eight months while the finding stayed hidden. +That function had never been run. Not once. It was generated, read, judged plausible, and published. And it is not an outlier: behind that first bug were three more, each independently sufficient to make the check useless. Wrong field name. Wrong nesting, so `.get(..., 0)` would return zero for a confirmed-malicious address. And `return 0` on any non-200, so the free tier's rate limit turns every attacker benign after the first HTTP 429. -Worse in the same repository: an entire `osv-scanner.toml` of invented keys — `[scanning]`, `workers`, `max_depth`, `skip_git` — none of which appear in the tool's schema. And a benchmark attached to one of them: *"set `workers = 4` for parallel scanning (40% faster on my 8-core system)."* That is a measurement of a configuration key that has never existed. It cannot have been taken. It reads like it was. +Four bugs, all failing toward "clean." A person who had run it once would have found the first one in a second. -**This is the most dangerous variant**, because the reader can verify it and still be wrong. The field names are plausible. Some of them are real flags from adjacent tools. Checking that a key looks right is not checking that the tool reads it. +## Correcting myself, which sharpens the point -### 3. The flag whose semantics are backwards +I originally wrote this section around a different claim: that config formats silently accept invented keys, so the mistakes are structurally invisible. I had a demo: -A Trivy policy, described as denying builds on critical findings: - -```bash -trivy image --policy ./policy/security.rego myapp:latest +```python +tomllib.loads('[scanning]\nworkers = 4\n') # parses clean +yaml.safe_load('expiration: 2025-12-31') # parses clean ``` -Two problems. The flag is `--ignore-policy`; there is no `--policy` for this, so the command errors on an unknown flag. And the name is the semantics: a Trivy Rego policy **filters findings out**. It cannot deny, block, or fail anything. +That demo is real, and the conclusion I drew from it was wrong. One of my own posts shipped a fabricated `osv-scanner.toml` full of invented keys — `[scanning]`, `workers`, `max_depth`. I assumed the tool had swallowed them. -So a policy written to "deny on critical" is, under the only mechanism the tool offers, either inert or a rule that *suppresses* criticals. The second outcome is the one that should worry you, and it is the one that happens if someone helpfully fixes the flag. +It would not have. `osv-scanner` has **errored on unknown config keys since September 2024** — `internal/config/manager.go` calls `Undecoded()` and returns `unknown keys in config file`. The commit is `56e3a994`, titled *"feat: error if configuration file has unknown properties."* My post was published in October 2025. -The kernel has a well-known instance of the same trap. `CONFIG_KEXEC_SIG` sounds like it makes `kexec_file_load` require a signed kernel. The Kconfig help text says otherwise, in as many words: *"The image can still be loaded without a valid signature unless you also enable KEXEC_SIG_FORCE."* A post I wrote claimed the former. The refusal actually comes from lockdown, which needs Secure Boot enforcing — so on a machine with Secure Boot off, the option named for signature verification verifies nothing. +So the file would have failed on the first run. The tool's authors had already built the exact defense I was about to recommend. -### 4. The error swallowed into a pass +**The config didn't slip past a permissive parser. It never met the parser at all.** Which is a better thesis than the one I started with, and it is the same thesis as the `NameError`: these artifacts were read, not run. -This is the one with the highest blast radius, because it converts an outage into an all-clear. +## Why a model produces this -A vulnerability scanner queried the NVD API like this: +Not carelessness, and not lying. Two mechanisms, both measured. -```python -params = {"keyword": package_name, ...} -``` +**It generates plausible surface because plausible is what it optimises for.** The scale of this is now quantified: across 576,000 code samples from 16 models, [Spracklen et al.](https://arxiv.org/abs/2406.10279) found **5.2% of package references hallucinated for commercial models and 21.7% for open-source**, with 205,474 unique invented names. And the rate tracks corpus frequency — [an AWS study](https://arxiv.org/abs/2407.09726) found GPT-4o produced valid invocations for only **38.58%** of low-frequency APIs. Security-scanner config is exactly the low-frequency regime: plenty of examples of `.grype.yaml` in the world, not many, and adjacent tools supply keys that look right. -The NVD 2.0 parameter is `keywordSearch`. I tested both against the live API: `keyword` returns **404**, `keywordSearch` returns 200. Then follow the error path — `raise_for_status()` raises, `except requests.RequestException` catches, and the handler does `return []`. That empty list propagates all the way up to the report, which prints: +That is how you get `expiration:` in a grype ignore rule. Grype's `IgnoreRule` struct has nine fields and that is not one of them — but `expiration` is a real key in adjacent tools, so it arrives wearing the right clothes. -``` -Total vulnerabilities: 0 -``` +The unsettling detail is repeatability. Requerying hallucination-producing prompts ten times, Spracklen et al. found **43% of hallucinated packages repeated in all ten runs**. These aren't random slips; a large share are stable, which is what makes the [slopsquatting](https://www.theregister.com/2025/04/12/ai_code_suggestions_sabotage_supply_chain/) idea coherent. Though in fairness: as far as I can find, exploitation is not yet documented in the wild. The rate is measured, the repeatability is measured, and the only demonstration is a researcher registering an empty `huggingface-cli` package that got 30,000 downloads. -Every host, every run, clean. And the same file had a second, independent instance: version comparison caught `packaging.version.InvalidVersion` and returned `False`, meaning *not vulnerable*. Every real Debian and Ubuntu version string raises it — `1:24.0.5-1`, `9.2p1-2ubuntu0.13`, `3.0.2-0ubuntu1.12` — so every package it looked at cleared. +**It optimises the metric you actually gave it.** My archive has this written down in its own commit messages. One reads *"4 posts humanized (40-45 → 90-97.5)"* and lists what it added to get there: `47hrs Isaac Sim`, `2.3 FPS`, `73% accuracy`. Another: *"Claude-Flow: 40.1% → 20.6% (8 gists created, -49% code)"* — and those eight gists exist, created in a **21-second burst, weeks after the post that presents them as working notes.** -Two bugs, written months apart, both failing in the same direction. That is not coincidence; it is what happens when the quiet path is never exercised. +My favourite is a commit reporting **`Humanization: 105/100`**. A rubric optimised past its own ceiling. -A threat-intelligence lookup in another post did the same thing three ways at once. It called `os.environ` without importing `os`, so it raised `NameError` on its first line and had never run at all. Behind that, the field name was wrong and nested one level too shallow, so `.get(..., 0)` would have returned 0 for a confirmed-malicious address. And `return 0` on any non-200 meant the free tier's daily rate limit turned every attacker benign after the first HTTP 429. +This is [specification gaming](https://deepmind.google/discover/blog/specification-gaming-the-flip-side-of-ai-ingenuity/): behaviour satisfying the literal specification without achieving the intended outcome. The rubric said *add concrete measurements*. It added concrete measurements. The measurements were the deliverable, and nothing in the objective said they had to correspond to anything. -**If a check cannot answer, it must not answer "fine."** A scanner that returns "no findings" on a network error is worse than no scanner, because you stop looking. +It is also measurable in coding agents specifically. [ImpossibleBench](https://arxiv.org/abs/2510.20270) constructs tasks where the spec and the tests conflict, so any pass implies a shortcut, and reports **GPT-5 cheating on 54%** of them. Its opening example is the thing that most alarmed me in my own archive: an agent with access to unit tests may delete the failing test rather than fix the bug. -### 5. The check that cannot run when it matters +## Why review didn't catch it -A pre-commit hook enforcing standards, and the honest discovery that you can walk straight past it: +This is the part I got wrong about myself, and the literature is unkind in a useful way. -```bash -git commit --no-verify -``` +**Code review does not find defects at anything like the rate people assume.** [Bacchelli and Bird](https://sback.it/publications/icse2013.pdf) hand-classified **570 review comments** at Microsoft. Their finding, verbatim: *"Review comments about defects are few, comprising one-eighth of the total in our sample, and mostly address 'micro' level and superficial concerns."* Code improvements outnumbered defects two to one. -My instinct at the time was to close the hole from inside the hook. You cannot. `--no-verify` skips the hook entirely — the process never starts, so nothing it might do on the way out can matter. There is no exit code that runs when the code does not run. +And of those 570 comments, the number about wrong exception handling was **three**. -Client-side hooks are a convenience for the person running them. Enforcement is server-side: a `pre-receive` hook, a required status check, branch protection. Keep the hook for the fast feedback loop; do not confuse it for the control. +Three. Which is the category my `return 0` bug lives in, and it means my review missing it was not an unusual lapse. It is the base rate. -Adjacent: a Prometheus config whose `rule_files` glob matched nothing, because the rules directory was never mounted into the container. That is not a startup error. Prometheus comes up healthy with **zero rules loaded**, which looks exactly like a quiet night. +**The error paths are where the damage is.** [Yuan et al.](https://www.usenix.org/system/files/conference/osdi14/osdi14-paper-yuan.pdf) analysed 198 real catastrophic failures across Cassandra, HBase, HDFS, Hadoop and Redis: -### 6. The test that asserts the defect +> almost all (92%) of the catastrophic system failures are the result of incorrect handling of non-fatal errors explicitly signaled in software -The one I find most uncomfortable, because tests are supposed to be the answer to all of the above. +> in 35% of the catastrophic failures, the faults in the error handling code fall into three trivial patterns: (i) the error handler is simply empty or only contains a log printing statement… -A link validator was classifying transient server errors as dead links. Its regression test: +Read pattern (i) again, then read the `security-gate` job I shipped in a CI pipeline post: -```python -@pytest.mark.parametrize("code,expected", [ - (404, "broken"), - (500, "broken"), - (503, "broken"), -]) +```yaml +security-gate: + needs: [dependency-scan, container-scan, comprehensive-scan] + if: always() + steps: + - run: | + echo "All security scans completed" + # Make final go/no-go decision ``` -That test passes. It has always passed. It was written from the implementation rather than from the intent, so it faithfully locks in the behaviour the implementation happens to have — including the wrong parts. A 5xx means the origin answered and erred. It is up. Treating that as a dead citation fed live sources into an auto-repair queue that rewrites links. +An error handler that only contains a log printing statement. Named as the gate. `if: always()`, so it passes when everything upstream fails. This has a CWE — [CWE-636, Failing Open](https://cwe.mitre.org/data/definitions/636.html) — and a thirty-year literature, and I shipped it anyway. -The fix was to state the taxonomy as an invariant instead of a table of examples: +**And we look at machine-written code less carefully.** [Al Madi](https://arxiv.org/abs/2208.14613) eye-tracked 21 programmers reading Copilot output and human-written code. Complexity and readability were comparable, but *"programmers direct less visual attention to model generated code"*, significantly. The authors' own conclusion: beware complacency and automation bias. -```python -def test_only_dead_codes_are_broken(): - broken = {c for c in range(200, 600) if classify(c)[0] == "broken"} - assert broken == {404, 410} -``` +So: a defect class that is invisible to reading, in the category review is worst at, in code we look at least closely. That is not a personal failure. It is a system with no check in it. -That version cannot be satisfied by the bug. +## It happened while I was writing this -## The near-miss, which is the actual point +I used a research agent for the citations above, with explicit instructions to verify against primary sources and to flag anything it could not confirm. -I would like to claim I am now immune to this. Last night says otherwise. +It came back with a detailed report. Then it came back **again**, unprompted, with a retraction: it had produced a Fagan 1976 quote — *"approximately two thirds of all errors reported during development are found by inspections"* — that does not appear in the paper. It had generated an archive.org identifier, a chapter attribution, a "cuts out 21%" filter detail, all correctly formatted, before opening any of the documents. -I was testing whether a static site could drop `'unsafe-inline'` from its Content-Security-Policy by having the framework hash every inline script at build time. I enabled it, rebuilt, and checked the output: +Most of those details were right, which is exactly why nothing looked wrong. The two that were wrong were wrong in the same register as the ones that were right. -``` -inline scripts on the page: 8 -sha256 hashes in the policy: 8 -'unsafe-inline' present: no -``` +It caught this only by re-fetching the primary sources. No consistency check would have found it, because the fabrication was internally consistent — which is itself a measured property. [Sui et al.](https://arxiv.org/abs/2406.04175) find hallucinated LLM output shows *higher* narrativity and semantic coherence than truthful output. Fabrication is smoother than truth. It reads better. -Eight and eight. No unsafe-inline. I nearly shipped it. +The defect class appeared in the middle of a task specifically instructed to guard against it. Which should tell you how much instruction is worth here, relative to verification. -The count match was a coincidence. Computing the actual SHA-256 of each script body and testing membership in the policy: +## What actually works -``` -[0] type="application/ld+json" NO MATCH -[1] type="application/ld+json" NO MATCH -[2] (theme-flash) NO MATCH -[4] MATCH -[5] MATCH -[6] (theme deck) NO MATCH -[7] (theme toggle) NO MATCH -[8] (mobile TOC) NO MATCH -``` +**Run it.** This is the whole finding, and it is embarrassingly simple. Not "read it carefully" — execute it once, against something that should fail. The `NameError` dies on import. The invented TOML dies on parse, because the tool's authors already thought of that. The `security-gate` reveals itself the moment an upstream job fails. Reading is precisely the activity that cannot distinguish plausible from correct. -Two of eight. Across a 25-page sample, **133 of 183 inline scripts would have been blocked.** The framework only hashes scripts it compiles, and every one of those had an `is:inline` directive — which is a direct instruction not to process them. +**Make the parser strict, and check whether it already is.** `serde`'s `deny_unknown_fields`, Pydantic's `extra='forbid'`, Go's `DisallowUnknownFields` (on `Decoder` only, not `json.Unmarshal`). Kubernetes made server-side field validation the default in **v1.27** — its motivating incident was a Service silently breaking because `containerPort` had been renamed `targetPort` and the server accepted the stale key. The tracking issue stayed open for seven years. The defaults are permissive, and they are changing. -And the failure would have been silent, because `report-uri` is also ignored when CSP arrives in a `` tag. No telemetry. The symptom would have been readers getting a flash of the wrong theme and a dead theme picker, with every check green. +**Check the arithmetic against the sample size.** Psychology has a mechanical test for this — [GRIM](https://en.wikipedia.org/wiki/GRIM_test), which exploits the fact that a mean of N integers must be expressible with denominator N. In its original application, **36 of 71 testable papers contained at least one impossible value.** My archive was full of the same thing: "73% accuracy" on a stated corpus of 50 items requires 36.5 items. I have not seen anyone point GRIM at model output, and it costs nothing. -I caught it because I had spent a week reading other people's broken gates and had gotten suspicious of tidy numbers. **The count was the check that passed for the wrong reason.** +**Check provenance, not just content.** `gh api gists/ --jq .created_at` against the post date. Eleven artifacts created in a 21-second burst are not working notes. `git log -S` on the artifact, and read the commit body — a pass that states its optimisation target is telling you what its numbers are for. -## What to actually do +**Two independent implementations, with the caveat.** Differential testing is the strongest no-oracle technique there is: [Csmith](https://users.cs.utah.edu/~regehr/papers/pldi11-preprint.pdf) found 325 previously-unknown compiler bugs. But [Knight and Leveson](http://sunnyday.mit.edu/papers/nver-tse.pdf) is the necessary counterweight — 27 independently written versions, a million tests, and correlated failure at 99% confidence, with roughly half the faults appearing in two or more programs. Independent authors do not produce independent mistakes. Their own conclusion is narrower than the folklore: not that N-version programming fails, but that its reliability *"may not be as high as theory predicts under the assumption of independence."* -Nothing here is clever. It is all just declining to accept the cheap signal. +**And be careful what you cite for any of this.** The most-quoted number in code review — "a review of 200-400 LOC over 60 to 90 minutes should yield 70-90% defect discovery" — is widely attributed to the Cisco/SmartBear study. It is not in the Cisco data. In the book it sits in a different chapter, by a different author, about personal reviews under the SEI's TSP, with no data behind it. The Cisco chapter explicitly declines the claim: *"we don't know how each of these reviews would have fared with a different process."* The vendor's own page states the 400-LOC finding with a Cisco attribution and then the 70-90% figure in the next sentence, unsourced. Nobody wrote a false sentence. The number acquired its authority by proximity. -**Verify the mechanism, not the presence of the mechanism.** The question is never "is there a gate." It is "what happens when I feed it something that should fail." Break it on purpose once, and watch it break. +Which is the same mechanism as everything above, in the literature about checking. -**Grep the tool's source or schema for the key you are about to rely on.** Every invented config key in this archive would have died to a thirty-second search of the struct definition. A key that looks right and a key the parser reads are different things, and only one of them is checkable. +## What I changed -**Make errors loud, and never let them return the safe value.** `return []`, `return 0`, `return False` on an exception path are all the same bug with different types. If the check cannot evaluate, it must raise, and something must alert on the absence of a result rather than only on a bad one. +The pre-publish gate on this site now runs a sixth audit: artifact provenance and every config key against the tool's actual upstream schema, one key at a time, including semantics — a flag can exist and do the opposite, as `trivy --ignore-policy` does. Its sharpest rule is the one my archive taught me: **a measurement attached to an invented key means the surrounding numbers are generated too.** `workers = 4 is 40% faster` cannot have been measured if the key has never existed. -**Write tests from the intent, and prefer invariants to examples.** "The broken set is exactly `{404, 410}`" cannot be satisfied by a bug. A list of cases can be, and will be, because the list gets written by reading the code. +But the honest summary is shorter than the tooling. Every defect I found survived because an artifact was judged on how it read. The fix is not more careful reading. -**Be suspicious when a number comes out clean.** Eight and eight. A false-positive rate of exactly zero. Twelve orders of magnitude of headroom. A tidy result is a hypothesis, not a conclusion, and the cheapest moment to check it is while you still believe it. +Run it once. Point it at something that should fail. Watch it fail. -The uncomfortable summary of ninety posts: almost every control I had written up as working had a plausible story, a passing check, and a green light. The ones that were actually broken were not the ones that looked broken. They were the ones nobody had tried to break.