posts: rewrite "Nobody Ran It" around verified research

[williamzujkowski]

Reframes the post from a confession into what the evidence actually shows. The
old version read as "I shipped broken things"; the artifacts were
agent-generated, passed every gate that existed, and looked right on review. The
useful story is the defect class and why review is structurally blind to it.

The thesis changed, because research refuted the old one

The previous version argued that config formats silently accept invented keys,
so the mistakes are invisible. I had a working demo of it.

The demo is real and the conclusion was wrong. osv-scanner — the exact tool in
my example — has errored on unknown config keys since 2024-09-23 (commit
56e3a994, "feat: error if configuration file has unknown properties";
manager.go calls Undecoded()). My post shipped that fabricated config in
October 2025. It would have failed on the first run.

Which gives the sharper thesis, and the one my own evidence supports: the
artifacts were never executed. The decisive receipt is a threat-intel function
that calls os.environ with no import os — NameError on line one, on every
input, since the day it was written. That function had never been run. Not once.

Research, verified to primary sources

• Package/API hallucination is measured. 576,000 samples, 16 models: 5.2%
  hallucinated package references for commercial models, 21.7% open-source,
  205,474 unique invented names. And the repeatability figure, read out of the
  PDF body rather than the abstract: "43% of hallucinated packages were
  repeated in all 10 queries, while 39% did not repeat at all." Rate tracks
  corpus frequency — GPT-4o produces valid invocations for only 38.58% of
  low-frequency APIs, which is exactly where security-scanner config lives.
• Specification gaming, with my own commits as the exhibit. "4 posts
  humanized (40-45 → 90-97.5)", "8 gists created, -49% code", and one reading
  Humanization: 105/100 — a rubric optimised past its own ceiling.
  ImpossibleBench measures GPT-5 cheating on 54% of spec-vs-test conflicts.
• Why review misses it. Bacchelli & Bird hand-classified 570 review comments
  at Microsoft: defects are one-eighth of the total and "mostly address 'micro'
  level and superficial concerns." Three of the 570 concerned wrong
  exception handling — the exact category of my return 0 bug. Yuan et al.
  (198 real catastrophic failures): 92% stem from incorrect handling of
  non-fatal errors, and 35% have handlers that are "simply empty or only
  contain a log printing statement" — which is verbatim my security-gate job.
  Al Madi's eye-tracking: programmers give model-generated code significantly
  less visual attention.

The demonstration that happened mid-write

The research agent retracted itself, unprompted, three times. It had produced a
Fagan 1976 quote that does not exist in the paper, with correct formatting, a
real DOI, an archive.org identifier and a plausible chapter attribution —
generated before it opened any document. Most surrounding details were right,
which is why nothing looked wrong. Caught only by re-fetching primary sources.

That is the defect class occurring inside a task explicitly instructed to guard
against it, and it is now the post's centrepiece. It pairs with the measured
finding that hallucinated LLM output shows higher coherence than truthful
output — fabrication reads better than truth.

I did not use any retracted material. Verified by grep: the only mentions of
Fagan and "two thirds" are inside the passage describing the fabrication. I also
dropped property-based testing entirely, since the agent flagged that its own
"no controlled comparison exists" claim was an exhausted-search null result
doing real argumentative work.

One more citation-laundering example, verified

The most-quoted number in code review — "200-400 LOC over 60-90 minutes yields
70-90% defect discovery" — is not in the Cisco/SmartBear data. It sits in a
different chapter, by a different author, about personal reviews under the SEI
TSP, with no data behind it. The Cisco chapter explicitly declines the claim.
I fetched SmartBear's page and confirmed the shape: the 400-LOC finding carries
a Cisco attribution, the 70-90% appears in the next sentence unsourced, and the
study scale (2,500 reviews, 3.2M LOC, 50 developers) appears nowhere on the
page. Nobody wrote a false sentence; the number acquired authority by proximity.

Prevention, already shipped

blog-artifact-check (merged in #486) is now the sixth pre-publish audit:
artifact provenance, and every config key against upstream schema one at a time
including semantics. Sharpest rule: a measurement attached to an invented key
means the surrounding numbers are generated too.

Title changed to "Nobody Ran It"; slug kept, so the two forward-links still
resolve. Build passes.

[github-actions]

❌ Link Validation Results

• Total Links: 973
• Broken Links: 36
• Status: failed

View Full Report

Link Validation Report

Generated: 2026-08-19 06:40:46

Executive Summary

• Total Links Checked: 1514
• Valid Links: 1161 (76.7%)
• Broken Links: 44 (2.9%)
• Redirects: 0
• Requires Manual Review: 38
• Auto-Fixable: 1

Issue Breakdown

Issue Type	Count	Percentage
connection_error	197	13.0%
None	1161	76.7%
http_403	100	6.6%
not_found	18	1.2%
dns_error	26	1.7%
http_400	7	0.5%
timeout	2	0.1%
http_401	1	0.1%
http_429	1	0.1%
http_402	1	0.1%

Top Affected Domains

Domain	Broken Links	Total Links
vault.example.com	5	5
example.local	4	4
raw.githubusercontent.com	4	5
wazuh-manager:55000	2	2
vaultwarden:80	2	2
www.intelrealsense.com	2	2
docs.vllm.ai	2	2
culorijs.org	1	3
azure.microsoft.com	1	4
climatemodeling.science.energy.gov	1	1

Repair Statistics

• High Confidence Fixes: 1
• Medium Confidence Fixes: 5
• Low Confidence Fixes: 0
• No Fix Available: 38

Files with Most Issues

File	Broken Links	Total Links
2025-09-01-self-hosted-bitwarden-migration-guide.md	8	19
2025-11-05-siem-homelab-wazuh-graylog-comparison.md	4	22
2026-07-23-agent-controls-as-oscal.md	4	11
2025-03-10-raspberry-pi-security-projects.md	3	8
2024-09-25-gvisor-container-sandboxing-security.md	3	45
2025-12-10-homelab-security-dashboard-grafana-prometheus.md	2	15
2025-10-13-embodied-ai-robots-physical-world.md	2	32
2025-10-29-privacy-first-ai-lab-local-llms.md	2	28
2025-08-09-ai-cognitive-infrastructure.md	2	24
2026-07-19-oklch-terminal-themes.md	1	11