Harden synthesis pipeline: rate limits, Qwen agent, neighbor cascade, remote access - #18
Merged
Conversation
…o synthesize_images
…mp, row-by-row retry
atharvas
added a commit
that referenced
this pull request
Aug 23, 2026
Making timeout FATAL at the existing 720s default would convert a 34% silent-pass into a 34% hard-fail on stage 6. 720 is the outlier: every other timeout in the tree is 3600, and dataset/CLAUDE.md documents 3600 in four places. The two halves land together. Also: note the duration data is censored at 720 so the true tail is unknown (adds --calibrate to the audit script); defer #12/#18 since expected_n has no source in this tree; scope reward_formula_id to the parser each pipeline actually bakes; define missing-manifest behavior; flag rl-prep citations as external and unversioned. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LH8qaoE2CQenNmzJPMKh94
atharvas
added a commit
that referenced
this pull request
Aug 23, 2026
…ed severity The build-time invariant table advertised gates that cannot currently fire. Add footnote markers for benchmark_dest_missing (no BENCHMARK_DEST producer), dilution_ratio / reward_formula_unknown / image_identity_missing (no expected_n / reward-hash / image-identity producer yet, per the existing "#12 and #18 are deferred" section), and flip pytest_collect_ok's severity from FATAL to warn to match the code change, with a footnote recording the return-to-FATAL condition. Also documents the new manifest_empty check, left unnumbered since the table's #N values are cross-referenced extensively elsewhere in the doc's prose.
atharvas
added a commit
that referenced
this pull request
Aug 23, 2026
…raint Measured the blast radius rather than assuming it: 5 Grafana panels read candidate_containers and all 5 are COUNT(*) over rows. manifest_warnings and build_manifest appear 0 times in the dashboard; the 3 panels reading resource_metrics/test_duration_s all source from error_logs. So changing a column is invisible and deleting a row moves 5 figures at once -- which puts Plan 4's follow-on delete permanently out of scope. Two spec assumptions found false against the real data: 1. The overrides record has NO expected_n field. Its 10 real keys are benchmark_dest, benchmark_storage_key, extra_*_commands, issue_number, oracle_h, owner, repo, pip_pins, restore_regex. So "create the table" and "make #18 live" are different tasks; expected_n ships hand-declared and nullable per operator ruling. 2. Plan 4's ranking scheme is degenerate: zero of the 636 suspect rows have a harbor_runs row (the join is sound -- 7 containers do, all of them fast rows under 720s). Ranking by repo instead. Also found: parser.py never receives base_commit, so invariant #15 would ship inert. test.sh already has it as a Jinja var and must pass it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
atharvas
added a commit
that referenced
this pull request
Aug 23, 2026
Both were open producer gaps recorded in the 2026-08-13 ledger: the invariant, the column and the tests all existed, but nothing supplied the input, so each check skipped forever while reading as live. benchmark_dest_missing (FATAL) had been inert since the day it shipped -- nothing in the tree ever set $BENCHMARK_DEST. Now: overrides lookup -> _build_pr_image -> ImageManager.build_pr_image -> BENCHMARK_DEST build arg -> docker_build_run.sh's existing conditional breadcrumb. The ARG is declared in the `run` stage specifically, since ARGs do not cross FROM boundaries and docker_build_run.sh would not otherwise see it. Verified by running the emission block in real containers under all three conditions, then feeding the resulting breadcrumbs through the real evaluator: declared+present -> passes, declared+missing -> FATAL fires (ok=False), undeclared -> skips. That is now pinned as a test; before this change all three cases skipped identically. dilution_ratio (#18) reads FORMULACODE_EXPECTED_N, which nothing injected. Now supplied per task via [verifier.env] from the override row. None injects NOTHING rather than an empty value -- emitting a key that is always empty would make the wiring look live when it is not. The lookup is cached per process. Consumers sit in per-item loops (stage 6 enqueues neighbours mid-flight, so there is no point where the full task set is known), and an uncached read would be one round-trip per task. A failed read caches {} too, since an absent table stays absent. Behaviour change worth knowing: the 5 override tasks now hard-fail stage 6 if their declared benchmark file does not survive git clean. That is the gate doing its job -- it is the joblib/asv_benchmarks.txt wipe it was written for -- but it can reject containers that passed yesterday. Every task without an override row is unaffected. Also fixed: adding the lookup made the "mocked" synthesize_images tests open a REAL Supabase connection, because fetch_all resolves its own client. They passed only because a local DB happened to be running. Now patched. All three producers proven by deletion: removing the ARG, the adapter injection, or the build-arg pass each fails its test. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Test plan