fix: Ali's four engineering findings, a uuid7 production defect, the Arabic scope audit, and the joint deposit - #1079
Merged
Hashevolution merged 32 commits intoAug 26, 2026
Conversation
…stale UI assertions Two clusters of the standing test backlog, both "the code moved, the test did not" rather than product defects. 101 -> 82 expected. LRB cluster (12 -> 1). `eval/external/_fixtures/` is gitignored (.gitignore:82), so every LRB scenario test died on FileNotFoundError and had in fact never run in CI since the suite landed (PR #1027, 2026-06-23). The builders under scripts/research/build_lrb_scenario_s*.py are deterministic and stdlib-only — no network, no model, hardcoded vocabulary — so the fixtures are rebuilt on demand by a new tests/_lrb_fixtures.py helper rather than skipped or added to the workflow's ignore list. Building S1 + S2 costs well under a second and keeps the coverage real. UI cluster (9 -> 1). v0.6.1 attached cache-busting query strings to the static assets (mobile.css?v=v22-20260625-upload), so assertions matching the literal href="/static/mobile.css" stopped matching pages that do link the stylesheet. Both call sites now match the path with an optional query. Two further cases in the same files were not stale: - test_no_full_page_left_unaccounted was correctly red — intro.html landed in PR #1062 and was never registered in the mobile.css coverage contract. It links mobile.css, so it joins the contract and gains its own test rather than being excused. - test_neighbor_panel_reposition_present pinned `top: 56px`, but the panel was redesigned into a bottom sheet (top: auto; bottom: 12px; max-height: 50vh — see the "sheet stays at ~50vh" note in mobile.css). The assertion now pins the invariant the rule exists for (anchored to the viewport bottom, height-capped) instead of one superseded offset. Two failures are deliberately left red because neither is a stale assertion: - test_no_excessive_important — mobile.css uses !important 29 times against a budget of 25. That is real CSS debt; raising the budget to match would be quarantining the test. - test_token_mode_s2_reproduces_phase_b_baseline — JAMES token-mode R@1 on LRB-S2 comes out 0.6875 against the pinned 0.7125 (vanilla 0.225 and naive 0.5375 both reproduce exactly, and the whole S1 triple reproduces). The recorded fixture_sha in reports/external/lrb/v021-s2-token-baseline-token-20260611T133018Z.james.result.json is 9f40d2e0..., while the committed builder now emits 1ddfb6c0..., so the published run used a different S2 scenario than the builder produces today. 0.7125 is the number in the LRB preprint abstract, so neither side may be edited to match the other without a decision. Verification: from a clean tree with eval/external/_fixtures deleted, the three LRB modules go 12 failed -> 1 failed, 37 passed; the two UI modules go 9 failed -> 1 failed, 29 passed. `ruff check .` clean. Run with --noconftest, since conftest instantiates the embedding model and huggingface.co is blocked in this environment. Quality delta: exempt (label: fix). Test-side only; no core/ change. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Jttr1R2aJ3YjJkS1odiLC3
…it text Ali circulated the final v1.0.0 record for approval, having measured the separating cell the first reply asked for: an instruction-removed arm (80 control + 80 with the prompt's only length instruction deleted, same frozen payloads, 160/160 clean) moving per-cap medians 124-128.5 -> 133.5-144.5, pooled 127 -> 138.5. Everything requested in the first reply landed verbatim - the axis-2 wording with its eval_count / thinking-trace caveat, PRs #461/#463 as the seven-tier closure with #440 as the earlier split and #448 as Converse's leg, the three v0.3.x DOIs cited individually, the reason not to use a concept DOI, and the managed-Gemini -> gpt-4o-mini forward-pointer correction. Robin's ORCID is filled in, so she has already replied. The draft gives an OK conditional on two sentence changes, both places where the body claims more than the record's own "does not claim" section allows: - The 127 -> 138.5 shift is dismissed as "smaller than within-cell spread". Within-cell spread is per-sample dispersion; a shift in centres has to clear the standard error of the centre. At the reported 48-63 spread the per-sample SD is around 15, so an 80-call arm's SE is near 1.7 tokens - the observed +11.5 is several times that and points the same way in all four cap cells. Proposed wording keeps the finding and bounds it: the instruction is worth about 9%, the model's own answer floor owns the rest. - "a hosted model with no reasoning trace floors near its visible answer" reads as established, but the reasoning axis was never varied on that stack and every trace-side number comes from e4b. Proposed wording matches the body to the claim-scope section. Two checks rather than changes: Robin's model-scale efficiency axis (her ~9x finding in #448) is absent from this version and should be her call before submission, and the control-arm medians differ from the figures in Ali's first message and repo README, so the 2026-08-19 sweep should be identified as fresh calls if it is. Two optional notes: 20363998 is deposited as `software`, not a dataset, and the three-author ten-word line is missing. A stitch-clause draft is included for all three forward-pointers. Not sent; operator confirm required. Vadym is deliberately not raised here - his phrasing does not appear in the final text, so this record carries no attribution defect; the Phase 2 notice is separate. Docs only - no core/ change. Quality delta: exempt (label: docs). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Jttr1R2aJ3YjJkS1odiLC3
…e flag, harden the rest Pre-send check of the conditional-OK draft against the underlying records changed three things. Dropped a wrong objection. The draft was about to say Converse's "6-9/20 unique" synthesis figure is unverifiable because Issue #448 does not carry it. It is in the triavalabs/gemma4-26b-mode-split README (6 unique at cap 400, 9 at 4096), and her DOI 10.5281/zenodo.20570701 archives that repo, so the figure is traceable through a record the deposit already cites. Removed, and noted in passing so she is not asked to re-justify it. Corrected a quotation. The draft quoted her third-axis finding in its LinkedIn phrasing while citing Issue #448 as the source; the issue states "Parameter count appears to buy reasoning efficiency, not just reasoning capacity." Now quoted from the cited artifact. Rebuilt the statistical argument on firmer ground. The first version leaned on a standard-error estimate. Ali's own reported per-cap medians carry a stronger, assumption-free fact: the four control medians (124-128.5) and the four instruction-removed medians (133.5-144.5) are disjoint, so every cell moved the same way in a same-day paired arm on frozen payloads. The SE calculation is kept as corroboration and fixed - the earlier 1.7 was the error on a mean, while the comparison is between medians (~2 tokens at 80 calls), with the SD derivation from his 48-63 spread now shown so it can be checked. The control-arm question was also reframed. He does say "80 control", so the arm is fresh calls; the useful observation is that the two controls disagree - 127.5-130.5 / 118.5-122.5 in the first sweep against 124-128.5 across all four caps in the same-day arm, a roughly six-token move at matched caps with the cap trend flattened. That does not touch the paired result but bounds how precisely the 9% can be quoted. Our own axis numbers were re-checked against source and all match: 62 -> 1681, 27x, 3-5% per tier, 5 of 7 tiers 82-98% trace, substitution 2%, reflect 61% with 580 visible tokens, PRs #461/#463/#440, Issue #448, and the three v0.3.x DOIs. Not sent; operator confirm required. Docs only - no core/ change. Quality delta: exempt (label: docs). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Jttr1R2aJ3YjJkS1odiLC3
A pre-send pass over the draft's own claims found three places where it
was not exact.
The opening line said both flagged sentences overstep Ali's "does not
claim" section. That holds for the second (the body states the
no-reasoning-trace floor as general, while the claims section confines
the visible-answer-floor explanation to one measured pair and puts the
reasoning axis on the middleware leg) but not the first — the claims
section says nothing about the instruction's effect size. The first item
is a wrong test plus wording that overshoots into "no effect", and the
letter now separates the two failure modes and grants outright that his
conclusion stands: an instruction worth 11.5 tokens is not what holds a
call an order of magnitude below a 400-4096 cap.
The draft asserted per-cell shifts of "+9 to +16". Ali reports the
medians as ranges (124-128.5 -> 133.5-144.5) without saying which cap
pairs with which, so per-cell shifts are unknown and could be as small
as +5 under the worst pairing, which would not clear the ~4-token
per-cell median error. Removed; the argument now rests on the
disjointness of the two median sets, which holds under any pairing, plus
the pooled +11.5 against a ~2-token error at 80 calls.
The drift note quantified a "roughly six-token move" between the two
control arms. That number depends on the same unknown pairing (the gap
runs 1.5 to 10 tokens across plausible alignments), so the letter now
states only what is observable — the first sweep's high-cap band sits
below the same-day control band with no overlap, and the downward cap
trend is gone — and asks him to name the cause.
Two sharpenings while there: the no-trace replacement is now drop-in
prose for the bullet rather than claim-scope phrasing, and the
model-scale point cites his own previous draft ("synthesis grows
markedly more token-efficient with model scale") rather than our
internal drafts, since that is the version he can check.
Not sent; operator confirm required.
Docs only - no core/ change. Quality delta: exempt (label: docs).
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Jttr1R2aJ3YjJkS1odiLC3
…tions Last pass before sending, checking every remaining figure against the artifact it came from rather than against my own summary of it. The drift item was wrong twice. Ali's repo README gives the first sweep's per-cap medians as 127.5 / 130.5 / 118.5 / 122.5 and calls them non-monotonic in as many words; the draft had described a "downward cap trend" that does not exist in his data. And the two controls do not in fact disagree - the same-day band (124-128.5) sits inside the first sweep's span (118.5-130.5), with only the two lowest cells failing to reproduce. "The two controls do not line up" overstated it. Rewritten to what is actually observable, and re-aimed at the point that survives: the control centre carries a few tokens of run-to-run play, which is what a reader weighs the 9% against. The ten-word line was mis-attributed. The draft called it "your phrasing"; launch-tracker.md:109 records it as the candidate headline from our own v3prime-e-substitution-synthesis-result.md that Ali independently re-derived and endorsed on the PR #440 thread, his own contribution being the "cost asymmetry in ten words" framing. In a letter whose subject is citation accuracy this could not stand. Restated as the three-way convergence it was. Our README has no "Context" section - the draft named a section that does not exist. The forward-pointer's real home is Papers & Reproducibility. Also scoped one sentence in item 1: "you report the medians as ranges" now says "this arm", since the first sweep does publish a per-cap table and the ranges-only reporting is specific to the new instruction-removed arm. Not sent; operator confirm required. Docs only - no core/ change. Quality delta: exempt (label: docs). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Jttr1R2aJ3YjJkS1odiLC3
…e letter Final pass caught a wrong number in my own suggested wording, plus three smaller defects. The second replacement sentence offered "bounds the prompt's length instruction to roughly 9% of that distance". The distance in that sentence is the gap between the natural stop and the cap; 11.5 tokens is 9% of the reply's own length (127), not of that gap, which would be 4% at cap 400 and 0.3% at 4096. Both replacements now say what the number is: removing the instruction lengthens the reply by about 9%. The item-2 drop-in opened with "…while synthesis never approaches any cap", which is from earlier in Ali's bullet, so it would not have slotted where the clause it replaces actually sits. Rewritten to close his own sentence. The opening said "two sentences" while the letter proposes three replacements across two changes; it now says two changes. The rest is compression: item 1 goes from three paragraphs to two with the arithmetic folded into the first, and the control-arm check is cut to about half. No argument dropped - the disjointness of the two median sets is still the load-bearing evidence, with the standard-error estimate as corroboration. Every premise in the letter is now traceable to an artifact: Ali's final text and his repo README for the production figures, Issue #448 for Converse's wording, her repo README for the 6-9/20 fan-out, our launch-tracker for the ten-word line's provenance, the v0.3.1 tag's .zenodo.json for upload_type, and README.md for the section the stitch clause lands in. Not sent; operator confirm required. Docs only - no core/ change. Quality delta: exempt (label: docs). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Jttr1R2aJ3YjJkS1odiLC3
Adversarial re-read of the Ali reply found the letter committing, in miniature, the error it raises with him. Wrong comparator. The draft measured the +11.5 shift against the error on a single 80-call median (~2 tokens) and called it "some five times that". The shift is a difference between two independently estimated medians, so the comparator is the error on the difference - larger by √2, about 3 tokens - which puts it near four times, not five. The SD estimate is also tightened: the per-cap spreads are 63/48/48/51, mean 52.5, which over the n=20 range constant of 3.735 gives SD ≈ 14 rather than 15. Both arms ran the same frozen payloads, so pairing can only help, but the letter now states the conservative independent-arm figure. Overstated the disjointness. The draft called the non-overlap of the two median sets "the stronger evidence" and asserted that "a cause that does nothing does not do that". On its own it is a sign test over four pairs, which an inert cause reproduces about one time in sixteen - well short of conventional significance, and weaker than the magnitude argument it was placed above. Reordered: magnitude carries the weight, non-overlap shows the effect is consistent across cells rather than driven by one, and the one-in-sixteen figure is stated rather than glossed. Neither change touches the conclusion - the instruction is a real but small contributor at about 9%, and the model's own answer floor is what holds the call an order of magnitude below the cap. Not sent; operator confirm required. Docs only - no core/ change. Quality delta: exempt (label: docs). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Jttr1R2aJ3YjJkS1odiLC3
Three cosmetic and consistency fixes after an end-to-end read. One line overran the wrap the rest of the letter keeps. The control-arm paragraph still pointed at "the disjointness above" as what protects the paired result, left over from the ordering before magnitude was made the load-bearing argument; it now names the actual protection - control and treatment ran the same day on the same payloads. And Converse's ~9% token-efficiency finding and her interpretation of it were run together in one clause; they are now distinguished. All figures re-derived independently and agree with the letter: pooled shift 11.5, ratio 1.0906, +9.1%, set gap 5.0, SD 14.06 from the 63/48/48/51 spreads over the n=20 range constant, 80-call median error 1.97, difference error 2.79, ratio 4.13x, sign-test 0.0625. The discarded "9% of that distance" phrasing would have been 4.2% at cap 400 and 0.29% at 4096. Not sent; operator confirm required. Docs only - no core/ change. Quality delta: exempt (label: docs). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Jttr1R2aJ3YjJkS1odiLC3
…air to Converse One premise had been carried through every previous pass without being checked at source: whether Ali's "48-63" is a within-cell range or a standard deviation. The whole of item 1 turns on it. Read as a range, SD is about 14, the error on the difference of two 80-call medians is about 2.8, and the observed +11.5 is 4.1x it - the effect is real. Read as an SD, SD is about 55, the error is about 10.9, and +11.5 is 1.06x - no effect at all, which would have made the letter tell him he is wrong where he is right. His repo's table settles it: the row is labelled "within-cell range" with values 63/48/48/51. Range confirmed from the source's own column header rather than inferred, so the arithmetic stands. Noted alongside that token lengths are right-skewed, which raises the range-to-SD ratio above the normal 3.735 and therefore makes our SD estimate conservative. Separately, the Converse item was unfair by omission. Ali's first draft tagged that bullet "Robin - this is your axis; edit freely", and her ORCID is now filled in, so the most likely account of the missing model-scale axis is that she rewrote her own bullet. The letter now says so first and asks only in the alternative. Not sent; operator confirm required. Docs only - no core/ change. Quality delta: exempt (label: docs). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Jttr1R2aJ3YjJkS1odiLC3
…olidate Checked the letter by trying to apply it rather than by reading it, and the two replacements conflict. Ali's medians, his spread claim, his "neither the cap nor the instruction" conclusion and the no-trace clause are all one sentence. Change (1) replaced that sentence from "per-cap medians" to its end, which takes the no-trace clause with it; change (2) then offered wording that assumed the sentence was still in its original shape. Applied in order they produce a broken sentence, and the letter had called them "two changes, both in the production bullet" while one of them actually lands in the later boundary-condition paragraph. Restructured to what it should have been: one consolidated drop-in for that sentence carrying both fixes, and one separate drop-in for the echo of the same phrasing further down. Item (2) now diagnoses and points at the consolidated block instead of competing with it, and the opening says where the two edits land. Also matched the "[Robin - this is your axis; edit freely.]" quote to his original punctuation, and restored "on this stack" over "here" in the drop-in, since the record has three stacks in it. Not sent; operator confirm required. Docs only - no core/ change. Quality delta: exempt (label: docs). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Jttr1R2aJ3YjJkS1odiLC3
Cosmetic only - the Converse paragraph had a line running past the 78 column wrap the rest of the letter keeps. Docs only - no core/ change. Quality delta: exempt (label: docs). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Jttr1R2aJ3YjJkS1odiLC3
…t wording fixes Read as prose rather than as claims, which surfaced a different class of problem. One was factual. The letter said the phrase "not what puts it there" "recurs further down", implying it appears twice. It appears only in the boundary-condition paragraph; the production bullet expresses the same idea in different words. Rewritten to say where the phrase actually is. The rest is language. "Smaller by roughly √n" says smaller by an amount when the point is a factor. "Which an inert cause reproduces" attached the relative clause to the sign test rather than to the pattern. "It isn't one cell carrying it" used one pronoun for two referents. "Close to four times that" followed three different figures. Partial quotations carried their punctuation inside on two lines and outside on a third; they are outside now, with the one fully-quoted sentence keeping its own period. The single-drop-in paragraph repeated what the opening had already said. "This reads stronger" pointed at two blocks. "Went out with the restructuring" could be read as published rather than removed. "My OK on the text as it stands" sat oddly next to a request for two changes. And the same draft of Ali's was called "previous" in one sentence and "first" in the next. Not sent; operator confirm required. Docs only - no core/ change. Quality delta: exempt (label: docs). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Jttr1R2aJ3YjJkS1odiLC3
One angle the earlier passes had not taken: effect size. The letter argued that Ali's comparison of an 11.5-token shift against a 48-63 within-cell range is the wrong test for significance, but stopped there. Standardised, the shift is about 0.8 SD on the reply's own length (11.5 / 14.06), which is not a small effect by any convention. So two yardsticks are being blurred in his sentence. Against the distance to any cap the shift is negligible - that is his point and it holds. Against the reply's own variability it is substantial. "Smaller than within-cell spread" states the second while arguing the first, which is why it reads as "no effect anywhere". The paragraph now makes that split explicit, and concedes his half of it first. Not sent; operator confirm required. Docs only - no core/ change. Quality delta: exempt (label: docs). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Jttr1R2aJ3YjJkS1odiLC3
Checked the draft for completeness against his closing ask - "one OK each on this exact text, plus the stitch clause, anything that reads wrong one line back" - rather than for correctness. The OK, the stitch clause and the corrections were all there; four things were not. Identifier transcription was never confirmed, and it is the one check only we can perform. He copied our three DOIs and four repository pointers into the record; all seven are correct (20363998 / 20372649 / 20374227, and #461, #463, #440, #448). Given that this entire exchange began with a mis-transcribed pointer, saying so is not a courtesy. The procedure was left open. He needs two OKs to submit and Robin's ORCID is already filled in, so ours is the last one; "make those two changes and you have my OK" left it ambiguous whether a revised text has to come back for re-approval, which would cost a round trip for nothing. It now says to make the changes and submit. Also confirmed our author row as printed, and that the managed-Gemini forward-pointer parenthetical he added lands correctly - both requested in the first reply, neither acknowledged in this one. Deliberately still silent on the four engineering findings: the first reply opened that slot with no scope or deadline, he did not ask, and reopening it here would create an obligation before the work exists. Not sent; operator confirm required. Docs only - no core/ change. Quality delta: exempt (label: docs). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Jttr1R2aJ3YjJkS1odiLC3
…ollision
Ran option (a): find out why the committed generator cannot reproduce
the published fixture SHA, rather than editing either side to match.
The generator is deterministic (two clean rebuilds, identical SHA), and
`fixture_sha` is a plain sha256 of the file, so the mismatch against the
2026-06-11 run's 9f40d2e0... is a real content difference. The
2026-06-11 generator was never committed - the whole LRB suite arrived
in one commit twelve days later - so it cannot be recovered, and the
adapter imports only math/re/Counter, which rules out any JAMES-side
change.
Comparing a fresh run against the published result JSON localises the
difference completely: R@5, R@10, P@5, P@10 and temporal_accuracy are
identical, only rank-1 ordering moves, and all of the -0.025 sits in one
four-query cell, historical-mid-policy, which drops 0.5 -> 0.0.
The cause is a token collision inside the fixture. The query template
hard-codes the offset - "What was the text of {policy} 16 weeks ago?" -
so every historical-mid query carries a bare "16", and co-pol-016 is
titled "Policy 16: Operating Standard". All four queries retrieve it
regardless of gold. Only policies collide because only policy titles
carry a bare number; the same template over departments ("Department of
Public Works") and projects ("Project Riverwalk Renewal") reproduces
exactly. That cell is measuring a numeric collision, not time-travel
retrieval - and at 2 of 4 already failing, it was doing so in the
published run too.
build_lrb_scenario_s3.py carries the same template and the same numbered
policy titles. S3 is the publication-scale rung and was not measured
here; flagged, not claimed.
Nothing was changed: no generator edit, no re-baselining, no touching
0.7125 in the preprint. The ordering claim survives either way (0.6875
still clears naive-supersede by +0.15), what does not is the
reproducibility of the absolute figure. Four operator decisions are laid
out in the report, and the test stays red because it is reporting a true
non-reproduction.
Docs only - no core/ change. Quality delta: exempt (label: fix).
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Jttr1R2aJ3YjJkS1odiLC3
…e the arc Ali Afana submitted the joint deposit on 2026-08-19: 10.5281/zenodo.22030935, report, CC-BY-4.0, Afana / Converse / Seo in alphabetical order, carrying the two JAMES precision fixes. This lands the repository side of that. Forward-pointer. README and README.ko.md gain a "Cross-stack convergence record" entry under Papers & Reproducibility, using the middleware variant of the stitch clause drafted on this side - Ali's README already resolves to the live DOI with the production clause, and his issue #448 note goes up the same day. The clause names the JAMES leg as the v0.3.1 seven-tier closure (PRs #461/#463) so a reader lands on the right artifact rather than on #440. Archive record. reports/promo-assets/m9-joint-deposit-record.md is the read-only outcome the prep folder's own "what changes after publish" section calls for: the three axes and their owners, the citation corrections and where they originated, the eval_count / thinking-trace caveat that survived into the published text verbatim, both precision fixes, the identifier verification, and what remains open. The prep folder is marked historical rather than deleted - it is the audit trail for how the text got there. Metadata. .zenodo.json cites the joint DOI and its notes no longer describe the collab arc as a pending separate trajectory; the cycle-vs-collab separation rule is restated rather than dropped, since it still governs which JAMES findings may enter that record. CLAUDE.md gets an index row and its Track 2c line closes with the DOI instead of "next Ali action = mid-June 3-author joint-piece". Vadym Arnaut. The record went out three-author. His phrasing ("each variant has its own tax") does not appear in the published text, so it carries no attribution defect and the 3-vs-4 question the catalog was gating is moot for it. The catalog now says so explicitly, and keeps phases 1-3 open only for a future deposit that actually uses the phrase - otherwise a later session reopens a settled question. Still open with Ali: the four engineering findings follow-up promised in the first reply. Docs and metadata only - no core/ change. Quality delta: exempt (label: docs). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Jttr1R2aJ3YjJkS1odiLC3
Ali announced the deposit is live (10.5281/zenodo.22030935) and asked for nothing, so this is a short acknowledgement rather than a reply with content: our forward-pointer is up in both READMEs with the middleware clause and the seven-tier closure named as the leg, thanks for turning both precision fixes around in a day, the four engineering findings stay in their own message once measured, and the arXiv thread is his to open. Deliberately omits three things. The LRB-S2 non-reproduction is a JAMES preprint matter, not his. The "Robin's (PR #440) clause" attribution in his email reads slightly off - that pointer came from our correction - but it is in his prose rather than in the record, and raising it after publication would be petty. And no interim status on the engineering findings, since the first reply opened that slot with no deadline attached. Front matter carries the one operator recommendation: open the DOI and eyeball that both precision fixes actually landed, since zenodo.org and doi.org are blocked here and we could not read the published text back. Not sent; operator confirm required. Docs only - no core/ change. Quality delta: exempt (label: docs). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Jttr1R2aJ3YjJkS1odiLC3
… characters Ali Afana's first correction of the four he sent on 2026-08-19: "Stripping bidi control characters removes the concealment but not the concealed text; removing the whole marked span is a different operation with a different result." He is right, and the weaker version was our implementation choice rather than his advice - the recommendation came from his Track 2c report and we shipped the character strip. Under v1, an RLO attack lost its wrapper and kept its payload: the concealed instruction arrived at the model as ordinary cleartext. The gate was removing the evidence of the attack and forwarding the attack. v2 splits the treatment by what each control actually does. Override characters - LRO (U+202D) and RLO (U+202E) - now take their whole span: opener, contents and terminating PDF together, to the matching PDF or to end of input if unterminated, with depth tracking so an inner embedding's PDF cannot close an outer override. An override forces direction regardless of the characters' own properties, which is precisely the concealment primitive, and it has no legitimate use inside a user's question. Everything else keeps its contents. Embeddings (LRE/RLE) and isolates (LRI/RLI/FSI/PDI) are how legitimate bidirectional text carries a directional run - an English product name inside an Arabic sentence - and deleting their contents would destroy real input. Marks and the zero-width set are single characters with no span at all. This is deliberately destructive for override spans, and Ali's own bidi_04 case shows the cost: three per-digit RLO spans mean the spoofed "120" is removed rather than mis-parsed. That is the safer failure - a validator seeing no number asks again, one seeing the wrong number does not - and both counts land in the audit dict, so the removal stays forensically visible. The audit dict gains override_spans_removed and override_span_chars, and chars_dropped now includes span characters so the existing caller gate in routes/query.py:176 keeps firing unchanged. Verification: tests/test_input_normalization.py 29 -> 35 tests, all passing. New TestOverrideSpanRemoval covers span contents, unterminated spans, nesting against an inner embedding, multiple spans, and the isolate/embedding contents that must survive. The four Ali cases are rewritten against the v2 contract - bidi_01 and bidi_03 now assert the concealed instruction is absent, which is the whole point, and bidi_02's LRE-wrapped digits still survive. Clean multilingual input is byte-identical. ruff clean. The adversarial fixtures are untouched: the runner still must not normalise, since the fixture-to-server boundary is what those cases test. Noted while here, not fixed: the module docstring cites reports/research-runs/bidi-normalization-audit-20260602.md, which is not in the repository. The v2 rationale is written into the docstring itself rather than relying on that reference. Does not touch core/retrieval, core/graph or core/reasoning. Quality delta: exempt (label: fix). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Jttr1R2aJ3YjJkS1odiLC3
…e scorer Ali Afana's third finding: "Keyword gates over Arabic break on ordinary orthography — tatweel, alef maqsura, presentation forms — variants real keyboards produce every day. Where a check is gated behind such matching, ordinary traffic goes unchecked and nothing is logged." Checked before building anything: JAMES has no Arabic keyword gate. ATTACK_PATTERNS in core/security_layer/_policies.py is English and Korean only, so there is no bypass to close in the security layer and this commit does not claim one. Where his point does land is our own measurement path, plus the gate that lets one word arrive in several byte forms in the first place. Runtime gate. Tatweel (U+0640) is display-only elongation and survives both NFC *and* NFKC, so it is removed explicitly. Arabic presentation forms (U+FB50-FDFF, U+FE70-FEFF) fold to base letters only under NFKC, so NFKC is applied per character inside those two blocks. It is not applied globally on purpose - measured, a global NFKC also rewrites circled numerals to digits, ligatures like fi, and full-width forms to half-width, which is not a change a Korean-first system should absorb as part of an Arabic fix. A test pins that: Korean, circled numerals, half-width katakana and full-width digits must come through untouched. The gate deliberately stops short of folding letters. Alef maqsura, the alef family and teh marbuta are what the user actually typed, and some of those pairs are distinct letters rather than variants; rewriting them in the text forwarded to the model changes the input. That belongs at comparison time. Scorer. scripts/adversarial_sweep.py compared substring criteria with a plain .lower(), so a reply containing the forbidden phrase written with tatweel, a presentation form, harakat or an alef variant scored as a clean resist - a false negative in the numbers we report, which is exactly the shape of his finding. _fold_for_match now normalises both sides and does fold letters, which is safe at comparison time in a way it is not in the gate. Kept local rather than imported from core/: the runner is a black-box client and the fixture-to-server boundary is what the bidi cases exist to test. Verification: 68 tests pass across the two suites (was 51). New TestArabicOrthographicVariants covers tatweel, presentation forms, the lam-alef ligature, untouched ordinary Arabic, the no-global-NFKC guard, the letters that must survive, harakat, and idempotence. New TestArabicMatchingFold drives the four variant classes through evaluate_criterion and pins that unrelated Arabic still passes and that the fold is a casefold-only for English and Korean. Clean multilingual input stays byte-identical. ruff clean - a dead reply_lower left by the substitution was removed rather than left for the gate to catch. Does not touch core/retrieval, core/graph or core/reasoning. Quality delta: exempt (label: fix). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Jttr1R2aJ3YjJkS1odiLC3
…he wider exposure
Ali Afana's fourth finding: "Salt your run identities. Ours were keyed
by a human-readable name, and the stack silently find-or-created the
same conversations across four sweeps — turning what were labelled
before and after columns into turns 2 to 5 of a single conversation."
The mechanism is live here and now demonstrated rather than asserted.
routes/query.py:178 defaults session_id to the literal "default",
core/reasoning/engine_memory.py:83 injects that session's last five
turns into the prompt, routes/query.py:231 writes every answered turn
back, and core/reasoning/engine.py:233 runs the memory build *before*
the mode dispatch, so the retrieval path is included too. The engine's
own comment at engine.py:271 already described the failure in as many
words.
scripts/adversarial_sweep.py posted only {"question": ...}, so it took
the server default: all 18 cases of a sweep shared one conversation and
case N was answered with cases 1..N-1 in its context. It now mints
advsweep-<8 hex run salt>-<case_id>, with the salt drawn once per
process so a re-run cannot rejoin a previous sweep, and records the key
on CaseResult so it reaches the run JSON and can be grepped against
audit_log.
Surveying the rest turned the finding out wider than Track 2c. Only four
things POST to /query/ - eval/qvt/oracle.py makes no HTTP call and
bench_lc_scope_arms.py only polls /healthz - and the other three all use
stable, human-readable keys that re-join across runs: bench.py
(bench_<suite>_<qid>), run_ragas.py (ragas_live_<i>) and
q15_repeat_audit.py (q15_audit_<run_idx>). bench.py is the STEP 7 gate
for every PR touching core/{retrieval,graph,reasoning}.
Those three are deliberately not touched. Salting bench.py changes the
conditions under which future numbers are produced, so they would no
longer be comparable to eval/qvt/baseline_<sha>.json without a
re-baseline, and its current key was chosen on purpose (bench.py:323)
to keep suites distinguishable in the trace log - a salt has to preserve
that. That is an operator call.
Whether any published number was actually affected depends on whether
the machine that produced it carried prior turns in
memory/james_memory.db, which is not in the repository and cannot be
inspected from a session container. The report gives the one sqlite
query that settles it, plus the runbook for the Track 2c re-measurement,
which needs a live server and Ollama and stays operator-gated. Until it
runs, no figure from ar_ecommerce-cross-stack-comparison.md is
re-confirmed to Ali - and its "bidi_02 slipped, most likely single-run
noise" note now has order contamination as a live alternative.
Verification: 11 new tests in tests/test_sweep_run_identity.py cover key
uniqueness, run-stability, the salt's shape, and the store-level bleed
itself against a temp DB, plus a guard that the runner still posts the
fixture text byte-for-byte. 79 tests pass across the three suites
touched this session. ruff clean.
Does not touch core/retrieval, core/graph or core/reasoning.
Quality delta: exempt (label: fix).
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Jttr1R2aJ3YjJkS1odiLC3
JavaScript's \d matches [0-9] only — an Arabic-Indic or extended
Arabic-Indic enumerated reply was invisible to the four model-output
facing patterns. Ali Afana's second engineering finding.
Changed to an explicit [0-9٠-٩۰-۹] class at:
- truncation detection (~2092)
- SUGGESTION_PATTERNS ①②③ literals (~2156-2158)
- the new RegExp string form (~2233, \u escapes since it is a string)
- markdown ordered-list rendering (~2970)
Explicit class rather than \p{Nd}: tests/test_suggestion_click.py
extracts the JS literals and recompiles them with Python re, which
rejects \p.
Internal sentinels (\x00TABLE\d+\x00 / \x00CODE\d+\x00) deliberately
left ASCII-only — they are our own emitted markers, not model output.
Verification: node v22 behavioural check on all six extraction cases
(ASCII unchanged, Arabic-Indic now extracted), truncation detection and
ordered-list rendering for both digit systems. Pinned the new spelling
in test_chat_ux_n4_n5.py + added test_numeric_patterns_are_not_ascii_only.
118 passed across the four touched suites; the one red
(test_cluster_header_emits_when_suggestions_exist, missing star icon) is
pre-existing and unrelated — confirmed by git stash.
Quality delta: exempt (label: fix)
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Jttr1R2aJ3YjJkS1odiLC3
Hashevolution
changed the title
The first reply promised a separate message reporting what each finding did or did not reproduce, once measured. Three reproduced as live defects and are fixed on this branch; the fourth reproduced as a mechanism but the measurement it invalidates has not been re-run, so it is reported unfinished rather than closed. Deliberately withheld: any re-confirmation of the Track 2c figures. The one slipped verdict in that table now has order contamination as a live alternative explanation, and the re-run is operator-gated (live server + Ollama). Reporting it as still open is what the first reply committed to. Two accuracy fixes against the code while drafting: the engine injects the last 5 turns, not every prior turn (engine_memory.py:84), so case N carried the five before it, not 1..N-1; and the scorer's letter fold is alef family + alef maqsura + ta marbuta, named exactly. Quality delta: exempt (label: docs) Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Jttr1R2aJ3YjJkS1odiLC3
`start_trace()` called `uuid.uuid7()` unguarded. That function landed in the standard library in Python 3.14; pyproject declares `requires-python = ">=3.10"` and CI pins 3.11, so the call raised AttributeError on every interpreter this project claims to support — taking down the `/query/` edge for any caller that does not mint its own trace_id. Invisible until now for two reasons: the browser client always sends a trace_id (the reasoning-stream poller mints it before the request, so it takes the passthrough branch), and tests/test_observability.py is on the CI ignore list. API clients and our own measurement runners (scripts/bench.py, scripts/adversarial_sweep.py) post without one. `_new_uuid7_hex()` uses stdlib uuid7 when present and otherwise builds the RFC 9562 §5.7 layout directly: 48-bit big-endian Unix-millisecond timestamp, version 7, variant 0b10, 74 random bits from `secrets`. Ids minted either way sort together, which is the only property the rest of the module relies on. Verification: 5 new tests pin version, variant, timestamp freshness, lexicographic time-ordering across a sleep, uniqueness over 5k draws, and that a client-supplied id still passes through untouched. One test forces the fallback even on an interpreter that has uuid7. test_observability 13 passed, 1 remaining red needs the embedding model (huggingface.co is blocked here), not a code path. Also two stale source readers, same root cause and unrelated to the above: `pipeline_synth` became a package under the 20 KB module-size gate, and `inspect.getsource()` on a package returns only its `__init__.py`. Every structural assertion grepping for a symbol in the split-out body was reporting the feature deleted when it had only moved. tests/_pipeline_src.py now walks a package, so the next split is absorbed the same way. Restores test_planner_terse_skip 6/6 and test_longterm_save_admin_confirm 11/11. Quality delta: exempt (label: fix) Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Jttr1R2aJ3YjJkS1odiLC3
… → 32 green) Four independent causes, none of them a live defect in the markup: 1. `LogoTaglineTests` pinned a `<div class="logo">` header on the chat page. v0.6.1 (2026-06-15) restructured that page Claude-style — short brand form moved to `.sidebar-header-brand`, full positioning phrase to the welcome hero. The intent (the brand label is the positioning string, not a bare codename) is intact, so the assertions follow it to where it lives. Added a test pinning the split itself. 2. `ReportCardBubbleTests` required a left accent rail and a `--shadow-card` elevation on the assistant bubble. chat.css records why they are gone: "dropped per operator catch — the bubble was visually crowding the reading area on phones". Restoring them to turn the test green would re-introduce exactly what the operator rejected, so the tests now pin the decision instead, plus the asymmetry (the user bubble keeps its pill). 3. Two modal assertions and the live-dot assertion compared a literal `class="modal"` / `class="live-dot"`. The inline-style extraction rollout appends a generated utility class, so markup that does carry the class stopped matching. Compare class tokens, which is what a browser does and what survives the next utility class. 4. The firstrun modal's `max-height:88vh` / `overflow-y:auto` moved out of the inline style attribute into the extracted utility class (`tokens.css:596`). The test follows the declarations into the stylesheet and also checks the referenced class is actually declared. One real gap surfaced and is recorded rather than asserted away: the chat page carries no "system live" cue at all now, because the live-dot lived inside the header logo the redesign removed. Closing it means adding markup to the sidebar header — a UI decision, not a test fix. `test_chat_page_has_no_logo_header_by_design` fails the moment a `.logo` header returns, which is when index.html should rejoin the loop. Verification: 32 passed. Mutation-checked that the two new guards bite — re-adding the rail fails test_james_bubble_has_no_chrome, and re-adding a `.logo` div to index.html fails the by-design test. Tree restored clean afterwards. ruff clean. Quality delta: exempt (label: fix) Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Jttr1R2aJ3YjJkS1odiLC3
Two stale points in tests/test_local_vs_cloud_paired.py.
`call_local` grew a `local_backend` kwarg. Four monkeypatch stubs kept
the old signature, so every stubbed call raised TypeError inside the
harness's `except Exception` and was recorded as a *candidate* error —
which makes the harness skip the judge entirely. One test failed loudly
on that ("skipped: candidate error" where it expected the judge's own
message); the other three kept passing while no longer exercising the
path they were written for. All four stubs now take the kwarg.
The caveat block gained `chat_mode_lenient_judge` — chat-mode fixtures
are intrinsically judge-only, since only the factual_chat sub-class
carries gold_signals — and the exact-set assertion was never updated.
Added rather than loosened: dropping a caveat silently is the failure
this guards against, so adding one should be a conscious edit here too.
Renamed off "five", which the list stopped being some time ago.
Verification: 19 passed (was 17 passed / 2 failed), and the three
previously-vacuous tests now reach the judge branch they assert about.
ruff clean.
Quality delta: exempt (label: fix)
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Jttr1R2aJ3YjJkS1odiLC3
…its non-reproduction Ali's 2026-08-21 message specifies the shape: "One message per finding is the right shape, and I would rather have a slow measured answer than a fast impression. Please send the non-reproductions with the same weight as the reproductions." The 4-in-1 draft (16a638e) is deleted rather than kept alongside — git holds it, and leaving a superseded draft in the tree invites a later session sending the wrong one. Four letters plus an index replace it, in their own folder: this is a separate thread from the joint deposit, whose prep folder stays CLOSED. Splitting unblocks three. ①②③ are measured and can go now; only ④ waits on the Track 2c re-run, which is operator-gated. Batching would have held the three hostage to it. ③ now opens on the non-reproduction, as he asked. The half of his finding about keyword gates does not hold here: our injection detector carries 31 literal patterns and 13 regexes with zero Arabic-script characters between them, so there is no Arabic check for a variant spelling to slip past. That is a coverage gap of a different kind and the letter names it as such rather than letting it pass as a confirmation. The earlier draft had deliberately omitted this to avoid giving him something to answer; his request overrides that. Five corrections found by checking each claim against the source rather than against the prior draft: - ① had the attribution backwards, inherited from commit e19f239's own note. His Track 2c report recommends "strip/normalize bidi control characters at input" (X3, and the Provia list) — the character strip is what we built, following his advice. Span removal is his 2026-08-19 revision of his own recommendation. The letter now says so. - ③ claimed presentation forms already matched under the old scorer. They did not; the first test used a malformed string. Rebuilt the variant from codepoints that NFKC folds back to each base letter: tatweel, harakat, presentation forms and the alef family all failed to match before and all match now. - ③ said the scoring fault "is not zero by construction". It may well be zero — it only fires if a reply actually spelled a forbidden phrase with a variant. Now: the scoring was capable of crediting us wrongly, not that it did. - ③ described global NFKC as folding "full-width forms to half-width". It goes both ways (ｱ→ア). Narrowed to the three effects verified. - ② described the four sites wrongly ("two enumerated-list parsers"). It is six literals in four places, and the fourth is the pass that strips an enumerated line once it has been lifted into a chip — which is why the four had to move together. Also: ②③ were swapped against Ali's own numbering in the old draft body (confirmed from the commit messages: ①bidi ②digits ③Arabic ④salt), and ④ ended by asking whether de-confounding was worth the cycles, which breaks the no-new-obligations rule the letters are written under. Docs only. Quality delta: exempt (label: docs) Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Jttr1R2aJ3YjJkS1odiLC3
… errors out The first pass produced five corrections, which is a symptom, not a result. Root cause: the letters were drafted against the previous draft and the commit messages rather than against source, and commit e19f239's own note is self-contradictory, so its error propagated. This pass enumerated every checkable claim in the four letters and verified each against source by command, including executing the pre-fix modules out of git history. Two real errors found and removed: 1. ④ claimed "our own code comment already described the outcome in as many words" and quoted it. That comment (engine.py:266-274) is about an unrelated bug — the 2026-05-09 force_web_search chip click taking the chat path. The quote was also inexact. The most self-critical sentence in the letter, "documented in our source and still shipped", rested on a misreading Ali could have caught by opening the file. Removed; replaced with what is actually true — none of the three lines is wrong alone, which is why the measurement consequence went unnoticed. 2. ① asserted his August letter "revised your own earlier recommendation". That attributes intent we cannot verify: his report's "strip/normalize" may have meant span removal all along. Replaced with the two texts side by side and the responsibility left on our side, where it belongs. One claim strengthened: ③'s non-reproduction now rests on a whole-tree scan rather than counting two lists. The only lines in core/ containing Arabic-script characters are four in the docstring 8c6f726 added, which git blame confirms. Verified as accurate and left unchanged: v1 stripped controls and left the payload as cleartext (confirmed by running the pre-fix module on an RLO attack) · v1 applied NFC · bidi_04's three per-digit RLO/PDF spans and the surviving cleartext 120 · the old scorer's plain .lower() substring compare · the old sweep posting {"question": text} only · the truncation helper's return-true semantics · zero Arabic-Indic digit coverage in tests before the fix · exactly six changed lines carrying \d across four places · the bench/ragas/q15 key shapes read from source rather than from our own report · the bidi_02 slip record · and that the bidi_01..04 test bodies were genuinely rewritten, v1 asserting only that the controls were gone where v2 asserts the concealed instruction is. Also: ④'s two Korean warning blocks merged into one, and its claim that finding ③ "broke the same scorer" tightened — ③ broke the scorer, ④ contaminated the runs that go through it. Docs only. Quality delta: exempt (label: docs) Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Jttr1R2aJ3YjJkS1odiLC3
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Jttr1R2aJ3YjJkS1odiLC3
…es and does not do to ④ An audit of the Arabic path was checked claim-by-claim against source rather than taken at face value. Eight of its nine findings reproduce exactly, line numbers included; its one conclusion — "Arabic RAG quality cannot be measured" — is wider than the code supports. New report: reports/research-runs/arabic-pipeline-capability-audit-20260822.md The substance. core/i18n.py::detect_language counts Hangul syllables against ASCII letters and takes the larger; Arabic scores zero on both, so the tie-break sends MSA down the Korean branch and a few Latin characters flip it to English. Seven modules consume that verdict — planner, query rewriter, synthesiser, memory builder, verifier, reflection loop, softener — so an Arabic question is planned, rewritten, synthesised and verified under Korean scaffolding, and verify.py:459 hands a blocked Arabic user a Korean refusal. Three tokenisers match [가-힣A-Za-z0-9]+ and the query-time entity sanitiser strips everything outside Hangul and ASCII, reducing an Arabic entity name to one space. ③ now carries this, because stopping at "we have no Arabic keyword gate" under-reports what is true, and because its old line "widening the detector is a policy change" reads as though the detector were the only gap. This changes the letter's character from "we fixed your finding" to "our Arabic support was never an equal-conditions measurement", which is the honest answer he asked for by name. Two of my own claims were wrong in opposite directions and are fixed: - Under-stated: I wrote that zero tokens costs "one of the three query variants". Traced it — query_expander.py:183 returns the query unchanged on zero tokens, so the expanded variant deduplicates against the original at orchestrator.py:101, and the empty keyword variant is dropped by the truthiness check. Arabic retrieves on one query where Korean retrieves on two. Verified by running the trio. - Over-stated: "Arabic text does not build graph nodes here" covers the ingestion path, which I never traced. _safe_json_load has exactly one consumer (retrieval_engine.py:172), so the claim is now scoped to query-time extraction, and the ingestion path is named as untraced. ④ gets the audit's consequence stated precisely rather than as a second contamination. Running the eighteen Track 2c cases through the classifier splits them 12 Korean / 6 English, and the split does not follow the fixture's own language labels — six ar-LV cases go five/one, three msa cases go two/one. For the re-run that is a constant, not a variable: the text does not change, so it cannot mask the salt effect and the re-run stays valid. Where it bites is comparison across rows, which may reflect scaffolding rather than language. Calling it a second contamination of the re-run would have been wrong. No promise of a fix is made to Ali. Adding Arabic classification means the detector, seven consumers, three tokenisers, the sanitiser, the refusal message and the RTL/lang layer — a v0.6 scope conversation, not a line in a message about four unrelated findings. Docs only, no core/ change. Quality delta: exempt (label: docs) Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Jttr1R2aJ3YjJkS1odiLC3
… in the runbook it replaces The re-measurement blocking finding ④ cannot run in a session container: ollama is not installed and ollama.com is not reachable through the proxy, huggingface.co is blocked so the embedding model cannot load and the server will not boot. Verified rather than assumed. So this makes it one command on the operator's machine instead. Building it surfaced two faults in the three-line runbook it replaces. **Wrong order.** The runbook wiped conversation_history first. That wipe is exactly what destroys the answer to the question the wipe is meant to serve — whether any published number was actually produced with prior turns in context. The script captures per-session turn counts first, writes them to disk, and changes nothing at all without --yes. **Wrong comparison.** Diffing a fresh run against §2 of the Track 2c table cannot isolate the salt. That table was last written 2026-06-23; 19 commits to core/ and 73 in total have landed since, so any verdict that moves is confounded by drift and any verdict that holds proves nothing. The finding ③ scorer fix, on this same branch, confounds it again. Replaced with a paired design on one build: arm A — every case shares one conversation key (the pre-fix behaviour) arm B — salted per-case keys (the fix) Same build, same fixture, same model, history wiped before each arm. Drift, the scorer fix and our Arabic language misclassification are all present in both arms and cancel. The A↔B difference is the contamination effect, measured instead of argued. The published table is still printed for reference, explicitly flagged as drift-confounded. Arm A needs the old behaviour on demand, so adversarial_sweep.py gains --shared-session-key: every case sends that one key. It prints a warning when used, records run_identity_mode / shared_session_key / run_salt in the run payload so the two arms are distinguishable after the fact, and its docstring says it is for the paired A-arm and nothing else. One real bug caught by dry-running the parts that work here: Path.relative_to raises for anything outside the repo, and the only caller that mattered ran at step 5 — after the wipe and both sweeps. An operator pointing --out-dir at /tmp would have lost the run to a formatting call. Replaced with a helper that falls back to the absolute path. Verified without a server: preflight correctly reports all four blockers in this container and exits 1; --evidence-only captures and exits having changed nothing; the baseline table parser reads all 18 cases; compare and the letter-block emitter were driven with synthetic arms for both outcomes. When nothing moves, the emitted block says so in words that resist hedging — that result is a live possibility and the letter names it in advance. ④'s gate block and the arc README now carry the one command. The 2026-08-19 report's runbook is marked superseded with both faults named. Does not touch core/. Quality delta: exempt (label: fix) — measurement-side. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Jttr1R2aJ3YjJkS1odiLC3
The four letters each carried "assumes the third reply has gone out" as a precondition. It has: Ali' fourth message is the answer to it. Three of its points map one-to-one onto the third reply' body — the README entry checked in both English and Korean, the leg resolving to #461/#463 as v0.3.1 rather than #440, and "one message per finding" answering our "in its own message". Recorded as fact rather than assumption, so the next session does not re-open it or hold the letters for it. Send order: 1, 2, 3 now; then the Track 2c re-measurement; then 4. Docs only. Quality delta: exempt (label: docs) Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Jttr1R2aJ3YjJkS1odiLC3
…letters Operator chose to send findings 1-3 as one message. Ali asked for one message per finding, so this does not go out silently: the letter opens by naming the choice, says the three arrived together and read better as one, and says the fourth still comes on its own because it depends on a measurement — which is the case where his shape genuinely holds. Content is identical to finding-1/2/3. Verified by normalising both and diffing every substantive sentence: 17, 11 and 47 sentences, three differences, all deliberate. - "so I am putting that half first" → "so that half goes first", moving under a section heading. - The tokeniser character class is spelled out in words rather than pasted, so it cannot break in a mail client. - "a message about four unrelated findings" → "about findings that are not about it". Merging made "four" wrong: this letter carries three. The per-finding letters stay. Finding 4 goes out in that form, and a later session needs to know which version was sent. Docs only. Quality delta: exempt (label: docs) Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Jttr1R2aJ3YjJkS1odiLC3
Hashevolution
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Ali Afana's four engineering findings, all worked. Three reproduced as live defects. The fourth reproduced as a mechanism; its measurement is operator-gated and still owed.
override_spans_removed/override_span_chars.\dis ASCII-only, so an Arabic-Indic enumerated reply was invisible to the chip extractorchat.jsmoved to an explicit class. Internal sentinels left ASCII-only on purpose.session_id="default", so history bled between casesA live production defect, found while working the findings.
start_trace()calleduuid.uuid7()unguarded. That landed in the stdlib in Python 3.14;pyprojectdeclares>=3.10and CI pins 3.11, so it raisedAttributeErroron every supported interpreter — taking down the/query/edge for any caller that does not mint its own trace_id. Invisible because the browser always sends one andtest_observability.pyis on the CI ignore list; our own measurement runners do not. Replaced with an RFC 9562 §5.7 fallback, five tests pinning version, variant, timestamp, ordering and uniqueness.Arabic pipeline capability audit (
reports/research-runs/arabic-pipeline-capability-audit-20260822.md).detect_languagecounts Hangul against ASCII and takes the larger; Arabic scores zero on both, so MSA falls through to the Korean branch and a few Latin characters flip it to English. Seven modules consume that verdict. Three tokenisers yield zero tokens for Arabic — retrieval drops from three query variants to one. The query-time entity sanitiser reduces an Arabic entity name to a single space. Nothing is fixed here; the audit exists so ③ reports scope honestly and so a v0.6 conversation has evidence.Paired Track 2c re-measurement harness (
scripts/research/track2c_remeasure.py). Replaces a three-line runbook with two faults: it wipedconversation_historybefore capturing the evidence that wipe destroys, and it diffed against a table from 2026-06-23 that ~19core/commits of drift have since made unusable for isolating the salt. Now runs both arms on one build — A shares a key, B salts — so drift, the ③ scorer fix and the language misclassification are present in both and cancel.adversarial_sweep.pygains--shared-session-keyfor the A arm.The three-author convergence record is published —
10.5281/zenodo.22030935, Afana / Converse / Seo, carrying two JAMES precision fixes. Repository side: forward-pointer in both READMEs, archive record, metadata, Vadym attribution chain closed.LRB-S2's published R@1 does not reproduce from the committed generator — root-caused, nothing changed.
Standing test failures 101 → 71 in CI, across four clusters, each traced to a cause rather than silenced.
Verification
Local runs use
--noconftest;tests/conftest.pyinstantiates the embedding model andhuggingface.cois blocked here.test_cluster_header_emits_when_suggestions_exist, a missing star icon) is pre-existing, confirmed bygit stash.TestHistoryBleedMechanismpointscore.memory.db.DB_PATHat a temp file and shows turn N+1 seeing turn N under a shared key and not under salted keys.TestRunnerStillPostsRawTextkeeps the "MUST NOT normalizetext" discipline.uuid7exists.--evidence-onlychanges nothing; the table parser reads all 18 cases;compare_armsand the letter-block emitter driven with synthetic arms for both outcomes.ruff check .andbanditclean.Test clusters restored
test_planner_terse_skip+test_longterm_save_admin_confirmtest_ui_report_polishThe
pipeline_synthcluster was one root cause: it became a package under the 20 KB module-size gate, andinspect.getsource()on a package returns only__init__.py, so structural tests reported features deleted when they had only moved.tests/_pipeline_src.pynow walks a package.Quality Delta vs baseline
Quality delta: exempt (label: fix)
core/input_normalization.pyandcore/observability.pyare the input gate and the trace edge — notcore/retrieval,core/graphorcore/reasoning. No traversal, retrieval or reasoning line changes. The rest is test-side, measurement-side and docs.Out of scope
langlayer. That is a v0.6 scope conversation under the mother-platform rule, not a line in this PR.0.7125. Four options are laid out for an operator.test_token_mode_s2_reproduces_phase_b_baselinestays red on purpose — it reports a true non-reproduction.!important29 vs a budget of 25 inmobile.css— real CSS debt; raising the budget would be quarantining the test..logoheader returns.🤖 Generated with Claude Code
https://claude.ai/code/session_01Jttr1R2aJ3YjJkS1odiLC3