Repository navigation
Fix served-model CI: record lineage for the July v5 incumbent - #100
Merged
Merged
Conversation
The served-model job has been red since it landed (before PR #99): master serves f32 blob a4918f0 with no mtnn_lineage.json, and the original promote.py run directory is not in this repo. Record that incumbent's bytes, keys hash, and existing meta metrics, stamp sidecar run ids, and keep the check strict — a different f32 is still refused. Co-authored-by: Cam Davis <jcdavis131@gmail.com>
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
The recorder already refused a non-incumbent f32; the check did not, so a hand-written forced lineage could bless a future model. Reject forced adoption unless embedding_f32_sha256 is the pinned July v5 sha. Stop stamping mtnn_jacobian.json and mtnn_attr_pop.json with the v5 run id: they are 48-d Jul 11–14 exports. Mark them not_from_served_net and have network-viz refuse to treat them as the shipped net. Co-authored-by: Cam Davis <jcdavis131@gmail.com>
This branch was successfully deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.


Root cause
served-modelhas been red since it landed, not since PR #99.budgetsis a separate workflow;siteandoffline-buildare green on those red master runs).The served tree is still the July 2026 v5 export (
assets/mtnn_embeddings.f32git bloba4918f09, content sha2562574ef58…).pipeline/data/is gitignored, the originalpromote.pyrun directory is not in this repo, and PR #97 showed that retraining v5 does not reproduce the incumbent. The check was correctly reporting “no lineage file.” Making that job a required check before anyone could write the file is why master and PRs #96/#97/#98 stay red.What changed
Did not skip, weaken, or delete the check.
scripts/check_served_model.pystill hashes the f32, the vectors.json key order, meta keys/metrics, sidecarlineage.run_id, and assets/ vs public/ parity.scripts/record_served_lineage.py(stdlib). It writesmtnn_lineage.jsonfrom the served bytes (f32 sha256, keys sha256, metrics already inmtnn_meta.json) for runjuly-v5-a4918f09, withforced: trueand a written reason. A replacement model still has to go throughpromote.py+export_mtnn_embeddings.py.sync_publicso?v=tokens and the Vercelpublic/tree match.Review follow-up
Check-side pin for forced adoption.
served_model.forced_adoption_pin_problem(run bycheck_served_model.py) rejects a lineage withforced: trueand nocheckpoint_sha256, oradoption.kind == served-incumbent, unlessembedding_f32_sha256is the pinned July v5 sha (2574ef581ac3…). The recorder is no longer the only gate: a hand-written forced lineage for a different model fails the check.promote.py --forceof a real bundle still writescheckpoint_sha256and is not this path (needed fortests/test_e2e_smoke.py). Test:test_a_forced_lineage_with_a_different_sha_fails.Mislabeled 48-d sidecars.
mtnn_jacobian.json(built 2026-07-14, dEmb 48) andmtnn_attr_pop.json(built 2026-07-11, different checkpoint bytes) are no longer stampedrun_id: july-v5-a4918f09. They carrystale/not_from_served_netand their own origin run ids (pre-v5-48d-2026-07-14/pre-v5-48d-2026-07-11). The check fails if a sidecar marked not-from-served-net claims the served run.network-viz.jsthrowsnot from served net (stale 48-d export)and leaves Jacobian/attribution unused rather than treating them as the shipped 64-d net.Evidence that CI passes on this PR
Follow-up commit
e11c2a5b. All required checks green:PR #96 (opening-night 2026-27)
This PR does not touch
index.html/public/index.html. Once this lands, #96 needs only a rebase. Anyindex.htmlconflicts would be from #99’s nav work, not from this lineage fix. No model/asset changes required on #96.