Repository navigation
Description
Observation
The documented reasoning model (CIC CLAUDE.md: graph-first, relationship-type filtering, local subgraph) is more advanced than the retrieval implementation. The bridge breaks at graph traversal: the model exists, the implementation does not follow it.
Verified at 270d6b6 (feat/quality-gates), data kb_data/pkl:
- 3873 nodes, 124 333 edges. Edge types:
related-to103 734 (83%),references15 358,refers-to3872,calls1369.kb_data/edge_types.mdlists onlyrefers-toandrelated-to, so it is stale. refers-tois exactly the chunk-order chain: 3872/3872 edges aren_i -> n_{i+1}, across file boundaries.focus_pack(mcp-server/server.py:1008) andimpact_analysis(:1231) follow every outgoing edge regardless of type.guided_path(:1309) readsedge_typeonly for the explanation string.neighborscan filter by type, but nothing above it uses that.- Nodes carry
id, chunk_id, type, label, tags, category, used_in, entrypoint: nostatus,provenanceorauthority. Edges do carryevidence(file + line range),weightandevidence_chunk_idon all 124 333 edges, so provenance exists at edge level only.
Question
How much does untyped traversal change what focus_pack returns, and is the effect concentrated in the refers-to chain (as first suspected) or elsewhere?
Method
Real focus_pack imported from mcp-server/server.py with KB_DATA_DIR=kb_data/pkl, top_k=5, depth 1 and 2. The query seeds are identical in every variant; only kb["adj"] differs. Queries were fixed beforehand (five, taken from a review):
- Mi a CIC-Relay?
- Mi a primitive layer szerepe?
- Hol van az authority boundary?
- Mi számít igazságnak a CIC-ben?
- Milyen kapcsolat van schema, runtime és proof között?
Variants: A all edges (current), B without refers-to, C only calls + references (exploratory, added after seeing B).
Results
A vs B (drop refers-to only)
| query | depth | related nodes A→B | final chunks only via refers-to (of ≤50) |
files only via refers-to |
|---|---|---|---|---|
| Relay | 1 / 2 | 83→80 / 478→450 | 1 / 2 | 0 / 5 |
| primitive layer | 1 / 2 | 15→11 / 43→23 | 4 / 20 (of 43) | 1 / 12 |
| authority boundary | 1 / 2 | 99→94 / 401→391 | 2 / 1 | 1 / 0 |
| truth | 1 / 2 | 144→140 / 479→461 | 3 / 1 | 1 / 4 |
| schema-runtime-proof | 1 / 2 | 36→32 / 130→103 | 4 / 14 | 0 / 6 |
A vs C (only calls + references)
| query | depth | related nodes A→C | chunk overlap (of A) | files A→C |
|---|---|---|---|---|
| Relay | 1 / 2 | 83→42 / 478→153 | 30 / 18 | 38→11 / 114→23 |
| primitive layer | 1 / 2 | 15→5 / 43→5 | 5 / 5 | 8→4 / 31→4 |
| authority boundary | 1 / 2 | 99→73 / 401→182 | 29 / 19 | 26→14 / 88→23 |
| truth | 1 / 2 | 144→89 / 479→202 | 35 / 23 | 39→16 / 98→28 |
| schema-runtime-proof | 1 / 2 | 36→21 / 130→43 | 21 / 16 | 12→7 / 45→9 |
Findings
- The
refers-tochain alone matters less than suspected. At depth 1 it adds 1–4 chunks per query; the 50-chunk cap infocus_packhides most of the rest. It matters when the result is small:primitive layerat depth 2 has 20 of 43 chunks (47%) reachable only through chunk-order links. - The dominant mass is
related-to(embedding similarity). Restricting tocalls+referenceschanges the result a lot: at depth 2 only 16–23 of 50 chunks overlap in the four capped queries, and the file set shrinks by roughly 1.7–8x in every query (e.g. 114→23, 88→23, 31→4). - These numbers show that typing changes the context. They do not show that the typed context is better: no ground truth or answer-quality rating exists yet, and C is also much poorer (5 nodes for
primitive layer), which may just meancalls/referencesis sparse for that topic. - Depth 1 is mostly capped, so the real effect is hidden at depth 1 and visible at depth 2.
Caveats
- 5 queries, one KB snapshot, one seeding method;
related-toedges are embedding-derived, so the seed neighborhood is partly self-reinforcing. - Results are file/chunk-set differences, not relevance judgements.
- Variant C was chosen after seeing B.
Proposed next steps
- Regenerate
edge_types.mdfrom the actual edge set (4 types, not 2). - Give
focus_pack/impact_analysis/guided_pathanedge_typesallow-list parameter (default = current behavior) so the effect can be switched per call. - Decide whether
refers-to(sequence) belongs in reasoning closure at all. - Add
status/provenance/authorityto nodes (or define them as derived from edges/evidence). - Quality experiment: same model, same questions, context from A vs a typed variant, rated against fixed criteria (CIC-own terms kept, false external abstractions, claims with a source). Only then decide defaults per reasoning mode.
Origin: an external AI review of the KB package; each claim above was re-checked locally, and one was corrected (provenance exists on edges).
Activity
sinkog commentedon Oct 17, 2026
Quality experiment: first result (inconclusive, leaning against variant C as a default)
Setup. A = current
focus_pack(all edges), C = onlycalls+references;depth=2,top_k=5; the 5 fixed questions. Each cell: one fresh agent (Sonnet) that saw only its context file (chunk texts, 1200 chars each, in returned order) and had to answer in Hungarian with chunk citations and say what it cannot know. 10 answers total, 1 run per cell.Context size (chars), A / C: q1 18.9k / 19.0k, q2 32.1k / 2.9k, q3 20.1k / 20.2k, q4 19.3k / 18.4k, q5 21.3k / 15.8k.
Deviations from the registered protocol (be careful with this):
M1 own-term recall (checklist e.g. q1: ProofTrace, workflow_id, Cabinet, Vault; mere string presence, counted even where the answer says the term is missing): A 14/22, C 12/22. Per question A/C: q1 3/4 vs 1/4, q2 4/5 vs 4/5, q3 3/4 vs 4/4, q4 1/4 vs 0/4, q5 3/5 vs 3/5. Too small and too noisy to conclude anything.
Reading (unblinded):
Setx->validateInputSchema,ProofArtifactFromTrace, scaffoldpki_verify.go) and names the exact point where the bridge breaks. C stays at the documentation level and says there is no call-path evidence.What this does and does not show.
related-toentirely (C) did not produce better answers on these 5 questions; on the ones that need implementation evidence it was worse. Sorelated-tocarries useful signal and "calls+references only" is not a good default.refers-tochain pollutes context) was not tested for quality: variant B (all edges exceptrefers-to) was not run through the answerers, and the measurement above showedrefers-tomatters mainly for small result sets.Next, if we continue: run B through the same pipeline, repeat each cell 3x, run the blind judge with the registered criteria, and write the term checklists into the protocol first.
sinkog commentedon Oct 17, 2026
Quality experiment v2 (A / B / C, 3 runs per cell, blind judge) -- result: no variant beats A; the judge could not discriminate
Protocol: registered before any v2 answer existed (term checklists and decision rule included). Variants: A all edges (current), B all edges except
refers-to, C onlycalls+references. 5 fixed questions x 3 variants x 3 independent answerer runs = 45 answers; contexts as in the previous comment (depth=2,top_k=5). Blind judge: one agent per question, 9 answers shuffled under neutral ids, evidence pool = union of the three contexts.Decision rule (fixed beforehand): a variant is "better" than A only if its mean precision (supported/total claims) is higher on >= 3 of 5 questions AND its mean unsupported count is not higher on >= 3 of 5.
Judge result: every one of the 45 answers was rated 100% supported, 0 unsupported, bridge_stated = 1 (each answer says what it cannot determine). Mean claims per answer: A 11.3, B 11.4, C 9.7. By the pre-registered rule neither B nor C is better than A, and A is not better than them either: the judge measures nothing here. Likely reasons: the answerers were told to answer only from the context and to state gaps, the evidence pool is the union of all three contexts (a claim from any variant is "supported"), and "supported by the cited chunk" is a weak test (it is not "true"). I have not shown that the judge can fail (no seeded false claim), so this null result is a blind spot of the instrument, not evidence that the variants are equal.
Automated own-term recall (M1, string presence, mean over 3 runs):
M2 (external tech names absent from the context) is 0 for all 45 answers, so it does not discriminate. Answer length is equal (219 / 223 / 216 words).
What is left standing:
refers-toremoved) is also worse on q1 (0.25 vs 0.67 for A; the A answers mentionworkflow_idand the cabinet scope, the B answers do not). I did not check why; plausible cause is that the chunk-order chain pulls in the next chunks of the same document, i.e. continuity, which is useful here, not only noise. That contradicts the initial suspicion in this issue ("refers-topollutes the context") for at least this question. Single question, not conclusive.Not shown: that typed traversal improves answers; also not shown that it does not. n = 5 questions, one model family answering and judging, an insensitive judge, and a keyword-presence metric.
Proposed next step (if worth continuing): first make the instrument able to fail -- seed known-false and known-unsupported claims into some answers and check that the judge catches them; use a pool per variant instead of the union; use reference facts from authoritative CIC docs (e.g.
SYSTEM_CONTEXT) for a correctness check instead of "supported by pool". Until then I would not change the defaults offocus_pack/impact_analysis/guided_path; the allow-list parameter from the earlier proposal is still cheap and lets the effect be switched per call.sinkog commentedon Oct 17, 2026
Quality experiment v3: the instrument can now fail, and the answer is still "no difference in precision"
Registered before running (
PROTOCOL_v3): (1) validate the judge by seeding known-bad claims; (2) judge every answer against its own variant's context only (not the union); (3) the judge counts as usable only if sensitivity >= 0.8.Judge validation. For each question I took the rep-1 answer of A, B and C (15 answers) and appended 2 false/unsupported sentences, each citing a real chunk of that context (e.g. "SHA-512" where the chunk says SHA-256; "build only warns and continues" where the chunk says the build fails; "Let's Encrypt root" where it says CICRootCA; "12 atomic primitives" where it says 7; "the applier knows the policy" where the context says the opposite). Strict fact-checker judges (one per question, neutral ids, own pool per answer).
So the instrument is usable by the pre-registered criterion, and the earlier "100% supported" result was not just a dead judge.
Re-judge of the 45 original answers, own pool per answer, strict:
refers-to)calls+references)Per question, A/B/C are all 1.000 on q1-q3; q4: C 0.971; q5: A 0.978. By the pre-registered rule (higher mean precision on >= 3 of 5 questions) no variant is better than A; B is higher on 1 question (q5), equal elsewhere; the single unsupported claims in A and C are one-off.
Reading. Precision is not where the variants differ: the answerers were told to answer only from the context and to state gaps, so they almost never invent. The measurable difference is coverage (own-term recall, previous comment: A 0.65, B 0.55, C 0.51; the q1 gap A 0.67 vs B/C 0.25 is the largest). I have not yet shown that higher coverage means a more correct or more useful answer, only that it names more of the expected CIC terms.
What this closes and what it does not.
SYSTEM_CONTEXT) is still missing.Recommendation for #8: do not change the traversal defaults on the strength of this. The
edge_typesallow-list parameter (default = current behaviour) and the regeneratededge_types.mdstay worthwhile as plumbing; the claim that untyped traversal measurably hurts answer quality is not supported by these experiments. If you want to push on the original hypothesis, the next experiment should be completeness against reference facts and a retrieval-level metric (recall of the chunks a reference answer needs), not answer-level precision.sinkog commentedon Oct 17, 2026
Fix list (what is confirmed and gets fixed in the PR)
Scope decision: the experiments (comments above) did not show a quality effect of untyped traversal, so the traversal defaults are not changed. Only the verified, behaviour-neutral items are fixed:
kb_data/edge_types.mdis stale (tracked artifact, 2 types listed, graph has 4). The generatormake_source.py:generate_edge_types_docalready enumerates all types; the committed file simply predatesreferences/calls. Fix: regenerate, and make the generator also state, per type, what it means (refers-to= chunk-order chain,related-to= embedding similarity,references= explicit reference,calls= call graph) and its edge count, so that readers do not treat them as equivalent.focus_pack,impact_analysis,guided_pathfollow every outgoing edge;guided_pathonly prints the type. Fix: optionaledge_typesallow-list parameter on all three, defaultNone= current behaviour;guided_path/impact_analysis/focus_packapply the same filter helper; unknown or empty lists are handled explicitly. Tests for default-unchanged and filtered behaviour.server.yamltool descriptions andMANIFEST.sha256updated accordingly (repo requiresmanifest-verify).Not in this PR (observations, no action):
refers-toappears to give useful continuity for q1; q4 ("what is truth") is weak in all variants (looks like a seed/ranking problem); nodestatus/provenance/authorityand epistemic edge types are ideas without evidence yet; reference-based completeness measure is open.feat(mcp): optional edge_types allow-list for graph traversal, docume…
sinkog commentedon Oct 17, 2026
Closed: PR #9 merged to main (
b845d70), main CI green on that commit.What was fixed: optional
edge_typesallow-list onfocus_pack/impact_analysis/guided_path(default unchanged), andkb_data/edge_types.md+ its generator now cover all 4 edge types with their meaning.What was investigated and not found: no measurable quality effect of untyped traversal (see the experiment comments; precision identical across variants with a judge validated by seeded faults, 30/30 caught). Traversal defaults are therefore unchanged.
Left open as observations, not tracked here:
refers-tocontinuity benefit (q1), weak q4 ("what is truth") in all variants (seed/ranking), node status/provenance/authority and epistemic edge types, a reference-based completeness measure, and the pre-existing drift ofserver.yaml/make_source.yamlfrom the code.