Skip to content

Typed edge traversal: focus_pack/impact_analysis/guided_path ignore edge types (measurement + proposal) #8

Closed

Description

@sinkog

Observation

The documented reasoning model (CIC CLAUDE.md: graph-first, relationship-type filtering, local subgraph) is more advanced than the retrieval implementation. The bridge breaks at graph traversal: the model exists, the implementation does not follow it.

Verified at 270d6b6 (feat/quality-gates), data kb_data/pkl:

  • 3873 nodes, 124 333 edges. Edge types: related-to 103 734 (83%), references 15 358, refers-to 3872, calls 1369. kb_data/edge_types.md lists only refers-to and related-to, so it is stale.
  • refers-to is exactly the chunk-order chain: 3872/3872 edges are n_i -> n_{i+1}, across file boundaries.
  • focus_pack (mcp-server/server.py:1008) and impact_analysis (:1231) follow every outgoing edge regardless of type. guided_path (:1309) reads edge_type only for the explanation string. neighbors can filter by type, but nothing above it uses that.
  • Nodes carry id, chunk_id, type, label, tags, category, used_in, entrypoint: no status, provenance or authority. Edges do carry evidence (file + line range), weight and evidence_chunk_id on all 124 333 edges, so provenance exists at edge level only.

Question

How much does untyped traversal change what focus_pack returns, and is the effect concentrated in the refers-to chain (as first suspected) or elsewhere?

Method

Real focus_pack imported from mcp-server/server.py with KB_DATA_DIR=kb_data/pkl, top_k=5, depth 1 and 2. The query seeds are identical in every variant; only kb["adj"] differs. Queries were fixed beforehand (five, taken from a review):

  1. Mi a CIC-Relay?
  2. Mi a primitive layer szerepe?
  3. Hol van az authority boundary?
  4. Mi számít igazságnak a CIC-ben?
  5. Milyen kapcsolat van schema, runtime és proof között?

Variants: A all edges (current), B without refers-to, C only calls + references (exploratory, added after seeing B).

Results

A vs B (drop refers-to only)

query depth related nodes A→B final chunks only via refers-to (of ≤50) files only via refers-to
Relay 1 / 2 83→80 / 478→450 1 / 2 0 / 5
primitive layer 1 / 2 15→11 / 43→23 4 / 20 (of 43) 1 / 12
authority boundary 1 / 2 99→94 / 401→391 2 / 1 1 / 0
truth 1 / 2 144→140 / 479→461 3 / 1 1 / 4
schema-runtime-proof 1 / 2 36→32 / 130→103 4 / 14 0 / 6

A vs C (only calls + references)

query depth related nodes A→C chunk overlap (of A) files A→C
Relay 1 / 2 83→42 / 478→153 30 / 18 38→11 / 114→23
primitive layer 1 / 2 15→5 / 43→5 5 / 5 8→4 / 31→4
authority boundary 1 / 2 99→73 / 401→182 29 / 19 26→14 / 88→23
truth 1 / 2 144→89 / 479→202 35 / 23 39→16 / 98→28
schema-runtime-proof 1 / 2 36→21 / 130→43 21 / 16 12→7 / 45→9

Findings

  1. The refers-to chain alone matters less than suspected. At depth 1 it adds 1–4 chunks per query; the 50-chunk cap in focus_pack hides most of the rest. It matters when the result is small: primitive layer at depth 2 has 20 of 43 chunks (47%) reachable only through chunk-order links.
  2. The dominant mass is related-to (embedding similarity). Restricting to calls + references changes the result a lot: at depth 2 only 16–23 of 50 chunks overlap in the four capped queries, and the file set shrinks by roughly 1.7–8x in every query (e.g. 114→23, 88→23, 31→4).
  3. These numbers show that typing changes the context. They do not show that the typed context is better: no ground truth or answer-quality rating exists yet, and C is also much poorer (5 nodes for primitive layer), which may just mean calls/references is sparse for that topic.
  4. Depth 1 is mostly capped, so the real effect is hidden at depth 1 and visible at depth 2.

Caveats

  • 5 queries, one KB snapshot, one seeding method; related-to edges are embedding-derived, so the seed neighborhood is partly self-reinforcing.
  • Results are file/chunk-set differences, not relevance judgements.
  • Variant C was chosen after seeing B.

Proposed next steps

  • Regenerate edge_types.md from the actual edge set (4 types, not 2).
  • Give focus_pack / impact_analysis / guided_path an edge_types allow-list parameter (default = current behavior) so the effect can be switched per call.
  • Decide whether refers-to (sequence) belongs in reasoning closure at all.
  • Add status / provenance / authority to nodes (or define them as derived from edges/evidence).
  • Quality experiment: same model, same questions, context from A vs a typed variant, rated against fixed criteria (CIC-own terms kept, false external abstractions, claims with a source). Only then decide defaults per reasoning mode.

Origin: an external AI review of the KB package; each claim above was re-checked locally, and one was corrected (provenance exists on edges).

Activity

  1. sinkog commented on Oct 17, 2026

    @sinkog
    MemberAuthor

    Quality experiment: first result (inconclusive, leaning against variant C as a default)

    Setup. A = current focus_pack (all edges), C = only calls + references; depth=2, top_k=5; the 5 fixed questions. Each cell: one fresh agent (Sonnet) that saw only its context file (chunk texts, 1200 chars each, in returned order) and had to answer in Hungarian with chunk citations and say what it cannot know. 10 answers total, 1 run per cell.

    Context size (chars), A / C: q1 18.9k / 19.0k, q2 32.1k / 2.9k, q3 20.1k / 20.2k, q4 19.3k / 18.4k, q5 21.3k / 15.8k.

    Deviations from the registered protocol (be careful with this):

    • The per-question term checklist for M1 was only fixed in my plan, not written to the protocol file, and I scored it after reading the answers.
    • The blind LLM judge (claim support) was not run. What follows in "Reading" is my own unblinded judgement.
    • M2 (external-tech names absent from the context) is 0 for all 10 answers, so it does not discriminate. (The one mention, Terraform in q3/A, is in its context.)

    M1 own-term recall (checklist e.g. q1: ProofTrace, workflow_id, Cabinet, Vault; mere string presence, counted even where the answer says the term is missing): A 14/22, C 12/22. Per question A/C: q1 3/4 vs 1/4, q2 4/5 vs 4/5, q3 3/4 vs 4/4, q4 1/4 vs 0/4, q5 3/5 vs 3/5. Too small and too noisy to conclude anything.

    Reading (unblinded):

    • q5 (schema/runtime/proof): A is clearly better. It reaches code level (Setx -> validateInputSchema, ProofArtifactFromTrace, scaffold pki_verify.go) and names the exact point where the bridge breaks. C stays at the documentation level and says there is no call-path evidence.
    • q2 (primitive layer): C's context is 11x smaller (2.9k vs 32k chars) and the answer is about as good on the what (7 atoms, aggregates, ManagedEntity). A additionally explains the why (50 incompatible domain schemas, git-based composition, the rejected override-rules). So C suffices for "what", not for "why".
    • q1 (Relay): A is more concrete (proof artifact fields, cabinet scope, chain hash); C is more architectural (relay graph, responsibilities).
    • q3 (authority boundary): roughly a tie; both say there is no explicit definition. A has OIS gate detail, C has broader scope and names the missing Vault/CICSourceCA layer.
    • q4 (truth): both stay conceptual and neither identifies a source of truth; C's context contained no ProofTrace at all.

    What this does and does not show.

    • Dropping related-to entirely (C) did not produce better answers on these 5 questions; on the ones that need implementation evidence it was worse. So related-to carries useful signal and "calls+references only" is not a good default.
    • The hypothesis from Typed edge traversal: focus_pack/impact_analysis/guided_path ignore edge types (measurement + proposal) #8 (the refers-to chain pollutes context) was not tested for quality: variant B (all edges except refers-to) was not run through the answerers, and the measurement above showed refers-to matters mainly for small result sets.
    • n=1 per cell, same model answering and (informally) me judging; differences of this size could be run-to-run noise.

    Next, if we continue: run B through the same pipeline, repeat each cell 3x, run the blind judge with the registered criteria, and write the term checklists into the protocol first.

  2. sinkog commented on Oct 17, 2026

    @sinkog
    MemberAuthor

    Quality experiment v2 (A / B / C, 3 runs per cell, blind judge) -- result: no variant beats A; the judge could not discriminate

    Protocol: registered before any v2 answer existed (term checklists and decision rule included). Variants: A all edges (current), B all edges except refers-to, C only calls + references. 5 fixed questions x 3 variants x 3 independent answerer runs = 45 answers; contexts as in the previous comment (depth=2, top_k=5). Blind judge: one agent per question, 9 answers shuffled under neutral ids, evidence pool = union of the three contexts.

    Decision rule (fixed beforehand): a variant is "better" than A only if its mean precision (supported/total claims) is higher on >= 3 of 5 questions AND its mean unsupported count is not higher on >= 3 of 5.

    Judge result: every one of the 45 answers was rated 100% supported, 0 unsupported, bridge_stated = 1 (each answer says what it cannot determine). Mean claims per answer: A 11.3, B 11.4, C 9.7. By the pre-registered rule neither B nor C is better than A, and A is not better than them either: the judge measures nothing here. Likely reasons: the answerers were told to answer only from the context and to state gaps, the evidence pool is the union of all three contexts (a claim from any variant is "supported"), and "supported by the cited chunk" is a weak test (it is not "true"). I have not shown that the judge can fail (no seeded false claim), so this null result is a blind spot of the instrument, not evidence that the variants are equal.

    Automated own-term recall (M1, string presence, mean over 3 runs):

    question A B C
    q1 Relay 0.67 0.25 0.25
    q2 primitive layer 0.80 0.80 0.80
    q3 authority boundary 0.83 0.75 0.75
    q4 truth 0.17 0.17 0.17
    q5 schema/runtime/proof 0.80 0.80 0.60
    all 0.65 0.55 0.51

    M2 (external tech names absent from the context) is 0 for all 45 answers, so it does not discriminate. Answer length is equal (219 / 223 / 216 words).

    What is left standing:

    1. C (calls + references only) gives no gain and is somewhat worse on recall (q5: 0.60 vs 0.80; q1: 0.25 vs 0.67). Same direction as the first run.
    2. Unexpected: B (only refers-to removed) is also worse on q1 (0.25 vs 0.67 for A; the A answers mention workflow_id and the cabinet scope, the B answers do not). I did not check why; plausible cause is that the chunk-order chain pulls in the next chunks of the same document, i.e. continuity, which is useful here, not only noise. That contradicts the initial suspicion in this issue ("refers-to pollutes the context") for at least this question. Single question, not conclusive.
    3. q4 (truth) is low in all variants (0.17): the answers never reach ProofTrace/canonical/digest regardless of edge filtering; this looks like a seed/ranking problem, not an edge-type problem.

    Not shown: that typed traversal improves answers; also not shown that it does not. n = 5 questions, one model family answering and judging, an insensitive judge, and a keyword-presence metric.

    Proposed next step (if worth continuing): first make the instrument able to fail -- seed known-false and known-unsupported claims into some answers and check that the judge catches them; use a pool per variant instead of the union; use reference facts from authoritative CIC docs (e.g. SYSTEM_CONTEXT) for a correctness check instead of "supported by pool". Until then I would not change the defaults of focus_pack / impact_analysis / guided_path; the allow-list parameter from the earlier proposal is still cheap and lets the effect be switched per call.

  3. sinkog commented on Oct 17, 2026

    @sinkog
    MemberAuthor

    Quality experiment v3: the instrument can now fail, and the answer is still "no difference in precision"

    Registered before running (PROTOCOL_v3): (1) validate the judge by seeding known-bad claims; (2) judge every answer against its own variant's context only (not the union); (3) the judge counts as usable only if sensitivity >= 0.8.

    Judge validation. For each question I took the rep-1 answer of A, B and C (15 answers) and appended 2 false/unsupported sentences, each citing a real chunk of that context (e.g. "SHA-512" where the chunk says SHA-256; "build only warns and continues" where the chunk says the build fails; "Let's Encrypt root" where it says CICRootCA; "12 atomic primitives" where it says 7; "the applier knows the policy" where the context says the opposite). Strict fact-checker judges (one per question, neutral ids, own pool per answer).

    • Sensitivity 30/30: every seeded sentence was flagged, in every answer.
    • Two additional flags beyond the seeded ones (a "level 4" wording and a "per-step hash input/output" detail) were stricter-reading cases, not seeded.
    • Caveat: the seeded errors are blatant and sit at the end of the answer; sensitivity to subtle errors is not tested.

    So the instrument is usable by the pre-registered criterion, and the earlier "100% supported" result was not just a dead judge.

    Re-judge of the 45 original answers, own pool per answer, strict:

    variant precision (supported/claims) unsupported claims (total over 15 answers)
    A (all edges) 0.994 1 (q5)
    B (no refers-to) 1.000 0
    C (calls+references) 0.993 1 (q4)

    Per question, A/B/C are all 1.000 on q1-q3; q4: C 0.971; q5: A 0.978. By the pre-registered rule (higher mean precision on >= 3 of 5 questions) no variant is better than A; B is higher on 1 question (q5), equal elsewhere; the single unsupported claims in A and C are one-off.

    Reading. Precision is not where the variants differ: the answerers were told to answer only from the context and to state gaps, so they almost never invent. The measurable difference is coverage (own-term recall, previous comment: A 0.65, B 0.55, C 0.51; the q1 gap A 0.67 vs B/C 0.25 is the largest). I have not yet shown that higher coverage means a more correct or more useful answer, only that it names more of the expected CIC terms.

    What this closes and what it does not.

    • Closed: "typed traversal reduces false/unsupported claims" -- no effect found with a working judge on 5 questions, 45 answers.
    • Not closed: whether C or B yields less complete answers in a way that matters, because completeness was only measured by keyword presence; a reference-based correctness/completeness check against authoritative CIC docs (e.g. SYSTEM_CONTEXT) is still missing.
    • Same limits as before: one model family answers and judges, 5 questions, one KB snapshot.

    Recommendation for #8: do not change the traversal defaults on the strength of this. The edge_types allow-list parameter (default = current behaviour) and the regenerated edge_types.md stay worthwhile as plumbing; the claim that untyped traversal measurably hurts answer quality is not supported by these experiments. If you want to push on the original hypothesis, the next experiment should be completeness against reference facts and a retrieval-level metric (recall of the chunks a reference answer needs), not answer-level precision.

  4. sinkog commented on Oct 17, 2026

    @sinkog
    MemberAuthor

    Fix list (what is confirmed and gets fixed in the PR)

    Scope decision: the experiments (comments above) did not show a quality effect of untyped traversal, so the traversal defaults are not changed. Only the verified, behaviour-neutral items are fixed:

    1. kb_data/edge_types.md is stale (tracked artifact, 2 types listed, graph has 4). The generator make_source.py:generate_edge_types_doc already enumerates all types; the committed file simply predates references/calls. Fix: regenerate, and make the generator also state, per type, what it means (refers-to = chunk-order chain, related-to = embedding similarity, references = explicit reference, calls = call graph) and its edge count, so that readers do not treat them as equivalent.
    2. Traversal tools cannot be restricted by edge type. focus_pack, impact_analysis, guided_path follow every outgoing edge; guided_path only prints the type. Fix: optional edge_types allow-list parameter on all three, default None = current behaviour; guided_path/impact_analysis/focus_pack apply the same filter helper; unknown or empty lists are handled explicitly. Tests for default-unchanged and filtered behaviour.
    3. server.yaml tool descriptions and MANIFEST.sha256 updated accordingly (repo requires manifest-verify).

    Not in this PR (observations, no action): refers-to appears to give useful continuity for q1; q4 ("what is truth") is weak in all variants (looks like a seed/ranking problem); node status/provenance/authority and epistemic edge types are ideas without evidence yet; reference-based completeness measure is open.

  5. sinkog commented on Oct 17, 2026

    @sinkog
    MemberAuthor

    Closed: PR #9 merged to main (b845d70), main CI green on that commit.

    What was fixed: optional edge_types allow-list on focus_pack / impact_analysis / guided_path (default unchanged), and kb_data/edge_types.md + its generator now cover all 4 edge types with their meaning.

    What was investigated and not found: no measurable quality effect of untyped traversal (see the experiment comments; precision identical across variants with a judge validated by seeded faults, 30/30 caught). Traversal defaults are therefore unchanged.

    Left open as observations, not tracked here: refers-to continuity benefit (q1), weak q4 ("what is truth") in all variants (seed/ranking), node status/provenance/authority and epistemic edge types, a reference-based completeness measure, and the pre-existing drift of server.yaml / make_source.yaml from the code.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions