Skip to content

Grounding fingerprints are hex-encoded JSON and take up to 39% of a scaffold #233

Description

@theyashasvipandey

Grounding fingerprints are committed to Markdown as hex-encoded JSON, and in a
setup-populated scaffold they take up more than a third of the scaffold's bytes. Every agent that
reads a context file pays for them in tokens.

Measured

Share of .mex/**/*.md bytes on lines that are fingerprint: or bodyHash::

scaffold fingerprint/hash share
project set up with 0.8.2 39.3%
Hono, populated by setup 37.4%
two older populated projects 27.0%, 26.3%
mex itself 14.2%

The average fingerprint across 66 entries is 1,704 characters.

Cause

serializeFingerprint() (src/graph/fingerprint.ts:81):

const payload = Buffer.from(JSON.stringify(fingerprint), "utf8").toString("hex");
return `${FINGERPRINT_PREFIX}:${K}:${payload}`;

JSON of 64 decimal uint32 values plus full neighbour ids (function:<32 hex>, 5.3 on average),
then hex-encoded, which doubles it.

Options (sizes are averages over the same 66 entries)

encoding avg chars risk
today: hex(JSON) 1,704 —
A. plain JSON, or base64url(JSON) ~850 / ~1,140 none; same data
B. base64url of the 64 minhash values as binary, plus neighbour ids shortened ~400 neighbour shortening changes what reconciliation compares; needs a collision check
C. move fingerprints to a committed sidecar file, keeping only node + bodyHash in frontmatter ~2–3% of scaffold new file format; needs a design

Whatever the encoding, use a new prefix (for example mh2:64:) so deserializeFingerprint
reads both and existing scaffolds keep working. Rewrite old entries only when a file's groundings
are already being rewritten.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    code-graphWork related to the AST and Tree-sitter code graphenhancementNew feature or requestscaffoldRelated to scaffold structure

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions