🔮 glean

A tiny, file-based protocol for memory and distillation.

You glean a finding from larger work. A finding is one small unit of current guidance — what's worth carrying forward.

.glean/
  glean.sh
  in/                            ← raw inbox; anything goes
  findings/
    INDEX.md                     ← generated; always-loaded surface
    example-finding-001.md       ← one flat file per finding
    example-finding-002.md
  out/                           ← inbox items after distillation; swept on retention
    foo.md
  dropped/                       ← retired findings; durable reflection drawer
    bar.md
    bar.reason.md                ← why the finding was retired
  distil.md                      ← host-local distillation instructions, editable

The file system is the protocol.

The finding contract

A finding is one markdown file at findings/<id>.md, it only needs two things:

• Title — first # line.
• Description — first non-empty line between the title and the next heading. One sentence.

Everything else is free prose. Findings can link to other resources. The supplied distil.md suggests common sections:

• Triggers — comma separated items under ## Triggers are keywords searched by the fetch command.
• Associations — wikilinks [[id]] under ##Associations that resolve to findings/<id>.md. Associations are just bullet links - nothing parses them, but they read well and an agent can grep for backlinks trivially.
• Context — examples, references, and longer notes can live under ## Context, and can be loaded on demand for leaner systems.
• Why - reasoning or historical context.

Disclosure layers

Three tiers of cost, by design:

Tier	What	When
Always loaded	findings/INDEX.md	A host symlinks this into its always-loaded context surface. The agent sees one bullet per finding (id + title + description).
Fetched on demand	findings/<id>.md body	Agent runs fetch to get matching paths, then reads the bodies.
Read by hand	Anything else	Long examples, references — surface a finding as a starting point and follow links.

Promotion to always-loaded is the host's call (e.g. by symlinking a specific finding into the host's always-context surface). Glean stays unaware of how its output is consumed.

Procedure

The default flow is: ingest raw material, distil it into findings, fetch findings on demand. Periodically, curate the corpus.

Ingest

Ingest raw material into in/.

./glean.sh ingest some/note.md          # file or directory
echo "rough thought" | ./glean.sh ingest - rough

Distil

Distil is the act of turning raw material in in/ into findings.

Read each item in in/ and choose one of three outcomes:

• revise an existing finding — edit findings/<id>.md in place;
• create a new finding — write findings/<new-id>.md directly (a finding is just a file);
• nothing earned — the material doesn't merit carry-forward.

In all three cases, close the inbox item:

./glean.sh complete <in-id>     # in/<id> → out/<id>

out/ is the audit residue — every inbox item that was considered passes through it, regardless of whether a finding was produced. It's swept on retention. Items remaining in in/ are still awaiting distillation.

Run ./glean.sh index after writing or revising findings to refresh INDEX.md.

The distil.md brief seeded by init is host-local: edit it freely to shape distillation for this glean system. The protocol doesn't care what the brief says — it only cares about the finding contract.

Fetch

Return findings matching the query.

Fetch has two modes.

• strict (default): matches against id, title, description, and contents of ## Triggers.
• all (--allor -a): grep the whole file.

Strict by default keeps the agent's context lean and rewards writers who curate their Triggers section.

Curate

Separately from inbox work, periodically sit with findings/ as a whole: merge findings that overlap, split findings that have grown to cover two ideas, add wikilinks under ## Associations, and retire findings that no longer earn their place.

./glean.sh drop <finding-id> "reason..."   # findings/<id>.md → dropped/

dropped/ is the reflection drawer for retired ideas. It's durable — not swept — so old reasoning remains available to read later.

Vendoring

To add glean to another project, copy glean.sh and README.md into the project's .glean/ directory, then run ./.glean/glean.sh init to seed the trays:

mkdir -p <project>/.glean
cp glean.sh README.md <project>/.glean/
<project>/.glean/glean.sh init

init creates in/, findings/, out/, and dropped/ next to itself, and seeds distil.md if one is not already present. glean.sh operates on the .glean/ directory it lives in, so each vendored copy is self-contained.

Commands

./glean.sh init
./glean.sh ingest <src> [name]      # land raw material in in/ (use - for stdin)
./glean.sh complete <id>            # in/<id> → out/<id>; closes a distillation
./glean.sh drop <id> [reason...]    # findings/<id>.md → dropped/, with reason
./glean.sh index                    # regenerate findings/INDEX.md
./glean.sh fetch [--all] <q...>     # paths of findings matching query
./glean.sh status                   # list trays
./glean.sh sweep [days]             # remove out/ entries older than N days (default 14)