slod-probe — Is Semantic Level of Detail linearly encoded in frozen embeddings?

A controlled probing study for the Temporal Knowledge Hypergraph (TKH) project.

Research question. Scientific papers are written at several granularity levels: an abstract states a contribution ("we propose a method for X"), a methods paragraph states a procedure ("a 3-layer transformer with dropout 0.1"). Call that the Semantic Level of Detail (SLoD) of a span. Can a linear probe on frozen language-model embeddings recover SLoD from weak labels derived from document structure alone, with no manual annotation?

Answer. Yes for a domain-matched encoder, marginally for a general-purpose retrieval encoder, not at all for a general-purpose sentence encoder. SciBERT carries +0.209 nats of information about SLoD beyond what token length and document position already supply, against a dimension-matched randomly-initialised floor; BGE carries +0.064; MiniLM carries nothing distinguishable from zero. Five independent controls.

---

Where to find what the brief asked for

the brief asks for	where it is
Deliverable 1 — Literature Summary, 2–3 pages	docs/LITERATURE.pdf · .md — 3 pages
Deliverable 2 — GitHub repository	this repository; layout below
Deliverable 3 — Technical Report, 2–4 pages	docs/REPORT.pdf · .md — 4 pages
AI usage log	AI_USAGE.md
§2.1 weak labels: ≥500 spans/class, 50+ papers, 1–2 domains, balance within 10%	data/spans/spans.jsonl — 1,705 per class, 300 papers, 2 domains, exactly balanced. Stats: data/spans/spans.stats.json
§2.1 span metadata: paper_id, section_name, label, text, token_count	all present, plus domain, section_type, rel_position, span_kind
§2.2 frozen embeddings, mean-pooled, no fine-tuning	src/embed.py — parameters checksummed before and after extraction, so frozen-ness is enforced
§2.3 the three conditions + macro-F1, confusion matrix, per-class P/R, accuracy	results/summary.md §4–6; report §4
§2.4 required length control	report §3.1 and §4.2; results/length_window_survival.json documents why the brief's 100–150 token window is infeasible here and what was done instead
§3.1 qualitative: 3 correct + 3 failed per class	results/qualitative_examples__scibert.md and __bge.md
§3.1 UMAP/t-SNE coloured by label	results/figures/umap.png; report §4.3
§3.2 "Is SLoD linearly decodable? What is your conclusion?"	report §6
§3.2 "how would you use this as a routing signal in a RAG system?"	report §6.1
§3.2 "What additional experiments would strengthen or refute this finding?"	report §6, final paragraph
the runnable command in Deliverable 2	python src/probe.py --train --eval --condition in_domain — works as written

Bonus items from the brief: UMAP visualisation ✅ · more than one embedding model compared ✅ (four, dimension-matched) · confound across domains and paper ages ✅ (docs/BONUS.md §1) · continuous SLoD regression ✅ (report §6, ρ = 0.609) · synthetic paired-data alternative ❌ not done, named as the first item of future work.

Work beyond the brief

docs/BONUS.pdf · .md — seven executed experiments that do not fit the page caps: the confound across domains and paper ages (with a pre-registered prediction that was falsified), a norm-versus- dimension ablation, a mean-pooling arithmetic check, a negative control outcome imported from epidemiology, auxiliary confound-decodability probes, a permutation sanity check on the headline metric, a document-provenance checker, and an abstraction-direction analysis showing the axis is not rank-1 (18–25 dimensions) yet two independently trained encoders order all 5,115 spans the same way on it (Spearman +0.709, random-direction null −0.203).

docs/appendix/ — process and provenance: the full 54-paper literature sweep, the pre-registered design specification, and an adversarial red-team of that specification which found four errors before they shipped. Not deliverables; read only if the process is of interest.

---

Quick start

# 1. environment (CPU-only is fine; no GPU required)
python -m venv .venv && source .venv/bin/activate
pip install torch --index-url https://download.pytorch.org/whl/cpu   # avoids ~2.5GB of CUDA wheels
pip install -r requirements.txt

# 2. credentials for the corpus (see "Data" below)
export S2_API_KEY=s2k-...

# 3. full pipeline
export PYTHONPATH=src
python src/s2orc_ingest.py --target-per-domain 150   # ~5 min, streams S2ORC shards
python src/dataset.py      --build                   # weak SLoD labels from structure

# encoders: two primaries, each with a floor at ITS OWN width, 5 seeds each
python src/embed.py --spans data/spans/spans.jsonl --model bge
python src/embed.py --spans data/spans/spans.jsonl --model scibert
python src/embed.py --spans data/spans/spans.jsonl --model random         --seeds 0 1 2 3 4
python src/embed.py --spans data/spans/spans.jsonl --model random_scibert --seeds 0 1 2 3 4

python src/balancing.py                              # entropy balancing + validity gate
python src/conditional_probe.py                      # THE HEADLINE: dV
python src/probe.py       --train --eval --all-conditions
python src/controls.py    --all --model bge
python src/controls.py    --all --model scibert      # every control, both primaries
python src/analysis.py    --all                      # figures, examples, ordinal regression
python src/report_tables.py                          # -> results/summary.md
python src/make_pdf.py --all --combined              # -> docs/*.pdf

The exact invocation named in the task specification works as given:

python src/probe.py --train --eval --condition in_domain

data/spans/spans.jsonl is committed, so every step after dataset.py runs without an API key. The ingest and label steps re-derive the corpus and need one — see Licensing below for why the full paper texts are not shipped.

---

Data

Why S2ORC via the Datasets API, and not the HuggingFace mirror

allenai/s2orc on the HuggingFace Hub is gated — it returns HTTP 401 without an approved access request. The Semantic Scholar Datasets API serves the same release as ~700 gzipped JSONL shards and only needs a free API key, so that is the route src/s2orc_ingest.py takes. Each record provides content.text plus content.annotations: character-offset spans marking title, abstract, sectionheader, paragraph, figurecaption and table. Those offsets are what the weak labelling scheme consumes.

Why both domains come from the same source

The cheaper option would be QASPER (ungated, clean, NLP-only) for the NLP domain and S2ORC for CV. That would confound the cross-domain condition: QASPER text is derived from LaTeX source, S2ORC text is GROBID's parse of a PDF. A cross-domain F1 drop would then be unattributable — domain shift, or parser shift? Drawing both domains from one release holds the extraction pipeline fixed so that only the research field varies.

Domain assignment

Each paper's arXiv primary category is resolved through the arXiv API; cs.CL becomes nlp and cs.CV becomes cv. This is an author-assigned, mutually exclusive label — cleaner than a coarse "fields of study" tag, and reproducible by anyone.

Getting a key

Free, from https://www.semanticscholar.org/product/api. Set S2_API_KEY, or put a line S2_API_Key: s2k-... in a keys.txt one directory above the repo (the path is git-ignored so the key is never committed).

Yield

One shard of the 2026-07-21 release contained 56,316 records, of which 6,899 passed the structural gate (English, has abstract + title, ≥4 section headers, ≥8 paragraphs, 8k–150k chars) and 560 were cs.CL or cs.CV. One shard is therefore enough for the 150 papers per domain used here; --max-shards raises the cap if a larger corpus is wanted.

---

Weak SLoD labels (SH0)

Label	Source
macro	title, abstract, first two introduction paragraphs, conclusion paragraphs
meso	section lead — first sentence of each non-intro / non-conclusion section
micro	non-lead paragraphs of methods / experiments / results sections

Four decisions worth knowing before reading any number:

1. meso and micro never share text. meso is the first sentence of a section's lead paragraph; micro is drawn only from non-lead paragraphs, and the rest of the lead paragraph is discarded. Assigning it to micro would make every meso span a literal prefix of a micro span from the same section.
2. The scheme has a length asymmetry built in. macro and micro spans are paragraphs; meso spans are single sentences. Length is not merely correlated with the label, it is partly definitional. See controls.py.
3. Titles are kept but tagged. A ~10-token title against ~200-token paragraphs is the most length-separable span in the corpus. It is included because the task specifies it, and span_kind == "title" allows the ablation.
4. Figure captions and tables are dropped. GROBID emits them as paragraph-like blocks; they concentrate in results sections and would hand the probe a trivial micro cue.

Classes are balanced exactly (not merely to within 10%), by round-robin over papers rather than uniform random sampling — a random subsample would let a few long papers dominate micro, so that "micro" would partly mean "written by these authors". Exact balance also fixes the majority-class baseline at 1/3 in every condition.

---

Experimental design

Every split is grouped by paper_id. Spans from one paper share topic, style and parse quality; a span-level random split would place a paper's abstract and its methods paragraphs on opposite sides of the train/test boundary and let the probe score by recognising the paper. --ungrouped-split reproduces the naive version so the report can quantify what leakage would have bought. Confidence intervals resample test papers, not test spans, for the same reason.

Condition	Train	Test
in_domain	5 grouped CV folds within one domain	held-out papers, same domain
cross_domain	the same training folds	the entire other domain
length_controlled	5 grouped CV folds on a length-controlled span set	held-out papers, same domain

Reusing the in-domain training folds for cross_domain keeps the training-set size identical across conditions, so a cross-domain drop cannot be blamed on having trained on more or less data.

Encoders (all frozen, mean-pooled, torch.no_grad(), weights checksummed before and after extraction): BAAI/bge-small-en-v1.5, sentence-transformers/all-MiniLM-L6-v2, allenai/scibert_scivocab_uncased, plus a randomly-initialised BGE as a floor.

Probes: multinomial logistic regression and linear SVM only. C is chosen by grouped inner CV on training folds; the scaler lives inside the pipeline so it never sees test-fold statistics.

---

Controls

src/controls.py implements the confound analysis. The task names one confound (length); on its own it is not sufficient to support the headline claim.

Check	Question it answers
length_only	What does a classifier given only token_count score?
length_head	Fixed 32-token window from the start of each span.
length_rand	Fixed 32-token window from a random offset — head-truncating a micro paragraph preserves its lead sentence, which is exactly what a meso span is, so head truncation makes micro look more meso.
length_matched	No truncation: spans subsampled so the three classes share a token-count distribution. Text stays natural, and all three classes survive at the same rate (24.1%).

Why the window is 32 and not the 100–150 the brief suggests. A meso span is defined as a section's first sentence, and single sentences are almost never that long — median meso length here is 29 tokens. At a 128-token window exactly 8 of 1244 meso spans survive (0.6%), against 673 macro and 466 micro; balancing then leaves 24 spans in total. Truncating to a fixed 100–150 token range does not control the length confound, it deletes the class most entangled with it. The full survival curve is written to results/length_window_survival.json so the choice is auditable; 32 tokens retains 745 spans per class. Because even that leaves an asymmetric selection effect (75% of macro, 41% of meso, 86% of micro survive), the report leans on length_matched instead, which is the only variant where all three classes survive at an identical rate. | tfidf | Does a sparse bag-of-ngrams model match the dense probe? If so the claim is about vocabulary, not representation geometry. | | masked | Digits, citations and discourse cues replaced by placeholders, then re-embedded. | | control_task | Hewitt & Liang (2019) selectivity, adapted to span level: random labels per span, and per paper. | | leakage | Grouped vs ungrouped split — the cost of getting the split wrong. | | paired | Within-paper macro-vs-micro pairs: topic, authors and parse held fixed, only level varies. | | position_only | What does rel_position alone score? |

src/annotate.py additionally tests the premise the whole study rests on — that document structure proxies abstraction level — by scoring a blind re-annotation of ~120 spans against the weak labels. Those annotations were produced by a language model, not a human, and the report says so wherever the number appears.

---

Repository layout

slod-probe/
├── README.md                  ← you are here
├── AI_USAGE.md                ← required AI-assistance log
├── requirements.txt
├── src/
│   ├── s2orc_ingest.py        corpus acquisition (S2 Datasets API + arXiv categories)
│   ├── dataset.py             weak SLoD label assignment
│   ├── embed.py               frozen embedding extraction + cache
│   ├── stats.py               ← paired bootstrap, permutation tests, Kish ESS
│   ├── balancing.py           ← entropy balancing, positivity, the validity gate
│   ├── conditional_probe.py   ← conditional V-information: the headline metric
│   ├── probe.py               linear probe, the three conditions
│   ├── controls.py            length / lexical / selectivity / leakage / paired controls,
│   │                          plus the negative-control LF, norm/dimension ablation,
│   │                          auxiliary confound probes and mean-pooling arithmetic
│   ├── analysis.py            error analysis, figures, UMAP, ordinal regression
│   ├── annotate.py            weak-label validation sheet + scoring
│   ├── report_tables.py       aggregates results/*.json into results/summary.md
│   ├── make_pdf.py            markdown -> self-contained PDF (no pandoc, no LaTeX)
│   └── utils.py               paths, IO, seeding, sentence splitting
├── scripts/run_all.sh         the whole experiment, in dependency order
├── tests/test_units.py        regression tests for the text-processing functions
├── data/
│   ├── raw/paper_ids.json     provenance for all 300 papers (committed)
│   ├── raw/papers.jsonl       full S2ORC texts (NOT committed — see Licensing)
│   └── spans/                 labelled spans + length/masked variants
├── embeddings/                cached .npy (regenerable; git-ignored)
├── results/                   metrics, confusion matrices, figures, examples
├── docs/
│   ├── LITERATURE.md          literature summary
│   ├── REPORT.md              technical report
│   └── papers/                the papers cited, for reference
└── notebooks/analysis.ipynb   embedding-space exploration

src/s2orc_ingest.py is an addition to the layout given in the task specification: keeping HTTP/streaming/credential code out of dataset.py leaves that module about labelling only.

Licensing and what is committed

S2ORC records carry per-paper open-access licences which this pipeline does not retain, so the repository does not redistribute the 300 full paper texts (data/raw/papers.jsonl, ~16 MB, git-ignored). Committed instead:

file	what	why
data/raw/paper_ids.json	every corpusid, arXiv id, primary category and title	factual provenance; lets anyone re-derive the identical corpus
data/spans/spans.jsonl	the 3,732 labelled spans	short excerpts; every experiment runs from these, so omitting them would make the results unreproducible

The cited papers in docs/papers/ are likewise not committed — run bash scripts/fetch_papers.sh to download them from arXiv.

To rebuild the full corpus:

export S2_API_KEY=s2k-...
python src/s2orc_ingest.py --target-per-domain 150

The scan is deterministic given the same release, and paper_ids.json records exactly which papers this run used, so any divergence is detectable.

Tests

PYTHONPATH=src python tests/test_units.py

No test framework required. The suite covers the pure text-processing functions where a silent bug changes the data rather than raising — sentence splitting, section-name normalisation, cue masking, token-window truncation and class balancing. Every case is a regression test for a real defect or a real corpus pattern; the two sentence-splitter bugs documented in docs/REPORT.md §2.3 survived into a complete experimental run before spot-checking caught them, which is what these tests exist to prevent.

Reproducibility

Everything stochastic (splits, subsampling, random truncation windows, control labels, UMAP) derives from utils.SEED = 20260727. Embeddings are cached under embeddings/ and keyed by a SHA-256 of the span texts, so a rebuilt spans file invalidates its cache instead of silently reusing stale vectors.

Measured runtime on 8 CPU cores with no GPU, for the 3,732-span corpus:

stage	time
s2orc_ingest.py	~4 min shard scan + ~4 min arXiv category resolution
dataset.py --build	~5 s
embed.py — bge-small / MiniLM-L6 / random-init	~3.5 / ~2 / ~4 min
embed.py — SciBERT (110M params)	~8 min
probe.py --all-conditions	~8 min
controls.py --all	~25 min (includes embedding four span variants)
analysis.py --all	~5 min

A GPU is not required; the brief's estimate of ~2 h on CPU for 1,500 spans is conservative by roughly an order of magnitude for encoders this size. The dominant cost is controls.py, because each length variant and the masked variant is a separate corpus that has to be embedded.