Detect AI-generated academic papers by the geometry of their sentence-window embedding trajectory. The detector uses a single feature — F4, the fraction of consecutive embedding step-pairs with negative cosine — together with a logistic regression calibrated for FPR ≤ 5%.
This is the reference implementation for the paper: "Trajectory Geometry Detects AI-Authored Papers" (working title).
Each paper's body is concatenated in canonical IMRaD order
(Abstract → Introduction → Methods → Results → Discussion → Conclusion),
split into sentences, and embedded in 3-sentence sliding windows (stride 1)
with qwen3-embedding:8b. From the window trajectory E_1, …, E_N ∈ R^4096
we compute step vectors B_k = E_{k+1} − E_k, unit-normalize them, then
measure the cosine between consecutive steps and report
F4 = fraction of
cos(B_k, B_{k+1}) < 0.
AI-authored papers tend to produce smoother trajectories (fewer
backward-turning steps) than human-authored ones. A logistic regression on F4
maps this to p(AI), and a threshold of 0.5244 (validation-calibrated for
FPR ≤ 5%) yields the binary verdict.
git clone https://github.com/faltland/trajectory-geometry-detector.git
cd trajectory-geometry-detector
pip install -r requirements.txtYou will also need Ollama running locally with the embedding model pulled:
ollama pull qwen3-embedding:8b
ollama serve # default port 11434The model is ~8 GB. Any port works; pass --port to detect.py if you run
Ollama somewhere else.
The first run will auto-download the NLTK punkt tokenizer (a few MB).
Two commands:
# 1. Detect on a single paper (any folder with section .txt files)
python detect.py example/genuine/hist_0185/
# 2. Detect on an AI-generated sample
python detect.py example/ai/lla_13/For batch processing, loop in your shell:
for d in path/to/papers/*/; do python detect.py "$d"; doneRunning python detect.py example/genuine/hist_0185/ --port 11445 --verbose on
the bundled known-genuine paper (185 windows, qwen3-embedding:8b on a P40 GPU,
≈ 81 s for the full embedding pass):
[detect] sections present: ['Abstract', 'Introduction', 'Methods', 'Results', 'Discussion', 'Conclusion']
[detect] n_sentences: 187
[detect] embedded 50/185 windows (29.5s)
[detect] embedded 100/185 windows (48.0s)
[detect] embedded 150/185 windows (67.8s)
[detect] embedding done: 185 windows in 81.1s
{
"paper": "example/genuine/hist_0185",
"verdict": "GENUINE",
"p_ai": 0.001407,
"p_genuine": 0.998593,
"F4": 0.52459,
"threshold": 0.5244014101938493,
"n_windows": 185,
"n_sentences": 187,
"sections_present": [
"Abstract", "Introduction", "Methods",
"Results", "Discussion", "Conclusion"
],
"embed_model": "qwen3-embedding:8b",
"model_path": "models/f4_logistic.pkl"
}
p_ai = 0.0014 is well below the 0.5244 decision threshold → GENUINE.
F4 = 0.5246 is right at the decision boundary, which is typical for
genuine papers (the AI-genuine separation in F4-space happens around 0.52).
The bundled model models/f4_logistic.pkl was trained on the paper's full
corpus (480 genuine + 198 AI papers, stratified 60/20/20 split, seed 42).
Reported test-set performance:
| Metric | Value |
|---|---|
| AUC | 0.922 |
| Threshold (val-calibrated) | 0.5244 |
| Test FPR | 2.08% |
| Test Recall | 70.0% |
| Balanced accuracy | 0.84 |
To re-train from scratch on the same kind of data:
python train.py \
--genuine path/to/genuine_papers/ \
--ai path/to/ai_papers/ \
--out models/f4_logistic.pkltrain.py performs the same 60/20/20 stratified split (seed 42), calibrates the
threshold on validation, and writes both f4_logistic.pkl and a sibling
.meta.json with the new threshold and test metrics.
Wild-test re-evaluation. When applied to 845 papers from outside the training distribution (mixed PMC sample, post-2024), the same model with threshold 0.5244 flags 10.06%. Length-stratified (papers with n_windows ≥ 188, ≈ 4,500 words), the rate drops to 6.96% — versus a Test-Genuine FPR estimate ≤ 6.5% (Wilson 95% upper bound) at the same length. The net AI signal on long papers is therefore weak but positive (≈ +0.5 to +7 percentage points). The short-paper end carries a length confound that is not yet corrected for.
| Constant | Value |
|---|---|
| Section order | Abstract → Introduction → Methods → Results → Discussion → Conclusion |
| Window size | 3 sentences |
| Window stride | 1 sentence |
| Embedder | qwen3-embedding:8b (4096-d) |
| Threshold | 0.5244 (validation-calibrated) |
Changing the embedder (e.g. to BGE-M3) shifts the F4 distribution and
requires a new threshold and likely a re-trained model. train.py handles
this automatically when given a different --embed-model.
trajectory-geometry-detector/
├── detect.py # inference CLI
├── train.py # train on your own corpus
├── models/
│ ├── f4_logistic.pkl # logistic regression, scikit-learn 1.8.0 pickle
│ └── f4_logistic.meta.json
├── example/
│ ├── genuine/ # 5 known-genuine papers (section .txt folders)
│ ├── ai/ # 5 known-AI papers
│ └── MANIFEST.json
├── requirements.txt
├── LICENSE
└── README.md
A folder qualifies as "a paper" if it contains at least one of the six section
.txt files. Sections may be missing; the pipeline concatenates what it
finds. Minimum requirement: enough sentences to produce ≥ 4 windows (i.e.
≥ 6 sentences total).
- Embedder lock-in. All numbers are conditional on
qwen3-embedding:8b. Other embedders give other F4 distributions. - Length confound. F4 has a residual dependence on document length; short papers (~1500 words / < 100 sentences) are over-flagged. The paper reports length-stratified results; the bundled threshold is unaware of this.
- Single feature. F4 alone reaches AUC ≈ 0.92 on the paper's test set, but generalises imperfectly to out-of-distribution corpora — wild-test rates drop substantially when controlled for length.
- Static threshold. No online recalibration; threshold is fixed at training time.
@article{koerber2026trajectory,
title = {Trajectory Geometry Detects AI-Authored Papers},
author = {Koerber, Hanno},
affiliation = {Independent Researcher, Berlin, Germany},
year = {2026},
arXiv = {forthcoming}
}MIT — see LICENSE.