Reproduce "Proactive Thought Partners" (arXiv:2609.01588) as a lab page

[kcarnold]

An attempt at reproducing Zhang, Davis, Chen & Hsu, Designing Proactive Thought Partners for Writing (arXiv:2609.01588v1) inside this add-in, plus an honest log of where the reproduction breaks down.

What the paper built

A Next.js app with its own BlockNote editor. Writers configure "partners" (name + emoji, role, event triggers, contextual heuristic); as they write, keystroke-level triggers fire, an LLM decision engine picks at most two whose heuristics match, and each produces an acknowledgement plus a question-style suggestion. Writers can ignore it, open it as inspiration, or have it edit the document.

What is here

A Partners lab page (frontend/src/pages/partners/), reachable from the Labs (···) menu. Configure partners, switch watching on, and tags appear when a heuristic is satisfied; clicking one opens the suggestion and a short follow-up conversation; ignoring one lets it fade after fifteen seconds.

Paper §4	Here
Partner customization	✅
Long Pause / Sentence End / Text Selection triggers	✅ from polled snapshots, not keystrokes
Session goal	✅ as the existing document brief
Keystroke log fed to the decision engine	⚠️ coarse activity trace instead
Decision engine, at most two partners	✅ one model instead of two
Acknowledgement + question	✅ generated on click, not on activation
Tag aligned with the cursor	⚠️ fixed position in the panel
Ignoring / Inspiring	✅
Executing — partner writes into the document	❌ deliberately not built

docs/proactive-partners-reproduction.md is the map and the challenge log.

The interesting problems

There are no keystrokes. The paper's triggers run on a live keystroke stream inside its own editor. A task pane's writer types into Word or Google Docs; Office.js offers only DocumentSelectionChanged, and Google Docs offers no selection event at all. The triggers are rebuilt from diffs of polled getDocContext() snapshots, and the 15-second keystroke log becomes a much coarser activity trace. Burst/pause rhythm, backspace runs, and the rest of what the keystroke-analysis literature the paper cites depends on are not observable here.

"Execute" is deliberately not implemented. docs/design/interface-concepts.md opens its covenant with "the writer's sentences are the writer's", and frontend/CLAUDE.md states that nothing in the add-in rewrites the writer's prose. This is the one place where a faithful reproduction and this project's stated values genuinely conflict, so it is flagged rather than quietly resolved — and it means this build cannot speak to the paper's findings about Executing. The machinery exists (EditorAPI.applyEdit), so it is a decision to revisit, not a capability gap.

The paper's prompts are not in the PDF. §4.5 says they are in supplementary materials; the arXiv appendix is references only. Every prompt here is reconstructed from prose, so a behavioural difference could be a prompt difference we cannot detect.

Two problems only running it found

Both are consequences of polling rather than listening, so they are evidence about the compromise rather than ordinary bugs.

An unwatched trigger was spending the cooldown. A writer whose only partner listened for Long pause received nothing, ever: every sentence they finished fired a Sentence end that no partner listened for, which produced no activation but started the 45-second cooldown and suppressed the pause behind it. The feature looked dead while every part of it worked. Fixed, with a regression test — but the general point stands and shapes any study run on this build: the cooldown is a scarce resource the noisiest trigger wins by default, so a writer with one sentence-end partner and one pause partner is not running two partners at equal odds. The paper has no cooldown and no such effect.

A polled observer has no history from before it starts. Its first snapshot is indistinguishable from "text just typed", so a writer who switched watching on and immediately typed a sentence and stopped got no pause — the burst had been absorbed into the baseline. Mitigated by sampling at once rather than on the first interval; the blind window shrinks to one host read but cannot close.

Verification

frontend/tests/partners-flows.spec.ts drives the whole loop in a browser — configure a partner, type, stop, tag appears, open it, get the suggestion and a follow-up. That spec is what found both problems above. 45 unit tests cover the state machine and the two response parsers. Typecheck, lint, and the full 284-test suite pass.

Not verified: any real host. Everything ran against the standalone Lexical editor, where reading the document is a function call. On Word each poll is a Word.run sync and on Google Docs an Apps Script round-trip; whether a 1.5-second poll is tolerable there is an open question.

🤖 Generated with Claude Code

https://claude.ai/code/session_01LzPdByDv9KAZyJd6kmTYpR