fix(chatbook): resolve @ mentions followed by punctuation or naming paths with spaces

[pjdoland]

Summary

Chatbook @ mentions dropped the referenced file in two common cases. Punctuation typed right after a mention, as in Using @file:data/README.md, convert..., became part of the path. And a path containing a space, which the mention menu itself offers, could not be expressed at all. In both cases the model received the mention as [unavailable] and generated code anyway, with nothing shown to the user. This follows the direction agreed on #503: match how Claude Code and Codex treat @ paths.

Solution

Server (notebook_intelligence/chatbook_mentions.py)

• Quoted form. @file:"data/my notes.md", @dir:"My Folder", and @ext:"provider:value" name a value containing whitespace or @, following Claude Code's @"path" and Codex's habit of quoting a picked path that contains whitespace. A quoted value is taken literally. It cannot hold a double quote or a line break, and a closing quote glued to a word character (@file:"a"b.csv) keeps the old single-token reading.

• Trailing punctuation. Sentence punctuation typed after an unquoted file or dir mention is not part of the path. That means . , ; : ! ? …, closing brackets and quotes, and their full-width forms. Resolution tries the name as written, then with up to five of those characters removed one at a time, and stops at the first name that exists, even if it cannot be read. So:

  • a file whose real name ends in punctuation still resolves (notes. is found before notes);
  • a shorter, unrelated name is never reached past one that exists (a binary data. does not fall through to a text file data);
  • +, -, combining marks, and invisible or bidi characters are never trimmed. foo.c++ stays foo.c++, and a trailing U+202E is refused rather than stripped;
  • every candidate still goes through _safe_relative_path, so trimming cannot reach a dotfile, a skipped directory, or anything outside the root.

  This is deliberately narrower than Claude Code's \b, which backtracks off any non-word character.

• ext: values are quoted-capable but never trimmed. The extension docs describe provider values as opaque, and the example provider returns "" for an unknown value, so there is no reliable miss signal. Trimming could quietly resolve a different item. docs/chatbook-extensions.md now says this.

• Duplicates are sent once. Mentions are deduplicated by resolved path, or by provider value for ext:, before any file is read or provider is called. So @file:big.csv, … @file:"big.csv" reads the file once.

Frontend (src/chatbook-mentions.ts)

• The menu inserts the quoted form when the value contains whitespace, @, or U+001C to U+001F (which Python's \s also treats as separators). It leaves alone values it cannot quote.
• select() and applyChatbookMention now share chatbookMentionInsertText, and the caret is placed from the inserted text's length. The previous item.value.length + 1 + … arithmetic would land two characters short inside a quoted token.
• CHATBOOK_MENTION_TOKEN_RE is unchanged. It only answers whether a prompt has any mention, and every quoted token already matches it. Its comment now says so.

The existing test that pinned @dir:docs, as unavailable ("NUI-compatible tokens") now expects it to resolve, per the #503 decision.

Testing

• Python: new tests cover:
  • each punctuation character, including …, .”, and 。;
  • progressive trimming ((@dir:data(2)) finds data(2));
  • the five-character cap;
  • the name as written winning over the trimmed one;
  • never reaching a different file (foo.c++, NFD é,, binary data.);
  • path safety under trimming (dotfiles, skipped dirs, .., bidi and zero-width suffixes);
  • quoted paths with spaces and @;
  • unreadable files reporting the real name;
  • dedupe, including that a repeated mention is read once, that a refused mention does not hide a real sibling, and that ext: quoted and unquoted call the provider once;
  • linear-time behavior on a 200,000-character punctuation run.
• Jest: quoting rules (@, whitespace, the U+001C to U+001F separators, and the quote and line-break exclusions), plus a menu test that picks a spaced path through the real EditorView and asserts both the inserted text and the caret position.
• Mutation-checked: every new test fails against the bug it targets. That includes a greedy \W+$ trim, trimming before the exact name, skipping the safety check on a trimmed candidate, the old caret arithmetic, keying a refused mention on its trimmed name, and dropping the ext: dedupe. The linear-time test hits pytest-timeout against a regex trim.
• Gates: pytest tests/ -q (2181 passed, plus 77 in test_claude_client.py), jlpm tsc --noEmit, jlpm lint:check, jlpm jest (595 passed).
• Live, in JupyterLab with this branch built: a stand-in OpenAI-compatible server echoed back the MENTION_CONTEXT it received. Through the real menu I picked data/my notes.md and logo@2x.txt, which were inserted as @file:"data/my notes.md" and @file:"logo@2x.txt", with text typed afterwards landing after the closing quote. I then wrote @file:data/README.md. Done. All three reached the model with the right path and the file's contents.
• Two review rounds (a correctness review plus parser and path-safety, CodeMirror, and contract and test reviewers) reshaped the first implementation. It trimmed with \W+$, which was quadratic on long punctuation runs, stripped combining marks and bidi characters, and could fall through to a different file. The punctuation set, one-at-a-time trimming, the "first existing name wins" rule, and the pre-read dedupe all came out of that review.

Risks / follow-ups

• NUI's matching change is fix(mentions): resolve @ mentions followed by punctuation or naming paths with spaces nui#6. It brings nui's tokenizer to the same quoting and punctuation rules. nui keeps requiring whitespace before @, because Claude Code, which receives nui's rewritten @path mentions, requires the same.
• Behavior change for saved prompts. A prompt like @file:data.csv, used to send [unavailable] and now sends the file, so regenerated code can differ. That is the intended fix.
• Pathological names. A name that itself begins and ends with a double quote cannot be mentioned, since the quoted form cannot hold a quote. docs/chatbook.md says so.
• Pre-existing, not changed here:
  • An in-root symlink pointing at a dotfile is followed.
  • A refused first candidate ends the search, so @dir:data/... is not trimmed back to data/. That is conservative, and rare.
  • Running any Chatbook cell logs an uncaught No active debugger session from JupyterLab's debugger. I reproduced this on released 6.0.0 with a plain prompt, no mentions, and filed it as Running a Chatbook cell throws an uncaught "No active debugger session" error #507.
  • There is no cap on the number of mentions per prompt.
• CHANGELOG.md is untouched, since entries are curated at release and its top section still reads [6.0.0] - unreleased.

Screenshots

The confirm bar from the live run, showing which mentions the stand-in model received, is available to add here.

Closes #503