fix(mentions): resolve @ mentions followed by punctuation or naming paths with spaces

[pjdoland]

Summary

This is the nui half of plmbr/notebook-intelligence#503, where we agreed to bring @ mentions in line with Claude Code and Codex and to update nui to match. The Chatbook side is plmbr/notebook-intelligence#505.

Before this change, a mention ran to the next whitespace. So Summarize @file:notes.md, then chart it looked up notes.md,, and a path with a space could not be named at all, even though the mention menu offers such paths. Either way the token reached the agent unresolved.

What changes

• Quoted mentions. @file:"data/my notes.md" (and @dir:"...", @ext:"...") names a value containing whitespace or @. The value is taken exactly as written, leading or trailing whitespace included, and cannot hold a double quote or a line break. A closing quote glued to a word character (@file:"a"b.csv) keeps the old single-token reading.
• Trailing punctuation. Sentence punctuation typed after an unquoted file or dir mention is not part of the path: . , ; : ! ? …, closing brackets and quotes, and their full-width forms. The name as written is tried first, then with up to five of those characters removed one at a time, and the first name that exists wins. A real name ending in punctuation (notes.) still resolves, and a shorter name is never reached past one that exists. foo.c++ is never trimmed to foo.c. Paths outside the working directory end the search. The trimmed punctuation is written back after the resolved path, so @file:notes.md, becomes @/abs/notes.md, and the sentence is unchanged.
• ext: values accept the quoted form but are never trimmed. Provider values are opaque, so trailing characters may belong to them.
• Output for the agent. A resolved path containing whitespace is emitted as @"/abs/path", the form Claude Code reads. Other paths are unchanged (@/abs/path).
• Mention menu. useMentionMenu inserts the quoted form when a value needs it (ui/src/lib/mentionToken.ts).

Tokens are found with a small rune scanner (internal/mentions/tokens.go) rather than a regexp. RE2 has no lookaround, and its \s is ASCII-only, while Chatbook's Python tokenizer is Unicode-aware; the scanner matches Chatbook's behavior. A mention still has to follow whitespace, which is also what Claude Code requires of the @path mentions nui hands it. That check is now done by rune rather than by byte, so a full-width space before @ counts.

Testing

• internal/mentions/tokens_test.go covers:
  • each punctuation character;
  • one-at-a-time trimming (@dir:data(2)). resolves data(2));
  • the name as written winning;
  • never reaching a different file (foo.c++, an existing data. folder, too much punctuation);
  • trimming staying inside the working directory;
  • quoted paths with spaces and @;
  • quoted values staying exact ("notes.md ");
  • malformed quotes;
  • the whitespace-before-@ rule;
  • ext: values quoted but never trimmed.
• ui/src/lib/mentionToken.test.ts covers the quoting rules.
• Mutation-checked: the tests fail if trimming runs before the exact name, if a spaced path is emitted unquoted, if the whitespace rule is loosened, or if quoted values are trimmed again.
  • One mutation cannot be caught: continuing past a path outside the working directory instead of stopping. Trimming punctuation from the last segment can never turn an outside path into an inside one, so both behave the same; the stop is there to match Chatbook.
• Gates, as in CI: go vet ./internal/mentions/ and go test . ./cmd/... ./internal/... (Go 1.27.1 locally). The UI passes npm run lint, npm run build and npm test: 36 files, 229 tests.
• One Go test fails locally on untouched main as well: TestOrchestratorDefinitionAPIHasNoCLIAllowlist. It expects the api/anthropic harness, which is unavailable in a shell with no ANTHROPIC_API_KEY and the Claude CLI installed. main's CI is green, so it only depends on the local environment.

Not changed here

• Pasted absolute-path mentions keep their current handling. A resolved absolute mention renders back as the same @/abs/path, so trimming punctuation from it would change nothing the agent sees.
• Absolute paths containing whitespace, such as an upload under a Windows temp directory with a space in the username, still cannot be written, because the quoted form covers only file:, dir: and ext:. That was already the case; adding a bare @"..." form is worth a separate decision.
• A path containing both whitespace and a double quote cannot be quoted in either nui or Claude Code's form.