Skip to content

feat(core): Tokenizer fits the pointer — a chip row and a sheet on a finger - #5315

Draft
imdreamrunner wants to merge 2 commits into
mainfrom
feat/tokenizer-touch
Draft

feat(core): Tokenizer fits the pointer — a chip row and a sheet on a finger#5315
imdreamrunner wants to merge 2 commits into
mainfrom
feat/tokenizer-touch

Conversation

@imdreamrunner

Copy link
Copy Markdown
Contributor

Builds the Tokenizer mobile prototype into Tokenizer itself, as a second surface the component picks from the primary pointer — not a parallel component to adopt.

Why the pointer control cannot just be made bigger

Both of its core gestures need a hardware keyboard. You type between the chips, in an input sharing their line, and you remove the last one with Backspace on an empty input. On a phone, focusing that input raises the virtual keyboard over the bottom half of the screen — where the suggestion popover opens — and each chip added grows the field by a line and pushes the page under your thumb.

The touch surface

Chips scroll sideways The field is exactly one line tall however many there are, so adding and removing never reflow the form below it.
Add is a fixed target Outside the scroller at the trailing edge: same place with two chips or twenty, and a tap on it can never be mistaken for the start of a sideways drag. (The prototype has it inside the scroller, where it scrolls away once the chips overflow — the one deliberate departure.)
Suggestions are a pinned-tall sheet Search at the top where the keyboard cannot cover it, full-width rows a thumb can hit. tall is the one BottomSheet height that is keyboard-aware. The list populates before anything is typed: in a sheet the list is the content.

Tapping a row adds that token and leaves the sheet up, so building a set of five is five taps.

Nothing changes at the call site — same props, same values, no new import, no media query to write. The switch is pointer: coarse alone with no width bound, the same test and the same reasoning as the DateInput touch surface: pointer is the PRIMARY device, so a touchscreen laptop keeps the typable field and a narrowed desktop window is still a mouse, while a tablet gets the sheet at any width.

Notable

  • One selection engine, not two. useTokenSelection holds what a token is — including the Create "X" sentinel, a synthetic item recognised on the way back in and turned into a real one. That is the part that must not fork. The pointer surface's rendered output is unchanged; its diff is the extraction and a rename to PointerTokenizerField.
  • Two bugs only a browser could show, both now covered by a test:
    • The sheet's sticky header sits between two layers in one stacking context. A bare z-index: 1 put it over BottomSheet's grab handle and hid the pill; no z-index at all let List's position: relative rows paint straight through its background. It needs an isolated wrapper to scope the layer.
    • minBlockSize compiles to min-height, so a 44px thumb floor declared next to the size scale is the same property and simply loses the merge. Folded into the scale as max(44px, <size>).
  • Cost: 16.4 KB gzipped on top of Tokenizer's 58.6 KB — mostly BottomSheet and List — for every consumer including desktop-only ones, since the choice is made at runtime. (Measured: bundle the component alone, minified, from dist before and after.)
  • New public API is one export, TokenizerTouchSurface (the touch half with the pointer test skipped, so the stories are reviewable on a laptop), plus three @astryx.tokenizer.* catalog keys. No media-query constant: feat(core)!: DateInput fits the pointer — a touch picker on a finger, the text field on a mouse #5243's DateInput surface already publishes TOUCH_POINTER_QUERY, and two of those in the root barrel would collide.
  • The plus glyph is drawn in the component rather than registered as an icon name — structural, the way CheckboxIndicator draws its own tick; registering a name would put all seven bundled themes on the hook for an icon.

Test plan

  • pnpm lint:strict clean (0 errors; the 53 warnings are pre-existing on main), pnpm build, pnpm test — 7,530 core+lab and 2,801 cli+apps tests pass.
  • CI's extra typechecks: core/lab/charts typecheck:docs, cli typecheck:strict + typecheck:template-docs, storybook typecheck + build, lab:readiness:check.
  • 26 new tests in TokenizerTouch.test.tsx covering the surface switch (including that width alone does not switch it, and that a tablet does), the closed field's contract, the sheet, hasCreate, maxEntries, and the disabled-with-a-reason path. The existing Tokenizer.test.tsx is untouched and still passes — the shared setup answers (pointer: coarse) false, so it keeps testing the pointer surface.
  • Verified in a browser (Chromium, iPhone 15 profile, built dist): the row scrolls (scrollWidth 540 vs clientWidth 253) while the field stays 52px and does not move; the sheet opens at 92dvh with the pill visible above the header; the header covers rows scrolling under it; tapping the search field through it still focuses it.

Storybook: Core/Tokenizer/Touch surface — the first story reports which surface you are actually looking at; the rest force the touch half.

…finger

Adds a second surface to `Tokenizer`, chosen at runtime from the primary
pointer, so a phone gets a control shaped for a thumb instead of one shaped for
a keyboard.

The pointer control needs a keyboard for both of its core gestures. You type
BETWEEN the chips, in an input sharing their line, and you remove the last one
with Backspace on an empty input. On a phone, focusing that input raises the
virtual keyboard over the bottom half of the screen — where the suggestion
popover opens — and each chip added grows the field by a line and pushes the
page under your thumb.

The touch surface is three ideas:

- The chips scroll sideways instead of wrapping, so the field is exactly one
  line tall however many there are and nothing below it moves.
- Add is a separate, fixed target OUTSIDE the scroller, at the trailing edge:
  same place with two chips or twenty, and a tap on it can never be mistaken
  for the start of a sideways drag. (The prototype had it inside the scroller,
  where it scrolls away once the chips overflow.)
- Suggestions are a pinned-tall sheet: search at the top where the keyboard
  cannot cover it, full-width rows a thumb can hit, and `tall` is the one
  BottomSheet height that is keyboard-aware. The list populates before anything
  is typed — in a sheet the list IS the content, so `hasEntriesOnFocus`, which
  is about not putting a popover over the page unbidden, does not apply.

Built from the mobile prototype at sandbox/pages/mobile-prototypes/?p=tokenizer.

The switch is `pointer: coarse` alone, with no width bound — same test, and same
reasoning, as the DateInput touch surface: `pointer` is the PRIMARY device, so a
touchscreen laptop keeps the typable field and a narrowed desktop window is
still a mouse, while a tablet gets the sheet at any width.

The selection logic is now one hook, `useTokenSelection`, rather than two
copies. That is what keeps the "Create X" sentinel — a synthetic item that has
to be recognised on the way back in and turned into a real one — from forking
between the surfaces. The pointer surface's rendered output is unchanged; its
diff is the imports, the extraction, and a rename to `PointerTokenizerField`.

Two things a browser caught that jsdom could not, both now covered by a test:

- The sheet's sticky header sits between two layers in ONE stacking context. A
  bare `z-index: 1` put it over BottomSheet's grab handle and hid the pill; no
  z-index at all let List's `position: relative` rows paint straight through
  its background. It needs an isolated wrapper to scope the layer.
- `minBlockSize` compiles to `min-height`, so a 44px floor declared next to the
  size scale is the same property and simply loses the merge. The floor is
  folded into the scale as `max(44px, <size>)`.

Costs 16.4 KB gzipped on top of Tokenizer's 58.6 KB (measured: bundle the
component alone, minified, from dist before and after), for every consumer
including desktop-only ones, since the choice is made at runtime.

New public API is one export, `TokenizerTouchSurface` — the touch half with the
pointer test skipped, so the Storybook stories are reviewable on a laptop — plus
three `@astryx.tokenizer.*` catalog keys. No media-query constant is exported:
the DateInput surface already publishes `TOUCH_POINTER_QUERY`, and two of those
in the root barrel would collide.

Verified on an iPhone 15 profile in Chromium: the row scrolls (scrollWidth 540
vs clientWidth 253) while the field stays 52px and does not move, the sheet
opens at 92dvh with the pill visible over the header, the header covers rows
scrolling under it, and tapping the search field through it still focuses it.
@meta-cla meta-cla Bot added the CLA Signed This label is managed by the Meta Open Source bot. label Aug 22, 2026
@vercel

vercel Bot commented Aug 22, 2026

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated (UTC)
astryx Ready Ready Preview Aug 22, 2026 7:12am

Request Review

@github-actions

github-actions Bot commented Aug 22, 2026

Copy link
Copy Markdown
Contributor

PR Analysis Report

📚 Storybook Preview

View Storybook for this PR
GitHub Pages may take up to a minute to hydrate after deploy.

🧪 Sandbox Preview

View Sandbox for this PR
GitHub Pages may take up to a minute to hydrate after deploy.

Modified Components

Tokenizer (@astryxdesign/core) · View in Storybook
Metric Before After Delta
Bundle Size (ESM) N/A N/A N/A
Lines of Code N/A 1280 -
Complexity N/A Very High (162) -

Bundle Size Summary

Package Size (ESM) Size (CJS) Gzipped
@astryxdesign/core N/A 4.8KB 1.2KB

Accessibility Audit

Status: 1 accessibility violation(s) found — 1 serious.

Tokenizer - 1 issue(s)
  • 🟠 serious: Ensure the contrast between foreground and background colors meets WCAG 2 AA minimum contrast ratio thresholds
    • Rule: color-contrast · Affects 3/29 stories · Learn more
    • WCAG: 1.4.3 (Level AA)

Generated by PR Enrichment workflow | Storybook | Sandbox | View full report

github-actions Bot added a commit that referenced this pull request Aug 22, 2026
@imdreamrunner
imdreamrunner marked this pull request as draft August 22, 2026 07:00
…nt of their own

`pr-a11y` failed, and the interesting part is why: the story title
`Core/Tokenizer/Touch surface` is the only three-segment title in the repo, and
the a11y tooling reads those two ends differently. The audit FILTERS by
`title.split('/')[1]` — `Tokenizer`, so `--components Tokenizer` found the
stories — but KEYS its report by `title.split('/').pop()`, so every violation
was filed under a component named "Touch surface", which does not exist. The
weekly a11y summary is built from the same report, so it would have grown a
phantom component too.

Folded the eight stories into `Tokenizer.stories.tsx` behind a section comment
instead. Two files can share a title (`Vega/VegaChart` does), but one file per
component is the repo's shape and leaves nothing to reason about.

That leaves one real violation, now correctly keyed
`Tokenizer::Touch: disabled, with a reason::color-contrast`, and it is the
disabled treatment the whole system already has rather than anything this
surface does. Measured, the three nodes are the field label at 2.52 and the two
chips at 1.72 — the same nodes and the same ratios as the already-baselined
`Tokenizer::Disabled` and `Tokenizer::Disabled With Message`. Both surfaces
compose the same two dims: `inputWrapperStyles.disabled` puts `opacity: 0.5` on
the field and `Token` adds its own, so a chip is at 0.25 effective on the
pointer surface today (verified in the built Storybook: `core-tokenizer--disabled`
reports the identical 0.25). Fixing it means changing what disabled looks like
system-wide, which is not this PR, and WCAG 1.4.3 exempts inactive controls. So
it joins the two entries beside it in the baseline.

The disabled Add button lands at the same 0.25 as the chips around it, which is
why axe does not flag it separately and why it should not be exempted from the
field's dim — it would be the one bright thing in a disabled field.

Verified: a11y gate passes (0 new, 3 baselined), RTL audit passes, the modal
close-visibility guard passes, storybook typecheck and build pass, eslint clean.
@imdreamrunner

Copy link
Copy Markdown
Contributor Author

Follow-up commit: pr-a11y was red, and the cause was worth writing down.

The story title broke the tooling. Core/Tokenizer/Touch surface was the only three-segment story title in the repo, and the a11y script reads the two ends of it differently — it filters by title.split("/")[1] (Tokenizer, so the stories were found) but keys its report by title.split("/").pop(), so every violation was filed under a component called "Touch surface" that does not exist. The weekly a11y summary is built from the same report, so it would have grown a phantom component too. Folded the eight stories into Tokenizer.stories.tsx behind a section comment; two files can share a title (Vega/VegaChart does) but one file per component is the repo shape.

The remaining violation is the system-wide disabled treatment, not this surface. Measured, the three nodes on Touch: disabled, with a reason are the field label at 2.52 and the two chips at 1.72 — the same nodes and the same ratios as the already-baselined Tokenizer::Disabled and Tokenizer::Disabled With Message. Both surfaces compose the same two dims: inputWrapperStyles.disabled puts opacity: 0.5 on the field and Token adds its own, so a chip is at 0.25 effective on the pointer surface today — verified against the built Storybook, core-tokenizer--disabled reports the identical 0.25. Changing that means changing what disabled looks like system-wide, which is not this PR, and WCAG 1.4.3 exempts inactive controls. So it joins the two entries beside it in the baseline rather than being fixed here.

The disabled Add button sits at that same 0.25, which is why axe does not flag it separately — and why it should not be exempted from the field dim: it would be the one bright thing in a disabled field.

Also worth flagging for #5243: the DateInput touch branch uses the same Core/DateInput/Touch surface title and will hit the same keying bug when it runs pr-a11y.

Locally: a11y gate passes (0 new, 3 baselined), RTL audit, modal close-visibility guard, storybook typecheck + build, eslint all clean.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

CLA Signed This label is managed by the Meta Open Source bot. needs:design-review Affects visuals — Design should review

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant