fix: preserve grapheme clusters (combining marks) in cell emission by natemoo-re · Pull Request #114 · bombshell-dev/tty

natemoo-re · 2026-08-23T03:35:32Z

cells with combining marks (zero-width codepoints) were silently dropped when emitted because render_text skipped codepoints with cw == 0. the cell model stored only a single uint32_t, meaning combining accents, ZWJ emoji, skin-tone modifiers, flag variation emoji, and kitty graphics were broken.

the fix here is to update Cell's model to track uint32_t combining[8] and have render_text append zero-width codepoints to the current column before cursor output. this requires updating OUT_BYTES_PER_CELL from 64 to 128 to cover worst-case scenario (base char + 8 combining marks, silently dropping marks beyond 8).

spec §8.3.3 has been updated with a grapheme-cluster preservation requirement. §13 now describes cells as "a grapheme cluster" rather than "a Unicode codepoint" and defines the capacity and truncation behavior.

semi-related to #84

natemoo-re · 2026-08-23T11:45:59Z

+/* Maximum combining marks stored per cell. Marks beyond this limit are
+ * silently truncated from the end; the first CELL_MAX_COMBINING are kept. */
+#define CELL_MAX_COMBINING 8


this gives us plenty of headroom for the common case and is unlikely to be a problem in the short-term, but I have a slight concern that it will be a long-term issue if any terminal protocols introduce complex metadata via ZWJ that requires >8 codepoints

an alternative design would be bumping this cieling and keeping a dynamic map of combining size per cell rather than reserving a flat 8 per cell.

pkg-pr-new · 2026-08-23T11:49:36Z

Open in StackBlitz

npm i https://pkg.pr.new/@bomb.sh/tty@114

commit: 5e775b5

github-actions · 2026-08-23T11:49:37Z

Size Increased — +1.0 KB

112.6 KB unpacked

codspeed · 2026-08-23T11:58:28Z

Merging this PR will not alter performance

✅ 10 untouched benchmarks

_{Comparing nm/cell-grapheme (5e775b5) with main (00aee53)¹}

No successful run was found on main (2864d7d) during the generation of this report, so 00aee53 was used instead as the comparison base. There might be some changes unrelated to this pull request in this report. ↩

The Cell struct stored only a single uint32_t codepoint, so render_text silently dropped every combining mark (wcwidth ≤ 0): the emitted ANSI stream contained bare base codepoints with no continuations. Root cause: kitty-graphics Unicode placeholder cells require two combining diacritics (row/col index from the rowcolumn-diacritics table) to follow the base U+10EEEE codepoint. Because those marks were dropped, every placeholder cell emitted as row 0 — producing the N-repeated-top-band banding artifact in multi-row placements. The same drop affected any combining-mark or ZWJ content: accented characters (e + U+0301), flag pairs, skin-tone modifiers. Fix: - Cell gains `uint32_t combining[8]` (zero-terminated). Marks beyond 8 are silently truncated from the end; the first marks always survive. - cells_fill uses a designated-init template so combining[] is zeroed on every back-buffer reset, and setcell clears it on every base write. - render_text tracks the last-written column; zero-width codepoints go to append_combining() instead of being discarded. - present_cups / present_lines emit combining[] bytes immediately after the base character, before any cursor repositioning. - OUT_BYTES_PER_CELL bumped 64→128 to cover base + 8 combining marks. - cell_cmp checks combining[] so a changed mark triggers a diff/rediff. - Spec updated: §8.3.3 normative preservation requirement, §13 cluster cell representation with truncation semantics, §13 measurement note. Four new tests: kitty placeholder (U+10EEEE + 2 marks), combining accent (e + U+0301), ZWJ family emoji (ZWJ preserved per-cell; following emoji start new cells — inherent cell-model constraint, documented), and truncation-from-end pinned at 8 marks.

427deb4 (#36) inserted the test-alt-os job above the existing test step, absorbing it into the new job — Linux stopped running tests entirely while macos/windows kept them. Re-add the step after the wasm build.

cowboyd

Can we make the tests visual as in other test cases? It would involve enhancing our print() helper (yet again!)

expect(trim(print(ansi, 12, 4))).toEqual(`
┌──────────┐
│café      │
│          │
└──────────┘`.trim());

We really ought to see if we can use libghostty for this (not now ofc, but sometime in the unspecified future)

setcell() is a no-op when the base codepoint falls outside the screen or the active clip region, but render_text recorded last_x unconditionally. A following combining mark was then appended to whatever cell sat at that column, leaking it onto a cell the text never owned. setcell now reports whether it wrote, and last_x is only set on a successful write.

print() now attaches zero-width codepoints (combining marks, ZWJ, variation selectors) to the most recently written cell instead of advancing the column, so grapheme tests can compare against a picture of the screen rather than counting substrings in the raw bytes.

Truncating marks beyond a cell's 8 slots was silent. Surface the first truncation in each frame through the existing per-render error list, mirroring CLIP_DEPTH_EXCEEDED, so callers can detect lost marks without the renderer doing IO.

append_combining now raises COMBINING_MARKS_EXCEEDED the first time a frame drops a mark, using a per-frame flag reset alongside clip_depth_exceeded. Truncation behavior is unchanged.

natemoo-re · 2026-10-08T14:13:54Z

@cowboyd updated print() to support zero-width codepoints so all the grapheme tests can render visually in 9454736. +1 on libghostty someday.

cowboyd

nice work!

cowboyd

I think you're going to need a changeset however

natemoo-re commented Aug 23, 2026

View reviewed changes

natemoo-re marked this pull request as ready for review August 23, 2026 11:49

natemoo-re force-pushed the nm/cell-grapheme branch from 07a84e9 to d7c2d69 Compare August 28, 2026 00:17

natemoo-re added 3 commits August 27, 2026 22:20

style: clang-format

414d3ed

fix(ci): restore deno task test on the ubuntu verify job

1a1f83d

427deb4 (#36) inserted the test-alt-os job above the existing test step, absorbing it into the new job — Linux stopped running tests entirely while macos/windows kept them. Re-add the step after the wasm build.

natemoo-re force-pushed the nm/cell-grapheme branch from d7c2d69 to 1a1f83d Compare August 28, 2026 02:21

natemoo-re requested review from cowboyd and rauhryan September 1, 2026 00:22

cowboyd reviewed Sep 1, 2026

View reviewed changes

Comment thread specs/renderer-spec.md Outdated

cowboyd reviewed Sep 1, 2026

View reviewed changes

Comment thread src/clayterm.c Outdated

cowboyd reviewed Sep 1, 2026

View reviewed changes

natemoo-re added 4 commits October 8, 2026 09:03

feat: report COMBINING_MARKS_EXCEEDED when a cell truncates marks

5e775b5

append_combining now raises COMBINING_MARKS_EXCEEDED the first time a frame drops a mark, using a per-frame flag reset alongside clip_depth_exceeded. Truncation behavior is unchanged.

cowboyd approved these changes Oct 8, 2026

View reviewed changes

cowboyd requested changes Oct 8, 2026

View reviewed changes

Navigation Menu

Repository navigation

fix: preserve grapheme clusters (combining marks) in cell emission - #114

fix: preserve grapheme clusters (combining marks) in cell emission#114
natemoo-re wants to merge 7 commits into
mainfrom
nm/cell-grapheme

natemoo-re commented Aug 23, 2026 •

edited

Loading

Uh oh!

natemoo-re Aug 23, 2026

Uh oh!

pkg-pr-new Bot commented Aug 23, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Aug 23, 2026 •

edited

Loading

Uh oh!

codspeed Bot commented Aug 23, 2026 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

cowboyd left a comment •

edited

Loading

Uh oh!

natemoo-re commented Oct 8, 2026

Uh oh!

cowboyd left a comment

Uh oh!

cowboyd left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

natemoo-re commented Aug 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

natemoo-re Aug 23, 2026

Choose a reason for hiding this comment

Uh oh!

pkg-pr-new Bot commented Aug 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Aug 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codspeed Bot commented Aug 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Merging this PR will not alter performance

Footnotes

Uh oh!

Uh oh!

Uh oh!

cowboyd left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

natemoo-re commented Oct 8, 2026

Uh oh!

cowboyd left a comment

Choose a reason for hiding this comment

Uh oh!

cowboyd left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

natemoo-re commented Aug 23, 2026 •

edited

Loading

pkg-pr-new Bot commented Aug 23, 2026 •

edited

Loading

github-actions Bot commented Aug 23, 2026 •

edited

Loading

codspeed Bot commented Aug 23, 2026 •

edited

Loading

cowboyd left a comment •

edited

Loading