Skip to content

Latest commit

 

History

History
324 lines (248 loc) · 13.7 KB

unicode.md

File metadata and controls

324 lines (248 loc) · 13.7 KB

unicode

a pith String is a sequence of utf-8 bytes. s.len() counts bytes, s[i] reads a byte, and s.substring(a, b) cuts between bytes. that is the same model Go and Rust use, and it is the right one for a server language: string indexing stays O(1) and no program pays for character bookkeeping it never asked for.

the cost of that model is that byte offsets and character offsets agree only while the text is ascii. std.text is the layer that closes the gap.

import std.text as text

text.char_count("Ärger")     # 5, where "Ärger".len() is 6
text.truncate("Ärger", 1)    # "Ä"
text.is_valid(input)         # is this really utf-8?

substring aborts rather than corrupt a character

a cut inside a character would leave half of it, a lead byte with its continuation missing, which no longer round-trips through utf-8. so substring aborts instead:

pith runtime error: substring(0, 1) would split the character 'Ä' at byte 1 (it occupies bytes 0..2)
  a String is indexed in bytes, so a byte offset can land inside a multi-byte character. use std.text for character-aware work:
    text.slice(s, a, b)         slice by character index
    text.truncate(s, n)         keep n characters
    text.truncate_bytes(s, n)   keep n bytes, ending on a boundary

this follows the strict accessors already in the runtime — pith_list_get_value_strict, pith_bytes_get_strict, and the strict map accessors — which all abort with a diagnostic rather than return a quietly wrong value. substring keeps its String return type, and code that cuts on boundaries never reaches the abort.

the check only fires when the cut lands strictly inside a well-formed multi-byte character. a String is a byte string, and code holding binary data in one may slice it anywhere; only real text has boundaries to respect. if you are slicing arbitrary bytes, use Bytes, which is what the rest of the standard library does.

a length check does not make an offset safe

the abort is correct and the caller is what needs fixing, so it is worth naming the mistake that produces one. len() counts bytes and substring takes byte offsets, so a length check never proves an offset falls on a character boundary. s.len() >= 2 says nothing about where character two starts. neither does "this field is thirteen characters by the spec", when the field came off a socket and the spec is the peer's to ignore.

three things do prove it:

  • an ascii byte you matched. an ascii byte can never appear inside a multi-byte character, so the position of one is always a boundary. this covers most parsing: index_of("="), a scan that stops on :, a cursor that only moved across digits it checked. it is why slicing at a delimiter is safe without thinking about it.
  • an ascii prefix or suffix you matched. starts_with("Bearer ") proves offset 7 is a boundary; ends_with("Z") proves len - 1 is.
  • a boundary helper. text.floor_boundary, text.ceil_boundary, text.byte_offset_of and the std.text slicing functions answer the question directly.

what does not prove it: a fixed offset on unvalidated input, an arithmetic index like len - 1 behind nothing but a length check, a count of bytes read or written, or a position from a scan that steps one byte at a time over arbitrary text. when you have one of those and need to cut, either move to the boundary beside it or read the byte in place with s[i], which returns one byte and never cuts anything.

comparing is the common case, and comparing does not need a cut at all. testing whether a word sits at a position by slicing it out is the shape that produced the worst of these: compare the bytes one at a time instead, which is both safe and cheaper.

picking a unit

three units, in increasing order of how much they cost and how closely they match what a reader sees:

unit what it is use it for
byte s.len(), s[i], substring protocol framing, storage limits, ascii parsing
character text.char_count, text.slice anything that must not corrupt text
grapheme graphemes.count, graphemes.truncate text a person reads

a "character" here is a unicode scalar value — the unit Go's for range and Rust's chars() yield. it is not always one mark on screen: e followed by a combining acute accent is two scalar values but one thing a reader sees, and a family emoji is several. when you are truncating a display name or a chat message, graphemes are the right unit.

there is a fourth unit pith never uses for its own strings but has to speak at protocol boundaries: the utf-16 code unit, which the language server protocol and the javascript world count positions in. std.text.utf16 converts between utf-16 offsets and byte offsets (length, to_byte_offset, from_byte_offset); a code point outside the basic multilingual plane counts as two units, everything else as one.

validation at the edge

bytes arriving from a socket, a file, or a form field are not text until you have checked them.

if not text.is_valid(field):
    fail "field is not utf-8"

or repair them, when rejecting is too strict:

safe := text.sanitize(field)   # every undecodable byte becomes U+FFFD

decode_at rejects overlong encodings, surrogates, and values above U+10FFFF, not just structurally broken sequences. those shapes are well-formed enough to fool a lenient decoder, which is how a string gets past a validation check as one thing and is read downstream as another.

find_invalid returns the byte offset of the first bad byte, for when the error message should say where.

truncation

three different questions, three functions:

text.truncate(s, 20)              # at most 20 characters
text.truncate_bytes(s, 255)       # at most 255 bytes, ending on a boundary
text.truncate_with(s, 20, "...")  # at most 20 characters, and say it was cut

truncate_bytes is the one to reach for when the limit comes from storage — a varchar(255), a header size cap, a log field budget. it never returns a partial character, so the result is always valid utf-8.

fold to compare, to_lower to display

to_lower and to_upper on String are ascii-only. "Ärger".to_lower() is "Ärger", unchanged. "ПРИВЕТ".to_lower() is "ПРИВЕТ", unchanged.

that matters more than it looks, because most to_lower() calls in auth, routing and header code are not lowercasing at all — they are caseless comparisons, written with the only tool that was there:

if header.to_lower() == "content-type":   # fine, headers are ascii
if username.to_lower() == stored:         # broken the moment a name is not

the second one is the bug. use fold:

text.fold("Ärger")              # "ärger"
text.eq_fold("Ärger", "ärger")  # true

folding is not lowercasing, and the difference is not pedantic. greek final sigma lowercases to itself but folds to a plain sigma, so "ΣΣ" and "σς" are the same word and only folding says so. folding produces a comparison key; lowercasing produces text for a reader. keep to_lower for the second job.

this is simple folding: every character folds to exactly one character. "ß" folds to itself rather than to "ss", so fold("straße") and fold("STRASSE") are not equal. full folding would close that gap with a mapping that changes length; it is not implemented, and the tests pin the current behaviour so it cannot drift by accident.

but do not fold a protocol token

fold is for text a person typed. It is the wrong tool for an identifier a specification defines, and reaching for it there is a security regression rather than a fix.

Two non-ASCII code points fold into ASCII:

code point folds to
U+017F LATIN SMALL LETTER LONG S ſ s
U+212A KELVIN SIGN K k

So text.eq_fold("cloſe", "close") is true, and so is text.eq_fold("websocKet", "websocket") with a Kelvin sign. HTTP, WebSocket and TLS all define their keywords as ASCII case-insensitive and nothing wider; folding them would accept values the RFC says are different, which is exactly the shape a header-smuggling or filter-bypass bug takes.

For those, strings.equals_ignore_case and to_lower are correct, and this is why the standard library's own Connection: close, Upgrade: websocket, Bearer and header-name comparisons were deliberately left alone:

strings.equals_ignore_case(header_name, "content-type")   # protocol token
text.eq_fold(display_name, stored_name)                   # text a user typed

The rule is about where the value came from, not how it looks. A specification wrote the token; a person wrote the text.

normalize so equal-looking text is equal

"é" can be one code point or an "e" followed by a combining acute accent. they render identically, a user cannot tell them apart, and they are different bytes — so they hash differently, compare unequal, and land in a database as two rows.

text.normalize(input)   # NFC, the composed form

normalize at the edge, where text arrives, and store the result. for an identifier a user typed, you usually want both operations:

text.eq_fold_normalized(typed, stored)

only NFC is implemented. NFD, NFKC and NFKD are not. NFC is the form to store and compare; the compatibility forms discard information — they flatten a fullwidth letter and its ascii form into the same string — which is the wrong default for text you are keeping.

graphemes, for anything a person reads

a scalar value is still not what a reader counts. "é" written as an e plus a combining accent is two scalar values and one mark on screen. a family emoji is five. a flag is two. truncating by scalar value can strip an accent off its letter, take the skin tone off a person, or leave half a flag.

std.text.graphemes splits on what a reader would call a character:

import std.text.graphemes as graphemes

graphemes.count("👨‍👩‍👧")            # 1, where text.char_count is 5
graphemes.truncate(name, 20)     # never cuts a cluster in half
graphemes.split(message)

use graphemes when the number or the cut is user-visible — a display name, a chat message, a character counter next to an input box. use the scalar functions in std.text when you are working with the text itself: parsing, validating, encoding. graphemes cost a table lookup per code point and a small amount of state, which is worth paying for a truncation and not for a parser.

these are the extended grapheme cluster rules from unicode annex 29, the same definition Swift's Character and a text editor's arrow keys use. the implementation is checked against the standard's own test file: all 1187 cases of GraphemeBreakTest.txt run as a colocated test, from data the generator packs into the tables, so regenerating for a new unicode release re-verifies the segmenter rather than just replacing its data.

display width, for terminal layout

a terminal cell is a fourth unit, alongside bytes, characters and graphemes: 日 fills two columns, a combining accent fills none, and a family emoji built from several code points fills two. std.text.width counts columns and lays text out against that count:

import std.text.width as width

width.of("日本")                        # 4 — two columns per character
width.pad_right("日", 5)                # "日   "
width.center("abc", 6)                  # " abc  "
width.truncate_to_with("日本語です", 7, "…")  # "日本語…"

counting is grapheme-aware: a flag pair measures 2, an accented letter 1, and truncation never splits a cluster — a two-column character that half-fits is dropped, so the result can come up one column short but never overflows. widths follow east asian width with Ambiguous counted narrow (wcwidth's default); variation selector 16 forces emoji presentation and two columns. the byte-based padders in std.strings are unchanged — they are for byte work, and terminal layout should use this module instead.

the tables

the case folding and normalization data is generated from the unicode character database, pinned at 15.1.0, by tools/unicodegen/generate.py:

python3 tools/unicodegen/generate.py

the generated modules are checked in, so building pith never needs the network. to move to a new unicode release, change UNICODE_VERSION in that script, rerun it, and commit the diff. the version is recorded in text.unicode_version() and in the header of every generated file.

the generator checks every table it emits against python's unicodedata before writing, and refuses to run if that module's version does not match the pin — a generator that only parses the UCD files can confirm it read them consistently, not that it read them correctly.

the tables are packed into string constants rather than list literals, and binary-searched in place, so they cost nothing at startup.

table records size
case folding 205 runs (from 1457 pairs) 4.0 KB
combining classes, decompositions, compositions 388 + 2061 + 941 42.0 KB
grapheme break properties, plus the conformance cases 733 ranges + 3009 25.0 KB
display width (wide and zero-width runs) 120 + 356 runs 4.7 KB

76 KB in total, all of it compiled in rather than loaded, and none of it touched by a program that does not call into these functions.

what is not here

std.text carries no locale. tags, message catalogs and plural rules belong in std.intl — see i18n.md — which builds on this module. std.text never depends on std.intl, so a program that only needs correct text does not pull in locale data.

also not implemented, deliberately: full and special case mapping (ß → SS, turkish dotted and dotless i), NFD/NFKC/NFKD, word and sentence segmentation, and bidirectional text.