DX2: the shell and the first bindings

[tamnd]

Milestone DX2 of the developer-experience program (Spec/2064g/dx/00-overview.md). Specs: dx/04, dx/06, dx/12 §2, §5, dx/03 §5, dx/13 §1. Runs with M2 (#3, closed).

M2 landed the query engine, so this is where the SDKs become real. Python goes first, ahead of TypeScript and Go, for one reason: an analytical database is evaluated from a notebook, and the notebook is where a bad first impression is unrecoverable. The 60-second snippet in dx/06 §1 is the target, import zudb, zudb.connect, .to_pandas(), nothing else.

Two things here are corrections from the 2026 audit (Spec/2064g/AUDIT-2026-08.md §2.1). The wheel matrix is not one abi3 wheel: the free-threaded build has no stable ABI until CPython 3.15 and PEP 803's abi3t, so 3.14t needs a version-specific cp314-cp314t wheel, and PyO3 0.29 dropped 3.13t entirely. And the Python floor is 3.11, not 3.9.

shell (dx/12 §2):

• multi-line editing, syntax highlighting, completion, paging
• backslash commands, Ctrl-C cancel, progress reporting
• JSONL session protocol v1, versioned, tested, documented (dx/12 §5)

Rust SDK v1 (dx/04):

• query, parameters, typed rows, streaming
• explain and profile returning structured plans, not strings

Python SDK v0 (dx/06):

• PyO3 0.29 + maturin 1.14, three wheels per platform: cp311-abi3, cp314-cp314t, cp315-abi3.abi3t
• the release job asserts the exact expected wheel tag set and fails on an unrequested version-specific wheel
• context managers, iteration, to_arrow, to_pandas with Arrow-backed dtypes for pandas 3
• .pyi stubs inside the wheel with py.typed, checked against the runtime by griffe
• GIL released around every query, load, and appender flush
• cold import zudb under 50 ms, gated

error model end to end (dx/03 §5):

• GQLSTATUS through the ABI to Python and Rust as structured fields, not a formatted string
• code, severity, position with offset/line/column, excerpt, doc_url, retryable

cancellation and progress:

• zu_conn_interrupt, progress callbacks
• Ctrl-C raises KeyboardInterrupt in Python within 50 ms, and interrupts in the shell

corpus and grammar:

• results, errors, parameters: ~940 cases total, Python and Rust runners
• tree-sitter-gql and the Shiki grammar, shared by docs, shell, and editors (dx/13 §1)

done when:

• the 60-second snippet works verbatim in Python and Rust
• the misuse suite passes in both: no crash, no leak, a clear error for every deliberately wrong program
• Ctrl-C interrupts a 10-second query within 50 ms in the shell, in Python, and through the ABI

[tamnd]

Multi-line editing landed in #259. The box stays open because the line it is on also names highlighting, completion and paging, which are the next two changes.

zu shell now asks whether standard input is a terminal instead of assuming it is a pipe. A person gets a prompt, a history and statements that span lines, a program gets exactly the JSON frames it always got, and --format jsonl forces the frames for a harness that runs zu under a pty.

Return runs a statement whose brackets and quotes are closed or that ends in a semicolon, and adds a line when they are not. Ctrl-J adds a line whatever the text looks like, which is the way out when the rule and the author disagree. Up and down walk the lines of the buffer first and the history second, so editing the middle of a three-line MATCH does not throw it away. Ctrl-C abandons the line and keeps the session, Ctrl-D on an empty line leaves, and so do quit and exit.

The shell is written here rather than taken from a crate, and that is the part worth recording. term.rs is the one file that talks to a driver, about thirty lines of termios on unix and console modes on Windows, with a fallback that reports no terminal so a tier 2 platform falls back to the line loop. Everything above it is pure and tested without a terminal anywhere: keys.rs turns bytes into keys and answers "not yet" rather than guessing when an escape sequence arrives split across reads, and line.rs turns keys into a buffer and returns the escape sequence that shows it, so a test asks where the cursor would be drawn on an eighty column window. Twenty eight unit tests, and the whole thing driven end to end under a pty, which is where the one real bug turned up: raw mode has to clear ICRNL, or the driver turns the return key's carriage return into a newline and return and Ctrl-J become the same key.

Keystroke cost is gated, since a redraw rebuilds the whole buffer on every key. crates/zu-cli/benches/editor.rs measures 0.21 to 0.46 microseconds a key on a one-line statement and 0.47 to 1.03 on a six-line one, against a 50 microsecond ceiling in bench/budgets.toml. The bench compiles the editor's own source rather than a copy of it, so the number cannot drift from what the shell runs.

Next on this line: highlighting and completion, which want the tree-sitter grammar this milestone also lists, then paging with the backslash commands and Ctrl-C cancel.

[tamnd]

Colour and completion landed in #261, the second and third of the four things the shell line names.

The highlighter is a scanner rather than a parser, and the reason is that a shell repaints on every keystroke and the only thing that can be said about MATCH (a:Per is what its characters are. A parser's answer to a half typed statement is a syntax error, and a syntax error is not a colour. Keywords, strings, numbers, comments and $parameters get one of the sixteen colours every terminal has had for fifty years, since the 256 colour forms are the ones that come out unreadable on somebody's light background. The escapes go into the text that is written and never into the text the geometry is measured from, so a painted statement wraps exactly where a plain one does, and NO_COLOR or TERM=dumb turns all of it off.

Completion is the half only a database can give. Tab reads the catalog rather than a word list, so a label that comes back is a label that is in the file, and a tab that offers nothing has told you the thing you are typing is not there. The character in front of the word is the context: a colon takes labels and rel table names, a dot takes the properties the graph types declare, and anything else takes keywords, tables and graphs. One candidate is inserted, several insert what they share and print the rest in columns above the prompt, and a keyword comes back in the case it was typed in, so matc finishes as match. The names are gathered again after every statement, because a statement is how a table comes into being.

Both are held to the keystroke budget. On this machine a key costs 0.31 microseconds on a one-line statement, 0.73 on a six-line one, 2.17 to paint the six-line one, and a tab weighed against a hundred labels and a hundred tables costs 2.85, against the 50 microsecond ceiling. A pty run is what says it works: MATCH (a:n and tab completes to node, ret and tab to return, co and tab prints the five keywords that start with it, and a tab inside a string leaves the string alone.

The box stays unticked because the same line asks for paging, which is next, together with the backslash commands, Ctrl-C on a running statement, and progress. The tree-sitter and Shiki grammars this milestone lists separately are still worth having for editors and for the documentation site, where a real grammar earns what it costs; the shell does not want one on a keystroke.

[tamnd]

Paging landed in #263, which is the fourth of the four things on the editing line, so that box is ticked.

A result of forty rows in a window of twenty four used to be thirty six rows a user watched go past. The shell now shows what fits, prints --more-- in reverse video, and waits: space takes the next screen, return takes one row, q or Ctrl-C stops. There is no going back and no alternate screen, and the reason is that the rows go into the terminal's own scrollback, where the terminal's scrolling and the terminal's search already work; a pager that took the screen away would owe the user both of those back. $PAGER is not spawned either, since that hands a user's data to a program zu did not choose and means leaving raw mode, forking, and hoping the child gives the terminal back the way it found it.

The arithmetic is in page.rs and nothing else is, so wrapping and stepping are tested without a terminal: a long line is cut where the terminal would cut it, counting columns rather than characters so a wide character starts the next row instead of being halved, and the trailing newline every answer ends with is not a row of its own. What fits is printed the way it always was, so a one-row result waits for nothing, and a terminal that will not enter raw mode gets the whole answer rather than an error. term.rs learned the window's height alongside its width, falling back to the VT100's twenty four rows where the terminal does not know its own size.

A pty run over a forty edge graph is what says it works end to end: twenty three rows, the prompt, and the rest of the answer on the space that follows.

Next on this milestone is the line under it: the backslash commands, Ctrl-C on a running statement, and progress reporting. The first is a parse and a table, the second wants the interrupt the cancellation line also asks for, so the two will land together.

[tamnd]

The backslash commands have landed in #265, which is the first third of the shell line here.

\? lists them, \d lists the tables with what each holds and what a rel table joins, \d NAME prints one table's columns with their types and whether any row leaves them out, \l lists the graphs, \i FILE runs a file and stops at the first statement that fails, \timing reports how long each statement took, and \q leaves. The spellings are psql's, since a database user arrives with those in their fingers. The columns come from the property directory rather than from the declared graph type, because a table loaded from a CSV has columns and no declaration.

Driving it turned up a real bug and fixed it: the shell was treating -- as a line comment, which the lexer is not, so it dimmed text its own engine still ran and its completeness rule skipped the brackets after it. // and /* */ are the comments, in the scanner, in the completeness rule and in the file splitter.

The line stays unticked, because the other two thirds of it are still open. Ctrl-C on a running statement needs an interrupt the engine does not have yet: zu::session::Session runs a statement to the end with nothing to ask whether it should stop. That is the next piece, a flag the execution loops read at their boundaries plus a signal handler in the one file allowed to make syscalls, and progress reporting rides on the same check, since whatever is in a position to notice a cancellation is in a position to say how far it has got.

[tamnd]

Ctrl-C cancel and progress reporting landed in #268, which finishes the shell line above and ticks it: the backslash commands went in with #265 and multi-line editing, highlighting, completion and paging were already there.

Ctrl-C on a running statement used to be the signal it has always been, so it ended the process and took the open file, the plan cache and the session with it. What it does now is stop the statement. The word the engine reads is zu_common::Interrupt, one allocation per session that anyone can clone, and it rides in exec::Options with the rest of the switches rather than as an argument on the execute entry points, because every path already carries the options and a cancellation wired into only some of them is a feature that works on only some statements. Both executors already stop at a boundary, so that is where the check goes: the first thing next() does in the chunk executor, which is the single pull entry point every operator's inner loop goes through, and StopState::stopped() in the push pipeline, the flag its workers already test per morsel. A drained run returns an error rather than the piece of an answer its workers happened to finish.

Progress rides on the same handle, counted in rows read from storage, once per chunk at the scan and once per claimed morsel. The shell paints a line after four hundred milliseconds, polls twenty times a second, writes only when the line would say something different, and erases it before the answer. A statement that was stopped says so and says the session survived it.

Stopping is not failing, so Interrupted carries no GQLSTATUS, the standard having no condition for it, the session answers the next statement, the process exits 130, and a host on the C ABI reads ZU_INTERRUPTED, the number the header has been holding for it since the status enum went in.

The two cancellation lines further down stay open: zu_conn_interrupt and progress callbacks are the ABI half, and KeyboardInterrupt in Python needs the Python SDK first. The engine half they both need is now in place.

Next is the JSONL session protocol v1, versioned, tested and documented.

[tamnd]

JSONL session protocol v1 is in, #269.

The shell already answered frames on a pipe, but nothing about that wire was a promise: a client had to run a query to find out whether it was talking to a build it understood, and there was no number to check. So the session now writes a greeting before it reads anything, {"protocol":1,"zu":"0.0.1","c_abi":"0.5","format_version":1,"features":[],"file":"social.zu1"}, and {"op":"hello"} answers with the same object at any time for the client that attached to a session somebody else started. The version moves when a frame changes meaning and stays put when one is added, which is the rule that lets a client ignore fields it does not know.

Parameters got the other half of the fix. A JSON array now arrives as a list and a JSON object as a record, nested as deep as it is written, which is the mapping the data model already implied and which makes the corpus's IN $ids predicates expressible over the wire at last. That removed the last parameter fault worth having: the only one left is a params that is not an object. Temporals deliberately have no wire spelling, since JSON has no date and a string that looks like one is a string, so a statement that wants one takes a string and calls date($when), naming the calendar type instead of leaving the wire to guess it.

The whole thing is written down in docs/api/jsonl-protocol.md: framing, every frame, the result shape, the two failure shapes and how to tell an engine condition from a protocol fault, parameters, a worked session, and what version 1 does not do. That last section is the honest part. A result arrives whole on one line, there is no in-band cancellation, and a column of a million values travels as a million JSON numbers. Each is a version 2 question and none of them is a reason to hold v1 back from the callers it already serves.

The integration test drives the real binary over pipes and reads the greeting first, then asks for it again and checks it is the same bytes. Next up is the Rust SDK, typed rows first.

[tamnd]

The first Rust SDK line is done. #271 gave a result typed rows, row.get::<(&str, i64)>() and get_by_name, so reading an answer stops being a match on Value. #274 gives it streaming, which is the half of that line that changes what the SDK can be used for.

query_stream takes a sink and hands it a batch at a time, borrowed for the length of the call, and the sink says whether it wants more. query_stream_batched names the batch size, for a caller writing Arrow record batches or HTTP chunks of a size of its own. The call returns a Streamed instead of rows, since the rows are gone by then: the columns, the count, whether the caller stopped it, and whether the engine made them as it went.

It is a sink and not a cursor because a cursor here would have to hold a borrow of the graph and the operator state across the caller's call, and the crate forbids unsafe. The callback already holds both, so it is the version of the idea that compiles.

Both executors stream through the same handoff and a caller cannot tell which one ran its statement, which a parity test over four statements pins. The pipeline executor was worth the extra work: streaming through the pull executor alone measured six times the buffered read, and through the pipeline one it is 1.18 to 1.75 times, for 0.018 times the bytes. A caller that reads one batch and stops pays 0.015 of the scan, because a streamed statement sizes its morsels by the caller's batch rather than by a load balance there is no second worker to strike.

ORDER BY, DISTINCT and the aggregates cannot answer a first row before their last, so they run whole and are cut into batches afterwards. The caller's loop does not change and streamed reports which happened. SKIP and LIMIT do stream, spent across the batches rather than inside each one, so a streamed answer is the buffered answer row for row.

Next is the other Rust line, explain and profile returning structured plans rather than strings. exec::Profile already exists inside the engine, so most of that is deciding what the public shape of it should be.

[tamnd]

The Rust SDK is done. #275 makes EXPLAIN and EXPLAIN ANALYZE structures rather than text, which was the last line of it.

conn.explain_plan returns a QueryPlan: the operator tree, the columns the statement answers with, the parameters it wants bound, and the notes compiling it raised. An operator says what it is and what it is working on in separate fields, which bracket group it belongs to, which variables it introduces and which tables it reads. The listing is that tree rendered, so conn.explain is one call away from it and the two cannot describe different plans.

conn.profile returns the counters, which the engine had kept as a structure all along with no way out to a caller. An operator's kind and its detail are two fields there too, so grouping profiles by operator is a comparison rather than a parse, and qerror and bound_violation are the two derived numbers worth naming: how far the estimate was out, and whether a count passed a ceiling the optimizer promised.

A write is still refused by profile, since it runs as the parts it was split at its write into rather than as the one plan a profile describes. Reading a plan is not running one, so a read-only connection does that.

That closes both Rust lines. Python SDK v0 is next: PyO3 0.29 and maturin, the three wheels per platform, the release job that asserts the tag set, context managers and iteration, to_arrow and to_pandas, stubs inside the wheel, the GIL released around every query and flush, and the cold import under 50 ms.

[tamnd]

The error model is now every field dx/03 §5 asks for, through the engine, the C ABI and the Rust SDK (#277).

A condition carries its code, the class those first two digits name, the standard's text, the severity, the place, the line that place is on, the page it is written up on, and whether running the same statement again could get anywhere. Three of those are new. The page is https://zu.dev/docs/errors/ and the code, built once here rather than nine times in nine bindings, half of them differently. Retryable is true for 40000, a transaction rolled back with nothing done, and false for 40003, where the statement's completion is unknown and running it again could do the work twice; a Conflict, which carries no condition at all, answers true too, because that is the failure a retry loop exists for. The place is now said twice, a line and a column for the person reading and an offset for the caller slicing the text or highlighting a span, since the offset is the one number neither of the others yields without walking the statement again.

The excerpt is the piece I want to explain, because it is the one that could have been done three ways. It is the line the position falls on, quoted at the moment the condition was raised, and it is there for the caller holding an error and no longer holding the statement, which is every driver that logged the failure and every notebook that ran the cell an hour ago. It is the whole line rather than a window around the column, so the column stays a valid index into it and a caret built from the pair lands where the pair says it should. A line over four kilobytes is dropped whole rather than cut, because a cut line would put the caret somewhere else entirely, and no excerpt is better than a wrong one.

Filling it happens at the raise sites in the lexer and the parser rather than in a wrapper at some boundary, which is why it is free for the shell, the corpus runner and every SDK at once.

On the C ABI this is five additive functions, zu_error_standard_text, zu_error_doc_url, zu_error_excerpt, zu_error_retryable and zu_error_offset, so the revision moves to 0.6 and nothing compiled against 0.5 stops compiling. The offset accessor answers ZU_DONE wherever the position accessor does, and they answer together, so a caller tests one. The C smoke program reads all of it and passes under -Wall -Wextra -Werror -pedantic.

I ticked the second line of this section and left the first, since it names Python and there is no Python SDK yet to read these fields; it gets ticked when the wheel lands and its exception carries them.

One gap worth naming: the binder raises its conditions after the text has been read and thrown away, so an unknown label comes back with a code and a page but no place and no excerpt. That is a change of its own, carrying the source through the binder, and it is on the list.