Give every published crate its own README, release 0.0.2

CHANGELOG.md

@@ -5,6 +5,14 @@ the release workflow refuses a tag without one and copies it onto the
 GitHub release. The crates are versioned together, so one number is the
 version of all of them.
 
+## 0.0.2
+
+Every crate now has its own README, so its page on crates.io says what it is. In 0.0.1 only `zudb` and `zudb-cli` had one, and both showed the repository README. There are no code changes in this release.
+
+- Each `zudb-*` crate has a README covering what it does, how it is put together, and a short example against its real API. Each also says the crate is published for `zudb` to depend on, and that the import keeps the workspace name (`use zu_query` for `zudb-query`).
+- `zudb` has a README of its own with the quickstart, databases and connections, statements, transactions, bulk loading, errors, and the storage engines. Its Rust examples are programs in `crates/zu-snippets`, and the snippet test compiles and runs them against the README.
+- `zudb-cli` has a README covering the commands of the `zu` binary, the shell, output formats, and exit codes.
+
 ## 0.0.1
 
 The first release, and the first time zu is on crates.io. It is early:

Cargo.toml

@@ -4,27 +4,27 @@ members = ["crates/*"]
 exclude = ["fuzz"]
 
 [workspace.package]
-version = "0.0.1"
+version = "0.0.2"
 edition = "2024"
 rust-version = "1.98"
 license = "Apache-2.0"
 repository = "https://github.com/tamnd/zu"
 authors = ["Tam Nguyen <tamnd87@gmail.com>"]
 
 [workspace.dependencies]
-zu-common = { package = "zudb-common", path = "crates/zu-common", version = "=0.0.1" }
-zu-encoding = { package = "zudb-encoding", path = "crates/zu-encoding", version = "=0.0.1" }
-zu-json = { package = "zudb-json", path = "crates/zu-json", version = "=0.0.1" }
-zu-vector = { package = "zudb-vector", path = "crates/zu-vector", version = "=0.0.1" }
-zu-storage = { package = "zudb-storage", path = "crates/zu-storage", version = "=0.0.1" }
-zu-zu1 = { package = "zudb-zu1", path = "crates/zu-zu1", version = "=0.0.1" }
-zu-sqlite = { package = "zudb-sqlite", path = "crates/zu-sqlite", version = "=0.0.1" }
-zu-s3 = { package = "zudb-s3", path = "crates/zu-s3", version = "=0.0.1" }
-zu-query = { package = "zudb-query", path = "crates/zu-query", version = "=0.0.1" }
-zu-exec = { package = "zudb-exec", path = "crates/zu-exec", version = "=0.0.1" }
-zu = { package = "zudb", path = "crates/zu", version = "=0.0.1" }
-zu-arrow = { package = "zudb-arrow", path = "crates/zu-arrow", version = "=0.0.1" }
-zu-corpus = { package = "zudb-corpus", path = "crates/zu-corpus", version = "=0.0.1" }
+zu-common = { package = "zudb-common", path = "crates/zu-common", version = "=0.0.2" }
+zu-encoding = { package = "zudb-encoding", path = "crates/zu-encoding", version = "=0.0.2" }
+zu-json = { package = "zudb-json", path = "crates/zu-json", version = "=0.0.2" }
+zu-vector = { package = "zudb-vector", path = "crates/zu-vector", version = "=0.0.2" }
+zu-storage = { package = "zudb-storage", path = "crates/zu-storage", version = "=0.0.2" }
+zu-zu1 = { package = "zudb-zu1", path = "crates/zu-zu1", version = "=0.0.2" }
+zu-sqlite = { package = "zudb-sqlite", path = "crates/zu-sqlite", version = "=0.0.2" }
+zu-s3 = { package = "zudb-s3", path = "crates/zu-s3", version = "=0.0.2" }
+zu-query = { package = "zudb-query", path = "crates/zu-query", version = "=0.0.2" }
+zu-exec = { package = "zudb-exec", path = "crates/zu-exec", version = "=0.0.2" }
+zu = { package = "zudb", path = "crates/zu", version = "=0.0.2" }
+zu-arrow = { package = "zudb-arrow", path = "crates/zu-arrow", version = "=0.0.2" }
+zu-corpus = { package = "zudb-corpus", path = "crates/zu-corpus", version = "=0.0.2" }
 
 thiserror = "2"
 crc32c = "0.6"

conformance.toml

@@ -9,7 +9,7 @@
 
 [engine]
 name = "zu"
-version = "0.0.1"
+version = "0.0.2"
 
 # What the storage can hold.
 [data]

crates/zu-arrow/Cargo.toml

@@ -7,6 +7,7 @@ rust-version.workspace = true
 license.workspace = true
 repository.workspace = true
 authors.workspace = true
+readme = "README.md"
 
 # Published as zudb-*, because zu is taken on crates.io, but the
 # library keeps the name every crate in this workspace already uses.

crates/zu-arrow/README.md

@@ -0,0 +1,67 @@
+# zudb-arrow
+
+Turns a zu query result into Arrow arrays. This is the fast way to get data out of zu: a result becomes one Arrow buffer per column instead of one object per row, and pandas, polars, DuckDB, and the JavaScript dataframe libraries can read it without copying it again.
+
+## How it works
+
+The engine does most of the work before this crate sees anything. On a plain projection the sink in `zudb-exec` fills column buffers, not rows, and they are already in the layout Arrow uses: values end to end, a validity bitmap that is left out when nothing is null, and strings as bytes plus offsets. `zudb-query` exposes them through `QueryResult::columnar` (borrowed) and `QueryResult::into_columns` (owned).
+
+This crate puts an Arrow type and a bitmap around those buffers.
+
+- `Table::of(&result, &tables)` borrows the result, so the buffers are copied once into arrays.
+- `Table::taken(result, &tables)` consumes the result and moves the buffers in. For integers, floats, booleans, strings, dates, times, datetimes, and durations this costs a pointer, not a copy. If the sink built rows instead of columns, it quietly falls back to `of`.
+- Nodes, rels, paths, lists, and records have no buffer to move. They are built by hand as Arrow structs and lists in the `values` module.
+- A `Table` has a schema, one array per column, and a row count. `batches(n)` cuts it into `RecordBatch`es by slicing, which copies nothing. An empty result still gives one empty batch, so the reader learns the schema.
+
+Node and rel values carry a table id. To turn that into a name, pass something that implements the `Tables` trait. `()` knows no names and is fine when the result has no node or rel columns.
+
+A column has one type, which the engine decides. Two things are refused here because Arrow has no type for them: a time with a UTC offset, and a handle to a graph or a binding table. Errors come back as `Error::Type`, `Error::Value`, or `Error::Arrow`.
+
+The code lives with the engine rather than in a client because every client should convert a column the same way. The Python and JavaScript clients both use it.
+
+## Using it
+
+```rust
+use zu_arrow::{BATCH, Table};
+
+fn main() -> Result<(), Box<dyn std::error::Error>> {
+    let db = zudb::Database::memory()?;
+    let mut conn = db.connect()?;
+    conn.execute("INSERT (p:person {uid: 1, name: 'ada'})")?;
+    conn.execute("INSERT (p:person {uid: 2, name: 'grace'})")?;
+
+    let result = conn.query("MATCH (p:person) RETURN p.name AS name, p.uid AS uid")?;
+
+    // `taken` consumes the result and moves its column buffers into
+    // Arrow. `()` is the table-name lookup, which only node and rel
+    // columns need.
+    let table = Table::taken(result, &())?;
+    println!("{} rows, schema {:?}", table.rows(), table.schema());
+    for batch in table.batches(BATCH) {
+        let batch = batch?;
+        println!("{} columns, {} rows", batch.num_columns(), batch.num_rows());
+    }
+    Ok(())
+}
+```
+
+That prints a schema with `name` as `Utf8` and `uid` as `Int64`, and one batch of two rows. `BATCH` is the default batch size, 65,536 rows.
+
+## Features
+
+Both features are off by default.
+
+- `ffi` adds `Table::into_stream`, `stream`, and `stream_taken`, which return an `FFI_ArrowArrayStream`. That is the Arrow C Data Interface, the way to hand a result to pyarrow, DuckDB, Go, or the JVM in the same process. The buffers are shared with the reader and released when it is done.
+- `ipc` adds `Table::into_ipc`, `ipc`, and `ipc_taken`, which return the bytes of an Arrow IPC stream. That is for runtimes that cannot read a C struct, which in practice means JavaScript.
+
+## Should you depend on this?
+
+Probably not directly. This crate is one part of [zu](https://github.com/tamnd/zu), an embedded property-graph database, and it is published so that [`zudb`](https://crates.io/crates/zudb) can depend on it. Its API changes whenever the engine needs it to, with no deprecation period, and the versions of all the zu crates move together. If you want to use zu from Rust, add `zudb` and use what it exports.
+
+The package is named `zudb-arrow` because `zu` was already taken on crates.io. The library keeps the name the workspace uses, so the import is `use zu_arrow`.
+
+zu is early and not ready for real data yet. See the [repository](https://github.com/tamnd/zu) for the status and the specification.
+
+## License
+
+Apache-2.0

crates/zu-cli/Cargo.toml

@@ -7,7 +7,7 @@ rust-version.workspace = true
 license.workspace = true
 repository.workspace = true
 authors.workspace = true
-readme = "../../README.md"
+readme = "README.md"
 keywords = ["database", "graph", "gql", "cli"]
 categories = ["command-line-utilities", "database-implementations"]
 

crates/zu-cli/README.md

@@ -0,0 +1,95 @@
+# zudb-cli
+
+The `zu` command-line tool for [zu](https://github.com/tamnd/zu), an embedded property-graph database. It loads data into `.zu1` files, runs queries against them, checks them, and gives you an interactive shell and a language server for editors.
+
+```
+cargo install zudb-cli
+```
+
+That installs one binary, `zu`. The package is named `zudb-cli` because `zu` was already taken on crates.io. Reading Parquet edge lists needs the `arrow` feature: `cargo install zudb-cli --features arrow`.
+
+zu is early and not ready for data you care about. Flags and output can change between 0.0.x releases.
+
+## A first session
+
+```
+zu copy edges.csv graph.zu1
+zu stat graph.zu1
+zu query graph.zu1 -c 'MATCH (n:node) RETURN count(n) AS n'
+zu query graph.zu1 -c 'MATCH (n:node {id: $id}) RETURN n.id' -p id=10 --format json
+zu shell graph.zu1
+```
+
+`zu copy` builds a new file from an edge list, `zu query` runs one statement and exits, and `zu shell` keeps the file open so the catalog, statistics, plan cache and decoded blocks are paid for once.
+
+## Commands
+
+| Command | What it does |
+| --- | --- |
+| `zu shell [<file.zu1>]` | open an interactive session on a file, or on nothing |
+| `zu query <file.zu1> -c <zuQL>` | run one statement and print the result |
+| `zu lsp --stdio [--db <file.zu1>]` | speak the language server protocol, for an editor |
+| `zu copy <edges> <out.zu1>` | bulk load an edge list, or a whole dataset, into a new file |
+| `zu convert <in> <out>` | rewrite an edge list in another format, or a database in another engine |
+| `zu verify <file.zu1>` | walk every checksum and cross-check the structure |
+| `zu stat <file.zu1>` | print the size breakdown: schema, free space, and data |
+| `zu analyze <file.zu1>` | rebuild every rel table's optimizer summary |
+| `zu neighbors [--in] [--key] <file.zu1> <node>` | print one node's neighbor list, in either direction |
+| `zu lookup <file.zu1> <key>` | resolve an original id through the primary-key index |
+| `zu edge [--in] <file.zu1> <src> <dst>` | ask whether one edge exists, without decoding the list |
+| `zu conformance ...` | declare, verify, tally, and score conformance reports |
+| `zu corpus <dir>` | run the shared corpus cases against this build |
+| `zu version` | print the version, the C ABI revision, and what this build supports |
+
+`zu help` lists them and `zu help <command>` (or `zu <command> --help`) prints the synopsis, examples and related commands for one. All of it is generated from one table in the source, so the usage line you get after a wrong command line is the same one the help shows.
+
+### Loading and converting
+
+`zu copy` takes an edge list as whitespace separated text, CSV, or Parquet, and writes one node table and one rel table. `--nodes nodes.csv` adds node properties, and `--reorder degree|bfs|none` renumbers nodes before writing so that adjacency compresses better. Original ids stay readable. For labelled data, give one file per table: `zu copy --node Account=nodes/Account.csv --rel transfer=Account:Account:rels/transfer.csv fin.zu1`.
+
+`zu convert` moves an edge list between text, CSV and Parquet, or a database between the `zu1` format and SQLite (`graph.zu1` to `graph.db` and back). Both engines number rows the same way, so the converted file answers queries with the same ids.
+
+### The shell
+
+At a terminal, `zu shell` is an editor with history, multi-line statements, syntax highlighting, and tab completion that offers the labels and properties that are actually in the open file. Backslash commands answer questions about the file without going through the query language:
+
+| Command | What it shows |
+| --- | --- |
+| `\d [NAME]` | the tables in the file, or one table's columns |
+| `\l` | the graphs in the file |
+| `\session` | the schema, graph, time zone and parameters of this session |
+| `\i FILE` | runs the statements in a file |
+| `\timing [on\|off]` | how long each statement took |
+| `\?` and `\q` | help, and quit |
+
+When standard input is not a terminal, or with `--format jsonl`, the shell speaks a line protocol for programs instead. The first line out is a greeting with the protocol and build versions. Each line in is either a bare statement or a JSON frame such as `{"op":"query","q":"...","params":{...}}`, and the other ops are `prepare`, `execute`, `close_stmt`, `explain`, `explain_analyze`, `hello` and `quit`. Each reply is one JSON object with the GQLSTATUS, columns and rows, or an error. A failed statement does not end the session. The protocol is described in `docs/api/jsonl-protocol.md` in the repository.
+
+### Editors
+
+`zu lsp --stdio` is a language server in the same binary. Completion comes from the catalog of the file passed with `--db`, diagnostics come from the same parser that would reject the statement, and highlighting uses the same scanner as the shell prompt.
+
+## Output and exit codes
+
+Commands that print results take `--format`. `query` takes `table` or `json`. `stat`, `verify`, `convert`, `copy` and `version` take `text` or `json`, and `zu help --format json` prints the whole command table for tools. The flag is spelled and read the same way by every command.
+
+The exit code depends on the kind of error, not on which command hit it:
+
+| Code | Meaning |
+| --- | --- |
+| 0 | success |
+| 1 | the statement failed with a GQL condition |
+| 2 | bad usage or an invalid argument |
+| 3 | an I/O error |
+| 4 | the file is corrupt, or was written by something this build cannot read |
+| 5 | lost a write conflict to another writer |
+| 130 | interrupted |
+
+## How it is built
+
+The binary is a thin layer over [`zudb`](https://crates.io/crates/zudb). Every query goes through the same `Session` the Rust API uses, so the CLI and an embedding program get the same answers. Argument parsing is hand-written rather than clap, because the command surface is small and the binary has a 15 MiB size budget. JSON reading and writing come from the small `zudb-json` crate for the same reason. The shell is split into a terminal driver (`term`), key decoding (`keys`), a line editor (`line`) and the loop that joins them (`repl`), so everything except the loop is tested without a terminal. The `corpus` and `conformance` commands are the test harnesses the release process runs, and they use `zudb-corpus`.
+
+To use zu from Rust code instead of the shell, depend on [`zudb`](https://crates.io/crates/zudb).
+
+## License
+
+Apache-2.0

crates/zu-common/Cargo.toml

@@ -7,6 +7,7 @@ rust-version.workspace = true
 license.workspace = true
 repository.workspace = true
 authors.workspace = true
+readme = "README.md"
 
 # Published as zudb-*, because zu is taken on crates.io, but the
 # library keeps the name every crate in this workspace already uses.

crates/zu-common/README.md

@@ -0,0 +1,79 @@
+# zudb-common
+
+The shared vocabulary of [zu](https://github.com/tamnd/zu), an embedded property-graph database: ids, the error type, the type model, and the value types (decimals, dates and times, byte strings) that every other crate in the workspace needs to agree on.
+
+It is the bottom of the dependency graph. It depends only on `thiserror`, and every other zu crate depends on it.
+
+## What is in it
+
+| Item | What it is |
+| --- | --- |
+| `NodeId`, `RelId`, `TableId`, `NodeGroupId`, `NodeOffset`, `Epoch` | Graph identifiers. `NodeId` packs a table, a node group and a row into one `u64`. |
+| `ZuError`, `Result` | The error every zu crate returns. Query errors carry a GQLSTATUS diagnostic. |
+| `gqlstatus` | The ISO/IEC 39075 condition codes, generated from the standard's own table, and the `DiagnosticRecord` that carries one back to a caller. |
+| `types` | The two layer type model: `LogicalType` is what the language talks about, `PhysicalType` is what a vector holds. |
+| `Decimal` | Exact decimals as an `i128` and a scale, up to 38 digits. |
+| `temporal` | Dates, times, datetimes and durations, stored as plain counts (days, nanoseconds, months). |
+| `bytes`, `unicode`, `keywords` | Byte string literals, Unicode normalization (NFC, NFD, NFKC, NFKD), and the GQL reserved word lists. |
+| `Interrupt` | A handle a caller keeps on a running statement to stop it or read its progress. |
+| `IdMap`, `IdSet` | Hash maps and sets with a cheap hasher for keys that come from the catalog, not from users. |
+
+## Ids
+
+`NodeId` is the one layout that is written to disk by every storage engine, so it does not change between versions:
+
+```text
+bits 63..50  table_id     (14 bits, up to 16 384 tables)
+bits 49..28  node_group   (22 bits, up to 4 194 304 groups per table)
+bits 27..11  row          (17 bits, GROUP_ROWS = 131 072 rows per group)
+bits 10..0   reserved     (must be zero)
+```
+
+A node group is the unit storage works in: a table is split into groups of `GROUP_ROWS` rows, and each group's columns are stored and compressed together. The full data model is in `docs/03-data-model.md`.
+
+## Values
+
+The value types are built so that comparing two values is comparing two integers. A date is a day count since 1970-01-01, a local time is nanoseconds since midnight, and a duration is either a month count or a nanosecond count depending on its kind. Calendar arithmetic only happens when text is parsed or printed. `Decimal` works the same way: `1.20` and `1.2` compare equal, and each prints back the way it was written.
+
+## Errors
+
+`ZuError` has a variant for I/O, corrupt data, unsupported format ids, invalid arguments, write conflicts and interrupts. Errors raised by a query are `ZuError::Gql` and carry a GQLSTATUS code, a severity, and optionally a position in the statement text, so a client can point at the token that failed instead of parsing the message.
+
+## Example
+
+```rust
+use zu_common::gqlstatus::codes;
+use zu_common::{Decimal, LogicalType, NodeId, Temporal, ZuError};
+
+fn main() {
+    // A node id packs the table, the node group and the row into one u64.
+    let id = NodeId::new(3, 7, 42);
+    assert_eq!((id.table(), id.group(), id.row()), (3, 7, 42));
+
+    // Decimals are exact: an i128 and a scale, never a float.
+    let price = Decimal::parse("19.99", 2).unwrap();
+    let tax = Decimal::parse("1.60", 2).unwrap();
+    println!("{}", price.add(&tax).unwrap()); // 21.59
+
+    // A date is a day count, so comparing two dates compares two integers.
+    let day = Temporal::parse(&LogicalType::Date, "2026-10-02").unwrap();
+    println!("{day}");
+
+    // Errors carry a GQLSTATUS code from ISO/IEC 39075.
+    let err = ZuError::gql(codes::C22012, "x / 0");
+    assert_eq!(err.gqlstatus().map(|s| s.code()), Some("22012"));
+    println!("{err}");
+}
+```
+
+## Should you depend on this?
+
+Probably not directly. This crate is published so that [`zudb`](https://crates.io/crates/zudb) can depend on it. Its API changes whenever the engine needs it to, with no deprecation period, and all the zu crates are released together with exact version pins between them. If you want to use zu from Rust, add `zudb` and use what it exports.
+
+The package is named `zudb-common` because `zu` was already taken on crates.io. The library keeps the name the workspace uses, so the import is `use zu_common`.
+
+zu is early and not ready for real data yet. See the [repository](https://github.com/tamnd/zu) for the status and the specification.
+
+## License
+
+Apache-2.0