diff --git a/CHANGELOG.md b/CHANGELOG.md index 065de5a..5947eae 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -14,5 +14,8 @@ Version 0.3.0 is prepared but has not been published. Its stabilization work inc - SQLite safety: upgrade the bundled SQLite and verify its runtime version. - Release integrity: validate release versions, locked builds, archive contents, checksums, and installer verification. - Project governance: document private security reporting and contribution checks, and add issue and pull request templates. +- 查询性能:`cite` 现在使用本地建立的 `cbeta_id` 索引,不再全表扫描;索引在安装/更新时建立,已有数据在首次 `cite` 时补建,数据目录不可写时静默退回扫描。 +- 中断恢复:写入数据时被中断(Ctrl-C、断电等)留下的回滚日志现在会在查询打开数据时由 SQLite 自动回滚;数据确实不可写而无法回滚时,报错会说明真实原因与处理办法,不再谎称数据不兼容。`fojin data clean` 与 `fojin data update` 也会一并清理活跃数据的 `-journal`/`-shm`/`-wal` 边车,新下载的数据不会再被上一份数据的陈旧日志污染。 +- 输出可读性:置信度显示为 `1.00` 时不再逐行标注 `[MITRA 1.00]`(当前数据集全部如此);`--json` 的 `confidence` 字段不变。 These changes are not released until the 0.3.0 release tag is published. diff --git a/README.md b/README.md index 6d9f8fe..75a29c8 100644 --- a/README.md +++ b/README.md @@ -12,8 +12,8 @@ ``` $ fojin parallel "色即是空" 汉 色不異空，空不異色，色即是空，空即是色； (《般若波羅蜜多心經》T0251 卷1) -梵 śūnyat'aiva rūpaṃ, rūpān na pṛthak śūnyatā … [MITRA 1.00] -藏 གཟུགས་ལས་སྟོང་པ་ཉིད་གཞན་མ་ཡིན༏ … [MITRA 1.00] +梵 śūnyat'aiva rūpaṃ, rūpān na pṛthak śūnyatā … +藏 གཟུགས་ལས་སྟོང་པ་ཉིད་གཞན་མ་ཡིན༏ … … 还有 38 组匹配,加 --all 查看全部 @@ -135,7 +135,7 @@ fojin parallel "色即是空" --json fojin texts "心经" # 模糊查经名(简繁均可) → Taishō 编号 + 各语对齐条数 fojin cite T0251 # 按编号列出一部经的对齐,经文顺序;--juan N 限定卷 fojin data status # 本地数据状态(位置/大小/版本/行数统计) -fojin data clean # 删除本地数据,释放 561 MB +fojin data clean # 删除本地数据,释放 578 MB fojin data update # 重新下载数据(覆盖本地) fojin data verify # 校验版本、SQLite 与 FTS 完整性 ``` @@ -143,6 +143,11 @@ fojin data verify # 校验版本、SQLite 与 FTS 完整性 `texts` 与 `cite` 支持与 `parallel` 一致的 `--json` / `--data-dir` / `--offline`; `cite` 另有 `--lang` / `--top` / `--limit` / `--all`。典型工作流:`texts` 找到编号 → `cite` 通读对齐。 +按经号查询依赖一个本地索引(约 17 MB)。v0.3.0 之后**全新下载**的数据在安装时就已带上它, +首次 `fojin cite` 不会有额外提示或等待;**升级前已经下载过的数据**则在首次运行 `fojin cite` +时补建一次(约 1–2 秒,数据目录相应增加约 17 MB),之后按经号查询为毫秒级。 +数据目录不可写时会跳过建索引,查询结果不受影响,只是较慢。 + ``` $ fojin texts "心经" | head -3 T0249 佛說帝釋般若波羅蜜多心經 (藏 50 · 梵 25) @@ -192,9 +197,11 @@ fojin parallel "<汉文短语>" --json --offline - 藏 / Tibetan:676,898 条 - 梵 / Sanskrit:231,722 条 - 来源:Dharmamitra 的 [MITRA-parallel](https://github.com/dharmamitra/mitra-parallel) 对齐数据集([Nehrdich & Keutzer, 2026](https://arxiv.org/pdf/2601.06400)),以 GitHub Release(`data-v1`)形式分发;学术使用请引用原论文(BibTeX 见 [`DATA_LICENSE`](DATA_LICENSE))。 +- 当前数据集(`data-v1`)所有对齐的置信度均为 1.00,不具区分度,因此人类可读输出不再逐行标注; + `--json` 的 `confidence` 字段保持输出。未来数据若提供真实分数,低于 1.00 的会自动显示。 - 当前二进制把官方下载地址、SHA-256 与兼容元数据固定在 `data-v1`;`fojin data update` 只会重新获取这份固定数据,不会自动切换到未来的数据主版本。版本、归一化规则或查询所需 schema 不兼容的数据会被拒绝。 -- 首次运行时下载,压缩包约 **183 MB**,解压后约 **561 MB**(SQLite)。下载后完全离线可用。 -- 安装和更新采用有界的磁盘流式传输,不再在内存中缓冲完整压缩包或数据库;压缩响应上限为 **256 MiB**,解压后的数据库上限为 **768 MiB**。更新期间可能临时需要现用数据库所占空间,外加约 **744 MiB** 暂存磁盘空间(约 183 MiB 压缩包 + 约 561 MiB 候选数据库)。 +- 首次运行时下载,压缩包约 **183 MB**,解压并建好按经号索引后约 **578 MB**(SQLite)。下载后完全离线可用。 +- 安装和更新采用有界的磁盘流式传输,不再在内存中缓冲完整压缩包或数据库;压缩响应上限为 **256 MiB**,解压后的数据库上限为 **768 MiB**。更新期间可能临时需要现用数据库所占空间,外加约 **761 MiB** 暂存磁盘空间(约 183 MiB 压缩包 + 约 578 MiB 候选数据库)。 - HTTP DNS 解析与连接超时均为 **30 秒**,响应头和响应体的空闲读取超时均为 **60 秒**,并以跨重定向、覆盖 DNS 到响应体读取的 **15 分钟端到端硬时限** 为上限。 - 同一数据目录上的首次安装、更新和清理操作按 single-flight 串行执行;等待者最多等待 **20 分钟**。永久保留的 `data.sqlite.lock` 文件是无害的协调文件,`fojin data clean` 会有意保留它。 - 离线查询行为及固定到 `data-v1` 的校验和契约保持不变。 @@ -233,7 +240,7 @@ fojin data verify # verify version, SQLite, and FTS integrity - **Build/install integrity**: building from crates.io or source requires Rust 1.95+ (MSRV 1.95). Starting with v0.3.0, the shell installer requires the target binary release to provide `SHA256SUMS` and verifies the archive before extraction. It fails closed for an older latest or explicitly selected release without that file, including the transition before v0.3.0 is published; use the currently published crates.io version or a source build instead. This does not state that v0.3.0 has been released. - **For AI agents**: pure-JSON stdout, semantic exit codes (`0` ok / `1` runtime / `2` usage), zero network with `--offline`. Ready-made Claude Code integration in [`examples/claude/`](examples/claude/). - **Data**: 908,620 zh↔sa/bo alignments from Dharmamitra's [MITRA-parallel](https://github.com/dharmamitra/mitra-parallel) dataset, redistributed under CC BY-SA 4.0. The official URL, checksum, and compatibility contract remain pinned to `data-v1`; rendering support for future language rows does not mean the official update channel can adopt them without a binary upgrade. Academic use: please cite [Nehrdich & Keutzer (2026)](https://arxiv.org/pdf/2601.06400) — BibTeX in [`DATA_LICENSE`](DATA_LICENSE). -- **Data transfer resources**: installs and updates use bounded, disk-streamed transfers and no longer buffer the complete archive or database in memory. Compressed responses are capped at **256 MiB** and decompressed databases at **768 MiB**. An update can temporarily require the live database plus roughly **744 MiB** of staging disk (about 183 MiB for the archive and 561 MiB for the candidate database). +- **Data transfer resources**: installs and updates use bounded, disk-streamed transfers and no longer buffer the complete archive or database in memory. Compressed responses are capped at **256 MiB** and decompressed databases at **768 MiB**. An update can temporarily require the live database plus roughly **761 MiB** of staging disk (about 183 MiB for the archive and 578 MiB for the candidate database). - **Data timeouts**: HTTP DNS resolution and connection timeouts are both **30 seconds**, response-header and response-body idle-read timeouts are both **60 seconds**, and a **15-minute hard end-to-end deadline** spans redirects from DNS through the final body read. - **Concurrent data operations**: initial install, update, and clean operations on one data directory are single-flight; a waiter may wait up to **20 minutes**. The permanent `data.sqlite.lock` file is harmless coordination state and intentionally survives `fojin data clean`. - **Stable query contract**: offline queries and the checksum contract pinned to `data-v1` are unchanged. diff --git a/docs/superpowers/plans/2026-07-26-cite-index-and-confidence.md b/docs/superpowers/plans/2026-07-26-cite-index-and-confidence.md new file mode 100644 index 0000000..d446553 --- /dev/null +++ b/docs/superpowers/plans/2026-07-26-cite-index-and-confidence.md @@ -0,0 +1,828 @@ +# cite 索引与 confidence 显示 实施计划 + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** 给 `cite` 消除全表扫描(2.34 s → ~0 ms),并让恒为 1.0 的 `confidence` 不再污染每一行输出。 + +**Architecture:** 索引是对**已下载产物**的本地优化,不进 `schema.sql`(那是兼容性参照物,且被 Python 导出管线执行)。两条创建路径:安装/更新时在候选文件上建好;`cite` 发现缺失时懒建一次。任何失败都静默退回全表扫描。`confidence` 只改渲染,JSON 与排序逻辑一字不动。 + +**Tech Stack:** Rust 2021、rusqlite 0.40.1(bundled SQLite)、anyhow。**不新增依赖。** + +**依据 spec:** `docs/superpowers/specs/2026-07-26-index-and-confidence-design.md` + +## Global Constraints + +- **MSRV 1.95。本机默认 stable 是 1.88,低于 MSRV,所有 cargo 命令必须写成 `rustup run 1.96.0 cargo ...`**,否则以 "requires rustc 1.95" 失败。MSRV 校验用 `rustup run 1.95.0 cargo test --all --locked`。 +- 不新增 Cargo 依赖,不改 `Cargo.toml`。 +- 所有面向用户的字符串用中文,与既有输出风格一致。 +- 退出码契约不变:`0` 成功、`1` 运行期错误、`2` 用法错误。 +- `--json` 时 stdout 必须是纯 JSON;提示走 stderr。 +- **索引的存在与否绝不改变任何查询结果,只改变速度。** +- **索引构建的任何失败都不得让查询失败**,一律静默退回全表扫描。 +- `cargo clippy --all-targets -- -D warnings` 无警告,`cargo fmt` 已格式化。 +- 全量测试基线 **213 全绿**,只应变多不应变红。 + +## 文件结构 + +| 文件 | 本轮职责 | +| --- | --- | +| `src/data/operation_lock.rs` | 新增 `try_acquire`:单次尝试、不等待、不打印 | +| `src/data.rs` | 索引 DDL 常量、`ensure_cbeta_index`(懒建)、安装路径建索引 | +| `src/cli.rs` | `Cite` 分支调用 `ensure_cbeta_index` | +| `src/render.rs` | `conf_tag` 改为显示精度为 `1.00` 时不输出标签 | +| `README.md` / `CHANGELOG.md` | 首屏示例去掉 `[MITRA 1.00]`;说明首次 `cite` 的一次性停顿 | + +--- + +### Task 1: `operation_lock::try_acquire` + +**Files:** +- Modify: `src/data/operation_lock.rs` + +**Interfaces:** +- Produces: `pub(super) fn try_acquire(data_path: &Path) -> Result` —— 单次 `try_lock`,锁被占用时立即返回 `Err`,**不等待、不向 stderr 打印任何内容**。 + +现有的 `acquire` 会最长等待 20 分钟,并在首次等待时打印"检测到另一个 fojin 数据操作,正在等待..."。这两条对查询路径都是错的:用户只是想查一部经,不该因为后台在更新数据而卡住或看到无关提示。 + +- [ ] **Step 1: 写失败测试** + +在 `src/data/operation_lock.rs` 的 `mod tests` 中追加: + +```rust + #[test] + fn try_acquire_returns_immediately_when_held() { + let directory = tempfile::tempdir().unwrap(); + let data = directory.path().join("data.sqlite"); + let held = acquire(&data, Duration::from_millis(100)).unwrap(); + + let started = Instant::now(); + let result = try_acquire(&data); + let waited = started.elapsed(); + + assert!(result.is_err(), "must not acquire a held lock"); + assert!( + waited < Duration::from_millis(50), + "try_acquire must not wait; waited {waited:?}" + ); + + drop(held); + assert!(try_acquire(&data).is_ok(), "must acquire once released"); + } +``` + +- [ ] **Step 2: 运行,确认失败** + +Run: `rustup run 1.96.0 cargo test --lib operation_lock` +Expected: FAIL,`cannot find function try_acquire in this scope`。 + +- [ ] **Step 3: 实现** + +把 `acquire` 中打开锁文件的那段抽成共享函数,并新增 `try_acquire`。将 `acquire` 的开头替换为调用共享函数: + +```rust +fn open_lock_file(data_path: &Path) -> Result<(File, PathBuf)> { + let lock_path = sibling_path(data_path, ".lock")?; + let file = OpenOptions::new() + .read(true) + .write(true) + .create(true) + .truncate(false) + .open(&lock_path) + .with_context(|| format!("打开数据操作锁失败: {}", lock_path.display()))?; + Ok((file, lock_path)) +} + +/// Single attempt, no waiting and no stderr output. The query path uses this: +/// a user running `cite` should never block on a background data operation, +/// nor see a message about one. +pub(super) fn try_acquire(data_path: &Path) -> Result { + let (file, lock_path) = open_lock_file(data_path)?; + match file.try_lock() { + Ok(()) => Ok(OperationLock { _file: file }), + Err(TryLockError::WouldBlock) => { + Err(anyhow!("数据操作锁被占用: {}", lock_path.display())) + } + Err(TryLockError::Error(error)) => Err(error) + .with_context(|| format!("获取数据操作锁失败: {}", lock_path.display())), + } +} + +pub(super) fn acquire(data_path: &Path, timeout: Duration) -> Result { + let (file, lock_path) = open_lock_file(data_path)?; + let started = Instant::now(); + let mut reported_wait = false; + loop { + match file.try_lock() { + Ok(()) => return Ok(OperationLock { _file: file }), + Err(TryLockError::WouldBlock) => { + if started.elapsed() >= timeout { + return Err(anyhow!( + "等待其他 fojin 数据操作超时: {}", + lock_path.display() + )); + } + if !reported_wait { + eprintln!("检测到另一个 fojin 数据操作,正在等待..."); + reported_wait = true; + } + let remaining = timeout.saturating_sub(started.elapsed()); + std::thread::sleep(POLL_INTERVAL.min(remaining)); + } + Err(TryLockError::Error(error)) => { + return Err(error) + .with_context(|| format!("获取数据操作锁失败: {}", lock_path.display())); + } + } + } +} +``` + +文件顶部的 `use` 需要 `PathBuf`: + +```rust +use std::path::{Path, PathBuf}; +``` + +- [ ] **Step 4: 运行,确认通过** + +Run: `rustup run 1.96.0 cargo test --lib operation_lock` +Expected: 2 passed(既有的 `competing_lock_times_out_then_succeeds_after_drop` 与新增的)。 + +- [ ] **Step 5: 全量 + clippy,提交** + +Run: `rustup run 1.96.0 cargo test && rustup run 1.96.0 cargo clippy --all-targets -- -D warnings` + +```bash +git add src/data/operation_lock.rs +git commit -m "feat(data): 新增不等待、不打印的 try_acquire 供查询路径使用" +``` + +--- + +### Task 2: `cite` 缺失时懒建索引 + +**Files:** +- Modify: `src/data.rs` +- Modify: `src/cli.rs` +- Test: `tests/command.rs`(追加) + +**Interfaces:** +- Consumes: `operation_lock::try_acquire`(Task 1) +- Produces: + - `pub const CBETA_INDEX_NAME: &str = "idx_parallels_cbeta"` + - `pub fn ensure_cbeta_index(conn: &rusqlite::Connection)` —— 尽力而为,无返回值,永不 panic、永不传播错误 + - `fn create_cbeta_index(conn: &rusqlite::Connection) -> rusqlite::Result<()>`(私有,Task 3 复用) + +**关键实现顺序:先以读写方式打开,成功后才打印提示。** 打开读写是最可靠且最便宜的可写性判据(SQLite 需要**所在目录**可写以创建回滚日志,光看文件权限不够)。把打开放在打印之前,只读安装就**永远不会**看到任何提示,而能建索引的用户能看到那 1–2 秒停顿的解释。 + +- [ ] **Step 1: 写失败测试** + +追加到 `tests/command.rs`: + +```rust +fn write_cite_fixture(dir: &std::path::Path) { + let db_path = dir.join("data.sqlite"); + let conn = Connection::open(db_path).unwrap(); + init_schema(&conn).unwrap(); + for (k, v) in [("version", "v1"), ("norm_ruleset", "t2s-char-1to1-v1")] { + conn.execute( + "INSERT INTO meta(key,value) VALUES (?1,?2)", + rusqlite::params![k, v], + ) + .unwrap(); + } + for (zt, zn, lang, f, juan) in [ + ("色即是空", "色即是空", "sa", "rūpaṃ śūnyatā", 1), + ("受想行識", "受想行识", "bo", "gzugs stong pa", 2), + ] { + conn.execute( + "INSERT INTO parallels(zh_text,zh_norm,foreign_lang,foreign_text,confidence,cbeta_id,title_zh,juan_num) + VALUES (?1,?2,?3,?4,0.9,'T0251','心經',?5)", + rusqlite::params![zt, zn, lang, f, juan], + ) + .unwrap(); + } +} + +fn index_exists(dir: &std::path::Path) -> bool { + let conn = Connection::open(dir.join("data.sqlite")).unwrap(); + conn.query_row( + "SELECT COUNT(*) FROM sqlite_master WHERE type='index' AND name='idx_parallels_cbeta'", + [], + |row| row.get::<_, i64>(0), + ) + .unwrap() + > 0 +} + +#[test] +fn cite_builds_the_missing_index_and_output_is_unchanged() { + let dir = tempfile::tempdir().unwrap(); + write_cite_fixture(dir.path()); + assert!(!index_exists(dir.path()), "fixture must start without the index"); + + let first = std::process::Command::new(env!("CARGO_BIN_EXE_fojin")) + .args(["cite", "T0251", "--offline", "--data-dir"]) + .arg(dir.path()) + .output() + .unwrap(); + assert_eq!(first.status.code(), Some(0)); + assert!(index_exists(dir.path()), "cite must have built the index"); + + // Second run: index already present, so no rebuild and no notice. This is + // also the idempotency check — `ensure_cbeta_index` returns before it even + // opens the file read-write once the index exists. + let second = std::process::Command::new(env!("CARGO_BIN_EXE_fojin")) + .args(["cite", "T0251", "--offline", "--data-dir"]) + .arg(dir.path()) + .output() + .unwrap(); + assert_eq!( + String::from_utf8(first.stdout).unwrap(), + String::from_utf8(second.stdout).unwrap() + ); + assert!( + !String::from_utf8(second.stderr).unwrap().contains("建立索引"), + "the notice must appear only when actually building" + ); +} + +#[test] +fn a_held_lock_skips_the_build_and_leaves_results_identical() { + // Holding the operation lock forces `cite` down the un-indexed path, which + // is the only portable way to compare indexed vs un-indexed output — the + // first run of the previous test already has the index by the time it + // queries, so it cannot make this comparison. + let indexed_dir = tempfile::tempdir().unwrap(); + write_cite_fixture(indexed_dir.path()); + let indexed = std::process::Command::new(env!("CARGO_BIN_EXE_fojin")) + .args(["cite", "T0251", "--offline", "--data-dir"]) + .arg(indexed_dir.path()) + .output() + .unwrap(); + assert!(index_exists(indexed_dir.path())); + + let plain_dir = tempfile::tempdir().unwrap(); + write_cite_fixture(plain_dir.path()); + let lock_file = std::fs::OpenOptions::new() + .read(true) + .write(true) + .create(true) + .truncate(false) + .open(plain_dir.path().join("data.sqlite.lock")) + .unwrap(); + lock_file.try_lock().expect("test must own the lock"); + + let plain = std::process::Command::new(env!("CARGO_BIN_EXE_fojin")) + .args(["cite", "T0251", "--offline", "--data-dir"]) + .arg(plain_dir.path()) + .output() + .unwrap(); + drop(lock_file); + + assert_eq!(plain.status.code(), Some(0), "a busy lock must not fail the query"); + assert!( + !index_exists(plain_dir.path()), + "a busy lock must skip the build rather than wait for it" + ); + assert_eq!( + String::from_utf8(indexed.stdout).unwrap(), + String::from_utf8(plain.stdout).unwrap(), + "the index changes speed only — never results" + ); +} + +#[test] +fn parallel_does_not_build_the_cite_index() { + let dir = tempfile::tempdir().unwrap(); + write_cite_fixture(dir.path()); + + let output = std::process::Command::new(env!("CARGO_BIN_EXE_fojin")) + .args(["parallel", "色即是空", "--offline", "--data-dir"]) + .arg(dir.path()) + .output() + .unwrap(); + + assert_eq!(output.status.code(), Some(0)); + assert!( + !index_exists(dir.path()), + "only the cite path needs this index; parallel must not pay for it" + ); +} + +#[cfg(unix)] +#[test] +fn cite_degrades_gracefully_when_the_data_dir_is_read_only() { + use std::os::unix::fs::PermissionsExt; + + let dir = tempfile::tempdir().unwrap(); + write_cite_fixture(dir.path()); + let original = std::fs::metadata(dir.path()).unwrap().permissions(); + std::fs::set_permissions(dir.path(), std::fs::Permissions::from_mode(0o500)).unwrap(); + + let output = std::process::Command::new(env!("CARGO_BIN_EXE_fojin")) + .args(["cite", "T0251", "--offline", "--data-dir"]) + .arg(dir.path()) + .output() + .unwrap(); + + // Restore before asserting so a failure still leaves a removable tempdir. + std::fs::set_permissions(dir.path(), original).unwrap(); + + assert_eq!( + output.status.code(), + Some(0), + "a read-only data dir must not fail the query: {}", + String::from_utf8_lossy(&output.stderr) + ); + assert!(String::from_utf8(output.stdout).unwrap().contains("色即是空")); +} +``` + +- [ ] **Step 2: 运行,确认失败** + +Run: `rustup run 1.96.0 cargo test --test command` +Expected: `cite_builds_the_missing_index_and_output_is_unchanged` FAIL —— 索引未被创建。另两条会通过(尚未有任何建索引逻辑)。 + +- [ ] **Step 3: 在 `src/data.rs` 实现** + +在 `EXPECTED_NORM_RULESET` 常量之后加入: + +```rust +/// Name of the local `cite` index. Public so tests and diagnostics can look it +/// up by the same string the builder uses. +pub const CBETA_INDEX_NAME: &str = "idx_parallels_cbeta"; + +/// `cite` filters on `cbeta_id = ?1 COLLATE NOCASE` and orders by +/// `juan_num, id`. The NOCASE collation on the leading column is required: +/// a BINARY index cannot serve that comparison as an equality seek and +/// degrades to an index scan. +/// +/// Deliberately NOT in `schema.sql`: that file is the compatibility reference +/// (`validate_compatibility` checks against it) and is executed verbatim by +/// the Python export pipeline. Every already-downloaded dataset lacks this +/// index and must stay fully valid, so it can never be a compatibility +/// requirement. +const CBETA_INDEX_DDL: &str = "CREATE INDEX IF NOT EXISTS idx_parallels_cbeta \ + ON parallels(cbeta_id COLLATE NOCASE, juan_num, id)"; + +fn create_cbeta_index(conn: &rusqlite::Connection) -> rusqlite::Result<()> { + conn.execute_batch(CBETA_INDEX_DDL) +} + +fn cbeta_index_exists(conn: &rusqlite::Connection) -> rusqlite::Result { + conn.query_row( + "SELECT 1 FROM sqlite_master WHERE type = 'index' AND name = ?1", + [CBETA_INDEX_NAME], + |_| Ok(true), + ) + .optional() + .map(|found| found.unwrap_or(false)) +} + +/// Best-effort local optimization for `cite`. Without this index the query +/// scans all 908,620 rows; with it, SQLite seeks directly. +/// +/// Every failure degrades to that scan: a read-only filesystem, a busy data +/// operation, or any SQLite error leaves the query correct and merely slower. +/// Nothing here can fail the caller. +pub fn ensure_cbeta_index(conn: &rusqlite::Connection) { + let Some(path) = conn.path().map(Path::to_path_buf) else { + return; // no backing file (in-memory) + }; + if path.as_os_str().is_empty() { + return; + } + // On a read error, assume present rather than risk a pointless rebuild. + if cbeta_index_exists(conn).unwrap_or(true) { + return; + } + // Never wait: a background `data update`/`clean` means we simply scan. + let Ok(_lock) = operation_lock::try_acquire(&path) else { + return; + }; + // Open read-write BEFORE announcing. Opening is the only reliable + // writability signal — SQLite needs the containing directory writable for + // its rollback journal, which file permissions alone do not tell us — and + // doing it first keeps a read-only install completely silent. + let Ok(writable) = rusqlite::Connection::open_with_flags( + &path, + OpenFlags::SQLITE_OPEN_READ_WRITE, + ) else { + return; + }; + eprintln!("首次按经号查询,正在建立索引(一次性,约 1-2 秒,数据目录约 +17 MB)..."); + let _ = create_cbeta_index(&writable); +} +``` + +- [ ] **Step 4: 在 `src/cli.rs` 的 `Command::Cite` 分支接上** + +在 `Command::Cite` 分支中,把 + +```rust + let conn = open_ensured(data_dir, offline)?; + let groups_all = query::by_cbeta(&conn, cbeta_id.trim(), juan, langs.as_deref(), top)?; +``` + +替换为 + +```rust + let conn = open_ensured(data_dir, offline)?; + data::ensure_cbeta_index(&conn); + let groups_all = query::by_cbeta(&conn, cbeta_id.trim(), juan, langs.as_deref(), top)?; +``` + +只加在 `Cite` 分支。`parallel` 与 `texts` 用不上这个索引,不应为它付检测成本。 + +- [ ] **Step 5: 运行,确认通过** + +Run: `rustup run 1.96.0 cargo test --test command --test golden` +Expected: 全部 PASS,包括三条新测试。 + +- [ ] **Step 6: 全量 + clippy,提交** + +Run: `rustup run 1.96.0 cargo test && rustup run 1.96.0 cargo clippy --all-targets -- -D warnings` + +```bash +git add src/data.rs src/cli.rs tests/command.rs +git commit -m "perf(cite): 缺失时懒建 cbeta_id 索引,失败静默退回扫描" +``` + +--- + +### Task 3: 安装与更新时建索引 + +**Files:** +- Modify: `src/data.rs` +- Test: `tests/data.rs`(追加) + +**Interfaces:** +- Consumes: `create_cbeta_index`、`CBETA_INDEX_NAME`(Task 2) + +在候选文件上建索引,让新装用户与 `data update` 之后的用户不必经过懒建路径。位置在 `verify_dataset_file` 通过之后、原子替换之前:失败时走既有的 `candidate.cleanup_with`,活跃数据不受影响。 + +- [ ] **Step 1: 写失败测试** + +追加到 `tests/data.rs`。把 `CBETA_INDEX_NAME` 加进顶部的 `use fojin_cli::data::{...}` 列表。 + +这条测试沿用同文件 `ensure_data_downloads_verifies_and_unpacks` 的本地 HTTP server 模式(以及既有的 `gzip_bytes` / `replacement_database_bytes` / `sha256_hex` 三个 helper),走完整的下载 → 校验 → 解压 → 发布路径: + +```rust +#[test] +fn install_leaves_the_cite_index_in_place() { + let gz = gzip_bytes(&replacement_database_bytes()); + let sha = sha256_hex(&gz); + + let listener = std::net::TcpListener::bind("127.0.0.1:0").unwrap(); + let port = listener.local_addr().unwrap().port(); + let body = gz.clone(); + let server = std::thread::spawn(move || { + let (mut stream, _) = listener.accept().unwrap(); + let mut req = [0u8; 4096]; + let _ = std::io::Read::read(&mut stream, &mut req); + let head = format!( + "HTTP/1.1 200 OK\r\nContent-Length: {}\r\nContent-Type: application/gzip\r\nConnection: close\r\n\r\n", + body.len() + ); + stream.write_all(head.as_bytes()).unwrap(); + stream.write_all(&body).unwrap(); + }); + + let dir = tempfile::tempdir().unwrap(); + let path = dir.path().join("data.sqlite"); + let source = DataSource { + url: &format!("http://127.0.0.1:{port}/data.gz"), + sha256: &sha, + }; + ensure_data(&path, false, &source).unwrap(); + server.join().unwrap(); + + let conn = open_read_only_db(&path).unwrap(); + let present: i64 = conn + .query_row( + "SELECT COUNT(*) FROM sqlite_master WHERE type='index' AND name=?1", + [CBETA_INDEX_NAME], + |row| row.get(0), + ) + .unwrap(); + assert_eq!(present, 1, "a fresh install must ship the cite index"); + + // The index must not disturb the compatibility contract. + verify_dataset_file(&path).unwrap(); +} +``` + +- [ ] **Step 2: 运行,确认失败** + +Run: `rustup run 1.96.0 cargo test --test data install_leaves_the_cite_index_in_place` +Expected: FAIL,`assertion \`left == right\` failed: a fresh install must ship the cite index`(left: 0)。 + +- [ ] **Step 3: 实现** + +在 `src/data.rs` 的 `install_candidate` 中,`verify_dataset_file` 之后、同步之前插入建索引: + +```rust +fn install_candidate(path: &Path, source: &DataSource<'_>) -> Result<()> { + let candidate = transfer::stage_candidate(path, source, transfer::PRODUCTION_POLICY)?; + if let Err(error) = verify_dataset_file(candidate.path()).map(|_| ()) { + return Err(candidate.cleanup_with(error)); + } + // Build the cite index on the candidate so a fresh install never has to + // take the lazy path. ~1.6 s against a 183 MB download. + if let Err(error) = rusqlite::Connection::open_with_flags( + candidate.path(), + OpenFlags::SQLITE_OPEN_READ_WRITE, + ) + .map_err(anyhow::Error::from) + .and_then(|conn| create_cbeta_index(&conn).map_err(anyhow::Error::from)) + .with_context(|| format!("为候选数据建立索引失败: {}", candidate.path().display())) + { + return Err(candidate.cleanup_with(error)); + } + if let Err(error) = std::fs::OpenOptions::new() + .read(true) + .write(true) + .open(candidate.path()) + .and_then(|file| file.sync_all()) + .with_context(|| format!("同步候选数据失败: {}", candidate.path().display())) + { + return Err(candidate.cleanup_with(error)); + } + let candidate_path = candidate.path().to_path_buf(); + finish_replacement(candidate, replace_with_candidate(path, &candidate_path)) +} +``` + +- [ ] **Step 4: 运行,确认通过** + +Run: `rustup run 1.96.0 cargo test --test data` +Expected: 全部 PASS。 + +- [ ] **Step 5: 全量 + clippy,提交** + +Run: `rustup run 1.96.0 cargo test && rustup run 1.96.0 cargo clippy --all-targets -- -D warnings` + +```bash +git add src/data.rs tests/data.rs +git commit -m "perf(data): 安装与更新时为候选数据建立 cite 索引" +``` + +--- + +### Task 4: `confidence` 为 1.00 时不显示标签 + +**Files:** +- Modify: `src/render.rs` +- Test: `tests/render.rs`(追加) + +**Interfaces:** +- 无新公开签名。`conf_tag` 是 `render.rs` 私有函数,仅行为改变。 + +数据集全部 908,620 行的 `confidence` 都是 1.0,所以每行末尾的 `[MITRA 1.00]` 不携带任何信息。改为**按显示精度**判断:格式化成两位小数后若为 `1.00` 则不输出标签。按格式化结果而非原始值比较,避免 0.995 这类值既显示成 `[MITRA 1.00]` 又被判定为"有信息量"的自相矛盾。 + +JSON 的 `confidence` 字段与 `group_and_rank` / `cap_per_lang` 的排序逻辑**一律不动**:前者是 agent 契约,后者对未来有真实分数的数据是正确的。 + +- [ ] **Step 1: 写失败测试** + +追加到 `tests/render.rs`: + +```rust +#[test] +fn perfect_confidence_shows_no_tag() { + let group = MatchGroup { + zh_text: "色即是空".into(), + cbeta_id: Some("T0251".into()), + title_zh: Some("心經".into()), + juan_num: Some(1), + parallels: vec![Parallel { + lang: "sa".into(), + text: "rūpaṃ śūnyatā".into(), + confidence: Some(1.0), + }], + }; + let out = render_human(&[group], None, 0); + assert!(out.contains("梵 rūpaṃ śūnyatā"), "the parallel itself stays: {out}"); + assert!( + !out.contains("MITRA"), + "a uniform 1.00 carries no information and must not be printed: {out}" + ); +} + +#[test] +fn imperfect_confidence_still_shows_the_tag() { + let group = MatchGroup { + zh_text: "色即是空".into(), + cbeta_id: Some("T0251".into()), + title_zh: Some("心經".into()), + juan_num: Some(1), + parallels: vec![Parallel { + lang: "sa".into(), + text: "rūpaṃ śūnyatā".into(), + confidence: Some(0.87), + }], + }; + let out = render_human(&[group], None, 0); + assert!(out.contains("梵 rūpaṃ śūnyatā [MITRA 0.87]"), "got: {out}"); +} + +#[test] +fn confidence_rounding_to_one_hides_the_tag() { + // 0.995 formats as "1.00"; printing `[MITRA 1.00]` while calling the value + // informative would contradict itself, so the rule keys on what would be + // displayed, not on the raw float. + let group = MatchGroup { + zh_text: "色即是空".into(), + cbeta_id: Some("T0251".into()), + title_zh: Some("心經".into()), + juan_num: Some(1), + parallels: vec![Parallel { + lang: "sa".into(), + text: "rūpaṃ śūnyatā".into(), + confidence: Some(0.995), + }], + }; + let out = render_human(&[group], None, 0); + assert!(!out.contains("MITRA"), "got: {out}"); +} + +#[test] +fn absent_confidence_shows_no_tag() { + let group = MatchGroup { + zh_text: "色即是空".into(), + cbeta_id: Some("T0251".into()), + title_zh: Some("心經".into()), + juan_num: Some(1), + parallels: vec![Parallel { + lang: "sa".into(), + text: "rūpaṃ śūnyatā".into(), + confidence: None, + }], + }; + let out = render_human(&[group], None, 0); + assert!(!out.contains("MITRA"), "got: {out}"); +} + +#[test] +fn json_still_carries_confidence_when_the_tag_is_hidden() { + let group = MatchGroup { + zh_text: "色即是空".into(), + cbeta_id: Some("T0251".into()), + title_zh: Some("心經".into()), + juan_num: Some(1), + parallels: vec![Parallel { + lang: "sa".into(), + text: "rūpaṃ śūnyatā".into(), + confidence: Some(1.0), + }], + }; + let out = render_json(&[group], 1); + let v: serde_json::Value = serde_json::from_str(&out).unwrap(); + assert_eq!( + v["groups"][0]["parallels"][0]["confidence"], + serde_json::json!(1.0), + "the JSON contract is unchanged; only the human tag is suppressed" + ); +} +``` + +- [ ] **Step 2: 运行,确认失败** + +Run: `rustup run 1.96.0 cargo test --test render` +Expected: `perfect_confidence_shows_no_tag`、`confidence_rounding_to_one_hides_the_tag` FAIL(输出里仍有 `[MITRA 1.00]`);其余通过。 + +- [ ] **Step 3: 实现** + +把 `src/render.rs` 的 `conf_tag` 替换为: + +```rust +/// The tag appears only when it would show a number other than 1.00. +/// +/// Keying on the formatted string rather than the raw float keeps the rule +/// self-consistent: a value like 0.995 would render as "1.00", so treating it +/// as informative would print a tag that says exactly what we suppress +/// elsewhere. Absence of the tag reads as "no caveat". +fn conf_tag(c: Option) -> String { + match c { + Some(v) => { + let shown = format!("{v:.2}"); + if shown == "1.00" { + String::new() + } else { + format!(" [MITRA {shown}]") + } + } + None => String::new(), + } +} +``` + +- [ ] **Step 4: 运行,确认通过** + +Run: `rustup run 1.96.0 cargo test --test render --test golden --test cli` +Expected: 全部 PASS。既有断言用的都是 0.91 / 0.88 / 0.75,均 < 1.00,不受影响 —— 包括 `tests/golden.rs` 的逐字节守卫。 + +- [ ] **Step 5: 全量 + clippy,提交** + +Run: `rustup run 1.96.0 cargo test && rustup run 1.96.0 cargo clippy --all-targets -- -D warnings` + +```bash +git add src/render.rs tests/render.rs +git commit -m "feat(render): 置信度显示为 1.00 时不再输出标签" +``` + +--- + +### Task 5: 真实数据验证与文档 + +**Files:** +- Modify: `README.md` +- Modify: `CHANGELOG.md` + +- [ ] **Step 1: 真实数据冒烟(必须实测,不得跳过)** + +数据集是只读资源。**先复制到临时目录再测,不要直接对 `~/.cache/fojin/data.sqlite` 操作,也不要运行 `fojin data clean` / `data update`。** + +```bash +SP=/tmp/claude-1000/-home-lqsxi-projects-fojin-cli/88b0dd32-1d6d-4140-a736-9dae215864d7/scratchpad/verify +mkdir -p "$SP" && cp ~/.cache/fojin/data.sqlite "$SP/" +rustup run 1.96.0 cargo build --release +B=./target/release/fojin + +# 1) 首次 cite:应打印一次建索引提示,并明显变慢一次 +time $B cite T0251 --limit 3 --offline --data-dir "$SP" + +# 2) 再次 cite:无提示,应为毫秒级 +time $B cite T0251 --limit 3 --offline --data-dir "$SP" + +# 3) 另一部经,确认索引对所有 cbeta_id 生效 +time $B cite T0220 --limit 2 --offline --data-dir "$SP" + +# 4) --juan 走复合前缀 +time $B cite T0251 --juan 1 --limit 2 --offline --data-dir "$SP" + +# 5) confidence:真实数据全为 1.0,输出中不应再出现 MITRA 标签 +$B parallel "色即是空" --offline --data-dir "$SP" | grep -c "MITRA" || echo "0 处 MITRA(符合预期)" + +# 6) 索引不改变结果:与一份保持未建索引的副本对比同一查询。 +# 不能拿 ~/.cache 那份做对照 —— cite 会就地把索引建起来, +# 而且那是用户的真实数据,不该被本次验证改动。 +# 用只读目录强制建索引静默失败,从而保持无索引状态: +mkdir -p "$SP/plain" && cp ~/.cache/fojin/data.sqlite "$SP/plain/" +chmod 500 "$SP/plain" +$B cite T0251 --limit 3 --offline --data-dir "$SP/plain" > "$SP/no-index.txt" +chmod 700 "$SP/plain" +$B cite T0251 --limit 3 --offline --data-dir "$SP" > "$SP/with-index.txt" +diff "$SP/no-index.txt" "$SP/with-index.txt" && echo "有无索引输出完全一致" + +rm -rf "$SP" +``` + +Expected:第 1 步打印提示且耗时约 1–3 s;第 2/3/4 步无提示且为毫秒级;第 5 步为 0 处;第 6 步 diff 无差异,且只读目录那次不打印任何提示、退出码为 0(顺带在真实数据上验证了降级路径)。**把实际输出与耗时记录到报告中。** + +- [ ] **Step 2: 更新 README** + +(a) 第 15–16 行的首屏示例输出去掉 ` [MITRA 1.00]`: + +```markdown +梵 śūnyat'aiva rūpaṃ, rūpān na pṛthak śūnyatā … +藏 གཟུགས་ལས་སྟོང་པ་ཉིད་གཞན་མ་ཡིན༏ … +``` + +(b) 在「其他子命令」一节 `fojin cite` 的说明之后补一句: + +```markdown +首次运行 `fojin cite` 会为按经号查询建立一次本地索引(约 1–2 秒,数据目录增加约 17 MB), +之后按经号查询为毫秒级。数据目录不可写时会跳过建索引,查询结果不受影响,只是较慢。 +``` + +(c) 在「数据集」一节补一条,说明置信度: + +```markdown +- 当前数据集(`data-v1`)所有对齐的置信度均为 1.00,不具区分度,因此人类可读输出不再逐行标注; + `--json` 的 `confidence` 字段保持输出。未来数据若提供真实分数,低于 1.00 的会自动显示。 +``` + +- [ ] **Step 3: 更新 CHANGELOG** + +在 `## [0.3.0] - Unreleased` 的列表中追加: + +```markdown +- 查询性能:`cite` 现在使用本地建立的 `cbeta_id` 索引,不再全表扫描;索引在安装/更新时建立,已有数据在首次 `cite` 时补建,数据目录不可写时静默退回扫描。 +- 输出可读性:置信度显示为 `1.00` 时不再逐行标注 `[MITRA 1.00]`(当前数据集全部如此);`--json` 的 `confidence` 字段不变。 +``` + +- [ ] **Step 4: 最终验证并提交** + +Run: +```bash +rustup run 1.95.0 cargo test --all --locked +rustup run 1.96.0 cargo fmt --all --check +rustup run 1.96.0 cargo clippy --all-targets --locked -- -D warnings +``` +Expected:全部通过,测试数应为 213 + 本轮新增。 + +```bash +git add README.md CHANGELOG.md +git commit -m "docs: 记录 cite 索引与置信度显示变更" +``` diff --git a/docs/superpowers/specs/2026-07-26-index-and-confidence-design.md b/docs/superpowers/specs/2026-07-26-index-and-confidence-design.md new file mode 100644 index 0000000..d725089 --- /dev/null +++ b/docs/superpowers/specs/2026-07-26-index-and-confidence-design.md @@ -0,0 +1,168 @@ +# cite 索引与 confidence 显示设计 + +日期:2026-07-26 +状态:已批准,待实施 + +## 背景 + +两个独立的小改进,共同点是都在修正"实测暴露、但没人报告过"的问题。 + +**1. `cite` 是全表扫描。** `parallels` 表除 FTS 外没有任何索引,`cite` 的 `WHERE cbeta_id = ?1 COLLATE NOCASE` 因此扫描全部 908,620 行(588 MB)。 + +**2. `confidence` 全库恒为 1.0。** 908,620 行无一例外。于是人类输出里每一行末尾的 `[MITRA 1.00]` 是纯噪音,而 `group_and_rank` 按 `max_conf` 排序、`cap_per_lang` 按置信度降序,都是在做无区分度的比较。 + +## 实测依据 + +| 场景 | 无索引 | 有索引 | +| --- | --- | --- | +| `cite T0251`(端到端) | 2.34 s | ~0 ms | +| `cite T0220`(端到端) | 0.19 s | ~0 ms | +| 索引构建耗时 | — | 1.60 s | +| 数据文件增量 | — | +17.2 MB | + +查询计划:无索引为 `SCAN parallels` + `USE TEMP B-TREE FOR ORDER BY`; +BINARY 索引仅得到 `SCAN parallels USING INDEX`(仍非等值查找,因为比较带 `COLLATE NOCASE`); +NOCASE 索引得到 `SEARCH parallels USING INDEX idx_parallels_cbeta (cbeta_id=?)`, +带 `--juan` 时为 `(cbeta_id=? AND juan_num=?)`,且 ORDER BY 的临时 B 树消失。 + +`confidence` 分布:`SELECT ROUND(confidence,1), COUNT(*) FROM parallels GROUP BY 1` → 仅一行 `(1.0, 908620)`。 + +## 已确定的决策 + +| 决策点 | 结论 | 理由 | +| --- | --- | --- | +| 索引创建时机 | **安装/更新时建 + `cite` 路径缺失时懒建** | 现有用户本地已有 588 MB 数据,不应为一个 17 MB 的索引重下 183 MB。 | +| 索引 DDL 的位置 | **Rust 侧常量,不进 `schema.sql`** | 见下节两条理由。 | +| `texts` 是否一并加索引 | **不做** | 0.913→0.121 s 要付 +34 MB,性价比远不如 cite;且 `texts` 真正的病因是把全部分组取回 Rust 再按归一化标题过滤,预聚合小表才是根治,那需要重新导出数据。 | +| `confidence` 处理 | **显示精度为 `1.00` 时不输出标签** | 当前数据集下噪音清零,未来有真实分数时自动显现,无需再改代码。 | +| JSON 的 `confidence` | **一字不动** | agent 契约的一部分,改动会破坏现有消费者。 | +| 排序逻辑 | **一字不动** | 现在是空转,但对未来有真实分数的数据是正确的,删掉是退步。 | + +## 索引设计 + +### DDL + +```sql +CREATE INDEX IF NOT EXISTS idx_parallels_cbeta + ON parallels(cbeta_id COLLATE NOCASE, juan_num, id) +``` + +`COLLATE NOCASE` 不可省略。`cite` 的比较是 `cbeta_id = ?1 COLLATE NOCASE`,BINARY 排序规则的索引无法用于该比较的等值查找,实测只能退化成索引扫描。 + +列顺序 `(cbeta_id, juan_num, id)` 对应 `by_cbeta` 的 `WHERE cbeta_id = ?` [`AND juan_num = ?`] `ORDER BY juan_num, id`,使过滤与排序都由索引直接满足。 + +### 为什么不写进 `schema.sql` + +1. **Python 导出管线直接执行该文件**(`export_parallels.py` 读 `SCHEMA_PATH` 并 `executescript`)。索引若在 schema 里,会在 908,620 行插入**之前**建立,显著拖慢导出。 +2. **`schema.sql` 是兼容性校验的参照物**(`validate_compatibility` 依它检查必需的表与 FTS 声明)。索引绝不能成为兼容性要求——所有已下载的数据文件都没有它,把它列入参照物会让"缺索引"看起来像"数据不兼容"。 + +索引是对已下载产物的**本地优化**,不是发布 schema 的一部分。DDL 作为常量定义在 `src/data.rs`。 + +### 创建路径一:安装与更新 + +在 `install_candidate` 中,于**候选文件**上创建索引——即在 `verify_dataset_file` 通过之后、原子替换之前。这样: + +- 失败时候选文件被清理,活跃数据不受影响(沿用既有的 `candidate.cleanup_with` 路径)。 +- 替换是原子的,不存在"索引建到一半的活跃库"。 +- 成本 1.6 s,相对 183 MB 下载可忽略。 + +`data update` 与 `data clean` 后的重新安装都走这条路径,因此更新后索引依然存在。 + +### 创建路径二:`cite` 缺失时懒建 + +只挂在 `cite` 上。`parallel` 与 `texts` 用不到这个索引,不应为它付检测成本。 + +流程: + +1. 用**已经打开的只读连接**查一次 `sqlite_master`,判断索引是否存在(亚毫秒,不额外开连接)。 +2. 存在则直接继续查询。 +3. 缺失则尝试构建: + a. `operation_lock::try_acquire`(新增:单次 `try_lock`,不等待、不打印)。拿不到说明有 `data update`/`clean` 在跑 —— **跳过,用全表扫描**。 + b. 向 stderr 打一行一次性提示,说明正在建立索引及大致耗时。 + c. 另开读写连接执行 DDL,完成后关闭,释放锁。 +4. 继续用原只读连接查询。SQLite 会在后续语句准备时看到新索引。 + +`operation_lock` 现有的 `acquire` 会等待最长 20 分钟并在首次等待时向 stderr 打印提示 —— 这两者对查询路径都是错的,故新增 `try_acquire`。 + +### 降级是硬要求 + +只读文件系统、权限不足、磁盘空间不足、锁被占用、任何 SQLite 错误 —— **一律静默退回全表扫描,绝不让查询失败**。索引是优化,不是功能前提。 + +注意其中一种容易漏掉的失败:以读写方式打开 SQLite 需要**所在目录**可写(要创建回滚日志),而不只是数据文件本身可写。因此"文件可写但目录只读"也必须落入降级路径,不能假定文件权限足以代表可写性。 + +唯一允许的输出是步骤 3(b) 的一次性 stderr 提示(让用户理解那一次 1–2 秒的停顿)。构建**失败**时不打印任何东西:用户没要求建索引,失败对他没有可操作性,而查询结果完全正确。 + +`--json` 模式下 stdout 仍是纯 JSON —— 提示走 stderr,契约不变。 + +### 与既有机制的关系 + +- `validate_compatibility`:只检查必需的表与 FTS 声明,多一个索引不影响。 +- `verify_dataset`(`PRAGMA quick_check`)与 `verify_dataset_file`(FTS integrity-check):多一个索引不影响。 +- `schema.sql` 的 "write-once artifact" 注释针对的是 `UPDATE`/`DELETE` 会让 FTS 索引失步。创建索引不写 `parallels` 的任何行,不涉及该风险。 +- 数据文件的 SHA-256 只在下载的压缩包上校验,安装后不再校验文件本身,故本地增建索引不破坏任何校验契约。 + +## confidence 设计 + +`render::conf_tag` 改为按**显示精度**判断: + +```rust +fn conf_tag(c: Option) -> String { + match c { + Some(v) => { + let shown = format!("{v:.2}"); + if shown == "1.00" { + String::new() + } else { + format!(" [MITRA {shown}]") + } + } + None => String::new(), + } +} +``` + +按格式化后的字符串而非原始值比较,是为了避免 0.995 这类值既显示成 `[MITRA 1.00]`、又被判定为"有信息量"的自相矛盾。规则因此可以一句话说清:**标签只在它能显示出非 1.00 的数字时才出现。** + +标签的**缺席**读作"无保留",语义自然,不需要额外解释。 + +不改动的部分: + +- JSON 的 `confidence` 字段照常输出。 +- `group_and_rank` 的 `max_conf` 排序键、`cap_per_lang` 的置信度降序,均保持原样。 + +## 测试策略 + +**索引** + +- 单元:DDL 常量能在内存库上成功执行且幂等(连跑两次)。 +- 集成(临时目录 + 真实文件): + - 索引缺失时,`cite` 触发构建,构建后索引存在,查询结果与构建前**完全一致**。 + - 索引已存在时不重复构建、不打印提示。 + - 数据文件设为只读时,`cite` 仍返回正确结果、退出码 0、不报错(降级)。 + - 锁被占用时跳过构建、查询照常成功。 + - 安装完成后的数据带索引。`tests/data.rs` 已有走通 `ensure_data` 成功路径的测试,在其基础上追加断言即可;若该路径不便复用,则新增一条。 +- 关键不变式:**索引的存在与否不改变任何查询结果**,只改变速度。需要一条测试同时对有/无索引的库跑同一 `cite` 查询并断言输出逐字节相同。 + +**confidence** + +- `conf_tag` 的直接测试:`1.0` → 空;`0.87` → ` [MITRA 0.87]`;`0.995` → 空(显示精度为 1.00);`None` → 空。 +- 渲染层:一组置信度全为 1.0 的分组,输出中不含 `[MITRA`。 +- 现有测试全部使用 <1.0 的值(0.91 / 0.88 / 0.75),预期零破坏,包括 `tests/golden.rs`。 + +**真实数据冒烟** + +- `cite T0251` 与 `cite T0220` 在索引前后的耗时与输出对比。 +- `parallel "色即是空"` 确认输出中不再出现 `[MITRA 1.00]`。 + +## 文档 + +- README 第 15–16 行的首屏示例输出仍挂着 `[MITRA 1.00]`,必须更新为无标签形式。 +- README 需说明:首次 `cite` 可能有一次约 1–2 秒的索引构建停顿,数据目录随之增加约 17 MB。 +- CHANGELOG 在 `## [0.3.0] - Unreleased` 追加条目,措辞不得暗示已发布。 + +## 明确不在本轮范围 + +- `texts` 的索引或预聚合表。 +- `data status` 展示索引状态。 +- 删除或改变 `confidence` 的排序用途。 +- 重新导出数据(data-v2)。 diff --git a/src/cli.rs b/src/cli.rs index 62138b5..b4961ff 100644 --- a/src/cli.rs +++ b/src/cli.rs @@ -300,6 +300,7 @@ pub fn run() -> Result { return Ok(code); } let conn = open_ensured(data_dir, offline)?; + data::ensure_cbeta_index(&conn); let groups_all = query::by_cbeta(&conn, cbeta_id.trim(), juan, langs.as_deref(), top)?; let total = groups_all.len(); if total == 0 && !json { diff --git a/src/data.rs b/src/data.rs index 7c2bd98..8798880 100644 --- a/src/data.rs +++ b/src/data.rs @@ -13,6 +13,91 @@ static REPLACEMENT_BACKUP_SEQUENCE: std::sync::atomic::AtomicU64 = pub const EXPECTED_DATA_VERSION: &str = "v1"; pub const EXPECTED_NORM_RULESET: &str = "t2s-char-1to1-v1"; +/// Single source of truth for the index name. A macro rather than a `const` +/// because `concat!` needs literals: this is what lets `CBETA_INDEX_DDL` and +/// `CBETA_INDEX_NAME` below be built from the same token, so they cannot +/// drift apart. If they ever did, `CREATE INDEX IF NOT EXISTS` would keep +/// returning `Ok` while `cbeta_index_exists` kept returning `false`, and every +/// single `cite` would take the lock, open read-write and print the notice +/// without building anything — the leak commit `be07b85` fixed, re-armed. +macro_rules! cbeta_index_name { + () => { + "idx_parallels_cbeta" + }; +} + +/// Name of the local `cite` index. Public so tests and diagnostics can look it +/// up by the same string the builder uses. +pub const CBETA_INDEX_NAME: &str = cbeta_index_name!(); + +/// `cite` filters on `cbeta_id = ?1 COLLATE NOCASE` and orders by +/// `juan_num, id`. The NOCASE collation on the leading column is required: +/// a BINARY index cannot serve that comparison as an equality seek and +/// degrades to an index scan. +/// +/// Deliberately NOT in `schema.sql`: that file is the compatibility reference +/// (`validate_compatibility` checks against it) and is executed verbatim by +/// the Python export pipeline. Every already-downloaded dataset lacks this +/// index and must stay fully valid, so it can never be a compatibility +/// requirement. +const CBETA_INDEX_DDL: &str = concat!( + "CREATE INDEX IF NOT EXISTS ", + cbeta_index_name!(), + " ON parallels(cbeta_id COLLATE NOCASE, juan_num, id)" +); + +fn create_cbeta_index(conn: &rusqlite::Connection) -> rusqlite::Result<()> { + conn.execute_batch(CBETA_INDEX_DDL) +} + +fn cbeta_index_exists(conn: &rusqlite::Connection) -> rusqlite::Result { + conn.query_row( + "SELECT 1 FROM sqlite_master WHERE type = 'index' AND name = ?1", + [CBETA_INDEX_NAME], + |_| Ok(true), + ) + .optional() + .map(|found| found.unwrap_or(false)) +} + +/// Best-effort local optimization for `cite`. Without this index the query +/// scans all 908,620 rows; with it, SQLite seeks directly. +/// +/// Every failure degrades to that scan: a read-only filesystem, a busy data +/// operation, or any SQLite error leaves the query correct and merely slower. +/// Nothing here can fail the caller. +pub fn ensure_cbeta_index(conn: &rusqlite::Connection) { + let Some(path) = conn.path().map(PathBuf::from) else { + return; // no backing file (in-memory) + }; + if path.as_os_str().is_empty() { + return; // rusqlite returns `Some("")` for a temporary/in-memory database + } + // On a read error, assume present rather than risk a pointless rebuild. + if cbeta_index_exists(conn).unwrap_or(true) { + return; + } + // Never wait: a background `data update`/`clean` means we simply scan. + let Ok(_lock) = operation_lock::try_acquire(&path) else { + return; + }; + // Writability cannot be reliably predicted before we actually try to + // write: SQLite defers creating its rollback journal until the write + // itself, so a successful read-write `open` (and even a `SELECT`) tells + // us nothing — a directory that forbids new files will still let an + // existing, writable database file open cleanly. So we don't announce + // anything up front; we only report the build after `CREATE INDEX` has + // actually succeeded. + let Ok(writable) = + rusqlite::Connection::open_with_flags(&path, OpenFlags::SQLITE_OPEN_READ_WRITE) + else { + return; + }; + if create_cbeta_index(&writable).is_ok() { + eprintln!("已为按经号查询建立索引(一次性,数据目录增加约 17 MB)。"); + } +} + pub struct DataSource<'a> { pub url: &'a str, pub sha256: &'a str, @@ -145,6 +230,12 @@ pub fn clean_data(path: &Path) -> Result> { if size.is_some() { std::fs::remove_file(path).with_context(|| format!("删除数据失败: {}", path.display()))?; } + // The database this journal belonged to is gone, so the journal is now + // meaningless — and actively harmful: left behind, it would be replayed + // into whatever gets downloaded next. Main file first, sidecars second: + // the reverse order would, if interrupted, leave a half-written database + // with no journal left to repair it. + transfer::remove_sqlite_sidecars(path)?; Ok(size) } @@ -153,6 +244,16 @@ fn install_candidate(path: &Path, source: &DataSource<'_>) -> Result<()> { if let Err(error) = verify_dataset_file(candidate.path()).map(|_| ()) { return Err(candidate.cleanup_with(error)); } + // Build the cite index on the candidate so a fresh install never has to + // take the lazy path. ~1.6 s against a 183 MB download. + if let Err(error) = + rusqlite::Connection::open_with_flags(candidate.path(), OpenFlags::SQLITE_OPEN_READ_WRITE) + .map_err(anyhow::Error::from) + .and_then(|conn| create_cbeta_index(&conn).map_err(anyhow::Error::from)) + .with_context(|| format!("为候选数据建立索引失败: {}", candidate.path().display())) + { + return Err(candidate.cleanup_with(error)); + } if let Err(error) = std::fs::OpenOptions::new() .read(true) .write(true) @@ -162,8 +263,28 @@ fn install_candidate(path: &Path, source: &DataSource<'_>) -> Result<()> { { return Err(candidate.cleanup_with(error)); } + // The replace only renames the main file, so any journal sitting next to + // the live path would survive it and be replayed into the brand-new + // database (`database disk image is malformed` on the first read-write + // open). Clearing it here is safe precisely because the old database is + // about to be replaced wholesale, and because `operation_lock` is held + // for the whole function, excluding concurrent fojin data operations. + // + // One caveat this rationale does not cover on its own: if the replace + // below then fails, we have discarded a journal that `open_read_only_db` + // could otherwise have rolled back, turning a recoverable dataset into an + // unrecoverable one. Accepted, and bounded — the file is a re-downloadable + // cache, and the remedy is re-running the command that just failed. + if let Err(error) = transfer::remove_sqlite_sidecars(path) { + return Err(candidate.cleanup_with(error)); + } let candidate_path = candidate.path().to_path_buf(); - finish_replacement(candidate, replace_with_candidate(path, &candidate_path)) + finish_replacement(candidate, replace_with_candidate(path, &candidate_path))?; + // And again afterwards, so what gets published is always "new database, + // no foreign journal" — whatever appeared during the replacement window + // belongs to the database that just went away. Failing here is louder + // than leaving the freshly installed dataset next to a live landmine. + transfer::remove_sqlite_sidecars(path) } fn sibling_path(path: &Path, suffix: &str) -> Result { @@ -572,11 +693,112 @@ pub fn open_db(path: &Path) -> Result { rusqlite::Connection::open(path).with_context(|| format!("打开数据失败: {}", path.display())) } -pub fn open_read_only_db(path: &Path) -> Result { +fn open_read_only_connection(path: &Path) -> Result { rusqlite::Connection::open_with_flags(path, OpenFlags::SQLITE_OPEN_READ_ONLY) .with_context(|| format!("打开数据失败: {}", path.display())) } +/// Force SQLite to actually touch the file. Opening is lazy: a hot rollback +/// journal is only discovered when the pager takes its first SHARED lock and +/// reads page 1, which is why the failure used to surface far from here — in +/// `require_schema`'s first `prepare`. +fn probe_first_read(conn: &rusqlite::Connection) -> rusqlite::Result<()> { + conn.query_row("SELECT count(*) FROM sqlite_schema", [], |_| Ok(())) +} + +/// A read attempt blocked by a hot rollback journal: the database was left +/// mid-write (an interrupted `CREATE INDEX`, `data update`, power loss…) and +/// rolling that journal back needs write access, which a read-only connection +/// does not have. SQLite reports `SQLITE_READONLY_ROLLBACK`. +/// +/// Only ever consult this on a *read*: a genuine write against a read-only +/// connection carries the same "attempt to write a readonly database" text +/// under the plain `SQLITE_READONLY` code, and that one is not a hot journal. +fn is_hot_journal_error(error: &rusqlite::Error) -> bool { + let rusqlite::Error::SqliteFailure(failure, message) = error else { + return false; + }; + failure.extended_code == rusqlite::ffi::SQLITE_READONLY_ROLLBACK + || message + .as_deref() + .is_some_and(|text| text.contains("attempt to write a readonly database")) +} + +/// Let SQLite roll a hot journal back by itself. +/// +/// The whole discrimination between "interrupted write on a writable dataset" +/// and "genuinely read-only install" is this reopen with +/// `SQLITE_OPEN_READ_WRITE` (deliberately *without* `CREATE`, so self-healing +/// can never conjure an empty database) followed by a real read: +/// +/// * Writable dataset: the reopen succeeds and the first read takes an +/// EXCLUSIVE lock, replays the journal, deletes it — the file is back to its +/// pre-interruption state and the caller's re-opened read-only connection +/// simply works. +/// * Read-only file: the kernel refuses `O_RDWR`, and SQLite *silently* +/// downgrades the open to read-only, so the read fails with +/// `SQLITE_READONLY_ROLLBACK` again. +/// * Read-only directory (writable file): SQLite replays the journal but +/// cannot unlink it, so the read fails with `SQLITE_IOERR_DELETE` and the +/// journal stays hot. +/// +/// Either failure path falls back to the pre-existing error. Neither damages +/// anything: the read-only file is left byte-identical, and the read-only +/// directory is left holding a *replayed* — that is, restored — database +/// whose journal simply gets replayed again, idempotently, next time. +/// +/// This never deletes a file. Rolling back is SQLite's job: removing a journal +/// by hand would destroy the in-flight transaction of whatever process wrote +/// it and freeze the database half-written. +/// +/// Nor can this fire against a live transaction: a journal whose writer still +/// holds a RESERVED lock is by definition *not* hot, so a concurrent fojin +/// index build never even reaches here — that case reports `SQLITE_BUSY`, +/// which is not the signature we act on. +fn rollback_hot_journal(path: &Path) { + let Ok(conn) = rusqlite::Connection::open_with_flags(path, OpenFlags::SQLITE_OPEN_READ_WRITE) + else { + return; + }; + let _ = probe_first_read(&conn); +} + +pub fn open_read_only_db(path: &Path) -> Result { + let conn = open_read_only_connection(path)?; + match probe_first_read(&conn) { + Err(error) if is_hot_journal_error(&error) => { + // Drop the SHARED lock first: the rollback needs EXCLUSIVE. + drop(conn); + rollback_hot_journal(path); + // Healed, the fresh connection reads fine. Not healed, nothing on + // disk changed and it fails exactly where it failed before — see + // the hot-journal branch of `require_schema`. + open_read_only_connection(path) + } + // Anything else (including "file is not a database") is left to the + // caller's existing checks, so no error text moves for it. + _ => Ok(conn), + } +} + +/// Wording for the state a hot journal actually leaves the CLI in. +/// +/// It is *not* a dataset incompatibility, and `fojin data update` cannot fix +/// it: the atomic replace only renames the main file, so the stale journal +/// stays exactly where it was and takes the freshly downloaded database down +/// with it. Reaching this point also means self-healing already tried and +/// failed, so the only real cause left is that the dataset cannot be written. +fn hot_journal_error(conn: &rusqlite::Connection) -> anyhow::Error { + let data = conn.path().unwrap_or("data.sqlite"); + anyhow!( + "上次写入数据时被中断,`{data}` 旁留下了一份未回滚的日志(`{data}-journal`)。\ + 回滚这份日志需要写权限,而当前数据不可写,因此无法自动修复 \ + —— 数据本身没有损坏,也不需要重新下载。\ + 请恢复该数据文件及其所在目录的写权限后重新运行(届时会自动回滚),\ + 或改用一个可写的 `--data-dir`。" + ) +} + #[derive(Debug, serde::Serialize)] pub struct DatasetCompatibility { pub version: String, @@ -727,6 +949,13 @@ pub fn dataset_stats(conn: &rusqlite::Connection) -> Result { fn require_schema(conn: &rusqlite::Connection, name: &str, sql: &str) -> Result<()> { conn.prepare(sql).map(|_| ()).map_err(|e| { + // A hot journal blocks *every* read, so it always lands on this first + // check of `validate_compatibility` — this is the single funnel where + // it can surface, and calling it an incompatibility was wrong twice + // over: wrong diagnosis, and advice that cannot work. + if is_hot_journal_error(&e) { + return hot_journal_error(conn); + } anyhow!( "dataset incompatibility: required schema `{name}` is missing or invalid: {e}. Run `fojin data update`." ) diff --git a/src/data/operation_lock.rs b/src/data/operation_lock.rs index 8f79242..0e81581 100644 --- a/src/data/operation_lock.rs +++ b/src/data/operation_lock.rs @@ -1,7 +1,7 @@ use super::sibling_path; use anyhow::{anyhow, Context, Result}; use std::fs::{File, OpenOptions, TryLockError}; -use std::path::Path; +use std::path::{Path, PathBuf}; use std::time::{Duration, Instant}; const POLL_INTERVAL: Duration = Duration::from_millis(100); @@ -12,7 +12,7 @@ pub(super) struct OperationLock { _file: File, } -pub(super) fn acquire(data_path: &Path, timeout: Duration) -> Result { +fn open_lock_file(data_path: &Path) -> Result<(File, PathBuf)> { let lock_path = sibling_path(data_path, ".lock")?; let file = OpenOptions::new() .read(true) @@ -21,6 +21,25 @@ pub(super) fn acquire(data_path: &Path, timeout: Duration) -> Result Result { + let (file, lock_path) = open_lock_file(data_path)?; + match file.try_lock() { + Ok(()) => Ok(OperationLock { _file: file }), + Err(TryLockError::WouldBlock) => Err(anyhow!("数据操作锁被占用: {}", lock_path.display())), + Err(TryLockError::Error(error)) => { + Err(error).with_context(|| format!("获取数据操作锁失败: {}", lock_path.display())) + } + } +} + +pub(super) fn acquire(data_path: &Path, timeout: Duration) -> Result { + let (file, lock_path) = open_lock_file(data_path)?; let started = Instant::now(); let mut reported_wait = false; loop { @@ -69,4 +88,24 @@ mod tests { acquire(&data, Duration::from_millis(100)).unwrap(); assert!(data.with_file_name("data.sqlite.lock").exists()); } + + #[test] + fn try_acquire_returns_immediately_when_held() { + let directory = tempfile::tempdir().unwrap(); + let data = directory.path().join("data.sqlite"); + let held = acquire(&data, Duration::from_millis(100)).unwrap(); + + let started = Instant::now(); + let result = try_acquire(&data); + let waited = started.elapsed(); + + assert!(result.is_err(), "must not acquire a held lock"); + assert!( + waited < Duration::from_millis(50), + "try_acquire must not wait; waited {waited:?}" + ); + + drop(held); + assert!(try_acquire(&data).is_ok(), "must acquire once released"); + } } diff --git a/src/data/transfer.rs b/src/data/transfer.rs index f0df5e8..987d566 100644 --- a/src/data/transfer.rs +++ b/src/data/transfer.rs @@ -10,6 +10,9 @@ use std::time::{Duration, Instant as StdInstant}; const MIB: u64 = 1024 * 1024; const BUFFER_SIZE: usize = 64 * 1024; +/// 边车文件:SQLite 在主数据库旁自己派生的回滚日志与 WAL 组。它们只在 +/// 与那份主文件配对时才有意义,因此永远跟着主文件一起被处置。 +const SQLITE_SIDECAR_SUFFIXES: [&str; 3] = ["-journal", "-shm", "-wal"]; static ARTIFACT_SEQUENCE: AtomicU64 = AtomicU64::new(0); #[derive(Clone, Copy, Debug)] @@ -433,7 +436,7 @@ fn try_create_candidate_artifact_with( where F: FnMut(&Path) -> Result, { - let sidecars = ["-journal", "-shm", "-wal"] + let sidecars = SQLITE_SIDECAR_SUFFIXES .map(|suffix| sibling_path(path, suffix)) .into_iter() .collect::>>()?; @@ -508,7 +511,7 @@ fn path_entry_exists(path: &Path) -> Result { } fn remove_artifact_family(path: &Path) -> Result<()> { - for suffix in ["", "-journal", "-shm", "-wal"] { + for suffix in std::iter::once("").chain(SQLITE_SIDECAR_SUFFIXES) { let artifact = if suffix.is_empty() { path.to_path_buf() } else { @@ -526,6 +529,31 @@ fn remove_artifact_family(path: &Path) -> Result<()> { Ok(()) } +/// 只删 SQLite 为一个数据库派生的边车,主文件原封不动。 +/// +/// 与 `remove_artifact_family` 是同一套纪律,区别只在于主文件不归它管: +/// 两个调用点要么正在删除主文件(`clean_data`),要么正要用 rename 覆盖它 +/// (`install_candidate`)。 +/// +/// **只能用在这两处。** 对一个正在使用的数据库删日志,等于摧毁另一个进程 +/// 正在进行的事务,并让数据库永久停在半写状态 —— 查询打开路径上绝不能 +/// 出现这个调用。让路径安全的是这两条:主文件即将消失或被整体替换,以及 +/// 两处都在 `operation_lock` 之下,排除了并发的 fojin 数据操作。 +pub(super) fn remove_sqlite_sidecars(live_path: &Path) -> Result<()> { + for suffix in SQLITE_SIDECAR_SUFFIXES { + let sidecar = sibling_path(live_path, suffix)?; + match std::fs::remove_file(&sidecar) { + Ok(()) => {} + Err(error) if error.kind() == io::ErrorKind::NotFound => {} + Err(error) => { + return Err(error) + .with_context(|| format!("删除陈旧数据日志失败: {}", sidecar.display())); + } + } + } + Ok(()) +} + pub(super) fn remove_known_artifacts(live_path: &Path) -> Result<()> { let legacy = live_path.with_extension("tmp"); match std::fs::remove_file(&legacy) { @@ -579,7 +607,7 @@ fn is_owned_artifact_name(live_name: &str, artifact_name: &str) -> bool { let Some(candidate) = suffix.strip_prefix(".candidate.") else { return false; }; - let generation = ["-journal", "-shm", "-wal"] + let generation = SQLITE_SIDECAR_SUFFIXES .into_iter() .find_map(|sidecar| candidate.strip_suffix(sidecar)) .unwrap_or(candidate); diff --git a/src/render.rs b/src/render.rs index a4396cc..b252d2a 100644 --- a/src/render.rs +++ b/src/render.rs @@ -21,8 +21,24 @@ pub fn lang_label(code: &str) -> &str { } } +/// The tag appears only when it would show a number other than 1.00. +/// +/// Keying on the formatted string rather than the raw float keeps the rule +/// self-consistent: a value like 0.9951 would render as "1.00", so treating it +/// as informative would print a tag that says exactly what we suppress +/// elsewhere. Absence of the tag reads as "no caveat". fn conf_tag(c: Option) -> String { - c.map(|v| format!(" [MITRA {v:.2}]")).unwrap_or_default() + match c { + Some(v) => { + let shown = format!("{v:.2}"); + if shown == "1.00" { + String::new() + } else { + format!(" [MITRA {shown}]") + } + } + None => String::new(), + } } /// Renders groups only — no footer, no "还有 N 组" line. Shared by the plain diff --git a/tests/command.rs b/tests/command.rs index 9154a90..d2fe47e 100644 --- a/tests/command.rs +++ b/tests/command.rs @@ -555,3 +555,436 @@ fn reverse_lookup_finds_chinese_from_sanskrit() { assert_eq!(v["matched"], serde_json::json!(true)); assert_eq!(v["groups"][0]["zh_text"], serde_json::json!("色即是空")); } + +fn write_cite_fixture(dir: &std::path::Path) { + let db_path = dir.join("data.sqlite"); + let conn = Connection::open(db_path).unwrap(); + init_schema(&conn).unwrap(); + for (k, v) in [("version", "v1"), ("norm_ruleset", "t2s-char-1to1-v1")] { + conn.execute( + "INSERT INTO meta(key,value) VALUES (?1,?2)", + rusqlite::params![k, v], + ) + .unwrap(); + } + for (zt, zn, lang, f, juan) in [ + ("色即是空", "色即是空", "sa", "rūpaṃ śūnyatā", 1), + ("受想行識", "受想行识", "bo", "gzugs stong pa", 2), + ] { + conn.execute( + "INSERT INTO parallels(zh_text,zh_norm,foreign_lang,foreign_text,confidence,cbeta_id,title_zh,juan_num) + VALUES (?1,?2,?3,?4,0.9,'T0251','心經',?5)", + rusqlite::params![zt, zn, lang, f, juan], + ) + .unwrap(); + } +} + +fn index_exists(dir: &std::path::Path) -> bool { + let conn = Connection::open(dir.join("data.sqlite")).unwrap(); + conn.query_row( + "SELECT COUNT(*) FROM sqlite_master WHERE type='index' AND name='idx_parallels_cbeta'", + [], + |row| row.get::<_, i64>(0), + ) + .unwrap() + > 0 +} + +#[test] +fn cite_builds_the_missing_index_and_output_is_unchanged() { + let dir = tempfile::tempdir().unwrap(); + write_cite_fixture(dir.path()); + assert!( + !index_exists(dir.path()), + "fixture must start without the index" + ); + + let first = std::process::Command::new(env!("CARGO_BIN_EXE_fojin")) + .args(["cite", "T0251", "--offline", "--data-dir"]) + .arg(dir.path()) + .output() + .unwrap(); + assert_eq!(first.status.code(), Some(0)); + assert!(index_exists(dir.path()), "cite must have built the index"); + assert!( + String::from_utf8_lossy(&first.stderr).contains("建立索引"), + "the first run actually built the index, so it must announce it: {}", + String::from_utf8_lossy(&first.stderr) + ); + + // Second run: index already present, so no rebuild and no notice. This is + // also the idempotency check — `ensure_cbeta_index` returns before it even + // opens the file read-write once the index exists. + let second = std::process::Command::new(env!("CARGO_BIN_EXE_fojin")) + .args(["cite", "T0251", "--offline", "--data-dir"]) + .arg(dir.path()) + .output() + .unwrap(); + assert_eq!( + String::from_utf8(first.stdout).unwrap(), + String::from_utf8(second.stdout).unwrap() + ); + assert!( + !String::from_utf8(second.stderr) + .unwrap() + .contains("建立索引"), + "the notice must appear only when actually building" + ); +} + +#[test] +fn a_held_lock_skips_the_build_and_leaves_results_identical() { + // Holding the operation lock forces `cite` down the un-indexed path, which + // is the only portable way to compare indexed vs un-indexed output — the + // first run of the previous test already has the index by the time it + // queries, so it cannot make this comparison. + let indexed_dir = tempfile::tempdir().unwrap(); + write_cite_fixture(indexed_dir.path()); + let indexed = std::process::Command::new(env!("CARGO_BIN_EXE_fojin")) + .args(["cite", "T0251", "--offline", "--data-dir"]) + .arg(indexed_dir.path()) + .output() + .unwrap(); + assert!(index_exists(indexed_dir.path())); + + let plain_dir = tempfile::tempdir().unwrap(); + write_cite_fixture(plain_dir.path()); + let lock_file = std::fs::OpenOptions::new() + .read(true) + .write(true) + .create(true) + .truncate(false) + .open(plain_dir.path().join("data.sqlite.lock")) + .unwrap(); + lock_file.try_lock().expect("test must own the lock"); + + let plain = std::process::Command::new(env!("CARGO_BIN_EXE_fojin")) + .args(["cite", "T0251", "--offline", "--data-dir"]) + .arg(plain_dir.path()) + .output() + .unwrap(); + drop(lock_file); + + assert_eq!( + plain.status.code(), + Some(0), + "a busy lock must not fail the query" + ); + assert!( + !index_exists(plain_dir.path()), + "a busy lock must skip the build rather than wait for it" + ); + assert_eq!( + String::from_utf8(indexed.stdout).unwrap(), + String::from_utf8(plain.stdout).unwrap(), + "the index changes speed only — never results" + ); +} + +#[test] +fn parallel_does_not_build_the_cite_index() { + let dir = tempfile::tempdir().unwrap(); + write_cite_fixture(dir.path()); + + let output = std::process::Command::new(env!("CARGO_BIN_EXE_fojin")) + .args(["parallel", "色即是空", "--offline", "--data-dir"]) + .arg(dir.path()) + .output() + .unwrap(); + + assert_eq!(output.status.code(), Some(0)); + assert!( + !index_exists(dir.path()), + "only the cite path needs this index; parallel must not pay for it" + ); +} + +// Verifies the earliest fallback path: data directory not writable, no +// pre-existing lock file. `operation_lock::try_acquire` fails at lock-file +// creation (which needs directory write access) and returns early, before +// reaching the read-write open, the announcement, or the index build. +// +// Because that path exits so early, the stderr and index assertions below are +// trivially true by construction: this test CANNOT distinguish graceful +// degradation from the wiring being absent altogether. The tests that do pin +// the wiring down are `cite_builds_the_missing_index_and_output_is_unchanged` +// and `a_held_lock_skips_the_build_and_leaves_results_identical`. +#[cfg(unix)] +#[test] +fn cite_degrades_gracefully_when_the_data_dir_is_read_only() { + use std::os::unix::fs::PermissionsExt; + + let dir = tempfile::tempdir().unwrap(); + write_cite_fixture(dir.path()); + let original = std::fs::metadata(dir.path()).unwrap().permissions(); + std::fs::set_permissions(dir.path(), std::fs::Permissions::from_mode(0o500)).unwrap(); + + let output = std::process::Command::new(env!("CARGO_BIN_EXE_fojin")) + .args(["cite", "T0251", "--offline", "--data-dir"]) + .arg(dir.path()) + .output() + .unwrap(); + + // Restore before asserting so a failure still leaves a removable tempdir. + std::fs::set_permissions(dir.path(), original).unwrap(); + + assert_eq!( + output.status.code(), + Some(0), + "a read-only data dir must not fail the query: {}", + String::from_utf8_lossy(&output.stderr) + ); + assert!(String::from_utf8(output.stdout) + .unwrap() + .contains("色即是空")); + assert!( + !String::from_utf8_lossy(&output.stderr).contains("建立索引"), + "a fresh read-only directory must never build the index, so it must not announce one: {}", + String::from_utf8_lossy(&output.stderr) + ); + assert!( + !index_exists(dir.path()), + "a read-only directory must not have the index" + ); +} + +#[cfg(unix)] +#[test] +fn cite_stays_silent_when_a_preexisting_lock_file_meets_a_read_only_directory() { + use std::os::unix::fs::PermissionsExt; + + let dir = tempfile::tempdir().unwrap(); + write_cite_fixture(dir.path()); + + // A lock file left behind by any earlier `cite` or `data update` run + // already exists here, unlocked. `try_acquire` succeeds on it, so the + // code reaches the read-write open, which also succeeds — the file + // itself is writable. Only the later `CREATE INDEX` fails, once the + // directory itself blocks the new file SQLite needs for its rollback + // journal. A pre-success notice would fire every single time despite the + // index never actually being built. + std::fs::File::create(dir.path().join("data.sqlite.lock")).unwrap(); + + let original = std::fs::metadata(dir.path()).unwrap().permissions(); + std::fs::set_permissions(dir.path(), std::fs::Permissions::from_mode(0o500)).unwrap(); + + let output = std::process::Command::new(env!("CARGO_BIN_EXE_fojin")) + .args(["cite", "T0251", "--offline", "--data-dir"]) + .arg(dir.path()) + .output() + .unwrap(); + + // Restore before asserting so a failure still leaves a removable tempdir. + std::fs::set_permissions(dir.path(), original).unwrap(); + + assert_eq!( + output.status.code(), + Some(0), + "a read-only data dir must not fail the query: {}", + String::from_utf8_lossy(&output.stderr) + ); + assert!(String::from_utf8(output.stdout) + .unwrap() + .contains("色即是空")); + assert!( + !String::from_utf8(output.stderr) + .unwrap() + .contains("建立索引"), + "must not announce an index build that never actually completed" + ); + assert!( + !index_exists(dir.path()), + "the read-only directory must have blocked index creation" + ); +} + +/// SQLite 回滚日志的文件头 magic。日志一旦带上它就是"热"的:任何新进程 +/// 打开这个数据库时都必须先把它回放掉,而回放需要写权限。 +const JOURNAL_MAGIC: [u8; 8] = [0xd9, 0xd5, 0x05, 0xf9, 0x20, 0xa1, 0x63, 0xd7]; + +/// 在 `dir` 里布置"写事务进行到一半、进程猝死"留下的现场:一份已经被 +/// 部分改写的 `data.sqlite`,外加一份仍然有效的 `data.sqlite-journal`。 +/// +/// 做法不是杀进程(时序不可重复),而是把一个**尚未提交**的事务在磁盘上 +/// 的状态原样复制出来。SQLite 的不变量是"任何被写回主库文件的页,其原始 +/// 内容必定已经先同步进日志",所以复制出来的这一对,恰好就是进程死在该 +/// 窗口内时留在磁盘上的东西。文件锁不会跟着字节被复制走,因此复制品里 +/// 的日志对任何新进程都是"热"的。 +/// +/// 触发点是页缓存溢出:缓存放不下时 SQLite 必须先把日志(含 magic 头) +/// 同步落盘,才能把脏页写回主库 —— 这正是 588 MB 数据上 `CREATE INDEX` +/// 会走到的那一步。 +fn plant_hot_journal(dir: &std::path::Path) { + let scratch = tempfile::tempdir().unwrap(); + write_cite_fixture(scratch.path()); + let source = scratch.path().join("data.sqlite"); + let source_journal = scratch.path().join("data.sqlite-journal"); + + let conn = Connection::open(&source).unwrap(); + conn.pragma_update(None, "cache_size", 10_i64).unwrap(); + conn.execute_batch("BEGIN IMMEDIATE").unwrap(); + for index in 0..4000 { + conn.execute( + "INSERT INTO parallels(zh_text,zh_norm,foreign_lang,foreign_text,confidence,cbeta_id,title_zh,juan_num) + VALUES (?1,?1,'sa','x',0.9,'T9999','未提交',1)", + rusqlite::params![format!("未提交的垃圾行{index}")], + ) + .unwrap(); + } + + // 既不提交也不回滚:此刻磁盘上就是崩溃现场。先复制主库再复制日志, + // 保证日志不会比主库更旧。用 read/write 而不是 `fs::copy`,因为前者 + // 的共享模式在各平台上都允许读取另一个连接正打开着的文件。 + let data = dir.join("data.sqlite"); + let journal = dir.join("data.sqlite-journal"); + std::fs::write(&data, std::fs::read(&source).unwrap()).unwrap(); + std::fs::write(&journal, std::fs::read(&source_journal).unwrap()).unwrap(); + drop(conn); // scratch 自行回滚,与 dir 里的复制品无关 + + let planted_journal = std::fs::read(&journal).unwrap(); + assert!( + planted_journal.starts_with(&JOURNAL_MAGIC), + "现场必须留下一份带 magic 头的热日志,否则这个测试什么都没复现" + ); + let planted_data = std::fs::read(&data).unwrap(); + assert!( + planted_data + .windows("未提交的垃圾行".len()) + .any(|window| window == "未提交的垃圾行".as_bytes()), + "主库文件里必须真的落下了未提交的页,否则回滚有没有发生无从区分" + ); +} + +fn parallel_rows(dir: &std::path::Path) -> (i64, i64) { + let conn = Connection::open(dir.join("data.sqlite")).unwrap(); + let total = conn + .query_row("SELECT COUNT(*) FROM parallels", [], |row| row.get(0)) + .unwrap(); + let uncommitted = conn + .query_row( + "SELECT COUNT(*) FROM parallels WHERE cbeta_id = 'T9999'", + [], + |row| row.get(0), + ) + .unwrap(); + (total, uncommitted) +} + +// 回归 CRITICAL:懒建索引被中断后留下的热日志,曾经让所有查询命令永久 +// 退出 1 并谎称数据不兼容。只读连接回滚不了日志,但数据目录本身是可写 +// 的 —— 查询路径必须自己把它回滚掉,而不是把用户推去重下 183 MB。 +#[test] +fn cite_rolls_back_a_hot_journal_left_by_an_interrupted_index_build() { + let dir = tempfile::tempdir().unwrap(); + plant_hot_journal(dir.path()); + + let output = run_fojin(&["cite", "T0251", "--offline"], dir.path()); + + let stderr = String::from_utf8(output.stderr).unwrap(); + assert_eq!( + output.status.code(), + Some(0), + "a rollback-able journal must not fail the query: {stderr}" + ); + assert!( + String::from_utf8(output.stdout) + .unwrap() + .contains("色即是空"), + "the recovered dataset must answer normally" + ); + assert!( + !stderr.contains("dataset incompatibility"), + "an interrupted write is not a dataset incompatibility: {stderr}" + ); + assert!( + !dir.path().join("data.sqlite-journal").exists(), + "SQLite must have rolled the journal back and removed it" + ); + assert_eq!( + parallel_rows(dir.path()), + (2, 0), + "rollback must restore the pre-transaction rows, not commit the junk" + ); +} + +// 同一份现场对每条读路径都成立:`parallel` 与 `data verify` 曾经一起 +// 退出 1,而 `data verify` 正是用户会用来诊断的那条命令。 +#[test] +fn every_read_command_recovers_from_a_hot_journal() { + for args in [ + vec!["parallel", "色即是空", "--offline"], + vec!["data", "verify"], + ] { + let dir = tempfile::tempdir().unwrap(); + plant_hot_journal(dir.path()); + + let output = run_fojin(&args, dir.path()); + + let stderr = String::from_utf8(output.stderr).unwrap(); + assert_eq!( + output.status.code(), + Some(0), + "`{args:?}` must recover instead of failing: {stderr}" + ); + assert!( + !stderr.contains("dataset incompatibility"), + "`{args:?}` must not blame the dataset: {stderr}" + ); + assert_eq!(parallel_rows(dir.path()), (2, 0), "`{args:?}`"); + } +} + +// 自愈的边界:数据真的不可写时,回滚做不到,查询也**不能**假装成功。 +// 区分两者的唯一判据是以读写方式重开能不能真的读到数据 —— 这里内核会 +// 拒绝写入(SQLite 静默降级为只读打开),于是自愈失败,命令仍然报错。 +#[cfg(unix)] +#[test] +fn a_read_only_dataset_still_fails_instead_of_pretending_to_recover() { + use std::os::unix::fs::PermissionsExt; + + let dir = tempfile::tempdir().unwrap(); + plant_hot_journal(dir.path()); + let data = dir.path().join("data.sqlite"); + let journal = dir.path().join("data.sqlite-journal"); + let data_before = std::fs::read(&data).unwrap(); + let journal_before = std::fs::read(&journal).unwrap(); + for path in [&data, &journal] { + std::fs::set_permissions(path, std::fs::Permissions::from_mode(0o444)).unwrap(); + } + let directory_permissions = std::fs::metadata(dir.path()).unwrap().permissions(); + std::fs::set_permissions(dir.path(), std::fs::Permissions::from_mode(0o500)).unwrap(); + + let output = run_fojin(&["cite", "T0251", "--offline"], dir.path()); + + // 先恢复权限,失败时 tempdir 才删得掉。 + std::fs::set_permissions(dir.path(), directory_permissions).unwrap(); + for path in [&data, &journal] { + std::fs::set_permissions(path, std::fs::Permissions::from_mode(0o644)).unwrap(); + } + + let stderr = String::from_utf8(output.stderr).unwrap(); + assert_eq!( + output.status.code(), + Some(1), + "an unrecoverable dataset must not report success: {stderr}" + ); + assert!( + stderr.contains("日志") && stderr.contains("写权限"), + "the message must name the real cause and what to do: {stderr}" + ); + assert!( + !stderr.contains("dataset incompatibility") && !stderr.contains("data update"), + "neither the diagnosis nor the advice was ever true here: {stderr}" + ); + assert_eq!( + std::fs::read(&data).unwrap(), + data_before, + "a failed self-heal must leave the dataset byte-identical" + ); + assert_eq!( + std::fs::read(&journal).unwrap(), + journal_before, + "a failed self-heal must never delete the journal it could not replay" + ); +} diff --git a/tests/data.rs b/tests/data.rs index 5c262b5..ccdb125 100644 --- a/tests/data.rs +++ b/tests/data.rs @@ -1,7 +1,7 @@ use fojin_cli::data::{ clean_data, ensure_data, gunzip, open_compatible_db, open_read_only_db, resolve_data_path, update_data, validate_compatibility, verify_dataset, verify_dataset_file, verify_sha256, - DataSource, EXPECTED_DATA_VERSION, EXPECTED_NORM_RULESET, + DataSource, CBETA_INDEX_NAME, EXPECTED_DATA_VERSION, EXPECTED_NORM_RULESET, }; use std::io::{Read, Write}; use std::path::PathBuf; @@ -181,10 +181,15 @@ fn clean_data_removes_known_file_artifacts_and_preserves_other_entries() { let download = sibling_path(&data, ".download.123.0.gz"); let candidate = sibling_path(&data, ".candidate.123.1"); let candidate_journal = sibling_path(&data, ".candidate.123.1-journal"); + // 活跃库自己的边车。被中断的建索引会留下 `-journal`;删掉主库却把它 + // 留在原地,下一次重新下载的数据会被这份陈旧日志重放成一堆乱码。 + let live_sidecars = ["-journal", "-shm", "-wal"].map(|suffix| sibling_path(&data, suffix)); let matching_directory = sibling_path(&data, ".download.keep"); let lock_path = sibling_path(&data, ".lock"); let unrelated = sibling_path(&data, ".unrelated"); let lookalikes = [ + sibling_path(&data, "-journal.notes"), + sibling_path(&data, ".journal"), sibling_path(&data, ".candidate.notes"), sibling_path(&data, ".candidate.123"), sibling_path(&data, ".candidate.123.x"), @@ -195,7 +200,10 @@ fn clean_data_removes_known_file_artifacts_and_preserves_other_entries() { sibling_path(&data, ".download.123.0.gz.notes"), ]; std::fs::write(&data, b"live").unwrap(); - for artifact in [&legacy, &download, &candidate, &candidate_journal] { + for artifact in [&legacy, &download, &candidate, &candidate_journal] + .into_iter() + .chain(live_sidecars.iter()) + { std::fs::write(artifact, b"temporary").unwrap(); } std::fs::create_dir(&matching_directory).unwrap(); @@ -206,7 +214,10 @@ fn clean_data_removes_known_file_artifacts_and_preserves_other_entries() { } assert_eq!(clean_data(&data).unwrap(), Some(4)); - for removed in [&data, &legacy, &download, &candidate, &candidate_journal] { + for removed in [&data, &legacy, &download, &candidate, &candidate_journal] + .into_iter() + .chain(live_sidecars.iter()) + { assert!(!removed.exists(), "artifact remains: {}", removed.display()); } assert!(matching_directory.is_dir()); @@ -314,6 +325,49 @@ fn ensure_data_downloads_verifies_and_unpacks() { verify_dataset_file(&path).unwrap(); } +#[test] +fn install_leaves_the_cite_index_in_place() { + let gz = gzip_bytes(&replacement_database_bytes()); + let sha = sha256_hex(&gz); + + let listener = std::net::TcpListener::bind("127.0.0.1:0").unwrap(); + let port = listener.local_addr().unwrap().port(); + let body = gz.clone(); + let server = std::thread::spawn(move || { + let (mut stream, _) = listener.accept().unwrap(); + let mut req = [0u8; 4096]; + let _ = std::io::Read::read(&mut stream, &mut req); + let head = format!( + "HTTP/1.1 200 OK\r\nContent-Length: {}\r\nContent-Type: application/gzip\r\nConnection: close\r\n\r\n", + body.len() + ); + stream.write_all(head.as_bytes()).unwrap(); + stream.write_all(&body).unwrap(); + }); + + let dir = tempfile::tempdir().unwrap(); + let path = dir.path().join("data.sqlite"); + let source = DataSource { + url: &format!("http://127.0.0.1:{port}/data.gz"), + sha256: &sha, + }; + ensure_data(&path, false, &source).unwrap(); + server.join().unwrap(); + + let conn = open_read_only_db(&path).unwrap(); + let present: i64 = conn + .query_row( + "SELECT COUNT(*) FROM sqlite_master WHERE type='index' AND name=?1", + [CBETA_INDEX_NAME], + |row| row.get(0), + ) + .unwrap(); + assert_eq!(present, 1, "a fresh install must ship the cite index"); + + // The index must not disturb the compatibility contract. + verify_dataset_file(&path).unwrap(); +} + #[test] fn concurrent_first_install_downloads_once() { if std::env::var_os("FOJIN_CONCURRENT_WORKER").is_some() { @@ -1252,6 +1306,64 @@ fn update_data_replaces_live_dataset_without_backup_path_dependency() { assert_no_candidate_artifacts(&path); } +// 被中断的建索引会在活跃路径旁留下一份"热"回滚日志。`install_candidate` +// 的原子替换只搬主文件,日志会原地不动地等着被重放进那份刚下载的新库 —— +// 于是用户白下 183 MB,第一次读写打开就把新库改成 `database disk image is +// malformed`。发布出去的必须是"新库 + 没有任何外来日志"。 +#[test] +fn update_data_publishes_without_the_previous_datasets_journal() { + let dir = tempfile::tempdir().unwrap(); + let path = dir.path().join("data.sqlite"); + std::fs::write(&path, b"old live dataset").unwrap(); + let journal = sibling_path(&path, "-journal"); + std::fs::write(&journal, foreign_hot_journal_bytes()).unwrap(); + + let gz = gzip_bytes(&replacement_database_bytes()); + let sha = sha256_hex(&gz); + serve_update(&path, gz, &sha).unwrap(); + + assert!( + !journal.exists(), + "the replaced dataset's journal must not survive the publish: {}", + journal.display() + ); + assert_replacement_marker(&path); + assert_no_candidate_artifacts(&path); +} + +/// 一份来自**另一个**数据库、且仍然有效("热")的回滚日志的字节。 +/// +/// 触发点是页缓存溢出:缓存放不下时 SQLite 必须先把日志(含 magic 头) +/// 同步落盘才能把脏页写回主库,日志由此变"热"。事务既不提交也不回滚, +/// 读出来的就是进程猝死在该窗口时留在磁盘上的字节。 +fn foreign_hot_journal_bytes() -> Vec { + const JOURNAL_MAGIC: [u8; 8] = [0xd9, 0xd5, 0x05, 0xf9, 0x20, 0xa1, 0x63, 0xd7]; + + let dir = tempfile::tempdir().unwrap(); + let path = dir.path().join("interrupted.sqlite"); + let conn = rusqlite::Connection::open(&path).unwrap(); + fojin_cli::schema::init_schema(&conn).unwrap(); + insert_compat_meta(&conn); + conn.pragma_update(None, "cache_size", 10_i64).unwrap(); + conn.execute_batch("BEGIN IMMEDIATE").unwrap(); + for index in 0..2000 { + conn.execute( + "INSERT INTO parallels(zh_text, zh_norm, foreign_lang, foreign_text) \ + VALUES (?1, ?1, 'sa', 'x')", + rusqlite::params![format!("未提交的垃圾行{index}")], + ) + .unwrap(); + } + let bytes = std::fs::read(sibling_path(&path, "-journal")).unwrap(); + drop(conn); + + assert!( + bytes.starts_with(&JOURNAL_MAGIC), + "the fixture must be a journal SQLite would actually replay" + ); + bytes +} + fn replacement_database_bytes() -> Vec { compatible_database_bytes(|conn| { conn.execute( diff --git a/tests/render.rs b/tests/render.rs index 3b1b213..ed6b8c4 100644 --- a/tests/render.rs +++ b/tests/render.rs @@ -272,3 +272,114 @@ fn json_segments_appear_only_when_splitting_happened() { ); assert!(v.get("fallback").is_none()); } + +#[test] +fn perfect_confidence_shows_no_tag() { + let group = MatchGroup { + zh_text: "色即是空".into(), + cbeta_id: Some("T0251".into()), + title_zh: Some("心經".into()), + juan_num: Some(1), + parallels: vec![Parallel { + lang: "sa".into(), + text: "rūpaṃ śūnyatā".into(), + confidence: Some(1.0), + }], + }; + let out = render_human(&[group], None, 0); + assert!( + out.contains("梵 rūpaṃ śūnyatā"), + "the parallel itself stays: {out}" + ); + assert!( + !out.contains("MITRA"), + "a uniform 1.00 carries no information and must not be printed: {out}" + ); +} + +#[test] +fn imperfect_confidence_still_shows_the_tag() { + let group = MatchGroup { + zh_text: "色即是空".into(), + cbeta_id: Some("T0251".into()), + title_zh: Some("心經".into()), + juan_num: Some(1), + parallels: vec![Parallel { + lang: "sa".into(), + text: "rūpaṃ śūnyatā".into(), + confidence: Some(0.87), + }], + }; + let out = render_human(&[group], None, 0); + assert!( + out.contains("梵 rūpaṃ śūnyatā [MITRA 0.87]"), + "got: {out}" + ); +} + +#[test] +fn confidence_rounding_to_one_hides_the_tag() { + // 0.9951 formats as "1.00"; printing `[MITRA 1.00]` while calling the value + // informative would contradict itself, so the rule keys on what would be + // displayed, not on the raw float. + let group = MatchGroup { + zh_text: "色即是空".into(), + cbeta_id: Some("T0251".into()), + title_zh: Some("心經".into()), + juan_num: Some(1), + parallels: vec![Parallel { + lang: "sa".into(), + text: "rūpaṃ śūnyatā".into(), + confidence: Some(0.9951), + }], + }; + let out = render_human(&[group], None, 0); + assert!( + out.contains("梵 rūpaṃ śūnyatā"), + "the parallel itself stays: {out}" + ); + assert!(!out.contains("MITRA"), "got: {out}"); +} + +#[test] +fn absent_confidence_shows_no_tag() { + let group = MatchGroup { + zh_text: "色即是空".into(), + cbeta_id: Some("T0251".into()), + title_zh: Some("心經".into()), + juan_num: Some(1), + parallels: vec![Parallel { + lang: "sa".into(), + text: "rūpaṃ śūnyatā".into(), + confidence: None, + }], + }; + let out = render_human(&[group], None, 0); + assert!( + out.contains("梵 rūpaṃ śūnyatā"), + "the parallel itself stays: {out}" + ); + assert!(!out.contains("MITRA"), "got: {out}"); +} + +#[test] +fn json_still_carries_confidence_when_the_tag_is_hidden() { + let group = MatchGroup { + zh_text: "色即是空".into(), + cbeta_id: Some("T0251".into()), + title_zh: Some("心經".into()), + juan_num: Some(1), + parallels: vec![Parallel { + lang: "sa".into(), + text: "rūpaṃ śūnyatā".into(), + confidence: Some(1.0), + }], + }; + let out = render_json(&[group], 1); + let v: serde_json::Value = serde_json::from_str(&out).unwrap(); + assert_eq!( + v["groups"][0]["parallels"][0]["confidence"], + serde_json::json!(1.0), + "the JSON contract is unchanged; only the human tag is suppressed" + ); +}