diff --git a/.github/workflows/release.yml b/.github/workflows/release.yml index bc2d164f..2090ea48 100644 --- a/.github/workflows/release.yml +++ b/.github/workflows/release.yml @@ -369,6 +369,8 @@ jobs: steps: - uses: actions/checkout@v7 + with: + fetch-depth: 0 - name: Extract version id: version @@ -435,6 +437,34 @@ jobs: } > release-notes.md fi + - name: Append contributor credits + env: + GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} + run: | + { + echo "" + echo "" + echo "## Community contributors" + echo "" + } >> release-notes.md + FOUND=0 + for PR in $(grep -oE '#[0-9]+' changelog-body.md 2>/dev/null | sort -u); do + NUM="${PR#\#}" + LOGIN="$(gh pr view "${NUM}" --repo "${GITHUB_REPOSITORY}" --json author --jq '.author.login' 2>/dev/null || true)" + [ -z "${LOGIN}" ] && continue + case "${LOGIN}" in ajianaz|codecoradev|app/*|*"[bot]") continue ;; esac + ASSOC="$(gh api "repos/${GITHUB_REPOSITORY}/pulls/${NUM}" --jq '.author_association' 2>/dev/null || true)" + case "${ASSOC}" in MEMBER|COLLABORATOR|OWNER) continue ;; esac + TITLE="$(gh pr view "${NUM}" --repo "${GITHUB_REPOSITORY}" --json title --jq '.title' 2>/dev/null || true)" + echo "- @${LOGIN} — ${TITLE} (${PR})" >> release-notes.md + FOUND=1 + done + if [ "${FOUND}" -eq 0 ]; then + echo "(No external contributions in this cycle. Full contributor list: https://github.com/${GITHUB_REPOSITORY}/graphs/contributors)" >> release-notes.md + fi + echo "--- release-notes.md tail ---" + tail -n 10 release-notes.md + - name: Create GitHub Release uses: softprops/action-gh-release@v2 with: diff --git a/AGENT.md b/AGENT.md index cf24d585..f4dae3e3 100644 --- a/AGENT.md +++ b/AGENT.md @@ -7,7 +7,7 @@ **Uteke** is a local-first semantic memory engine for AI agents. Single Rust binary, fully offline, ~30ms recall. No API key, Docker, or cloud service needed. - **Repo:** `codecoradev/uteke` (remote GitHub), local clone -- **Version:** 0.18.0 +- **Version:** 0.18.1 - **License:** Apache 2.0 - **Main branches:** `develop` (default branch, all PRs go here), `main` (release mirror) diff --git a/CHANGELOG.md b/CHANGELOG.md index 38b5e32c..3be9f7bc 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,25 @@ ## [Unreleased] +## [0.18.1] - 2026-09-15 + +Patch release. Theme: **correctness and operational hardening** - structural +export round-trips cleanly in stores with soft-deleted memories, HTTP write +payloads reject unknown keys, `uteke upgrade` keeps every installed artifact +in sync with the release bundle, and `/health` no longer leaks instance +details to unauthenticated probes. + +### Fixed + +- **Structural export drops soft-deleted memory references (#1243, #1253)** - `room_memories`, `memory_edges`, and `timeline_events` dumps are filtered by the same liveness predicate as memories; `graph_nodes.memory_id` links to soft-deleted memories are nulled; manifest section counts match the rows actually exported. FK-enforced import into a fresh store now works for real production stores with deprecated memories. +- **HTTP write endpoints reject unknown keys (#1251, #1254)** - `POST /remember` and `POST /room/remember` return 400 naming the offending key (e.g. a v1-style `ns`) instead of silently dropping it and storing the memory in the default namespace. +- **`uteke upgrade` updates companion binaries and ORT libs (#1245, #1255)** - `uteke-serve`, `uteke-mcp`, and the bundled `libonnxruntime*` files are replaced from the same checksum-verified archive (atomically, preserving symlinks); the version comparison normalizes the release tag's `v` prefix so an up-to-date install is no longer re-offered the same release. +- **`GET /health` minimal payload for tokenless requests (#1252, #1257)** - without a valid token the endpoint answers `{"status":"ok"}` only (200, load-balancer friendly); version, memory counts, and update info require a valid bearer token or auth disabled. Server behavior only - CLI/MCP unchanged. + +### Added + +- **`uteke doctor` success footer (#1246, #1256)** - one quiet line after all checks pass (repo + managed cloud), human output only: skipped for `--json`, non-TTY (CI/pipes), and `doctor_footer = false` in `uteke.toml`. + ## [0.18.0] - 2026-09-13 Minor release. Theme: **agent-operable memory plumbing** - ingest with explicit timestamps, full room lifecycle management, and a recall payload contract that keeps benchmark harnesses honest. Retrieval behavior is unchanged from 0.17.0 (revalidated at the published config: recall@5 non-abs 0.9457, identical per-question rankings). @@ -26,6 +45,11 @@ Minor release. Theme: **agent-operable memory plumbing** - ingest with explicit - **Room rename, room update, and memory room-move (#1202)** — rooms now have sanctioned lifecycle ops matching the namespace family: `Uteke::rename_room(old, new)` rewrites the registry row and every `room_memories`/`room_documents` reference in ONE transaction (FK-safe insert→repoint→delete order; no `ON UPDATE CASCADE` needed), preserving title/description/namespace; `Uteke::update_room(id, title?, description?)` edits room metadata (schema v19 adds the additive `rooms.description` column; structural export/import carry it and stay backward-compatible with 5-column exports); `Uteke::move_memory_to_room(id, from, to)` moves a memory between rooms preserving the link's `author`/`role`/`joined_at` provenance (`Ok(0)` when not a member of `from`). Surfaces: HTTP `POST /room/rename|/room/update|/room/memory/move` (404 on missing room/link), CLI `uteke room rename|update|move-memory`, MCP `uteke_room_rename|uteke_room_update|uteke_room_memory_move`. - **`uteke update ` CLI** — in-place memory edit (content/tags/importance/pinned/type), closing the CLI surface gap: HTTP `PUT /memory` and MCP `uteke_update` already existed (#1202 item 5). +### Contributors + +- [@KazamiHazaki](https://github.com/KazamiHazaki) — binary install ORT lib fixes (#1221) +- [@dependabot](https://github.com/apps/dependabot) — dependency updates (dirs, tokenizers, toml, usearch) + ## [0.17.0] — 2026-09-06 Minor release. Theme: **inspectable, trustworthy memory** — explain recall on diff --git a/Cargo.lock b/Cargo.lock index 61f5a758..b37af4b3 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -1991,9 +1991,9 @@ dependencies = [ [[package]] name = "rustls" -version = "0.23.43" +version = "0.23.45" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "0283386ce02abc0151e1761d08802dfe86c173b0b494af5cbc086574e453da06" +checksum = "0d41d731c7d2f962d1ccc364cec258de3c0e93b38c2fb3ba97ac74513048d634" dependencies = [ "once_cell", "ring", @@ -2015,9 +2015,9 @@ dependencies = [ [[package]] name = "rustls-webpki" -version = "0.103.13" +version = "0.103.15" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "61c429a8649f110dddef65e2a5ad240f747e85f7758a6bccc7e5777bd33f756e" +checksum = "f3c3cf1d8b1e7d4927e2d154c3fcb02979afb9939629c62cd9048d4f07b60ac2" dependencies = [ "ring", "rustls-pki-types", @@ -2767,7 +2767,7 @@ dependencies = [ [[package]] name = "uteke-cli" -version = "0.18.0" +version = "0.18.1" dependencies = [ "chrono", "clap", @@ -2793,7 +2793,7 @@ dependencies = [ [[package]] name = "uteke-core" -version = "0.18.0" +version = "0.18.1" dependencies = [ "chrono", "dirs", @@ -2817,7 +2817,7 @@ dependencies = [ [[package]] name = "uteke-mcp" -version = "0.18.0" +version = "0.18.1" dependencies = [ "chrono", "dirs", @@ -2829,7 +2829,7 @@ dependencies = [ [[package]] name = "uteke-server" -version = "0.18.0" +version = "0.18.1" dependencies = [ "chrono", "ctrlc", diff --git a/Cargo.toml b/Cargo.toml index 12087796..7c935a64 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -9,7 +9,7 @@ members = [ ] [workspace.package] -version = "0.18.0" +version = "0.18.1" edition = "2024" license = "Apache-2.0" repository = "https://github.com/codecoradev/uteke" diff --git a/README.id.md b/README.id.md index d3da0a64..521ec1ee 100644 --- a/README.id.md +++ b/README.id.md @@ -82,7 +82,7 @@ menemukan buktinya. 500 pertanyaan hand-curated, lima kemampuan memori. Uteke menjalankan suite penuh dengan **nol panggilan LLM di jalur retrieval**: embedding lokal, satu CPU, deterministik. -| Metrik | Uteke **v0.17.0** | agentmemory¹ | BM25-only¹ | +| Metrik | Uteke **v0.18.1** | agentmemory¹ | BM25-only¹ | |---|---|---|---| | **recall_any@5** (bukti di top-5) | **98.4%** | 95.2% | 86.2% | | recall_any@10 | 98.8% | 98.6% | 94.6% | @@ -92,7 +92,7 @@ lokal, satu CPU, deterministik. ¹ Angka publikasi agentmemory, benchmark yang sama, split 500 pertanyaan yang sama (basis recall_any@5 mereka; terverifikasi apples-to-apples di [head-to-head](docs/benchmarks.md#head-to-head-vs-published-systems)). ² Strict = *semua* sesi gold harus masuk top-5; 65% pertanyaan butuh beberapa sesi. Ceiling matematis 99.4%. ³ MRR, bukan recall_all (tidak bisa dibandingkan langsung; ditampilkan untuk kelengkapan).

- LongMemEval-S recall@5: uteke 98.4% (revalidasi v0.18.0) vs MemPalace 96.6% dan agentmemory 95.2% (hasil raw di-commit di repo) + LongMemEval-S recall@5: uteke 98.4% (revalidasi v0.18.1) vs MemPalace 96.6% dan agentmemory 95.2% (hasil raw di-commit di repo)

**Per kategori pertanyaan** (recall_any@5: cerita per kategori yang jarang ditampilkan tool lain): @@ -384,7 +384,7 @@ Bisa. Uteke punya MCP server yang langsung pakai dengan Claude Code, Cursor, dan
Sudah production-ready? -Uteke sekarang v0.17.0 dengan 200+ test, CI/CD di setiap commit, dan benchmark harness. Dipakai production oleh tim CodeCora dan early adopter lain. Masih di versi 0.x, jadi mungkin ada rough edges, tapi core-nya udah stabil. +Uteke sekarang v0.18.1 dengan 200+ test, CI/CD di setiap commit, dan benchmark harness. Dipakai production oleh tim CodeCora dan early adopter lain. Masih di versi 0.x, jadi mungkin ada rough edges, tapi core-nya udah stabil.
--- diff --git a/README.md b/README.md index 43742605..edf19496 100644 --- a/README.md +++ b/README.md @@ -78,7 +78,7 @@ finds the evidence. 500 hand-curated questions, five memory abilities. Uteke run the full suite with **zero LLM calls in the retrieval path**: local embeddings, one CPU, deterministic. -| Metric | uteke **v0.18.0** | agentmemory¹ | BM25-only¹ | +| Metric | uteke **v0.18.1** | agentmemory¹ | BM25-only¹ | |---|---|---|---| | **recall_any@5** (evidence in top-5) | **98.4%** | 95.2% | 86.2% | | recall_any@10 | 98.8% | 98.6% | 94.6% | @@ -88,7 +88,7 @@ one CPU, deterministic. ¹ agentmemory's published numbers, same benchmark, same 500-question split (their recall_any@5 basis; verified apples-to-apples in our [head-to-head](docs/benchmarks.md#head-to-head-vs-published-systems)). ² Strict = *every* gold session in top-5; 65% of questions have multiple gold sessions. Mathematical ceiling 99.4%. ³ MRR, not recall_all (not directly comparable; shown for completeness).

- LongMemEval-S recall@5: uteke 98.4% (revalidated v0.18.0) vs MemPalace 96.6% and agentmemory 95.2% (raw results committed in-repo) + LongMemEval-S recall@5: uteke 98.4% (revalidated v0.18.1) vs MemPalace 96.6% and agentmemory 95.2% (raw results committed in-repo)

**By question category** (recall_any@5: the category-level story most tools don't show): @@ -345,7 +345,7 @@ Yes. Uteke ships with an MCP server that works with Claude Code, Cursor, and Her
Is it production-ready? -Uteke is at v0.17.0 with 200+ tests, CI/CD on every commit, and a benchmark harness. It's used in production by the CodeCora team and other early adopters. Still in 0.x, so expect rough edges, but the core is stable. +Uteke is at v0.18.1 with 200+ tests, CI/CD on every commit, and a benchmark harness. It's used in production by the CodeCora team and other early adopters. Still in 0.x, so expect rough edges, but the core is stable.
--- diff --git a/benchmarks/longmemeval/.gitignore b/benchmarks/longmemeval/.gitignore index 70c9fb25..6583fca5 100644 --- a/benchmarks/longmemeval/.gitignore +++ b/benchmarks/longmemeval/.gitignore @@ -17,9 +17,3 @@ results_*/ __pycache__/ *.pyc .venv/ - -# Fast-eval datasets are derived from the parent dataset (regenerate with make_fast_eval.py) -data/longmemeval_fast50.json -data/longmemeval_fast10.json -data/longmemeval_temporal15.json -data/longmemeval_multisession15.json diff --git a/benchmarks/longmemeval/check_volume.py b/benchmarks/longmemeval/check_volume.py deleted file mode 100644 index 69d76da3..00000000 --- a/benchmarks/longmemeval/check_volume.py +++ /dev/null @@ -1,27 +0,0 @@ -#!/usr/bin/env python3 -"""Quick check: shard mana saja yang sudah tersimpan di Modal Volume.""" -import modal - -vol = modal.Volume.from_name("uteke-longmemeval", create_if_missing=True) - -app = modal.App("check-lmeval-volume") - -@app.function(volumes={"/root/vol": vol}, timeout=120) -def check(strategy: str) -> list: - import pathlib - base = pathlib.Path("/root/vol") / strategy - out = [] - if base.exists(): - for f in sorted(base.glob("shard_*.jsonl")): - n = sum(1 for l in f.read_text().splitlines() if l.strip()) - out.append((f.name, n)) - return out - -@app.local_entrypoint() -def main(): - for strategy in ["hybrid", "vector"]: - res = check.remote(strategy) - total = sum(n for _, n in res) - print(f"{strategy}: {len(res)}/20 shard selesai, total {total} pertanyaan") - for name, n in res: - print(f" {name}: {n} Q") diff --git a/benchmarks/longmemeval/compare_fast50.py b/benchmarks/longmemeval/compare_fast50.py deleted file mode 100644 index 39f90a07..00000000 --- a/benchmarks/longmemeval/compare_fast50.py +++ /dev/null @@ -1,73 +0,0 @@ -#!/usr/bin/env python3 -"""Compare baseline vs rerank fast-eval results (#1118). - -Usage: python3 compare_fast50.py -""" -import json -import sys -from pathlib import Path -from collections import defaultdict - - -def load(d): - rows = {} - p = Path(d) / "retrieval_results.jsonl" - for line in p.read_text().splitlines(): - if not line.strip(): - continue - r = json.loads(line) - m = (r.get("retrieval_results", {}) or {}).get("metrics") or {} - sm = (m.get("session") or {}) - rows[r["question_id"]] = { - "type": r["question_type"], - "r5": sm.get("recall_all@5"), - "r10": sm.get("recall_all@10"), - } - return rows - - -def agg(rows): - by_type = defaultdict(list) - for v in rows.values(): - if v["r5"] is not None: - by_type[v["type"]].append(v["r5"]) - out = {} - allv = [] - for t, vals in by_type.items(): - out[t] = sum(vals) / len(vals) - allv += vals - out["OVERALL"] = sum(allv) / len(allv) if allv else 0.0 - return out - - -def main(): - base = load(sys.argv[1]) - rerank = load(sys.argv[2]) - a, b = agg(base), agg(rerank) - - print(f"{'type':30s} {'n':>3s} {'baseline':>9s} {'rerank':>9s} {'delta':>8s}") - types = sorted(set(a) | set(b)) - for t in types: - d = b.get(t, 0) - a.get(t, 0) - n = sum(1 for v in rerank.values() if v["type"] == t and v["r5"] is not None) - flag = " <<<" if d >= 0.03 and t != "OVERALL" else "" - print(f"{t:30s} {n:3d} {a.get(t, 0):9.4f} {b.get(t, 0):9.4f} {d:+8.4f}{flag}") - - dd = b["OVERALL"] - a["OVERALL"] - print() - print(f"OVERALL: {a['OVERALL']:.4f} -> {b['OVERALL']:.4f} ({dd:+.4f})") - gate = dd >= 0.03 - print(f"GATE (+3pp): {'PASS ✅' if gate else 'FAIL ❌'}") - - # per-Q movers - up = sum(1 for q in base if q in rerank and rerank[q]["r5"] is not None - and base[q]["r5"] is not None and rerank[q]["r5"] > base[q]["r5"]) - down = sum(1 for q in base if q in rerank and rerank[q]["r5"] is not None - and base[q]["r5"] is not None and rerank[q]["r5"] < base[q]["r5"]) - print(f"Q improved: {up} Q regressed: {down}") - - return 0 if gate else 1 - - -if __name__ == "__main__": - sys.exit(main()) diff --git a/benchmarks/longmemeval/contradiction_segment.py b/benchmarks/longmemeval/contradiction_segment.py deleted file mode 100644 index b6f205dd..00000000 --- a/benchmarks/longmemeval/contradiction_segment.py +++ /dev/null @@ -1,271 +0,0 @@ -#!/usr/bin/env python3 -""" -Contradiction-segment benchmark (#1172 Fase 3). - -Measures how conflict resolution (supersede, #1053/#1172) affects retrieval -quality on a synthetic knowledge-update workload. This is the ACTIVE-store -counterpart to LongMemEval's passive knowledge-update subset (subset_kupd): -instead of asking whether stale sessions are retrieved, we resolve conflicts -in the store first and ask whether recall surfaces the WINNER. - -Design (deterministic, N topics): - 1. Seed 2 memories per topic: - - stale: the OLD fact ("… uses tool X") - - winner: the NEW fact ("… switched to tool Y") - plus D distractor memories per topic (same domain vocabulary, no conflict). - 2. Baseline run: both facts active (no supersede). Query each topic - semantically ("what does topic use now?"). Measures how often the - stale fact pollutes top-k when nothing resolved the conflict. - 3. Resolved run: supersede(stale → winner) via the CLI, then re-query. - Measures winner@k and stale@k on the RESOLVED store. - 4. Ledger sanity: contradiction_resolutions lists the pair; undo restores - (audited, then re-superseded). - -Metrics per strategy: - winner@{1,3,5} — winner memory ranked in top-k - stale@{1,5} — stale memory present in top-k (0.0 expected post-resolve) - MRR (winner) — reciprocal rank of the winner - -Usage: - python3 contradiction_segment.py --binary ../../target/release/uteke --topics 40 - python3 contradiction_segment.py --topics 40 --json out.json -""" - -import argparse -import json -import shutil -import subprocess -import sys -import tempfile -from pathlib import Path - -TOPICS = [ - ("acme-corp", "build tooling", "gradle", "bazel"), - ("brightpath", "package manager", "yarn", "pnpm"), - ("cloudline", "hosting provider", "heroku", "fly.io"), - ("dataworks", "message queue", "rabbitmq", "kafka"), - ("everhost", "web server", "apache", "nginx"), - ("fintrak", "ledger database", "postgresql", "cockroachdb"), - ("gadgethub", "mobile framework", "cordova", "flutter"), - ("heliosoft", "ci system", "jenkins", "github actions"), - ("innodata", "search engine", "elasticsearch", "meilisearch"), - ("jetstream", "api style", "soap", "grpc"), - ("kobalt", "css framework", "bootstrap", "tailwind"), - ("lumenpath", "state management", "redux", "zustand"), - ("metriq", "observability stack", "graphite", "prometheus"), - ("novabyte", "language runtime", "java", "kotlin"), - ("orbita", "container runtime", "docker swarm", "kubernetes"), - ("pixelbay", "image format", "jpeg-xl", "avif"), - ("quantex", "config format", "ini", "toml"), - ("riverbend", "version control", "svn", "git"), - ("saltmarsh", "auth protocol", "basic auth", "oauth2"), - ("tidewater", "template engine", "ejs", "handlebars"), - ("umbracloud", "storage layer", "mongodb", "sqlite"), - ("vellum", "docs generator", "javadoc", "rustdoc"), - ("wharfside", "package registry", "nexus", "ghcr"), - ("xenolith", "testing framework", "junit4", "junit5"), - ("yarrow", "scheduler", "cron", "systemd timers"), - ("zephyr", "linting tool", "tslint", "eslint"), - ("argonhold", "secret store", "env files", "vault"), - ("basaltix", "logging library", "log4j", "tracing"), - ("cobaltrun", "runtime monitor", "new relic", "otel"), - ("duskfield", "error tracker", "rollbar", "sentry"), - ("emberfall", "feature flags", "launchdarkly", "unleash"), - ("frostline", "cache layer", "memcached", "redis"), - ("glacierpeak", "object store", "s3 class", "r2"), - ("hollowpine", "markdown parser", "marked", "comrak"), - ("irisvale", "date library", "moment", "dayjs"), - ("jaderock", "http client", "axios", "fetch"), - ("kelpforest", "orm", "sequelize", "drizzle"), - ("lavaglass", "bundler", "webpack", "vite"), - ("mistvale", "type checker", "flow", "typescript"), - ("nightsky", "charting library", "chart.js", "d3"), -] - -# Question phrasings deliberately avoid the exact "uses/switched to" verbs -# so the query is semantic, not keyword lookup. -QUESTION = "which {thing} does {topic} use now?" - - -def resolve_binary(cli_path: str) -> str: - if cli_path: - p = Path(cli_path) - if p.exists(): - return str(p) - print(f"warning: --binary {cli_path} missing; falling back", file=sys.stderr) - repo = Path(__file__).resolve().parent.parent.parent - cand = repo / "target" / "release" / "uteke" - if cand.exists(): - return str(cand) - for c in ("/opt/data/.local/bin/uteke",): - if Path(c).exists(): - return str(c) - return shutil.which("uteke") or "uteke" - - -def uteke(binary: str, store: Path, namespace: str, args: list[str]) -> dict | list: - cmd = [ - binary, - "--store", str(store), - "--namespace", namespace, - "--json", - *args, - ] - result = subprocess.run(cmd, capture_output=True, text=True, timeout=600) - if result.returncode != 0: - raise RuntimeError(f"uteke {' '.join(args)} failed: {result.stderr[:400]}") - return json.loads(result.stdout) - - -def remember(binary, store, ns, content: str) -> str: - out = uteke(binary, store, ns, ["remember", content]) - return str(out["id"]) # type: ignore[no-any-return] - - -def supersede(binary, store, ns, old: str, new: str) -> None: - uteke(binary, store, ns, ["supersede", old, new, "--reason", "benchmark conflict resolution"]) - - -def recall_ids(binary, store, ns, query: str, strategy: str, k: int) -> list[str]: - out = uteke(binary, store, ns, [ - "recall", query, - "--limit", str(k), - "--min", "0.0", - "--strategy", strategy, - ]) - return [m["memory_id"] for m in out] - - -def recall_map(binary, store, ns) -> dict[str, str]: - """id → content for the namespace (to map ids back to roles).""" - out = uteke(binary, store, ns, ["list", "--limit", "500"]) - return {m["id"]: m["content"] for m in out} - - -def metrics_at(ranking: list[str], winner: str, stale: str) -> tuple[float, float, float, float, float]: - w = lambda k: 1.0 if winner in ranking[:k] else 0.0 - s = lambda k: 1.0 if stale in ranking[:k] else 0.0 - rr = 0.0 - for i, mid in enumerate(ranking, start=1): - if mid == winner: - rr = 1.0 / i - break - return w(1), w(5), rr, s(1), s(5) - - -def main() -> int: - ap = argparse.ArgumentParser(description="#1172 F3 contradiction segment") - ap.add_argument("--binary", default="", help="uteke binary path") - ap.add_argument("--store", default="", help="existing store to reuse (default: temp)") - ap.add_argument("--namespace", default="bench-contradiction") - ap.add_argument("--distractors", type=int, default=3, help="distractors per topic") - ap.add_argument("--topics", type=int, default=0, help="limit topics (0 = all)") - ap.add_argument("--strategy", default="fusion") - ap.add_argument("--json", default="", help="write metrics JSON here") - args = ap.parse_args() - - topics = TOPICS[: args.topics] if args.topics > 0 else TOPICS - binary = resolve_binary(args.binary) - print(f"binary: {binary}") - - tmp = None - if args.store: - store = Path(args.store) - else: - tmp = tempfile.TemporaryDirectory(prefix="uteke-contradiction-") - store = Path(tmp.name) / "bench.uteke" - - ns = args.namespace - roles: dict[str, tuple[str, str]] = {} # winner id → (stale id, topic) - topic_thing = {t: th for t, th, _o, _n in topics} - - # ── Seed ──────────────────────────────────────────────────────────── - print(f"seeding {len(topics)} topics (+{args.distractors} distractors each)…") - distractor_pool = [ - "weekly sync notes and standup summaries", - "onboarding checklist for new engineers", - "retro action items from the last sprint", - "vendor invoice and billing contacts", - "conference talk notes and takeaways", - ] - for topic, thing, old_tool, new_tool in topics: - stale = remember(binary, store, ns, - f"{topic} uses {old_tool} for {thing}. Decision recorded after evaluation.") - winner = remember(binary, store, ns, - f"{topic} switched to {new_tool} for {thing}. The old {old_tool} setup is retired.") - roles[winner] = (stale, topic) - for d in range(args.distractors): - remember(binary, store, ns, - f"{topic} {distractor_pool[d % len(distractor_pool)]} ({thing} context {d})") - - strategies = [s.strip() for s in args.strategy.split(",") if s.strip()] - results: dict[str, dict] = {} - - def measure(strategy: str) -> dict: - w1s = w5s = rrs = ss1 = ss5 = 0.0 - n = 0 - for winner, (stale, topic) in roles.items(): - q = QUESTION.format(thing=topic_thing[topic], topic=topic) - ranking = recall_ids(binary, store, ns, q, strategy, 10) - w1, w5, rr, s1, s5 = metrics_at(ranking, winner, stale) - w1s += w1; w5s += w5; rrs += rr; ss1 += s1; ss5 += s5 - n += 1 - return { - "winner@1": w1s / n, "winner@5": w5s / n, "winner_mrr": rrs / n, - "stale@1": ss1 / n, "stale@5": ss5 / n, "n": n, - } - - # ── Baseline for ALL strategies on the UNRESOLVED store first ────── - # (code-scanning fix: resolving per-strategy left later strategies - # measuring "baseline" on an already-resolved store.) - baselines = {s: measure(s) for s in strategies} - - # ── Resolve once: supersede every stale → winner ─────────────────── - for winner, (stale, _topic) in roles.items(): - supersede(binary, store, ns, stale, winner) - - # Ledger sanity: every resolution is listed (F2 surface, in-loop). - ledger_raw = subprocess.run( - [binary, "--store", str(store), "--json", - "contradictions", "list", "--namespace", ns, "--limit", "500"], - capture_output=True, text=True, timeout=600, - ) - ledger = json.loads(ledger_raw.stdout) - listed = {e["id"] for e in ledger} - ledger_ok = all(stale in listed for _w, (stale, _t) in roles.items()) - - # ── Resolved metrics for all strategies ──────────────────────────── - resolved = {s: measure(s) for s in strategies} - - for strategy in strategies: - baseline = {k: v for k, v in baselines[strategy].items() if k != "n"} - res = {k: v for k, v in resolved[strategy].items() if k != "n"} - n = baselines[strategy]["n"] - results[strategy] = { - "questions": n, - "baseline_unresolved": baseline, - "resolved": res, - "ledger_listed_all": ledger_ok, - } - print(f"\n[{strategy}] n={n}") - print(f" baseline (unresolved): winner@1={baseline['winner@1']:.3f} " - f"winner@5={baseline['winner@5']:.3f} MRR={baseline['winner_mrr']:.3f} " - f"stale@1={baseline['stale@1']:.3f} stale@5={baseline['stale@5']:.3f}") - print(f" resolved (superseded): winner@1={res['winner@1']:.3f} " - f"winner@5={res['winner@5']:.3f} MRR={res['winner_mrr']:.3f} " - f"stale@1={res['stale@1']:.3f} stale@5={res['stale@5']:.3f}") - print(f" ledger lists all resolutions: {ledger_ok}") - - if args.json: - out = Path(args.json) - out.parent.mkdir(parents=True, exist_ok=True) - out.write_text(json.dumps(results, indent=2)) - print(f"\nmetrics → {out}") - - if tmp: - tmp.cleanup() - return 0 - - -if __name__ == "__main__": - sys.exit(main()) diff --git a/benchmarks/longmemeval/fast_eval_ids.json b/benchmarks/longmemeval/fast_eval_ids.json deleted file mode 100644 index 79ed15fb..00000000 --- a/benchmarks/longmemeval/fast_eval_ids.json +++ /dev/null @@ -1,100 +0,0 @@ -{ - "fast50": [ - "7527f7e2", - "75499fd8", - "8e9d538c", - "3d86fd0a", - "0862e8bf_abs", - "bc8a6e93_abs", - "29f2956b_abs", - "gpt4_a56e767c", - "b3c15d39", - "60472f9c", - "2b8f3739", - "c2ac3c61", - "eeda8a6d_abs", - "35a27287", - "57f827a0", - "75f70248", - "2311e44b", - "85fa3a3f", - "ef66a6e5", - "92a0aa75", - "bc149d6b", - "55241a1f", - "a08a253f", - "gpt4_f49edff3", - "gpt4_468eb063", - "6e984301", - "gpt4_8e165409", - "6e984302", - "gpt4_8279ba03", - "gpt4_b5700ca0", - "08f4fc43", - "gpt4_385a5000", - "a3838d2b", - "gpt4_b4a80587", - "gpt4_8c8961ae", - "982b5123_abs", - "5a4f22c0", - "6071bd76", - "8fb83627", - "b01defab", - "4b24c848", - "a2f3aa27", - "031748ae_abs", - "0ddfec37_abs", - "e9327a54", - "58470ed2", - "3e321797", - "e982271f", - "5809eb10", - "c8f1aeed" - ], - "fast10": [ - "7527f7e2", - "75499fd8", - "8e9d538c", - "3d86fd0a", - "0862e8bf_abs", - "bc8a6e93_abs", - "29f2956b_abs", - "gpt4_a56e767c", - "b3c15d39", - "60472f9c" - ], - "temporal15": [ - "gpt4_f49edff3", - "gpt4_468eb063", - "6e984301", - "gpt4_8e165409", - "6e984302", - "gpt4_8279ba03", - "gpt4_b5700ca0", - "08f4fc43", - "gpt4_385a5000", - "a3838d2b", - "gpt4_b4a80587", - "gpt4_8c8961ae", - "982b5123_abs", - "gpt4_7ddcf75f", - "gpt4_7a0daae1" - ], - "multisession15": [ - "gpt4_a56e767c", - "b3c15d39", - "60472f9c", - "2b8f3739", - "c2ac3c61", - "eeda8a6d_abs", - "2311e44b", - "85fa3a3f", - "ef66a6e5", - "92a0aa75", - "bc149d6b", - "55241a1f", - "a08a253f", - "81507db6", - "8979f9ec" - ] -} \ No newline at end of file diff --git a/benchmarks/longmemeval/lmeval_final_metrics.txt b/benchmarks/longmemeval/lmeval_final_metrics.txt deleted file mode 100644 index 6b575165..00000000 --- a/benchmarks/longmemeval/lmeval_final_metrics.txt +++ /dev/null @@ -1,65 +0,0 @@ -============================================================ -Session-Level Retrieval Metrics -============================================================ - -Overall (470 questions): - recall_all@5 = 0.9093 - ndcg_any@5 = 0.8749 - recall_all@10 = 0.9635 - ndcg_any@10 = 0.8972 - -knowledge-update (72 questions): - recall_all@5 = 0.9861 - ndcg_any@5 = 0.9694 - recall_all@10 = 0.9931 - ndcg_any@10 = 0.9719 - -multi-session (121 questions): - recall_all@5 = 0.8869 - ndcg_any@5 = 0.8580 - recall_all@10 = 0.9623 - ndcg_any@10 = 0.8920 - -single-session-assistant (56 questions): - recall_all@5 = 0.9464 - ndcg_any@5 = 0.8955 - recall_all@10 = 0.9643 - ndcg_any@10 = 0.9015 - -single-session-preference (30 questions): - recall_all@5 = 0.9000 - ndcg_any@5 = 0.7405 - recall_all@10 = 0.9333 - ndcg_any@10 = 0.7524 - -single-session-user (64 questions): - recall_all@5 = 0.9844 - ndcg_any@5 = 0.9619 - recall_all@10 = 1.0000 - ndcg_any@10 = 0.9675 - -temporal-reasoning (127 questions): - recall_all@5 = 0.8350 - ndcg_any@5 = 0.8162 - recall_all@10 = 0.9362 - ndcg_any@10 = 0.8566 - -============================================================ -Turn-Level Retrieval Metrics -============================================================ - -(Not measured — this harness indexes sessions, not individual turns.) - -============================================================ -Summary Table (for README) -============================================================ - -| Question Type | Count | Recall@5 | Recall@10 | NDCG@5 | NDCG@10 | -|---------------|-------|----------|-----------|--------|---------| -| **Overall** | 470 | 0.909 | 0.963 | 0.875 | 0.897 | -| knowledge-update | 72 | 0.986 | 0.993 | 0.969 | 0.972 | -| multi-session | 121 | 0.887 | 0.962 | 0.858 | 0.892 | -| single-session-assistant | 56 | 0.946 | 0.964 | 0.896 | 0.901 | -| single-session-preference | 30 | 0.900 | 0.933 | 0.741 | 0.752 | -| single-session-user | 64 | 0.984 | 1.000 | 0.962 | 0.967 | -| temporal-reasoning | 127 | 0.835 | 0.936 | 0.816 | 0.857 | diff --git a/benchmarks/longmemeval/lmeval_relaunch.log b/benchmarks/longmemeval/lmeval_relaunch.log deleted file mode 100644 index 360f9215..00000000 --- a/benchmarks/longmemeval/lmeval_relaunch.log +++ /dev/null @@ -1,359 +0,0 @@ -✓ Initialized. View run at -https://modal.com/apps/ajianaz/main/ap-B81Y5ELh4Z92idroPfgB5j -✓ Created objects. -├── 🔨 Created mount -│ /opt/data/github/codecoradev/uteke/benchmarks/longmemeval/modal_fanout.py -├── 🔨 Created mount -│ /opt/data/github/codecoradev/uteke/benchmarks/longmemeval/run_eval.py -├── 🔨 Created mount -│ /opt/data/github/codecoradev/uteke/benchmarks/longmemeval/data/longmemeval_s -│ _cleaned.json -├── 🔨 Created mount /opt/data/.codecora/uteke/models/embeddinggemma-q4 -├── 🔨 Created function run_shard. -└── 🔨 Created function list_volume. -Fan-out: strategy=hybrid shards=20 limit=ALL -Stopping app - user stopped from CLI. -Runner terminated. -Runner terminated. -Runner terminated. -Runner terminated. -Runner terminated. -Runner terminated. -Runner terminated. -Runner terminated. -Runner terminated. -Runner terminated. -Runner terminated. -Runner terminated. -Runner terminated. -Runner terminated. -Runner terminated. -Runner terminated. -Runner terminated. -╭───────────────────── Traceback (most recent call last) ──────────────────────╮ -│ /opt/data/github/codecoradev/uteke/benchmarks/longmemeval/modal_fanout.py:19 │ -│ 7 in main │ -│ │ -│ 196 │ merged, seen = [], set() │ -│ ❱ 197 │ for res in run_shard.map(inputs): │ -│ 198 │ │ n_eval = 0 │ -│ │ -│ /opt/data/.local/lib/python3.13/site-packages/modal/_utils/async_utils.py:83 │ -│ 9 in __iter__ │ -│ │ -│ 838 │ │ │ with Runner() as runner: │ -│ ❱ 839 │ │ │ │ yield from run_async_gen(runner, self._async_iterable │ -│ 840 │ │ except NestedEventLoops: │ -│ │ -│ /opt/data/.local/lib/python3.13/site-packages/modal/_utils/async_utils.py:91 │ -│ 7 in run_async_gen │ -│ │ -│ 916 │ │ │ else: │ -│ ❱ 917 │ │ │ │ next_yield = runner.run(gen.asend(next_send)) # type │ -│ 918 │ │ except KeyboardInterrupt as e: │ -│ │ -│ /usr/lib/python3.13/asyncio/base_events.py:725 in run_until_complete │ -│ │ -│ 724 │ │ │ -│ ❱ 725 │ │ return future.result() │ -│ 726 │ -│ │ -│ /opt/data/.local/lib/python3.13/site-packages/modal/_utils/async_utils.py:74 │ -│ 9 in asend │ -│ │ -│ 748 │ │ self.iterated = True │ -│ ❱ 749 │ │ return await self.gen.asend(value) │ -│ 750 │ -│ │ -│ /opt/data/.local/lib/python3.13/site-packages/modal/parallel_map.py:1036 in │ -│ _map_async │ -│ │ -│ 1035 │ async_input_gen = async_zip(*[sync_or_async_iter(it) for it in in │ -│ ❱ 1036 │ async for output in _map_helper( │ -│ 1037 │ │ self, │ -│ │ -│ /opt/data/.local/lib/python3.13/site-packages/modal/parallel_map.py:1009 in │ -│ _map_helper │ -│ │ -│ 1008 │ ) as map_output_stream: │ -│ ❱ 1009 │ │ async for output in map_output_stream: │ -│ 1010 │ │ │ yield output │ -│ │ -│ /opt/data/.local/lib/python3.13/site-packages/modal/_utils/async_utils.py:10 │ -│ 91 in async_merge │ -│ │ -│ 1090 │ │ │ │ │ assert_type(item, ExceptionWrapper) │ -│ ❱ 1091 │ │ │ │ │ raise item.value │ -│ 1092 │ -│ │ -│ /opt/data/.local/lib/python3.13/site-packages/modal/_utils/async_utils.py:10 │ -│ 70 in producer │ -│ │ -│ 1069 │ │ │ async with aclosing(generator) as stream: │ -│ ❱ 1070 │ │ │ │ async for item in stream: │ -│ 1071 │ │ │ │ │ await queue.put(ValueWrapper(item)) │ -│ │ -│ /opt/data/.local/lib/python3.13/site-packages/modal/_object.py:55 in wrapped │ -│ │ -│ 54 │ │ async with aclosing(method(self, *args, **kwargs)) as stream: │ -│ ❱ 55 │ │ │ async for item in stream: │ -│ 56 │ │ │ │ yield item │ -│ │ -│ /opt/data/.local/lib/python3.13/site-packages/modal/_functions.py:1650 in │ -│ _map │ -│ │ -│ 1649 │ │ │ ) as stream: │ -│ ❱ 1650 │ │ │ │ async for item in stream: │ -│ 1651 │ │ │ │ │ yield item │ -│ │ -│ /opt/data/.local/lib/python3.13/site-packages/modal/parallel_map.py:965 in │ -│ _map_invocation_inputplane │ -│ │ -│ 964 │ ) as merged: │ -│ ❱ 965 │ │ async for maybe_output in merged: │ -│ 966 │ │ │ if maybe_output is not None: # ignore None sentinels │ -│ │ -│ /opt/data/.local/lib/python3.13/site-packages/modal/_utils/async_utils.py:10 │ -│ 91 in async_merge │ -│ │ -│ 1090 │ │ │ │ │ assert_type(item, ExceptionWrapper) │ -│ ❱ 1091 │ │ │ │ │ raise item.value │ -│ 1092 │ -│ │ -│ /opt/data/.local/lib/python3.13/site-packages/modal/_utils/async_utils.py:10 │ -│ 70 in producer │ -│ │ -│ 1069 │ │ │ async with aclosing(generator) as stream: │ -│ ❱ 1070 │ │ │ │ async for item in stream: │ -│ 1071 │ │ │ │ │ await queue.put(ValueWrapper(item)) │ -│ │ -│ /opt/data/.local/lib/python3.13/site-packages/modal/parallel_map.py:921 in │ -│ poll_outputs │ -│ │ -│ 920 │ │ ) as streamer: │ -│ ❱ 921 │ │ │ async for idx, output in streamer: │ -│ 922 │ │ │ │ if not order_outputs: │ -│ │ -│ /opt/data/.local/lib/python3.13/site-packages/modal/_utils/async_utils.py:12 │ -│ 20 in async_map_ordered │ -│ │ -│ 1219 │ async with aclosing(async_map(async_zip(counter(), input_generato │ -│ ❱ 1220 │ │ async for output_idx, output_item in stream: │ -│ 1221 │ │ │ buffer[output_idx] = output_item │ -│ │ -│ /opt/data/.local/lib/python3.13/site-packages/modal/_utils/async_utils.py:11 │ -│ 96 in async_map │ -│ │ -│ 1195 │ ) as stream: │ -│ ❱ 1196 │ │ async for item in stream: │ -│ 1197 │ │ │ yield item │ -│ │ -│ /opt/data/.local/lib/python3.13/site-packages/modal/_utils/async_utils.py:10 │ -│ 91 in async_merge │ -│ │ -│ 1090 │ │ │ │ │ assert_type(item, ExceptionWrapper) │ -│ ❱ 1091 │ │ │ │ │ raise item.value │ -│ 1092 │ -│ │ -│ /opt/data/.local/lib/python3.13/site-packages/modal/_utils/async_utils.py:10 │ -│ 70 in producer │ -│ │ -│ 1069 │ │ │ async with aclosing(generator) as stream: │ -│ ❱ 1070 │ │ │ │ async for item in stream: │ -│ 1071 │ │ │ │ │ await queue.put(ValueWrapper(item)) │ -│ │ -│ /opt/data/.local/lib/python3.13/site-packages/modal/_utils/async_utils.py:11 │ -│ 85 in worker │ -│ │ -│ 1184 │ │ │ if isinstance(item, ValueWrapper): │ -│ ❱ 1185 │ │ │ │ res = await prevent_cancellation_abortion(async_mappe │ -│ 1186 │ │ │ │ yield res │ -│ │ -│ /opt/data/.local/lib/python3.13/site-packages/modal/_utils/async_utils.py:11 │ -│ 50 in prevent_cancellation_abortion │ -│ │ -│ 1149 │ try: │ -│ ❱ 1150 │ │ return await asyncio.shield(t) │ -│ 1151 │ except asyncio.CancelledError: │ -│ │ -│ /opt/data/.local/lib/python3.13/site-packages/modal/_utils/async_utils.py:12 │ -│ 09 in mapper_func_wrapper │ -│ │ -│ 1208 │ async def mapper_func_wrapper(tup: tuple[int, T]) -> tuple[int, V │ -│ ❱ 1209 │ │ return (tup[0], await async_mapper_func(tup[1])) │ -│ 1210 │ -│ │ -│ /opt/data/.local/lib/python3.13/site-packages/modal/parallel_map.py:910 in │ -│ fetch_output │ -│ │ -│ 909 │ │ │ else: │ -│ ❱ 910 │ │ │ │ raise e │ -│ 911 │ │ return (item.idx, output) │ -│ │ -│ /opt/data/.local/lib/python3.13/site-packages/modal/parallel_map.py:905 in │ -│ fetch_output │ -│ │ -│ 904 │ │ try: │ -│ ❱ 905 │ │ │ output = await _process_result(item.result, item.data_for │ -│ 906 │ │ except Exception as e: │ -│ │ -│ /opt/data/.local/lib/python3.13/site-packages/modal/_utils/function_utils.py │ -│ :550 in _process_result │ -│ │ -│ 549 │ │ │ -│ ❱ 550 │ │ raise RemoteError(result.exception) │ -│ 551 │ -╰──────────────────────────────────────────────────────────────────────────────╯ -RemoteError -✓ Initialized. View run at -https://modal.com/apps/ajianaz/main/ap-19sx5htEuCgLw7mifYtwhV -✓ Created objects. -├── 🔨 Created mount -│ /opt/data/github/codecoradev/uteke/benchmarks/longmemeval/modal_fanout.py -├── 🔨 Created mount -│ /opt/data/github/codecoradev/uteke/benchmarks/longmemeval/data/longmemeval_s -│ _cleaned.json -├── 🔨 Created mount -│ /opt/data/github/codecoradev/uteke/benchmarks/longmemeval/run_eval.py -├── 🔨 Created mount /opt/data/.codecora/uteke/models/embeddinggemma-q4 -├── 🔨 Created function run_shard. -└── 🔨 Created function list_volume. -Fan-out: strategy=hybrid shards=20 limit=ALL -✓ Initialized. View run at -https://modal.com/apps/ajianaz/main/ap-H2Wp8hhTGBEmLMBx9TXYzW -✓ Created objects. -├── 🔨 Created mount -│ /opt/data/github/codecoradev/uteke/benchmarks/longmemeval/modal_fanout.py -├── 🔨 Created mount -│ /opt/data/github/codecoradev/uteke/benchmarks/longmemeval/data/longmemeval_s -│ _cleaned.json -├── 🔨 Created mount -│ /opt/data/github/codecoradev/uteke/benchmarks/longmemeval/run_eval.py -├── 🔨 Created mount /opt/data/.codecora/uteke/models/embeddinggemma-q4 -├── 🔨 Created function run_shard. -└── 🔨 Created function list_volume. -Fan-out: strategy=hybrid shards=20 limit=ALL -Container terminated due to preemption. Your Function will be restarted with the same input. For more details, see https://modal.com/docs/guide/preemption -Container terminated due to preemption. Your Function will be restarted with the same input. For more details, see https://modal.com/docs/guide/preemption -Runner failed with exception: Worker disappeared, in-progress inputs will be re-scheduled. -Container terminated due to preemption. Your Function will be restarted with the same input. For more details, see https://modal.com/docs/guide/preemption -Container terminated due to preemption. Your Function will be restarted with the same input. For more details, see https://modal.com/docs/guide/preemption -Container terminated due to preemption. For more details, see https://modal.com/docs/guide/preemption -Runner terminated. - shard 0: 25/25 questions in 4190.6s - shard 1: 25/25 questions in 4059.8s - shard 2: 25/25 questions in 4094.0s -Container terminated due to preemption. Your Function will be restarted with the same input. For more details, see https://modal.com/docs/guide/preemption -Container terminated due to preemption. Your Function will be restarted with the same input. For more details, see https://modal.com/docs/guide/preemption -Container terminated due to preemption. Your Function will be restarted with the same input. For more details, see https://modal.com/docs/guide/preemption -Container terminated due to preemption. Your Function will be restarted with the same input. For more details, see https://modal.com/docs/guide/preemption -Container terminated due to preemption. Your Function will be restarted with the same input. For more details, see https://modal.com/docs/guide/preemption -Container terminated due to preemption. Your Function will be restarted with the same input. For more details, see https://modal.com/docs/guide/preemption - shard 3: 25/25 questions in 5802.8s - shard 4: 25/25 questions in 4099.4s - shard 5: 25/25 questions in 4246.5s - shard 6: 25/25 questions in 4175.9s - shard 7: 25/25 questions in 5948.5s - shard 8: 25/25 questions in 5910.1s - shard 9: 25/25 questions in 4138.0s - shard 10: 25/25 questions in 5969.8s - shard 11: 25/25 questions in 4112.3s - shard 12: 25/25 questions in 5010.1s - shard 13: 25/25 questions in 5690.3s - shard 14: 25/25 questions in 4276.1s - shard 15: 25/25 questions in 5263.5s -WARNING: Logs may not be continuous -WARNING: Logs may not be continuous -WARNING: Logs may not be continuous -WARNING: Logs may not be continuous -WARNING: Logs may not be continuous -WARNING: Logs may not be continuous -WARNING: Logs may not be continuous -WARNING: Logs may not be continuous -WARNING: Logs may not be continuous -WARNING: Logs may not be continuous -WARNING: Logs may not be continuous -WARNING: Logs may not be continuous -WARNING: Logs may not be continuous -WARNING: Logs may not be continuous -WARNING: Logs may not be continuous -WARNING: Logs may not be continuous -WARNING: Logs may not be continuous -WARNING: Logs may not be continuous - shard 16: 25/25 questions in 5084.4s - shard 17: 25/25 questions in 4178.2s - shard 18: 25/25 questions in 4103.4s - shard 19: 25/25 questions in 5038.7s - -Done: 500 questions in 15929s wall clock -Results: results_modal_hybrid/retrieval_results.jsonl - -Run: python print_metrics.py results_modal_hybrid/retrieval_results.jsonl -============================================================ -Session-Level Retrieval Metrics -============================================================ - -Overall (470 questions): - recall_all@5 = 0.9093 - ndcg_any@5 = 0.8749 - recall_all@10 = 0.9635 - ndcg_any@10 = 0.8972 - -knowledge-update (72 questions): - recall_all@5 = 0.9861 - ndcg_any@5 = 0.9694 - recall_all@10 = 0.9931 - ndcg_any@10 = 0.9719 - -multi-session (121 questions): - recall_all@5 = 0.8869 - ndcg_any@5 = 0.8580 - recall_all@10 = 0.9623 - ndcg_any@10 = 0.8920 - -single-session-assistant (56 questions): - recall_all@5 = 0.9464 - ndcg_any@5 = 0.8955 - recall_all@10 = 0.9643 - ndcg_any@10 = 0.9015 - -single-session-preference (30 questions): - recall_all@5 = 0.9000 - ndcg_any@5 = 0.7405 - recall_all@10 = 0.9333 - ndcg_any@10 = 0.7524 - -single-session-user (64 questions): - recall_all@5 = 0.9844 - ndcg_any@5 = 0.9619 - recall_all@10 = 1.0000 - ndcg_any@10 = 0.9675 - -temporal-reasoning (127 questions): - recall_all@5 = 0.8350 - ndcg_any@5 = 0.8162 - recall_all@10 = 0.9362 - ndcg_any@10 = 0.8566 - -============================================================ -Turn-Level Retrieval Metrics -============================================================ - -(Not measured — this harness indexes sessions, not individual turns.) - -============================================================ -Summary Table (for README) -============================================================ - -| Question Type | Count | Recall@5 | Recall@10 | NDCG@5 | NDCG@10 | -|---------------|-------|----------|-----------|--------|---------| -| **Overall** | 470 | 0.909 | 0.963 | 0.875 | 0.897 | -| knowledge-update | 72 | 0.986 | 0.993 | 0.969 | 0.972 | -| multi-session | 121 | 0.887 | 0.962 | 0.858 | 0.892 | -| single-session-assistant | 56 | 0.946 | 0.964 | 0.896 | 0.901 | -| single-session-preference | 30 | 0.900 | 0.933 | 0.741 | 0.752 | -| single-session-user | 64 | 0.984 | 1.000 | 0.962 | 0.967 | -| temporal-reasoning | 127 | 0.835 | 0.936 | 0.816 | 0.857 | -Stopping app - local entrypoint completed. -✓ App completed. View run at -https://modal.com/apps/ajianaz/main/ap-H2Wp8hhTGBEmLMBx9TXYzW diff --git a/benchmarks/longmemeval/lmeval_watch.log b/benchmarks/longmemeval/lmeval_watch.log deleted file mode 100644 index 9d04e3c2..00000000 --- a/benchmarks/longmemeval/lmeval_watch.log +++ /dev/null @@ -1,104 +0,0 @@ -[15:59:42] watcher start: strategy=hybrid shards=20 interval=600s -[15:59:53] pull: 0/20 shard, 0 Q merged → retrieval_results.jsonl -[15:59:55] app tidak aktif → relaunch modal_fanout (resume dari Volume) -[16:03:15] watcher start: strategy=hybrid shards=20 interval=600s -[16:03:24] pull: 0/20 shard, 0 Q merged → retrieval_results.jsonl -[16:03:26] app masih aktif -[16:13:35] pull: 0/20 shard, 0 Q merged → retrieval_results.jsonl -[16:13:37] app masih aktif -[16:22:04] watcher start: strategy=hybrid shards=20 interval=600s -[16:22:13] pull: 0/20 shard, 0 Q merged → retrieval_results.jsonl -[16:22:14] report: OK -[16:22:16] app masih aktif -[16:32:26] pull: 0/20 shard, 0 Q merged → retrieval_results.jsonl -[16:32:27] report: OK -[16:32:29] app masih aktif -[16:42:38] pull: 0/20 shard, 0 Q merged → retrieval_results.jsonl -[16:42:39] report: OK -[16:42:42] app tidak aktif → relaunch modal_fanout (resume dari Volume) -[16:52:52] pull: 0/20 shard, 0 Q merged → retrieval_results.jsonl -[16:52:53] report: OK -[16:52:55] app masih aktif -[17:00:10] watcher start: strategy=hybrid shards=20 interval=600s -[17:00:24] pull: 0/20 shard, 0 Q merged → retrieval_results.jsonl -[17:00:25] report: OK -[17:00:28] app tidak aktif → relaunch modal_fanout (resume dari Volume) -[17:10:37] pull: 0/20 shard, 0 Q merged → retrieval_results.jsonl -[17:10:38] report: OK -[17:10:40] app masih aktif -[17:20:49] pull: 0/20 shard, 0 Q merged → retrieval_results.jsonl -[17:20:50] report: OK -[17:20:52] app masih aktif -[17:31:02] pull: 0/20 shard, 0 Q merged → retrieval_results.jsonl -[17:31:03] report: OK -[17:31:05] app masih aktif -[17:41:14] pull: 0/20 shard, 0 Q merged → retrieval_results.jsonl -[17:41:14] report: OK -[17:41:17] app masih aktif -[17:51:27] pull: 0/20 shard, 0 Q merged → retrieval_results.jsonl -[17:51:28] report: OK -[17:51:31] app masih aktif -[18:01:39] pull: 0/20 shard, 0 Q merged → retrieval_results.jsonl -[18:01:40] report: OK -[18:01:42] app masih aktif -[18:11:52] pull: 5/20 shard, 125 Q merged → retrieval_results.jsonl -[18:11:53] report: OK -[18:11:55] app masih aktif -[18:22:06] pull: 5/20 shard, 125 Q merged → retrieval_results.jsonl -[18:22:07] report: OK -[18:22:09] app masih aktif -[18:32:19] pull: 7/20 shard, 175 Q merged → retrieval_results.jsonl -[18:32:19] report: OK -[18:32:21] app masih aktif -[18:42:32] pull: 12/20 shard, 300 Q merged → retrieval_results.jsonl -[18:42:33] report: OK -[18:42:36] app masih aktif -[18:52:46] pull: 13/20 shard, 325 Q merged → retrieval_results.jsonl -[18:52:46] report: OK -[18:52:49] app masih aktif -[19:02:59] pull: 13/20 shard, 325 Q merged → retrieval_results.jsonl -[19:03:00] report: OK -[19:03:02] app masih aktif -[19:13:11] pull: 15/20 shard, 375 Q merged → retrieval_results.jsonl -[19:13:12] report: OK -[19:13:14] app masih aktif -[19:23:25] pull: 15/20 shard, 375 Q merged → retrieval_results.jsonl -[19:23:26] report: OK -[19:23:28] app masih aktif -[19:33:38] pull: 16/20 shard, 400 Q merged → retrieval_results.jsonl -[19:33:39] report: OK -[19:33:41] app masih aktif -[19:43:52] pull: 16/20 shard, 400 Q merged → retrieval_results.jsonl -[19:43:53] report: OK -[19:43:56] app masih aktif -[19:54:07] pull: 18/20 shard, 450 Q merged → retrieval_results.jsonl -[19:54:08] report: OK -[19:54:11] app masih aktif -[20:04:20] pull: 18/20 shard, 450 Q merged → retrieval_results.jsonl -[20:04:21] report: OK -[20:04:25] app masih aktif -[20:14:36] pull: 18/20 shard, 450 Q merged → retrieval_results.jsonl -[20:14:37] report: OK -[20:14:39] app masih aktif -[20:24:50] pull: 19/20 shard, 475 Q merged → retrieval_results.jsonl -[20:24:51] report: OK -[20:24:53] app masih aktif -[20:35:06] pull: 19/20 shard, 475 Q merged → retrieval_results.jsonl -[20:35:07] report: OK -[20:35:10] app masih aktif -[20:45:29] pull: 19/20 shard, 475 Q merged → retrieval_results.jsonl -[20:45:34] report: OK -[20:45:40] app masih aktif -[20:55:54] pull: 19/20 shard, 475 Q merged → retrieval_results.jsonl -[20:55:55] report: OK -[20:56:00] app masih aktif -[21:06:14] pull: 19/20 shard, 475 Q merged → retrieval_results.jsonl -[21:06:15] report: OK -[21:06:19] app masih aktif -[21:16:34] pull: 19/20 shard, 475 Q merged → retrieval_results.jsonl -[21:16:35] report: OK -[21:16:39] app masih aktif -[21:26:53] pull: 20/20 shard, 500 Q merged → retrieval_results.jsonl -[21:26:55] report: OK -[21:26:55] SEMUA SHARD SELESAI — watcher berhenti -[21:26:55] metrik final tersimpan: lmeval_final_metrics.txt diff --git a/benchmarks/longmemeval/lmeval_watch_state.json b/benchmarks/longmemeval/lmeval_watch_state.json deleted file mode 100644 index 3ff2fa91..00000000 --- a/benchmarks/longmemeval/lmeval_watch_state.json +++ /dev/null @@ -1,5 +0,0 @@ -{ - "last_pull": "21:26:53", - "shards_done": 20, - "questions": 500 -} \ No newline at end of file diff --git a/benchmarks/longmemeval/make_fast_eval.py b/benchmarks/longmemeval/make_fast_eval.py deleted file mode 100644 index be83b51a..00000000 --- a/benchmarks/longmemeval/make_fast_eval.py +++ /dev/null @@ -1,87 +0,0 @@ -#!/usr/bin/env python3 -"""Regenerate the fast-eval datasets from the parent LongMemEval-S dataset. - -The fast-eval JSON files (fast50 / temporal15 / multisession15 / fast10) are -derived subsets of ``data/longmemeval_s_cleaned.json`` — each question object -is byte-identical to its parent entry. Committing ~45MB of derived JSON bloats -the repository and triggers secret-scanning false positives (synthetic chat -text occasionally matches token-shaped patterns). Git therefore tracks only -this generator plus the per-file question_id lists; the JSON files themselves -are regenerated locally on demand: - - python3 make_fast_eval.py # regenerate all four files - python3 make_fast_eval.py --verify # check existing files match the lists - -The id lists live in fast_eval_ids.json. Selection was stratified per -question_type when first built (2026-08-27); the lists are the source of -truth, so regeneration is fully deterministic — no randomness. - -Requires the parent dataset. If missing, run download_data.sh first and fetch -longmemeval_s_cleaned.json manually (see the note in that script). -""" -import argparse -import json -import sys -from pathlib import Path - -HERE = Path(__file__).parent -DATA = HERE / "data" -PARENT = DATA / "longmemeval_s_cleaned.json" -IDS = HERE / "fast_eval_ids.json" - -# id-list file name -> output dataset file name -NAME_MAP = { - "fast50": "longmemeval_fast50.json", - "fast10": "longmemeval_fast10.json", - "temporal15": "longmemeval_temporal15.json", - "multisession15": "longmemeval_multisession15.json", -} - - -def load_parent(): - if not PARENT.exists(): - sys.exit(f"Parent dataset not found: {PARENT}\nRun download_data.sh and fetch longmemeval_s_cleaned.json first (see script notes).") - with open(PARENT) as f: - return json.load(f) - - -def main(): - ap = argparse.ArgumentParser() - ap.add_argument("--verify", action="store_true", help="verify existing files instead of regenerating") - args = ap.parse_args() - - with open(IDS) as f: - id_lists = json.load(f) - parent = load_parent() - by_id = {q["question_id"]: q for q in parent} - - ok = True - for key, fname in NAME_MAP.items(): - ids = id_lists[key] - missing = [i for i in ids if i not in by_id] - if missing: - print(f"[FAIL] {fname}: {len(missing)} ids not in parent (first: {missing[0]})") - ok = False - continue - rebuilt = [by_id[i] for i in ids] - out_path = DATA / fname - if args.verify: - if not out_path.exists(): - print(f"[FAIL] {fname}: missing (run without --verify to regenerate)") - ok = False - continue - with open(out_path) as f: - cur = json.load(f) - match = json.dumps(cur) == json.dumps(rebuilt) - print(f"[{'OK' if match else 'FAIL'}] {fname}: {'identical' if match else 'DIFFERS from id-list rebuild'}") - ok &= match - else: - with open(out_path, "w") as f: - json.dump(rebuilt, f) - print(f"[OK] wrote {out_path} ({len(rebuilt)} questions)") - - sys.exit(0 if ok else 1) - - -if __name__ == "__main__": - main() diff --git a/benchmarks/longmemeval/mmr.py b/benchmarks/longmemeval/mmr.py deleted file mode 100644 index 69eac5ff..00000000 --- a/benchmarks/longmemeval/mmr.py +++ /dev/null @@ -1,105 +0,0 @@ -#!/usr/bin/env python3 -"""MMR (Maximal Marginal Relevance) diversity for session ranking (#1120). - -Zero-model post-processing: reorders an already-ranked candidate list to -penalise redundancy among top results. Relevance is approximated from the -input rank order (RRF-style 1/(60+rank)); similarity between candidates is -a TF-IDF cosine over their raw texts, computed over the candidate pool -itself — no external model, no runtime dependencies. - -Usage: - from mmr import mmr_rerank - out = mmr_rerank(ids, texts, lam=0.7) # MMR active - out = mmr_rerank(ids, texts, lam=None) # no-op (flag off) - -Unit tests: test_mmr.py -""" -from __future__ import annotations - -import math -import re -from collections import Counter - -_RRF_K = 60 # same constant as the RRF fusion used elsewhere in the harness -_WORD = re.compile(r"[a-z0-9']+") - - -def _tokens(text: str) -> list: - return _WORD.findall(text.lower()) - - -def _tfidf_vectors(texts: list) -> list: - """TF-IDF vectors over the candidate pool. Pure python, pool-sized idf.""" - tokenized = [_tokens(t) for t in texts] - n_docs = len(tokenized) or 1 - df = Counter() - for toks in tokenized: - df.update(set(toks)) - idf = {w: math.log((n_docs + 1) / (c + 1)) + 1.0 for w, c in df.items()} - vecs = [] - for toks in tokenized: - tf = Counter(toks) - vec = {w: (1.0 + math.log(c)) * idf[w] for w, c in tf.items()} - norm = math.sqrt(sum(v * v for v in vec.values())) or 1.0 - vecs.append({w: v / norm for w, v in vec.items()}) - return vecs - - -def _cosine(a: dict, b: dict) -> float: - if len(b) < len(a): - a, b = b, a - return sum(v * b.get(w, 0.0) for w, v in a.items()) - - -def mmr_rerank(session_ids: list, session_texts: list, lam: float | None, - relevance: list | None = None) -> list: - """Greedy MMR reorder of session_ids. - - lam=None → no-op (returns input order; flag off). - lam=1.0 → pure relevance, reproduces input order. - lam<1.0 → trades relevance for diversity. - - session_texts[i] is the full text of session_ids[i]. Optional - `relevance` overrides the default rank-based scores (len must match). - """ - if lam is None or len(session_ids) < 2: - return list(session_ids) - if len(session_ids) != len(session_texts): - raise ValueError("session_ids/session_texts length mismatch") - - rel = relevance if relevance is not None else [ - 1.0 / (_RRF_K + i + 1) for i in range(len(session_ids)) - ] - # Normalise relevance to [0,1] so it is scale-comparable with the cosine - # similarity term. Raw RRF scores span ~0.016 with a spread of ~0.002 — - # without normalisation the diversity term (~0.1-1.0) dominates by ~50x - # and MMR degenerates into pure diversity (measured: R@5 1.0 -> 0.33 - # on the mini smoke dataset). - rmin, rmax = min(rel), max(rel) - if rmax > rmin: - rel = [(r - rmin) / (rmax - rmin) for r in rel] - else: - rel = [1.0] * len(rel) - vecs = _tfidf_vectors(session_texts) - - selected: list = [] - remaining = list(range(len(session_ids))) - # Track max similarity of each remaining candidate to the selected set. - max_sim = {i: 0.0 for i in remaining} - - while remaining: - best_i = remaining[0] - best_score = lam * rel[best_i] - (1.0 - lam) * max_sim[best_i] - for i in remaining[1:]: - score = lam * rel[i] - (1.0 - lam) * max_sim[i] - if score > best_score: - best_i, best_score = i, score - selected.append(best_i) - remaining.remove(best_i) - bv = vecs[best_i] - for j in remaining: - s = _cosine(bv, vecs[j]) - if s > max_sim[j]: - max_sim[j] = s - - return [session_ids[i] for i in selected] diff --git a/benchmarks/longmemeval/modal_fanout.py b/benchmarks/longmemeval/modal_fanout.py deleted file mode 100644 index 3aacb591..00000000 --- a/benchmarks/longmemeval/modal_fanout.py +++ /dev/null @@ -1,279 +0,0 @@ -#!/usr/bin/env python3 -""" -Modal fan-out runner for the LongMemEval benchmark (uteke). - -Runs the EXISTING harness (run_eval.py) unchanged, in parallel, on Modal CPU -containers. Same uteke binary (v0.15.0 GitHub release), same embedding model -(EmbeddingGemma 300M Q4 ONNX baked into the image), same dataset file → -results are directly comparable with local throttled runs. - -Why CPU and not GPU: the embedding model is a 300M Q4 ONNX model designed for -CPU inference. The historical bottleneck was (a) 2-core throttling + -sequential execution, not (b) compute. Fan-out over N containers divides -wall-clock by ~N at CPU pricing (<$1 for a full 500Q run). - -Usage: - # Sanity check: 10 questions, 2 shards - modal run modal_fanout.py --limit 10 --num-shards 2 - - # Full 500Q hybrid run (the publishable number) - modal run modal_fanout.py --strategy hybrid --num-shards 10 - - # Full 500Q vector run - modal run modal_fanout.py --strategy vector --num-shards 10 - -Results land in results_modal_[_q]/retrieval_results.jsonl -(merged from all shards), ready for: python print_metrics.py - -Notes: -- Each shard runs in its own container with its own temp store → no - cross-contamination between shards (harness wipes per question anyway). -- cpu=2 matches the local throttled run (taskset -c 0-1) for fair per-question - latency comparison. -- The embedding model is baked at /root/.codecora/uteke/models/ so no - first-run download happens inside containers. -""" - -import json -import os -import pathlib -import subprocess -import sys -import time - -import modal - -REPO_DIR = pathlib.Path(__file__).parent -UTEKE_VERSION = "v0.15.0" -# When set, the image builds uteke from this git ref instead of downloading -# the UTEKE_VERSION release. Used for pre-release validation (e.g. develop tip -# before tagging). The exact SHA must be used, not a branch name, so the image -# build is reproducible. -UTEKE_GIT_REF = os.environ.get("UTEKE_GIT_REF", "") -DATA_FILE = "longmemeval_s_cleaned.json" -# Local source of the embedding model (must contain onnx/ + tokenizer.json). -MODEL_SOURCE = pathlib.Path("/opt/data/.codecora/uteke/models/embeddinggemma-q4") - -app = modal.App("uteke-longmemeval") - -# Volume for durable shard outputs: if the local map() call dies (network, -# spend limit mid-run, Ctrl-C), completed shards are preserved and a rerun -# resumes instead of starting from zero. -vol = modal.Volume.from_name("uteke-longmemeval", create_if_missing=True) - -image = ( - # Ubuntu 24.04 base: glibc 2.39 — the uteke release binary needs it - # (debian slim/bookworm ships glibc 2.36 → too old, incl. the "legacy" build). - modal.Image.from_registry("ubuntu:24.04", add_python="3.11") - .apt_install("curl", "ca-certificates", "util-linux") -) -if UTEKE_GIT_REF: - # Pre-release validation: build uteke from the exact SHA (reproducible), - # but reuse the official ORT 1.24.4 x86_64 (SSE4.2) shared lib from the - # latest release tarball — ort is load-dynamic, so the lib is decoupled - # from the binary and versioned identically to official releases. - # build-essential: cargo needs a native linker (cc) on the base image. - image = image.apt_install("build-essential", "pkg-config").run_commands( - "curl -sL https://github.com/codecoradev/uteke/releases/download/" - f"{UTEKE_VERSION}/uteke-x86_64-unknown-linux-gnu-legacy-{UTEKE_VERSION}.tar.gz" - " | tar xz -C /tmp && " - "cp /tmp/libonnxruntime* /usr/local/bin/ && " - "curl -sL https://github.com/codecoradev/uteke/archive/" + UTEKE_GIT_REF + ".tar.gz | tar xz -C /tmp && " - "cd /tmp/uteke-*/ && " - "curl -sSf https://sh.rustup.rs | sh -s -- -y --profile minimal --default-toolchain stable && " - "$HOME/.cargo/bin/cargo build --release -p uteke-cli && " - "cp target/release/uteke /usr/local/bin/uteke && " - "chmod +x /usr/local/bin/uteke && " - "uteke --version" - ) -else: - # uteke release bundle: binary + libonnxruntime colocated (exe-dir lookup). - image = image.run_commands( - f"curl -sL https://github.com/codecoradev/uteke/releases/download/{UTEKE_VERSION}/" - f"uteke-x86_64-unknown-linux-gnu-legacy-{UTEKE_VERSION}.tar.gz | tar xz -C /tmp && " - "cp /tmp/uteke /usr/local/bin/uteke && cp /tmp/libonnxruntime* /usr/local/bin/ && " - "chmod +x /usr/local/bin/uteke && uteke --version" - ) -image = ( - image - .pip_install("tqdm>=4.65.0", "numpy>=1.24.0") - # Bake the embedding model so containers never download at runtime. - # (add_local_* must come last — Modal mounts them at container start.) - .add_local_dir(str(MODEL_SOURCE), "/root/.codecora/uteke/models/embeddinggemma-q4") - .add_local_file(str(REPO_DIR / "run_eval.py"), "/root/harness/run_eval.py") - # run_eval.py imports temporal/mmr experiment modules at import time — - # they must be baked into the image or every shard dies on ModuleNotFoundError. - .add_local_file(str(REPO_DIR / "temporal.py"), "/root/harness/temporal.py") - .add_local_file(str(REPO_DIR / "mmr.py"), "/root/harness/mmr.py") - .add_local_file(str(REPO_DIR / "data" / DATA_FILE), "/root/harness/data.json") -) - - -@app.function(image=image, timeout=14400, cpu=2, volumes={"/root/vol": vol}) -def run_shard(spec: dict) -> dict: - """Evaluate one strided slice of the dataset with the unmodified harness. - - spec: {"strategy": str, "shard_idx": int, "num_shards": int, "limit": int} - Writes shard output durably to the Modal Volume as soon as it finishes, - so progress survives local map() interruptions and reruns can resume. - """ - import json as _json - - strategy = spec["strategy"] - shard_idx = spec["shard_idx"] - num_shards = spec["num_shards"] - limit = spec["limit"] - - vol_path = pathlib.Path("/root/vol") / strategy / f"shard_{shard_idx:02d}.jsonl" - data = _json.loads(pathlib.Path("/root/harness/data.json").read_text()) - if limit and limit > 0: - data = data[:limit] - shard = data[shard_idx::num_shards] # strided → even question-type mix - expected = len(shard) - if not shard: - return {"shard_idx": shard_idx, "n": 0, "elapsed": 0.0, "jsonl": "", "stderr_tail": ""} - - # Volume cache: skip kalau shard SUDAH LENGKAP; resume kalau PARTIAL (dari timeout) - resume_args = [] - if vol_path.exists(): - prior = [l for l in vol_path.read_text().splitlines() if l.strip()] - if len(prior) >= expected: - return { - "shard_idx": shard_idx, - "n": len(prior), - "evaluated": len(prior), - "elapsed": 0.0, - "jsonl": "\n".join(prior), - "stderr_tail": "(resumed from volume)", - } - if prior: - # partial: seed output dir dari Volume agar run_eval --resume melanjutkan - out_seed = pathlib.Path(f"/tmp/out_{shard_idx}") - out_seed.mkdir(parents=True, exist_ok=True) - (out_seed / "retrieval_results.jsonl").write_text("\n".join(prior) + "\n") - resume_args = ["--resume"] - - pathlib.Path("/tmp/shard.json").write_text(_json.dumps(shard)) - out_dir = f"/tmp/out_{shard_idx}" - - t0 = time.time() - try: - proc = subprocess.run( - [ - sys.executable, "/root/harness/run_eval.py", - "--data", "/tmp/shard.json", - "--output", out_dir, - "--strategy", strategy, - "--namespace", "lmeval", - *resume_args, - ], - capture_output=True, text=True, timeout=14100, # 14400 - buffer commit - ) - except subprocess.TimeoutExpired as e: - # partial-save: run_eval menulis streaming ke out_dir; simpan apa adanya - partial = pathlib.Path(out_dir) / "retrieval_results.jsonl" - jsonl = partial.read_text() if partial.exists() else "" - vol_path.parent.mkdir(parents=True, exist_ok=True) - vol_path.write_text(jsonl) - vol.commit() - return { - "shard_idx": shard_idx, - "n": len([l for l in jsonl.splitlines() if l.strip()]), - "evaluated": len([l for l in jsonl.splitlines() if l.strip()]), - "elapsed": round(time.time() - t0, 1), - "jsonl": jsonl, - "stderr_tail": f"TIMEOUT partial-save: {len(jsonl.splitlines())}/{len(shard)} Q evaluated", - } - elapsed = time.time() - t0 - if proc.returncode != 0: - raise RuntimeError(f"shard {shard_idx} failed:\n{proc.stderr[-2000:]}") - - jsonl_path = pathlib.Path(out_dir) / "retrieval_results.jsonl" - jsonl = jsonl_path.read_text() if jsonl_path.exists() else "" - - # Persist this shard durably before returning — see vol comment above. - vol_path.parent.mkdir(parents=True, exist_ok=True) - vol_path.write_text(jsonl) - vol.commit() - - return { - "shard_idx": shard_idx, - "n": len(shard), - "evaluated": jsonl.count("\n"), - "elapsed": round(elapsed, 1), - "jsonl": jsonl, - "stderr_tail": proc.stderr[-500:], - } - - -@app.function(image=image, timeout=600, cpu=1, volumes={"/root/vol": vol}) -def list_volume(strategy: str) -> list: - """List completed shards for a strategy on the volume (progress check).""" - base = pathlib.Path("/root/vol") / strategy - out = [] - if base.exists(): - for f in sorted(base.glob("shard_*.jsonl")): - n = sum(1 for l in f.read_text().splitlines() if l.strip()) - out.append({"shard": f.name, "questions": n}) - return out - - -@app.local_entrypoint() -def main( - strategy: str = "hybrid", - num_shards: int = 10, - limit: int = 0, - outdir: str = "", -): - if not MODEL_SOURCE.exists(): - sys.exit(f"Model source not found: {MODEL_SOURCE}") - if not (REPO_DIR / "data" / DATA_FILE).exists(): - sys.exit(f"Dataset not found: {REPO_DIR / 'data' / DATA_FILE}") - - outdir = outdir or f"results_modal_{strategy}" + (f"_{limit}q" if limit else "") - out_path = pathlib.Path(outdir) / "retrieval_results.jsonl" - out_path.parent.mkdir(parents=True, exist_ok=True) - - # Progress check from the volume (if any shards completed before a prior - # interruption, they'll be resumed instead of recomputed). - try: - done = list_volume.remote(strategy) - if done: - print(f"Volume state: {len(done)} shard(s) already on volume:") - for d in done: - print(f" {d['shard']}: {d['questions']} questions") - except Exception as e: - print(f"(volume check skipped: {e})") - - print(f"Fan-out: strategy={strategy} shards={num_shards} limit={limit or 'ALL'}") - t0 = time.time() - - # Strided slicing means shard i covers data[i::num_shards]. - inputs = [ - {"strategy": strategy, "shard_idx": i, "num_shards": num_shards, "limit": limit} - for i in range(num_shards) - ] - merged, seen = [], set() - for res in run_shard.map(inputs): - n_eval = 0 - for line in res["jsonl"].splitlines(): - if not line.strip(): - continue - qid = json.loads(line).get("question_id") - if qid in seen: - continue # dedupe guard (retries) - seen.add(qid) - merged.append(line) - n_eval += 1 - print( - f" shard {res['shard_idx']:2d}: {n_eval}/{res['n']} questions " - f"in {res['elapsed']}s" - ) - - out_path.write_text("\n".join(merged) + ("\n" if merged else "")) - wall = time.time() - t0 - print(f"\nDone: {len(merged)} questions in {wall:.0f}s wall clock") - print(f"Results: {out_path}") - print(f"\nRun: python print_metrics.py {out_path}") - # Print metrics right away (numpy available locally). - subprocess.run([sys.executable, str(REPO_DIR / "print_metrics.py"), str(out_path)]) diff --git a/benchmarks/longmemeval/modal_fast50.py b/benchmarks/longmemeval/modal_fast50.py deleted file mode 100644 index 47f4d2b6..00000000 --- a/benchmarks/longmemeval/modal_fast50.py +++ /dev/null @@ -1,306 +0,0 @@ -#!/usr/bin/env python3 -""" -Modal fan-out runner for the LongMemEval benchmark (uteke). - -Runs the EXISTING harness (run_eval.py) unchanged, in parallel, on Modal CPU -containers. Same uteke binary (v0.15.0 GitHub release), same embedding model -(EmbeddingGemma 300M Q4 ONNX baked into the image), same dataset file → -results are directly comparable with local throttled runs. - -Why CPU and not GPU: the embedding model is a 300M Q4 ONNX model designed for -CPU inference. The historical bottleneck was (a) 2-core throttling + -sequential execution, not (b) compute. Fan-out over N containers divides -wall-clock by ~N at CPU pricing (<$1 for a full 500Q run). - -Usage: - # Sanity check: 10 questions, 2 shards - modal run modal_fanout.py --limit 10 --num-shards 2 - - # Full 500Q hybrid run (the publishable number) - modal run modal_fanout.py --strategy hybrid --num-shards 10 - - # Full 500Q vector run - modal run modal_fanout.py --strategy vector --num-shards 10 - -Results land in results_modal_[_q]/retrieval_results.jsonl -(merged from all shards), ready for: python print_metrics.py - -Notes: -- Each shard runs in its own container with its own temp store → no - cross-contamination between shards (harness wipes per question anyway). -- cpu=2 matches the local throttled run (taskset -c 0-1) for fair per-question - latency comparison. -- The embedding model is baked at /root/.codecora/uteke/models/ so no - first-run download happens inside containers. -""" - -import json -import hashlib -import pathlib -import subprocess -import os -import sys -import time - -import modal - -REPO_DIR = pathlib.Path(__file__).parent -UTEKE_VERSION = "v0.15.0" -DATA_FILE = os.environ.get("LMEVAL_DATA", "longmemeval_fast50.json") -RERANK = False # removed (#1118 cancelled — single-model direction); kept as False for compat -RERANK_DEPTH = 20 -# Local source of the embedding model (must contain onnx/ + tokenizer.json). -MODEL_SOURCE = pathlib.Path("/opt/data/.codecora/uteke/models/embeddinggemma-q4") - -app = modal.App("uteke-lmeval-fast") - -# Volume for durable shard outputs: if the local map() call dies (network, -# spend limit mid-run, Ctrl-C), completed shards are preserved and a rerun -# resumes instead of starting from zero. -vol = modal.Volume.from_name("uteke-lmeval-fast", create_if_missing=True) - -image = ( - # Ubuntu 24.04 base: glibc 2.39 — the uteke release binary needs it - # (debian slim/bookworm ships glibc 2.36 → too old, incl. the "legacy" build). - modal.Image.from_registry("ubuntu:24.04", add_python="3.11") - .apt_install("curl", "ca-certificates", "util-linux") - # uteke release bundle: binary + libonnxruntime colocated (exe-dir lookup). - .run_commands( - f"curl -sL https://github.com/codecoradev/uteke/releases/download/{UTEKE_VERSION}/" - f"uteke-x86_64-unknown-linux-gnu-legacy-{UTEKE_VERSION}.tar.gz | tar xz -C /tmp && " - "cp /tmp/uteke /usr/local/bin/uteke && cp /tmp/libonnxruntime* /usr/local/bin/ && " - "chmod +x /usr/local/bin/uteke && uteke --version" - ) - .pip_install("tqdm>=4.65.0", "numpy>=1.24.0", "onnxruntime>=1.17.0", "tokenizers>=0.19.0") - # Bake the embedding model so containers never download at runtime. - # (add_local_* must come last — Modal mounts them at container start.) - .add_local_dir(str(MODEL_SOURCE), "/root/.codecora/uteke/models/embeddinggemma-q4") - .add_local_file(str(REPO_DIR / "run_eval.py"), "/root/harness/run_eval.py") - .add_local_file(str(REPO_DIR / "data" / DATA_FILE), "/root/harness/data.json") - .add_local_file(str(REPO_DIR / "temporal.py"), "/root/harness/temporal.py") - .add_local_file(str(REPO_DIR / "mmr.py"), "/root/harness/mmr.py") - .add_local_file(str(REPO_DIR / "compare_fast50.py"), "/root/harness/compare_fast50.py") -) - - -@app.function(image=image, timeout=14400, cpu=2, volumes={"/root/vol": vol}) -def run_shard(spec: dict) -> dict: - """Evaluate one strided slice of the dataset with the unmodified harness. - - spec: {"strategy": str, "shard_idx": int, "num_shards": int, "limit": int} - Writes shard output durably to the Modal Volume as soon as it finishes, - so progress survives local map() interruptions and reruns can resume. - """ - import json as _json - - strategy = spec["strategy"] - shard_idx = spec["shard_idx"] - num_shards = spec["num_shards"] - limit = spec["limit"] - temporal = bool(spec.get("temporal", False)) - mmr_lambda = spec.get("mmr_lambda") # None = OFF (default) - fusion = bool(spec.get("fusion", False)) - fusion_w = spec.get("fusion_weight") # primary weight, None = default - - # Variant-keyed volume path: baseline and temporal shards must never - # share cache entries, or resume would silently return stale results. - variant = f"{strategy}_temporal" if temporal else strategy - if mmr_lambda is not None: - variant += f"_mmr{mmr_lambda}" - if fusion: - variant += f"_fusion{fusion_w}" - # Dataset fingerprint: sha256 of the mounted dataset file (non-security - # fingerprinting, but avoids scanner flags). Cross-dataset - # stale hits are possible without it (fast50 shard satisfies a 15Q run's - # `len(prior) >= expected` check and is returned verbatim) — see #1129. - data_bytes = pathlib.Path("/root/harness/data.json").read_bytes() - data_tag = hashlib.sha256(data_bytes).hexdigest()[:8] - vol_path = pathlib.Path("/root/vol") / variant / data_tag / f"shard_{shard_idx:02d}.jsonl" - data = _json.loads(data_bytes) - if limit and limit > 0: - data = data[:limit] - shard = data[shard_idx::num_shards] # strided → even question-type mix - expected = len(shard) - if not shard: - return {"shard_idx": shard_idx, "n": 0, "elapsed": 0.0, "jsonl": "", "stderr_tail": ""} - - # Volume cache: skip kalau shard SUDAH LENGKAP; resume kalau PARTIAL (dari timeout) - resume_args = [] - if vol_path.exists(): - prior = [] - for l in vol_path.read_text().splitlines(): - if not l.strip(): - continue - try: - json.loads(l) - except json.JSONDecodeError: - # Truncated tail line from a killed container mid-write: drop - # it rather than abort the shard (#1130 review finding). - continue - prior.append(l) - prior_qids = {json.loads(l).get("question_id") for l in prior} - expected_qids = {e.get("question_id") for e in shard} - # Complete = same question set, not just same count (guard #1129 even - # if two datasets share a prefix and length by coincidence). - if len(prior) >= expected and prior_qids == expected_qids: - return { - "shard_idx": shard_idx, - "n": len(prior), - "evaluated": len(prior), - "elapsed": 0.0, - "jsonl": "\n".join(prior), - "stderr_tail": "(resumed from volume)", - } - if prior: - # partial: seed output dir dari Volume agar run_eval --resume melanjutkan - out_seed = pathlib.Path(f"/tmp/out_{shard_idx}") - out_seed.mkdir(parents=True, exist_ok=True) - (out_seed / "retrieval_results.jsonl").write_text("\n".join(prior) + "\n") - resume_args = ["--resume"] - - pathlib.Path("/tmp/shard.json").write_text(_json.dumps(shard)) - out_dir = f"/tmp/out_{shard_idx}" - - t0 = time.time() - try: - proc = subprocess.run( - [ - sys.executable, "/root/harness/run_eval.py", - "--data", "/tmp/shard.json", - "--output", out_dir, - "--strategy", strategy, - "--namespace", "lmeval", - *(["--temporal"] if temporal else []), - *(["--mmr-lambda", f"{mmr_lambda}"] if mmr_lambda is not None else []), - *(["--fusion", "--fusion-primary-weight", f"{fusion_w}"] if fusion and fusion_w is not None else (["--fusion"] if fusion else [])), - *resume_args, - ], - capture_output=True, text=True, timeout=14100, # 14400 - buffer commit - ) - except subprocess.TimeoutExpired as e: - # partial-save: run_eval menulis streaming ke out_dir; simpan apa adanya - partial = pathlib.Path(out_dir) / "retrieval_results.jsonl" - jsonl = partial.read_text() if partial.exists() else "" - vol_path.parent.mkdir(parents=True, exist_ok=True) - vol_path.write_text(jsonl) - vol.commit() - return { - "shard_idx": shard_idx, - "n": len([l for l in jsonl.splitlines() if l.strip()]), - "evaluated": len([l for l in jsonl.splitlines() if l.strip()]), - "elapsed": round(time.time() - t0, 1), - "jsonl": jsonl, - "stderr_tail": f"TIMEOUT partial-save: {len(jsonl.splitlines())}/{len(shard)} Q evaluated", - } - elapsed = time.time() - t0 - if proc.returncode != 0: - raise RuntimeError(f"shard {shard_idx} failed:\n{proc.stderr[-2000:]}") - - jsonl_path = pathlib.Path(out_dir) / "retrieval_results.jsonl" - jsonl = jsonl_path.read_text() if jsonl_path.exists() else "" - - # Persist this shard durably before returning — see vol comment above. - vol_path.parent.mkdir(parents=True, exist_ok=True) - vol_path.write_text(jsonl) - vol.commit() - - return { - "shard_idx": shard_idx, - "n": len(shard), - "evaluated": jsonl.count("\n"), - "elapsed": round(elapsed, 1), - "jsonl": jsonl, - "stderr_tail": proc.stderr[-500:], - } - - -@app.function(image=image, timeout=600, cpu=1, volumes={"/root/vol": vol}) -def list_volume(strategy: str) -> list: - """List completed shards for a strategy on the volume (progress check). - - Glob is dataset-tag-agnostic (shard_*.jsonl under any tag dir) so the - progress view still works after #1129 introduced per-dataset subdirs. - """ - base = pathlib.Path("/root/vol") / strategy - out = [] - if base.exists(): - for f in sorted(base.glob("*/shard_*.jsonl")): - n = sum(1 for l in f.read_text().splitlines() if l.strip()) - out.append({"shard": f"{f.parent.name}/{f.name}", "questions": n}) - return out - - -@app.local_entrypoint() -def main( - strategy: str = "hybrid", - num_shards: int = 2, - limit: int = 0, - outdir: str = "", - temporal: bool = False, - mmr_lambda: float = 0.0, - fusion: bool = False, - fusion_weight: float = 0.0, -) -> None: - if not MODEL_SOURCE.exists(): - sys.exit(f"Model source not found: {MODEL_SOURCE}") - if not (REPO_DIR / "data" / DATA_FILE).exists(): - sys.exit(f"Dataset not found: {REPO_DIR / 'data' / DATA_FILE}") - - lam: float | None = mmr_lambda if mmr_lambda > 0 else None # 0/omitted = OFF - fw: float | None = fusion_weight if fusion_weight > 0 else None # None = harness default - variant = f"{strategy}_temporal" if temporal else strategy - if lam is not None: - variant += f"_mmr{lam}" - if fusion: - variant += f"_fusion{fw}" - outdir = outdir or "results_modal_" + variant + (f"_{limit}q" if limit else "") - out_path = pathlib.Path(outdir) / "retrieval_results.jsonl" - out_path.parent.mkdir(parents=True, exist_ok=True) - - # Progress check from the volume (if any shards completed before a prior - # interruption, they'll be resumed instead of recomputed). - try: - variant = f"{strategy}_temporal" if temporal else strategy - if lam is not None: - variant += f"_mmr{lam}" - done = list_volume.remote(variant) - if done: - print(f"Volume state: {len(done)} shard(s) already on volume:") - for d in done: - print(f" {d['shard']}: {d['questions']} questions") - except Exception as e: - print(f"(volume check skipped: {e})") - - print(f"Fan-out: strategy={strategy} shards={num_shards} limit={limit or 'ALL'}") - t0 = time.time() - - # Strided slicing means shard i covers data[i::num_shards]. - inputs = [ - {"strategy": strategy, "shard_idx": i, "num_shards": num_shards, "limit": limit, - "temporal": temporal, "mmr_lambda": lam, "fusion": fusion, "fusion_weight": fw} - for i in range(num_shards) - ] - merged, seen = [], set() - for res in run_shard.map(inputs): - n_eval = 0 - for line in res["jsonl"].splitlines(): - if not line.strip(): - continue - qid = json.loads(line).get("question_id") - if qid in seen: - continue # dedupe guard (retries) - seen.add(qid) - merged.append(line) - n_eval += 1 - print( - f" shard {res['shard_idx']:2d}: {n_eval}/{res['n']} questions " - f"in {res['elapsed']}s" - ) - - out_path.write_text("\n".join(merged) + ("\n" if merged else "")) - wall = time.time() - t0 - print(f"\nDone: {len(merged)} questions in {wall:.0f}s wall clock") - print(f"Results: {out_path}") - print(f"\nRun: python print_metrics.py {out_path}") - # Print metrics right away (numpy available locally). - subprocess.run([sys.executable, str(REPO_DIR / "print_metrics.py"), str(out_path)]) diff --git a/benchmarks/longmemeval/preview_scores.py b/benchmarks/longmemeval/preview_scores.py deleted file mode 100644 index b17ab606..00000000 --- a/benchmarks/longmemeval/preview_scores.py +++ /dev/null @@ -1,85 +0,0 @@ -#!/usr/bin/env python3 -"""Partial preview: 2-way fusion scores from already-collected vec+hyb rankings. - -Also joins whatever fts5 shards are already on disk (results_modal_fts5/) for -an early 3-way peek, question-subset only. -""" -import glob -import json -import os -from collections import defaultdict - -BASE = os.path.dirname(os.path.abspath(__file__)) - - -def load(p): - return [json.loads(l) for l in open(p) if l.strip()] - - -def rrf(rankings, weights, k=60, topn=45): - s = defaultdict(float) - for ids, w in zip(rankings, weights): - for rank, sid in enumerate(ids[:topn]): - s[sid] += w / (k + rank + 1) - return [x for x, _ in sorted(s.items(), key=lambda t: -t[1])] - - -def r_at(ids, t, k): - return len(set(ids[:k]) & set(t)) / len(t) - - -def main(): - vec = {r['question_id']: r['retrieval_results']['retrieved_session_ids'] - for r in load(os.path.join(BASE, 'results_modal_vector', 'retrieval_results.jsonl'))} - hyb = {r['question_id']: r['retrieval_results']['retrieved_session_ids'] - for r in load(os.path.join(BASE, 'results_modal_hybrid_fresh_post1130', 'retrieval_results.jsonl'))} - - fts = {} - fts_files = sorted(glob.glob(os.path.join(BASE, 'results_modal_fts5_partial/shard_*.jsonl'))) - if not fts_files: - merged = os.path.join(BASE, 'results_modal_fts5', 'retrieval_results.jsonl') - if os.path.exists(merged): - fts_files = [merged] - for fp in fts_files: - for r in load(fp): - fts[r['question_id']] = r['retrieval_results']['retrieved_session_ids'] - - with open(os.path.join(BASE, 'data', 'longmemeval_fast50.json')) as f: - ds = json.load(f) - gt = {q['question_id']: q['answer_session_ids'] for q in ds} - - qids2 = [q for q in vec if q in hyb and gt.get(q)] - print(f'== 2-way preview (full fast50, n={len(qids2)}) ==') - rows = [] - for name, src in [('vector-only', 'v'), ('hybrid-only', 'h')]: - tot = sum(r_at(vec[q] if src == 'v' else hyb[q], gt[q], 5) for q in qids2) - rows.append((name, tot / len(qids2))) - for name, w in [('fusion 1.7/1.0 (shipped)', (1.7, 1.0)), ('fusion 1.0/1.0', (1.0, 1.0))]: - tot = sum(r_at(rrf([vec[q], hyb[q]], w), gt[q], 5) for q in qids2) - rows.append((name, tot / len(qids2))) - for name, v in rows: - print(f' {name:26s} R@5 = {v:.4f}') - - qids3 = [q for q in qids2 if q in fts] - if qids3: - print(f'\n== 3-way EARLY peek (subset with fts5, n={len(qids3)}/50) ==') - base2 = sum(r_at(rrf([vec[q], hyb[q]], (1.7, 1.0)), gt[q], 5) for q in qids3) - print(f' fusion 1.7/1.0 on subset R@5 = {base2 / len(qids3):.4f}') - fts_only = sum(r_at(fts[q], gt[q], 5) for q in qids3) - print(f' fts5-only on subset R@5 = {fts_only / len(qids3):.4f}') - best = (None, -1.0) - for wv in (1.4, 1.7, 2.0): - for wf in (0.2, 0.4, 0.6, 0.8, 1.0): - w = (wv, 1.0, wf) - tot = sum(r_at(rrf([vec[q], hyb[q], fts[q]], w), gt[q], 5) for q in qids3) - if tot > best[1]: - best = (w, tot) - print(f' best 3-way on subset: w={best[0]} R@5 = {best[1] / len(qids3):.4f} ' - f'(delta vs 2way {best[1] / len(qids3) - base2 / len(qids3):+.4f})') - print(' (ANGKA SUBSET — bisa berubah saat 50Q lengkap, jangan dipakai putus weight)') - else: - print('\n(fts5 rankings belum ada di disk — 3-way peek menyusul)') - - -if __name__ == '__main__': - main() diff --git a/benchmarks/longmemeval/pull_fast.py b/benchmarks/longmemeval/pull_fast.py deleted file mode 100644 index 5db23f4c..00000000 --- a/benchmarks/longmemeval/pull_fast.py +++ /dev/null @@ -1,47 +0,0 @@ -#!/usr/bin/env python3 -"""Pull fast-eval shards from Modal Volume uteke-lmeval-fast → local merged jsonl. - -Usage: modal run pull_fast.py --out results_fast50_baseline -Writes: /retrieval_results.jsonl (merged, dedup by question_id) -Prints: single summary line (shards, questions). -""" -import json -import pathlib -import modal - -vol = modal.Volume.from_name("uteke-lmeval-fast", create_if_missing=True) - -app = modal.App("uteke-lmeval-pull-fast") - - -@app.function(volumes={"/root/vol": vol}, timeout=300) -def pull(strategy: str) -> dict: - base = pathlib.Path("/root/vol") / strategy - out = {} - if base.exists(): - for f in sorted(base.glob("shard_*.jsonl")): - out[f.name] = f.read_text() - vol.commit() - return out - - -@app.local_entrypoint() -def main(out: str = "results_fast50_baseline", strategy: str = "hybrid"): - shards = pull.remote(strategy) - seen, rows = set(), [] - for fname in sorted(shards): - for line in shards[fname].splitlines(): - line = line.strip() - if not line: - continue - try: - qid = json.loads(line).get("question_id") - except json.JSONDecodeError: - continue - if qid and qid not in seen: - seen.add(qid) - rows.append(line) - p = pathlib.Path(out) / "retrieval_results.jsonl" - p.parent.mkdir(parents=True, exist_ok=True) - p.write_text("\n".join(rows) + ("\n" if rows else "")) - print(f"PULL_FAST: {len(shards)} shards, {len(rows)} Q → {p}") diff --git a/benchmarks/longmemeval/pull_shards.py b/benchmarks/longmemeval/pull_shards.py deleted file mode 100644 index 65d46262..00000000 --- a/benchmarks/longmemeval/pull_shards.py +++ /dev/null @@ -1,29 +0,0 @@ -#!/usr/bin/env python3 -"""Pull completed LongMemEval shards from the Modal Volume. - -Prints a JSON dict {shard_file: content} on the last line (parsed by -watch_and_pull.py). -""" -import json - -import modal - -vol = modal.Volume.from_name("uteke-longmemeval", create_if_missing=True) - -app = modal.App("lmeval-pull") - - -@app.function(volumes={"/root/vol": vol}, timeout=120) -def pull(strategy: str) -> dict: - import pathlib - base = pathlib.Path("/root/vol") / strategy - out = {} - if base.exists(): - for f in sorted(base.glob("shard_*.jsonl")): - out[f.name] = f.read_text() - return out - - -@app.local_entrypoint() -def main(strategy: str = "hybrid"): - print(json.dumps(pull.remote(strategy))) diff --git a/benchmarks/longmemeval/sim3way.py b/benchmarks/longmemeval/sim3way.py deleted file mode 100644 index bdafd29d..00000000 --- a/benchmarks/longmemeval/sim3way.py +++ /dev/null @@ -1,156 +0,0 @@ -#!/usr/bin/env python3 -"""3-way RRF simulation: vector + hybrid + fts5-only rankings. - -Offline replay over saved rankings (fast50, Modal x86) — no uteke calls, no -embedding burns. Question: does adding fts5-only as a third RRF arm lift -R@5 over the shipped 2-way fusion (vec*1.7 + hyb*1, k=60)? - -Inputs (relative to this dir): - results_modal_vector/retrieval_results.jsonl — vector-only rankings - results_modal_hybrid_fresh_post1130/...jsonl — hybrid rankings - results_modal_fts5/retrieval_results.jsonl — fts5-only rankings - data/longmemeval_fast50.json — ground truth - -Usage: - python3 sim3way.py rankings # fail-overlap analysis (2-way) - python3 sim3way.py fusion # 3-way RRF grid search (weights, k) -""" -import json -import os -import sys -from collections import defaultdict - -BASE = os.path.dirname(os.path.abspath(__file__)) -K_DEFAULT = 60 - - -def load_jsonl(path): - with open(path) as f: - return [json.loads(line) for line in f if line.strip()] - - -def r_at_k(ids, truth, k): - if not truth: - return None - return len(set(ids[:k]) & truth) / len(truth) - - -def rrf_fuse(rankings, weights, k=K_DEFAULT, topn=45): - """Weighted RRF over multiple id lists. Mirrors rrf_fuse_weighted semantics.""" - scores = defaultdict(float) - for ids, w in zip(rankings, weights): - for rank, sid in enumerate(ids[:topn]): - scores[sid] += w / (k + rank + 1) - return [sid for sid, _ in sorted(scores.items(), key=lambda x: -x[1])] - - -def evaluate(fused, truth): - m = {} - m['recall_all@5'] = r_at_k(fused, truth, 5) - m['recall_all@10'] = r_at_k(fused, truth, 10) - return m - - -def load_all(): - vec = load_jsonl(os.path.join(BASE, 'results_modal_vector', 'retrieval_results.jsonl')) - hyb = load_jsonl(os.path.join(BASE, 'results_modal_hybrid_fresh_post1130', 'retrieval_results.jsonl')) - fts = load_jsonl(os.path.join(BASE, 'results_modal_fts5', 'retrieval_results.jsonl')) - ds = json.load(open(os.path.join(BASE, 'data', 'longmemeval_fast50.json'))) - if isinstance(ds, dict): - ds = [ds] - gt = {q['question_id']: set(q['answer_session_ids']) for q in ds} - - by_qid = defaultdict(dict) - for name, rows in (('vec', vec), ('hyb', hyb), ('fts', fts)): - for r in rows: - by_qid[r['question_id']][name] = r['retrieval_results']['retrieved_session_ids'] - - out = [] - for qid, d in by_qid.items(): - t = gt.get(qid) - if not t or len(d) < 3: - continue - out.append((qid, t, d['vec'], d['hyb'], d['fts'])) - return out - - -def mode_rankings(): - """Fail-overlap between the three base rankings.""" - data = load_all() - print(f'questions with all 3 rankings + truth: {len(data)}') - # simpler: recompute directly - res = {} - for name, idx in (('vec', 2), ('hyb', 3), ('fts', 4)): - fails = [] - total = 0.0 - for qid, t, *rankings in data: - r5 = r_at_k(rankings[idx - 2], t, 5) - assert r5 is not None - total += r5 - if r5 < 1.0: - fails.append((qid[:12], r5)) - res[name] = (total / len(data), fails) - print(f'{name}: mean R@5 {total / len(data):.4f}, fails {len(fails)}: {fails}') - - v = {q for q, _ in res["vec"][1]} - h = {q for q, _ in res["hyb"][1]} - f = {q for q, _ in res["fts"][1]} - print() - print(f'vec∩hyb fails: {len(v & h)} {sorted(v & h)}') - print(f'vec∩fts fails: {len(v & f)} {sorted(v & f)}') - print(f'hyb∩fts fails: {len(h & f)} {sorted(h & f)}') - print(f'all three fail: {len(v & h & f)} {sorted(v & h & f)}') - print(f'union of fails: {len(v | h | f)}') - - -def mode_fusion(): - data = load_all() - n = len(data) - print(f'questions: {n}') - - def run(weights, k=K_DEFAULT): - r5 = r10 = 0.0 - fails = [] - for qid, t, *rk in data: - fused = rrf_fuse(rk, weights, k) - m = evaluate(fused, t) - r5 += m['recall_all@5'] - r10 += m['recall_all@10'] - if m['recall_all@5'] < 1.0: - fails.append((qid[:12], m['recall_all@5'])) - return r5 / n, r10 / n, fails - - # Baseline: shipped 2-way (vec*1.7, hyb*1) - b = run((1.7, 1.0, 0.0)) - print(f'2-way shipped (1.7/1.0/0) R@5 {b[0]:.4f} R@10 {b[1]:.4f} fails {len(b[2])}: {b[2]}') - - # 3-way grid: fts weight 0.2–1.0, vec weight around 1.7 - print() - print('3-way grid (vec / hyb / fts):') - best = (None, -1) - for wv in (1.4, 1.7, 2.0): - for wf in (0.2, 0.4, 0.6, 0.8, 1.0): - w = (wv, 1.0, wf) - r5, r10, fails = run(w) - mark = '' - if r5 > best[1]: - best = (w, r5) - mark = ' <-- best so far' - print(f' {wv:.1f}/{1.0:.1f}/{wf:.1f} R@5 {r5:.4f} R@10 {r10:.4f} fails {len(fails)}{mark}') - print() - print(f'BEST: weights={best[0]} R@5={best[1]:.4f} (shipped 2-way: {b[0]:.4f}, delta {best[1]-b[0]:+.4f})') - - -def main(): - mode = sys.argv[1] if len(sys.argv) > 1 else 'rankings' - if mode == 'rankings': - mode_rankings() - elif mode == 'fusion': - mode_fusion() - else: - print(f'unknown mode: {mode}', file=sys.stderr) - sys.exit(1) - - -if __name__ == '__main__': - main() diff --git a/benchmarks/longmemeval/sim3way_final.py b/benchmarks/longmemeval/sim3way_final.py deleted file mode 100644 index fdd0b304..00000000 --- a/benchmarks/longmemeval/sim3way_final.py +++ /dev/null @@ -1,97 +0,0 @@ -#!/usr/bin/env python3 -"""Final 3-way RRF simulation on full 500Q (fts5) x fast50 (vec, hyb). - -Question: does adding fts5-only as a third RRF arm beat the shipped 2-way -fusion (vec*1.7 + hyb*1.0, k=60)? - -Data availability reality: - - fts5 rankings: 500 Q (just collected, Modal x86) - - vector + hybrid rankings: 50 Q (fast50, Modal x86) -=> 3-way grid runs on the 50-Q intersection; the fts5 arm at 500 Q is used - for the fts5-only baseline and failure analysis. -""" -import json -import os -from collections import defaultdict - -BASE = os.path.dirname(os.path.abspath(__file__)) - - -def load_jsonl(p): - with open(p) as f: - return [json.loads(l) for l in f if l.strip()] - - -def rrf(rankings, weights, k=60, topn=45): - s = defaultdict(float) - for ids, w in zip(rankings, weights): - for rank, sid in enumerate(ids[:topn]): - s[sid] += w / (k + rank + 1) - return [x for x, _ in sorted(s.items(), key=lambda t: -t[1])] - - -def r_at(ids, t, k): - return len(set(ids[:k]) & set(t)) / len(t) - - -def main(): - fts = {} - for s in range(10): - for r in load_jsonl(os.path.join(BASE, 'results_modal_fts5_partial', f'shard_{s:02d}.jsonl')): - fts[r['question_id']] = r['retrieval_results']['retrieved_session_ids'] - vec = {r['question_id']: r['retrieval_results']['retrieved_session_ids'] - for r in load_jsonl(os.path.join(BASE, 'results_modal_vector', 'retrieval_results.jsonl'))} - hyb = {r['question_id']: r['retrieval_results']['retrieved_session_ids'] - for r in load_jsonl(os.path.join(BASE, 'results_modal_hybrid_fresh_post1130', 'retrieval_results.jsonl'))} - - with open(os.path.join(BASE, 'data', 'longmemeval_s_cleaned.json')) as f: - ds = json.load(f) - gt = {q['question_id']: q['answer_session_ids'] for q in ds} - qtype = {q['question_id']: q.get('question_type', '?') for q in ds} - - # fts5-only baseline on full 500 - qids500 = [q for q in fts if gt.get(q)] - tot5 = sum(r_at(fts[q], gt[q], 5) for q in qids500) - tot10 = sum(r_at(fts[q], gt[q], 10) for q in qids500) - print(f'fts5-only 500Q: n={len(qids500)} R@5={tot5/len(qids500):.4f} R@10={tot10/len(qids500):.4f}') - - # 3-way grid on 50-Q intersection - qids = [q for q in vec if q in hyb and q in fts and gt.get(q)] - n = len(qids) - print(f'\n3-way grid on intersection n={n}') - base2 = {q: r_at(rrf([vec[q], hyb[q]], (1.7, 1.0)), gt[q], 5) for q in qids} - b2 = sum(base2.values()) / n - print(f' 2-way shipped (1.7,1.0) R@5 = {b2:.4f}') - - best = (None, -1.0) - for wv in (1.3, 1.4, 1.7, 2.0): - for wf in (0.1, 0.2, 0.4, 0.6, 1.0): - w = (wv, 1.0, wf) - tot = sum(r_at(rrf([vec[q], hyb[q], fts[q]], w), gt[q], 5) for q in qids) - if tot > best[1]: - best = (w, tot) - print(f' best 3-way w={best[0]} R@5 = {best[1]/n:.4f} (delta {best[1]/n - b2:+.4f})') - - # per-question delta at shipped 2-way vs best 3-way - wv, _, wf = best[0] - flips = [] - for q in qids: - d = r_at(rrf([vec[q], hyb[q], fts[q]], best[0]), gt[q], 5) - base2[q] - if abs(d) > 1e-9: - flips.append((q[:8], qtype.get(q, '?'), round(d, 2))) - print(f' flipped questions: {len(flips)}') - for f in flips: - print(' ', f) - - # failure overlap analysis on 500Q fts5 - fts_fail = {q for q in qids500 if r_at(fts[q], gt[q], 5) < 1.0} - print(f'\nfts5 fails R@5<1 @500Q: {len(fts_fail)}/{len(qids500)}') - by_type = defaultdict(int) - for q in fts_fail: - by_type[qtype.get(q, '?')] += 1 - for t, c in sorted(by_type.items(), key=lambda t: -t[1]): - print(f' {t}: {c}') - - -if __name__ == '__main__': - main() diff --git a/benchmarks/longmemeval/temporal.py b/benchmarks/longmemeval/temporal.py deleted file mode 100644 index cc84a06a..00000000 --- a/benchmarks/longmemeval/temporal.py +++ /dev/null @@ -1,224 +0,0 @@ -#!/usr/bin/env python3 -"""Temporal expression parsing + date-window boost for LongMemEval (#1119). - -Harness-side only: no uteke/Rust changes. When a question contains a temporal -expression ("last month", "before January", "in February", "between X and Y"), -we compute a date window relative to the question date and softly boost -retrieved sessions whose session date falls inside that window, before -Recall@k is computed. Boost is a small additive constant on an RRF-style rank -score, so it can lift an in-window session from rank ~10 into the top-5 but -never completely overrides semantic ranking. - -Default OFF (--temporal flag in run_eval.py); when OFF or when no temporal -expression is parsed, output ordering is identical to baseline. -""" -import re -from datetime import date, timedelta - -MONTHS = { - "january": 1, "february": 2, "march": 3, "april": 4, "may": 5, "june": 6, - "july": 7, "august": 8, "september": 9, "october": 10, "november": 11, "december": 12, -} -MONTH_RE = "|".join(MONTHS.keys()) -UNITS = {"day": 1, "week": 7, "month": 30, "year": 365} - -# Question date format in LongMemEval: "2023/03/22 (Wed) 21:49" -QDATE_RE = re.compile(r"(\d{4})/(\d{2})/(\d{2})") - - -def _days_in_month(year, month): - if month == 12: - return 31 - return (date(year, month + 1, 1) - timedelta(days=1)).day - - -def _month_start_abs(month, year): - return date(year, month, 1) - - -def _month_end_abs(month, year): - return date(year, month, _days_in_month(year, month)) - - -def _resolve_year(month, year, q): - """Bare month names resolve within the 12 months preceding the question. - - LongMemEval haystacks span roughly the year before the question date, so - 'January' asked in 2023/03 means 2023-01, while 'May' means 2022-05. - """ - if year is not None: - return year - return q.year if month <= q.month else q.year - 1 - - -def parse_question_date(qdate_str): - """'2023/03/22 (Wed) 21:49' -> date(2023,3,22). None if unparseable.""" - m = QDATE_RE.search(qdate_str or "") - if not m: - return None - return date(int(m.group(1)), int(m.group(2)), int(m.group(3))) - - -# Each pattern: (compiled regex, name, resolver(match, q) -> (lo, hi)). -# lo/hi are datetime.date or None for open-ended bounds. Ordered most -# specific first. -_RX_BETWEEN = re.compile( - r"\bbetween\s+(" + MONTH_RE + r")(?:\s+(\d{4}))?\s+and\s+(" + MONTH_RE + r")(?:\s+(\d{4}))?\b", - re.I, -) -_RX_BEFORE = re.compile(r"\bbefore\s+(" + MONTH_RE + r")(?:\s+(\d{4}))?\b", re.I) -_RX_AFTER = re.compile(r"\b(?:after|since)\s+(" + MONTH_RE + r")(?:\s+(\d{4}))?\b", re.I) -_RX_IN_MONTH = re.compile(r"\bin\s+(" + MONTH_RE + r")(?:\s+(\d{4}))?\b", re.I) -_RX_LAST_N = re.compile(r"\b(?:last|past|previous)\s+(\d+)\s+(day|week|month|year)s?\b", re.I) -_RX_N_AGO = re.compile( - r"\b(\d+|one|two|three|four|five|six|seven|eight|nine|ten|eleven|twelve)" - r"\s+(day|week|month)s?\s+ago\b", - re.I, -) -_RX_LAST_WEEK = re.compile(r"\blast\s+week\b", re.I) -_RX_LAST_MONTH = re.compile(r"\blast\s+month\b", re.I) -_RX_THIS_MONTH = re.compile(r"\bthis\s+month\b", re.I) -_RX_THIS_YEAR = re.compile(r"\bthis\s+year\b", re.I) - - -def _res_between(m, q): - m1, y1 = MONTHS[m.group(1).lower()], m.group(2) - m2, y2 = MONTHS[m.group(3).lower()], m.group(4) - lo = _month_start_abs(m1, _resolve_year(m1, int(y1) if y1 else None, q)) - hi_year = _resolve_year(m2, int(y2) if y2 else None, q) - hi = _month_end_abs(m2, hi_year) - if hi < lo: - # Range crossing the question date (e.g. "between February and April" - # asked in March): bare-month resolution can put hi a year before lo. - # The range is stated in forward order, so bump hi into the next year. - hi = _month_end_abs(m2, hi_year + 1) - return (lo, hi) - - -def _res_before(m, q): - mm, yy = MONTHS[m.group(1).lower()], m.group(2) - hi = _month_end_abs(mm, _resolve_year(mm, int(yy) if yy else None, q)) - return (None, hi) - - -def _res_after(m, q): - mm, yy = MONTHS[m.group(1).lower()], m.group(2) - lo = _month_start_abs(mm, _resolve_year(mm, int(yy) if yy else None, q)) - return (lo, None) - - -def _res_in_month(m, q): - mm, yy = MONTHS[m.group(1).lower()], m.group(2) - y = _resolve_year(mm, int(yy) if yy else None, q) - return (_month_start_abs(mm, y), _month_end_abs(mm, y)) - - -def _res_last_n(m, q): - n, unit = int(m.group(1)), m.group(2).lower() - return (q - timedelta(days=n * UNITS[unit]), q) - - -def _res_last_month(m, q): - first_this = _month_start_abs(q.month, q.year) - last_prev = first_this - timedelta(days=1) # final day of previous month - return (_month_start_abs(last_prev.month, last_prev.year), last_prev) - - -_WORDS = {"one": 1, "two": 2, "three": 3, "four": 4, "five": 5, "six": 6, - "seven": 7, "eight": 8, "nine": 9, "ten": 10, "eleven": 11, "twelve": 12} - - -def _res_n_ago(m, q): - # "10 days ago" / "four weeks ago": the referenced event sits ~N units - # before the question date. Window = [q - 1.5*span, q - 0.5*span] so the - # nominal day lands mid-window with generous slack for fuzzy memories. - g = m.group(1).lower() - n = int(g) if g.isdigit() else _WORDS[g] - span = n * UNITS[m.group(2).lower()] - return (q - timedelta(days=int(1.5 * span)), q - timedelta(days=span // 2)) - - -def _res_last_week(m, q): - # Previous calendar week (Mon-Sun) relative to the question date. - monday_this = q - timedelta(days=q.weekday()) - return (monday_this - timedelta(days=7), monday_this - timedelta(days=1)) - - -def _res_this_month(m, q): - return (_month_start_abs(q.month, q.year), q) - - -def _res_this_year(m, q): - return (date(q.year, 1, 1), q) - - -_PATTERNS = [ - (_RX_BETWEEN, "between", _res_between), - (_RX_BEFORE, "before", _res_before), - (_RX_AFTER, "after/since", _res_after), - (_RX_IN_MONTH, "in-month", _res_in_month), - (_RX_N_AGO, "n-ago", _res_n_ago), - (_RX_LAST_N, "last-N", _res_last_n), - (_RX_LAST_WEEK, "last-week", _res_last_week), - (_RX_LAST_MONTH, "last-month", _res_last_month), - (_RX_THIS_MONTH, "this-month", _res_this_month), - (_RX_THIS_YEAR, "this-year", _res_this_year), -] - - -def parse_temporal(question, qdate): - """Parse the first matching temporal expression from a question. - - Returns (lo, hi, kind) — lo/hi are datetime.date or None (open bound) — - or None when the question carries no recognized temporal expression. - """ - if not question or qdate is None: - return None - for rx, name, fn in _PATTERNS: - m = rx.search(question) - if m: - lo, hi = fn(m, qdate) - return (lo, hi, name) - return None - - -def in_window(d, window): - """True if date d is within (lo, hi) inclusive; open ends are unbounded.""" - if not window: - return False - lo, hi, _ = window - if lo and d < lo: - return False - if hi and d > hi: - return False - return True - - -RRF_K = 60 # standard RRF constant - - -def boost_ranking(session_ids, session_dates, window, boost=0.0022, rrf_k=RRF_K): - """Re-rank deduped session ids with an additive temporal boost. - - session_ids: recall rank order (index 0 = rank 1) - session_dates: dict session_id -> datetime.date (metadata 'date'); missing ok - window: output of parse_temporal(), or None -> returned unchanged - - Rank r gets RRF score 1/(k+r); an in-window session gets +`boost`. - Calibration goal (recall@5): an in-window session ranked 6..15 must pass - an out-of-window session holding the rank-5 slot — 1/(k+15)+boost > - 1/(k+5) holds for boost=0.0022, k=60. Reordering *within* the top-5 is - possible and harmless: recall@k membership only changes at the k/k+1 - boundary. Ties keep recall order (stable). - """ - if not window or not session_ids: - return session_ids - scored = [] - for rank, sid in enumerate(session_ids, start=1): - s = 1.0 / (rrf_k + rank) - d = session_dates.get(sid) - if d is not None and in_window(d, window): - s += boost - scored.append((s, -rank, sid)) # -rank: stable tie-break toward recall order - scored.sort(key=lambda t: (t[0], t[1]), reverse=True) - return [sid for _, _, sid in scored] diff --git a/benchmarks/longmemeval/test_mmr.py b/benchmarks/longmemeval/test_mmr.py deleted file mode 100644 index f59b5804..00000000 --- a/benchmarks/longmemeval/test_mmr.py +++ /dev/null @@ -1,73 +0,0 @@ -#!/usr/bin/env python3 -"""Unit tests for mmr.py (#1120). Run: python3 test_mmr.py""" -import sys - -from mmr import mmr_rerank, _tfidf_vectors, _cosine - -PASS = 0 -FAIL = 0 - - -def check(name, cond, detail=""): - global PASS, FAIL - if cond: - PASS += 1 - print(f" ok {name}") - else: - FAIL += 1 - print(f" FAIL {name} {detail}") - - -# --- no-op semantics (flag off) --- -ids = ["s1", "s2", "s3"] -texts = ["alpha beta", "gamma delta", "epsilon zeta"] -check("lam=None returns input unchanged", mmr_rerank(ids, texts, None) == ids) -check("lam=1.0 pure relevance keeps order", mmr_rerank(ids, texts, 1.0) == ids) -check("single item passthrough", mmr_rerank(["s1"], ["text"], 0.5) == ["s1"]) -check("empty list passthrough", mmr_rerank([], [], 0.5) == []) - -# --- diversity behavior --- -# Pool: A and A2 near-duplicates, B distinct. Rank order: A, A2, B. -dup = ["cat sat mat", "cat sat mat cat", "dogs bark loudly"] -out = mmr_rerank(["A", "A2", "B"], dup, lam=0.5) -check("near-duplicate demoted below distinct", out[0] == "A" and out[1] == "B", - f"got {out}") - -# With lam=1.0 the duplicate stays at rank 2 (pure relevance). -out = mmr_rerank(["A", "A2", "B"], dup, lam=1.0) -check("lam=1.0 keeps duplicate at rank 2", out[:2] == ["A", "A2"], f"got {out}") - -# --- relevance override respected --- -# Force B's relevance to dominate: even at lam=0.9 B must come first. -rel = [0.01, 0.01, 0.99] -out = mmr_rerank(["A", "A2", "B"], dup, lam=0.9, relevance=rel) -check("relevance override dominates", out[0] == "B", f"got {out}") - -# --- similarity helpers --- -v = _tfidf_vectors(["same same same", "same same same"]) -check("identical texts cosine=1.0", abs(_cosine(v[0], v[1]) - 1.0) < 1e-9, - f"got {_cosine(v[0], v[1])}") -v2 = _tfidf_vectors(["cat dog", "bird fish"]) -check("disjoint texts cosine=0.0", _cosine(v2[0], v2[1]) == 0.0) - -# --- length mismatch raises --- -try: - mmr_rerank(["A", "B"], ["only one text"], 0.5) - check("length mismatch raises", False) -except ValueError: - check("length mismatch raises", True) - -# --- deterministic --- -r1 = mmr_rerank(ids, texts, 0.7) -r2 = mmr_rerank(ids, texts, 0.7) -check("deterministic output", r1 == r2) - -# --- pool sizes: sanity on a bigger pool --- -big_ids = [f"s{i}" for i in range(30)] -big_texts = [f"doc {i} topic {i % 5}" for i in range(30)] -big_out = mmr_rerank(big_ids, big_texts, 0.6) -check("big pool: permutation valid", sorted(big_out) == sorted(big_ids)) -check("big pool: top item is rank-1", big_out[0] == "s0") - -print(f"\n{PASS} passed, {FAIL} failure(s)") -sys.exit(1 if FAIL else 0) diff --git a/benchmarks/longmemeval/test_temporal.py b/benchmarks/longmemeval/test_temporal.py deleted file mode 100644 index beeab1e2..00000000 --- a/benchmarks/longmemeval/test_temporal.py +++ /dev/null @@ -1,164 +0,0 @@ -#!/usr/bin/env python3 -"""Unit tests for temporal.py (#1119). - -Run: python3 -m pytest test_temporal.py -v (or python3 test_temporal.py) -""" -import sys -from datetime import date -from pathlib import Path - -sys.path.insert(0, str(Path(__file__).parent)) -from temporal import (parse_temporal, parse_question_date, in_window, - boost_ranking, RRF_K) - - -# ---------- parse_question_date ---------- - -def test_qdate_standard(): - assert parse_question_date("2023/03/22 (Wed) 21:49") == date(2023, 3, 22) - -def test_qdate_none(): - assert parse_question_date(None) is None - assert parse_question_date("") is None - assert parse_question_date("no date here") is None - - -# ---------- parse_temporal: resolver coverage ---------- - -Q = date(2023, 3, 22) - -def test_between(): - w = parse_temporal("What did I do between February and April?", Q) - assert w[2] == "between" - assert w[0] == date(2023, 2, 1) - assert w[1] == date(2023, 4, 30) - -def test_before(): - w = parse_temporal("What did I buy before January?", Q) - assert w[2] == "before" - assert w[0] is None and w[1] == date(2023, 1, 31) - -def test_after(): - w = parse_temporal("What changed after March?", Q) - assert w[2] == "after/since" - assert w[0] == date(2023, 3, 1) and w[1] is None - -def test_since_with_year(): - w = parse_temporal("since May 2022", Q) - assert w[0] == date(2022, 5, 1) and w[1] is None - -def test_in_month(): - w = parse_temporal("What did I eat in February?", Q) - assert w[2] == "in-month" - assert w[0] == date(2023, 2, 1) and w[1] == date(2023, 2, 28) - -def test_in_month_wrap_year(): - # 'May' asked in March 2023 -> 2022-05 (haystack year precedes question) - w = parse_temporal("in May", Q) - assert w[0] == date(2022, 5, 1) - -def test_last_n_months(): - w = parse_temporal("What did I cook in the last 3 months?", Q) - assert w[2] == "last-N" - assert w[0] == date(2022, 12, 22) and w[1] == Q - -def test_last_month(): - w = parse_temporal("last month", Q) - assert w[2] == "last-month" - assert w[0] == date(2023, 2, 1) and w[1] == date(2023, 2, 28) - -def test_last_month_january_edge(): - w = parse_temporal("last month", date(2023, 1, 15)) - assert w[0] == date(2022, 12, 1) and w[1] == date(2022, 12, 31) - -def test_this_month(): - w = parse_temporal("this month", Q) - assert w[0] == date(2023, 3, 1) - assert w[1] == Q # window extends through question date itself - -def test_this_year(): - w = parse_temporal("this year", Q) - assert w[0] == date(2023, 1, 1) and w[1] == Q - -def test_no_temporal(): - assert parse_temporal("What is my favorite sushi?", Q) is None - # Month name used as a proper noun must NOT parse (strict patterns - # require a preposition, e.g. "in January", not "January Jones"). - assert parse_temporal("January Jones is an actress", Q) is None - assert parse_temporal("", Q) is None - assert parse_temporal("last month", None) is None - - -# ---------- in_window ---------- - -def test_in_window_basic(): - w = (date(2023, 2, 1), date(2023, 2, 28), "t") - assert in_window(date(2023, 2, 15), w) - assert not in_window(date(2023, 3, 1), w) - -def test_in_window_open_bounds(): - w = (None, date(2023, 1, 31), "t") - assert in_window(date(2020, 6, 1), w) - assert not in_window(date(2023, 2, 1), w) - - -# ---------- boost_ranking ---------- - -def test_boost_lifts_in_window_session(): - # session at rank 9 (RRF 1/69 = 0.01449) with boost 0.002 -> 0.01649 - # rank 5 (1/65 = 0.01538) without boost stays below it? 0.01538 < 0.01649 yes - sids = [f"s{i}" for i in range(1, 16)] - dates = {f"s{i}": date(2023, 2, 10) if i == 9 else date(2022, 6, 1) for i in range(1, 16)} - w = (date(2023, 2, 1), date(3, 1, 1) and date(2023, 2, 28), "t") - out = boost_ranking(sids, dates, w) - assert out[0] == "s9", "in-window session at rank 9 should lift to top-5 (and here to #1 vs out-of-window ranks 2-4)" - -def test_boost_no_window_is_noop(): - sids = [f"s{i}" for i in range(1, 10)] - out = boost_ranking(sids, {}, None) - assert out == sids - -def test_boost_without_dates_is_noop(): - sids = [f"s{i}" for i in range(1, 10)] - w = (date(2023, 2, 1), date(2023, 2, 28), "t") - out = boost_ranking(sids, {}, w) - assert out == sids - -def test_boost_preserves_order_within_window(): - # two in-window sessions keep their recall order relative to each other - sids = [f"s{i}" for i in range(1, 12)] - dates = {sid: date(2023, 2, 5) for sid in sids} - w = (date(2023, 2, 1), date(2023, 2, 28), "t") - out = boost_ranking(sids, dates, w) - assert out == sids - -def test_boost_stable_semantic_top_ranked_not_demoted(): - # rank-1 session, in-window, must stay #1 even if many others boosted - sids = [f"s{i}" for i in range(1, 26)] - dates = {sid: date(2023, 2, 5) for sid in sids} - w = (date(2023, 2, 1), date(2023, 2, 28), "t") - out = boost_ranking(sids, dates, w) - assert out[0] == "s1" - -def test_rank_math(): - # Boundary property for recall@5: an in-window rank-15 must beat an - # out-of-window rank-5 (membership crossing), while a rank-1 out-of-window - # still beats an in-window rank-15 (head of ranking stays semantic). - r15_boosted = 1 / (RRF_K + 15) + 0.0022 - r5 = 1 / (RRF_K + 5) - r1 = 1 / (RRF_K + 1) - assert r15_boosted > r5, "in-window rank-15 must cross the rank-5 boundary" - assert r1 > r15_boosted, "rank-1 must stay above a boosted rank-15" - - -if __name__ == "__main__": - fails = 0 - for name, fn in [(n, f) for n, f in sorted(globals().items()) if n.startswith("test_") and callable(f)]: - try: - fn() - print(f"PASS {name}") - except AssertionError as e: - fails += 1 - print(f"FAIL {name}: {e}") - print(f"\n{fails} failure(s)") - sys.exit(1 if fails else 0) diff --git a/benchmarks/longmemeval/watch_and_pull.py b/benchmarks/longmemeval/watch_and_pull.py deleted file mode 100644 index 4019930e..00000000 --- a/benchmarks/longmemeval/watch_and_pull.py +++ /dev/null @@ -1,173 +0,0 @@ -#!/usr/bin/env python3 -"""Watch & pull LongMemEval shards from Modal Volume every 10 minutes. - -- Merges completed shards into results_modal_/retrieval_results.jsonl - (partial analysis possible anytime) -- Auto-relaunches `modal run modal_fanout.py` if the Modal run died - (shards on the Volume are resumed, not recomputed) -- Writes progress to lmeval_watch_state.json; log to lmeval_watch.log - -Run: nohup-style background via Hermes terminal(background=true). -Stop: kill the process (state persists on Volume + local jsonl). -""" -import json -import pathlib -import subprocess -import time - -BASE = pathlib.Path("/opt/data/github/codecoradev/uteke/benchmarks/longmemeval") -STRATEGY = "hybrid" -NUM_SHARDS = 20 -INTERVAL = 600 # 10 menit -STATE = BASE / "lmeval_watch_state.json" -LOG = BASE / "lmeval_watch.log" -RESULTS = BASE / f"results_modal_{STRATEGY}" / "retrieval_results.jsonl" - - -def log(msg): - line = f"[{time.strftime('%H:%M:%S')}] {msg}" - print(line, flush=True) - with open(LOG, "a") as f: - f.write(line + "\n") - - -def pull_shards(): - """Run pull_shards.py, parse JSON from last line.""" - r = subprocess.run( - ["modal", "run", "pull_shards.py", "--strategy", STRATEGY], - capture_output=True, text=True, cwd=BASE, timeout=300, - ) - if r.returncode != 0: - log(f"pull failed: {(r.stderr or r.stdout)[-200:]}") - return {} - for line in reversed((r.stdout + "\n" + r.stderr).splitlines()): - line = line.strip() - if line.startswith("{") and "shard_" in line: - try: - return json.loads(line) - except json.JSONDecodeError: - continue - return {} - - -def merge(shards: dict): - seen, rows = set(), [] - for fname in sorted(shards): - for line in shards[fname].splitlines(): - line = line.strip() - if not line: - continue - try: - qid = json.loads(line).get("question_id") - except json.JSONDecodeError: - continue - if qid and qid not in seen: - seen.add(qid) - rows.append(line) - RESULTS.parent.mkdir(parents=True, exist_ok=True) - RESULTS.write_text("\n".join(rows) + ("\n" if rows else "")) - return len(rows) - - -def app_running(): - """True kalau ada app uteke-longmemeval state=ephemeral DAN punya tasks > 0. - Pakai --json biar tidak kena format tabel terpotong.""" - import json as _json - r = subprocess.run(["modal", "app", "list", "--json"], capture_output=True, text=True, timeout=60) - try: - apps = _json.loads(r.stdout) - except _json.JSONDecodeError: - return True # gagal parse → asumsi jalan (jangan salah relaunch) - for a in apps: - desc = str(a.get("description", "")) - if "uteke-longmemeval" in desc and a.get("state") == "ephemeral": - return True - return False - - -def relaunch(): - log("app tidak aktif → relaunch modal_fanout (resume dari Volume)") - subprocess.Popen( - ["modal", "run", "modal_fanout.py", "--strategy", STRATEGY, - "--num-shards", str(NUM_SHARDS)], - cwd=BASE, - stdout=open(BASE / "lmeval_relaunch.log", "a"), - stderr=subprocess.STDOUT, - start_new_session=True, # lepas dari process group watcher — kill watcher ≠ kill run - ) - - -def report(n_shards: int, n_q: int, metrics_stdout: str): - """Progress report ke Discord thread benchmark (via discord_notify pattern).""" - import sys - sys.path.insert(0, "/opt/data/scripts") - try: - from discord_notify import send_embed - except ImportError: - log("report: discord_notify tidak importable — skip") - return - - recall5 = ndcg5 = recall10 = "?" - for line in metrics_stdout.splitlines(): - if "recall_all@5" in line: - recall5 = line.split("=")[1].strip() - elif "ndcg_any@5" in line: - ndcg5 = line.split("=")[1].strip() - elif "recall_all@10" in line: - recall10 = line.split("=")[1].strip() - - pct = n_q * 100 // 500 - desc = ( - f"Shards selesai: **{n_shards}/20** · Q merged: **{n_q}/500** ({pct}%)\n" - f"recall_all@5 = **{recall5}** · ndcg_any@5 = **{ndcg5}** · recall_all@10 = {recall10}" - ) - r = send_embed("1541981760668176484", { # thread benchmark ini - "title": f"📊 Progress {time.strftime('%H:%M')} — LongMemEval 500Q hybrid", - "description": desc, - "color": 0x3498DB, - }, bot="cmo") - log(f"report: {'OK' if r.get('success') else r.get('error')}") - - -def main(): - log(f"watcher start: strategy={STRATEGY} shards={NUM_SHARDS} interval={INTERVAL}s") - while True: - try: - shards = pull_shards() - n_shards = len(shards) - n_q = merge(shards) - state = {"last_pull": time.strftime("%H:%M:%S"), - "shards_done": n_shards, "questions": n_q} - json.dump(state, open(STATE, "w"), indent=1) - log(f"pull: {n_shards}/{NUM_SHARDS} shard, {n_q} Q merged → {RESULTS.name}") - - # hitung partial metrics + report ke Discord - pm = subprocess.run( - ["python3", "print_metrics.py", str(RESULTS)], - capture_output=True, text=True, cwd=BASE, timeout=120, - ) - report(n_shards, n_q, pm.stdout) - - if n_shards >= NUM_SHARDS: - if n_q >= 480: - # 480/500 = toleransi beberapa Q gagal evaluasi (metrics None) - log("SEMUA SHARD SELESAI — watcher berhenti") - # final: simpan metrik lengkap (pm sudah dihitung di atas) - open(BASE / "lmeval_final_metrics.txt", "w").write(pm.stdout) - log("metrik final tersimpan: lmeval_final_metrics.txt") - return - else: - log(f"20 file shard ada tapi baru {n_q}/500 Q — ada shard partial; " - f"tunggu relaunch resume (kode 14400s) menuntaskan sisa Q") - - if not app_running(): - relaunch() - else: - log("app masih aktif") - except Exception as e: - log(f"error: {type(e).__name__}: {e}") - time.sleep(INTERVAL) - - -if __name__ == "__main__": - main() diff --git a/crates/uteke-cli/Cargo.toml b/crates/uteke-cli/Cargo.toml index eb2cdb87..a0c5868f 100644 --- a/crates/uteke-cli/Cargo.toml +++ b/crates/uteke-cli/Cargo.toml @@ -22,7 +22,7 @@ usearch = ["uteke-core/usearch"] vecq = ["uteke-core/vecq"] [dependencies] -uteke-core = { path = "../uteke-core", version = "0.18.0", default-features = false, features = ["onnx"] } +uteke-core = { path = "../uteke-core", version = "0.18.1", default-features = false, features = ["onnx"] } clap = { version = "4", features = ["derive"] } serde_json = "1" tracing = "0.1" diff --git a/crates/uteke-cli/src/commands/upgrade.rs b/crates/uteke-cli/src/commands/upgrade.rs index 3c3e519d..fce89522 100644 --- a/crates/uteke-cli/src/commands/upgrade.rs +++ b/crates/uteke-cli/src/commands/upgrade.rs @@ -11,6 +11,8 @@ use sha2::{Digest, Sha256}; const REPO: &str = "codecoradev/uteke"; const BINARY_NAME: &str = "uteke"; +const SERVER_BINARY_NAME: &str = "uteke-serve"; +const MCP_BINARY_NAME: &str = "uteke-mcp"; /// Entry point for `uteke upgrade`. pub fn run(yes: bool) -> Result<(), String> { @@ -35,8 +37,11 @@ pub fn run(yes: bool) -> Result<(), String> { // 4. Get latest release version let latest_version = get_latest_version()?; - // 5. Check if already up to date - if latest_version == current_version { + // 5. Check if already up to date. The release tag carries a leading `v` + // that CARGO_PKG_VERSION does not — compare normalized, or an up-to-date + // install is re-offered the same release (#1245). + let latest_clean = latest_version.trim_start_matches('v'); + if latest_clean == current_version { println!("[INFO] Already up to date ({current_version})"); return Ok(()); } @@ -178,21 +183,56 @@ pub fn run(yes: bool) -> Result<(), String> { .unpack(&temp_dir) .map_err(|e| format!("Failed to extract archive: {e}"))?; - // 11. Find and replace binary - let extracted_binary = temp_dir.join(BINARY_NAME); - if !extracted_binary.exists() { - let _ = fs::remove_dir_all(&temp_dir); - return Err(format!("Binary '{BINARY_NAME}' not found in archive")); - } - + // 11. Replace binaries and bundled libs from the same verified archive. + // `uteke upgrade` must keep every installed artifact in sync with the + // release bundle (#1245): replacing only the CLI left `uteke-serve` and + // `uteke-mcp` on the old version and discarded the freshly downloaded + // ONNX Runtime libs (install.sh has installed those since #1221). let install_dir = current_exe .parent() .ok_or_else(|| "Cannot determine install directory".to_string())?; + // CLI binary first — hard-fail on any problem (existing behavior). + replace_binary(&temp_dir, BINARY_NAME, install_dir, true)?; + + // Companion binaries — replace when present in the archive; warn+skip + // otherwise so old installs without them still upgrade cleanly. + for name in [SERVER_BINARY_NAME, MCP_BINARY_NAME] { + replace_binary(&temp_dir, name, install_dir, false)?; + } + + // Bundled ONNX Runtime shared libs — refresh from the archive when present. + refresh_ort_libs(&temp_dir, install_dir)?; + + // 12. Cleanup + let _ = fs::remove_dir_all(&temp_dir); + + println!("[INFO] Update complete. ({current_version} → {latest_version})"); + + Ok(()) +} + +/// Verify a freshly extracted binary runs, then atomically move it into +/// `install_dir`. With `required = false`, a missing artifact is skipped +/// with a warning (companion binaries absent from older bundles). +fn replace_binary( + temp_dir: &std::path::Path, + name: &str, + install_dir: &std::path::Path, + required: bool, +) -> Result<(), String> { + let extracted = temp_dir.join(name); + if !extracted.exists() { + if required { + return Err(format!("Binary '{name}' not found in archive")); + } + println!("[WARN] {name} not in bundle — skipping (left at its installed version)"); + return Ok(()); + } + // Copy to temp file first, then rename (atomic on POSIX) - let temp_new = install_dir.join(format!("{BINARY_NAME}.new")); - fs::copy(&extracted_binary, &temp_new) - .map_err(|e| format!("Failed to copy new binary: {e}"))?; + let temp_new = install_dir.join(format!("{name}.new")); + fs::copy(&extracted, &temp_new).map_err(|e| format!("Failed to copy new {name}: {e}"))?; // Verify the new binary runs match std::process::Command::new(&temp_new) @@ -203,31 +243,81 @@ pub fn run(yes: bool) -> Result<(), String> { let new_version = String::from_utf8_lossy(&output.stdout).trim().to_string(); // Extract version from clap output like "uteke 0.6.7" let extracted_version = new_version.split_whitespace().nth(1).unwrap_or("unknown"); - println!("[INFO] Verified new binary: {extracted_version}"); + println!("[INFO] Verified new {name}: {extracted_version}"); } Ok(output) => { let _ = fs::remove_file(&temp_new); - let _ = fs::remove_dir_all(&temp_dir); return Err(format!( - "New binary failed to run: {}", + "New {name} failed to run: {}", String::from_utf8_lossy(&output.stderr) )); } Err(e) => { let _ = fs::remove_file(&temp_new); - let _ = fs::remove_dir_all(&temp_dir); - return Err(format!("Failed to verify new binary: {e}")); + return Err(format!("Failed to verify new {name}: {e}")); } } - // Atomic rename - fs::rename(&temp_new, ¤t_exe).map_err(|e| format!("Failed to replace binary: {e}"))?; - - // 12. Cleanup - let _ = fs::remove_dir_all(&temp_dir); - - println!("[INFO] Update complete. ({current_version} → {latest_version})"); + fs::rename(&temp_new, install_dir.join(name)) + .map_err(|e| format!("Failed to replace {name}: {e}"))?; + Ok(()) +} +/// Copy bundled ONNX Runtime shared libs from the extracted archive into the +/// install dir when present (mirrors install.sh since #1221). Symlinks are +/// preserved; a bundle without libs (very old releases) leaves existing libs +/// untouched. +fn refresh_ort_libs( + temp_dir: &std::path::Path, + install_dir: &std::path::Path, +) -> Result<(), String> { + let mut refreshed = 0usize; + let entries = fs::read_dir(temp_dir).map_err(|e| format!("Failed to read bundle dir: {e}"))?; + for entry in entries.flatten() { + let fname = entry.file_name(); + let fname = fname.to_string_lossy().to_string(); + if !fname.starts_with("libonnxruntime") { + continue; + } + let dest = install_dir.join(&fname); + let ft = entry + .file_type() + .map_err(|e| format!("Failed to stat {fname}: {e}"))?; + if ft.is_file() { + // Stage to a temp name, then rename over the target. Writing in + // place would truncate the live lib under a running `uteke-serve` + // (mmap -> SIGBUS) and an interrupted copy would leave a corrupt + // lib behind; rename is atomic on POSIX. + let staged = install_dir.join(format!("{fname}.new")); + fs::copy(entry.path(), &staged).map_err(|e| format!("Failed to stage {fname}: {e}"))?; + fs::rename(&staged, &dest).map_err(|e| format!("Failed to install {fname}: {e}"))?; + refreshed += 1; + } else if ft.is_symlink() { + #[cfg(unix)] + { + let target = fs::read_link(entry.path()) + .map_err(|e| format!("Failed to read link {fname}: {e}"))?; + let staged = install_dir.join(format!("{fname}.new")); + let _ = fs::remove_file(&staged); + std::os::unix::fs::symlink(&target, &staged) + .map_err(|e| format!("Failed to link {fname}: {e}"))?; + fs::rename(&staged, &dest) + .map_err(|e| format!("Failed to install {fname}: {e}"))?; + refreshed += 1; + } + #[cfg(not(unix))] + { + let _ = dest; + // Symlinked libs are a Unix packaging detail; other platforms + // keep their installed libs untouched. + } + } + } + if refreshed > 0 { + println!("[INFO] Refreshed {refreshed} ONNX Runtime lib file(s)"); + } else { + println!("[WARN] No ONNX Runtime libs in bundle — existing libs left untouched"); + } Ok(()) } @@ -321,3 +411,57 @@ fn sha256_file(path: &PathBuf) -> Result { .map_err(|e| format!("Failed to read file for hashing: {e}"))?; Ok(format!("{:x}", hasher.finalize())) } + +#[cfg(all(test, unix))] +mod tests { + use super::*; + use std::os::unix::fs::PermissionsExt; + + #[test] + fn replace_binary_verifies_and_installs() { + let dir = std::env::temp_dir().join(format!("uteke-upgrade-test-{}", std::process::id())); + let bundle = dir.join("bundle"); + let inst = dir.join("inst"); + fs::create_dir_all(&bundle).unwrap(); + fs::create_dir_all(&inst).unwrap(); + + // Fake binary that runs successfully and reports a version. + let fake = bundle.join("uteke"); + fs::write(&fake, "#!/bin/sh\necho \"uteke 9.9.9\"\n").unwrap(); + fs::set_permissions(&fake, fs::Permissions::from_mode(0o755)).unwrap(); + + replace_binary(&bundle, "uteke", &inst, true).unwrap(); + assert!(inst.join("uteke").exists(), "binary installed"); + + // Optional artifact missing -> skipped with a warning, no error. + replace_binary(&bundle, "uteke-not-shipped", &inst, false).unwrap(); + assert!(!inst.join("uteke-not-shipped").exists()); + + // Required artifact missing -> hard error. + assert!(replace_binary(&bundle, "uteke-required", &inst, true).is_err()); + + let _ = fs::remove_dir_all(&dir); + } + + #[test] + fn replace_binary_rejects_broken_artifact() { + let dir = std::env::temp_dir().join(format!("uteke-upgrade-broken-{}", std::process::id())); + let bundle = dir.join("bundle"); + let inst = dir.join("inst"); + fs::create_dir_all(&bundle).unwrap(); + fs::create_dir_all(&inst).unwrap(); + + // Binary that exits non-zero must fail verification and not install. + let bad = bundle.join("uteke-broken"); + fs::write(&bad, "#!/bin/sh\nexit 1\n").unwrap(); + fs::set_permissions(&bad, fs::Permissions::from_mode(0o755)).unwrap(); + + assert!(replace_binary(&bundle, "uteke-broken", &inst, true).is_err()); + assert!( + !inst.join("uteke-broken").exists(), + "broken binary must not be installed" + ); + + let _ = fs::remove_dir_all(&dir); + } +} diff --git a/crates/uteke-cli/src/config.rs b/crates/uteke-cli/src/config.rs index a3dd3b1b..a1d673bc 100644 --- a/crates/uteke-cli/src/config.rs +++ b/crates/uteke-cli/src/config.rs @@ -462,6 +462,11 @@ pub struct Config { /// Set to `false` to disable. #[serde(default = "default_true")] pub update_check: bool, + /// Show a one-line "Like Uteke?" footer after `uteke doctor` succeeds + /// (#1246). Interactive terminals only. Default: true. Set to `false` + /// to disable. + #[serde(default = "default_true")] + pub doctor_footer: bool, } /// Serde default helper: returns `true`. @@ -487,6 +492,7 @@ impl Default for Config { dream: DreamConfig::default(), lifecycle: LifecycleConfig::default(), update_check: true, + doctor_footer: true, } } } diff --git a/crates/uteke-cli/src/output.rs b/crates/uteke-cli/src/output.rs index 0379a803..c1cc5aad 100644 --- a/crates/uteke-cli/src/output.rs +++ b/crates/uteke-cli/src/output.rs @@ -215,11 +215,26 @@ pub(crate) fn print_doctor_human(report: &uteke_core::DoctorReport) { } if all_ok { println!("\n All checks passed."); + print_doctor_footer(); } else { println!("\n Some checks failed. Run `uteke repair` if index is inconsistent."); } } +/// One-line post-`doctor` callout (#1246). Quiet by design: interactive +/// terminals only (CI/pipes stay machine-clean), config opt-out via +/// `doctor_footer = false` in `uteke.toml`. +fn print_doctor_footer() { + use std::io::IsTerminal; + if !std::io::stdout().is_terminal() { + return; + } + if !crate::Config::load().doctor_footer { + return; + } + println!(" Like Uteke? ★ github.com/codecoradev/uteke · Managed for you: uteke.cloud"); +} + /// Print verify report in human-readable format. pub(crate) fn print_verify_human(report: &uteke_core::VerifyReport) { println!("Verify Report"); diff --git a/crates/uteke-core/src/structural_export.rs b/crates/uteke-core/src/structural_export.rs index 508e4ae7..41e5c8e3 100644 --- a/crates/uteke-core/src/structural_export.rs +++ b/crates/uteke-core/src/structural_export.rs @@ -54,9 +54,36 @@ impl crate::Uteke { // Manifest first line — import detects this to route here. let mut sections = serde_json::Map::new(); for s in SECTIONS { - let count: i64 = conn - .query_row(&format!("SELECT COUNT(*) FROM {s}"), [], |r| r.get(0)) - .unwrap_or(0); + // #1243: memory-referencing sections are counted with the same + // liveness predicate the dump uses, so manifest counts match the + // rows actually exported when the store holds soft-deleted memories. + // All queries are literals — no identifier interpolation. + let count_sql: &str = match *s { + "memories" => "SELECT COUNT(*) FROM memories WHERE deprecated = 0", + "room_memories" => { + "SELECT COUNT(*) FROM room_memories rm \ + JOIN memories m ON m.id = rm.memory_id WHERE m.deprecated = 0" + } + "memory_edges" => { + "SELECT COUNT(*) FROM memory_edges me \ + JOIN memories s ON s.id = me.source_id \ + JOIN memories t ON t.id = me.target_id \ + WHERE s.deprecated = 0 AND t.deprecated = 0" + } + "timeline_events" => { + "SELECT COUNT(*) FROM timeline_events te \ + JOIN memories m ON m.id = te.memory_id WHERE m.deprecated = 0" + } + // Pure structural tables — no memory reference, nothing to filter. + "rooms" => "SELECT COUNT(*) FROM rooms", + "room_documents" => "SELECT COUNT(*) FROM room_documents", + "graph_nodes" => "SELECT COUNT(*) FROM graph_nodes", + "graph_edges" => "SELECT COUNT(*) FROM graph_edges", + "documents" => "SELECT COUNT(*) FROM documents", + "document_chunks" => "SELECT COUNT(*) FROM document_chunks", + _ => unreachable!("unknown export section: {s}"), + }; + let count: i64 = conn.query_row(count_sql, [], |r| r.get(0)).unwrap_or(0); sections.insert((*s).to_string(), serde_json::json!(count)); } let manifest = serde_json::json!({ @@ -144,28 +171,48 @@ impl crate::Uteke { FROM rooms", ), ( + // #1243: only memberships of live (non-deprecated) memories — + // junction rows must not outlive the memories they reference. "room_memory", - "SELECT room_id, memory_id, author, joined_at FROM room_memories", + "SELECT rm.room_id, rm.memory_id, rm.author, rm.joined_at \ + FROM room_memories rm \ + JOIN memories m ON m.id = rm.memory_id WHERE m.deprecated = 0", ), ( "room_document", "SELECT room_id, doc_slug, added_at FROM room_documents", ), ( + // #1243: nodes survive, but a memory link pointing at a + // soft-deleted memory is nulled (mirrors the schema's + // ON DELETE SET NULL behavior for hard deletes). "graph_node", - "SELECT id, label, entity_type, properties_json, memory_id, created_at FROM graph_nodes", + "SELECT gn.id, gn.label, gn.entity_type, gn.properties_json, \ + CASE WHEN m.id IS NOT NULL AND m.deprecated = 0 \ + THEN gn.memory_id ELSE NULL END, gn.created_at \ + FROM graph_nodes gn \ + LEFT JOIN memories m ON m.id = gn.memory_id", ), ( "graph_edge", "SELECT id, source_id, target_id, relation, weight, created_at FROM graph_edges", ), ( + // #1243: an edge exists only while BOTH endpoints are live; + // edges touching a soft-deleted memory are dropped. "memory_edge", - "SELECT source_id, target_id, edge_type, created_at FROM memory_edges", + "SELECT me.source_id, me.target_id, me.edge_type, me.created_at \ + FROM memory_edges me \ + JOIN memories s ON s.id = me.source_id \ + JOIN memories t ON t.id = me.target_id \ + WHERE s.deprecated = 0 AND t.deprecated = 0", ), ( + // #1243: events of a soft-deleted memory are dropped with it. "timeline_event", - "SELECT id, memory_id, event_type, event_data, created_at FROM timeline_events", + "SELECT te.id, te.memory_id, te.event_type, te.event_data, te.created_at \ + FROM timeline_events te \ + JOIN memories m ON m.id = te.memory_id WHERE m.deprecated = 0", ), ]; @@ -783,4 +830,149 @@ mod tests { "legacy rows carry no manifest marker" ); } + + /// #1243: junction rows must not outlive their memory in the export. + /// A store containing a soft-deleted (deprecated) memory must produce an + /// export that imports cleanly into a fresh store with FK enforcement on: + /// room memberships, memory edges, and timeline events referencing the + /// dead memory are filtered at export time, and graph-node memory links + /// to it are nulled (node survives, dangling link does not). + #[test] + fn test_structural_export_drops_soft_deleted_memory_references() { + let dir_a = std::env::temp_dir().join(format!("sx-da-{}", std::process::id())); + let dir_b = std::env::temp_dir().join(format!("sx-db-{}", std::process::id())); + std::fs::create_dir_all(&dir_a).unwrap(); + std::fs::create_dir_all(&dir_b).unwrap(); + + let src = Uteke::open(dir_a.join("t.db").to_str().unwrap()).unwrap(); + let conn = src.graph_store(); + + let now = "2026-09-15T00:00:00Z"; + // m1: live. m2: soft-deleted. + conn.execute( + "INSERT INTO memories (id, content, embedding, tags, metadata, created_at, updated_at, namespace, access_count, deprecated, memory_type, importance, pinned, content_type, source_type) VALUES ('m1','live memory',NULL,'[]','{}',?1,?1,'sx-dns',0,0,'fact',0.5,0,'text','user')", + [now], + ).unwrap(); + conn.execute( + "INSERT INTO memories (id, content, embedding, tags, metadata, created_at, updated_at, namespace, access_count, deprecated, memory_type, importance, pinned, content_type, source_type) VALUES ('m2','dead memory',NULL,'[]','{}',?1,?1,'sx-dns',0,1,'fact',0.5,0,'text','user')", + [now], + ).unwrap(); + conn.execute( + "INSERT INTO rooms (id, title, namespace, created_at, updated_at) VALUES ('r1','dead-ref room','sx-dns',?1,?1)", + [now], + ).unwrap(); + conn.execute( + "INSERT INTO room_memories (room_id, memory_id, author, joined_at) VALUES ('r1','m1','alice',?1)", + [now], + ).unwrap(); + conn.execute( + "INSERT INTO room_memories (room_id, memory_id, author, joined_at) VALUES ('r1','m2','alice',?1)", + [now], + ).unwrap(); + conn.execute( + "INSERT INTO memory_edges (source_id, target_id, edge_type, created_at) VALUES ('m1','m2','related',?1)", + [now], + ).unwrap(); + conn.execute( + "INSERT INTO timeline_events (memory_id, event_type, event_data, created_at) VALUES ('m2','created','{}',?1)", + [now], + ).unwrap(); + conn.execute( + "INSERT INTO graph_nodes (id, label, entity_type, properties_json, memory_id, created_at) VALUES ('gn1','dead-entity','concept','{}','m2',?1)", + [now], + ).unwrap(); + conn.execute( + "INSERT INTO graph_nodes (id, label, entity_type, properties_json, memory_id, created_at) VALUES ('gn2','live-entity','concept','{}','m1',?1)", + [now], + ).unwrap(); + + let exported = src.export_full().unwrap(); + let manifest: serde_json::Value = + serde_json::from_str(exported.lines().next().unwrap()).unwrap(); + let sections = &manifest["uteke_export"]["sections"]; + assert_eq!(sections["memories"], 1, "only the live memory is exported"); + assert_eq!( + sections["room_memories"], 1, + "dead memory's room link is filtered" + ); + assert_eq!( + sections["memory_edges"], 0, + "edge into the dead memory is filtered" + ); + assert_eq!( + sections["timeline_events"], 0, + "dead memory's timeline event is filtered" + ); + assert_eq!( + sections["graph_nodes"], 2, + "graph nodes survive with dead links nulled" + ); + + assert!( + !exported.contains("dead memory"), + "soft-deleted memory content must not appear in the dump" + ); + + // The node referencing the dead memory must be exported with a NULL + // memory link; the node referencing the live memory keeps its link. + let mut node_links = std::collections::HashMap::new(); + for line in exported.lines() { + let Ok(v) = serde_json::from_str::(line) else { + continue; + }; + if v["type"] == "graph_node" { + let id = v["row"][0].as_str().unwrap_or_default().to_string(); + node_links.insert(id, v["row"][4].clone()); + } + } + assert_eq!(node_links.len(), 2, "both nodes exported"); + assert!( + node_links["gn1"].is_null(), + "graph-node link to a soft-deleted memory must be nulled" + ); + assert_eq!( + node_links["gn2"], + serde_json::json!("m1"), + "graph-node link to a live memory survives" + ); + + // FK-enforced import into a fresh store must succeed. + let dst = Uteke::open(dir_b.join("t.db").to_str().unwrap()).unwrap(); + let result = dst.import_full(&exported).unwrap(); + assert_eq!(result["imported"]["memories"], 1); + assert_eq!(result["imported"]["room_memories"], 1); + assert_eq!(result["imported"]["graph_nodes"], 2); + // Sections with zero rows are absent from the counts map (not 0). + assert_eq!(result["imported"]["memory_edges"].as_u64().unwrap_or(0), 0); + assert_eq!( + result["imported"]["timeline_events"].as_u64().unwrap_or(0), + 0 + ); + + let dconn = dst.graph_store(); + let dead_link: Option = dconn + .query_row( + "SELECT memory_id FROM graph_nodes WHERE id='gn1'", + [], + |r| r.get(0), + ) + .unwrap(); + assert!( + dead_link.is_none(), + "restored node must not link the dead memory" + ); + let live_link: String = dconn + .query_row( + "SELECT memory_id FROM graph_nodes WHERE id='gn2'", + [], + |r| r.get(0), + ) + .unwrap(); + assert_eq!(live_link, "m1", "live link survives the round-trip"); + + drop(src); + drop(dst); + std::fs::remove_dir_all(&dir_a).ok(); + std::fs::remove_dir_all(&dir_b).ok(); + } } diff --git a/crates/uteke-mcp/Cargo.toml b/crates/uteke-mcp/Cargo.toml index 601f9a64..69aecdc1 100644 --- a/crates/uteke-mcp/Cargo.toml +++ b/crates/uteke-mcp/Cargo.toml @@ -29,7 +29,7 @@ usearch = ["uteke-core/usearch"] vecq = ["uteke-core/vecq"] [dependencies] -uteke-core = { path = "../uteke-core", version = "0.18.0", default-features = false, features = ["onnx"] } +uteke-core = { path = "../uteke-core", version = "0.18.1", default-features = false, features = ["onnx"] } serde = { version = "1", features = ["derive"] } serde_json = "1" dirs = "7" diff --git a/crates/uteke-server/Cargo.toml b/crates/uteke-server/Cargo.toml index 049d8200..f66a37e5 100644 --- a/crates/uteke-server/Cargo.toml +++ b/crates/uteke-server/Cargo.toml @@ -27,8 +27,8 @@ name = "uteke-serve" path = "src/main.rs" [dependencies] -uteke-core = { path = "../uteke-core", version = "0.18.0", default-features = false, features = ["onnx"] } -uteke-mcp = { path = "../uteke-mcp", version = "0.18.0", default-features = false } +uteke-core = { path = "../uteke-core", version = "0.18.1", default-features = false, features = ["onnx"] } +uteke-mcp = { path = "../uteke-mcp", version = "0.18.1", default-features = false } serde = { version = "1", features = ["derive"] } serde_json = "1" schemars = { version = "1", optional = true } diff --git a/crates/uteke-server/src/api_registry.rs b/crates/uteke-server/src/api_registry.rs index 541ed7e1..c5f03f39 100644 --- a/crates/uteke-server/src/api_registry.rs +++ b/crates/uteke-server/src/api_registry.rs @@ -40,7 +40,7 @@ pub const ENDPOINTS: &[Endpoint] = &[ Endpoint { method: "GET", path: "/health", - description: "Health check — returns server status and version", + description: "Health check — tokenless requests get {status} only; a valid bearer token (or auth-disabled server) returns status, version, memory counts, and update info", request_type: None, response_type: Some("HealthResponse"), excludes_deprecated: false, diff --git a/crates/uteke-server/src/context.rs b/crates/uteke-server/src/context.rs index a51f9c65..c9d4c512 100644 --- a/crates/uteke-server/src/context.rs +++ b/crates/uteke-server/src/context.rs @@ -116,6 +116,9 @@ pub enum AuthResult { Disabled, /// Authenticated with a specific role. Authenticated(ApiRole), + /// No valid token presented. Only the unauthenticated-friendly surface + /// (health check) is served — with a minimal payload (#1252). + Anonymous, } #[derive(Clone)] diff --git a/crates/uteke-server/src/handlers.rs b/crates/uteke-server/src/handlers.rs index 768adb28..376319c5 100644 --- a/crates/uteke-server/src/handlers.rs +++ b/crates/uteke-server/src/handlers.rs @@ -46,17 +46,27 @@ pub fn route(uteke: &Mutex, ctx: &ReqCtx, req: &mut Request) -> Response< ); } - // Health endpoint — no auth required (useful for load balancers) + // Health endpoint — reachable WITHOUT a token (load balancers, uptime + // probes), but anonymous callers get a minimal body: version, memory + // counts, and update info leak the instance fingerprint to scanners (#1252). let is_health = matches!((&method, route_path), (Method::Get, "/health")); - // Authenticate all non-health requests + // Authenticate every request; a tokenless health probe degrades to the + // Anonymous role instead of a 401 so uptime checks keep working. + let mut health_anonymous = false; let auth_role = if !is_health { match context::check_auth(req, ctx) { Ok(role) => role, Err(resp) => return resp, } } else { - AuthResult::Disabled + match context::check_auth(req, ctx) { + Ok(role) => role, + Err(_) => { + health_anonymous = true; + AuthResult::Anonymous + } + } }; // Enforce read-only restriction (#409, #524): @@ -106,6 +116,16 @@ pub fn route(uteke: &Mutex, ctx: &ReqCtx, req: &mut Request) -> Response< match (method, route_path) { // ── Health ────────────────────────────────────────────────────── (Method::Get, "/health") => { + // #1252: anonymous probe gets liveness only — no version, no + // counts, no update info. Authenticated (or auth-disabled) + // callers get the full payload, unchanged. + if health_anonymous { + #[derive(serde::Serialize)] + struct MinimalHealthResponse { + status: &'static str, + } + return ctx.ok_response_for(req, &MinimalHealthResponse { status: "ok" }); + } let total = uteke.count(None).unwrap_or(0); let namespaces = uteke.list_namespaces().unwrap_or_default().len(); // Populate update_available from cache (non-blocking, no network). @@ -3948,4 +3968,96 @@ mod payload_conformance_tests { .expect("recall response must be a top-level JSON array, never a summary string/stub"); assert!(arr.is_empty(), "expected an empty array for a no-hit query"); } + /// #1252: an unauthenticated GET /health gets liveness only — no version, + /// no counts — while an authenticated request (or auth-disabled server) + /// still receives the full payload. + #[test] + fn test_health_anonymous_gets_minimal_payload() { + struct HealthAuthApp { + uteke: Mutex, + admin_token: Option<&'static str>, + } + impl HealthAuthApp { + fn with_auth(token: Option<&'static str>) -> Self { + Self { + uteke: Mutex::new( + Uteke::open_with_backend(":memory:", None) + .expect("open in-memory uteke without embedder"), + ), + admin_token: token, + } + } + + fn call(&self, auth_header: Option<&str>) -> (u16, serde_json::Value) { + let mut builder = tiny_http::TestRequest::new() + .with_method(tiny_http::Method::Get) + .with_path("/health"); + if let Some(h) = auth_header { + builder = builder + .with_header(tiny_http::Header::from_bytes("Authorization", h).unwrap()); + } + let mut req: tiny_http::Request = builder.into(); + let ctx = ReqCtx { + auth_token_hash: self.admin_token.map(|t| { + use sha2::{Digest, Sha256}; + Sha256::digest(t.as_bytes()).into() + }), + read_only_token_hash: None, + cors_origins: Vec::new(), + recall_config: None, + extraction_config: None, + }; + let resp = route(&self.uteke, &ctx, &mut req); + let status = resp.status_code().0; + let bytes = resp.into_reader().into_inner(); + let json = serde_json::from_slice(&bytes).unwrap_or(serde_json::Value::Null); + (status, json) + } + } + + // Auth ENABLED, no token -> minimal body only. + let app = HealthAuthApp::with_auth(Some("secret-token")); + let (status, json) = app.call(None); + assert_eq!(status, 200); + assert_eq!(json["status"], "ok", "liveness must stay available"); + assert!( + json.get("version").is_none() + && json.get("memories").is_none() + && json.get("namespaces").is_none() + && json.get("update_available").is_none(), + "anonymous health must not leak instance details: {json}" + ); + + // Auth ENABLED, valid token -> full payload. + let (status, json) = app.call(Some("Bearer secret-token")); + assert_eq!(status, 200); + assert_eq!(json["status"], "ok"); + assert!( + json.get("version").is_some(), + "authenticated health carries version: {json}" + ); + assert!( + json.get("memories").is_some(), + "authenticated health carries counts: {json}" + ); + + // Auth ENABLED, wrong token -> same minimal body as anonymous: + // an invalid credential must not see more than a scanner would, + // and uptime probes keep a 200 while tokens are being rotated. + let (status, json) = app.call(Some("Bearer wrong-token")); + assert_eq!(status, 200); + assert!( + json.get("version").is_none() && json.get("memories").is_none(), + "invalid credentials must get the minimal body: {json}" + ); + + // Auth DISABLED, no token -> full payload (single-operator local mode). + let app = HealthAuthApp::with_auth(None); + let (status, json) = app.call(None); + assert_eq!(status, 200); + assert!( + json.get("memories").is_some(), + "auth-disabled health stays full: {json}" + ); + } } diff --git a/crates/uteke-server/src/types.rs b/crates/uteke-server/src/types.rs index b1a21087..14d144fa 100644 --- a/crates/uteke-server/src/types.rs +++ b/crates/uteke-server/src/types.rs @@ -176,6 +176,7 @@ pub fn to_v1_flat(result: &uteke_core::memory::types::UnifiedSearchResult) -> se #[cfg_attr(feature = "docgen", derive(schemars::JsonSchema))] #[derive(Deserialize)] +#[serde(deny_unknown_fields)] // #1251: unknown keys (e.g. v1-style `ns`) must 400, not vanish pub struct RememberRequest { pub content: String, #[serde(default)] @@ -631,6 +632,7 @@ pub struct ImportRequest { #[cfg_attr(feature = "docgen", derive(schemars::JsonSchema))] #[derive(Deserialize)] +#[serde(deny_unknown_fields)] // #1251: same strictness as /remember pub struct RoomRememberRequest { pub room_id: String, pub content: String, @@ -794,6 +796,47 @@ mod tests { assert_eq!(url_decode("%41%42%43"), "ABC"); } + /// #1251: v2 write payloads must reject unknown keys instead of silently + /// dropping them — a v1-style `ns` key used to be accepted with 200 and + /// the memory silently landed in the default namespace. + #[test] + fn test_remember_request_rejects_unknown_fields() { + let payload = r#"{"content": "hello", "ns": "repo-vetio"}"#; + let mut cursor = payload.as_bytes(); + let err = match read_body::(&mut cursor) { + Err(e) => e, + Ok(_) => panic!("unknown key `ns` must be rejected"), + }; + assert!( + err.contains("`ns`"), + "error must name the offending key: {err}" + ); + } + + #[test] + fn test_room_remember_request_rejects_unknown_fields() { + let payload = r#"{"room_id": "r1", "content": "hello", "room": "r1"}"#; + let mut cursor = payload.as_bytes(); + let err = match read_body::(&mut cursor) { + Err(e) => e, + Ok(_) => panic!("unknown key `room` must be rejected"), + }; + assert!( + err.contains("`room`"), + "error must name the offending key: {err}" + ); + } + + #[test] + fn test_remember_request_still_accepts_valid_payload() { + let payload = + r#"{"content": "hello", "tags": ["t"], "namespace": "n1", "metadata": {"k": "v"}}"#; + let mut cursor = payload.as_bytes(); + let req = read_body::(&mut cursor).unwrap(); + assert_eq!(req.content, "hello"); + assert_eq!(req.namespace.as_deref(), Some("n1")); + } + #[test] fn test_url_decode_trailing_percent() { // Truncated percent-encoding at end of string — should not panic. diff --git a/docs/api-reference.md b/docs/api-reference.md index 0894b1c7..0532fbb1 100644 --- a/docs/api-reference.md +++ b/docs/api-reference.md @@ -213,7 +213,7 @@ Rebuild backlink indices for memory graph. #### 🟢 `GET` `/health` -Health check — returns server status and version +Health check — tokenless requests get {status} only; a valid bearer token (or auth-disabled server) returns status, version, memory counts, and update info **Response**: [`HealthResponse`](#healthresponse) diff --git a/docs/configuration.md b/docs/configuration.md index 16c8beba..dcc5ec94 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -436,6 +436,10 @@ UTEKE_AUTH_TOKEN=admin-secret UTEKE_READ_ONLY_TOKEN=viewer-key uteke-serve Read-only tokens can only access GET endpoints (recall, search, list, stats, graph, health). POST/DELETE operations return `403 Forbidden`. +`GET /health` also answers without a token (for load balancers and uptime probes), +but tokenless — or invalidly-authenticated — requests receive only `{"status":"ok"}`. +Version, memory counts, and update info require a valid token (or auth disabled). + ## Aging (#247) Controls automatic memory lifecycle management. Disabled by default — opt-in for long-running stores. diff --git a/docs/docker.md b/docs/docker.md index c2635866..3c222ca5 100644 --- a/docs/docker.md +++ b/docs/docker.md @@ -120,20 +120,6 @@ environment: pure Rust) builds indexes ~16x faster with ~3x smaller files. See [configuration](configuration.md#environment-variables) for details. -### Choosing the vector engine (v0.17.0+) - -Both engines ship in the image. Pick one via env var — switching leaves your -data untouched; the new engine rebuilds its index from SQLite on the next start: - -```yaml -environment: - - UTEKE_VECTOR_BACKEND=vecq # or "usearch" (default) -``` - -`usearch` (HNSW) has the best query latency at scale; `vecq` (4-bit + residual, -pure Rust) builds indexes ~16x faster with ~3x smaller files. See -[configuration](configuration.md#environment-variables) for details. - ## Multi-Architecture Images are built for: diff --git a/docs/install.md b/docs/install.md index 98635fc1..da91c1d5 100644 --- a/docs/install.md +++ b/docs/install.md @@ -21,7 +21,7 @@ echo 'export PATH="$HOME/.local/bin:$PATH"' >> ~/.bashrc Pin a specific version: ```bash -UTEKE_VERSION=v0.18.0 curl -fsSL https://raw.githubusercontent.com/codecoradev/uteke/main/install.sh | sh +UTEKE_VERSION=v0.18.1 curl -fsSL https://raw.githubusercontent.com/codecoradev/uteke/main/install.sh | sh ``` ## Install via Cargo @@ -40,23 +40,23 @@ Download from [GitHub Releases](https://github.com/codecoradev/uteke/releases): ```bash # Linux (x86_64) — tarball also contains libonnxruntime.so* (required for embeddings) -curl -sL https://github.com/codecoradev/uteke/releases/latest/download/uteke-x86_64-unknown-linux-gnu-v0.18.0.tar.gz | tar xz +curl -sL https://github.com/codecoradev/uteke/releases/latest/download/uteke-x86_64-unknown-linux-gnu-v0.18.1.tar.gz | tar xz mv uteke uteke-serve uteke-mcp ~/.local/bin/ mv libonnxruntime.so* libonnxruntime_providers_shared.so* ~/.local/bin/ # Linux (x86_64, legacy — no AVX2/SSE4.2) -curl -sL https://github.com/codecoradev/uteke/releases/latest/download/uteke-x86_64-unknown-linux-gnu-legacy-v0.18.0.tar.gz | tar xz +curl -sL https://github.com/codecoradev/uteke/releases/latest/download/uteke-x86_64-unknown-linux-gnu-legacy-v0.18.1.tar.gz | tar xz mv uteke uteke-serve uteke-mcp ~/.local/bin/ mv libonnxruntime.so* ~/.local/bin/ mkdir -p ~/.local/bin/ort-legacy && mv ort-legacy/libonnxruntime.so* ~/.local/bin/ort-legacy/ # Linux (aarch64 / ARM) -curl -sL https://github.com/codecoradev/uteke/releases/latest/download/uteke-aarch64-unknown-linux-gnu-v0.18.0.tar.gz | tar xz +curl -sL https://github.com/codecoradev/uteke/releases/latest/download/uteke-aarch64-unknown-linux-gnu-v0.18.1.tar.gz | tar xz mv uteke uteke-serve uteke-mcp ~/.local/bin/ mv libonnxruntime.so* libonnxruntime_providers_shared.so* ~/.local/bin/ # macOS (Apple Silicon) -curl -sL https://github.com/codecoradev/uteke/releases/latest/download/uteke-aarch64-apple-darwin-v0.18.0.tar.gz | tar xz +curl -sL https://github.com/codecoradev/uteke/releases/latest/download/uteke-aarch64-apple-darwin-v0.18.1.tar.gz | tar xz mv uteke uteke-serve uteke-mcp ~/.local/bin/ mv libonnxruntime*.dylib ~/.local/bin/ ``` @@ -118,6 +118,13 @@ uteke upgrade # Check + upgrade to latest release uteke upgrade --yes # Skip confirmation ``` +`upgrade` keeps every installed artifact in sync with the release bundle: the +CLI, `uteke-serve`, `uteke-mcp`, and the bundled ONNX Runtime shared libraries +are all replaced from the same checksum-verified archive (binaries are +run-verified and swapped atomically). The version comparison normalizes the +release tag's `v` prefix, so an up-to-date install reports +"Already up to date" instead of re-offering the same release. + ## What Gets Installed The install script deploys three binaries plus the bundled ONNX Runtime shared library (required for local embeddings, no API keys needed):