diff --git a/.agents/skills/complex-dev/SKILL.md b/.agents/skills/complex-dev/SKILL.md new file mode 100644 index 0000000..d1a8284 --- /dev/null +++ b/.agents/skills/complex-dev/SKILL.md @@ -0,0 +1,43 @@ +--- +name: complex-dev +description: Execute complex programming tasks with a rigorous, zero-compromise workflow. Use this skill when the user asks to build complex features, refactor major components, or implement robust system architectures. +--- + +# Complex Development Workflow + +You are an expert software engineer executing complex programming tasks. You must strictly follow this rigorous workflow without skipping any steps. + +## 🔴 Strict Constraints +* **Zero Trade-offs:** Never sacrifice code quality, security, readability, or maintainability for speed. Do not use hacky workarounds or leave critical "TODO" comments. +* **CLAUDE.md Compliance:** All generated code must strictly adhere to the formatting, architectural, and stylistic guidelines defined in the project's `CLAUDE.md` file. + +--- + +## Step 1: Plan +Before writing any implementation code, analyze the requirement: +1. Outline the system architecture, data flow, and necessary components. +2. Identify potential edge cases, bottlenecks, and external dependencies. +3. Draft a brief step-by-step execution plan and explicitly state it. + +## Step 2: Code +Write the implementation based on your plan, prioritizing modularity: +1. **Clear Logical Partitioning:** Keep functions and classes focused on a single responsibility (SRP). +2. **File Splitting:** Do not dump unrelated logic into a monolithic file. Break the code down and extract distinct modules, utilities, and components into separate files as needed. +3. Ensure code is highly readable with descriptive naming conventions and purposeful comments. + +## Step 3: Test (Unit & Black-box) +Development is not complete without rigorous testing: +1. **Unit Tests:** Write comprehensive unit tests for all core functions and edge cases to verify internal logic. +2. **Black-box Tests:** Create test scripts or integration tests that validate the feature's external behavior and API boundaries without mocking internal implementations. + +## Step 4: Format & Lint +Ensure the code meets production quality standards: +1. Run the project's designated code formatter. +2. Run the project's linter. +3. Automatically fix **all** linting errors and warnings. The code must be completely clean. + +## Step 5: Verify +Finalize the task with end-to-end validation: +1. Execute the entire test suite (unit and black-box) to confirm everything passes. +2. Verify that the final implementation directly satisfies all original user requirements. +3. Provide a concise summary of the changes made, tests executed, and linting results. diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index e386f5d..f6d1a00 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -43,14 +43,23 @@ jobs: - name: Check workspace run: cargo check --locked --workspace --all-targets + - name: Check SQLite feature set + run: cargo check --locked --no-default-features --features sqlite-fts --workspace --all-targets + - name: Run Clippy run: cargo clippy --locked --workspace --all-targets -- -D warnings + - name: Run Clippy for SQLite + run: cargo clippy --locked --no-default-features --features sqlite-fts --workspace --all-targets -- -D warnings + - name: Run unit tests run: cargo test --locked --workspace + - name: Run SQLite unit tests + run: cargo test --locked --no-default-features --features sqlite-fts --workspace + blackbox: - name: PostgreSQL black-box + name: Storage black-box runs-on: ubuntu-latest steps: - name: Checkout diff --git a/CLAUDE.md b/CLAUDE.md index b0bcfb6..9a09edf 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -1,8 +1,8 @@ # abyss-backend Development Guide `abyss-backend` is the standalone open-source Agent event service. It owns -event ingestion, PostgreSQL persistence, event queries, optional Elasticsearch -projection, and the HTTP API that exposes those capabilities. +event ingestion, compile-time-selected persistence and search, event queries, +and the HTTP API that exposes those capabilities. Keep the repository focused on these public event-service responsibilities. @@ -11,9 +11,11 @@ Keep the repository focused on these public event-service responsibilities. - `api` owns HTTP routing and response boundaries. - `identity` validates the deployment bearer token and returns the stable standalone owner identifier. -- `usage` owns event request types, validation, ingestion, and queries. -- `search` owns the Elasticsearch projection and search worker. -- `db` owns PostgreSQL pooling, the consolidated migration, and Diesel models. +- `usage` owns event contracts plus backend-independent validation and ordering. +- `search` owns shared search contracts and safe projection extraction. +- `storage` is the `dyn StorageBackend` compatibility boundary and contains the + conditionally compiled PostgreSQL/Elasticsearch and SQLite/FTS5 backends. +- `db` owns PostgreSQL-only pooling, migrations, and Diesel models. Keep modules focused and start new Rust module files with a `//!` responsibility comment. Keep the implementation self-contained within this repository. @@ -23,6 +25,8 @@ comment. Keep the implementation self-contained within this repository. ```bash cargo +nightly fmt --all -- --check cargo clippy --workspace --all-targets -- -D warnings +cargo clippy --no-default-features --features sqlite-fts --workspace --all-targets -- -D warnings cargo test --workspace +cargo test --no-default-features --features sqlite-fts --workspace make test-blackbox ``` diff --git a/Cargo.lock b/Cargo.lock index a1b7bc9..cad5933 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -12,6 +12,7 @@ dependencies = [ "diesel", "diesel_migrations", "hex", + "libsqlite3-sys", "pq-sys", "r2d2", "reqwest", @@ -270,6 +271,12 @@ dependencies = [ "syn 2.0.119", ] +[[package]] +name = "deranged" +version = "0.5.8" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "7cd812cc2bc1d69d4764bd80df88b4317eaef9e773c75226407d9bc0876b211c" + [[package]] name = "diesel" version = "2.3.12" @@ -282,9 +289,12 @@ dependencies = [ "diesel_derives", "downcast-rs", "itoa", + "libsqlite3-sys", "pq-sys", "r2d2", "serde_json", + "sqlite-wasm-rs", + "time", "uuid", ] @@ -390,6 +400,12 @@ version = "1.0.7" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "3f9eec918d3f24069decb9af1554cad7c880e2da24a9afd88aca000531ab82c1" +[[package]] +name = "foldhash" +version = "0.2.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "77ce24cb58228fbb8aa041425bb1050850ac19177686ea6e0f41a70416f56fdb" + [[package]] name = "form_urlencoded" version = "1.2.2" @@ -469,6 +485,15 @@ dependencies = [ "wasm-bindgen", ] +[[package]] +name = "hashbrown" +version = "0.16.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "841d1cc9bed7f9236f321df977030373f4a4163ae1a7dbfe1a51a2c1a51d9100" +dependencies = [ + "foldhash", +] + [[package]] name = "heck" version = "0.5.0" @@ -755,6 +780,17 @@ version = "0.2.189" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "3eaf3ede3fee6db1a4c2ee091bf8a8b4dccdc6d17f656fb07896ee72867612f2" +[[package]] +name = "libsqlite3-sys" +version = "0.35.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "133c182a6a2c87864fe97778797e46c7e999672690dc9fa3ee8e241aa4a9c13f" +dependencies = [ + "cc", + "pkg-config", + "vcpkg", +] + [[package]] name = "litemap" version = "0.8.3" @@ -850,6 +886,12 @@ dependencies = [ "windows-sys 0.61.2", ] +[[package]] +name = "num-conv" +version = "0.2.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "521739c6d2bac4aa25192232afe6841231376b2b26d4d9fae5ecf8ca5772e441" + [[package]] name = "num-traits" version = "0.2.19" @@ -915,6 +957,12 @@ dependencies = [ "zerovec", ] +[[package]] +name = "powerfmt" +version = "0.2.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "439ee305def115ba05938db6eb1644ff94165c5ab5e9420d1c1bcedbba909391" + [[package]] name = "pq-src" version = "0.3.11+libpq-18.3" @@ -1131,6 +1179,16 @@ dependencies = [ "windows-sys 0.52.0", ] +[[package]] +name = "rsqlite-vfs" +version = "0.1.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "c51c9ae4df8a7fba42103df5c621fa3c37eccf3a3c650879e90fc48b11cc192c" +dependencies = [ + "hashbrown", + "thiserror", +] + [[package]] name = "rustc-hash" version = "2.1.3" @@ -1332,6 +1390,18 @@ dependencies = [ "windows-sys 0.61.2", ] +[[package]] +name = "sqlite-wasm-rs" +version = "0.5.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "dc3efc0da82635d7e1ced0053bbbfa8c7ab9645d0bf36ceb4f7127bb85315d75" +dependencies = [ + "cc", + "js-sys", + "rsqlite-vfs", + "wasm-bindgen", +] + [[package]] name = "stable_deref_trait" version = "1.2.1" @@ -1421,6 +1491,36 @@ dependencies = [ "cfg-if", ] +[[package]] +name = "time" +version = "0.3.55" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "cdb87b95ec50ddfa440816d227a17b2ccbdda963a316a727fda0fc4334f7d134" +dependencies = [ + "deranged", + "num-conv", + "powerfmt", + "serde_core", + "time-core", + "time-macros", +] + +[[package]] +name = "time-core" +version = "0.1.9" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9e1c906769ad99c88eaa54e728060edef082f8e358ff32030cb7c7d315e81109" + +[[package]] +name = "time-macros" +version = "0.2.32" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "7e689342a48d2ea927c87ea50cabf8594854bf940e9310208848d680d668ed85" +dependencies = [ + "num-conv", + "time-core", +] + [[package]] name = "tinystr" version = "0.8.4" diff --git a/Makefile b/Makefile index 0cc5308..05d69f0 100644 --- a/Makefile +++ b/Makefile @@ -1,4 +1,4 @@ -.PHONY: check fmt clippy test test-blackbox docker-build k8s-render +.PHONY: check fmt clippy test test-blackbox test-blackbox-postgres test-blackbox-sqlite docker-build k8s-render check: fmt clippy test @@ -7,13 +7,20 @@ fmt: clippy: cargo clippy --locked --workspace --all-targets -- -D warnings + cargo clippy --locked --no-default-features --features sqlite-fts --workspace --all-targets -- -D warnings test: cargo test --locked --workspace + cargo test --locked --no-default-features --features sqlite-fts --workspace -test-blackbox: +test-blackbox: test-blackbox-postgres test-blackbox-sqlite + +test-blackbox-postgres: bash scripts/blackbox_abyss_backend.sh +test-blackbox-sqlite: + bash scripts/blackbox_sqlite_backend.sh + docker-build: docker build -t abyss-backend:local . diff --git a/README.md b/README.md index 286f1dc..29000f6 100644 --- a/README.md +++ b/README.md @@ -1,9 +1,8 @@ # abyss-backend `abyss-backend` is the open-source, self-hostable Agent event store for Abyss. -It accepts normalized Agent events, stores them in PostgreSQL, and exposes APIs -for event history, summaries, session timelines, attachments, and optional -full-text session search. +It accepts normalized Agent events and exposes APIs for event history, +summaries, session timelines, attachments, and full-text session search. This repository is intentionally focused on standalone event storage, queries, and optional full-text search. @@ -13,12 +12,17 @@ and optional full-text search. - Idempotent Agent event ingestion and raw event queries. - Token-usage summaries and ordered session timelines. - Validated image attachment storage and authorized download. -- Optional asynchronous Elasticsearch projection and session search. -- Health and PostgreSQL readiness probes. -- A consolidated PostgreSQL schema for new standalone deployments. +- Full-text session search through Elasticsearch or embedded SQLite FTS5. +- Health and storage-readiness probes. +- Embedded migrations for both supported storage profiles. -PostgreSQL is the only supported database in this migration step. SQLite is a -future addition and is not implemented here. +Exactly one storage profile is compiled into a binary: + +- `postgres-es` uses PostgreSQL as the source of truth and optionally projects + search documents to Elasticsearch. It is the default profile. +- `sqlite-fts` stores both authoritative data and its transactional FTS5 index + in one local SQLite file. It does not compile Diesel, PostgreSQL, reqwest, or + Elasticsearch worker code into the binary. ## Authentication @@ -34,7 +38,21 @@ export ABYSS_BACKEND_API_TOKEN_SHA256="$(printf '%s' "${ABYSS_API_TOKEN}" | open The plaintext token is never stored by `abyss-backend`. Put it in the Agent collector configuration and keep the digest in the backend secret store. -## Run locally +## Run locally with SQLite + +The SQLite profile has no external service dependency: + +```bash +export ABYSS_BACKEND_DATABASE_URL='./data/abyss.sqlite' +cargo run --locked --package abyss-backend \ + --no-default-features --features sqlite-fts +``` + +The backend creates the parent directory, database, schema, and FTS5 index on +startup. A `sqlite://` prefix is also accepted, for example +`sqlite://./data/abyss.sqlite`. + +## Run with PostgreSQL Start PostgreSQL, then configure the required environment variables: @@ -43,8 +61,8 @@ export ABYSS_BACKEND_DATABASE_URL='postgres://abyss:abyss@127.0.0.1:5432/abyss?s cargo run --locked --package abyss-backend ``` -The service listens on `0.0.0.0:8080` and runs its embedded migration by -default. Verify it with: +Both profiles listen on `0.0.0.0:8080` and run their embedded migrations by +default. Verify the selected store with: ```bash curl http://127.0.0.1:8080/healthz @@ -64,13 +82,13 @@ curl \ | Method | Path | Purpose | | --- | --- | --- | | `GET` | `/healthz` | Process liveness. | -| `GET` | `/readyz` | PostgreSQL readiness. | +| `GET` | `/readyz` | Selected storage readiness. | | `POST` | `/v1/agent-usage/events` | Ingest events and correlated diagnostic captures. | | `GET` | `/v1/agent-usage/events` | Query raw events. | | `GET` | `/v1/agent-usage/attachments/{id}` | Download stored image content. | | `GET` | `/v1/agent-usage/summary` | Aggregate event and token usage. | | `GET` | `/v1/agent-usage/sessions/{id}` | Read an ordered session timeline. | -| `GET` | `/v1/agent-usage/search` | Search sessions when Elasticsearch is configured. | +| `GET` | `/v1/agent-usage/search` | Search sessions through Elasticsearch or SQLite FTS5. | Every `/v1/agent-usage/*` endpoint requires the bearer token. Health and readiness endpoints are unauthenticated for orchestrator probes. @@ -79,12 +97,12 @@ readiness endpoints are unauthenticated for orchestrator probes. | Variable | Required | Default | Description | | --- | --- | --- | --- | -| `ABYSS_BACKEND_DATABASE_URL` | yes | — | PostgreSQL connection URL. | +| `ABYSS_BACKEND_DATABASE_URL` | yes | — | PostgreSQL URL or SQLite file path, depending on the compiled profile. | | `ABYSS_BACKEND_API_TOKEN_SHA256` | yes | — | Lowercase, 64-character SHA-256 bearer-token digest. | | `ABYSS_BACKEND_ADDR` | no | `0.0.0.0:8080` | HTTP listen address. | | `ABYSS_BACKEND_ENV` | no | `local` | Environment label reported at `/`. | | `ABYSS_BACKEND_LOG_LEVEL` | no | `info` | Default tracing filter. | -| `ABYSS_BACKEND_DATABASE_POOL_SIZE` | no | `10` | PostgreSQL pool size. | +| `ABYSS_BACKEND_DATABASE_POOL_SIZE` | no | `10` | Selected database pool size. | | `ABYSS_BACKEND_RUN_MIGRATIONS` | no | `true` | Run embedded migrations during startup. | | `ABYSS_BACKEND_MAX_INGEST_BATCH_SIZE` | no | `1000` | Maximum event count per ingest request. | | `ABYSS_BACKEND_SUMMARY_SCAN_LIMIT` | no | `100000` | Maximum source rows scanned by a summary query. | @@ -96,9 +114,10 @@ readiness endpoints are unauthenticated for orchestrator probes. | `ABYSS_BACKEND_SEARCH_POLL_INTERVAL_MILLISECONDS` | no | `500` | Search outbox polling interval. | | `ABYSS_BACKEND_SEARCH_BATCH_SIZE` | no | `100` | Search outbox batch size. | -Elasticsearch username and password must be provided together. Search remains -disabled when no URL is configured; event storage and queries continue to -work. +Elasticsearch settings exist only in `postgres-es` builds. Username and +password must be provided together. Search remains disabled in that profile +when no URL is configured; event storage and queries continue to work. The +`sqlite-fts` profile always provides search through the local FTS5 index. ## Containers and Kubernetes @@ -221,11 +240,13 @@ the example `:latest` reference for production promotion. ```bash make check make test-blackbox +make test-blackbox-sqlite make docker-build make k8s-render ``` -The black-box test starts an ephemeral PostgreSQL container and an -Elasticsearch contract double, then verifies the consolidated schema, -authentication, event ingestion, attachment retrieval, raw diagnostic -retention, summary, timeline, and search behavior. +`make test-blackbox` runs the same API contract against both storage profiles. +The PostgreSQL case starts an ephemeral container and an Elasticsearch contract +double; the SQLite case uses only a temporary local database file. Both verify +schema creation, authentication, event ingestion, attachment retrieval, raw +diagnostic retention, summary, timeline, and search behavior. diff --git a/crates/abyss-backend/Cargo.toml b/crates/abyss-backend/Cargo.toml index fb8f2cb..e926122 100644 --- a/crates/abyss-backend/Cargo.toml +++ b/crates/abyss-backend/Cargo.toml @@ -8,18 +8,38 @@ repository.workspace = true publish = false [features] -blackbox-bundled-pq = ["dep:pq-sys"] +default = ["postgres-es"] +blackbox-bundled-pq = ["postgres-es", "dep:pq-sys"] +postgres-es = [ + "dep:diesel", + "diesel/chrono", + "diesel/postgres", + "diesel/serde_json", + "diesel/uuid", + "dep:diesel_migrations", + "diesel_migrations/postgres", + "dep:reqwest", +] +sqlite-fts = [ + "dep:diesel", + "diesel/sqlite", + "diesel/returning_clauses_for_sqlite_3_35", + "dep:diesel_migrations", + "diesel_migrations/sqlite", + "dep:libsqlite3-sys", +] [dependencies] axum = "0.8" base64 = "0.21" chrono = { version = "0.4", features = ["serde"] } -diesel = { version = "2.2", features = ["chrono", "postgres", "r2d2", "serde_json", "uuid"] } -diesel_migrations = { version = "2.2", features = ["postgres"] } +diesel = { version = "2.2", default-features = false, features = ["32-column-tables", "r2d2", "with-deprecated"], optional = true } +diesel_migrations = { version = "2.2", default-features = false, optional = true } hex = "0.4" +libsqlite3-sys = { version = "0.35", features = ["bundled"], optional = true } pq-sys = { version = "0.7", features = ["bundled_without_openssl"], optional = true } r2d2 = "0.8" -reqwest = { version = "0.12", default-features = false, features = ["json", "rustls-tls"] } +reqwest = { version = "0.12", default-features = false, features = ["json", "rustls-tls"], optional = true } serde = { version = "1", features = ["derive"] } serde_json = "1" sha2 = "0.10" @@ -31,8 +51,5 @@ tracing = "0.1" tracing-subscriber = { version = "0.3", features = ["env-filter"] } uuid = { version = "1", features = ["serde", "v7"] } -[dev-dependencies] -pq-sys = { version = "0.7", features = ["bundled_without_openssl"] } - [lints] workspace = true diff --git a/crates/abyss-backend/migrations-sqlite/20260827000000_initial_schema/down.sql b/crates/abyss-backend/migrations-sqlite/20260827000000_initial_schema/down.sql new file mode 100644 index 0000000..6c08204 --- /dev/null +++ b/crates/abyss-backend/migrations-sqlite/20260827000000_initial_schema/down.sql @@ -0,0 +1,9 @@ +DROP TABLE usage_events_fts; +DROP TABLE agent_diagnostic_capture_events; +DROP TABLE agent_diagnostic_captures; +DROP TABLE llm_usage_event_attachments; +DROP TABLE llm_usage_events; +DROP TABLE agent_turns; +DROP TABLE agent_sessions; +DROP TABLE devices; +DROP TABLE app_users; diff --git a/crates/abyss-backend/migrations-sqlite/20260827000000_initial_schema/up.sql b/crates/abyss-backend/migrations-sqlite/20260827000000_initial_schema/up.sql new file mode 100644 index 0000000..cafdfbf --- /dev/null +++ b/crates/abyss-backend/migrations-sqlite/20260827000000_initial_schema/up.sql @@ -0,0 +1,160 @@ +CREATE TABLE app_users ( + id TEXT PRIMARY KEY, + email TEXT NOT NULL, + name TEXT, + created_at INTEGER NOT NULL +) STRICT; + +INSERT INTO app_users (id, email, name, created_at) +VALUES ( + '00000000-0000-0000-0000-000000000001', + 'owner@abyss.local', + 'Abyss Owner', + CAST(unixepoch('subsec') * 1000000 AS INTEGER) +); + +CREATE TABLE devices ( + id TEXT PRIMARY KEY, + user_id TEXT NOT NULL REFERENCES app_users(id) ON DELETE CASCADE, + host_name TEXT NOT NULL, + platform TEXT NOT NULL, + os_version TEXT, + first_seen_at INTEGER NOT NULL, + last_seen_at INTEGER NOT NULL, + created_at INTEGER NOT NULL, + updated_at INTEGER NOT NULL, + UNIQUE (user_id, host_name, platform) +) STRICT; + +CREATE INDEX devices_user_seen_idx + ON devices (user_id, last_seen_at DESC); + +CREATE TABLE agent_sessions ( + id TEXT PRIMARY KEY, + user_id TEXT NOT NULL REFERENCES app_users(id) ON DELETE CASCADE, + device_context_id TEXT NOT NULL REFERENCES devices(id), + agent_name TEXT NOT NULL, + agent_version TEXT, + session_id TEXT NOT NULL, + started_at INTEGER NOT NULL, + ended_at INTEGER, + metadata TEXT NOT NULL DEFAULT '{}' CHECK (json_valid(metadata)), + created_at INTEGER NOT NULL, + updated_at INTEGER NOT NULL, + UNIQUE (user_id, agent_name, session_id) +) STRICT; + +CREATE INDEX agent_sessions_user_time_idx + ON agent_sessions (user_id, started_at DESC); + +CREATE TABLE agent_turns ( + id TEXT PRIMARY KEY, + user_id TEXT NOT NULL REFERENCES app_users(id) ON DELETE CASCADE, + session_pk TEXT NOT NULL REFERENCES agent_sessions(id) ON DELETE CASCADE, + turn_index INTEGER NOT NULL CHECK (turn_index >= 1), + started_at INTEGER NOT NULL, + ended_at INTEGER, + created_at INTEGER NOT NULL, + updated_at INTEGER NOT NULL, + UNIQUE (session_pk, turn_index) +) STRICT; + +CREATE INDEX agent_turns_user_session_idx + ON agent_turns (user_id, session_pk, turn_index); + +CREATE TABLE llm_usage_events ( + id TEXT PRIMARY KEY, + user_id TEXT NOT NULL REFERENCES app_users(id) ON DELETE CASCADE, + device_context_id TEXT NOT NULL REFERENCES devices(id), + session_pk TEXT NOT NULL REFERENCES agent_sessions(id) ON DELETE CASCADE, + turn_pk TEXT NOT NULL REFERENCES agent_turns(id) ON DELETE CASCADE, + event_id TEXT NOT NULL UNIQUE, + agent_name TEXT NOT NULL, + agent_version TEXT, + session_id TEXT NOT NULL, + turn_index INTEGER NOT NULL CHECK (turn_index >= 1), + llm_provider TEXT NOT NULL, + llm_model TEXT NOT NULL, + event_type TEXT NOT NULL CHECK (event_type IN ('request', 'response')), + text TEXT, + text_sha256 BLOB, + input_tokens INTEGER NOT NULL DEFAULT 0 CHECK (input_tokens >= 0), + output_tokens INTEGER NOT NULL DEFAULT 0 CHECK (output_tokens >= 0), + cache_read_tokens INTEGER NOT NULL DEFAULT 0 CHECK (cache_read_tokens >= 0), + cache_write_tokens INTEGER NOT NULL DEFAULT 0 CHECK (cache_write_tokens >= 0), + reasoning_tokens INTEGER NOT NULL DEFAULT 0 CHECK (reasoning_tokens >= 0), + total_tokens INTEGER NOT NULL DEFAULT 0 CHECK (total_tokens >= 0), + observed_at INTEGER NOT NULL, + metadata TEXT NOT NULL DEFAULT '{}' CHECK (json_valid(metadata)), + created_at INTEGER NOT NULL +) STRICT; + +CREATE INDEX llm_usage_events_user_time_idx + ON llm_usage_events (user_id, observed_at DESC, id DESC); + +CREATE INDEX llm_usage_events_session_turn_idx + ON llm_usage_events (session_pk, turn_index, observed_at, id); + +CREATE INDEX llm_usage_events_agent_model_time_idx + ON llm_usage_events (agent_name, llm_provider, llm_model, observed_at DESC); + +CREATE TABLE llm_usage_event_attachments ( + id TEXT PRIMARY KEY, + user_id TEXT NOT NULL REFERENCES app_users(id) ON DELETE CASCADE, + event_pk TEXT NOT NULL REFERENCES llm_usage_events(id) ON DELETE CASCADE, + position INTEGER NOT NULL CHECK (position >= 0), + media_type TEXT NOT NULL CHECK ( + media_type IN ('image/png', 'image/jpeg', 'image/webp', 'image/gif') + ), + byte_size INTEGER NOT NULL CHECK (byte_size > 0 AND byte_size <= 8388608), + sha256 BLOB NOT NULL CHECK (length(sha256) = 32), + content BLOB, + created_at INTEGER NOT NULL, + UNIQUE (event_pk, position), + CHECK (content IS NULL OR length(content) = byte_size) +) STRICT; + +CREATE INDEX llm_usage_event_attachments_user_event_idx + ON llm_usage_event_attachments (user_id, event_pk, position); + +CREATE TABLE agent_diagnostic_captures ( + id TEXT PRIMARY KEY, + user_id TEXT NOT NULL REFERENCES app_users(id) ON DELETE CASCADE, + device_context_id TEXT NOT NULL REFERENCES devices(id) ON DELETE CASCADE, + session_pk TEXT NOT NULL REFERENCES agent_sessions(id) ON DELETE CASCADE, + capture_id TEXT NOT NULL, + flow_id TEXT NOT NULL, + captured_at INTEGER NOT NULL, + collector_version TEXT NOT NULL, + payload TEXT NOT NULL CHECK (json_valid(payload)), + created_at INTEGER NOT NULL, + UNIQUE (user_id, capture_id) +) STRICT; + +CREATE INDEX agent_diagnostic_captures_session_time_idx + ON agent_diagnostic_captures (session_pk, captured_at, id); + +CREATE INDEX agent_diagnostic_captures_flow_idx + ON agent_diagnostic_captures (user_id, flow_id, captured_at); + +CREATE TABLE agent_diagnostic_capture_events ( + capture_pk TEXT NOT NULL REFERENCES agent_diagnostic_captures(id) ON DELETE CASCADE, + event_pk TEXT NOT NULL REFERENCES llm_usage_events(id) ON DELETE CASCADE, + PRIMARY KEY (capture_pk, event_pk) +) STRICT; + +CREATE INDEX agent_diagnostic_capture_events_event_idx + ON agent_diagnostic_capture_events (event_pk); + +CREATE VIRTUAL TABLE usage_events_fts USING fts5( + event_pk UNINDEXED, + user_id UNINDEXED, + session_pk UNINDEXED, + session_id, + content, + tool_names, + tool_content, + commands, + file_paths, + tokenize = 'unicode61' +); diff --git a/crates/abyss-backend/src/api.rs b/crates/abyss-backend/src/api.rs index 0eb73d5..35a6629 100644 --- a/crates/abyss-backend/src/api.rs +++ b/crates/abyss-backend/src/api.rs @@ -2,8 +2,7 @@ //! //! The health endpoints are public, while every event, attachment, summary, //! timeline, and search endpoint authenticates the deployment bearer token. -//! Diesel is synchronous, so all database access is moved onto Tokio's blocking -//! pool through [`run_db`] rather than running on asynchronous executor threads. +//! Storage implementation details remain behind the configured backend trait. use axum::{ Json, Router, @@ -13,23 +12,18 @@ use axum::{ response::{IntoResponse, Response}, routing::{get, post}, }; -use diesel::PgConnection; use serde::Serialize; -use tokio::task; use tower_http::trace::TraceLayer; use uuid::Uuid; use crate::{ - db::{self, DbPool}, error::AppError, identity::IdentityAuthenticator, - search::{ - SearchService, SessionSearchQuery, SessionSearchResponse, outbox::SearchOutboxRepository, - }, + search::{SessionSearchQuery, SessionSearchResponse}, + storage::StorageBackend, usage::{ IngestEventsRequest, IngestEventsResponse, RawEventsQuery, RawEventsResponse, SessionTimelineResponse, SummaryFields, SummaryQuery, TokenUsageSummaryResponse, - repository as usage_repository, }, }; @@ -48,10 +42,8 @@ pub struct AppState { pub default_page_size: i64, /// Deployment-wide bearer-token validator. pub identity: IdentityAuthenticator, - /// Optional full-text search service. - pub search: Option, - /// PostgreSQL connection pool used by request handlers. - pub pool: DbPool, + /// Compile-time-selected event storage and full-text search implementation. + pub storage: std::sync::Arc, } /// Builds the complete HTTP router for one backend process. @@ -99,9 +91,10 @@ async fn health() -> Json { } async fn ready(State(state): State) -> Result, AppError> { - // Readiness checks the source of truth only. Search is an optional derived - // service and its temporary failure must not remove ingestion capacity. - run_db(state, db::check_ready).await?; + // Readiness checks the authoritative database. In the PostgreSQL profile, + // a temporary failure of the optional search projection must not remove + // ingestion capacity. + state.storage.ready().await?; Ok(Json(ServiceStatus { service: "abyss-backend", status: "ok", @@ -115,10 +108,10 @@ async fn ingest_events( ) -> Result, AppError> { let user_id = state.identity.authenticate(&headers)?; let max_batch_size = state.max_ingest_batch_size; - let response = run_db(state, move |connection| { - usage_repository::ingest_events(connection, &request, user_id, max_batch_size) - }) - .await?; + let response = state + .storage + .ingest_events(user_id, request, max_batch_size) + .await?; Ok(Json(response)) } @@ -130,10 +123,10 @@ async fn usage_summary( let user_id = state.identity.authenticate(&headers)?; let scan_limit = state.summary_scan_limit; let fields = query.fields.unwrap_or(SummaryFields::Full); - let response = run_db(state, move |connection| { - usage_repository::usage_summary(connection, &query, user_id, scan_limit) - }) - .await?; + let response = state + .storage + .usage_summary(user_id, query, scan_limit) + .await?; if fields == SummaryFields::TokenUsage { return Ok(Json(TokenUsageSummaryResponse::from_summary(response)).into_response()); } @@ -146,19 +139,7 @@ async fn session_search( Query(query): Query, ) -> Result, AppError> { let user_id = state.identity.authenticate(&headers)?; - let search = state - .search - .clone() - .ok_or_else(|| AppError::unavailable("session search is not configured".to_owned()))?; - let execution = search.search(user_id, query).await?; - let session_ids = execution.session_ids(); - // Elasticsearch contains only a bounded search projection. Authoritative - // session/device details are reloaded from PostgreSQL under the owner scope. - let details = run_db(state, move |connection| { - SearchOutboxRepository::session_details(connection, user_id, &session_ids) - }) - .await?; - Ok(Json(execution.hydrate(details))) + Ok(Json(state.storage.session_search(user_id, query).await?)) } async fn session_timeline( @@ -167,10 +148,7 @@ async fn session_timeline( Path(session_pk): Path, ) -> Result, AppError> { let user_id = state.identity.authenticate(&headers)?; - let response = run_db(state, move |connection| { - usage_repository::session_timeline(connection, user_id, session_pk) - }) - .await?; + let response = state.storage.session_timeline(user_id, session_pk).await?; Ok(Json(response)) } @@ -180,10 +158,10 @@ async fn image_attachment( Path(attachment_id): Path, ) -> Result { let user_id = state.identity.authenticate(&headers)?; - let attachment = run_db(state, move |connection| { - usage_repository::image_attachment(connection, user_id, attachment_id) - }) - .await?; + let attachment = state + .storage + .image_attachment(user_id, attachment_id) + .await?; let mut response = Bytes::from(attachment.content).into_response(); let response_headers = response.headers_mut(); @@ -230,28 +208,13 @@ async fn raw_events( ) -> Result, AppError> { let user_id = state.identity.authenticate(&headers)?; let default_page_size = state.default_page_size; - let response = run_db(state, move |connection| { - usage_repository::raw_events(connection, &query, user_id, default_page_size) - }) - .await?; + let response = state + .storage + .raw_events(user_id, query, default_page_size) + .await?; Ok(Json(response)) } -async fn run_db(state: AppState, task_fn: F) -> Result -where - T: Send + 'static, - F: FnOnce(&mut PgConnection) -> Result + Send + 'static, -{ - // Diesel and r2d2 are blocking APIs. Acquiring the pool connection inside - // spawn_blocking also keeps pool contention off Tokio worker threads. - task::spawn_blocking(move || { - let mut connection = state.pool.get()?; - task_fn(&mut connection) - }) - .await - .map_err(|error| AppError::internal(format!("database task failed: {error}")))? -} - #[derive(Serialize)] struct RootResponse { service: &'static str, diff --git a/crates/abyss-backend/src/config.rs b/crates/abyss-backend/src/config.rs index ea920dc..8c6d25c 100644 --- a/crates/abyss-backend/src/config.rs +++ b/crates/abyss-backend/src/config.rs @@ -16,8 +16,11 @@ const DEFAULT_POOL_SIZE: u32 = 10; const DEFAULT_MAX_INGEST_BATCH_SIZE: usize = 1_000; const DEFAULT_SUMMARY_SCAN_LIMIT: i64 = 100_000; const DEFAULT_PAGE_SIZE: i64 = 100; +#[cfg(feature = "postgres-es")] const DEFAULT_SEARCH_REQUEST_TIMEOUT_SECONDS: u64 = 10; +#[cfg(feature = "postgres-es")] const DEFAULT_SEARCH_POLL_INTERVAL_MILLISECONDS: u64 = 500; +#[cfg(feature = "postgres-es")] const DEFAULT_SEARCH_BATCH_SIZE: i64 = 100; /// Complete runtime configuration shared by process startup and HTTP state. @@ -30,11 +33,11 @@ pub struct Config { pub blackbox_allow_non_loopback: bool, /// Default tracing filter used when `RUST_LOG` is not set. pub log_level: String, - /// PostgreSQL connection string. + /// PostgreSQL URL or SQLite file path selected by the compiled backend. pub database_url: String, - /// Maximum number of PostgreSQL connections held by the r2d2 pool. + /// Maximum number of database connections held by the backend pool. pub database_pool_size: u32, - /// Whether embedded Diesel migrations run before the listener starts. + /// Whether embedded backend migrations run before the listener starts. pub run_migrations: bool, /// Maximum events and diagnostic captures accepted in one ingest request. pub max_ingest_batch_size: usize, @@ -45,6 +48,7 @@ pub struct Config { /// Deployment-wide bearer-token authentication configuration. pub identity: IdentityConfig, /// Optional Elasticsearch projection configuration. + #[cfg(feature = "postgres-es")] pub search: Option, } @@ -86,12 +90,14 @@ impl Config { DEFAULT_PAGE_SIZE, )?, identity: IdentityConfig::parse(&read_required_env("ABYSS_BACKEND_API_TOKEN_SHA256")?)?, + #[cfg(feature = "postgres-es")] search: SearchConfig::from_env()?, }) } } #[derive(Clone)] +#[cfg(feature = "postgres-es")] /// Settings required by the Elasticsearch client and projection worker. pub struct SearchConfig { /// Base Elasticsearch URL, without an index or API path suffix. @@ -108,6 +114,7 @@ pub struct SearchConfig { pub batch_size: i64, } +#[cfg(feature = "postgres-es")] impl SearchConfig { fn from_env() -> Result, AppError> { let endpoint = env_value("ABYSS_BACKEND_ELASTICSEARCH_URL"); @@ -189,6 +196,7 @@ fn read_positive_u32_env(key: &str, fallback: u32) -> Result { require_positive(key, value) } +#[cfg(feature = "postgres-es")] fn read_positive_u64_env(key: &str, fallback: u64) -> Result { let value = parse_env(key, fallback, str::parse::)?; require_positive(key, value) diff --git a/crates/abyss-backend/src/error.rs b/crates/abyss-backend/src/error.rs index 9784337..85ba807 100644 --- a/crates/abyss-backend/src/error.rs +++ b/crates/abyss-backend/src/error.rs @@ -20,7 +20,7 @@ pub enum AppError { /// Diesel query or transaction failure. #[error("database error: {0}")] Database(#[from] diesel::result::Error), - /// PostgreSQL connection-pool failure. + /// Database connection-pool failure. #[error("connection pool error: {0}")] Pool(#[from] r2d2::Error), /// Authenticated resource does not exist. @@ -30,6 +30,7 @@ pub enum AppError { #[error("unauthorized: {0}")] Unauthorized(String), /// Optional dependency is disabled or temporarily unavailable. + #[cfg(feature = "postgres-es")] #[error("unavailable: {0}")] Unavailable(String), /// Request data violates an API contract. @@ -62,6 +63,7 @@ impl AppError { } /// Constructs a dependency-unavailable error. + #[cfg(feature = "postgres-es")] pub const fn unavailable(message: String) -> Self { Self::Unavailable(message) } @@ -75,11 +77,13 @@ impl AppError { match self { Self::Validation(_) => StatusCode::BAD_REQUEST, Self::Unauthorized(_) => StatusCode::UNAUTHORIZED, + #[cfg(feature = "postgres-es")] Self::Unavailable(_) => StatusCode::SERVICE_UNAVAILABLE, Self::NotFound(_) => StatusCode::NOT_FOUND, - Self::Config(_) | Self::Database(_) | Self::Pool(_) | Self::Internal(_) => { + Self::Config(_) | Self::Pool(_) | Self::Internal(_) => { StatusCode::INTERNAL_SERVER_ERROR } + Self::Database(_) => StatusCode::INTERNAL_SERVER_ERROR, } } } @@ -88,14 +92,17 @@ impl IntoResponse for AppError { fn into_response(self) -> Response { let status = self.status_code(); let error = match &self { - Self::Config(_) | Self::Database(_) | Self::Pool(_) | Self::Internal(_) => { + Self::Config(_) | Self::Pool(_) | Self::Internal(_) => { tracing::error!(error = %self, "internal error"); "internal server error".to_owned() } - Self::NotFound(_) - | Self::Unauthorized(_) - | Self::Unavailable(_) - | Self::Validation(_) => self.to_string(), + Self::Database(_) => { + tracing::error!(error = %self, "internal error"); + "internal server error".to_owned() + } + Self::NotFound(_) | Self::Unauthorized(_) | Self::Validation(_) => self.to_string(), + #[cfg(feature = "postgres-es")] + Self::Unavailable(_) => self.to_string(), }; let body = ErrorResponse { error }; (status, Json(body)).into_response() diff --git a/crates/abyss-backend/src/main.rs b/crates/abyss-backend/src/main.rs index b643572..a8e24ba 100644 --- a/crates/abyss-backend/src/main.rs +++ b/crates/abyss-backend/src/main.rs @@ -2,9 +2,7 @@ //! //! Startup is deliberately ordered: configuration is validated before any //! external connection is opened, database migrations finish before the HTTP -//! listener accepts traffic, and the optional search worker shares the same -//! shutdown signal as the server. PostgreSQL remains the source of truth; -//! Elasticsearch is only a derived, eventually consistent projection. +//! listener accepts traffic, and backend-owned workers stop after the server. #![warn(missing_docs)] #![expect( @@ -14,16 +12,23 @@ mod api; mod config; +#[cfg(feature = "postgres-es")] mod db; mod error; mod identity; mod search; +mod storage; mod usage; -use std::{error::Error, time::Duration}; +#[cfg(all(feature = "postgres-es", feature = "sqlite-fts"))] +compile_error!("features postgres-es and sqlite-fts are mutually exclusive"); + +#[cfg(not(any(feature = "postgres-es", feature = "sqlite-fts")))] +compile_error!("one of features postgres-es or sqlite-fts must be enabled"); + +use std::error::Error; use config::Config; -use db::{create_pool, run_migrations}; use identity::IdentityAuthenticator; use tracing_subscriber::EnvFilter; @@ -33,28 +38,7 @@ async fn main() -> Result<(), Box> { validate_runtime_config(&config)?; init_tracing(&config); - let pool = create_pool(&config)?; - if config.run_migrations { - run_migrations(&pool)?; - } - - // Search is optional. Keeping the service itself in an Option makes the - // disabled state explicit all the way through routing and worker startup. - let search = config - .search - .as_ref() - .map(search::SearchService::new) - .transpose()?; - let (shutdown_sender, shutdown_receiver) = tokio::sync::watch::channel(false); - let search_worker = match (&search, config.search.as_ref()) { - (Some(search), Some(search_config)) => Some(search::worker::SearchIndexer::spawn( - pool.clone(), - search.client(), - search_config, - shutdown_receiver, - )), - _ => None, - }; + let storage = storage::build(&config)?; let address = config.addr; let app = api::router(api::AppState { @@ -63,8 +47,7 @@ async fn main() -> Result<(), Box> { summary_scan_limit: config.summary_scan_limit, default_page_size: config.default_page_size, identity: IdentityAuthenticator::new(config.identity), - search, - pool, + storage: storage.clone(), }); let listener = tokio::net::TcpListener::bind(&address).await?; @@ -73,33 +56,14 @@ async fn main() -> Result<(), Box> { listener, app.into_make_service_with_connect_info::(), ) - .with_graceful_shutdown(async move { - shutdown_signal().await; - if shutdown_sender.send(true).is_err() { - tracing::trace!("session search indexer already stopped"); - } - }) + .with_graceful_shutdown(shutdown_signal()) .await; - stop_search_worker(search_worker).await; + storage.shutdown().await; server_result?; Ok(()) } -async fn stop_search_worker(worker: Option>) { - let Some(mut worker) = worker else { - return; - }; - match tokio::time::timeout(Duration::from_secs(15), &mut worker).await { - Ok(Ok(())) => {} - Ok(Err(error)) => tracing::warn!(%error, "session search indexer task failed"), - Err(_elapsed) => { - tracing::warn!("session search indexer did not stop before timeout"); - worker.abort(); - } - } -} - fn validate_runtime_config(config: &Config) -> Result<(), error::AppError> { // Black-box tests use disposable credentials and data. Requiring loopback // prevents an accidentally started test instance from becoming reachable @@ -180,6 +144,7 @@ mod tests { summary_scan_limit: 1, default_page_size: 1, identity: IdentityConfig::parse(&"0".repeat(64)).expect("test token hash should parse"), + #[cfg(feature = "postgres-es")] search: None, } } diff --git a/crates/abyss-backend/src/search/mod.rs b/crates/abyss-backend/src/search/mod.rs index 26ede7a..c2e9164 100644 --- a/crates/abyss-backend/src/search/mod.rs +++ b/crates/abyss-backend/src/search/mod.rs @@ -1,29 +1,21 @@ -//! Traditional full-text session search backed by a derived Elasticsearch index. +//! Backend-independent full-text search contracts and safe source projection. //! -//! Search results are grouped by session in Elasticsearch, then hydrated with -//! authoritative session and device rows from PostgreSQL. The index never acts -//! as an authorization source: every query includes the authenticated owner and -//! missing PostgreSQL details cause a stale search hit to be omitted. +//! Concrete search engines live inside their selected storage backend. This +//! module defines only the query, response, highlight, and bounded projection +//! types shared by those implementations and the HTTP layer. -mod document; -mod elasticsearch; -/// PostgreSQL outbox leasing and hydration queries. -pub mod outbox; -/// Background outbox-to-Elasticsearch projection worker. -pub mod worker; - -use std::collections::{HashMap, HashSet}; +pub mod projection; use chrono::{DateTime, Utc}; use serde::{Deserialize, Serialize}; use uuid::Uuid; -use crate::{config::SearchConfig, error::AppError}; +use crate::error::AppError; -use self::{ - elasticsearch::{ElasticsearchClient, HIGHLIGHT_END, HIGHLIGHT_START, SearchMatchPage}, - outbox::SearchSessionDetails, -}; +/// Sentinel inserted before text matched by either search implementation. +pub const HIGHLIGHT_START: &str = "[[[ABYSS_SEARCH_HIGHLIGHT_START]]]"; +/// Sentinel inserted after text matched by either search implementation. +pub const HIGHLIGHT_END: &str = "[[[ABYSS_SEARCH_HIGHLIGHT_END]]]"; const DEFAULT_PAGE_SIZE: u32 = 20; const MAX_PAGE_SIZE: u32 = 50; @@ -31,41 +23,6 @@ const MAX_QUERY_CHARACTERS: usize = 256; const MAX_FILTER_CHARACTERS: usize = 256; const MAX_RESULT_WINDOW: u32 = 10_000; -#[derive(Clone)] -/// Validating facade over the Elasticsearch HTTP client. -pub struct SearchService { - client: ElasticsearchClient, -} - -impl SearchService { - /// Creates a search service from startup-validated settings. - pub fn new(config: &SearchConfig) -> Result { - Ok(Self { - client: ElasticsearchClient::new(config)?, - }) - } - - /// Returns a cloned client for the background indexer. - #[must_use] - pub fn client(&self) -> ElasticsearchClient { - self.client.clone() - } - - /// Validates and executes one owner-scoped session search. - pub async fn search( - &self, - user_id: Uuid, - query: SessionSearchQuery, - ) -> Result { - let query = query.validate()?; - let page = self.client.search(user_id, &query).await.map_err(|error| { - tracing::warn!(%error, %user_id, "session search request failed"); - AppError::unavailable("session search is temporarily unavailable".to_owned()) - })?; - Ok(SearchExecution { query, page }) - } -} - #[derive(Debug, Deserialize)] /// Query-string contract for session full-text search. pub struct SessionSearchQuery { @@ -90,7 +47,7 @@ pub struct SessionSearchQuery { } impl SessionSearchQuery { - fn validate(self) -> Result { + pub(crate) fn validate(self) -> Result { let text = normalized_required(&self.q, "q", MAX_QUERY_CHARACTERS)?; if let (Some(from), Some(to)) = (self.from, self.to) && from >= to @@ -109,7 +66,7 @@ impl SessionSearchQuery { "page_size must be between 1 and {MAX_PAGE_SIZE}" ))); } - // Elasticsearch's from/size pagination has a bounded result window. + // Both search implementations expose the same bounded result window. // Checked arithmetic makes oversized user input a validation error. let offset = page .saturating_sub(1) @@ -149,7 +106,7 @@ impl SessionSearchQuery { } } -/// Normalized query safe to translate directly into Elasticsearch JSON. +/// Normalized query safe to translate into a backend-specific search request. pub struct ValidatedSearchQuery { /// Trimmed full-text query. pub text: String, @@ -169,109 +126,16 @@ pub struct ValidatedSearchQuery { pub page: u32, /// Sessions returned per page. pub page_size: u32, - /// Zero-based Elasticsearch result offset. + /// Zero-based search result offset. pub offset: u32, } -/// Elasticsearch matches paired with the validated query that produced them. -pub struct SearchExecution { - query: ValidatedSearchQuery, - page: SearchMatchPage, -} - -impl SearchExecution { - /// Returns session keys that must be hydrated from PostgreSQL. - #[must_use] - pub fn session_ids(&self) -> Vec { - self.page - .sessions - .iter() - .map(|session| session.session_pk) - .collect() - } - - /// Combines search matches with authoritative session/device details. - /// - /// Stale index entries whose session no longer exists are omitted instead - /// of returning partially authorized or incomplete data. - #[must_use] - pub fn hydrate( - self, - mut details: HashMap, - ) -> SessionSearchResponse { - let items = self - .page - .sessions - .into_iter() - .filter_map(|matches| { - let details = details.remove(&matches.session_pk)?; - let mut models = matches - .events - .iter() - .map(|event| event.llm_model.clone()) - .collect::>() - .into_iter() - .collect::>(); - models.sort(); - let mut providers = matches - .events - .iter() - .map(|event| event.llm_provider.clone()) - .collect::>() - .into_iter() - .collect::>(); - providers.sort(); - Some(SessionSearchResult { - session_pk: details.session.id, - session_id: details.session.session_id, - agent_name: details.session.agent_name, - agent_version: details.session.agent_version, - host_name: details.device.host_name, - platform: details.device.platform, - started_at: details.session.started_at, - ended_at: details.session.ended_at, - providers, - models, - match_count: matches.match_count, - matches: matches - .events - .into_iter() - .map(|event| SessionSearchMatch { - event_pk: event.event_pk, - turn_pk: event.turn_pk, - turn_index: event.turn_index, - event_type: event.event_type, - llm_provider: event.llm_provider, - llm_model: event.llm_model, - observed_at: event.observed_at, - fragments: event - .fragments - .into_iter() - .map(|fragment| SearchFragment::parse(&fragment)) - .collect(), - }) - .collect(), - }) - }) - .collect::>(); - let consumed = u64::from(self.query.page).saturating_mul(u64::from(self.query.page_size)); - SessionSearchResponse { - query: self.query.text, - total_sessions: self.page.total_sessions, - page: self.query.page, - page_size: self.query.page_size, - has_more: consumed < self.page.total_sessions, - items, - } - } -} - #[derive(Serialize)] /// Paginated session search response. pub struct SessionSearchResponse { /// Normalized full-text query. pub query: String, - /// Approximate distinct session count returned by Elasticsearch cardinality. + /// Distinct matching-session count reported by the selected backend. pub total_sessions: u64, /// One-based current page. pub page: u32, @@ -334,14 +198,14 @@ pub struct SessionSearchMatch { } #[derive(Serialize)] -/// One Elasticsearch highlight fragment split into plain and matching text. +/// One search highlight fragment split into plain and matching text. pub struct SearchFragment { /// Ordered text segments suitable for structured UI rendering. pub segments: Vec, } impl SearchFragment { - fn parse(fragment: &str) -> Self { + pub(crate) fn parse(fragment: &str) -> Self { let mut segments = Vec::new(); let mut remainder = fragment; while let Some(start) = remainder.find(HIGHLIGHT_START) { diff --git a/crates/abyss-backend/src/search/projection.rs b/crates/abyss-backend/src/search/projection.rs new file mode 100644 index 0000000..4bba669 --- /dev/null +++ b/crates/abyss-backend/src/search/projection.rs @@ -0,0 +1,143 @@ +//! Bounded, allowlisted extraction shared by every full-text implementation. + +use serde_json::Value; + +use super::{HIGHLIGHT_END, HIGHLIGHT_START}; + +pub const MAX_CONTENT_CHARACTERS: usize = 1_000_000; +const MAX_METADATA_FIELD_CHARACTERS: usize = 100_000; +const MAX_EXTRACTED_VALUES: usize = 64; +const MAX_JSON_DEPTH: usize = 8; + +/// Searchable values derived from one source usage event. +pub struct SearchProjection { + pub content: Option, + pub tool_names: Vec, + pub tool_content: Vec, + pub commands: Vec, + pub file_paths: Vec, +} + +impl SearchProjection { + pub fn from_source(text: Option, metadata: &Value) -> Self { + let extracted = SearchableMetadata::extract(metadata); + Self { + content: text + .map(|value| sanitize_search_text(&value)) + .map(|value| truncate_characters(value, MAX_CONTENT_CHARACTERS)), + tool_names: extracted.tool_names, + tool_content: extracted.tool_content, + commands: extracted.commands, + file_paths: extracted.file_paths, + } + } +} + +struct SearchableMetadata { + tool_names: Vec, + tool_content: Vec, + commands: Vec, + file_paths: Vec, +} + +impl SearchableMetadata { + fn extract(metadata: &Value) -> Self { + let mut extracted = Self { + tool_names: Vec::new(), + tool_content: Vec::new(), + commands: Vec::new(), + file_paths: Vec::new(), + }; + + if let Some(working_directory) = metadata.get("working_directory").and_then(Value::as_str) { + extracted.push_file_path(working_directory); + } + + // Ignore every other top-level key so headers, credentials, and image + // data never enter a derived full-text index. + let Some(segments) = metadata.get("content_segments").and_then(Value::as_array) else { + return extracted; + }; + for segment in segments.iter().take(MAX_EXTRACTED_VALUES) { + let Some(object) = segment.as_object() else { + continue; + }; + match object.get("type").and_then(Value::as_str) { + Some("tool_call") => { + if let Some(name) = object.get("name").and_then(Value::as_str) { + push_bounded(&mut extracted.tool_names, name); + } + if let Some(input) = object.get("input").and_then(Value::as_str) { + push_bounded(&mut extracted.tool_content, input); + if let Ok(value) = serde_json::from_str::(input) { + extracted.extract_structured_input(&value, MAX_JSON_DEPTH); + } + } + } + Some("tool_result") => { + if let Some(output) = object.get("output").and_then(Value::as_str) { + push_bounded(&mut extracted.tool_content, output); + } + } + _ => {} + } + } + extracted + } + + fn extract_structured_input(&mut self, value: &Value, remaining_depth: usize) { + if remaining_depth == 0 { + return; + } + match value { + Value::Object(object) => { + for (key, child) in object.iter().take(MAX_EXTRACTED_VALUES) { + if let Some(text) = child.as_str() { + match key.as_str() { + "command" | "cmd" => push_bounded(&mut self.commands, text), + "file" | "file_path" | "path" => self.push_file_path(text), + _ => {} + } + } + self.extract_structured_input(child, remaining_depth.saturating_sub(1)); + } + } + Value::Array(items) => { + for item in items.iter().take(MAX_EXTRACTED_VALUES) { + self.extract_structured_input(item, remaining_depth.saturating_sub(1)); + } + } + _ => {} + } + } + + fn push_file_path(&mut self, value: &str) { + push_bounded(&mut self.file_paths, value); + } +} + +fn push_bounded(values: &mut Vec, value: &str) { + let value = sanitize_search_text(value); + let value = value.trim(); + if value.is_empty() || values.len() >= MAX_EXTRACTED_VALUES { + return; + } + let value = truncate_characters(value.to_owned(), MAX_METADATA_FIELD_CHARACTERS); + if !values.contains(&value) { + values.push(value); + } +} + +pub fn sanitize_search_text(value: &str) -> String { + value + .replace(HIGHLIGHT_START, "") + .replace(HIGHLIGHT_END, "") +} + +fn truncate_characters(mut value: String, maximum: usize) -> String { + let Some((byte_index, _character)) = value.char_indices().nth(maximum) else { + return value; + }; + value.truncate(byte_index); + value +} diff --git a/crates/abyss-backend/src/storage/mod.rs b/crates/abyss-backend/src/storage/mod.rs new file mode 100644 index 0000000..7cb6aa6 --- /dev/null +++ b/crates/abyss-backend/src/storage/mod.rs @@ -0,0 +1,107 @@ +//! Compile-time-selected persistence and full-text search compatibility boundary. +//! +//! HTTP handlers depend only on [`StorageBackend`]. Exactly one concrete +//! implementation is compiled into a binary, while dynamic dispatch keeps +//! connection pools, SQL dialects, search projection, and worker lifecycle out +//! of the API layer. + +#[cfg(feature = "postgres-es")] +mod postgres; +#[cfg(feature = "sqlite-fts")] +mod sqlite; + +use std::{future::Future, pin::Pin, sync::Arc}; + +use uuid::Uuid; + +use crate::{ + config::Config, + error::AppError, + search::{SessionSearchQuery, SessionSearchResponse}, + usage::{ + IngestEventsRequest, IngestEventsResponse, RawEventsQuery, RawEventsResponse, + SessionTimelineResponse, SummaryQuery, SummaryResponse, attachments::StoredImageAttachment, + }, +}; + +/// Sendable heap-allocated future returned across the dynamic storage boundary. +pub type BoxedFuture<'a, T> = Pin + Send + 'a>>; + +/// Complete object-safe event-store contract consumed by the HTTP layer. +pub trait StorageBackend: Send + Sync { + /// Verifies that the authoritative database can serve requests. + fn ready(&self) -> BoxedFuture<'_, Result<(), AppError>>; + + /// Validates and atomically ingests one owner-scoped event batch. + fn ingest_events( + &self, + user_id: Uuid, + request: IngestEventsRequest, + max_batch_size: usize, + ) -> BoxedFuture<'_, Result>; + + /// Returns one newest-first page of owner-scoped raw events. + fn raw_events( + &self, + user_id: Uuid, + query: RawEventsQuery, + default_page_size: i64, + ) -> BoxedFuture<'_, Result>; + + /// Aggregates owner-scoped event and token usage. + fn usage_summary( + &self, + user_id: Uuid, + query: SummaryQuery, + summary_limit: i64, + ) -> BoxedFuture<'_, Result>; + + /// Loads one owner-scoped session timeline. + fn session_timeline( + &self, + user_id: Uuid, + session_pk: Uuid, + ) -> BoxedFuture<'_, Result>; + + /// Loads authorized image bytes by attachment identifier. + fn image_attachment( + &self, + user_id: Uuid, + attachment_id: Uuid, + ) -> BoxedFuture<'_, Result>; + + /// Executes one owner-scoped full-text session search. + fn session_search( + &self, + user_id: Uuid, + query: SessionSearchQuery, + ) -> BoxedFuture<'_, Result>; + + /// Stops backend-owned background work and releases lifecycle resources. + fn shutdown(&self) -> BoxedFuture<'_, ()>; +} + +/// Builds the one storage implementation selected by Cargo features. +#[cfg(all(feature = "postgres-es", not(feature = "sqlite-fts")))] +pub fn build(config: &Config) -> Result, AppError> { + postgres::PostgresEsBackend::new(config) + .map(|backend| -> Arc { Arc::new(backend) }) +} + +/// Builds the one storage implementation selected by Cargo features. +#[cfg(all(feature = "sqlite-fts", not(feature = "postgres-es")))] +pub fn build(config: &Config) -> Result, AppError> { + sqlite::SqliteFtsBackend::new(config) + .map(|backend| -> Arc { Arc::new(backend) }) +} + +/// Reports an invalid storage feature selection after the compile-time error. +#[cfg(any( + all(feature = "postgres-es", feature = "sqlite-fts"), + not(any(feature = "postgres-es", feature = "sqlite-fts")) +))] +pub fn build(_config: &Config) -> Result, AppError> { + Err(AppError::config( + "one storage backend feature must be selected".to_owned(), + )) +} diff --git a/crates/abyss-backend/src/storage/postgres.rs b/crates/abyss-backend/src/storage/postgres.rs new file mode 100644 index 0000000..e9716cd --- /dev/null +++ b/crates/abyss-backend/src/storage/postgres.rs @@ -0,0 +1,197 @@ +//! PostgreSQL source-of-truth and Elasticsearch projection implementation. + +mod repository; +mod search; + +use std::time::Duration; + +use diesel::PgConnection; +use tokio::{sync::Mutex, task::JoinHandle}; +use uuid::Uuid; + +use crate::{ + config::Config, + db::{self, DbPool}, + error::AppError, + search::{SessionSearchQuery, SessionSearchResponse}, + storage::{BoxedFuture, StorageBackend}, + usage::{ + IngestEventsRequest, IngestEventsResponse, RawEventsQuery, RawEventsResponse, + SessionTimelineResponse, SummaryQuery, SummaryResponse, attachments::StoredImageAttachment, + }, +}; + +use self::search::{SearchIndexer, SearchOutboxRepository, SearchService}; + +struct SearchWorker { + shutdown: tokio::sync::watch::Sender, + handle: Mutex>>, +} + +/// Existing PostgreSQL and optional Elasticsearch storage stack. +pub struct PostgresEsBackend { + pool: DbPool, + search: Option, + search_worker: Option, +} + +impl PostgresEsBackend { + pub(super) fn new(config: &Config) -> Result { + let pool = db::create_pool(config)?; + if config.run_migrations { + db::run_migrations(&pool).map_err(|error| { + AppError::internal(format!("run PostgreSQL migrations: {error}")) + })?; + } + + let search = config.search.as_ref().map(SearchService::new).transpose()?; + let search_worker = match (&search, config.search.as_ref()) { + (Some(search), Some(search_config)) => { + let (shutdown, receiver) = tokio::sync::watch::channel(false); + let handle = + SearchIndexer::spawn(pool.clone(), search.client(), search_config, receiver); + Some(SearchWorker { + shutdown, + handle: Mutex::new(Some(handle)), + }) + } + _ => None, + }; + + Ok(Self { + pool, + search, + search_worker, + }) + } + + async fn run_db(&self, task_fn: F) -> Result + where + T: Send + 'static, + F: FnOnce(&mut PgConnection) -> Result + Send + 'static, + { + let pool = self.pool.clone(); + tokio::task::spawn_blocking(move || { + let mut connection = pool.get()?; + task_fn(&mut connection) + }) + .await + .map_err(|error| AppError::internal(format!("database task failed: {error}")))? + } +} + +impl StorageBackend for PostgresEsBackend { + fn ready(&self) -> BoxedFuture<'_, Result<(), AppError>> { + Box::pin(async move { self.run_db(db::check_ready).await }) + } + + fn ingest_events( + &self, + user_id: Uuid, + request: IngestEventsRequest, + max_batch_size: usize, + ) -> BoxedFuture<'_, Result> { + Box::pin(async move { + self.run_db(move |connection| { + repository::ingest_events(connection, &request, user_id, max_batch_size) + }) + .await + }) + } + + fn raw_events( + &self, + user_id: Uuid, + query: RawEventsQuery, + default_page_size: i64, + ) -> BoxedFuture<'_, Result> { + Box::pin(async move { + self.run_db(move |connection| { + repository::raw_events(connection, &query, user_id, default_page_size) + }) + .await + }) + } + + fn usage_summary( + &self, + user_id: Uuid, + query: SummaryQuery, + summary_limit: i64, + ) -> BoxedFuture<'_, Result> { + Box::pin(async move { + self.run_db(move |connection| { + repository::usage_summary(connection, &query, user_id, summary_limit) + }) + .await + }) + } + + fn session_timeline( + &self, + user_id: Uuid, + session_pk: Uuid, + ) -> BoxedFuture<'_, Result> { + Box::pin(async move { + self.run_db(move |connection| { + repository::session_timeline(connection, user_id, session_pk) + }) + .await + }) + } + + fn image_attachment( + &self, + user_id: Uuid, + attachment_id: Uuid, + ) -> BoxedFuture<'_, Result> { + Box::pin(async move { + self.run_db(move |connection| { + repository::image_attachment(connection, user_id, attachment_id) + }) + .await + }) + } + + fn session_search( + &self, + user_id: Uuid, + query: SessionSearchQuery, + ) -> BoxedFuture<'_, Result> { + Box::pin(async move { + let search = self.search.clone().ok_or_else(|| { + AppError::unavailable("session search is not configured".to_owned()) + })?; + let execution = search.search(user_id, query).await?; + let session_ids = execution.session_ids(); + let details = self + .run_db(move |connection| { + SearchOutboxRepository::session_details(connection, user_id, &session_ids) + }) + .await?; + Ok(execution.hydrate(details)) + }) + } + + fn shutdown(&self) -> BoxedFuture<'_, ()> { + Box::pin(async move { + let Some(worker) = &self.search_worker else { + return; + }; + if worker.shutdown.send(true).is_err() { + tracing::trace!("session search indexer already stopped"); + } + let Some(mut handle) = worker.handle.lock().await.take() else { + return; + }; + match tokio::time::timeout(Duration::from_secs(15), &mut handle).await { + Ok(Ok(())) => {} + Ok(Err(error)) => tracing::warn!(%error, "session search indexer task failed"), + Err(_elapsed) => { + tracing::warn!("session search indexer did not stop before timeout"); + handle.abort(); + } + } + }) + } +} diff --git a/crates/abyss-backend/src/usage/repository.rs b/crates/abyss-backend/src/storage/postgres/repository.rs similarity index 87% rename from crates/abyss-backend/src/usage/repository.rs rename to crates/abyss-backend/src/storage/postgres/repository.rs index 3285cf2..df586b8 100644 --- a/crates/abyss-backend/src/usage/repository.rs +++ b/crates/abyss-backend/src/storage/postgres/repository.rs @@ -1,4 +1,4 @@ -//! Diesel-backed repository functions for Agent usage APIs. +//! PostgreSQL-backed repository functions for Agent usage APIs. //! //! This module is the transactional boundary for the event hierarchy. Ingest //! validates the complete request before opening a transaction, upserts device, @@ -18,7 +18,6 @@ use diesel::{ Array, BigInt, Bool, Integer, Jsonb, Nullable, Text, Timestamptz, Uuid as SqlUuid, }, }; -use sha2::{Digest, Sha256}; use uuid::Uuid; use crate::{ @@ -41,14 +40,23 @@ use crate::{ SummaryRow, TurnTimeline, UsageEventResponse, attachments::{ ImageMediaType, StoredImageAttachment, UsageEventAttachmentResponse, - ValidatedImageAttachment, validate_image_attachments, + ValidatedImageAttachment, }, diagnostics::IngestDiagnosticCapture, event_order::UsageEventTimelineOrder, + persistence::{ + TurnIdentity, TurnIdentityKind, agent_name_filter_values, canonical_agent_name, + choose_turn_index, escape_like_pattern, non_empty, non_negative_count, normalize_limit, + normalize_slug, normalize_text, normalized_optional, normalized_tokens, parse_group_by, + sha256_bytes, turn_identity, unix_epoch, validate_batch, validate_event_type, + }, session_metadata_from_event, }, }; +#[cfg(test)] +use crate::usage::persistence::{turn_identity_from_metadata, validate_event}; + const SUMMARY_AGGREGATE_SQL: &str = "\ SELECT \ CASE WHEN $1 THEN to_char(e.observed_at AT TIME ZONE 'UTC', 'YYYY-MM-DD') END AS day, \ @@ -794,72 +802,6 @@ fn next_session_turn_index( .map_err(AppError::from) } -const fn choose_turn_index( - requested_turn_index: i32, - existing_turn_index: Option, - next_turn_index: i32, -) -> i32 { - // Events with the same stable provider identity must remain in one turn, - // even if collector-local numbering changes between observations. - if let Some(existing_turn_index) = existing_turn_index { - return existing_turn_index; - } - // A previously used local number indicates a restarted counter. Append at - // the next index; otherwise preserve intentional gaps from the collector. - if requested_turn_index < next_turn_index { - next_turn_index - } else { - requested_turn_index - } -} - -fn turn_identity(event: &IngestUsageEvent) -> Option { - turn_identity_from_metadata(&event.metadata) -} - -fn turn_identity_from_metadata(metadata: &serde_json::Value) -> Option { - // Prefer Agent-level turn identities over provider message/response ids - // because a single user turn may contain several provider round trips. - metadata_string(metadata, "codex_turn_id") - .map(|value| TurnIdentity { - kind: TurnIdentityKind::CodexTurnId, - value, - }) - .or_else(|| { - metadata_string(metadata, "claude_turn_id").map(|value| TurnIdentity { - kind: TurnIdentityKind::ClaudeTurnId, - value, - }) - }) - .or_else(|| { - metadata_string(metadata, "response_id").map(|value| TurnIdentity { - kind: TurnIdentityKind::ResponseId, - value, - }) - }) - .or_else(|| { - metadata_string(metadata, "message_id").map(|value| TurnIdentity { - kind: TurnIdentityKind::MessageId, - value, - }) - }) - .or_else(|| { - metadata_string(metadata, "request_hash").map(|value| TurnIdentity { - kind: TurnIdentityKind::RequestHash, - value, - }) - }) -} - -fn metadata_string(metadata: &serde_json::Value, key: &str) -> Option { - metadata - .get(key) - .and_then(serde_json::Value::as_str) - .map(str::trim) - .filter(|value| !value.is_empty()) - .map(str::to_owned) -} - fn upsert_turn( connection: &mut PgConnection, user_id: Uuid, @@ -1070,228 +1012,6 @@ const fn touch_turn_metadata(turn: &AgentTurn) { )); } -fn validate_batch( - request: &IngestEventsRequest, - max_batch_size: usize, -) -> Result>, AppError> { - if request.events.len() > max_batch_size { - return Err(AppError::validation(format!( - "events batch size must be <= {max_batch_size}" - ))); - } - - // Decode and validate all potentially expensive attachment content before - // obtaining a connection transaction and any table locks. - let mut attachments = Vec::with_capacity(request.events.len()); - for event in &request.events { - validate_event(event)?; - attachments.push(validate_image_attachments(&event.attachments)?); - } - if request.diagnostic_captures.len() > max_batch_size { - return Err(AppError::validation(format!( - "diagnostic captures batch size must be <= {max_batch_size}" - ))); - } - let request_event_ids = request - .events - .iter() - .map(|event| event.event_id.as_str()) - .collect::>(); - let mut capture_ids = HashSet::with_capacity(request.diagnostic_captures.len()); - for capture in &request.diagnostic_captures { - capture.validate_event_correlation(&request_event_ids)?; - if !capture_ids.insert(capture.capture_id.as_str()) { - return Err(AppError::validation( - "diagnostic capture ids must be unique within one ingest request".to_owned(), - )); - } - } - Ok(attachments) -} - -fn validate_event(event: &IngestUsageEvent) -> Result<(), AppError> { - require_non_empty(&event.event_id, "event_id")?; - require_non_empty(&event.device.host_name, "device.host_name")?; - require_non_empty(&event.device.platform, "device.platform")?; - require_non_empty(&event.agent.name, "agent.name")?; - require_non_empty(&event.session_id, "session_id")?; - require_non_empty(&event.llm.provider, "llm.provider")?; - require_non_empty(&event.llm.model, "llm.model")?; - if event.turn_index < 1_i32 { - return Err(AppError::validation("turn_index must be >= 1".to_owned())); - } - let tokens = normalized_tokens(event)?; - if tokens.total == 0 { - return Ok(()); - } - let visible_tokens = tokens - .input - .saturating_add(tokens.output) - .saturating_add(tokens.reasoning); - if tokens.total < visible_tokens { - tracing::debug!( - event_id = %event.event_id, - "provider total_tokens is lower than visible token components" - ); - } - Ok(()) -} - -fn require_non_empty(value: &str, field: &str) -> Result<(), AppError> { - if value.trim().is_empty() { - return Err(AppError::validation(format!("{field} is required"))); - } - Ok(()) -} - -fn normalized_tokens(event: &IngestUsageEvent) -> Result { - let input = normalize_token(event.token_usage.input_tokens, "input_tokens")?; - let output = normalize_token(event.token_usage.output_tokens, "output_tokens")?; - let cache_read = normalize_token(event.token_usage.cache_read_tokens, "cache_read_tokens")?; - let cache_write = normalize_token(event.token_usage.cache_write_tokens, "cache_write_tokens")?; - let reasoning = normalize_token(event.token_usage.reasoning_tokens, "reasoning_tokens")?; - // Preserve an explicit provider total because provider accounting may not - // equal the visible component sum; derive only when the field is absent. - let total = match event.token_usage.total_tokens { - Some(value) => normalize_token(Some(value), "total_tokens")?, - None => input - .saturating_add(output) - .saturating_add(cache_read) - .saturating_add(cache_write) - .saturating_add(reasoning), - }; - - Ok(NormalizedTokens { - input, - output, - cache_read, - cache_write, - reasoning, - total, - }) -} - -fn normalize_token(value: Option, field: &str) -> Result { - let value = value.unwrap_or(0); - if value < 0 { - return Err(AppError::validation(format!("{field} must be >= 0"))); - } - Ok(value) -} - -fn validate_event_type(value: &str) -> Result { - match value.trim().to_ascii_lowercase().as_str() { - "request" => Ok("request".to_owned()), - "response" => Ok("response".to_owned()), - _ => Err(AppError::validation( - "event_type must be request or response".to_owned(), - )), - } -} - -fn parse_group_by(raw: Option<&str>) -> Vec { - // Unknown dimensions are ignored and historical aliases remain readable. - // Falling back after filtering prevents an empty GROUP BY contract. - let parsed: Vec<_> = raw - .unwrap_or("user,agent,provider,model") - .split(',') - .map(str::trim) - .filter(|value| !value.is_empty()) - .map(|value| match value { - "employee" => "user".to_owned(), - "llm_provider" => "provider".to_owned(), - "llm_model" => "model".to_owned(), - other => other.to_owned(), - }) - .filter(|value| { - matches!( - value.as_str(), - "day" | "user" | "device" | "agent" | "provider" | "model" | "event_type" - ) - }) - .collect(); - - if parsed.is_empty() { - vec![ - "user".to_owned(), - "agent".to_owned(), - "provider".to_owned(), - "model".to_owned(), - ] - } else { - parsed - } -} - -fn normalize_slug(value: &str) -> String { - value - .trim() - .to_ascii_lowercase() - .chars() - .map(|character| match character { - '_' | ' ' => '-', - other => other, - }) - .collect() -} - -fn canonical_agent_name(value: &str) -> String { - let normalized = normalize_slug(value); - match normalized.as_str() { - "claude" | "claude-desktop" => "claude-code".to_owned(), - _ => normalized, - } -} - -fn agent_name_filter_values(value: &str) -> Vec { - let canonical = canonical_agent_name(value); - if canonical == "claude-code" { - vec![ - "claude-code".to_owned(), - "claude".to_owned(), - "claude-desktop".to_owned(), - ] - } else { - vec![canonical] - } -} - -fn normalize_text(value: &str) -> String { - value.trim().to_owned() -} - -fn normalized_optional(value: Option<&str>) -> Option { - value - .map(str::trim) - .filter(|value| !value.is_empty()) - .map(str::to_owned) -} - -fn non_empty(value: &str) -> Option<&str> { - (!value.trim().is_empty()).then_some(value) -} - -fn non_negative_count(value: i64) -> usize { - usize::try_from(value.max(0_i64)).unwrap_or(usize::MAX) -} - -const fn unix_epoch() -> DateTime { - DateTime::::from_timestamp(0_i64, 0_u32).expect("unix epoch must be a valid timestamp") -} - -fn escape_like_pattern(value: &str) -> String { - // The SQL uses LIKE ... ESCAPE '\\'; escape wildcard characters so a user - // filter remains a literal substring search rather than a pattern language. - let mut escaped = String::with_capacity(value.len()); - for character in value.chars() { - if matches!(character, '\\' | '%' | '_') { - escaped.push('\\'); - } - escaped.push(character); - } - escaped -} - #[cfg(test)] fn latest_optional( left: Option>, @@ -1304,14 +1024,6 @@ fn latest_optional( } } -fn normalize_limit(limit: Option, fallback: i64, maximum: i64) -> i64 { - limit.unwrap_or(fallback).clamp(1_i64, maximum) -} - -fn sha256_bytes(value: &str) -> Vec { - Sha256::digest(value.as_bytes()).to_vec() -} - struct EventFilters<'a> { user_id: Uuid, from: Option>, @@ -1504,34 +1216,12 @@ impl From for SummaryRow { } } -struct TurnIdentity { - kind: TurnIdentityKind, - value: String, -} - -enum TurnIdentityKind { - CodexTurnId, - ClaudeTurnId, - ResponseId, - MessageId, - RequestHash, -} - #[derive(QueryableByName)] struct TurnIndexRow { #[diesel(sql_type = Integer)] turn_index: i32, } -struct NormalizedTokens { - input: i64, - output: i64, - cache_read: i64, - cache_write: i64, - reasoning: i64, - total: i64, -} - struct UpsertSessionInput<'a> { device: &'a Device, user_id: Uuid, diff --git a/crates/abyss-backend/src/search/document.rs b/crates/abyss-backend/src/storage/postgres/search/document.rs similarity index 58% rename from crates/abyss-backend/src/search/document.rs rename to crates/abyss-backend/src/storage/postgres/search/document.rs index 8937d7d..0c0d7ec 100644 --- a/crates/abyss-backend/src/search/document.rs +++ b/crates/abyss-backend/src/storage/postgres/search/document.rs @@ -6,19 +6,17 @@ //! retains the complete source event and remains the recovery source. use serde::Serialize; -use serde_json::Value; use uuid::Uuid; -use crate::db::models::UsageEvent; +use crate::{ + db::models::UsageEvent, + search::projection::{SearchProjection, sanitize_search_text}, +}; -use super::elasticsearch::{HIGHLIGHT_END, HIGHLIGHT_START}; - -const MAX_CONTENT_CHARACTERS: usize = 1_000_000; -const MAX_METADATA_FIELD_CHARACTERS: usize = 100_000; -const MAX_EXTRACTED_VALUES: usize = 64; -const MAX_JSON_DEPTH: usize = 8; +#[cfg(test)] +use crate::search::{HIGHLIGHT_END, HIGHLIGHT_START, projection::MAX_CONTENT_CHARACTERS}; -/// Elasticsearch representation of one immutable usage event. +/// Full-text search document for one immutable usage event. #[derive(Serialize)] pub struct SearchDocument { /// Source usage-event primary key and Elasticsearch document identifier. @@ -64,7 +62,7 @@ impl SearchDocument { /// Projects one source event into the strict Elasticsearch mapping. #[must_use] pub fn from_event(event: UsageEvent) -> Self { - let metadata = SearchableMetadata::extract(&event.metadata); + let projection = SearchProjection::from_source(event.text, &event.metadata); Self { event_pk: event.id, user_id: event.user_id, @@ -77,132 +75,15 @@ impl SearchDocument { llm_model: event.llm_model, event_type: event.event_type, observed_at: event.observed_at, - content: event - .text - .map(|text| sanitize_search_text(&text)) - .map(|text| truncate_characters(text, MAX_CONTENT_CHARACTERS)), - tool_names: metadata.tool_names, - tool_content: metadata.tool_content, - commands: metadata.commands, - file_paths: metadata.file_paths, + content: projection.content, + tool_names: projection.tool_names, + tool_content: projection.tool_content, + commands: projection.commands, + file_paths: projection.file_paths, } } } -struct SearchableMetadata { - tool_names: Vec, - tool_content: Vec, - commands: Vec, - file_paths: Vec, -} - -impl SearchableMetadata { - fn extract(metadata: &Value) -> Self { - let mut extracted = Self { - tool_names: Vec::new(), - tool_content: Vec::new(), - commands: Vec::new(), - file_paths: Vec::new(), - }; - - if let Some(working_directory) = metadata.get("working_directory").and_then(Value::as_str) { - extracted.push_file_path(working_directory); - } - - // Deliberately ignore every top-level key except working_directory and - // content_segments so headers, credentials, and image data stay out of - // the derived index even when present in raw metadata. - let Some(segments) = metadata.get("content_segments").and_then(Value::as_array) else { - return extracted; - }; - for segment in segments.iter().take(MAX_EXTRACTED_VALUES) { - let Some(object) = segment.as_object() else { - continue; - }; - match object.get("type").and_then(Value::as_str) { - Some("tool_call") => { - if let Some(name) = object.get("name").and_then(Value::as_str) { - push_bounded(&mut extracted.tool_names, name); - } - if let Some(input) = object.get("input").and_then(Value::as_str) { - push_bounded(&mut extracted.tool_content, input); - if let Ok(value) = serde_json::from_str::(input) { - extracted.extract_structured_input(&value, MAX_JSON_DEPTH); - } - } - } - Some("tool_result") => { - if let Some(output) = object.get("output").and_then(Value::as_str) { - push_bounded(&mut extracted.tool_content, output); - } - } - _ => {} - } - } - extracted - } - - fn extract_structured_input(&mut self, value: &Value, remaining_depth: usize) { - if remaining_depth == 0 { - return; - } - match value { - Value::Object(object) => { - for (key, child) in object.iter().take(MAX_EXTRACTED_VALUES) { - if let Some(text) = child.as_str() { - match key.as_str() { - "command" | "cmd" => push_bounded(&mut self.commands, text), - "file" | "file_path" | "path" => self.push_file_path(text), - _ => {} - } - } - // Recurse after examining the current key so nested command - // or path fields are discoverable without indexing all JSON. - self.extract_structured_input(child, remaining_depth.saturating_sub(1)); - } - } - Value::Array(items) => { - for item in items.iter().take(MAX_EXTRACTED_VALUES) { - self.extract_structured_input(item, remaining_depth.saturating_sub(1)); - } - } - _ => {} - } - } - - fn push_file_path(&mut self, value: &str) { - push_bounded(&mut self.file_paths, value); - } -} - -fn push_bounded(values: &mut Vec, value: &str) { - let value = sanitize_search_text(value); - let value = value.trim(); - if value.is_empty() || values.len() >= MAX_EXTRACTED_VALUES { - return; - } - let value = truncate_characters(value.to_owned(), MAX_METADATA_FIELD_CHARACTERS); - if !values.contains(&value) { - values.push(value); - } -} - -fn sanitize_search_text(value: &str) -> String { - // Marker tokens are controlled by this service during highlighting. Strip - // collector-supplied copies so clients cannot forge highlighted segments. - value - .replace(HIGHLIGHT_START, "") - .replace(HIGHLIGHT_END, "") -} - -fn truncate_characters(mut value: String, maximum: usize) -> String { - let Some((byte_index, _character)) = value.char_indices().nth(maximum) else { - return value; - }; - value.truncate(byte_index); - value -} - #[cfg(test)] mod tests { use chrono::Utc; diff --git a/crates/abyss-backend/src/search/elasticsearch.rs b/crates/abyss-backend/src/storage/postgres/search/elasticsearch.rs similarity index 98% rename from crates/abyss-backend/src/search/elasticsearch.rs rename to crates/abyss-backend/src/storage/postgres/search/elasticsearch.rs index 28187f7..ee59dfa 100644 --- a/crates/abyss-backend/src/search/elasticsearch.rs +++ b/crates/abyss-backend/src/storage/postgres/search/elasticsearch.rs @@ -13,17 +13,16 @@ use serde::Deserialize; use serde_json::{Value, json}; use uuid::Uuid; -use crate::{config::SearchConfig, error::AppError}; +use crate::{ + config::SearchConfig, + error::AppError, + search::{HIGHLIGHT_END, HIGHLIGHT_START, ValidatedSearchQuery}, +}; -use super::{ValidatedSearchQuery, document::SearchDocument}; +use super::document::SearchDocument; /// Fixed name of the derived usage-event index. pub const SEARCH_INDEX: &str = "abyss_usage_events"; -/// Sentinel inserted before text matched by Elasticsearch highlighting. -pub const HIGHLIGHT_START: &str = "[[[ABYSS_SEARCH_HIGHLIGHT_START]]]"; -/// Sentinel inserted after text matched by Elasticsearch highlighting. -pub const HIGHLIGHT_END: &str = "[[[ABYSS_SEARCH_HIGHLIGHT_END]]]"; - /// One idempotent operation submitted through Elasticsearch's Bulk API. pub enum BulkOperation { /// Create or replace a document from current source-event state. diff --git a/crates/abyss-backend/src/storage/postgres/search/mod.rs b/crates/abyss-backend/src/storage/postgres/search/mod.rs new file mode 100644 index 0000000..1c2f4f3 --- /dev/null +++ b/crates/abyss-backend/src/storage/postgres/search/mod.rs @@ -0,0 +1,159 @@ +//! PostgreSQL-owned Elasticsearch projection and session-search implementation. +//! +//! Search results are grouped by session in Elasticsearch, then hydrated with +//! authoritative PostgreSQL session and device rows. Elasticsearch never acts +//! as an authorization source: every query includes the authenticated owner and +//! stale hits without matching PostgreSQL details are omitted. + +mod document; +mod elasticsearch; +mod outbox; +mod worker; + +use std::collections::{HashMap, HashSet}; + +use uuid::Uuid; + +use crate::{ + config::SearchConfig, + error::AppError, + search::{ + SearchFragment, SessionSearchMatch, SessionSearchQuery, SessionSearchResponse, + SessionSearchResult, ValidatedSearchQuery, + }, +}; + +use self::{ + elasticsearch::{ElasticsearchClient, SearchMatchPage}, + outbox::SearchSessionDetails, +}; + +pub(super) use self::{outbox::SearchOutboxRepository, worker::SearchIndexer}; + +#[derive(Clone)] +/// Validating facade over the Elasticsearch HTTP client. +pub(super) struct SearchService { + client: ElasticsearchClient, +} + +impl SearchService { + /// Creates a search service from startup-validated settings. + pub(super) fn new(config: &SearchConfig) -> Result { + Ok(Self { + client: ElasticsearchClient::new(config)?, + }) + } + + /// Returns a cloned client for the background indexer. + #[must_use] + pub(super) fn client(&self) -> ElasticsearchClient { + self.client.clone() + } + + /// Validates and executes one owner-scoped session search. + pub(super) async fn search( + &self, + user_id: Uuid, + query: SessionSearchQuery, + ) -> Result { + let query = query.validate()?; + let page = self.client.search(user_id, &query).await.map_err(|error| { + tracing::warn!(%error, %user_id, "session search request failed"); + AppError::unavailable("session search is temporarily unavailable".to_owned()) + })?; + Ok(SearchExecution { query, page }) + } +} + +/// Elasticsearch matches paired with the validated query that produced them. +pub(super) struct SearchExecution { + query: ValidatedSearchQuery, + page: SearchMatchPage, +} + +impl SearchExecution { + /// Returns session keys that must be hydrated from PostgreSQL. + #[must_use] + pub(super) fn session_ids(&self) -> Vec { + self.page + .sessions + .iter() + .map(|session| session.session_pk) + .collect() + } + + /// Combines search matches with authoritative session/device details. + /// + /// Stale index entries whose session no longer exists are omitted instead + /// of returning partially authorized or incomplete data. + #[must_use] + pub(super) fn hydrate( + self, + mut details: HashMap, + ) -> SessionSearchResponse { + let items = self + .page + .sessions + .into_iter() + .filter_map(|matches| { + let details = details.remove(&matches.session_pk)?; + let mut models = matches + .events + .iter() + .map(|event| event.llm_model.clone()) + .collect::>() + .into_iter() + .collect::>(); + models.sort(); + let mut providers = matches + .events + .iter() + .map(|event| event.llm_provider.clone()) + .collect::>() + .into_iter() + .collect::>(); + providers.sort(); + Some(SessionSearchResult { + session_pk: details.session.id, + session_id: details.session.session_id, + agent_name: details.session.agent_name, + agent_version: details.session.agent_version, + host_name: details.device.host_name, + platform: details.device.platform, + started_at: details.session.started_at, + ended_at: details.session.ended_at, + providers, + models, + match_count: matches.match_count, + matches: matches + .events + .into_iter() + .map(|event| SessionSearchMatch { + event_pk: event.event_pk, + turn_pk: event.turn_pk, + turn_index: event.turn_index, + event_type: event.event_type, + llm_provider: event.llm_provider, + llm_model: event.llm_model, + observed_at: event.observed_at, + fragments: event + .fragments + .into_iter() + .map(|fragment| SearchFragment::parse(&fragment)) + .collect(), + }) + .collect(), + }) + }) + .collect::>(); + let consumed = u64::from(self.query.page).saturating_mul(u64::from(self.query.page_size)); + SessionSearchResponse { + query: self.query.text, + total_sessions: self.page.total_sessions, + page: self.query.page, + page_size: self.query.page_size, + has_more: consumed < self.page.total_sessions, + items, + } + } +} diff --git a/crates/abyss-backend/src/search/outbox.rs b/crates/abyss-backend/src/storage/postgres/search/outbox.rs similarity index 100% rename from crates/abyss-backend/src/search/outbox.rs rename to crates/abyss-backend/src/storage/postgres/search/outbox.rs diff --git a/crates/abyss-backend/src/search/worker.rs b/crates/abyss-backend/src/storage/postgres/search/worker.rs similarity index 97% rename from crates/abyss-backend/src/search/worker.rs rename to crates/abyss-backend/src/storage/postgres/search/worker.rs index 4d1c9ee..71859f8 100644 --- a/crates/abyss-backend/src/search/worker.rs +++ b/crates/abyss-backend/src/storage/postgres/search/worker.rs @@ -10,14 +10,11 @@ use std::time::Duration; use tokio::{sync::watch, task::JoinHandle}; use uuid::Uuid; -use crate::{ - config::SearchConfig, - db::DbPool, - error::AppError, - search::{ - elasticsearch::ElasticsearchClient, - outbox::{OutboxTaskResult, SearchOutboxRepository}, - }, +use crate::{config::SearchConfig, db::DbPool, error::AppError}; + +use super::{ + elasticsearch::ElasticsearchClient, + outbox::{OutboxTaskResult, SearchOutboxRepository}, }; /// Factory for the detached search projection task. diff --git a/crates/abyss-backend/src/storage/sqlite/connection.rs b/crates/abyss-backend/src/storage/sqlite/connection.rs new file mode 100644 index 0000000..6d40f02 --- /dev/null +++ b/crates/abyss-backend/src/storage/sqlite/connection.rs @@ -0,0 +1,95 @@ +//! Diesel SQLite connection pooling and per-connection safety configuration. + +use std::{fs, path::Path}; + +use diesel::{ + QueryableByName, RunQueryDsl, SqliteConnection, + connection::SimpleConnection, + r2d2::{ConnectionManager, CustomizeConnection, Error as DieselPoolError}, + sql_query, + sql_types::Text, +}; + +use crate::error::AppError; + +pub(super) type SqlitePool = r2d2::Pool>; + +#[derive(Debug)] +struct SqliteConnectionCustomizer; + +impl CustomizeConnection for SqliteConnectionCustomizer { + fn on_acquire(&self, connection: &mut SqliteConnection) -> Result<(), DieselPoolError> { + connection + .batch_execute( + "PRAGMA foreign_keys = ON; + PRAGMA busy_timeout = 5000; + PRAGMA synchronous = NORMAL;", + ) + .map_err(DieselPoolError::from) + } +} + +#[derive(QueryableByName)] +struct JournalMode { + #[diesel(sql_type = Text)] + journal_mode: String, +} + +pub(super) fn create_pool(database_url: &str, pool_size: u32) -> Result { + let database_url = normalized_database_url(database_url)?; + let maximum_size = if database_url == ":memory:" { + 1 + } else { + pool_size + }; + let manager = ConnectionManager::::new(database_url); + r2d2::Pool::builder() + .max_size(maximum_size) + .connection_customizer(Box::new(SqliteConnectionCustomizer)) + .build(manager) + .map_err(AppError::from) +} + +pub(super) fn configure_database(connection: &mut SqliteConnection) -> Result<(), AppError> { + let journal_mode = sql_query("PRAGMA journal_mode = WAL") + .get_result::(connection)? + .journal_mode; + if journal_mode != "wal" && journal_mode != "memory" { + return Err(AppError::internal(format!( + "SQLite did not enable WAL journal mode: {journal_mode}" + ))); + } + Ok(()) +} + +fn normalized_database_url(database_url: &str) -> Result { + let normalized = database_url + .strip_prefix("sqlite://") + .unwrap_or(database_url); + if normalized == ":memory:" { + return Ok(normalized.to_owned()); + } + let path = Path::new(normalized); + if path.as_os_str().is_empty() { + return Err(AppError::config( + "ABYSS_BACKEND_DATABASE_URL must contain a SQLite file path".to_owned(), + )); + } + create_parent_directory(path)?; + Ok(normalized.to_owned()) +} + +fn create_parent_directory(path: &Path) -> Result<(), AppError> { + let Some(parent) = path + .parent() + .filter(|parent| !parent.as_os_str().is_empty()) + else { + return Ok(()); + }; + fs::create_dir_all(parent).map_err(|error| { + AppError::config(format!( + "create SQLite database directory {}: {error}", + parent.display() + )) + }) +} diff --git a/crates/abyss-backend/src/storage/sqlite/migrations.rs b/crates/abyss-backend/src/storage/sqlite/migrations.rs new file mode 100644 index 0000000..8000846 --- /dev/null +++ b/crates/abyss-backend/src/storage/sqlite/migrations.rs @@ -0,0 +1,15 @@ +//! Diesel-managed embedded migrations for the SQLite event store. + +use diesel::SqliteConnection; +use diesel_migrations::{EmbeddedMigrations, MigrationHarness, embed_migrations}; + +use crate::error::AppError; + +const MIGRATIONS: EmbeddedMigrations = embed_migrations!("migrations-sqlite"); + +pub(super) fn run(connection: &mut SqliteConnection) -> Result<(), AppError> { + connection + .run_pending_migrations(MIGRATIONS) + .map_err(|error| AppError::internal(format!("run SQLite migrations: {error}")))?; + Ok(()) +} diff --git a/crates/abyss-backend/src/storage/sqlite/mod.rs b/crates/abyss-backend/src/storage/sqlite/mod.rs new file mode 100644 index 0000000..4214d9c --- /dev/null +++ b/crates/abyss-backend/src/storage/sqlite/mod.rs @@ -0,0 +1,154 @@ +//! Self-contained SQLite source-of-truth with transactional FTS5 projection. + +mod connection; +mod migrations; +mod models; +mod repository; +mod schema; +mod search; + +use diesel::{QueryDsl, RunQueryDsl, SqliteConnection, dsl::count_star}; +use uuid::Uuid; + +use crate::{ + config::Config, + error::AppError, + search::{SessionSearchQuery, SessionSearchResponse}, + storage::{BoxedFuture, StorageBackend}, + usage::{ + IngestEventsRequest, IngestEventsResponse, RawEventsQuery, RawEventsResponse, + SessionTimelineResponse, SummaryQuery, SummaryResponse, attachments::StoredImageAttachment, + }, +}; + +use self::connection::SqlitePool; + +/// SQLite event store with an FTS5 projection committed beside each event. +pub(super) struct SqliteFtsBackend { + pool: SqlitePool, +} + +impl SqliteFtsBackend { + pub(super) fn new(config: &Config) -> Result { + let pool = connection::create_pool(&config.database_url, config.database_pool_size)?; + { + let mut database = pool.get()?; + connection::configure_database(&mut database)?; + if config.run_migrations { + migrations::run(&mut database)?; + } + } + Ok(Self { pool }) + } + + async fn run_db(&self, task_fn: F) -> Result + where + T: Send + 'static, + F: FnOnce(&mut SqliteConnection) -> Result + Send + 'static, + { + let pool = self.pool.clone(); + tokio::task::spawn_blocking(move || { + let mut connection = pool.get()?; + task_fn(&mut connection) + }) + .await + .map_err(|error| AppError::internal(format!("database task failed: {error}")))? + } +} + +impl StorageBackend for SqliteFtsBackend { + fn ready(&self) -> BoxedFuture<'_, Result<(), AppError>> { + Box::pin(async move { + self.run_db(|connection| { + schema::app_users::table + .select(count_star()) + .first::(connection)?; + Ok(()) + }) + .await + }) + } + + fn ingest_events( + &self, + user_id: Uuid, + request: IngestEventsRequest, + max_batch_size: usize, + ) -> BoxedFuture<'_, Result> { + Box::pin(async move { + self.run_db(move |connection| { + repository::ingest_events(connection, &request, user_id, max_batch_size) + }) + .await + }) + } + + fn raw_events( + &self, + user_id: Uuid, + query: RawEventsQuery, + default_page_size: i64, + ) -> BoxedFuture<'_, Result> { + Box::pin(async move { + self.run_db(move |connection| { + repository::raw_events(connection, &query, user_id, default_page_size) + }) + .await + }) + } + + fn usage_summary( + &self, + user_id: Uuid, + query: SummaryQuery, + summary_limit: i64, + ) -> BoxedFuture<'_, Result> { + Box::pin(async move { + self.run_db(move |connection| { + repository::usage_summary(connection, &query, user_id, summary_limit) + }) + .await + }) + } + + fn session_timeline( + &self, + user_id: Uuid, + session_pk: Uuid, + ) -> BoxedFuture<'_, Result> { + Box::pin(async move { + self.run_db(move |connection| { + repository::session_timeline(connection, user_id, session_pk) + }) + .await + }) + } + + fn image_attachment( + &self, + user_id: Uuid, + attachment_id: Uuid, + ) -> BoxedFuture<'_, Result> { + Box::pin(async move { + self.run_db(move |connection| { + repository::image_attachment(connection, user_id, attachment_id) + }) + .await + }) + } + + fn session_search( + &self, + user_id: Uuid, + query: SessionSearchQuery, + ) -> BoxedFuture<'_, Result> { + Box::pin(async move { + self.run_db(move |connection| search::session_search(connection, user_id, query)) + .await + }) + } + + fn shutdown(&self) -> BoxedFuture<'_, ()> { + Box::pin(async {}) + } +} diff --git a/crates/abyss-backend/src/storage/sqlite/models.rs b/crates/abyss-backend/src/storage/sqlite/models.rs new file mode 100644 index 0000000..ca70f21 --- /dev/null +++ b/crates/abyss-backend/src/storage/sqlite/models.rs @@ -0,0 +1,407 @@ +//! Diesel row and insert models for SQLite relational tables. + +use chrono::{DateTime, Utc}; +use diesel::{Insertable, Queryable, Selectable}; +use serde_json::Value; +use uuid::Uuid; + +use crate::{error::AppError, usage::event_order::TimelineEvent}; + +use super::schema::{ + agent_diagnostic_capture_events, agent_diagnostic_captures, agent_sessions, agent_turns, + devices, llm_usage_event_attachments, llm_usage_events, +}; + +#[derive(Queryable, Selectable)] +#[diesel(table_name = devices)] +#[diesel(check_for_backend(diesel::sqlite::Sqlite))] +pub(super) struct DeviceRow { + pub(super) id: String, + pub(super) user_id: String, + pub(super) host_name: String, + pub(super) platform: String, + pub(super) os_version: Option, + pub(super) first_seen_at: i64, + pub(super) last_seen_at: i64, + pub(super) created_at: i64, + pub(super) updated_at: i64, +} + +#[derive(Insertable)] +#[diesel(table_name = devices)] +pub(super) struct NewDevice { + pub(super) id: String, + pub(super) user_id: String, + pub(super) host_name: String, + pub(super) platform: String, + pub(super) os_version: Option, + pub(super) first_seen_at: i64, + pub(super) last_seen_at: i64, + pub(super) created_at: i64, + pub(super) updated_at: i64, +} + +#[derive(Queryable, Selectable)] +#[diesel(table_name = agent_sessions)] +#[diesel(check_for_backend(diesel::sqlite::Sqlite))] +pub(super) struct SessionRow { + pub(super) id: String, + pub(super) user_id: String, + pub(super) device_context_id: String, + pub(super) agent_name: String, + pub(super) agent_version: Option, + pub(super) session_id: String, + pub(super) started_at: i64, + pub(super) ended_at: Option, + pub(super) metadata: String, + pub(super) created_at: i64, + pub(super) updated_at: i64, +} + +#[derive(Insertable)] +#[diesel(table_name = agent_sessions)] +pub(super) struct NewSession { + pub(super) id: String, + pub(super) user_id: String, + pub(super) device_context_id: String, + pub(super) agent_name: String, + pub(super) agent_version: Option, + pub(super) session_id: String, + pub(super) started_at: i64, + pub(super) ended_at: Option, + pub(super) metadata: String, + pub(super) created_at: i64, + pub(super) updated_at: i64, +} + +#[derive(Queryable, Selectable)] +#[diesel(table_name = agent_turns)] +#[diesel(check_for_backend(diesel::sqlite::Sqlite))] +pub(super) struct TurnRow { + pub(super) id: String, + pub(super) user_id: String, + pub(super) session_pk: String, + pub(super) turn_index: i32, + pub(super) started_at: i64, + pub(super) ended_at: Option, + pub(super) created_at: i64, + pub(super) updated_at: i64, +} + +#[derive(Insertable)] +#[diesel(table_name = agent_turns)] +pub(super) struct NewTurn { + pub(super) id: String, + pub(super) user_id: String, + pub(super) session_pk: String, + pub(super) turn_index: i32, + pub(super) started_at: i64, + pub(super) ended_at: Option, + pub(super) created_at: i64, + pub(super) updated_at: i64, +} + +#[derive(Clone, Queryable, Selectable)] +#[diesel(table_name = llm_usage_events)] +#[diesel(check_for_backend(diesel::sqlite::Sqlite))] +pub(super) struct EventRow { + pub(super) id: String, + pub(super) user_id: String, + pub(super) device_context_id: String, + pub(super) session_pk: String, + pub(super) turn_pk: String, + pub(super) event_id: String, + pub(super) agent_name: String, + pub(super) agent_version: Option, + pub(super) session_id: String, + pub(super) turn_index: i32, + pub(super) llm_provider: String, + pub(super) llm_model: String, + pub(super) event_type: String, + pub(super) text: Option, + pub(super) text_sha256: Option>, + pub(super) input_tokens: i64, + pub(super) output_tokens: i64, + pub(super) cache_read_tokens: i64, + pub(super) cache_write_tokens: i64, + pub(super) reasoning_tokens: i64, + pub(super) total_tokens: i64, + pub(super) observed_at: i64, + pub(super) metadata: String, + pub(super) created_at: i64, +} + +#[derive(Insertable)] +#[diesel(table_name = llm_usage_events)] +pub(super) struct NewEvent { + pub(super) id: String, + pub(super) user_id: String, + pub(super) device_context_id: String, + pub(super) session_pk: String, + pub(super) turn_pk: String, + pub(super) event_id: String, + pub(super) agent_name: String, + pub(super) agent_version: Option, + pub(super) session_id: String, + pub(super) turn_index: i32, + pub(super) llm_provider: String, + pub(super) llm_model: String, + pub(super) event_type: String, + pub(super) text: Option, + pub(super) text_sha256: Option>, + pub(super) input_tokens: i64, + pub(super) output_tokens: i64, + pub(super) cache_read_tokens: i64, + pub(super) cache_write_tokens: i64, + pub(super) reasoning_tokens: i64, + pub(super) total_tokens: i64, + pub(super) observed_at: i64, + pub(super) metadata: String, + pub(super) created_at: i64, +} + +#[derive(Queryable, Selectable)] +#[diesel(table_name = llm_usage_event_attachments)] +#[diesel(check_for_backend(diesel::sqlite::Sqlite))] +pub(super) struct AttachmentRow { + pub(super) id: String, + pub(super) user_id: String, + pub(super) event_pk: String, + pub(super) position: i32, + pub(super) media_type: String, + pub(super) byte_size: i64, + pub(super) sha256: Vec, + pub(super) content: Option>, + pub(super) created_at: i64, +} + +#[derive(Insertable)] +#[diesel(table_name = llm_usage_event_attachments)] +pub(super) struct NewAttachment { + pub(super) id: String, + pub(super) user_id: String, + pub(super) event_pk: String, + pub(super) position: i32, + pub(super) media_type: String, + pub(super) byte_size: i64, + pub(super) sha256: Vec, + pub(super) content: Option>, + pub(super) created_at: i64, +} + +#[derive(Insertable)] +#[diesel(table_name = agent_diagnostic_captures)] +pub(super) struct NewDiagnosticCapture { + pub(super) id: String, + pub(super) user_id: String, + pub(super) device_context_id: String, + pub(super) session_pk: String, + pub(super) capture_id: String, + pub(super) flow_id: String, + pub(super) captured_at: i64, + pub(super) collector_version: String, + pub(super) payload: String, + pub(super) created_at: i64, +} + +#[derive(Insertable)] +#[diesel(table_name = agent_diagnostic_capture_events)] +pub(super) struct NewDiagnosticCaptureEvent { + pub(super) capture_pk: String, + pub(super) event_pk: String, +} + +pub(super) struct DeviceRecord { + pub(super) id: Uuid, + pub(super) host_name: String, + pub(super) platform: String, + pub(super) os_version: Option, +} + +pub(super) struct SessionRecord { + pub(super) id: Uuid, + pub(super) user_id: Uuid, + pub(super) device_context_id: Uuid, + pub(super) agent_name: String, + pub(super) agent_version: Option, + pub(super) session_id: String, + pub(super) started_at: DateTime, + pub(super) ended_at: Option>, + pub(super) metadata: Value, +} + +pub(super) struct TurnRecord { + pub(super) id: Uuid, + pub(super) turn_index: i32, + pub(super) started_at: DateTime, + pub(super) ended_at: Option>, +} + +#[derive(Clone)] +pub(super) struct EventRecord { + pub(super) id: Uuid, + pub(super) user_id: Uuid, + pub(super) device_context_id: Uuid, + pub(super) session_pk: Uuid, + pub(super) turn_pk: Uuid, + pub(super) event_id: String, + pub(super) agent_name: String, + pub(super) agent_version: Option, + pub(super) session_id: String, + pub(super) turn_index: i32, + pub(super) llm_provider: String, + pub(super) llm_model: String, + pub(super) event_type: String, + pub(super) text: Option, + pub(super) text_sha256: Option>, + pub(super) input_tokens: i64, + pub(super) output_tokens: i64, + pub(super) cache_read_tokens: i64, + pub(super) cache_write_tokens: i64, + pub(super) reasoning_tokens: i64, + pub(super) total_tokens: i64, + pub(super) observed_at: DateTime, + pub(super) metadata: Value, +} + +impl TimelineEvent for EventRecord { + fn turn_index(&self) -> i32 { + self.turn_index + } + + fn metadata(&self) -> &Value { + &self.metadata + } + + fn observed_at(&self) -> DateTime { + self.observed_at + } + + fn event_type(&self) -> &str { + &self.event_type + } + + fn event_id(&self) -> &str { + &self.event_id + } +} + +pub(super) struct AttachmentRecord { + pub(super) id: Uuid, + pub(super) event_pk: Uuid, + pub(super) position: i32, + pub(super) media_type: String, + pub(super) byte_size: i64, + pub(super) sha256: Vec, + pub(super) content: Option>, +} + +impl DeviceRow { + pub(super) fn into_record(self) -> Result { + let _ = (&self.user_id, self.created_at, self.updated_at); + Ok(DeviceRecord { + id: parse_uuid(&self.id, "device id")?, + host_name: self.host_name, + platform: self.platform, + os_version: self.os_version, + }) + } +} + +impl SessionRow { + pub(super) fn into_record(self) -> Result { + let _ = (self.created_at, self.updated_at); + Ok(SessionRecord { + id: parse_uuid(&self.id, "session id")?, + user_id: parse_uuid(&self.user_id, "session user id")?, + device_context_id: parse_uuid(&self.device_context_id, "session device id")?, + agent_name: self.agent_name, + agent_version: self.agent_version, + session_id: self.session_id, + started_at: timestamp_from_micros(self.started_at)?, + ended_at: self.ended_at.map(timestamp_from_micros).transpose()?, + metadata: parse_json(&self.metadata, "session metadata")?, + }) + } +} + +impl TurnRow { + pub(super) fn into_record(self) -> Result { + let _ = ( + &self.user_id, + &self.session_pk, + self.created_at, + self.updated_at, + ); + Ok(TurnRecord { + id: parse_uuid(&self.id, "turn id")?, + turn_index: self.turn_index, + started_at: timestamp_from_micros(self.started_at)?, + ended_at: self.ended_at.map(timestamp_from_micros).transpose()?, + }) + } +} + +impl EventRow { + pub(super) fn into_record(self) -> Result { + let _ = self.created_at; + Ok(EventRecord { + id: parse_uuid(&self.id, "event id")?, + user_id: parse_uuid(&self.user_id, "event user id")?, + device_context_id: parse_uuid(&self.device_context_id, "event device id")?, + session_pk: parse_uuid(&self.session_pk, "event session id")?, + turn_pk: parse_uuid(&self.turn_pk, "event turn id")?, + event_id: self.event_id, + agent_name: self.agent_name, + agent_version: self.agent_version, + session_id: self.session_id, + turn_index: self.turn_index, + llm_provider: self.llm_provider, + llm_model: self.llm_model, + event_type: self.event_type, + text: self.text, + text_sha256: self.text_sha256, + input_tokens: self.input_tokens, + output_tokens: self.output_tokens, + cache_read_tokens: self.cache_read_tokens, + cache_write_tokens: self.cache_write_tokens, + reasoning_tokens: self.reasoning_tokens, + total_tokens: self.total_tokens, + observed_at: timestamp_from_micros(self.observed_at)?, + metadata: parse_json(&self.metadata, "event metadata")?, + }) + } +} + +impl AttachmentRow { + pub(super) fn into_record(self) -> Result { + let _ = (&self.user_id, self.created_at); + Ok(AttachmentRecord { + id: parse_uuid(&self.id, "attachment id")?, + event_pk: parse_uuid(&self.event_pk, "attachment event id")?, + position: self.position, + media_type: self.media_type, + byte_size: self.byte_size, + sha256: self.sha256, + content: self.content, + }) + } +} + +pub(super) fn parse_uuid(value: &str, field: &str) -> Result { + Uuid::parse_str(value) + .map_err(|error| AppError::internal(format!("stored {field} is not a UUID: {error}"))) +} + +pub(super) fn timestamp_from_micros(value: i64) -> Result, AppError> { + DateTime::from_timestamp_micros(value).ok_or_else(|| { + AppError::internal(format!( + "stored timestamp microseconds are out of range: {value}" + )) + }) +} + +fn parse_json(value: &str, field: &str) -> Result { + serde_json::from_str(value) + .map_err(|error| AppError::internal(format!("stored {field} is invalid JSON: {error}"))) +} diff --git a/crates/abyss-backend/src/storage/sqlite/repository.rs b/crates/abyss-backend/src/storage/sqlite/repository.rs new file mode 100644 index 0000000..4c24f29 --- /dev/null +++ b/crates/abyss-backend/src/storage/sqlite/repository.rs @@ -0,0 +1,1215 @@ +//! Diesel ORM implementation of SQLite usage ingest and relational queries. + +use std::collections::{HashMap, HashSet}; + +use chrono::{DateTime, Utc}; +use diesel::{ + ExpressionMethods, QueryDsl, RunQueryDsl, SelectableHelper, SqliteConnection, dsl::max, + result::OptionalExtension, sql_query, sql_types::Text, +}; +use uuid::Uuid; + +use crate::{ + error::AppError, + search::projection::SearchProjection, + usage::{ + IngestEventsRequest, IngestEventsResponse, IngestUsageEvent, RawEventsQuery, + RawEventsResponse, SessionInfo, SessionTimelineResponse, SummaryQuery, SummaryResponse, + SummaryRow, TurnTimeline, UsageEventResponse, + attachments::{ + ImageMediaType, StoredImageAttachment, UsageEventAttachmentResponse, + ValidatedImageAttachment, + }, + diagnostics::IngestDiagnosticCapture, + event_order::UsageEventTimelineOrder, + persistence::{ + TurnIdentity, agent_name_filter_values, canonical_agent_name, choose_turn_index, + non_empty, normalize_limit, normalize_slug, normalize_text, normalized_optional, + normalized_tokens, parse_group_by, sha256_bytes, turn_identity, validate_batch, + validate_event_type, + }, + session_metadata_from_event, + }, +}; + +use super::{ + models::{ + AttachmentRow, DeviceRecord, DeviceRow, EventRecord, EventRow, NewAttachment, NewDevice, + NewDiagnosticCapture, NewDiagnosticCaptureEvent, NewEvent, NewSession, NewTurn, SessionRow, + TurnRow, parse_uuid, + }, + schema::{ + agent_diagnostic_capture_events, agent_diagnostic_captures, agent_sessions, agent_turns, + app_users, devices, llm_usage_event_attachments, llm_usage_events, + }, +}; + +pub(super) fn ingest_events( + connection: &mut SqliteConnection, + request: &IngestEventsRequest, + user_id: Uuid, + max_batch_size: usize, +) -> Result { + let validated_attachments = validate_batch(request, max_batch_size)?; + // Reserve the single SQLite writer before allocating turn indexes. + let (accepted, duplicates, accepted_captures, duplicate_captures) = connection + .immediate_transaction(|transaction| { + let mut accepted = 0_usize; + let mut duplicates = 0_usize; + for (event, attachments) in request.events.iter().zip(validated_attachments) { + if ingest_one_event(transaction, user_id, event, attachments)? { + accepted = accepted.saturating_add(1); + } else { + duplicates = duplicates.saturating_add(1); + } + } + + let mut accepted_captures = 0_usize; + let mut duplicate_captures = 0_usize; + for capture in &request.diagnostic_captures { + if ingest_one_diagnostic_capture(transaction, user_id, capture)? { + accepted_captures = accepted_captures.saturating_add(1); + } else { + duplicate_captures = duplicate_captures.saturating_add(1); + } + } + Ok::<_, AppError>((accepted, duplicates, accepted_captures, duplicate_captures)) + })?; + + Ok(IngestEventsResponse { + accepted, + duplicates, + rejected: 0, + errors: Vec::new(), + accepted_diagnostic_captures: accepted_captures, + duplicate_diagnostic_captures: duplicate_captures, + }) +} + +fn ingest_one_event( + connection: &mut SqliteConnection, + user_id: Uuid, + event: &IngestUsageEvent, + attachments: Vec, +) -> Result { + let event_id = normalize_text(&event.event_id); + let exists = llm_usage_events::table + .filter(llm_usage_events::event_id.eq(&event_id)) + .select(llm_usage_events::id) + .first::(connection) + .optional()? + .is_some(); + if exists { + return Ok(false); + } + + let observed_at = timestamp_to_micros(event.observed_at); + let now = timestamp_to_micros(Utc::now()); + let agent_name = canonical_agent_name(&event.agent.name); + let agent_version = normalized_optional(event.agent.version.as_deref()); + let provider = normalize_slug(&event.llm.provider); + let model = normalize_text(&event.llm.model); + let device_id = upsert_device(connection, user_id, event, observed_at, now)?; + let session_pk = upsert_session( + connection, + user_id, + device_id, + event, + &agent_name, + agent_version.as_deref(), + observed_at, + now, + )?; + let turn_index = resolve_turn_index(connection, session_pk, event)?; + let turn_pk = upsert_turn( + connection, + user_id, + session_pk, + turn_index, + observed_at, + now, + )?; + let tokens = normalized_tokens(event)?; + let text = normalized_optional(event.text.as_deref()); + let event_pk = Uuid::now_v7(); + let metadata = serialize_json(&event.metadata, "event metadata")?; + + let inserted = diesel::insert_into(llm_usage_events::table) + .values(NewEvent { + id: event_pk.to_string(), + user_id: user_id.to_string(), + device_context_id: device_id.to_string(), + session_pk: session_pk.to_string(), + turn_pk: turn_pk.to_string(), + event_id, + agent_name, + agent_version, + session_id: normalize_text(&event.session_id), + turn_index, + llm_provider: provider, + llm_model: model, + event_type: event.event_type.as_str().to_owned(), + text: text.clone(), + text_sha256: text.as_deref().map(sha256_bytes), + input_tokens: tokens.input, + output_tokens: tokens.output, + cache_read_tokens: tokens.cache_read, + cache_write_tokens: tokens.cache_write, + reasoning_tokens: tokens.reasoning, + total_tokens: tokens.total, + observed_at, + metadata, + created_at: now, + }) + .on_conflict(llm_usage_events::event_id) + .do_nothing() + .execute(connection)?; + if inserted != 1 { + return Ok(false); + } + + let attachment_rows = attachments + .into_iter() + .map(|attachment| NewAttachment { + id: Uuid::now_v7().to_string(), + user_id: user_id.to_string(), + event_pk: event_pk.to_string(), + position: attachment.position, + media_type: attachment.media_type.as_str().to_owned(), + byte_size: attachment.byte_size, + sha256: attachment.sha256, + content: attachment.content, + created_at: now, + }) + .collect::>(); + if !attachment_rows.is_empty() { + diesel::insert_into(llm_usage_event_attachments::table) + .values(&attachment_rows) + .execute(connection)?; + } + + insert_search_projection(connection, event_pk, user_id, session_pk, event)?; + Ok(true) +} + +fn upsert_device( + connection: &mut SqliteConnection, + user_id: Uuid, + event: &IngestUsageEvent, + observed_at: i64, + now: i64, +) -> Result { + let owner = user_id.to_string(); + let host_name = normalize_text(&event.device.host_name); + let platform = normalize_slug(&event.device.platform); + let os_version = normalized_optional(event.device.os_version.as_deref()); + let existing = devices::table + .filter(devices::user_id.eq(&owner)) + .filter(devices::host_name.eq(&host_name)) + .filter(devices::platform.eq(&platform)) + .select(DeviceRow::as_select()) + .first::(connection) + .optional()?; + if let Some(existing) = existing { + let id = parse_uuid(&existing.id, "device id")?; + diesel::update(devices::table.find(existing.id)) + .set(( + devices::os_version.eq(os_version.or(existing.os_version)), + devices::first_seen_at.eq(existing.first_seen_at.min(observed_at)), + devices::last_seen_at.eq(existing.last_seen_at.max(observed_at)), + devices::updated_at.eq(now), + )) + .execute(connection)?; + return Ok(id); + } + + let id = Uuid::now_v7(); + diesel::insert_into(devices::table) + .values(NewDevice { + id: id.to_string(), + user_id: owner, + host_name, + platform, + os_version, + first_seen_at: observed_at, + last_seen_at: observed_at, + created_at: now, + updated_at: now, + }) + .execute(connection)?; + Ok(id) +} + +#[expect( + clippy::too_many_arguments, + reason = "The session upsert receives one explicit value for every evolving boundary." +)] +fn upsert_session( + connection: &mut SqliteConnection, + user_id: Uuid, + device_id: Uuid, + event: &IngestUsageEvent, + agent_name: &str, + agent_version: Option<&str>, + observed_at: i64, + now: i64, +) -> Result { + let owner = user_id.to_string(); + let session_id = normalize_text(&event.session_id); + let existing = agent_sessions::table + .filter(agent_sessions::user_id.eq(&owner)) + .filter(agent_sessions::agent_name.eq(agent_name)) + .filter(agent_sessions::session_id.eq(&session_id)) + .select(SessionRow::as_select()) + .first::(connection) + .optional()?; + let incoming_metadata = session_metadata_from_event(&event.metadata); + if let Some(existing) = existing { + let id = parse_uuid(&existing.id, "session id")?; + let metadata = merge_json(&existing.metadata, &incoming_metadata, "session metadata")?; + diesel::update(agent_sessions::table.find(existing.id)) + .set(( + agent_sessions::device_context_id.eq(device_id.to_string()), + agent_sessions::agent_version + .eq(agent_version.map(str::to_owned).or(existing.agent_version)), + agent_sessions::started_at.eq(existing.started_at.min(observed_at)), + agent_sessions::ended_at.eq(Some( + existing.ended_at.unwrap_or(observed_at).max(observed_at), + )), + agent_sessions::metadata.eq(metadata), + agent_sessions::updated_at.eq(now), + )) + .execute(connection)?; + return Ok(id); + } + + let id = Uuid::now_v7(); + diesel::insert_into(agent_sessions::table) + .values(NewSession { + id: id.to_string(), + user_id: owner, + device_context_id: device_id.to_string(), + agent_name: agent_name.to_owned(), + agent_version: agent_version.map(str::to_owned), + session_id, + started_at: observed_at, + ended_at: Some(observed_at), + metadata: serialize_json(&incoming_metadata, "session metadata")?, + created_at: now, + updated_at: now, + }) + .execute(connection)?; + Ok(id) +} + +fn resolve_turn_index( + connection: &mut SqliteConnection, + session_pk: Uuid, + event: &IngestUsageEvent, +) -> Result { + let Some(identity) = turn_identity(event) else { + return Ok(event.turn_index); + }; + let existing = existing_turn_index_for_identity(connection, session_pk, &identity)?; + let next = agent_turns::table + .filter(agent_turns::session_pk.eq(session_pk.to_string())) + .select(max(agent_turns::turn_index)) + .first::>(connection)? + .unwrap_or(0_i32) + .saturating_add(1); + Ok(choose_turn_index(event.turn_index, existing, next)) +} + +fn existing_turn_index_for_identity( + connection: &mut SqliteConnection, + session_pk: Uuid, + identity: &TurnIdentity, +) -> Result, AppError> { + let rows = llm_usage_events::table + .filter(llm_usage_events::session_pk.eq(session_pk.to_string())) + .order_by(( + llm_usage_events::observed_at.asc(), + llm_usage_events::id.asc(), + )) + .select((llm_usage_events::turn_index, llm_usage_events::metadata)) + .load::<(i32, String)>(connection)?; + for (turn_index, metadata) in rows { + let metadata = serde_json::from_str::(&metadata).map_err(|error| { + AppError::internal(format!("stored event metadata is invalid JSON: {error}")) + })?; + if metadata + .get(identity.kind.metadata_key()) + .and_then(serde_json::Value::as_str) + == Some(identity.value.as_str()) + { + return Ok(Some(turn_index)); + } + } + Ok(None) +} + +fn upsert_turn( + connection: &mut SqliteConnection, + user_id: Uuid, + session_pk: Uuid, + turn_index: i32, + observed_at: i64, + now: i64, +) -> Result { + let session = session_pk.to_string(); + let existing = agent_turns::table + .filter(agent_turns::session_pk.eq(&session)) + .filter(agent_turns::turn_index.eq(turn_index)) + .select(TurnRow::as_select()) + .first::(connection) + .optional()?; + if let Some(existing) = existing { + let id = parse_uuid(&existing.id, "turn id")?; + diesel::update(agent_turns::table.find(existing.id)) + .set(( + agent_turns::started_at.eq(existing.started_at.min(observed_at)), + agent_turns::ended_at.eq(Some( + existing.ended_at.unwrap_or(observed_at).max(observed_at), + )), + agent_turns::updated_at.eq(now), + )) + .execute(connection)?; + return Ok(id); + } + + let id = Uuid::now_v7(); + diesel::insert_into(agent_turns::table) + .values(NewTurn { + id: id.to_string(), + user_id: user_id.to_string(), + session_pk: session, + turn_index, + started_at: observed_at, + ended_at: Some(observed_at), + created_at: now, + updated_at: now, + }) + .execute(connection)?; + Ok(id) +} + +/// Writes the SQLite FTS5 projection. Virtual tables are intentionally outside +/// Diesel's relational schema and use a fully bound raw statement. +fn insert_search_projection( + connection: &mut SqliteConnection, + event_pk: Uuid, + user_id: Uuid, + session_pk: Uuid, + event: &IngestUsageEvent, +) -> Result<(), AppError> { + let projection = + SearchProjection::from_source(normalized_optional(event.text.as_deref()), &event.metadata); + sql_query( + "INSERT INTO usage_events_fts ( + event_pk, user_id, session_pk, session_id, content, tool_names, + tool_content, commands, file_paths + ) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?)", + ) + .bind::(event_pk.to_string()) + .bind::(user_id.to_string()) + .bind::(session_pk.to_string()) + .bind::(normalize_text(&event.session_id)) + .bind::, _>(projection.content) + .bind::(projection.tool_names.join("\n")) + .bind::(projection.tool_content.join("\n")) + .bind::(projection.commands.join("\n")) + .bind::(projection.file_paths.join("\n")) + .execute(connection)?; + Ok(()) +} + +fn ingest_one_diagnostic_capture( + connection: &mut SqliteConnection, + user_id: Uuid, + capture: &IngestDiagnosticCapture, +) -> Result { + let event_ids = capture + .event_ids + .iter() + .map(|event_id| normalize_text(event_id)) + .collect::>(); + let events = llm_usage_events::table + .filter(llm_usage_events::user_id.eq(user_id.to_string())) + .filter(llm_usage_events::event_id.eq_any(&event_ids)) + .select(( + llm_usage_events::id, + llm_usage_events::device_context_id, + llm_usage_events::session_pk, + )) + .load::<(String, String, String)>(connection)?; + if events.len() != capture.event_ids.len() { + return Err(AppError::validation( + "diagnostic capture events were not ingested for the authenticated user".to_owned(), + )); + } + let Some((_, device_id, session_pk)) = events.first() else { + return Err(AppError::validation( + "diagnostic capture must reference at least one event".to_owned(), + )); + }; + if events + .iter() + .any(|(_, device, session)| device != device_id || session != session_pk) + { + return Err(AppError::validation( + "diagnostic capture events must belong to one session and device".to_owned(), + )); + } + + let capture_pk = Uuid::now_v7().to_string(); + let inserted = diesel::insert_into(agent_diagnostic_captures::table) + .values(NewDiagnosticCapture { + id: capture_pk.clone(), + user_id: user_id.to_string(), + device_context_id: device_id.clone(), + session_pk: session_pk.clone(), + capture_id: normalize_text(&capture.capture_id), + flow_id: normalize_text(&capture.flow_id), + captured_at: timestamp_to_micros(capture.captured_at), + collector_version: normalize_text(&capture.collector_version), + payload: serialize_json(&capture.payload, "diagnostic payload")?, + created_at: timestamp_to_micros(Utc::now()), + }) + .on_conflict(( + agent_diagnostic_captures::user_id, + agent_diagnostic_captures::capture_id, + )) + .do_nothing() + .execute(connection)?; + if inserted == 1 { + let links = events + .into_iter() + .map(|(event_pk, _, _)| NewDiagnosticCaptureEvent { + capture_pk: capture_pk.clone(), + event_pk, + }) + .collect::>(); + diesel::insert_into(agent_diagnostic_capture_events::table) + .values(&links) + .execute(connection)?; + } + Ok(inserted == 1) +} + +pub(super) fn raw_events( + connection: &mut SqliteConnection, + query: &RawEventsQuery, + user_id: Uuid, + default_limit: i64, +) -> Result { + let page_size = normalize_limit(query.limit, default_limit, 1_000); + let page_size_usize = usize::try_from(page_size) + .map_err(|error| AppError::internal(format!("invalid raw events page size: {error}")))?; + let offset = query.offset.unwrap_or(0).max(0); + let mut events = load_filtered_events( + connection, + &EventFilters { + user_id, + from: query.from, + to: query.to, + agent_name: query.agent_name.as_deref(), + session_id: query.session_id.as_deref(), + session_pk: query.session_pk, + turn_index: query.turn_index, + llm_provider: query.llm_provider.as_deref(), + llm_model: query.llm_model.as_deref(), + event_type: query.event_type.as_deref(), + limit: page_size.saturating_add(1), + offset, + }, + )?; + let has_next_page = events.len() > page_size_usize; + if has_next_page { + events.truncate(page_size_usize); + } + let devices = load_device_contexts(connection, &events)?; + let mut attachments = load_attachments(connection, &events)?; + let responses = events + .into_iter() + .map(|event| { + let event_pk = event.id; + let device_id = event.device_context_id; + event_response( + event, + devices.get(&device_id), + attachments.remove(&event_pk).unwrap_or_default(), + ) + }) + .collect(); + Ok(RawEventsResponse { + events: responses, + next_page_token: has_next_page.then(|| offset.saturating_add(page_size).to_string()), + }) +} + +struct EventFilters<'a> { + user_id: Uuid, + from: Option>, + to: Option>, + agent_name: Option<&'a str>, + session_id: Option<&'a str>, + session_pk: Option, + turn_index: Option, + llm_provider: Option<&'a str>, + llm_model: Option<&'a str>, + event_type: Option<&'a str>, + limit: i64, + offset: i64, +} + +fn load_filtered_events( + connection: &mut SqliteConnection, + filters: &EventFilters<'_>, +) -> Result, AppError> { + let mut query = llm_usage_events::table + .filter(llm_usage_events::user_id.eq(filters.user_id.to_string())) + .into_boxed::(); + if let Some(from) = filters.from { + query = query.filter(llm_usage_events::observed_at.ge(timestamp_to_micros(from))); + } + if let Some(to) = filters.to { + query = query.filter(llm_usage_events::observed_at.lt(timestamp_to_micros(to))); + } + if let Some(agent_name) = filters.agent_name.and_then(non_empty) { + query = + query.filter(llm_usage_events::agent_name.eq_any(agent_name_filter_values(agent_name))); + } + if let Some(session_id) = filters.session_id.and_then(non_empty) { + query = query.filter(llm_usage_events::session_id.eq(session_id.trim())); + } + if let Some(session_pk) = filters.session_pk { + query = query.filter(llm_usage_events::session_pk.eq(session_pk.to_string())); + } + if let Some(turn_index) = filters.turn_index { + query = query.filter(llm_usage_events::turn_index.eq(turn_index)); + } + if let Some(provider) = filters.llm_provider.and_then(non_empty) { + query = query.filter(llm_usage_events::llm_provider.eq(normalize_slug(provider))); + } + if let Some(model) = filters.llm_model.and_then(non_empty) { + query = query.filter(llm_usage_events::llm_model.eq(model.trim())); + } + if let Some(event_type) = filters.event_type.and_then(non_empty) { + query = query.filter(llm_usage_events::event_type.eq(validate_event_type(event_type)?)); + } + query + .order_by(( + llm_usage_events::observed_at.desc(), + llm_usage_events::id.desc(), + )) + .limit(filters.limit) + .offset(filters.offset) + .select(EventRow::as_select()) + .load::(connection)? + .into_iter() + .map(EventRow::into_record) + .collect() +} + +pub(super) fn session_timeline( + connection: &mut SqliteConnection, + user_id: Uuid, + session_pk: Uuid, +) -> Result { + let session = agent_sessions::table + .filter(agent_sessions::id.eq(session_pk.to_string())) + .filter(agent_sessions::user_id.eq(user_id.to_string())) + .select(SessionRow::as_select()) + .first::(connection) + .optional()? + .ok_or_else(|| AppError::not_found(format!("session not found: {session_pk}")))? + .into_record()?; + let turns = agent_turns::table + .filter(agent_turns::session_pk.eq(session.id.to_string())) + .order_by(agent_turns::turn_index.asc()) + .select(TurnRow::as_select()) + .load::(connection)? + .into_iter() + .map(TurnRow::into_record) + .collect::, _>>()?; + let mut events = llm_usage_events::table + .filter(llm_usage_events::session_pk.eq(session.id.to_string())) + .order_by(( + llm_usage_events::turn_index.asc(), + llm_usage_events::observed_at.asc(), + llm_usage_events::id.asc(), + )) + .select(EventRow::as_select()) + .load::(connection)? + .into_iter() + .map(EventRow::into_record) + .collect::, _>>()?; + UsageEventTimelineOrder::sort(&mut events); + let devices = load_device_contexts(connection, &events)?; + let mut attachments = load_attachments(connection, &events)?; + let mut events_by_turn: HashMap> = HashMap::new(); + for event in events { + let event_pk = event.id; + let device_id = event.device_context_id; + events_by_turn + .entry(event.turn_pk) + .or_default() + .push(event_response( + event, + devices.get(&device_id), + attachments.remove(&event_pk).unwrap_or_default(), + )); + } + let turn_timelines = turns + .into_iter() + .map(|turn| TurnTimeline { + turn_pk: turn.id, + turn_index: turn.turn_index, + started_at: turn.started_at, + ended_at: turn.ended_at, + events: events_by_turn.remove(&turn.id).unwrap_or_default(), + }) + .collect(); + Ok(SessionTimelineResponse { + session: SessionInfo { + session_pk: session.id, + user_id: session.user_id, + device_context_id: session.device_context_id, + agent_name: canonical_agent_name(&session.agent_name), + agent_version: session.agent_version, + session_id: session.session_id, + started_at: session.started_at, + ended_at: session.ended_at, + metadata: session.metadata, + }, + turns: turn_timelines, + }) +} + +pub(super) fn image_attachment( + connection: &mut SqliteConnection, + user_id: Uuid, + attachment_id: Uuid, +) -> Result { + let row = llm_usage_event_attachments::table + .filter(llm_usage_event_attachments::id.eq(attachment_id.to_string())) + .filter(llm_usage_event_attachments::user_id.eq(user_id.to_string())) + .select(AttachmentRow::as_select()) + .first::(connection) + .optional()? + .ok_or_else(|| AppError::not_found(format!("image attachment not found: {attachment_id}")))? + .into_record()?; + let content = row.content.ok_or_else(|| { + AppError::not_found(format!( + "image attachment content not found: {attachment_id}" + )) + })?; + if i64::try_from(content.len()).ok() != Some(row.byte_size) { + return Err(AppError::internal(format!( + "stored image attachment byte size mismatch: {attachment_id}" + ))); + } + Ok(StoredImageAttachment { + media_type: ImageMediaType::from_stored(&row.media_type)?, + sha256: hex::encode(row.sha256), + content, + }) +} + +pub(super) fn usage_summary( + connection: &mut SqliteConnection, + query: &SummaryQuery, + user_id: Uuid, + scan_limit: i64, +) -> Result { + let group_by = parse_group_by(query.group_by.as_deref()); + let dimensions = SummaryDimensions::from_group_by(&group_by); + let owner = load_owner(connection, user_id)?; + if !owner_matches_filter(user_id, &owner, query.user_filter.as_deref()) { + return Ok(empty_summary(query, group_by)); + } + let events = load_filtered_events( + connection, + &EventFilters { + user_id, + from: query.from, + to: query.to, + agent_name: query.agent_name.as_deref(), + session_id: query.session_id.as_deref(), + session_pk: None, + turn_index: None, + llm_provider: query.llm_provider.as_deref(), + llm_model: query.llm_model.as_deref(), + event_type: query.event_type.as_deref(), + limit: scan_limit, + offset: 0, + }, + )?; + let devices = load_device_contexts(connection, &events)?; + let mut buckets: HashMap = HashMap::new(); + for event in events { + let device = devices.get(&event.device_context_id); + let key = SummaryKey::new(&event, device, user_id, &owner, &dimensions); + buckets.entry(key).or_default().add(&event); + } + let mut rows = buckets + .into_iter() + .map(|(key, bucket)| bucket.into_row(key)) + .collect::>(); + rows.sort_by(|left, right| { + right + .total_tokens + .cmp(&left.total_tokens) + .then_with(|| left.agent_name.cmp(&right.agent_name)) + .then_with(|| left.user_email.cmp(&right.user_email)) + .then_with(|| left.day.cmp(&right.day)) + }); + let row_limit = usize::try_from(normalize_limit(query.limit, scan_limit, 100_000)) + .map_err(|error| AppError::internal(format!("invalid summary row limit: {error}")))?; + rows.truncate(row_limit); + Ok(SummaryResponse { + from: query.from, + to: query.to, + group_by, + rows, + next_page_token: None, + }) +} + +struct OwnerRecord { + email: String, + name: Option, +} + +fn load_owner(connection: &mut SqliteConnection, user_id: Uuid) -> Result { + let (email, name) = app_users::table + .find(user_id.to_string()) + .select((app_users::email, app_users::name)) + .first::<(String, Option)>(connection)?; + Ok(OwnerRecord { email, name }) +} + +fn owner_matches_filter(user_id: Uuid, owner: &OwnerRecord, user_filter: Option<&str>) -> bool { + let Some(filter) = user_filter.and_then(non_empty) else { + return true; + }; + if let Ok(filter_id) = Uuid::parse_str(filter.trim()) { + return filter_id == user_id; + } + let filter = filter.trim().to_lowercase(); + owner.email.to_lowercase().contains(&filter) + || owner + .name + .as_deref() + .unwrap_or_default() + .to_lowercase() + .contains(&filter) +} + +const fn empty_summary(query: &SummaryQuery, group_by: Vec) -> SummaryResponse { + SummaryResponse { + from: query.from, + to: query.to, + group_by, + rows: Vec::new(), + next_page_token: None, + } +} + +struct SummaryDimensions([bool; 7]); + +#[derive(Clone, Copy)] +enum SummaryDimension { + Day, + User, + Device, + Agent, + Provider, + Model, + EventType, +} + +impl SummaryDimension { + const fn index(self) -> usize { + match self { + Self::Day => 0, + Self::User => 1, + Self::Device => 2, + Self::Agent => 3, + Self::Provider => 4, + Self::Model => 5, + Self::EventType => 6, + } + } +} + +impl SummaryDimensions { + fn from_group_by(group_by: &[String]) -> Self { + let includes = |name: &str| group_by.iter().any(|value| value == name); + Self([ + includes("day"), + includes("user"), + includes("device"), + includes("agent"), + includes("provider"), + includes("model"), + includes("event_type"), + ]) + } + + const fn enabled(&self, dimension: SummaryDimension) -> bool { + self.0[dimension.index()] + } +} + +#[derive(Hash, Eq, PartialEq)] +struct SummaryKey { + day: Option, + user_id: Option, + user_name: Option, + user_email: Option, + host_name: Option, + platform: Option, + os_version: Option, + agent_name: Option, + llm_provider: Option, + llm_model: Option, + event_type: Option, +} + +impl SummaryKey { + fn new( + event: &EventRecord, + device: Option<&DeviceRecord>, + user_id: Uuid, + owner: &OwnerRecord, + dimensions: &SummaryDimensions, + ) -> Self { + let include_user = dimensions.enabled(SummaryDimension::User); + let include_device = dimensions.enabled(SummaryDimension::Device); + Self { + day: dimensions + .enabled(SummaryDimension::Day) + .then(|| event.observed_at.format("%Y-%m-%d").to_string()), + user_id: include_user.then_some(user_id), + user_name: include_user.then(|| owner.name.clone()).flatten(), + user_email: include_user.then(|| owner.email.clone()), + host_name: include_device + .then(|| device.map(|value| value.host_name.clone())) + .flatten(), + platform: include_device + .then(|| device.map(|value| value.platform.clone())) + .flatten(), + os_version: include_device + .then(|| device.and_then(|value| value.os_version.clone())) + .flatten(), + agent_name: dimensions + .enabled(SummaryDimension::Agent) + .then(|| canonical_agent_name(&event.agent_name)), + llm_provider: dimensions + .enabled(SummaryDimension::Provider) + .then(|| event.llm_provider.clone()), + llm_model: dimensions + .enabled(SummaryDimension::Model) + .then(|| event.llm_model.clone()), + event_type: dimensions + .enabled(SummaryDimension::EventType) + .then(|| event.event_type.clone()), + } + } +} + +#[derive(Default)] +struct SummaryBucket { + sessions: HashSet, + turns: HashSet, + requests: i64, + responses: i64, + input_tokens: i64, + output_tokens: i64, + cache_read_tokens: i64, + cache_write_tokens: i64, + reasoning_tokens: i64, + total_tokens: i64, +} + +impl SummaryBucket { + fn add(&mut self, event: &EventRecord) { + self.sessions.insert(event.session_pk); + self.turns.insert(event.turn_pk); + self.requests = self + .requests + .saturating_add(i64::from(event.event_type == "request")); + self.responses = self + .responses + .saturating_add(i64::from(event.event_type == "response")); + self.input_tokens = self.input_tokens.saturating_add(event.input_tokens); + self.output_tokens = self.output_tokens.saturating_add(event.output_tokens); + self.cache_read_tokens = self + .cache_read_tokens + .saturating_add(event.cache_read_tokens); + self.cache_write_tokens = self + .cache_write_tokens + .saturating_add(event.cache_write_tokens); + self.reasoning_tokens = self.reasoning_tokens.saturating_add(event.reasoning_tokens); + self.total_tokens = self.total_tokens.saturating_add(event.total_tokens); + } + + fn into_row(self, key: SummaryKey) -> SummaryRow { + SummaryRow { + day: key.day, + user_id: key.user_id, + user_name: key.user_name, + user_email: key.user_email, + host_name: key.host_name, + platform: key.platform, + os_version: key.os_version, + agent_name: key.agent_name, + llm_provider: key.llm_provider, + llm_model: key.llm_model, + event_type: key.event_type, + sessions: self.sessions.len(), + turns: self.turns.len(), + requests: self.requests, + responses: self.responses, + input_tokens: self.input_tokens, + output_tokens: self.output_tokens, + cache_read_tokens: self.cache_read_tokens, + cache_write_tokens: self.cache_write_tokens, + reasoning_tokens: self.reasoning_tokens, + total_tokens: self.total_tokens, + } + } +} + +fn load_device_contexts( + connection: &mut SqliteConnection, + events: &[EventRecord], +) -> Result, AppError> { + let ids = events + .iter() + .map(|event| event.device_context_id.to_string()) + .collect::>(); + if ids.is_empty() { + return Ok(HashMap::new()); + } + let devices = devices::table + .filter(devices::id.eq_any(ids)) + .select(DeviceRow::as_select()) + .load::(connection)? + .into_iter() + .map(DeviceRow::into_record) + .collect::, _>>()?; + Ok(devices + .into_iter() + .map(|device| (device.id, device)) + .collect()) +} + +fn load_attachments( + connection: &mut SqliteConnection, + events: &[EventRecord], +) -> Result>, AppError> { + if events.is_empty() { + return Ok(HashMap::new()); + } + let event_ids = events + .iter() + .map(|event| event.id.to_string()) + .collect::>(); + let rows = llm_usage_event_attachments::table + .filter(llm_usage_event_attachments::event_pk.eq_any(event_ids)) + .order_by(( + llm_usage_event_attachments::event_pk.asc(), + llm_usage_event_attachments::position.asc(), + )) + .select(AttachmentRow::as_select()) + .load::(connection)? + .into_iter() + .map(AttachmentRow::into_record) + .collect::, _>>()?; + let mut by_event = HashMap::new(); + for row in rows { + by_event + .entry(row.event_pk) + .or_insert_with(Vec::new) + .push(UsageEventAttachmentResponse { + id: row.id, + position: row.position, + media_type: ImageMediaType::from_stored(&row.media_type)?, + byte_size: row.byte_size, + sha256: hex::encode(row.sha256), + content_available: row.content.is_some(), + }); + } + Ok(by_event) +} + +fn event_response( + event: EventRecord, + device: Option<&DeviceRecord>, + attachments: Vec, +) -> UsageEventResponse { + UsageEventResponse { + id: event.id, + event_id: event.event_id, + user_id: event.user_id, + device_context_id: event.device_context_id, + host_name: device.map(|value| value.host_name.clone()), + platform: device.map(|value| value.platform.clone()), + session_pk: event.session_pk, + turn_pk: event.turn_pk, + agent_name: canonical_agent_name(&event.agent_name), + agent_version: event.agent_version, + session_id: event.session_id, + turn_index: event.turn_index, + llm_provider: event.llm_provider, + llm_model: event.llm_model, + event_type: event.event_type, + text: event.text, + text_sha256: event.text_sha256.map(hex::encode), + input_tokens: event.input_tokens, + output_tokens: event.output_tokens, + cache_read_tokens: event.cache_read_tokens, + cache_write_tokens: event.cache_write_tokens, + reasoning_tokens: event.reasoning_tokens, + total_tokens: event.total_tokens, + observed_at: event.observed_at, + metadata: event.metadata, + attachments, + } +} + +fn merge_json(stored: &str, incoming: &serde_json::Value, field: &str) -> Result { + let mut stored = serde_json::from_str::(stored) + .map_err(|error| AppError::internal(format!("stored {field} is invalid JSON: {error}")))?; + if let (Some(stored), Some(incoming)) = (stored.as_object_mut(), incoming.as_object()) { + stored.extend(incoming.clone()); + } + serialize_json(&stored, field) +} + +fn serialize_json(value: &serde_json::Value, field: &str) -> Result { + serde_json::to_string(value) + .map_err(|error| AppError::internal(format!("serialize {field}: {error}"))) +} + +pub(super) const fn timestamp_to_micros(value: DateTime) -> i64 { + value.timestamp_micros() +} + +#[cfg(test)] +mod tests { + use diesel::{Connection, QueryDsl, RunQueryDsl, SqliteConnection, dsl::count_star, sql_query}; + use serde_json::json; + use uuid::Uuid; + + use crate::{search::SessionSearchQuery, usage::IngestEventsRequest}; + + use super::{ + super::{migrations, schema::llm_usage_events, search}, + ingest_events, + }; + + const OWNER_ID: Uuid = Uuid::from_u128(1); + + #[derive(diesel::QueryableByName)] + struct Count { + #[diesel(sql_type = diesel::sql_types::BigInt)] + count: i64, + } + + fn database() -> SqliteConnection { + let mut database = SqliteConnection::establish(":memory:").expect("SQLite should open"); + migrations::run(&mut database).expect("migration should succeed"); + database + } + + #[test] + fn ingest_is_idempotent_and_searchable_in_the_same_transaction() { + let mut database = database(); + let request = request_with_text("sqlite searchable response"); + + let first = ingest_events(&mut database, &request, OWNER_ID, 100) + .expect("first ingest should succeed"); + assert_eq!(first.accepted, 1); + assert_eq!(first.duplicates, 0); + let duplicate = ingest_events(&mut database, &request, OWNER_ID, 100) + .expect("duplicate ingest should succeed"); + assert_eq!(duplicate.accepted, 0); + assert_eq!(duplicate.duplicates, 1); + + let projection_count = sql_query("SELECT count(*) AS count FROM usage_events_fts") + .get_result::(&mut database) + .expect("FTS projection should be queryable") + .count; + assert_eq!(projection_count, 1); + let results = search::session_search( + &mut database, + OWNER_ID, + SessionSearchQuery { + q: "searchable response".to_owned(), + from: None, + to: None, + agent_name: Some("codex".to_owned()), + llm_provider: None, + llm_model: None, + event_type: None, + page: None, + page_size: None, + }, + ) + .expect("committed event should be searchable"); + assert_eq!(results.total_sessions, 1); + assert_eq!(results.items.len(), 1); + assert_eq!(results.items[0].match_count, 1); + assert!( + results.items[0].matches[0] + .fragments + .iter() + .flat_map(|fragment| &fragment.segments) + .any(|segment| segment.highlighted), + "FTS result should contain a structured highlight" + ); + } + + #[test] + fn validation_failure_leaves_relational_and_fts_tables_empty() { + let mut database = database(); + let mut request = request_with_text("invalid event"); + request.events[0].token_usage.input_tokens = Some(-1); + + ingest_events(&mut database, &request, OWNER_ID, 100) + .expect_err("negative tokens should reject the batch"); + let relational_count = llm_usage_events::table + .select(count_star()) + .first::(&mut database) + .expect("relational table should be queryable"); + assert_eq!(relational_count, 0); + + let fts_count = sql_query("SELECT count(*) AS count FROM usage_events_fts") + .get_result::(&mut database) + .expect("FTS table should be queryable") + .count; + assert_eq!(fts_count, 0); + } + + fn request_with_text(text: &str) -> IngestEventsRequest { + serde_json::from_value(json!({ + "events": [{ + "event_id": "sqlite-event", + "observed_at": "2026-08-27T00:00:00Z", + "device": {"host_name": "local", "platform": "macos"}, + "agent": {"name": "Codex"}, + "session_id": "sqlite-session", + "turn_index": 1, + "llm": {"provider": "OpenAI", "model": "gpt-5"}, + "event_type": "response", + "text": text, + "token_usage": {"output_tokens": 3}, + "metadata": {"response_id": "sqlite-response"} + }] + })) + .expect("test ingest request should deserialize") + } +} diff --git a/crates/abyss-backend/src/storage/sqlite/schema.rs b/crates/abyss-backend/src/storage/sqlite/schema.rs new file mode 100644 index 0000000..80531fe --- /dev/null +++ b/crates/abyss-backend/src/storage/sqlite/schema.rs @@ -0,0 +1,162 @@ +//! Diesel schema for SQLite relational tables; the FTS5 virtual table stays raw SQL. + +diesel::table! { + use diesel::sql_types::{BigInt, Nullable, Text}; + + app_users (id) { + id -> Text, + email -> Text, + name -> Nullable, + created_at -> BigInt, + } +} + +diesel::table! { + use diesel::sql_types::{BigInt, Nullable, Text}; + + devices (id) { + id -> Text, + user_id -> Text, + host_name -> Text, + platform -> Text, + os_version -> Nullable, + first_seen_at -> BigInt, + last_seen_at -> BigInt, + created_at -> BigInt, + updated_at -> BigInt, + } +} + +diesel::table! { + use diesel::sql_types::{BigInt, Nullable, Text}; + + agent_sessions (id) { + id -> Text, + user_id -> Text, + device_context_id -> Text, + agent_name -> Text, + agent_version -> Nullable, + session_id -> Text, + started_at -> BigInt, + ended_at -> Nullable, + metadata -> Text, + created_at -> BigInt, + updated_at -> BigInt, + } +} + +diesel::table! { + use diesel::sql_types::{BigInt, Integer, Nullable, Text}; + + agent_turns (id) { + id -> Text, + user_id -> Text, + session_pk -> Text, + turn_index -> Integer, + started_at -> BigInt, + ended_at -> Nullable, + created_at -> BigInt, + updated_at -> BigInt, + } +} + +diesel::table! { + use diesel::sql_types::{BigInt, Binary, Integer, Nullable, Text}; + + llm_usage_events (id) { + id -> Text, + user_id -> Text, + device_context_id -> Text, + session_pk -> Text, + turn_pk -> Text, + event_id -> Text, + agent_name -> Text, + agent_version -> Nullable, + session_id -> Text, + turn_index -> Integer, + llm_provider -> Text, + llm_model -> Text, + event_type -> Text, + text -> Nullable, + text_sha256 -> Nullable, + input_tokens -> BigInt, + output_tokens -> BigInt, + cache_read_tokens -> BigInt, + cache_write_tokens -> BigInt, + reasoning_tokens -> BigInt, + total_tokens -> BigInt, + observed_at -> BigInt, + metadata -> Text, + created_at -> BigInt, + } +} + +diesel::table! { + use diesel::sql_types::{BigInt, Binary, Integer, Nullable, Text}; + + llm_usage_event_attachments (id) { + id -> Text, + user_id -> Text, + event_pk -> Text, + position -> Integer, + media_type -> Text, + byte_size -> BigInt, + sha256 -> Binary, + content -> Nullable, + created_at -> BigInt, + } +} + +diesel::table! { + use diesel::sql_types::{BigInt, Text}; + + agent_diagnostic_captures (id) { + id -> Text, + user_id -> Text, + device_context_id -> Text, + session_pk -> Text, + capture_id -> Text, + flow_id -> Text, + captured_at -> BigInt, + collector_version -> Text, + payload -> Text, + created_at -> BigInt, + } +} + +diesel::table! { + use diesel::sql_types::Text; + + agent_diagnostic_capture_events (capture_pk, event_pk) { + capture_pk -> Text, + event_pk -> Text, + } +} + +diesel::joinable!(agent_sessions -> app_users (user_id)); +diesel::joinable!(agent_sessions -> devices (device_context_id)); +diesel::joinable!(agent_turns -> agent_sessions (session_pk)); +diesel::joinable!(agent_turns -> app_users (user_id)); +diesel::joinable!(devices -> app_users (user_id)); +diesel::joinable!(llm_usage_events -> agent_sessions (session_pk)); +diesel::joinable!(llm_usage_events -> agent_turns (turn_pk)); +diesel::joinable!(llm_usage_events -> app_users (user_id)); +diesel::joinable!(llm_usage_events -> devices (device_context_id)); +diesel::joinable!(llm_usage_event_attachments -> app_users (user_id)); +diesel::joinable!(llm_usage_event_attachments -> llm_usage_events (event_pk)); +diesel::joinable!(agent_diagnostic_capture_events -> agent_diagnostic_captures (capture_pk)); +diesel::joinable!(agent_diagnostic_capture_events -> llm_usage_events (event_pk)); +diesel::joinable!(agent_diagnostic_captures -> agent_sessions (session_pk)); +diesel::joinable!(agent_diagnostic_captures -> app_users (user_id)); +diesel::joinable!(agent_diagnostic_captures -> devices (device_context_id)); + +diesel::allow_tables_to_appear_in_same_query!( + agent_diagnostic_capture_events, + agent_diagnostic_captures, + agent_sessions, + agent_turns, + app_users, + devices, + llm_usage_events, + llm_usage_event_attachments, +); diff --git a/crates/abyss-backend/src/storage/sqlite/search.rs b/crates/abyss-backend/src/storage/sqlite/search.rs new file mode 100644 index 0000000..bdd4ead --- /dev/null +++ b/crates/abyss-backend/src/storage/sqlite/search.rs @@ -0,0 +1,488 @@ +//! SQLite FTS5 raw queries with Diesel ORM hydration of relational session data. + +use std::collections::{HashMap, HashSet}; + +use chrono::{DateTime, Utc}; +use diesel::{ + ExpressionMethods, QueryDsl, QueryableByName, RunQueryDsl, SqliteConnection, + query_builder::{BoxedSqlQuery, SqlQuery}, + sql_query, + sql_types::{BigInt, Integer, Nullable, Text}, + sqlite::Sqlite, +}; +use uuid::Uuid; + +use crate::{ + error::AppError, + search::{ + HIGHLIGHT_END, HIGHLIGHT_START, SearchFragment, SessionSearchMatch, SessionSearchQuery, + SessionSearchResponse, SessionSearchResult, ValidatedSearchQuery, + }, + usage::persistence::canonical_agent_name, +}; + +use super::{ + models::{parse_uuid, timestamp_from_micros}, + repository::timestamp_to_micros, + schema::{agent_sessions, devices}, +}; + +pub(super) fn session_search( + connection: &mut SqliteConnection, + user_id: Uuid, + query: SessionSearchQuery, +) -> Result { + let query = query.validate()?; + let source = MatchSource::new(&query, user_id); + let total_sessions = count_sessions(connection, &source)?; + let session_ids = load_session_page(connection, &query, &source)?; + let matches = load_matches(connection, &session_ids, &source)?; + let mut details = load_session_details(connection, user_id, &session_ids)?; + + let items = session_ids + .into_iter() + .filter_map(|session_pk| { + let details = details.remove(&session_pk)?; + let session_matches = matches.get(&session_pk).cloned().unwrap_or_default(); + let match_count = session_matches + .first() + .map_or(0, |matched| matched.match_count); + let mut providers = session_matches + .iter() + .map(|matched| matched.llm_provider.clone()) + .collect::>() + .into_iter() + .collect::>(); + providers.sort(); + let mut models = session_matches + .iter() + .map(|matched| matched.llm_model.clone()) + .collect::>() + .into_iter() + .collect::>(); + models.sort(); + Some(SessionSearchResult { + session_pk, + session_id: details.session_id, + agent_name: canonical_agent_name(&details.agent_name), + agent_version: details.agent_version, + host_name: details.host_name, + platform: details.platform, + started_at: details.started_at, + ended_at: details.ended_at, + providers, + models, + match_count, + matches: session_matches + .into_iter() + .map(MatchedEvent::into_response) + .collect(), + }) + }) + .collect(); + let consumed = u64::from(query.offset).saturating_add(u64::from(query.page_size)); + Ok(SessionSearchResponse { + query: query.text, + total_sessions, + page: query.page, + page_size: query.page_size, + has_more: consumed < total_sessions, + items, + }) +} + +#[derive(Clone)] +enum FtsBind { + Text(String), + BigInt(i64), +} + +struct MatchSource { + sql: String, + binds: Vec, +} + +impl MatchSource { + fn new(query: &ValidatedSearchQuery, user_id: Uuid) -> Self { + let mut sql = String::from( + "FROM usage_events_fts + INNER JOIN llm_usage_events e ON e.id = usage_events_fts.event_pk + WHERE usage_events_fts MATCH ? AND e.user_id = ?", + ); + let mut binds = vec![ + FtsBind::Text(fts_query(&query.text)), + FtsBind::Text(user_id.to_string()), + ]; + if let Some(from) = query.from { + sql.push_str(" AND e.observed_at >= ?"); + binds.push(FtsBind::BigInt(timestamp_to_micros(from))); + } + if let Some(to) = query.to { + sql.push_str(" AND e.observed_at < ?"); + binds.push(FtsBind::BigInt(timestamp_to_micros(to))); + } + push_exact_filter( + &mut sql, + &mut binds, + "e.agent_name", + query.agent_name.as_deref(), + ); + push_exact_filter( + &mut sql, + &mut binds, + "e.llm_provider", + query.llm_provider.as_deref(), + ); + push_exact_filter( + &mut sql, + &mut binds, + "e.llm_model", + query.llm_model.as_deref(), + ); + push_exact_filter( + &mut sql, + &mut binds, + "e.event_type", + query.event_type.as_deref(), + ); + Self { sql, binds } + } +} + +fn push_exact_filter( + sql: &mut String, + binds: &mut Vec, + column: &str, + value: Option<&str>, +) { + if let Some(value) = value { + sql.push_str(" AND "); + sql.push_str(column); + sql.push_str(" = ?"); + binds.push(FtsBind::Text(value.to_owned())); + } +} + +fn bound_query(sql: String, binds: Vec) -> BoxedSqlQuery<'static, Sqlite, SqlQuery> { + let mut query = sql_query(sql).into_boxed::(); + for bind in binds { + query = match bind { + FtsBind::Text(value) => query.bind::(value), + FtsBind::BigInt(value) => query.bind::(value), + }; + } + query +} + +#[derive(QueryableByName)] +struct CountRow { + #[diesel(sql_type = BigInt)] + count: i64, +} + +fn count_sessions( + connection: &mut SqliteConnection, + source: &MatchSource, +) -> Result { + let sql = format!( + "SELECT COUNT(DISTINCT e.session_pk) AS count {}", + source.sql + ); + let count = bound_query(sql, source.binds.clone()) + .get_result::(connection)? + .count; + u64::try_from(count.max(0)) + .map_err(|error| AppError::internal(format!("invalid FTS session count: {error}"))) +} + +#[derive(QueryableByName)] +struct SessionIdRow { + #[diesel(sql_type = Text)] + session_pk: String, +} + +fn load_session_page( + connection: &mut SqliteConnection, + query: &ValidatedSearchQuery, + source: &MatchSource, +) -> Result, AppError> { + let sql = format!( + "SELECT e.session_pk AS session_pk + {} + ORDER BY bm25(usage_events_fts) ASC, e.observed_at DESC, e.session_pk ASC", + source.sql + ); + let rows = bound_query(sql, source.binds.clone()).load::(connection)?; + let page_start = usize::try_from(query.offset) + .map_err(|error| AppError::internal(format!("invalid FTS page offset: {error}")))?; + let page_size = usize::try_from(query.page_size) + .map_err(|error| AppError::internal(format!("invalid FTS page size: {error}")))?; + let page_end = page_start.saturating_add(page_size); + let mut seen = HashSet::new(); + let mut sessions = Vec::with_capacity(page_size); + for row in rows { + let session_pk = parse_uuid(&row.session_pk, "FTS session id")?; + if !seen.insert(session_pk) { + continue; + } + let position = seen.len().saturating_sub(1); + if position >= page_start { + sessions.push(session_pk); + } + if seen.len() >= page_end { + break; + } + } + Ok(sessions) +} + +fn load_matches( + connection: &mut SqliteConnection, + session_ids: &[Uuid], + source: &MatchSource, +) -> Result>, AppError> { + if session_ids.is_empty() { + return Ok(HashMap::new()); + } + let selected_placeholders = std::iter::repeat_n("?", session_ids.len()) + .collect::>() + .join(","); + let sql = format!( + "SELECT + e.id AS event_pk, + e.session_pk AS session_pk, + e.turn_pk AS turn_pk, + e.turn_index AS turn_index, + e.event_type AS event_type, + e.llm_provider AS llm_provider, + e.llm_model AS llm_model, + e.observed_at AS observed_at, + snippet(usage_events_fts, 4, '{HIGHLIGHT_START}', '{HIGHLIGHT_END}', '…', 32) + AS content_fragment, + snippet(usage_events_fts, 7, '{HIGHLIGHT_START}', '{HIGHLIGHT_END}', '…', 32) + AS command_fragment, + snippet(usage_events_fts, 8, '{HIGHLIGHT_START}', '{HIGHLIGHT_END}', '…', 32) + AS path_fragment, + snippet(usage_events_fts, 5, '{HIGHLIGHT_START}', '{HIGHLIGHT_END}', '…', 32) + AS tool_name_fragment, + snippet(usage_events_fts, 6, '{HIGHLIGHT_START}', '{HIGHLIGHT_END}', '…', 32) + AS tool_content_fragment + {} + AND e.session_pk IN ({selected_placeholders}) + ORDER BY bm25(usage_events_fts) ASC, e.observed_at DESC, e.id ASC", + source.sql + ); + let mut binds = source.binds.clone(); + binds.extend( + session_ids + .iter() + .map(|session_pk| FtsBind::Text(session_pk.to_string())), + ); + let rows = bound_query(sql, binds).load::(connection)?; + let mut by_session: HashMap)> = HashMap::new(); + for row in rows { + let matched = row.into_matched()?; + let entry = by_session + .entry(matched.session_pk) + .or_insert_with(|| (0, Vec::new())); + entry.0 = entry.0.saturating_add(1); + if entry.1.len() < 3 { + entry.1.push(matched); + } + } + Ok(by_session + .into_iter() + .map(|(session_pk, (match_count, mut matches))| { + for matched in &mut matches { + matched.match_count = match_count; + } + (session_pk, matches) + }) + .collect()) +} + +#[derive(QueryableByName)] +struct MatchedEventRow { + #[diesel(sql_type = Text)] + event_pk: String, + #[diesel(sql_type = Text)] + session_pk: String, + #[diesel(sql_type = Text)] + turn_pk: String, + #[diesel(sql_type = Integer)] + turn_index: i32, + #[diesel(sql_type = Text)] + event_type: String, + #[diesel(sql_type = Text)] + llm_provider: String, + #[diesel(sql_type = Text)] + llm_model: String, + #[diesel(sql_type = BigInt)] + observed_at: i64, + #[diesel(sql_type = Nullable)] + content_fragment: Option, + #[diesel(sql_type = Nullable)] + command_fragment: Option, + #[diesel(sql_type = Nullable)] + path_fragment: Option, + #[diesel(sql_type = Nullable)] + tool_name_fragment: Option, + #[diesel(sql_type = Nullable)] + tool_content_fragment: Option, +} + +impl MatchedEventRow { + fn into_matched(self) -> Result { + let fragments = [ + self.content_fragment, + self.command_fragment, + self.path_fragment, + self.tool_name_fragment, + self.tool_content_fragment, + ] + .into_iter() + .flatten() + .filter(|fragment| fragment.contains(HIGHLIGHT_START)) + .take(2) + .collect(); + Ok(MatchedEvent { + event_pk: parse_uuid(&self.event_pk, "FTS event id")?, + session_pk: parse_uuid(&self.session_pk, "FTS session id")?, + turn_pk: parse_uuid(&self.turn_pk, "FTS turn id")?, + turn_index: self.turn_index, + event_type: self.event_type, + llm_provider: self.llm_provider, + llm_model: self.llm_model, + observed_at: timestamp_from_micros(self.observed_at)?, + match_count: 0, + fragments, + }) + } +} + +#[derive(Clone)] +struct MatchedEvent { + event_pk: Uuid, + session_pk: Uuid, + turn_pk: Uuid, + turn_index: i32, + event_type: String, + llm_provider: String, + llm_model: String, + observed_at: DateTime, + match_count: u64, + fragments: Vec, +} + +impl MatchedEvent { + fn into_response(self) -> SessionSearchMatch { + SessionSearchMatch { + event_pk: self.event_pk, + turn_pk: self.turn_pk, + turn_index: self.turn_index, + event_type: self.event_type, + llm_provider: self.llm_provider, + llm_model: self.llm_model, + observed_at: self.observed_at, + fragments: self + .fragments + .into_iter() + .map(|fragment| SearchFragment::parse(&fragment)) + .collect(), + } + } +} + +struct SessionDetails { + session_id: String, + agent_name: String, + agent_version: Option, + host_name: String, + platform: String, + started_at: DateTime, + ended_at: Option>, +} + +fn load_session_details( + connection: &mut SqliteConnection, + user_id: Uuid, + session_ids: &[Uuid], +) -> Result, AppError> { + if session_ids.is_empty() { + return Ok(HashMap::new()); + } + let ids = session_ids.iter().map(Uuid::to_string).collect::>(); + let rows = agent_sessions::table + .inner_join(devices::table) + .filter(agent_sessions::user_id.eq(user_id.to_string())) + .filter(agent_sessions::id.eq_any(ids)) + .select(( + agent_sessions::id, + agent_sessions::session_id, + agent_sessions::agent_name, + agent_sessions::agent_version, + devices::host_name, + devices::platform, + agent_sessions::started_at, + agent_sessions::ended_at, + )) + .load::<( + String, + String, + String, + Option, + String, + String, + i64, + Option, + )>(connection)?; + rows.into_iter() + .map( + |( + session_pk, + session_id, + agent_name, + agent_version, + host_name, + platform, + started_at, + ended_at, + )| { + Ok(( + parse_uuid(&session_pk, "session id")?, + SessionDetails { + session_id, + agent_name, + agent_version, + host_name, + platform, + started_at: timestamp_from_micros(started_at)?, + ended_at: ended_at.map(timestamp_from_micros).transpose()?, + }, + )) + }, + ) + .collect() +} + +fn fts_query(value: &str) -> String { + value + .split_whitespace() + .map(|term| format!("\"{}\"", term.replace('"', "\"\""))) + .collect::>() + .join(" AND ") +} + +#[cfg(test)] +mod tests { + use super::fts_query; + + #[test] + fn fts_query_uses_literal_and_terms() { + assert_eq!( + fts_query("blackbox response"), + r#""blackbox" AND "response""# + ); + assert_eq!(fts_query(r#"quoted"value"#), r#""quoted""value""#); + } +} diff --git a/crates/abyss-backend/src/usage/event_order.rs b/crates/abyss-backend/src/usage/event_order.rs index 4fd3e45..5969899 100644 --- a/crates/abyss-backend/src/usage/event_order.rs +++ b/crates/abyss-backend/src/usage/event_order.rs @@ -12,7 +12,37 @@ use std::{ use chrono::{DateTime, Utc}; -use crate::db::models::UsageEvent; +/// Event fields required by deterministic timeline ordering. +pub trait TimelineEvent { + fn turn_index(&self) -> i32; + fn metadata(&self) -> &serde_json::Value; + fn observed_at(&self) -> DateTime; + fn event_type(&self) -> &str; + fn event_id(&self) -> &str; +} + +#[cfg(feature = "postgres-es")] +impl TimelineEvent for crate::db::models::UsageEvent { + fn turn_index(&self) -> i32 { + self.turn_index + } + + fn metadata(&self) -> &serde_json::Value { + &self.metadata + } + + fn observed_at(&self) -> DateTime { + self.observed_at + } + + fn event_type(&self) -> &str { + &self.event_type + } + + fn event_id(&self) -> &str { + &self.event_id + } +} /// Applies the authoritative event order used by owned and shared timelines. pub struct UsageEventTimelineOrder; @@ -22,18 +52,18 @@ impl UsageEventTimelineOrder { /// /// The final event identifier tie-breaker makes output deterministic even /// for corrupt cycles or collectors with identical timestamps. - pub fn sort(events: &mut [UsageEvent]) { + pub fn sort(events: &mut [T]) { let provider_ranks = ProviderResponseRanks::from_events(events); events.sort_by(|left, right| { - left.turn_index.cmp(&right.turn_index).then_with(|| { + left.turn_index().cmp(&right.turn_index()).then_with(|| { provider_ranks.compare_events(left, right).then_with(|| { - left.observed_at - .cmp(&right.observed_at) + left.observed_at() + .cmp(&right.observed_at()) .then_with(|| { - event_side_rank(&left.event_type) - .cmp(&event_side_rank(&right.event_type)) + event_side_rank(left.event_type()) + .cmp(&event_side_rank(right.event_type())) }) - .then_with(|| left.event_id.cmp(&right.event_id)) + .then_with(|| left.event_id().cmp(right.event_id())) }) }) }); @@ -46,31 +76,31 @@ struct ProviderResponseRanks { } impl ProviderResponseRanks { - fn from_events(events: &[UsageEvent]) -> Self { + fn from_events(events: &[T]) -> Self { let mut nodes_by_turn = BTreeMap::>::new(); // Native provider ordering is safe only when every event in a turn has // a response_id. A partially instrumented turn falls back as a whole so // ranked and unranked events cannot interleave unpredictably. let mut native_turns = events .iter() - .map(|event| event.turn_index) + .map(TimelineEvent::turn_index) .collect::>(); for event in events { - let Some(response_id) = metadata_string(&event.metadata, "response_id") else { - native_turns.remove(&event.turn_index); + let Some(response_id) = metadata_string(event.metadata(), "response_id") else { + native_turns.remove(&event.turn_index()); continue; }; let previous_response_id = - metadata_string(&event.metadata, "previous_response_id").map(str::to_owned); - let nodes = nodes_by_turn.entry(event.turn_index).or_default(); + metadata_string(event.metadata(), "previous_response_id").map(str::to_owned); + let nodes = nodes_by_turn.entry(event.turn_index()).or_default(); let node = nodes .entry(response_id.to_owned()) .or_insert_with(|| ProviderResponseNode { response_id: response_id.to_owned(), previous_response_id: previous_response_id.clone(), - first_observed_at: event.observed_at, - first_event_id: event.event_id.clone(), + first_observed_at: event.observed_at(), + first_event_id: event.event_id().to_owned(), }); node.merge_evidence(event, previous_response_id); } @@ -178,24 +208,27 @@ impl ProviderResponseRanks { } } - fn compare_events(&self, left: &UsageEvent, right: &UsageEvent) -> Ordering { - if left.turn_index != right.turn_index || !self.native_turns.contains(&left.turn_index) { + fn compare_events(&self, left: &T, right: &T) -> Ordering { + if left.turn_index() != right.turn_index() + || !self.native_turns.contains(&left.turn_index()) + { return Ordering::Equal; } - let left_response_id = metadata_string(&left.metadata, "response_id"); - let right_response_id = metadata_string(&right.metadata, "response_id"); + let left_response_id = metadata_string(left.metadata(), "response_id"); + let right_response_id = metadata_string(right.metadata(), "response_id"); // Request and response observations that describe the same provider // call remain adjacent and request-first regardless of timestamp ties. if left_response_id.is_some() && left_response_id == right_response_id { - return event_side_rank(&left.event_type).cmp(&event_side_rank(&right.event_type)); + return event_side_rank(left.event_type()).cmp(&event_side_rank(right.event_type())); } let native_order = left_response_id - .and_then(|response_id| self.ranks.get(&(left.turn_index, response_id.to_owned()))) + .and_then(|response_id| self.ranks.get(&(left.turn_index(), response_id.to_owned()))) .zip(right_response_id.and_then(|response_id| { - self.ranks.get(&(right.turn_index, response_id.to_owned())) + self.ranks + .get(&(right.turn_index(), response_id.to_owned())) })) .map(|(left_rank, right_rank)| left_rank.cmp(right_rank)); if let Some(ordering) = native_order.filter(|ordering| !ordering.is_eq()) { @@ -214,15 +247,19 @@ struct ProviderResponseNode { } impl ProviderResponseNode { - fn merge_evidence(&mut self, event: &UsageEvent, previous_response_id: Option) { + fn merge_evidence( + &mut self, + event: &T, + previous_response_id: Option, + ) { if self.previous_response_id.is_none() { self.previous_response_id = previous_response_id; } - if (event.observed_at, event.event_id.as_str()) + if (event.observed_at(), event.event_id()) < (self.first_observed_at, self.first_event_id.as_str()) { - self.first_observed_at = event.observed_at; - self.first_event_id.clone_from(&event.event_id); + self.first_observed_at = event.observed_at(); + event.event_id().clone_into(&mut self.first_event_id); } } @@ -250,7 +287,7 @@ const fn event_side_rank(event_type: &str) -> u8 { } } -#[cfg(test)] +#[cfg(all(test, feature = "postgres-es"))] mod tests { use chrono::{TimeZone as _, Timelike as _, Utc}; use serde_json::json; diff --git a/crates/abyss-backend/src/usage/mod.rs b/crates/abyss-backend/src/usage/mod.rs index 47625c7..0d055c2 100644 --- a/crates/abyss-backend/src/usage/mod.rs +++ b/crates/abyss-backend/src/usage/mod.rs @@ -1,17 +1,16 @@ //! API contracts and shared transformations for Agent usage collection. //! //! Ingest types preserve collector-provided observations, while response types -//! expose the normalized device/session/turn hierarchy stored by PostgreSQL. -//! Repository code owns persistence and validation that requires database state; -//! small deterministic metadata transformations remain in this module. +//! expose the normalized device/session/turn hierarchy returned by every storage +//! backend. Deterministic validation and metadata transformations live here; +//! backend-specific persistence stays behind the storage compatibility layer. /// Image attachment contracts and validation. pub mod attachments; /// Opaque diagnostic-capture ingest contracts. pub mod diagnostics; -mod event_order; -/// PostgreSQL-backed ingest and query operations. -pub mod repository; +pub mod event_order; +pub mod persistence; use chrono::{DateTime, Utc}; use serde::{Deserialize, Serialize}; @@ -105,7 +104,7 @@ pub enum UsageEventType { } impl UsageEventType { - /// Returns the canonical value stored in PostgreSQL and returned by APIs. + /// Returns the canonical value stored by backends and returned by APIs. pub const fn as_str(self) -> &'static str { match self { Self::Request => "request", diff --git a/crates/abyss-backend/src/usage/persistence.rs b/crates/abyss-backend/src/usage/persistence.rs new file mode 100644 index 0000000..1112eae --- /dev/null +++ b/crates/abyss-backend/src/usage/persistence.rs @@ -0,0 +1,352 @@ +//! Backend-independent validation and normalization for persisted usage data. + +use std::collections::HashSet; + +#[cfg(feature = "postgres-es")] +use chrono::{DateTime, Utc}; +use sha2::{Digest, Sha256}; + +use crate::error::AppError; + +use super::{ + IngestEventsRequest, IngestUsageEvent, + attachments::{ValidatedImageAttachment, validate_image_attachments}, +}; + +pub fn validate_batch( + request: &IngestEventsRequest, + max_batch_size: usize, +) -> Result>, AppError> { + if request.events.len() > max_batch_size { + return Err(AppError::validation(format!( + "events batch size must be <= {max_batch_size}" + ))); + } + + // Decode potentially expensive attachment content before a transaction and + // its table locks are acquired by either storage implementation. + let mut attachments = Vec::with_capacity(request.events.len()); + for event in &request.events { + validate_event(event)?; + attachments.push(validate_image_attachments(&event.attachments)?); + } + if request.diagnostic_captures.len() > max_batch_size { + return Err(AppError::validation(format!( + "diagnostic captures batch size must be <= {max_batch_size}" + ))); + } + let request_event_ids = request + .events + .iter() + .map(|event| event.event_id.as_str()) + .collect::>(); + let mut capture_ids = HashSet::with_capacity(request.diagnostic_captures.len()); + for capture in &request.diagnostic_captures { + capture.validate_event_correlation(&request_event_ids)?; + if !capture_ids.insert(capture.capture_id.as_str()) { + return Err(AppError::validation( + "diagnostic capture ids must be unique within one ingest request".to_owned(), + )); + } + } + Ok(attachments) +} + +pub fn validate_event(event: &IngestUsageEvent) -> Result<(), AppError> { + require_non_empty(&event.event_id, "event_id")?; + require_non_empty(&event.device.host_name, "device.host_name")?; + require_non_empty(&event.device.platform, "device.platform")?; + require_non_empty(&event.agent.name, "agent.name")?; + require_non_empty(&event.session_id, "session_id")?; + require_non_empty(&event.llm.provider, "llm.provider")?; + require_non_empty(&event.llm.model, "llm.model")?; + if event.turn_index < 1_i32 { + return Err(AppError::validation("turn_index must be >= 1".to_owned())); + } + let tokens = normalized_tokens(event)?; + if tokens.total == 0 { + return Ok(()); + } + let visible_tokens = tokens + .input + .saturating_add(tokens.output) + .saturating_add(tokens.reasoning); + if tokens.total < visible_tokens { + tracing::debug!( + event_id = %event.event_id, + "provider total_tokens is lower than visible token components" + ); + } + Ok(()) +} + +fn require_non_empty(value: &str, field: &str) -> Result<(), AppError> { + if value.trim().is_empty() { + return Err(AppError::validation(format!("{field} is required"))); + } + Ok(()) +} + +pub struct NormalizedTokens { + pub input: i64, + pub output: i64, + pub cache_read: i64, + pub cache_write: i64, + pub reasoning: i64, + pub total: i64, +} + +pub fn normalized_tokens(event: &IngestUsageEvent) -> Result { + let input = normalize_token(event.token_usage.input_tokens, "input_tokens")?; + let output = normalize_token(event.token_usage.output_tokens, "output_tokens")?; + let cache_read = normalize_token(event.token_usage.cache_read_tokens, "cache_read_tokens")?; + let cache_write = normalize_token(event.token_usage.cache_write_tokens, "cache_write_tokens")?; + let reasoning = normalize_token(event.token_usage.reasoning_tokens, "reasoning_tokens")?; + // Preserve an explicit provider total because provider accounting may not + // equal the visible component sum; derive only when the field is absent. + let total = match event.token_usage.total_tokens { + Some(value) => normalize_token(Some(value), "total_tokens")?, + None => input + .saturating_add(output) + .saturating_add(cache_read) + .saturating_add(cache_write) + .saturating_add(reasoning), + }; + + Ok(NormalizedTokens { + input, + output, + cache_read, + cache_write, + reasoning, + total, + }) +} + +fn normalize_token(value: Option, field: &str) -> Result { + let value = value.unwrap_or(0); + if value < 0 { + return Err(AppError::validation(format!("{field} must be >= 0"))); + } + Ok(value) +} + +pub fn validate_event_type(value: &str) -> Result { + match value.trim().to_ascii_lowercase().as_str() { + "request" => Ok("request".to_owned()), + "response" => Ok("response".to_owned()), + _ => Err(AppError::validation( + "event_type must be request or response".to_owned(), + )), + } +} + +pub fn parse_group_by(raw: Option<&str>) -> Vec { + let parsed: Vec<_> = raw + .unwrap_or("user,agent,provider,model") + .split(',') + .map(str::trim) + .filter(|value| !value.is_empty()) + .map(|value| match value { + "employee" => "user".to_owned(), + "llm_provider" => "provider".to_owned(), + "llm_model" => "model".to_owned(), + other => other.to_owned(), + }) + .filter(|value| { + matches!( + value.as_str(), + "day" | "user" | "device" | "agent" | "provider" | "model" | "event_type" + ) + }) + .collect(); + + if parsed.is_empty() { + vec![ + "user".to_owned(), + "agent".to_owned(), + "provider".to_owned(), + "model".to_owned(), + ] + } else { + parsed + } +} + +pub fn normalize_slug(value: &str) -> String { + value + .trim() + .to_ascii_lowercase() + .chars() + .map(|character| match character { + '_' | ' ' => '-', + other => other, + }) + .collect() +} + +pub fn canonical_agent_name(value: &str) -> String { + let normalized = normalize_slug(value); + match normalized.as_str() { + "claude" | "claude-desktop" => "claude-code".to_owned(), + _ => normalized, + } +} + +pub fn agent_name_filter_values(value: &str) -> Vec { + let canonical = canonical_agent_name(value); + if canonical == "claude-code" { + vec![ + "claude-code".to_owned(), + "claude".to_owned(), + "claude-desktop".to_owned(), + ] + } else { + vec![canonical] + } +} + +pub fn normalize_text(value: &str) -> String { + value.trim().to_owned() +} + +pub fn normalized_optional(value: Option<&str>) -> Option { + value + .map(str::trim) + .filter(|value| !value.is_empty()) + .map(str::to_owned) +} + +pub fn non_empty(value: &str) -> Option<&str> { + (!value.trim().is_empty()).then_some(value) +} + +#[cfg(feature = "postgres-es")] +pub fn non_negative_count(value: i64) -> usize { + usize::try_from(value.max(0_i64)).unwrap_or(usize::MAX) +} + +#[cfg(feature = "postgres-es")] +pub const fn unix_epoch() -> DateTime { + DateTime::::from_timestamp(0_i64, 0_u32).expect("unix epoch must be a valid timestamp") +} + +#[cfg(any(feature = "postgres-es", test))] +pub fn escape_like_pattern(value: &str) -> String { + let mut escaped = String::with_capacity(value.len()); + for character in value.chars() { + if matches!(character, '\\' | '%' | '_') { + escaped.push('\\'); + } + escaped.push(character); + } + escaped +} + +pub fn normalize_limit(limit: Option, fallback: i64, maximum: i64) -> i64 { + limit.unwrap_or(fallback).clamp(1_i64, maximum) +} + +pub fn sha256_bytes(value: &str) -> Vec { + Sha256::digest(value.as_bytes()).to_vec() +} + +pub struct TurnIdentity { + pub kind: TurnIdentityKind, + pub value: String, +} + +#[derive(Clone, Copy)] +pub enum TurnIdentityKind { + CodexTurnId, + ClaudeTurnId, + ResponseId, + MessageId, + RequestHash, +} + +impl TurnIdentityKind { + pub const fn metadata_key(self) -> &'static str { + match self { + Self::CodexTurnId => "codex_turn_id", + Self::ClaudeTurnId => "claude_turn_id", + Self::ResponseId => "response_id", + Self::MessageId => "message_id", + Self::RequestHash => "request_hash", + } + } +} + +pub fn turn_identity(event: &IngestUsageEvent) -> Option { + turn_identity_from_metadata(&event.metadata) +} + +pub fn turn_identity_from_metadata(metadata: &serde_json::Value) -> Option { + // Prefer Agent-level identities because one user turn may contain several + // provider round trips. + [ + TurnIdentityKind::CodexTurnId, + TurnIdentityKind::ClaudeTurnId, + TurnIdentityKind::ResponseId, + TurnIdentityKind::MessageId, + TurnIdentityKind::RequestHash, + ] + .into_iter() + .find_map(|kind| { + metadata_string(metadata, kind.metadata_key()).map(|value| TurnIdentity { kind, value }) + }) +} + +fn metadata_string(metadata: &serde_json::Value, key: &str) -> Option { + metadata + .get(key) + .and_then(serde_json::Value::as_str) + .map(str::trim) + .filter(|value| !value.is_empty()) + .map(str::to_owned) +} + +pub const fn choose_turn_index( + requested_turn_index: i32, + existing_turn_index: Option, + next_turn_index: i32, +) -> i32 { + if let Some(existing_turn_index) = existing_turn_index { + return existing_turn_index; + } + if requested_turn_index < next_turn_index { + next_turn_index + } else { + requested_turn_index + } +} + +#[cfg(test)] +mod tests { + use serde_json::json; + + use super::{ + TurnIdentityKind, canonical_agent_name, choose_turn_index, escape_like_pattern, + parse_group_by, turn_identity_from_metadata, + }; + + #[test] + fn shared_normalization_is_backend_independent() { + assert_eq!(canonical_agent_name(" Claude Desktop "), "claude-code"); + assert_eq!(parse_group_by(Some("day,llm_model")), ["day", "model"]); + assert_eq!(escape_like_pattern(r"a%b_c\d"), r"a\%b\_c\\d"); + } + + #[test] + fn turn_identity_and_restart_resolution_are_stable() { + let identity = turn_identity_from_metadata(&json!({ + "response_id": "response", + "codex_turn_id": "turn" + })) + .expect("turn identity should be extracted"); + assert!(matches!(identity.kind, TurnIdentityKind::CodexTurnId)); + assert_eq!(identity.value, "turn"); + assert_eq!(choose_turn_index(1_i32, None, 4_i32), 4_i32); + assert_eq!(choose_turn_index(1_i32, Some(2_i32), 4_i32), 2_i32); + } +} diff --git a/scripts/blackbox_sqlite_backend.sh b/scripts/blackbox_sqlite_backend.sh new file mode 100644 index 0000000..f40c7c8 --- /dev/null +++ b/scripts/blackbox_sqlite_backend.sh @@ -0,0 +1,126 @@ +#!/usr/bin/env bash + +set -euo pipefail + +for command_name in cargo curl python3; do + command -v "${command_name}" >/dev/null || { + echo "${command_name} is required for the SQLite black-box test." >&2 + exit 2 + } +done + +if [[ -n "${ABYSS_BACKEND_BLACKBOX_ADDR:-}" ]]; then + LISTEN_ADDR="${ABYSS_BACKEND_BLACKBOX_ADDR}" +else + BACKEND_PORT="$(python3 -c 'import socket; server = socket.socket(); server.bind(("127.0.0.1", 0)); print(server.getsockname()[1]); server.close()')" + LISTEN_ADDR="127.0.0.1:${BACKEND_PORT}" +fi +BASE_URL="http://${LISTEN_ADDR}" +RUN_ID="$(date -u +%Y%m%d%H%M%S)-$$" +API_TOKEN="blackbox-token-${RUN_ID}" +API_TOKEN_HASH="$(python3 -c 'import hashlib, sys; print(hashlib.sha256(sys.argv[1].encode()).hexdigest())' "${API_TOKEN}")" +TEMP_DIR="$(mktemp -d -t abyss-backend-sqlite-blackbox.XXXXXX)" +DATABASE_PATH="${TEMP_DIR}/abyss.sqlite" +SERVER_LOG="${TEMP_DIR}/backend.log" +SERVER_PID="" + +cleanup() { + local status=$? + if [[ "${status}" -ne 0 && -f "${SERVER_LOG}" ]]; then + echo "abyss-backend SQLite log:" >&2 + tail -n 200 "${SERVER_LOG}" >&2 || true + fi + if [[ -n "${SERVER_PID}" ]]; then + kill "${SERVER_PID}" 2>/dev/null || true + wait "${SERVER_PID}" 2>/dev/null || true + fi + rm -r "${TEMP_DIR}" +} +trap cleanup EXIT + +fail() { + echo "blackbox SQLite: $*" >&2 + exit 1 +} + +wait_for_backend() { + for _attempt in $(seq 1 180); do + if curl -fsS "${BASE_URL}/readyz" >/dev/null 2>&1; then + return + fi + kill -0 "${SERVER_PID}" 2>/dev/null || fail "abyss-backend exited before becoming ready" + sleep 1 + done + fail "abyss-backend did not become ready at ${BASE_URL}" +} + +ABYSS_BACKEND_ADDR="${LISTEN_ADDR}" \ +ABYSS_BACKEND_ENV="blackbox" \ +ABYSS_BACKEND_API_TOKEN_SHA256="${API_TOKEN_HASH}" \ +ABYSS_BACKEND_DATABASE_URL="${DATABASE_PATH}" \ +ABYSS_BACKEND_RUN_MIGRATIONS="true" \ +cargo run --locked --package abyss-backend --no-default-features --features sqlite-fts --quiet \ + >"${SERVER_LOG}" 2>&1 & +SERVER_PID=$! +wait_for_backend + +python3 - "${DATABASE_PATH}" <<'PY' +import sqlite3 +import sys + +database = sqlite3.connect(sys.argv[1]) +tables = { + row[0] + for row in database.execute( + "SELECT name FROM sqlite_master WHERE type IN ('table', 'view')" + ) +} +expected = { + "app_users", + "devices", + "agent_sessions", + "agent_turns", + "llm_usage_events", + "llm_usage_event_attachments", + "agent_diagnostic_captures", + "agent_diagnostic_capture_events", + "usage_events_fts", +} +missing = expected - tables +if missing: + raise SystemExit(f"SQLite migration is missing tables: {sorted(missing)}") +PY + +python3 scripts/tests/blackbox_api.py \ + --base-url "${BASE_URL}" \ + --token "${API_TOKEN}" \ + --run-id "${RUN_ID}" + +python3 - "${DATABASE_PATH}" "capture-${RUN_ID}" <<'PY' +import sqlite3 +import sys + +database = sqlite3.connect(sys.argv[1]) +capture_id = sys.argv[2] +capture_count = database.execute( + """SELECT count(*) + FROM agent_diagnostic_captures + WHERE capture_id = ? + AND json_extract(payload, '$.request_plaintext') = 'diagnostic request' + AND json_extract(payload, '$.response_plaintext') = 'diagnostic response'""", + (capture_id,), +).fetchone()[0] +if capture_count != 1: + raise SystemExit("diagnostic capture payload was not retained") +link_count = database.execute( + """SELECT count(*) + FROM agent_diagnostic_capture_events links + JOIN agent_diagnostic_captures captures ON captures.id = links.capture_pk + WHERE captures.capture_id = ?""", + (capture_id,), +).fetchone()[0] +if link_count != 2: + raise SystemExit("diagnostic capture was not linked to both events") +PY + +echo "blackbox: standalone SQLite, HTTP, attachment, diagnostics, and FTS contracts passed"