diff --git a/.claude/skills/gitnexus/gitnexus-cli/SKILL.md b/.claude/skills/gitnexus/gitnexus-cli/SKILL.md new file mode 100644 index 0000000..cd9a83b --- /dev/null +++ b/.claude/skills/gitnexus/gitnexus-cli/SKILL.md @@ -0,0 +1,83 @@ +--- +name: gitnexus-cli +description: "Use when the user needs to run GitNexus CLI commands like analyze/index a repo, check status, clean the index, generate a wiki, or list indexed repos. Examples: \"Index this repo\", \"Reanalyze the codebase\", \"Generate a wiki\"" +--- + +# GitNexus CLI Commands + +All commands work via `npx` — no global install required. + +## Commands + +### analyze — Build or refresh the index + +```bash +npx gitnexus analyze +``` + +Run from the project root. This parses all source files, builds the knowledge graph, writes it to `.gitnexus/`, and generates CLAUDE.md / AGENTS.md context files. + +| Flag | Effect | +| -------------- | ---------------------------------------------------------------- | +| `--force` | Force full re-index even if up to date | +| `--embeddings` | Enable embedding generation for semantic search (off by default) | +| `--drop-embeddings` | Drop existing embeddings on rebuild. By default, an `analyze` without `--embeddings` preserves them. | + +**When to run:** First time in a project, after major code changes, or when `gitnexus://repo/{name}/context` reports the index is stale. In Claude Code, a PostToolUse hook detects staleness after `git commit` and `git merge` and notifies the agent to run `analyze` — the hook does not run analyze itself, to avoid blocking the agent for up to 120s and risking KuzuDB corruption on timeout. + +### status — Check index freshness + +```bash +npx gitnexus status +``` + +Shows whether the current repo has a GitNexus index, when it was last updated, and symbol/relationship counts. Use this to check if re-indexing is needed. + +### clean — Delete the index + +```bash +npx gitnexus clean +``` + +Deletes the `.gitnexus/` directory and unregisters the repo from the global registry. Use before re-indexing if the index is corrupt or after removing GitNexus from a project. + +| Flag | Effect | +| --------- | ------------------------------------------------- | +| `--force` | Skip confirmation prompt | +| `--all` | Clean all indexed repos, not just the current one | + +### wiki — Generate documentation from the graph + +```bash +npx gitnexus wiki +``` + +Generates repository documentation from the knowledge graph using an LLM. Requires an API key (saved to `~/.gitnexus/config.json` on first use). + +| Flag | Effect | +| ------------------- | ----------------------------------------- | +| `--force` | Force full regeneration | +| `--model ` | LLM model (default: minimax/minimax-m2.5) | +| `--base-url ` | LLM API base URL | +| `--api-key ` | LLM API key | +| `--concurrency ` | Parallel LLM calls (default: 3) | +| `--gist` | Publish wiki as a public GitHub Gist | + +### list — Show all indexed repos + +```bash +npx gitnexus list +``` + +Lists all repositories registered in `~/.gitnexus/registry.json`. The MCP `list_repos` tool provides the same information. + +## After Indexing + +1. **Read `gitnexus://repo/{name}/context`** to verify the index loaded +2. Use the other GitNexus skills (`exploring`, `debugging`, `impact-analysis`, `refactoring`) for your task + +## Troubleshooting + +- **"Not inside a git repository"**: Run from a directory inside a git repo +- **Index is stale after re-analyzing**: Restart Claude Code to reload the MCP server +- **Embeddings slow**: Omit `--embeddings` (it's off by default) or set `OPENAI_API_KEY` for faster API-based embedding diff --git a/.claude/skills/gitnexus/gitnexus-debugging/SKILL.md b/.claude/skills/gitnexus/gitnexus-debugging/SKILL.md new file mode 100644 index 0000000..9510b97 --- /dev/null +++ b/.claude/skills/gitnexus/gitnexus-debugging/SKILL.md @@ -0,0 +1,89 @@ +--- +name: gitnexus-debugging +description: "Use when the user is debugging a bug, tracing an error, or asking why something fails. Examples: \"Why is X failing?\", \"Where does this error come from?\", \"Trace this bug\"" +--- + +# Debugging with GitNexus + +## When to Use + +- "Why is this function failing?" +- "Trace where this error comes from" +- "Who calls this method?" +- "This endpoint returns 500" +- Investigating bugs, errors, or unexpected behavior + +## Workflow + +``` +1. gitnexus_query({query: ""}) → Find related execution flows +2. gitnexus_context({name: ""}) → See callers/callees/processes +3. READ gitnexus://repo/{name}/process/{name} → Trace execution flow +4. gitnexus_cypher({query: "MATCH path..."}) → Custom traces if needed +``` + +> If "Index is stale" → run `npx gitnexus analyze` in terminal. + +## Checklist + +``` +- [ ] Understand the symptom (error message, unexpected behavior) +- [ ] gitnexus_query for error text or related code +- [ ] Identify the suspect function from returned processes +- [ ] gitnexus_context to see callers and callees +- [ ] Trace execution flow via process resource if applicable +- [ ] gitnexus_cypher for custom call chain traces if needed +- [ ] Read source files to confirm root cause +``` + +## Debugging Patterns + +| Symptom | GitNexus Approach | +| -------------------- | ---------------------------------------------------------- | +| Error message | `gitnexus_query` for error text → `context` on throw sites | +| Wrong return value | `context` on the function → trace callees for data flow | +| Intermittent failure | `context` → look for external calls, async deps | +| Performance issue | `context` → find symbols with many callers (hot paths) | +| Recent regression | `detect_changes` to see what your changes affect | + +## Tools + +**gitnexus_query** — find code related to error: + +``` +gitnexus_query({query: "payment validation error"}) +→ Processes: CheckoutFlow, ErrorHandling +→ Symbols: validatePayment, handlePaymentError, PaymentException +``` + +**gitnexus_context** — full context for a suspect: + +``` +gitnexus_context({name: "validatePayment"}) +→ Incoming calls: processCheckout, webhookHandler +→ Outgoing calls: verifyCard, fetchRates (external API!) +→ Processes: CheckoutFlow (step 3/7) +``` + +**gitnexus_cypher** — custom call chain traces: + +```cypher +MATCH path = (a)-[:CodeRelation {type: 'CALLS'}*1..2]->(b:Function {name: "validatePayment"}) +RETURN [n IN nodes(path) | n.name] AS chain +``` + +## Example: "Payment endpoint returns 500 intermittently" + +``` +1. gitnexus_query({query: "payment error handling"}) + → Processes: CheckoutFlow, ErrorHandling + → Symbols: validatePayment, handlePaymentError + +2. gitnexus_context({name: "validatePayment"}) + → Outgoing calls: verifyCard, fetchRates (external API!) + +3. READ gitnexus://repo/my-app/process/CheckoutFlow + → Step 3: validatePayment → calls fetchRates (external) + +4. Root cause: fetchRates calls external API without proper timeout +``` diff --git a/.claude/skills/gitnexus/gitnexus-exploring/SKILL.md b/.claude/skills/gitnexus/gitnexus-exploring/SKILL.md new file mode 100644 index 0000000..927a4e4 --- /dev/null +++ b/.claude/skills/gitnexus/gitnexus-exploring/SKILL.md @@ -0,0 +1,78 @@ +--- +name: gitnexus-exploring +description: "Use when the user asks how code works, wants to understand architecture, trace execution flows, or explore unfamiliar parts of the codebase. Examples: \"How does X work?\", \"What calls this function?\", \"Show me the auth flow\"" +--- + +# Exploring Codebases with GitNexus + +## When to Use + +- "How does authentication work?" +- "What's the project structure?" +- "Show me the main components" +- "Where is the database logic?" +- Understanding code you haven't seen before + +## Workflow + +``` +1. READ gitnexus://repos → Discover indexed repos +2. READ gitnexus://repo/{name}/context → Codebase overview, check staleness +3. gitnexus_query({query: ""}) → Find related execution flows +4. gitnexus_context({name: ""}) → Deep dive on specific symbol +5. READ gitnexus://repo/{name}/process/{name} → Trace full execution flow +``` + +> If step 2 says "Index is stale" → run `npx gitnexus analyze` in terminal. + +## Checklist + +``` +- [ ] READ gitnexus://repo/{name}/context +- [ ] gitnexus_query for the concept you want to understand +- [ ] Review returned processes (execution flows) +- [ ] gitnexus_context on key symbols for callers/callees +- [ ] READ process resource for full execution traces +- [ ] Read source files for implementation details +``` + +## Resources + +| Resource | What you get | +| --------------------------------------- | ------------------------------------------------------- | +| `gitnexus://repo/{name}/context` | Stats, staleness warning (~150 tokens) | +| `gitnexus://repo/{name}/clusters` | All functional areas with cohesion scores (~300 tokens) | +| `gitnexus://repo/{name}/cluster/{name}` | Area members with file paths (~500 tokens) | +| `gitnexus://repo/{name}/process/{name}` | Step-by-step execution trace (~200 tokens) | + +## Tools + +**gitnexus_query** — find execution flows related to a concept: + +``` +gitnexus_query({query: "payment processing"}) +→ Processes: CheckoutFlow, RefundFlow, WebhookHandler +→ Symbols grouped by flow with file locations +``` + +**gitnexus_context** — 360-degree view of a symbol: + +``` +gitnexus_context({name: "validateUser"}) +→ Incoming calls: loginHandler, apiMiddleware +→ Outgoing calls: checkToken, getUserById +→ Processes: LoginFlow (step 2/5), TokenRefresh (step 1/3) +``` + +## Example: "How does payment processing work?" + +``` +1. READ gitnexus://repo/my-app/context → 918 symbols, 45 processes +2. gitnexus_query({query: "payment processing"}) + → CheckoutFlow: processPayment → validateCard → chargeStripe + → RefundFlow: initiateRefund → calculateRefund → processRefund +3. gitnexus_context({name: "processPayment"}) + → Incoming: checkoutHandler, webhookHandler + → Outgoing: validateCard, chargeStripe, saveTransaction +4. Read src/payments/processor.ts for implementation details +``` diff --git a/.claude/skills/gitnexus/gitnexus-guide/SKILL.md b/.claude/skills/gitnexus/gitnexus-guide/SKILL.md new file mode 100644 index 0000000..937ac73 --- /dev/null +++ b/.claude/skills/gitnexus/gitnexus-guide/SKILL.md @@ -0,0 +1,64 @@ +--- +name: gitnexus-guide +description: "Use when the user asks about GitNexus itself — available tools, how to query the knowledge graph, MCP resources, graph schema, or workflow reference. Examples: \"What GitNexus tools are available?\", \"How do I use GitNexus?\"" +--- + +# GitNexus Guide + +Quick reference for all GitNexus MCP tools, resources, and the knowledge graph schema. + +## Always Start Here + +For any task involving code understanding, debugging, impact analysis, or refactoring: + +1. **Read `gitnexus://repo/{name}/context`** — codebase overview + check index freshness +2. **Match your task to a skill below** and **read that skill file** +3. **Follow the skill's workflow and checklist** + +> If step 1 warns the index is stale, run `npx gitnexus analyze` in the terminal first. + +## Skills + +| Task | Skill to read | +| -------------------------------------------- | ------------------- | +| Understand architecture / "How does X work?" | `gitnexus-exploring` | +| Blast radius / "What breaks if I change X?" | `gitnexus-impact-analysis` | +| Trace bugs / "Why is X failing?" | `gitnexus-debugging` | +| Rename / extract / split / refactor | `gitnexus-refactoring` | +| Tools, resources, schema reference | `gitnexus-guide` (this file) | +| Index, status, clean, wiki CLI commands | `gitnexus-cli` | + +## Tools Reference + +| Tool | What it gives you | +| ---------------- | ------------------------------------------------------------------------ | +| `query` | Process-grouped code intelligence — execution flows related to a concept | +| `context` | 360-degree symbol view — categorized refs, processes it participates in | +| `impact` | Symbol blast radius — what breaks at depth 1/2/3 with confidence | +| `detect_changes` | Git-diff impact — what do your current changes affect | +| `rename` | Multi-file coordinated rename with confidence-tagged edits | +| `cypher` | Raw graph queries (read `gitnexus://repo/{name}/schema` first) | +| `list_repos` | Discover indexed repos | + +## Resources Reference + +Lightweight reads (~100-500 tokens) for navigation: + +| Resource | Content | +| ---------------------------------------------- | ----------------------------------------- | +| `gitnexus://repo/{name}/context` | Stats, staleness check | +| `gitnexus://repo/{name}/clusters` | All functional areas with cohesion scores | +| `gitnexus://repo/{name}/cluster/{clusterName}` | Area members | +| `gitnexus://repo/{name}/processes` | All execution flows | +| `gitnexus://repo/{name}/process/{processName}` | Step-by-step trace | +| `gitnexus://repo/{name}/schema` | Graph schema for Cypher | + +## Graph Schema + +**Nodes:** File, Function, Class, Interface, Method, Community, Process +**Edges (via CodeRelation.type):** CALLS, IMPORTS, EXTENDS, IMPLEMENTS, DEFINES, MEMBER_OF, STEP_IN_PROCESS + +```cypher +MATCH (caller)-[:CodeRelation {type: 'CALLS'}]->(f:Function {name: "myFunc"}) +RETURN caller.name, caller.filePath +``` diff --git a/.claude/skills/gitnexus/gitnexus-impact-analysis/SKILL.md b/.claude/skills/gitnexus/gitnexus-impact-analysis/SKILL.md new file mode 100644 index 0000000..e19af28 --- /dev/null +++ b/.claude/skills/gitnexus/gitnexus-impact-analysis/SKILL.md @@ -0,0 +1,97 @@ +--- +name: gitnexus-impact-analysis +description: "Use when the user wants to know what will break if they change something, or needs safety analysis before editing code. Examples: \"Is it safe to change X?\", \"What depends on this?\", \"What will break?\"" +--- + +# Impact Analysis with GitNexus + +## When to Use + +- "Is it safe to change this function?" +- "What will break if I modify X?" +- "Show me the blast radius" +- "Who uses this code?" +- Before making non-trivial code changes +- Before committing — to understand what your changes affect + +## Workflow + +``` +1. gitnexus_impact({target: "X", direction: "upstream"}) → What depends on this +2. READ gitnexus://repo/{name}/processes → Check affected execution flows +3. gitnexus_detect_changes() → Map current git changes to affected flows +4. Assess risk and report to user +``` + +> If "Index is stale" → run `npx gitnexus analyze` in terminal. + +## Checklist + +``` +- [ ] gitnexus_impact({target, direction: "upstream"}) to find dependents +- [ ] Review d=1 items first (these WILL BREAK) +- [ ] Check high-confidence (>0.8) dependencies +- [ ] READ processes to check affected execution flows +- [ ] gitnexus_detect_changes() for pre-commit check +- [ ] Assess risk level and report to user +``` + +## Understanding Output + +| Depth | Risk Level | Meaning | +| ----- | ---------------- | ------------------------ | +| d=1 | **WILL BREAK** | Direct callers/importers | +| d=2 | LIKELY AFFECTED | Indirect dependencies | +| d=3 | MAY NEED TESTING | Transitive effects | + +## Risk Assessment + +| Affected | Risk | +| ------------------------------ | -------- | +| <5 symbols, few processes | LOW | +| 5-15 symbols, 2-5 processes | MEDIUM | +| >15 symbols or many processes | HIGH | +| Critical path (auth, payments) | CRITICAL | + +## Tools + +**gitnexus_impact** — the primary tool for symbol blast radius: + +``` +gitnexus_impact({ + target: "validateUser", + direction: "upstream", + minConfidence: 0.8, + maxDepth: 3 +}) + +→ d=1 (WILL BREAK): + - loginHandler (src/auth/login.ts:42) [CALLS, 100%] + - apiMiddleware (src/api/middleware.ts:15) [CALLS, 100%] + +→ d=2 (LIKELY AFFECTED): + - authRouter (src/routes/auth.ts:22) [CALLS, 95%] +``` + +**gitnexus_detect_changes** — git-diff based impact analysis: + +``` +gitnexus_detect_changes({scope: "staged"}) + +→ Changed: 5 symbols in 3 files +→ Affected: LoginFlow, TokenRefresh, APIMiddlewarePipeline +→ Risk: MEDIUM +``` + +## Example: "What breaks if I change validateUser?" + +``` +1. gitnexus_impact({target: "validateUser", direction: "upstream"}) + → d=1: loginHandler, apiMiddleware (WILL BREAK) + → d=2: authRouter, sessionManager (LIKELY AFFECTED) + +2. READ gitnexus://repo/my-app/processes + → LoginFlow and TokenRefresh touch validateUser + +3. Risk: 2 direct callers, 2 processes = MEDIUM +``` diff --git a/.claude/skills/gitnexus/gitnexus-refactoring/SKILL.md b/.claude/skills/gitnexus/gitnexus-refactoring/SKILL.md new file mode 100644 index 0000000..f48cc01 --- /dev/null +++ b/.claude/skills/gitnexus/gitnexus-refactoring/SKILL.md @@ -0,0 +1,121 @@ +--- +name: gitnexus-refactoring +description: "Use when the user wants to rename, extract, split, move, or restructure code safely. Examples: \"Rename this function\", \"Extract this into a module\", \"Refactor this class\", \"Move this to a separate file\"" +--- + +# Refactoring with GitNexus + +## When to Use + +- "Rename this function safely" +- "Extract this into a module" +- "Split this service" +- "Move this to a new file" +- Any task involving renaming, extracting, splitting, or restructuring code + +## Workflow + +``` +1. gitnexus_impact({target: "X", direction: "upstream"}) → Map all dependents +2. gitnexus_query({query: "X"}) → Find execution flows involving X +3. gitnexus_context({name: "X"}) → See all incoming/outgoing refs +4. Plan update order: interfaces → implementations → callers → tests +``` + +> If "Index is stale" → run `npx gitnexus analyze` in terminal. + +## Checklists + +### Rename Symbol + +``` +- [ ] gitnexus_rename({symbol_name: "oldName", new_name: "newName", dry_run: true}) — preview all edits +- [ ] Review graph edits (high confidence) and ast_search edits (review carefully) +- [ ] If satisfied: gitnexus_rename({..., dry_run: false}) — apply edits +- [ ] gitnexus_detect_changes() — verify only expected files changed +- [ ] Run tests for affected processes +``` + +### Extract Module + +``` +- [ ] gitnexus_context({name: target}) — see all incoming/outgoing refs +- [ ] gitnexus_impact({target, direction: "upstream"}) — find all external callers +- [ ] Define new module interface +- [ ] Extract code, update imports +- [ ] gitnexus_detect_changes() — verify affected scope +- [ ] Run tests for affected processes +``` + +### Split Function/Service + +``` +- [ ] gitnexus_context({name: target}) — understand all callees +- [ ] Group callees by responsibility +- [ ] gitnexus_impact({target, direction: "upstream"}) — map callers to update +- [ ] Create new functions/services +- [ ] Update callers +- [ ] gitnexus_detect_changes() — verify affected scope +- [ ] Run tests for affected processes +``` + +## Tools + +**gitnexus_rename** — automated multi-file rename: + +``` +gitnexus_rename({symbol_name: "validateUser", new_name: "authenticateUser", dry_run: true}) +→ 12 edits across 8 files +→ 10 graph edits (high confidence), 2 ast_search edits (review) +→ Changes: [{file_path, edits: [{line, old_text, new_text, confidence}]}] +``` + +**gitnexus_impact** — map all dependents first: + +``` +gitnexus_impact({target: "validateUser", direction: "upstream"}) +→ d=1: loginHandler, apiMiddleware, testUtils +→ Affected Processes: LoginFlow, TokenRefresh +``` + +**gitnexus_detect_changes** — verify your changes after refactoring: + +``` +gitnexus_detect_changes({scope: "all"}) +→ Changed: 8 files, 12 symbols +→ Affected processes: LoginFlow, TokenRefresh +→ Risk: MEDIUM +``` + +**gitnexus_cypher** — custom reference queries: + +```cypher +MATCH (caller)-[:CodeRelation {type: 'CALLS'}]->(f:Function {name: "validateUser"}) +RETURN caller.name, caller.filePath ORDER BY caller.filePath +``` + +## Risk Rules + +| Risk Factor | Mitigation | +| ------------------- | ----------------------------------------- | +| Many callers (>5) | Use gitnexus_rename for automated updates | +| Cross-area refs | Use detect_changes after to verify scope | +| String/dynamic refs | gitnexus_query to find them | +| External/public API | Version and deprecate properly | + +## Example: Rename `validateUser` to `authenticateUser` + +``` +1. gitnexus_rename({symbol_name: "validateUser", new_name: "authenticateUser", dry_run: true}) + → 12 edits: 10 graph (safe), 2 ast_search (review) + → Files: validator.ts, login.ts, middleware.ts, config.json... + +2. Review ast_search edits (config.json: dynamic reference!) + +3. gitnexus_rename({symbol_name: "validateUser", new_name: "authenticateUser", dry_run: false}) + → Applied 12 edits across 8 files + +4. gitnexus_detect_changes({scope: "all"}) + → Affected: LoginFlow, TokenRefresh + → Risk: MEDIUM — run tests for these flows +``` diff --git a/.env.example b/.env.example index b209b36..bb87908 100644 --- a/.env.example +++ b/.env.example @@ -1,5 +1,6 @@ # ── Server ─────────────────────────────────────────────────────── -HOST=0.0.0.0 +# Local/development default. Production containers override this behind the edge proxy. +HOST=127.0.0.1 PORT=8000 APP_ENV=dev CONFIG_PROFILE=dev @@ -16,8 +17,8 @@ LMSTUDIO_MODEL= # ── LLM: Anthropic (Cloud — for complex tasks) ────────────────── ANTHROPIC_API_KEY=sk-ant-your-key-here -ANTHROPIC_MODEL_MEDIUM=claude-3-5-haiku-20241022 -ANTHROPIC_MODEL_COMPLEX=claude-sonnet-4-20250514 +ANTHROPIC_MODEL_MEDIUM=claude-haiku-4-5-20251001 +ANTHROPIC_MODEL_COMPLEX=claude-sonnet-4-6 # ── LLM: OpenAI (Fallback) ────────────────────────────────────── OPENAI_API_KEY=sk-your-key-here @@ -29,31 +30,102 @@ DEFAULT_PROVIDER=lmstudio # ── Auth ───────────────────────────────────────────────────────── API_KEY=change-this-to-a-strong-key +# Server-side HMAC pepper for persisted enterprise API keys. Generate a +# separate 32+ character secret; never reuse API_KEY or expose it to clients. +API_KEY_HMAC_PEPPER= +# HMAC-authenticated deletion receipts. Production requires a stable key ID +# and a distinct 32+ byte secret encoded as URL-safe base64. During rotation, +# move retired keys into the JSON map until all retained receipts age out. +EXCEL_AGENT_LIFECYCLE_RECEIPT_KEY_ID= +EXCEL_AGENT_LIFECYCLE_RECEIPT_HMAC_KEY= +EXCEL_AGENT_LIFECYCLE_RECEIPT_PREVIOUS_KEYS={} AUTH_ENABLED=false RATE_LIMIT_RPM=60 TRUSTED_PROXY_HEADERS=false APPLY_TOKEN_TTL_SECONDS=900 ALLOW_LEGACY_APPLY_FINAL_STATUS=false +# ── Local desktop pairing (supported local mode only) ─────────── +# Never enable this in prod/production/enterprise profiles. Before enabling, +# keep the backend on HOST=127.0.0.1 (or ::1) above and set AUTH_ENABLED=true. +# The desktop owner supplies one 6-128 character [A-Za-z0-9_-] code at boot; +# it is removed from the process environment, expires, and is consumed once. +# Issued local.* bearers use an OS-keychain signing secret and expire quickly. +LOCAL_PAIRING_ENABLED=false +LOCAL_PAIRING_CODE= +LOCAL_PAIRING_CODE_TTL_SECONDS=120 +LOCAL_PAIRING_TOKEN_TTL_SECONDS=300 +LOCAL_PAIRING_MAX_ATTEMPTS=3 +LOCAL_PAIRING_LOCKOUT_SECONDS=60 +LOCAL_PAIRING_RATE_LIMIT_RPM=3 + # ── Database ───────────────────────────────────────────────────── DB_ENGINE=sqlite DATABASE_URL= DB_PATH=./data/sessions.db DATA_DIR=./data +# ── Data Connectors ───────────────────────────────────────────────────── +# SEC requires an identifying User-Agent with contact information. +SEC_EDGAR_USER_AGENT=FinanceAI/1.0 (internal-tool; contact@example.com) +SEC_EDGAR_ENABLE_LIVE_REQUESTS=true +SEC_EDGAR_TIMEOUT_SECONDS=15 +SEC_EDGAR_MAX_FILINGS=10 + +# Bounded local CSV parsing. Inputs beyond any configured limit fail closed. +CSV_CONNECTOR_MAX_INPUT_BYTES=8388608 +CSV_CONNECTOR_MAX_ROWS=100000 +CSV_CONNECTOR_MAX_COLUMNS=1024 +CSV_CONNECTOR_MAX_CELL_CHARACTERS=65536 +CSV_CONNECTOR_MAX_OUTPUT_CELLS=1000000 + +# Bounded XBRL/iXBRL parsing. DTD/entity expansion is rejected; only the plain +# HTML doctype is accepted for inline filings. +XBRL_CONNECTOR_MAX_INPUT_BYTES=16777216 +XBRL_CONNECTOR_MAX_FACTS=100000 +XBRL_CONNECTOR_MAX_FACT_CHARACTERS=1048576 +XBRL_CONNECTOR_MAX_XML_ELEMENTS=250000 + # Workbook worker integrations # sheetjs_exceljs_hyperformula uses workbook-worker/ Node CLI; python_fallback stays pure Python. WORKBOOK_WORKER_MODE=sheetjs_exceljs_hyperformula WORKBOOK_WORKER_DIR=./workbook-worker WORKBOOK_WORKER_TIMEOUT_SECONDS=30 +WORKBOOK_WORKER_MAX_OLD_SPACE_MB=512 +WORKBOOK_WORKER_MAX_OUTPUT_BYTES=16777216 OFFICECLI_SCHEMA_MODE=enabled -FORMULA_ENGINE=hyperformula -HYPERFORMULA_ENABLED=true -HYPERFORMULA_LICENSE_APPROVED=true +FORMULA_ENGINE=univer +# Disabled by default for the proprietary distribution. Enable only with an +# approved commercial license and an explicitly injected runtime key. +HYPERFORMULA_ENABLED=false +# The standard private-beta artifacts omit HyperFormula. A separately licensed +# custom build may set this true only when the runtime is actually included. +HYPERFORMULA_DISTRIBUTED=false +HYPERFORMULA_LICENSE_APPROVED=false +# Required for production/commercial HyperFormula deployments. HYPERFORMULA_LICENSE_KEY= WORKBOOK_WORKER_MAX_PAYLOAD_BYTES=12582912 WORKBOOK_WORKER_ALLOWED_EXTENSIONS=.xlsx,.xlsm,.xlsb,.xls,.csv,.tsv +# Shared bounded admission for workbook, managed-XLSX, and PDF subprocess work. +# Queue overflow and queue timeout fail with stable retryable errors; requests +# never fall back to an in-process parser after admission fails. +WORK_ADMISSION_MAX_CONCURRENCY=4 +WORK_ADMISSION_MAX_QUEUE=64 +WORK_ADMISSION_MAX_QUEUE_PER_SESSION=8 +WORK_ADMISSION_QUEUE_TIMEOUT_SECONDS=5 +WORKBOOK_WORKER_MAX_CONCURRENCY=2 +WORKBOOK_WORKER_MAX_QUEUE=32 + +# Bounded PDF parser process. Windows uses a Job Object; POSIX uses rlimits. +PDF_PARSE_TIMEOUT_SECONDS=30 +PDF_PARSE_CPU_SECONDS=20 +PDF_PARSE_MEMORY_BYTES=805306368 +PDF_PARSE_MAX_OUTPUT_BYTES=16777216 +PDF_PARSE_MAX_CONCURRENCY=2 +PDF_PARSE_MAX_QUEUE=16 +PDF_PARSE_QUEUE_TIMEOUT_SECONDS=5 + # ── Flagship Accuracy Rollout ─────────────────────────────────── PLAN_ENGINE_VERSION=v2 AGENT_RUNNER_VERSION=disabled @@ -71,10 +143,85 @@ RBAC_ENABLED=false SSO_ENABLED=false SSO_PROVIDER= SSO_IDP_METADATA_URL= +# On-prem user authentication. Users must also be pre-provisioned in the +# enterprise user store; token roles are ignored in favor of stored RBAC. +OIDC_ENABLED=false +OIDC_ISSUER= +OIDC_AUDIENCE= +OIDC_JWKS_URL= +OIDC_ALLOWED_TENANTS= +# Must match the Office client's WebCrypto verifier: RS256, PS256, and/or ES256. +OIDC_ALLOWED_ALGORITHMS=RS256 +OIDC_CLOCK_SKEW_SECONDS=60 +OIDC_REQUIRE_MFA=true +OIDC_ACCEPTED_MFA_ACR= +OIDC_JWKS_TIMEOUT_SECONDS=5 +OIDC_JWKS_CACHE_SECONDS=300 +# Public authorization-code + PKCE client metadata (never configure a client secret). +OIDC_CLIENT_ID= +OIDC_CLIENT_REDIRECT_URI=https://localhost:3443/src/taskpane/taskpane.html?oidc_callback=1 +OIDC_CLIENT_SCOPES=openid,profile,email +OIDC_DISCOVERY_URL= +OIDC_AUTHORIZATION_ENDPOINT= +OIDC_TOKEN_ENDPOINT= +OIDC_EXTRA_AUTHORIZE_PARAMS_JSON={} +# Edge-only CSP allowlist: exact space-separated HTTPS origins used by browser +# fetches to OIDC discovery, token, and JWKS endpoints. Never use wildcards or +# broad `https:`. Authorization-page navigation is not a connect-src fetch. +OIDC_BROWSER_CONNECT_ORIGINS= BYOK_ENABLED=false ZERO_DATA_RETENTION=false DATA_RETENTION_HOURS=0 +AI_ALLOWED_MODES=ask,plan,preview,action +AI_ALLOWED_PROVIDERS=lmstudio,ollama,anthropic,openai +AI_EXTERNAL_SENDING_ALLOWED=true +AI_EXTERNAL_DOCUMENTS_ALLOWED=true +AI_SOURCE_RETENTION_HOURS=0 +AI_EXPORT_REGENERATED_FALLBACK_ALLOWED=true +AI_EXPORT_MACRO_REGENERATED_FALLBACK_ALLOWED=false +LLM_EXACT_CACHE_ENABLED=false +LLM_VECTOR_CACHE_ENABLED=false # Optional database encryption at rest. Requires SQLCipher support. DB_ENCRYPT=false +# Required only when DB_ENCRYPT=true: exactly 32 random bytes encoded as 64 lowercase hex characters. DB_ENCRYPT_KEY= +# Stable non-secret identifier for the active SQLite key generation. Rotation +# receipts bind this identifier without disclosing key material. +DB_ENCRYPT_KEY_ID= + +# Optional detached platform/PostgreSQL encryption attestation. The trusted +# Ed25519 public key must be supplied out of repository and out of the evidence +# artifact directory; all fields must bind the exact release SHA. +STORAGE_ENCRYPTION_ATTESTATION_PATH= +STORAGE_ENCRYPTION_ATTESTATION_SIGNATURE_PATH= +STORAGE_ENCRYPTION_ATTESTATION_PUBLIC_KEY_PATH= +STORAGE_ENCRYPTION_ATTESTATION_KEY_ID= +STORAGE_ENCRYPTION_ATTESTATION_RELEASE_SHA= +STORAGE_ENCRYPTION_ATTESTATION_DEPLOYMENT_SHA256= +STORAGE_ENCRYPTION_ATTESTATION_RUNNER_SHA256= +STORAGE_ENCRYPTION_ATTESTATION_KEY_IDENTITY_SHA256= +STORAGE_ENCRYPTION_ATTESTATION_PUBLIC_KEY_SHA256= +# Defaults to seven days; accepted runtime values are bounded to 30 days. +STORAGE_ENCRYPTION_ATTESTATION_MAX_AGE_SECONDS=604800 + +# Optional enterprise managed-object storage substrate. It is disabled and is +# not wired into the default/local-first workbook path. Enabling it requires an +# explicit storage construction and lifecycle-purger registration by the +# enterprise deployment. Keys must be provisioned out of repository. +MANAGED_OBJECT_STORE_ENABLED=false +MANAGED_OBJECT_STORE_ROOT= +MANAGED_OBJECT_STORE_CURRENT_KEY_ID= +# Exactly 32 bytes encoded as 64 hexadecimal characters or URL-safe base64. +MANAGED_OBJECT_STORE_CURRENT_KEY= +# A distinct exact 32-byte address key; do not reuse an encryption key. +MANAGED_OBJECT_STORE_ADDRESS_KEY= +# Optional JSON object of at most four prior key-id -> encoded-key entries. +MANAGED_OBJECT_STORE_PREVIOUS_KEYS={} +MANAGED_OBJECT_STORE_MAX_OBJECT_BYTES=67108864 +MANAGED_OBJECT_STORE_TOTAL_BYTES=1073741824 +MANAGED_OBJECT_STORE_CHUNK_BYTES=262144 +# Rotation is an explicit offline operator action; leave these empty otherwise. +MANAGED_OBJECT_STORE_ROTATION_ID= +MANAGED_OBJECT_STORE_NEW_KEY_ID= +MANAGED_OBJECT_STORE_NEW_KEY= diff --git a/.github/CODEOWNERS b/.github/CODEOWNERS new file mode 100644 index 0000000..d25092d --- /dev/null +++ b/.github/CODEOWNERS @@ -0,0 +1,15 @@ +# Repository-wide ownership is intentionally explicit while the project is a +# private beta. Add domain owners only after they can review the complete +# security, workbook-correctness, and release-evidence boundary. +* @stag175 + +/.github/ @stag175 +/backend/ @stag175 +/plugin/ @stag175 +/workbook-worker/ @stag175 +/vnext/ @stag175 +/infra/ @stag175 +/scripts/ @stag175 +/docs/ @stag175 +/SECURITY.md @stag175 +/LICENSE @stag175 diff --git a/.github/ISSUE_TEMPLATE/bug.yml b/.github/ISSUE_TEMPLATE/bug.yml new file mode 100644 index 0000000..c12e6ab --- /dev/null +++ b/.github/ISSUE_TEMPLATE/bug.yml @@ -0,0 +1,54 @@ +name: Bug report +description: Report a reproducible product, workbook, or operational defect. +title: "[Bug] " +labels: ["bug"] +body: + - type: markdown + attributes: + value: Do not include workbook content, prompts, credentials, private evidence, or security vulnerabilities. + - type: dropdown + id: surface + attributes: + label: Surface + options: + - Excel taskpane + - Browser spreadsheet + - Backend/API + - Workbook worker/OOXML + - Identity/administration + - Storage/lifecycle + - Deployment/release + - vNext + validations: + required: true + - type: input + id: version + attributes: + label: Version or commit + placeholder: v0.5.0 or full commit SHA + validations: + required: true + - type: textarea + id: behavior + attributes: + label: Expected and actual behavior + description: Use synthetic/redacted examples only. + validations: + required: true + - type: textarea + id: reproduction + attributes: + label: Minimal reproduction + description: Include host/build, configuration profile, and exact steps. + validations: + required: true + - type: textarea + id: trust + attributes: + label: Trust-boundary impact + description: Note fidelity, rollback, ownership, privacy, egress, migration, or evidence implications. + - type: textarea + id: verification + attributes: + label: Local verification + description: List commands and sanitized results already attempted. diff --git a/.github/ISSUE_TEMPLATE/config.yml b/.github/ISSUE_TEMPLATE/config.yml new file mode 100644 index 0000000..8c5d340 --- /dev/null +++ b/.github/ISSUE_TEMPLATE/config.yml @@ -0,0 +1,8 @@ +blank_issues_enabled: false +contact_links: + - name: Private security report + url: https://github.com/stag175/excelaiplugin/security/policy + about: Do not disclose suspected vulnerabilities in a public issue. + - name: Support policy + url: https://github.com/stag175/excelaiplugin/blob/main/SUPPORT.md + about: Review supported versions, environments, and response objectives. diff --git a/.github/ISSUE_TEMPLATE/refinement.yml b/.github/ISSUE_TEMPLATE/refinement.yml new file mode 100644 index 0000000..3b8ebe6 --- /dev/null +++ b/.github/ISSUE_TEMPLATE/refinement.yml @@ -0,0 +1,41 @@ +name: Refinement proposal +description: Propose a product polish, reliability improvement, or vNext work package. +title: "[Refinement] " +labels: ["enhancement"] +body: + - type: dropdown + id: priority + attributes: + label: Proposed priority + options: + - P0 capability blocker + - P1 private-beta quality + - P2 enterprise/completeness + - P3 strategic bet + validations: + required: true + - type: textarea + id: outcome + attributes: + label: User or operator outcome + validations: + required: true + - type: textarea + id: acceptance + attributes: + label: Testable acceptance criteria + validations: + required: true + - type: textarea + id: dependencies + attributes: + label: Dependencies and non-goals + - type: checkboxes + id: boundary + attributes: + label: Boundary checks + options: + - label: This does not hide a release-evidence requirement. + required: true + - label: This does not introduce hidden legacy/vNext compatibility. + required: true diff --git a/.github/ISSUE_TEMPLATE/release-evidence.yml b/.github/ISSUE_TEMPLATE/release-evidence.yml new file mode 100644 index 0000000..1eca8ff --- /dev/null +++ b/.github/ISSUE_TEMPLATE/release-evidence.yml @@ -0,0 +1,41 @@ +name: Release evidence work item +description: Track genuine exact-SHA external evidence without storing private bytes in Git. +title: "[Evidence] " +labels: ["documentation"] +body: + - type: dropdown + id: evidence_class + attributes: + label: Evidence class + options: + - Windows Excel smoke + - macOS Excel smoke + - Excel Web smoke + - Private workbook corpus + - Production replay + - PostgreSQL migration/restore + - OIDC/mTLS topology + - Image scan/signature/provenance + - Independent penetration test + - Legal/release sign-off + validations: + required: true + - type: input + id: sha + attributes: + label: Exact 40-character source SHA + validations: + required: true + - type: textarea + id: owner + attributes: + label: Evidence owner, reviewer, and storage location + description: Do not paste private artifact content or credentials. + validations: + required: true + - type: textarea + id: acceptance + attributes: + label: Completion criteria + validations: + required: true diff --git a/.github/PULL_REQUEST_TEMPLATE.md b/.github/PULL_REQUEST_TEMPLATE.md new file mode 100644 index 0000000..028fdd9 --- /dev/null +++ b/.github/PULL_REQUEST_TEMPLATE.md @@ -0,0 +1,41 @@ +## Outcome + + + +## Root cause and trust boundary + + + +## Scope + +- [ ] Legacy 0.5 only +- [ ] vNext only +- [ ] Generated contracts/artifacts included +- [ ] No hidden legacy/vNext dependency introduced + +## Risk and compatibility + +- Workbook fidelity: +- Apply/rollback/recovery: +- Identity/ownership/privacy: +- Storage/migration/retention: +- Egress/licensing/dependencies: +- Performance/large-workbook behavior: +- Rollback or forward-fix plan: + +## Evidence + + + +- [ ] Regression failed before the change and passes after it +- [ ] Focused tests +- [ ] Static/type/lint gates +- [ ] Full affected verification profile +- [ ] GitNexus impact and detect-changes reviewed +- [ ] External/native evidence attached if this PR claims it + +## Release posture + +- [ ] This PR does not claim GA, signing, native-host certification, private + corpus coverage, or production evidence unless exact-SHA artifacts are attached +- [ ] No GitHub-hosted runner minutes are required diff --git a/.github/actionlint.yaml b/.github/actionlint.yaml new file mode 100644 index 0000000..396d1fe --- /dev/null +++ b/.github/actionlint.yaml @@ -0,0 +1,4 @@ +self-hosted-runner: + labels: + - excel-agent + - excel-agent-excel diff --git a/.github/dependabot.yml b/.github/dependabot.yml index 4e83a13..43ddcaa 100644 --- a/.github/dependabot.yml +++ b/.github/dependabot.yml @@ -31,6 +31,30 @@ updates: day: "monday" open-pull-requests-limit: 5 + - package-ecosystem: "npm" + directory: "/vnext" + schedule: + interval: "weekly" + day: "monday" + open-pull-requests-limit: 3 + groups: + vnext-typescript: + patterns: ["@bufbuild/*", "tsx", "typescript"] + + - package-ecosystem: "pip" + directory: "/vnext" + schedule: + interval: "weekly" + day: "monday" + open-pull-requests-limit: 3 + + - package-ecosystem: "cargo" + directory: "/vnext" + schedule: + interval: "weekly" + day: "monday" + open-pull-requests-limit: 3 + - package-ecosystem: "github-actions" directory: "/" schedule: @@ -42,3 +66,15 @@ updates: schedule: interval: "weekly" day: "monday" + + - package-ecosystem: "docker" + directory: "/infra/nginx" + schedule: + interval: "weekly" + day: "monday" + + - package-ecosystem: "docker" + directory: "/" + schedule: + interval: "weekly" + day: "monday" diff --git a/.github/workflows/codeql.yml b/.github/workflows/codeql.yml index 1a5e59a..45ec3d2 100644 --- a/.github/workflows/codeql.yml +++ b/.github/workflows/codeql.yml @@ -1,5 +1,7 @@ name: CodeQL +# CodeQL executes on an ephemeral self-hosted runner; there is no hosted fallback. + on: push: branches: [main] @@ -18,20 +20,21 @@ concurrency: jobs: analyze: + if: github.event_name != 'pull_request' || github.event.pull_request.head.repo.full_name == github.repository name: Analyze (${{ matrix.language }}) - runs-on: ubuntu-latest + runs-on: [self-hosted, linux, x64, excel-agent] timeout-minutes: 30 strategy: fail-fast: false matrix: language: [python, javascript-typescript] steps: - - uses: actions/checkout@v4 - - uses: github/codeql-action/init@v3 + - uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4.4.0 + - uses: github/codeql-action/init@c4dd10e44af883a891fe31ced449bcb4a6728b9b # v3.37.6 with: languages: ${{ matrix.language }} queries: security-and-quality - - uses: github/codeql-action/autobuild@v3 - - uses: github/codeql-action/analyze@v3 + - uses: github/codeql-action/autobuild@c4dd10e44af883a891fe31ced449bcb4a6728b9b # v3.37.6 + - uses: github/codeql-action/analyze@c4dd10e44af883a891fe31ced449bcb4a6728b9b # v3.37.6 with: category: "/language:${{ matrix.language }}" diff --git a/.github/workflows/secret-scan.yml b/.github/workflows/secret-scan.yml index c49af16..1067de9 100644 --- a/.github/workflows/secret-scan.yml +++ b/.github/workflows/secret-scan.yml @@ -1,5 +1,7 @@ name: Secret scan +# Secret scanning executes on an ephemeral self-hosted runner. + on: push: branches: [main] @@ -15,13 +17,14 @@ concurrency: jobs: gitleaks: - runs-on: ubuntu-latest + if: github.event_name != 'pull_request' || github.event.pull_request.head.repo.full_name == github.repository + runs-on: [self-hosted, linux, x64, excel-agent] timeout-minutes: 10 steps: - - uses: actions/checkout@v4 + - uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4.4.0 with: fetch-depth: 0 - name: Run gitleaks - uses: gitleaks/gitleaks-action@v2 + uses: gitleaks/gitleaks-action@ff98106e4c7b2bc287b24eaf42907196329070c7 # v2.3.9 env: GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} diff --git a/.github/workflows/v7-readiness.yml b/.github/workflows/v7-readiness.yml index 425df85..099c241 100644 --- a/.github/workflows/v7-readiness.yml +++ b/.github/workflows/v7-readiness.yml @@ -1,11 +1,31 @@ name: V7 Readiness Gates +# No job in this workflow may fall back to GitHub-hosted runners. The labels +# below are provisioned as ephemeral, clean-workspace runners by the operator. + on: push: branches: [main] pull_request: + workflow_dispatch: + inputs: + release_evidence: + description: Run strict release evidence gates against a previously uploaded evidence artifact + type: boolean + default: false + release_evidence_artifact_run_id: + description: Workflow run ID that uploaded the release-evidence artifact + type: string + required: false + default: "" + release_evidence_artifact_name: + description: Artifact containing the signed evidence index, detached signature, and only indexed allowlisted files + type: string + required: false + default: "" permissions: + actions: read contents: read concurrency: @@ -14,42 +34,46 @@ concurrency: jobs: backend-fast: - runs-on: ubuntu-latest + if: github.event_name != 'pull_request' || github.event.pull_request.head.repo.full_name == github.repository + runs-on: [self-hosted, linux, x64, excel-agent] timeout-minutes: 15 steps: - - uses: actions/checkout@v4 - - uses: actions/setup-python@v5 + - uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4.4.0 + - uses: actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065 # v5.6.0 with: python-version: "3.12" cache: pip - - uses: actions/setup-node@v4 + - uses: actions/setup-node@49933ea5288caeca8642d1e84afbd3f7d6820020 # v4.4.0 with: - node-version: "22" + node-version: "24" - name: Install backend dependencies - run: python -m pip install -r backend/requirements.txt + run: python -m pip install --require-hashes -r backend/requirements.lock.txt - name: Backend fast tests run: python -m pytest backend/tests -q -m "not slow" backend-slow-v7: - runs-on: ubuntu-latest + if: github.event_name != 'pull_request' || github.event.pull_request.head.repo.full_name == github.repository + runs-on: [self-hosted, linux, x64, excel-agent] timeout-minutes: 35 steps: - - uses: actions/checkout@v4 - - uses: actions/setup-python@v5 + - uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4.4.0 + - uses: actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065 # v5.6.0 with: python-version: "3.12" cache: pip - - uses: actions/setup-node@v4 + - uses: actions/setup-node@49933ea5288caeca8642d1e84afbd3f7d6820020 # v4.4.0 with: - node-version: "22" + node-version: "24" - name: Install backend dependencies - run: python -m pip install -r backend/requirements.txt + run: python -m pip install --require-hashes -r backend/requirements.lock.txt - name: Install workbook worker dependencies run: npm ci --prefix workbook-worker - name: Smoke workbook worker run: node workbook-worker/src/cli.mjs --help - name: Backend slow V7/workbook tests run: python -m pytest backend/tests -q -m slow + - name: Parser comparison corpus + run: python scripts/run_parser_comparison_corpus.py --fail-on-mismatch - name: Full V7 readiness modules run: | python -m backend.benchmarks.local_accuracy @@ -58,24 +82,45 @@ jobs: python -m backend.benchmarks.v7_readiness contracts: - runs-on: ubuntu-latest + if: github.event_name != 'pull_request' || github.event.pull_request.head.repo.full_name == github.repository + runs-on: [self-hosted, linux, x64, excel-agent] timeout-minutes: 15 steps: - - uses: actions/checkout@v4 - - uses: actions/setup-python@v5 + - uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4.4.0 + - name: Dependency review + if: github.event_name == 'pull_request' + uses: actions/dependency-review-action@2031cfc080254a8a887f58cffee85186f0e49e48 # v4.9.0 + with: + fail-on-severity: moderate + retry-on-snapshot-warnings: true + - uses: actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065 # v5.6.0 with: python-version: "3.12" cache: pip - - uses: actions/setup-node@v4 + - uses: actions/setup-node@49933ea5288caeca8642d1e84afbd3f7d6820020 # v4.4.0 with: - node-version: "22" + node-version: "24" - name: Install backend dependencies - run: python -m pip install -r backend/requirements.txt + run: python -m pip install --require-hashes -r backend/requirements.lock.txt + - name: Verify pinned toolchain and workflow syntax + run: | + python scripts/verify_toolchain_manifest.py + go run github.com/rhysd/actionlint/cmd/actionlint@v1.7.7 .github/workflows/*.yml + - name: Product and Office version synchronization + run: python scripts/check_version_sync.py + - name: Package and lock integrity + run: python scripts/check_package_integrity.py + - name: CycloneDX SBOM drift + run: python scripts/generate_sbom.py --check + - name: Backend semantic Ruff gate + run: python -m ruff check backend scripts --select F,E9,B,ASYNC + - name: Backend mypy gate + run: python -m mypy backend --ignore-missing-imports - name: OpenAPI type drift run: python scripts/generate_openapi_types.py --check - name: Upload generated OpenAPI type artifact on failure if: failure() - uses: actions/upload-artifact@v4 + uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4.6.2 with: name: taskpane-api-types path: plugin/src/taskpane/api-types.ts @@ -84,77 +129,124 @@ jobs: - name: pip-audit run: | python -m pip install pip-audit - python -m pip_audit -r backend/requirements.txt --strict --vulnerability-service osv + python -m pip_audit -r backend/requirements.lock.txt --strict --vulnerability-service osv - name: Dependency risk policy run: python scripts/check_dependency_risks.py plugin: - runs-on: ubuntu-latest + if: github.event_name != 'pull_request' || github.event.pull_request.head.repo.full_name == github.repository + runs-on: [self-hosted, linux, x64, excel-agent] timeout-minutes: 20 defaults: run: working-directory: plugin steps: - - uses: actions/checkout@v4 - - uses: actions/setup-node@v4 + - uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4.4.0 + - uses: actions/setup-node@49933ea5288caeca8642d1e84afbd3f7d6820020 # v4.4.0 with: - node-version: "22" + node-version: "24" cache: npm cache-dependency-path: plugin/package-lock.json - run: npm ci + - name: Install Playwright Chromium + run: npx playwright install --with-deps chromium + - name: npm audit critical + run: npm run audit:critical - run: npm run typecheck - run: npm run lint - run: npm run test:office-mock + - run: npm run test:frontend-smoke - run: npm run test:taskpane + - run: npm run test:spreadsheet-browser-smoke - run: npm run build - - run: npm audit --audit-level=high worker: - runs-on: ubuntu-latest + if: github.event_name != 'pull_request' || github.event.pull_request.head.repo.full_name == github.repository + runs-on: [self-hosted, linux, x64, excel-agent] timeout-minutes: 20 defaults: run: working-directory: workbook-worker steps: - - uses: actions/checkout@v4 - - uses: actions/setup-node@v4 + - uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4.4.0 + - uses: actions/setup-node@49933ea5288caeca8642d1e84afbd3f7d6820020 # v4.4.0 with: - node-version: "22" + node-version: "24" cache: npm cache-dependency-path: workbook-worker/package-lock.json - - uses: actions/setup-python@v5 + - uses: actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065 # v5.6.0 with: python-version: "3.12" - run: npm ci + - name: npm audit critical + run: npm run audit:critical - run: npm test - - run: npm audit --audit-level=high + - name: Parser comparison corpus + working-directory: ${{ github.workspace }} + run: python scripts/run_parser_comparison_corpus.py --fail-on-mismatch - name: Dependency risk policy run: python ../scripts/check_dependency_risks.py docker-build: - runs-on: ubuntu-latest + if: github.event_name != 'pull_request' || github.event.pull_request.head.repo.full_name == github.repository + runs-on: [self-hosted, linux, x64, excel-agent] timeout-minutes: 30 permissions: contents: read security-events: write steps: - - uses: actions/checkout@v4 - - name: Docker build + - uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4.4.0 + - name: Backend Docker build run: docker build -f backend/Dockerfile -t finance-ai-excel-plugin-backend:ci . - - name: Trivy image scan - uses: aquasecurity/trivy-action@0.24.0 + - name: Production edge Docker build + run: docker build -f infra/nginx/Dockerfile -t finance-ai-excel-plugin-edge:ci . + - name: Production edge package contract + run: | + docker run --rm --entrypoint sh \ + -e DEPLOY_ORIGIN=https://excel-agent.example.test \ + -e OIDC_BROWSER_CONNECT_ORIGINS=https://identity.example.test \ + -e MANIFEST_OUTPUT=/tmp/manifest.xml \ + finance-ai-excel-plugin-edge:ci -ec ' + /docker-entrypoint.d/40-render-production-manifest.sh + test -s /usr/share/nginx/html/src/taskpane/taskpane.html + test -s /usr/share/nginx/html/src/spreadsheet/spreadsheet.html + test -s /usr/share/nginx/html/assets/icon-80.png + grep -Fq "https://excel-agent.example.test" /tmp/manifest.xml + grep -Fq "connect-src '\''self'\'' https://identity.example.test" /tmp/nginx-runtime/nginx.conf + ! grep -Eq "connect-src[^;]*[[:space:]](https:|wss:|\\*)([[:space:];]|$)" /tmp/nginx-runtime/nginx.conf + ! grep -Eiq "localhost|\\$\\{" /tmp/manifest.xml + ! find /usr/share/nginx/html -type f -name "*.map" -print -quit | grep -q . + ' + - name: Trivy backend image scan + uses: aquasecurity/trivy-action@ed142fd0673e97e23eac54620cfb913e5ce36c25 # v0.36.0 with: + version: v0.73.0 image-ref: finance-ai-excel-plugin-backend:ci format: sarif - output: trivy-results.sarif + output: trivy-backend-results.sarif severity: HIGH,CRITICAL ignore-unfixed: true - exit-code: "0" - - name: Upload Trivy SARIF + exit-code: "1" + - name: Upload backend Trivy SARIF if: always() - uses: github/codeql-action/upload-sarif@v3 + uses: github/codeql-action/upload-sarif@c4dd10e44af883a891fe31ced449bcb4a6728b9b # v3.37.6 with: - sarif_file: trivy-results.sarif + sarif_file: trivy-backend-results.sarif + - name: Trivy production edge image scan + uses: aquasecurity/trivy-action@ed142fd0673e97e23eac54620cfb913e5ce36c25 # v0.36.0 + with: + version: v0.73.0 + image-ref: finance-ai-excel-plugin-edge:ci + format: sarif + output: trivy-edge-results.sarif + severity: HIGH,CRITICAL + ignore-unfixed: true + exit-code: "1" + - name: Upload production edge Trivy SARIF + if: always() + uses: github/codeql-action/upload-sarif@c4dd10e44af883a891fe31ced449bcb4a6728b9b # v3.37.6 + with: + sarif_file: trivy-edge-results.sarif - name: Worker packaged smoke run: docker run --rm finance-ai-excel-plugin-backend:ci node /app/workbook-worker/src/cli.mjs --help - name: Backend health smoke @@ -173,6 +265,151 @@ jobs: done docker logs backend-smoke exit 1 + - name: Release trust metadata snapshot (non-blocking) + run: python scripts/generate_release_trust_bundle.py --mode metadata-only --label ci-docker-snapshot || true + - name: Windows Excel smoke validation snapshot (non-blocking) + if: always() + run: | + mkdir -p data/release-trust + python scripts/validate_windows_excel_smoke.py --json > data/release-trust/windows-excel-smoke-validation.json || true + - name: Upload release trust snapshot artifacts + if: always() + uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4.6.2 + with: + name: release-trust-snapshot-${{ github.run_id }} + path: data/release-trust/** - name: Cleanup smoke container if: always() run: docker rm -f backend-smoke || true + + release-evidence: + if: github.event_name == 'workflow_dispatch' && inputs.release_evidence + runs-on: [self-hosted, windows, x64, excel-agent-excel] + timeout-minutes: 45 + env: + HYPERFORMULA_LICENSE_APPROVED: ${{ vars.HYPERFORMULA_LICENSE_APPROVED }} + HYPERFORMULA_LICENSE_KEY: ${{ secrets.HYPERFORMULA_LICENSE_KEY }} + steps: + - uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4.4.0 + - name: Validate release evidence artifact inputs + shell: pwsh + env: + EVIDENCE_ARTIFACT_NAME: ${{ inputs.release_evidence_artifact_name }} + EVIDENCE_RUN_ID: ${{ inputs.release_evidence_artifact_run_id }} + RELEASE_EVIDENCE_SIGNING_KEY_ID: ${{ vars.RELEASE_EVIDENCE_SIGNING_KEY_ID }} + RELEASE_EVIDENCE_ED25519_PUBLIC_KEY_B64: ${{ secrets.RELEASE_EVIDENCE_ED25519_PUBLIC_KEY_B64 }} + run: | + if ($env:EVIDENCE_RUN_ID -notmatch '^\d+$') { + throw 'release_evidence_artifact_run_id must be the numeric ID of a prior workflow run.' + } + if ($env:EVIDENCE_ARTIFACT_NAME -notmatch '^[A-Za-z0-9][A-Za-z0-9._-]{0,127}$') { + throw 'release_evidence_artifact_name must contain only letters, numbers, dots, underscores, and hyphens.' + } + if ([string]::IsNullOrWhiteSpace($env:RELEASE_EVIDENCE_ED25519_PUBLIC_KEY_B64)) { + throw 'RELEASE_EVIDENCE_ED25519_PUBLIC_KEY_B64 secret is required for strict evidence validation.' + } + if ([string]::IsNullOrWhiteSpace($env:RELEASE_EVIDENCE_SIGNING_KEY_ID)) { + throw 'RELEASE_EVIDENCE_SIGNING_KEY_ID variable is required for strict evidence validation.' + } + - name: Download release evidence artifact + uses: actions/download-artifact@d3f86a106a0bac45b974a628896c90dbdf5c8093 # v4.3.0 + with: + name: ${{ inputs.release_evidence_artifact_name }} + path: .release-evidence-input + repository: ${{ github.repository }} + run-id: ${{ inputs.release_evidence_artifact_run_id }} + github-token: ${{ github.token }} + - uses: actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065 # v5.6.0 + with: + python-version: "3.12" + cache: pip + - name: Install locked backend dependencies + run: python -m pip install --require-hashes -r backend/requirements.lock.txt + - name: Materialize out-of-band trusted evidence public key + shell: pwsh + env: + RELEASE_EVIDENCE_ED25519_PUBLIC_KEY_B64: ${{ secrets.RELEASE_EVIDENCE_ED25519_PUBLIC_KEY_B64 }} + run: | + $trustedKeyPath = Join-Path $env:RUNNER_TEMP 'release-evidence-ed25519-public.pem' + try { + $keyBytes = [Convert]::FromBase64String($env:RELEASE_EVIDENCE_ED25519_PUBLIC_KEY_B64) + } catch { + throw 'RELEASE_EVIDENCE_ED25519_PUBLIC_KEY_B64 must be valid base64-encoded PEM bytes.' + } + if ($keyBytes.Length -eq 0) { + throw 'RELEASE_EVIDENCE_ED25519_PUBLIC_KEY_B64 decoded to an empty key.' + } + [IO.File]::WriteAllBytes($trustedKeyPath, $keyBytes) + "RELEASE_EVIDENCE_TRUSTED_KEY_PATH=$trustedKeyPath" | Out-File -FilePath $env:GITHUB_ENV -Encoding utf8 -Append + - name: Validate exact-SHA evidence index and safely restore allowlisted files + shell: pwsh + env: + RELEASE_EVIDENCE_SIGNING_KEY_ID: ${{ vars.RELEASE_EVIDENCE_SIGNING_KEY_ID }} + run: | + $trustedKeyPath = $env:RELEASE_EVIDENCE_TRUSTED_KEY_PATH + New-Item -ItemType Directory -Force -Path data/release-trust | Out-Null + python scripts/validate_release_evidence_index.py ` + --index .release-evidence-input/release-evidence-index.json ` + --signature .release-evidence-input/release-evidence-index.ed25519 ` + --artifact-root .release-evidence-input ` + --release-sha $env:GITHUB_SHA ` + --trusted-public-key $trustedKeyPath ` + --trusted-signing-key-id $env:RELEASE_EVIDENCE_SIGNING_KEY_ID ` + --restore-root $env:GITHUB_WORKSPACE ` + --json | + Tee-Object -FilePath data/release-trust/evidence-index-validation.json + $exitCode = $LASTEXITCODE + if ($exitCode -ne 0) { exit $exitCode } + - uses: actions/setup-node@49933ea5288caeca8642d1e84afbd3f7d6820020 # v4.4.0 + with: + node-version: "24" + - name: Strict Windows Excel smoke artifact validation + shell: pwsh + run: | + New-Item -ItemType Directory -Force -Path data/release-trust | Out-Null + python scripts/validate_windows_excel_smoke.py --require-existing-artifacts --json | + Tee-Object -FilePath data/release-trust/windows-excel-smoke-validation.json + $exitCode = $LASTEXITCODE + if ($exitCode -ne 0) { exit $exitCode } + - name: Strict release evidence prerequisites + shell: pwsh + env: + RELEASE_EVIDENCE_SIGNING_KEY_ID: ${{ vars.RELEASE_EVIDENCE_SIGNING_KEY_ID }} + run: | + $trustedKeyPath = $env:RELEASE_EVIDENCE_TRUSTED_KEY_PATH + python scripts/verify_release_evidence_prereqs.py ` + --strict ` + --json ` + --evidence-index .release-evidence-input/release-evidence-index.json ` + --evidence-signature .release-evidence-input/release-evidence-index.ed25519 ` + --evidence-root .release-evidence-input ` + --release-sha $env:GITHUB_SHA ` + --trusted-public-key $trustedKeyPath ` + --trusted-signing-key-id $env:RELEASE_EVIDENCE_SIGNING_KEY_ID ` + --smoke-artifact-validation-json data/release-trust/windows-excel-smoke-validation.json | + Tee-Object -FilePath data/release-trust/release-evidence-prereqs.json + $exitCode = $LASTEXITCODE + if ($exitCode -ne 0) { exit $exitCode } + - name: Strict V7 readiness evidence + shell: pwsh + run: | + python -m backend.benchmarks.v7_readiness --fail-on-warning --require-production-replay --require-private-fixtures | + Tee-Object -FilePath data/release-trust/v7-readiness.json + $exitCode = $LASTEXITCODE + if ($exitCode -ne 0) { exit $exitCode } + - name: Strict release trust bundle validation + shell: pwsh + run: | + python scripts/generate_release_trust_bundle.py ` + --mode metadata-only ` + --label ci-release-evidence ` + --fail-on-warning ` + --require-private-fixtures ` + --smoke-artifact-validation-json data/release-trust/windows-excel-smoke-validation.json ` + --v7-readiness-json data/release-trust/v7-readiness.json + - name: Upload strict release trust artifacts + if: always() + uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4.6.2 + with: + name: release-trust-strict-${{ github.run_id }} + path: data/release-trust/** diff --git a/.github/workflows/vnext-contracts.yml b/.github/workflows/vnext-contracts.yml new file mode 100644 index 0000000..220b25a --- /dev/null +++ b/.github/workflows/vnext-contracts.yml @@ -0,0 +1,89 @@ +name: vNext Protocol Contracts + +# This workflow runs only on the operator's ephemeral self-hosted Linux pool. + +on: + push: + branches: [main] + paths: + - "vnext/**" + - "toolchain.lock.json" + - "scripts/verify_toolchain_manifest.py" + - "scripts/rust_toolchain.py" + - "scripts/verify-local.sh" + - "infra/ci/rust-tools.Dockerfile" + - ".github/workflows/vnext-contracts.yml" + pull_request: + paths: + - "vnext/**" + - "toolchain.lock.json" + - "scripts/verify_toolchain_manifest.py" + - "scripts/rust_toolchain.py" + - "scripts/verify-local.sh" + - "infra/ci/rust-tools.Dockerfile" + - ".github/workflows/vnext-contracts.yml" + workflow_dispatch: + +permissions: + contents: read + +concurrency: + group: vnext-contracts-${{ github.ref }} + cancel-in-progress: true + +jobs: + generated-contracts: + if: github.event_name != 'pull_request' || github.event.pull_request.head.repo.full_name == github.repository + runs-on: [self-hosted, linux, x64, excel-agent] + timeout-minutes: 15 + steps: + - uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4.4.0 + with: + fetch-depth: 0 + + - uses: actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065 # v5.6.0 + with: + python-version: "3.12" + cache: pip + + - name: Enforce clean-break dependency boundary + run: python vnext/scripts/check_clean_break_boundary.py + + - name: Verify Protobuf backward compatibility + env: + PR_BASE_SHA: ${{ github.event.pull_request.base.sha }} + PUSH_BEFORE_SHA: ${{ github.event.before }} + run: | + case "$GITHUB_EVENT_NAME" in + pull_request) BASE_SHA="$PR_BASE_SHA" ;; + push) BASE_SHA="$PUSH_BEFORE_SHA" ;; + workflow_dispatch) BASE_SHA="$(git rev-parse HEAD^)" ;; + *) echo "Unsupported GitHub event: $GITHUB_EVENT_NAME" >&2; exit 2 ;; + esac + ./vnext/scripts/verify-breaking.sh "$BASE_SHA" + + - name: Verify frozen vNext protocol baseline + run: ./vnext/scripts/verify-released-baseline.sh + + - name: Verify pinned Protobuf generation + run: ./vnext/scripts/verify-generated.sh + + - uses: actions/setup-node@49933ea5288caeca8642d1e84afbd3f7d6820020 # v4.4.0 + with: + node-version: "24" + cache: npm + cache-dependency-path: vnext/package-lock.json + - name: Verify TypeScript bindings + working-directory: vnext + run: | + npm ci + npm run verify + npm audit --audit-level=high + + - name: Verify Python bindings + run: | + python -m pip install ./vnext pytest==9.1.1 + python -m pytest -q vnext/test/python + + - name: Verify Rust bindings + run: python scripts/rust_toolchain.py verify-vnext --online --platform linux/amd64 diff --git a/.gitignore b/.gitignore index 2ab7cc8..89a35d7 100644 --- a/.gitignore +++ b/.gitignore @@ -3,12 +3,22 @@ .env.* !.env.example /data/ -backend/benchmarks/fixtures/private/ +backend/benchmarks/fixtures/private/* +!backend/benchmarks/fixtures/private/ +!backend/benchmarks/fixtures/private/README.md +!backend/benchmarks/fixtures/private/manifest.example.json +backend/benchmarks/fixtures/private-collaboration/* +!backend/benchmarks/fixtures/private-collaboration/ +!backend/benchmarks/fixtures/private-collaboration/README.md +!backend/benchmarks/fixtures/private-collaboration/manifest.example.json +!backend/benchmarks/fixtures/private-collaboration/manifest.schema.json +!backend/benchmarks/fixtures/private-collaboration/schedule.schema.json *.db *.sqlite *.sqlite3 *.db-shm *.db-wal +backend/storage/workflows/.locks/ # Python __pycache__/ @@ -20,6 +30,7 @@ __pycache__/ htmlcov/ .venv/ venv/ +*.egg-info/ # Node / frontend node_modules/ @@ -28,15 +39,21 @@ yarn-debug.log* yarn-error.log* pnpm-debug.log* plugin/dist/ +plugin/test-results/ workbook-worker/dist/ # Build and cache output dist/ build/ +target/ .cache/ .vite/ .turbo/ # Local Office/dev artifacts .office-addin-dev-certs/ +infra/nginx/certs/* +!infra/nginx/certs/README.md +infra/mtls/* +!infra/mtls/README.md *.log diff --git a/AGENTS.md b/AGENTS.md new file mode 100644 index 0000000..6a04e55 --- /dev/null +++ b/AGENTS.md @@ -0,0 +1,43 @@ + +# GitNexus — Code Intelligence + +This project is indexed by GitNexus as **excelaiplugin** (14992 symbols, 35436 relationships, 300 execution flows). Use the GitNexus MCP tools to understand code, assess impact, and navigate safely. + +> If any GitNexus tool warns the index is stale, run `npx gitnexus analyze` in terminal first. + +## Always Do + +- **MUST run impact analysis before editing any symbol.** Before modifying a function, class, or method, run `gitnexus_impact({target: "symbolName", direction: "upstream"})` and report the blast radius (direct callers, affected processes, risk level) to the user. +- **MUST run `gitnexus_detect_changes()` before committing** to verify your changes only affect expected symbols and execution flows. +- **MUST warn the user** if impact analysis returns HIGH or CRITICAL risk before proceeding with edits. +- When exploring unfamiliar code, use `gitnexus_query({query: "concept"})` to find execution flows instead of grepping. It returns process-grouped results ranked by relevance. +- When you need full context on a specific symbol — callers, callees, which execution flows it participates in — use `gitnexus_context({name: "symbolName"})`. + +## Never Do + +- NEVER edit a function, class, or method without first running `gitnexus_impact` on it. +- NEVER ignore HIGH or CRITICAL risk warnings from impact analysis. +- NEVER rename symbols with find-and-replace — use `gitnexus_rename` which understands the call graph. +- NEVER commit changes without running `gitnexus_detect_changes()` to check affected scope. + +## Resources + +| Resource | Use for | +|----------|---------| +| `gitnexus://repo/excelaiplugin/context` | Codebase overview, check index freshness | +| `gitnexus://repo/excelaiplugin/clusters` | All functional areas | +| `gitnexus://repo/excelaiplugin/processes` | All execution flows | +| `gitnexus://repo/excelaiplugin/process/{name}` | Step-by-step execution trace | + +## CLI + +| Task | Read this skill file | +|------|---------------------| +| Understand architecture / "How does X work?" | `.claude/skills/gitnexus/gitnexus-exploring/SKILL.md` | +| Blast radius / "What breaks if I change X?" | `.claude/skills/gitnexus/gitnexus-impact-analysis/SKILL.md` | +| Trace bugs / "Why is X failing?" | `.claude/skills/gitnexus/gitnexus-debugging/SKILL.md` | +| Rename / extract / split / refactor | `.claude/skills/gitnexus/gitnexus-refactoring/SKILL.md` | +| Tools, resources, schema reference | `.claude/skills/gitnexus/gitnexus-guide/SKILL.md` | +| Index, status, clean, wiki CLI commands | `.claude/skills/gitnexus/gitnexus-cli/SKILL.md` | + + diff --git a/CHANGELOG.md b/CHANGELOG.md new file mode 100644 index 0000000..69505a1 --- /dev/null +++ b/CHANGELOG.md @@ -0,0 +1,156 @@ +# Changelog + +All notable product changes are recorded here. This project uses semantic +versioning for product releases; Office manifests use the corresponding +four-part version. + +## [0.5.0] - Unreleased + +Supported private-beta release candidate. Repository implementation and local +verification are substantially complete; the exact-SHA external evidence in +the release checklist remains mandatory before a tag or release. + +### Excel and browser experience + +- Added a trust-first taskpane with explicit plan review, protected action + gating, typed error remedies, authentication recovery, accessible modal + focus management, command cancellation, first-run private-beta guidance, + preferences, contextual shortcuts, and a bounded local activity/recovery + ledger. +- Expanded the browser workbook workspace with sparse large-workbook viewport + reads, coalesced calculation, Office/coauthor event handling, target-scoped + preflight proofs, durable sync replay, host-drift reconciliation, progress + and retry state, and formula-result authority invalidation. +- Persisted workbook, sync queue, journal, recovery checkpoint, and bounded + undo/redo history in one IndexedDB compare-and-set transaction. Restored + history is bound to the exact workbook revision, fingerprint, and sync + generation; malformed, stale, oversized, or cross-workbook history fails + closed. +- Hardened normal and created-sheet rollback so Office mutation requires a + server-issued one-use capability and terminal acknowledgment requires exact + operation rows plus pre/post workbook, format, and host-snapshot proof. + +### Security + +- Replaced permissive production defaults with fail-closed identity, + configuration, parser, upload, lifecycle, and deletion controls. +- Disabled unscoped semantic caching, legacy RAG, and experimental capability + routes in production profiles. +- Added owner-aware lifecycle cleanup, deletion receipts, dependency policy, + a generated CycloneDX SBOM, and release trust evidence gates. +- Made cloud-provider requests opt out of provider-side request storage. +- Added process-local active-work cancellation and durable tombstone fences so + REST, WebSocket, diff, citation, retention, and migration writers cannot + resurrect a verified-deleted session. SQLite orphan reconciliation destroys + verified payloads child-first while preserving nonterminal cleanup + inventory. +- Added fail-closed SQLCipher runtime verification, signed rotation and restore + attestations, crash-resumable rekey receipts, strict file ownership/mode + checks, and an opt-in encrypted managed-object substrate with scoped + addressing, streaming AES-256-GCM, lifecycle purge hooks, recovery, and key + rotation. The managed-object substrate remains experimental and disabled by + default. +- Added bounded process admission, fair per-session scheduling, queue and + execution timeout taxonomy, graceful HTTP/WebSocket drain, subprocess-tree + cancellation, and privacy-safe low-cardinality metrics. + +### Correctness + +- Made planner, ReAct, validation, and tool execution fail closed on malformed + output, provider failures, timeouts, invalid operations, and incomplete runs. +- Added stable semantic idempotency keys and explicit Office acknowledgment for + agent-run rollback and undo recovery. +- Hardened workbook parsing and expanded spreadsheet operation, formula, + workbook identity, rollback, and synchronization checks. +- Added an append-only agent-run control plane with compare-and-set state + transitions, bounded event replay, idempotent mutation commands, one-use + digest-bound egress approval, explicit no-cloud-fallback receipts, terminal + fencing, and owner-scoped deletion. +- Added one-way v0.5 migration export, validation, staging, and lifecycle-owned + neutral intake with SQLite/PostgreSQL parity. It does not create a runtime + workbook, preserve a hidden compatibility layer, reverse-convert, or dual + write. +- Added an operator-only neutral-object materializer that copies a bounded + imported bundle into tenant/owner/session-scoped encrypted managed objects, + revalidates lifecycle and source state around every write, and publishes a + deterministic manifest last. It remains inert data rather than a domain + workbook, document, CRDT, or workflow import. + +### Experimental vNext clean break + +- Added frozen Protobuf v2 baselines and reproducible Rust, TypeScript, and + Python generation with breaking-change and cross-language wire fixtures. +- Added bounded native/WASM Rust cores for OPC package validation, sparse + atomic workbook transactions, stable sheet/row/column/cell identities, + deterministic snapshots, and restart-safe idempotency. +- Added an experimental XLSX semantic codec with exact no-edit passthrough, + bounded reviewed SpreadsheetML projection, preconditioned existing-cell + edits, opaque-part preservation proofs, signature/calc-chain fences, and a + restart-safe idempotency ledger. Export is authorized only for the exact + package and mutation proved by the codec; native Excel certification remains + an external gate. +- Added transport-neutral CRDT collaboration, fenced Excel bridge, and AI run + cores with actor-scoped undo, authority epochs, monotonic lease fencing, + read/hash/preflight/write/reread/acknowledge proofs, exact echo suppression, + append-only run events, and deterministic recovery. +- Added a lossless, bounded formula syntax/reference/dependency core and a tiny + reviewed evaluator. Authority receipts are exact-formula and exact-binding + scoped; Excel remains calculation authority in the supported build and no + Rust function family is promoted wholesale. +- Added experimental TypeScript and Python SDK adapters for every current vNext + Workbook, Run, Collaboration, and Formula RPC, with absolute deadlines, + cancellation, bounded safe/idempotent retries, request-replay integrity, + sanitized ambiguity, strict metadata, and one-shot streams. Authentication, + serialization, endpoints, and wire I/O remain caller-supplied transport + responsibilities. +- Added a native-only local service host that persists the five transport-neutral + domain cores through authenticated canonical snapshots and exact-generation + compare-and-set transactions. Its trust boundary is a private local POSIX + filesystem; it has no listener, provider, Office adapter, or production + deployment claim. +- Kept the vNext domain cores transport-neutral. Generated service contracts + and the local persistence host do not imply that a production network + service, provider, Office adapter, or collaboration deployment exists. + +### Operations and release evidence + +- Split liveness, readiness, and protected diagnostics; added route-template + metrics, request correlation, recursively redacted structured logs, and + privacy-safe operator reports. +- Added disposable SQLite backup/restore and live PostgreSQL 16 + migration/dump/restore rehearsals with exact manifests, logical/schema + digests, cleanup, and provider-free output. +- Added an entirely local/self-hosted verification plane with ephemeral Linux + and Windows+Excel runner contracts, exact toolchain checks, offline dependency + caches, clean-workspace enforcement, and no GitHub-hosted runner labels. +- Added deterministic signed promotion, offline distribution/install, + hash-chained release disposition, evidence portal, and native-host + certification tooling. Synthetic fixtures and local smoke runs cannot be + promoted as real host, private-corpus, production, signing, or legal + evidence. + +### Release engineering + +- Synchronized product version `0.5.0` and Office version `0.5.0.0` across both + manifests, packages, backend configuration, and CI. +- Added proprietary licensing, security reporting, ownership, support, + migration, third-party notice, dependency, and release-checklist documents. +- Added contributor bootstrap, architecture/API/operations documentation, + issue and pull-request templates, CODEOWNERS, expanded dependency update + coverage, deterministic package-integrity policy, generated SBOM license + review, and offline verification instructions. + +### Known release gates + +- `0.5.0` must not be tagged until every item in `docs/RELEASE_CHECKLIST.md` is + complete from the exact release SHA, including the host matrix, private + workbook and collaboration corpora, production-like replay, target + PostgreSQL/OIDC/mTLS topology, image scanning, signatures/provenance, + licensing/legal approval, security closure, and independent review. +- HyperFormula is physically omitted from the standard production dependency + graph. It may be distributed only in a separately approved build with the + required commercial license; otherwise Excel remains authoritative. + +## [0.4.0] - 2026-08-09 + +- Last pre-closure development baseline. This version is unsupported. diff --git a/CLAUDE.md b/CLAUDE.md new file mode 100644 index 0000000..6a04e55 --- /dev/null +++ b/CLAUDE.md @@ -0,0 +1,43 @@ + +# GitNexus — Code Intelligence + +This project is indexed by GitNexus as **excelaiplugin** (14992 symbols, 35436 relationships, 300 execution flows). Use the GitNexus MCP tools to understand code, assess impact, and navigate safely. + +> If any GitNexus tool warns the index is stale, run `npx gitnexus analyze` in terminal first. + +## Always Do + +- **MUST run impact analysis before editing any symbol.** Before modifying a function, class, or method, run `gitnexus_impact({target: "symbolName", direction: "upstream"})` and report the blast radius (direct callers, affected processes, risk level) to the user. +- **MUST run `gitnexus_detect_changes()` before committing** to verify your changes only affect expected symbols and execution flows. +- **MUST warn the user** if impact analysis returns HIGH or CRITICAL risk before proceeding with edits. +- When exploring unfamiliar code, use `gitnexus_query({query: "concept"})` to find execution flows instead of grepping. It returns process-grouped results ranked by relevance. +- When you need full context on a specific symbol — callers, callees, which execution flows it participates in — use `gitnexus_context({name: "symbolName"})`. + +## Never Do + +- NEVER edit a function, class, or method without first running `gitnexus_impact` on it. +- NEVER ignore HIGH or CRITICAL risk warnings from impact analysis. +- NEVER rename symbols with find-and-replace — use `gitnexus_rename` which understands the call graph. +- NEVER commit changes without running `gitnexus_detect_changes()` to check affected scope. + +## Resources + +| Resource | Use for | +|----------|---------| +| `gitnexus://repo/excelaiplugin/context` | Codebase overview, check index freshness | +| `gitnexus://repo/excelaiplugin/clusters` | All functional areas | +| `gitnexus://repo/excelaiplugin/processes` | All execution flows | +| `gitnexus://repo/excelaiplugin/process/{name}` | Step-by-step execution trace | + +## CLI + +| Task | Read this skill file | +|------|---------------------| +| Understand architecture / "How does X work?" | `.claude/skills/gitnexus/gitnexus-exploring/SKILL.md` | +| Blast radius / "What breaks if I change X?" | `.claude/skills/gitnexus/gitnexus-impact-analysis/SKILL.md` | +| Trace bugs / "Why is X failing?" | `.claude/skills/gitnexus/gitnexus-debugging/SKILL.md` | +| Rename / extract / split / refactor | `.claude/skills/gitnexus/gitnexus-refactoring/SKILL.md` | +| Tools, resources, schema reference | `.claude/skills/gitnexus/gitnexus-guide/SKILL.md` | +| Index, status, clean, wiki CLI commands | `.claude/skills/gitnexus/gitnexus-cli/SKILL.md` | + + diff --git a/CODE_OF_CONDUCT.md b/CODE_OF_CONDUCT.md new file mode 100644 index 0000000..0fe981a --- /dev/null +++ b/CODE_OF_CONDUCT.md @@ -0,0 +1,44 @@ +# Contributor Covenant Code of Conduct + +## Our pledge + +We pledge to make participation in this project a harassment-free experience +for everyone, regardless of age, body size, disability, ethnicity, sex +characteristics, gender identity and expression, level of experience, +education, socio-economic status, nationality, personal appearance, race, +religion, or sexual identity and orientation. + +We pledge to act and interact in ways that contribute to an open, welcoming, +diverse, inclusive, and healthy community. + +## Our standards + +Examples of behavior that contributes to a positive environment include: + +- demonstrating empathy and kindness; +- respecting differing opinions, viewpoints, and experiences; +- giving and gracefully accepting constructive feedback; +- accepting responsibility, apologizing, and learning from mistakes; +- focusing on what is best for the community and its users. + +Unacceptable behavior includes harassment, sexualized language or imagery, +personal or political attacks, insulting or derogatory comments, public or +private harassment, publishing private information without permission, and +other conduct that could reasonably be considered inappropriate. + +## Enforcement + +Project maintainers are responsible for clarifying and enforcing these +standards. They may remove, edit, or reject contributions and communication +that do not align with this code, and will communicate moderation reasons when +appropriate. + +Instances of abusive, harassing, or otherwise unacceptable behavior may be +reported using the private contact process in `SUPPORT.md`. Reports will be +reviewed promptly and handled with respect for the reporter's privacy and +security. + +## Attribution + +This code is adapted from the Contributor Covenant, version 2.1: +https://www.contributor-covenant.org/version/2/1/code_of_conduct.html diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md new file mode 100644 index 0000000..5dcff8d --- /dev/null +++ b/CONTRIBUTING.md @@ -0,0 +1,56 @@ +# Contributing to Excel Agent + +Excel Agent is a trust-first spreadsheet system. Correctness, reversibility, +tenant isolation, workbook fidelity, and evidence quality take priority over +feature breadth. + +## Before opening a change + +1. Read `SECURITY.md`, `docs/PROGRAM_COMPLETION_ROADMAP.md`, and the relevant + architecture decision records. +2. Keep legacy `0.5` and `vnext/` changes separate. vNext must not import + `/api/v1`, legacy SQLite models, V2–V7 implementation contracts, or the Node + workbook-worker CLI. +3. Trace affected symbols and execution flows with GitNexus before editing. +4. Add a failing regression or corpus case before changing semantic behavior. +5. Never commit workbook content, source documents, prompts, secrets, private + fixtures, release keys, runtime databases, or external evidence. + +## Verification + +Use the local/self-hosted command graph; the project does not depend on +GitHub-hosted runner minutes: + +```sh +scripts/verify-local.sh --profile fast +scripts/verify-local.sh --profile full +``` + +Run the narrowest relevant suite while iterating, then the full affected +profile. The strict `release` profile is expected to fail until genuine +exact-SHA external evidence exists. + +Generated OpenAPI types, Protobuf bindings, dependency locks, and the SBOM must +be regenerated by their checked-in tools and included in the same review. + +## Pull requests + +- Keep the title outcome-oriented. +- Explain the root cause and trust boundary, not only changed files. +- List user-visible behavior, compatibility, migration, privacy, licensing, + performance, and rollback effects. +- Include exact commands and results from a clean workspace. +- Separate deterministic/local evidence from native-host, private-corpus, or + production evidence. +- Do not describe scaffolding, mocks, or synthetic fixtures as production + readiness. +- Do not weaken fail-closed behavior to make a test or demo pass. + +Large changes require separate implementation, adversarial review, and evidence +owners. Reviewers may request a smaller semantic slice even when generated +artifacts make the textual diff large. + +## Security reports + +Do not open a public issue for a suspected vulnerability. Follow the private +reporting process in `SECURITY.md`. diff --git a/LICENSE b/LICENSE new file mode 100644 index 0000000..9b10260 --- /dev/null +++ b/LICENSE @@ -0,0 +1,127 @@ +Excel Agent Proprietary Software License Agreement +Version 1.0, effective August 9, 2026 + +Copyright (c) 2026 Charles Abderrahman. All rights reserved. + +This Proprietary Software License Agreement (the "Agreement") governs all use +of the Excel Agent software, source code, object code, documentation, models, +workbook assets, and related materials (collectively, the "Software"). By +accessing, installing, copying, or using the Software, you agree to this +Agreement. If you do not agree, do not use the Software. + +1. Limited license + +Subject to this Agreement and any written order, evaluation invitation, or +private-beta agreement issued by the copyright holder (an "Order"), the +copyright holder grants the named recipient a limited, non-exclusive, +non-transferable, non-sublicensable, revocable license during the applicable +term to install and use the Software solely for the recipient's internal +evaluation or internal business purposes. No license is granted to any person +who has not been expressly authorized in writing. + +2. Restrictions + +Except to the limited extent a restriction is prohibited by applicable law, +you must not: copy the Software other than for authorized backup; publish, +distribute, sell, lease, sublicense, host for third parties, or make the +Software available as a service; modify, translate, or create derivative works; +reverse engineer, decompile, disassemble, or attempt to discover source code or +non-public interfaces; remove proprietary notices; bypass access, licensing, +usage, security, or audit controls; use the Software to develop a competing +product; or permit an unauthorized person to access the Software. You may not +use the Software or its output in violation of law, third-party rights, or an +applicable Order. + +3. Ownership and reservation of rights + +The Software is licensed, not sold. The copyright holder and its licensors +retain all right, title, and interest in the Software, including all copyright, +patent, trade-secret, trademark, and other intellectual-property rights. No +rights are granted by implication, estoppel, or otherwise. All rights not +expressly granted are reserved. + +4. Confidentiality + +Non-public Software, documentation, benchmarks, security information, product +plans, and credentials are confidential information. You must protect them +using at least reasonable care, use them only for the authorized purpose, and +disclose them only to personnel who are bound by equivalent confidentiality +obligations and need access for that purpose. This section does not cover +information that you can document was lawfully public, already known without +restriction, independently developed, or rightfully received without a duty of +confidentiality. + +5. Third-party materials + +The Software may contain third-party materials governed by their own license +terms. Those terms apply to the corresponding materials and are identified in +the third-party notices and software bill of materials. This Agreement does not +grant rights that a third-party licensor has not authorized. You are responsible +for obtaining any separately required commercial license or service account. + +6. Data and security + +You are responsible for determining whether your data may be processed by the +Software, configuring the Software for your environment, protecting credentials +and encryption keys, reviewing proposed workbook changes, and maintaining +appropriate backups. Private-beta features may change and must not be treated +as a substitute for professional judgment or independent verification. + +7. Feedback + +If you voluntarily provide ideas, suggestions, or feedback about the Software, +you grant the copyright holder a perpetual, irrevocable, worldwide, +royalty-free right to use and incorporate that feedback without restriction or +payment, provided the feedback does not identify your confidential data. + +8. Term and termination + +This Agreement begins when you first access the Software and continues until +terminated. It terminates automatically if you breach it or when the applicable +Order expires. The copyright holder may terminate a private-beta license at any +time by written notice. On termination, you must stop using the Software and +delete all copies, except one archival copy may be retained solely where legally +required. Sections 2 through 12 survive termination where their nature requires. + +9. Disclaimer of warranties + +TO THE MAXIMUM EXTENT PERMITTED BY LAW, THE SOFTWARE IS PROVIDED "AS IS" AND +"AS AVAILABLE," WITH ALL FAULTS. THE COPYRIGHT HOLDER AND ITS LICENSORS DISCLAIM +ALL WARRANTIES, EXPRESS, IMPLIED, STATUTORY, OR OTHERWISE, INCLUDING +MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE, TITLE, NON-INFRINGEMENT, +ACCURACY, SECURITY, AVAILABILITY, AND RESULTS. THE SOFTWARE IS A PRIVATE BETA +AND MAY CONTAIN DEFECTS OR CAUSE LOSS OR CORRUPTION OF DATA. + +10. Limitation of liability + +TO THE MAXIMUM EXTENT PERMITTED BY LAW, THE COPYRIGHT HOLDER AND ITS LICENSORS +WILL NOT BE LIABLE FOR INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, PUNITIVE, OR +CONSEQUENTIAL DAMAGES, OR FOR LOST PROFITS, REVENUE, GOODWILL, DATA, OR BUSINESS +INTERRUPTION, ARISING FROM OR RELATED TO THE SOFTWARE, EVEN IF ADVISED OF THE +POSSIBILITY. THEIR TOTAL AGGREGATE LIABILITY WILL NOT EXCEED THE GREATER OF ONE +HUNDRED U.S. DOLLARS (US$100) OR THE AMOUNT YOU PAID FOR THE SOFTWARE DURING THE +TWELVE MONTHS BEFORE THE EVENT GIVING RISE TO THE CLAIM. Some jurisdictions do +not allow certain exclusions, so those exclusions apply only to the extent +permitted. + +11. Export and legal compliance + +You must comply with applicable export-control, sanctions, privacy, employment, +financial-services, and other laws. You represent that you are not prohibited +from receiving the Software under applicable trade restrictions. + +12. General + +This Agreement and any applicable Order are the entire agreement about the +Software and supersede prior discussions on that subject. An Order controls +over this Agreement only where it expressly identifies the conflicting term. +You may not assign this Agreement without prior written consent. If a provision +is unenforceable, it will be limited to the minimum extent necessary and the +remaining provisions will continue. A waiver must be in writing and is not a +continuing waiver. This Agreement is governed by the laws of the State of New +York, excluding conflict-of-law rules. The state and federal courts located in +New York County, New York have exclusive jurisdiction, and each party consents +to that jurisdiction and venue. + +For commercial licensing or other written authorization, contact the repository +owner through the private distribution channel that supplied the Software. diff --git a/MIGRATION.md b/MIGRATION.md new file mode 100644 index 0000000..993e25f --- /dev/null +++ b/MIGRATION.md @@ -0,0 +1,58 @@ +# Migration Guide: 0.4 to 0.5 + +Version `0.5.0` is a security and correctness closure release. It intentionally +turns several permissive or incomplete behaviors into hard failures. + +## Before upgrading + +1. Record the deployed revision and export an encrypted backup of the database, + managed document store, workbook artifacts, configuration, and signing keys. +2. Validate the backup restoration procedure in an isolated environment. +3. Inventory Office hosts, current manifests, provider credentials, local-model + profiles, worker dependencies, and HyperFormula licensing. +4. Review `SECURITY.md`, `SUPPORT.md`, `docs/THIRD_PARTY_NOTICES.md`, and the + release checklist. Do not deploy from `main`. + +## Required changes + +- Replace the Office manifest with the signed `0.5.0.0` manifest. Remove old + sideloaded copies before validating the new taskpane origin and permissions. +- Production profiles now require hardened authentication/authorization, + explicit HTTPS CORS origins, PostgreSQL, strong secrets, the constrained + workbook worker, explicit AI-egress policy, and validated runtime settings. +- Supply approved commercial HyperFormula licensing evidence and key material, + or disable HyperFormula and select a permitted formula-engine profile before + starting production. +- Legacy global RAG and experimental V7 routes are unavailable in production. + Do not depend on their previous endpoints or advertise them as capabilities. +- Cloud AI remains policy-controlled. OpenAI requests disable provider storage, + and no cloud fallback should be inferred from local-provider failure. + +## Behavior changes + +- Planner, ReAct, tool, and verifier failures no longer return a partially + successful executable plan. Partial work is a non-approvable draft requiring + fresh inspection. +- Agent-run rollback and undo enter `rollback_pending`. The client must send the + Office result to `/api/v1/agent-runs/{run_id}/rollback-result`; UI success is + valid only after the server verifies every expected operation result. +- Upload, parsing, workbook-worker, deletion, and retention paths fail closed + and may return new explicit errors instead of using unsafe fallbacks. +- Zero-hour retention is immediate. A terminal session may no longer be + retrievable after its deletion receipt is verified. + +## Database and rollback procedure + +Run the documented PostgreSQL migration and backfill rehearsal against a copy +of production data, then capture the evidence from the exact release SHA. +Application rollback across a completed schema/data migration is not automatic. +Restore the pre-upgrade application, database, and managed blobs together from +the validated backup; do not mix `0.4` code with a migrated `0.5` store. + +## vNext compatibility boundary + +The `0.5` API, V2–V7 models, SQLite schema, and worker CLI are closure-line +contracts only. They will not become permanent vNext compatibility layers. +The planned migration bridge is one-way: export supported `0.5` state and import +it into a new vNext workspace epoch. Do not treat that bridge as implemented +until its command, schema, tests, and signed migration evidence ship. diff --git a/README.md b/README.md index 04a0e87..e1ee9be 100644 --- a/README.md +++ b/README.md @@ -1,57 +1,156 @@ -# Finance AI Excel Plugin +# Excel Agent -An AI-powered Excel agent for investment banking and hedge fund workflows. Generates formula-driven, professionally formatted spreadsheet operations from natural language. +A local-first, review-before-apply Excel copilot for financial modeling and +workbook analysis. It turns natural-language requests into typed, auditable +operations while keeping Excel authoritative for live workbook mutation and +calculation in the standard build. + +## Release status + +The repository is a **supported private-beta release candidate**. The backend, +Excel taskpane, browser spreadsheet, workbook worker, production edge image, +dependency policy, and automated release gates are implemented and locally +verified. On-prem interactive access uses per-user OIDC authorization-code plus +PKCE; local desktop access uses loopback-only one-time pairing. It is not a GA +release until the commit-matched +Windows Excel smoke, executed private-workbook and 100-schedule collaboration +corpora, anonymized production replay, staging/production Postgres migration +and encryption archive, live identity/host-matrix evidence, formula-engine +licensing decision, and signed artifacts are complete. The +strict evidence gate is designed to remain red until those external artifacts +are supplied. + +See the [implementation status ledger](docs/IMPLEMENTATION_STATUS.md) for the +supported, experimental, designed, and externally blocked surfaces. In +particular, the [vNext clean-break line](vnext/README.md) contains generated +contracts, transport-neutral workbook/formula/run/collaboration/bridge cores, +a bounded OPC/XLSX codec, caller-transport SDKs, and a native local persistence +host. None of those experimental surfaces claims a deployed network service, +Office adapter, model provider, production-certified OOXML export, or Rust +calculation authority. ## Architecture ``` -User <-> Excel Add-in (Office.js + React + TypeScript) <-> FastAPI Backend <-> ReAct Planner/Tools <-> LLM (LM Studio/Ollama/Claude/GPT-4o) +User <-> Excel Add-in / Browser (React + TypeScript) <-> mTLS Edge <-> FastAPI policy/orchestration <-> typed tools <-> approved local/cloud model profile ``` -## Quick Start +The detailed authority, deletion, and verification flows are diagrammed in +[docs/ARCHITECTURE.md](docs/ARCHITECTURE.md). The +[documentation map](docs/README.md) routes contributors and operators without +using historical roadmaps as current status. -### 1. Backend (Python) +### Browser workbook preview -```bash -cd backend -pip install -r requirements.txt -cp ../.env.example ../.env # Edit with your API keys -python -m uvicorn backend.main:app --host 0.0.0.0 --port 8000 --reload -``` +![Synthetic browser workbook preview showing the review-before-apply workspace](docs/assets/browser-workbook-preview.png) -### 2. Ollama (Local LLM — Optional) +This screenshot is rendered locally from the browser workbook workspace with +synthetic values. It demonstrates the grid, formula bar, explicit AI preview, +sync status, and recovery controls; it is not native Excel host, private +workbook, production, or release-certification evidence. -```bash -# Install: https://ollama.com/download -ollama pull llama3.1:8b -ollama serve +## No-cloud local quick start + +On a host matching `toolchain.lock.json`, the fresh-contributor path targets a +useful result in under 15 minutes: + +```sh docs-check=syntax +./scripts/bootstrap-contributor.sh --profile fast +python3 -m venv .venv +. .venv/bin/activate +python -m pip install --require-hashes -r backend/requirements.lock.txt +EXCEL_AGENT_PYTHON=.venv/bin/python ./examples/no-cloud-local-quickstart.sh +./scripts/verify-local.sh --profile fast ``` -### 3. Excel Plugin +The example uses disposable SQLite state, loopback networking, and a supported +read-only API contract. It does not require a cloud model or retain data. See +[getting started](docs/GETTING_STARTED.md) for prerequisites and timing, and +the [generated OpenAPI cookbook](docs/API_COOKBOOK.md) for current protocol +discovery without copied schemas. -```bash -cd plugin -npm install -npm run dev # Starts HTTPS server on port 3443 -npm run generate:api-types -npm run verify # Typecheck + production build -npm run sideload # Sideloads into Excel +For the taskpane after the API path is green: + +```sh docs-check=syntax +npm --prefix plugin ci +npm --prefix plugin run verify +npm --prefix plugin run dev ``` -### 4. Verify Everything +Sideloading and release evidence require the real Excel host procedures; local +mock tests are not a substitute for the +[Windows Excel matrix](docs/WINDOWS_EXCEL_MANUAL_SMOKE.md). + +## Production add-in distribution + +`docker-compose.prod.yml` builds a single edge image that compiles the React +taskpane and browser spreadsheet, serves the icons and a generated production +manifest, and proxies same-origin `/api/` and `/ws/` traffic to the backend. +The edge forwards each user's short-lived bearer token and authenticates its +own backend connection with mTLS. No shared administrator credential is +injected into client requests or emitted in JavaScript. + +Before starting the stack: -```powershell -.\scripts\verify.ps1 -.\scripts\verify-full-stack.ps1 # slower V7 default-readiness gate +1. Put the public TLS certificate and key at + `infra/nginx/certs/server.crt` and `infra/nginx/certs/server.key`. +2. Provision the CA, edge-client, and backend certificate/key files described + under `infra/mtls/`; keep every private key outside source control. +3. Set `DEPLOY_ORIGIN` to the public origin only, such as + `https://excel.example.com` (no path or trailing slash). +4. Configure the OIDC issuer, audience, public PKCE client ID, JWKS/discovery + endpoints, explicit tenant allowlist, and same-origin callback. Set + `OIDC_BROWSER_CONNECT_ORIGINS` to only the exact HTTPS origins of the + discovery, token, and JWKS endpoints (for example, + `https://login.microsoftonline.com`); wildcards and broad `https:` are + rejected. Do not use a browser client secret. +5. Keep the reserved backend `API_KEY` offline for bootstrap/break-glass use; + it is not an edge or interactive-user credential. +6. Set the required Postgres, pinned Ollama image, and explicit AI/document + egress variables. The standard proprietary `0.5.0` artifacts physically + omit the HyperFormula runtime and keep `HYPERFORMULA_ENABLED=false`; Excel is + the calculation authority. A separately licensed custom build must include + the dependency and set `HYPERFORMULA_DISTRIBUTED=true`, approval, and a + commercial key. Production startup fails closed on inconsistent settings. + +Then build and start the deployment: + +```bash +docker compose -f docker-compose.prod.yml build nginx +docker compose -f docker-compose.prod.yml up -d ``` -### 5. Docker (Full Stack) +The deployable catalog manifest is available at +`$DEPLOY_ORIGIN/manifest.xml`; the browser spreadsheet is at +`$DEPLOY_ORIGIN/spreadsheet`. Run the external smoke check after DNS and TLS are +live: ```bash -cp .env.example .env # Edit with your API keys -docker-compose up -d +DEPLOY_ORIGIN=https://excel.example.com \ +SMOKE_OIDC_BEARER_TOKEN="$SHORT_LIVED_TEST_TOKEN" \ +SMOKE_OIDC_CONNECT_ORIGINS="$OIDC_BROWSER_CONNECT_ORIGINS" \ +sh infra/nginx/smoke-production.sh ``` +The smoke check verifies the production manifest and both UI surfaces, confirms +the CSP permits Office framing without `X-Frame-Options`, checks the backend +health route, proves unauthenticated requests remain unauthorized, and calls an +authenticated API route as the supplied test user. `SMOKE_TLS_INSECURE=1` is available only for a staging deployment +with a temporary self-signed certificate. + +The edge mTLS certificate is only a least-privileged transport identity. Each +HTTP/WebSocket request retains the user's validated bearer identity; the edge +never substitutes an administrator or shared API key. + +See [docs/PRODUCTION_RUNBOOK.md](docs/PRODUCTION_RUNBOOK.md) for first-admin +bootstrap, migration/backfill validation, monitoring, credential rotation, +rollback, and strict release-evidence procedures. + +Air-gapped operators should follow the authenticated +[offline distribution guide](docs/OFFLINE_DISTRIBUTION.md). That bundle stages +the Office manifest and loads exact container archives; it is not an MSI, +macOS pkg, desktop updater, or claim of platform signing/notarization. + ## Project Structure ``` @@ -72,6 +171,8 @@ excel-ai-plugin/ ├── plugin/ # Office.js Excel Plugin │ ├── manifest.xml # Add-in manifest │ └── src/taskpane/ # React trust workspace, API client, and Excel adapter +├── vnext/ # Experimental clean-break contracts, Rust cores, SDKs, and local host +├── infra/nginx/ # Production add-in build, manifest, edge proxy, and smoke check └── docker-compose.yml # Full stack orchestration ``` @@ -96,29 +197,39 @@ excel-ai-plugin/ - `GET /api/v1/sessions/{id}/citations` returns source/document/formula citations for the session. - `GET /api/v1/sessions/{id}/audit/export` exports audit events as CSV. - `POST /api/v1/sessions/{id}/rollback` returns restore operations for a saved snapshot. +- `POST /api/v1/sessions/{id}/rollback/result` records the Office.js restore + result; plans move through `rollback_pending` and are marked rolled back only + after a verified client acknowledgement. - Frontend API contracts are generated from FastAPI OpenAPI into `plugin/src/taskpane/api-types.ts`; do not edit that file by hand. ## Experimental Local Accuracy Workbench -- V7 stays experimental: defaults remain on `PLAN_ENGINE_VERSION=v2`, and live workbook mutation still goes only through Office.js after explicit approval. +- V7 stays experimental in product defaulting: opt-in readiness can report `ready`, but deterministic replay seed evidence is not production evidence and defaults remain on `PLAN_ENGINE_VERSION=v2` until a separate default-switch change is paired with anonymized production-like replay evidence and Windows Excel release evidence. Live workbook mutation still goes only through Office.js after explicit approval. - `POST /api/v1/experiments/v7/reviews` builds a read-only `V7WorkbookReview` combining truth model V5, formula graph V5, simulation V6, source evidence V3, registry coverage, confidence gaps, checkpoints, repair candidates, and next actions. - `GET /api/v1/experiments/v7/reviews/{review_id}` reloads the durable SQLite artifact; `/export` returns a JSON audit artifact. - `GET /api/v1/experiments/v7/readiness`, `POST /api/v1/experiments/v7/readiness/run`, and `GET /api/v1/experiments/v7/readiness/history` expose the persisted `V7DefaultReadinessReport` and compact history used to decide whether V7 is blocked, warning, or ready for a later default switch. -- `GET /api/v1/experiments/v7/replays`, `POST /api/v1/experiments/v7/replays/evaluate`, `POST /api/v1/experiments/v7/replays/capture`, `GET /api/v1/experiments/v7/replays/baselines`, and `POST /api/v1/experiments/v7/replays/baselines` expose anonymized local replay artifacts, deterministic drift evaluation, privacy-checked local capture, and pinned local replay baselines. Capture stores schema versions, operation types, counts, gate statuses, checkpoint counts, rollback availability, and stable hashes; payloads containing workbook values, formulas, raw prompts, sheet text, or source content are rejected. +- `GET /api/v1/experiments/v7/replays`, `POST /api/v1/experiments/v7/replays/evaluate`, `POST /api/v1/experiments/v7/replays/capture`, `GET /api/v1/experiments/v7/replays/baselines`, and `POST /api/v1/experiments/v7/replays/baselines` expose anonymized local replay artifacts, deterministic drift evaluation, privacy-checked local capture, and pinned local replay baselines. Capture stores schema versions, operation types, counts, gate statuses, checkpoint counts, rollback availability, and stable hashes; payloads containing workbook values, formulas, raw prompts, sheet text, or source content are rejected. `python -m backend.benchmarks.local_replay_seed` creates deterministic non-production plumbing evidence; `backend/benchmarks/fixtures/replays/synthetic_aggregate_replay_corpus.json` is a non-production importer contract fixture; `python -m backend.benchmarks.local_replay_corpus --production-evidence --evidence-class ` imports a fail-closed aggregate/anonymized corpus when real production-like evidence exists. - `GET /api/v1/experiments/v7/golden-cases` and `POST /api/v1/experiments/v7/golden-cases/evaluate` expose the workbook fixture gate as golden-case evidence for truth, formula, simulation, rollback, and repair expectations. - The taskpane Settings drawer includes a persisted local `V7 local accuracy mode` toggle and replay-capture controls. When the toggle is off, planning stays on the shipped V2 path; when on, V7 review, readiness, replay, simulation, and agent-run calls use the experimental surfaces while writes remain checkpointed Office.js handoffs. - The taskpane command palette includes a read-only V7 workbench across Understand, Trace, Simulate, Repair, and Audit plus readiness history, replay baselines, golden cases, retry/resume, rollback, and audit commands without switching the default planner. - The V7 registry now marks screenshot and sparkline operations as low-risk opt-in coverage with host-capability fallbacks; pivot mutations remain safely gated with explicit readiness reasons. - Local verification entrypoints: - `.\scripts\verify-full-stack.ps1` + - `.\scripts\verify-full-stack.ps1 -ReleaseEvidence` - `python -m backend.benchmarks.local_accuracy` - `python -m backend.benchmarks.local_replay --evaluate` + - `python -m backend.benchmarks.local_replay_seed` + - `python -m backend.benchmarks.local_replay_corpus ` - `python -m backend.benchmarks.v7_readiness` - `python -m backend.benchmarks.v7_golden_cases` + - `python scripts/check_package_integrity.py` + - `python scripts/run_parser_comparison_corpus.py --fail-on-mismatch` - `.\scripts\verify-local-accuracy.ps1` - `.\scripts\verify-v7-default-prep.ps1` - `cd plugin && npm run lint` - `cd plugin && npm run test:office-mock` + - `cd plugin && npm run test:frontend-smoke` + - `cd plugin && npm run test:spreadsheet-browser-smoke` - `cd plugin && npm run test:taskpane` ## Local Data Hygiene @@ -126,7 +237,7 @@ excel-ai-plugin/ - Runtime state lives under `data/` and is ignored by git. - To back up local sessions, stop the backend and copy `data/sessions.db` plus any matching `sessions.db-wal`/`sessions.db-shm` files to a secure backup location. - Avoid committing `.env`, cache folders, generated frontend `dist/`, or local database exports. -- Private local golden workbooks can live under `backend/benchmarks/fixtures/private/`; that folder is ignored so power-user regression cases stay local. +- Private local golden workbooks can live under `backend/benchmarks/fixtures/private/`; real manifests and workbooks are ignored so power-user regression cases stay local. Use `python -m backend.benchmarks.private_golden_fixtures` to validate the private manifest without echoing workbook names, formulas, prompts, or source values. ## Shortcut-Parity Surfaces @@ -137,15 +248,46 @@ excel-ai-plugin/ ## Verification -- `python -m pytest backend\tests -q` +The repository does not depend on GitHub-hosted runner minutes. Use the +cross-platform local/self-hosted entrypoint below; see +[Self-hosted verification](docs/SELF_HOSTED_VERIFICATION.md) for runner labels, +isolation requirements, and evidence retention. + +- `scripts/verify-local.sh --profile fast` +- `scripts/verify-local.sh --profile full` +- `scripts/verify-local.sh --profile vnext` (pinned Rust tools plus generated, + TypeScript, Python, SDK-loopback, native, and explicit WASM-package gates) +- `scripts/verify-local.sh --profile release` (strict; expected to remain red + until genuine external evidence exists) +- `PYTHON_EXECUTABLE=.venv/bin/python ./vnext/scripts/verify-sdk-examples.sh` + (sealed in-process transports only; reports `network=false`) + +- `python -m pytest backend/tests -q` - `python -m backend.benchmarks.local_accuracy` - `python -m backend.benchmarks.local_replay --evaluate` +- `python -m backend.benchmarks.local_replay_seed` +- `python -m backend.benchmarks.local_replay_corpus ` - `python -m backend.benchmarks.v7_readiness` - `python -m backend.benchmarks.v7_golden_cases` +- `python scripts/check_package_integrity.py` +- `python scripts/run_parser_comparison_corpus.py --fail-on-mismatch` - `python -c "import asyncio, json; from backend.benchmarks import get_benchmark_runner, deterministic_benchmark_pipeline, evaluate_gates; run=asyncio.run(get_benchmark_runner().run_suite(pipeline_fn=deterministic_benchmark_pipeline)); gate=evaluate_gates(run); print(json.dumps({'run_id': run.run_id, 'score': run.overall_score, 'gate': gate['status']}, indent=2)); raise SystemExit(0 if gate['status']=='passed' else 1)"` -- `cd plugin && npm run verify && npm run lint && npm run test:office-mock && npm run test:taskpane && npm audit --audit-level=critical` +- `cd plugin && npm run verify && npm run test:spreadsheet-browser-smoke` - `cd workbook-worker && npm test` - `python -c "from backend.main import app; print(app.title)"` - `.\scripts\verify.ps1` - `.\scripts\verify-v7-default-prep.ps1` - `.\scripts\verify-full-stack.ps1` +- `.\scripts\verify-full-stack.ps1 -ReleaseEvidence` + +## Contributing, security, and license + +Read [CONTRIBUTING.md](CONTRIBUTING.md) before opening a change and report +security issues through the private process in [SECURITY.md](SECURITY.md), not +through a public issue. Community behavior is governed by +[CODE_OF_CONDUCT.md](CODE_OF_CONDUCT.md). + +Excel Agent is proprietary software distributed only to explicitly authorized +recipients. See [LICENSE](LICENSE) for the controlling terms and +[third-party notices](docs/THIRD_PARTY_NOTICES.md) for separately licensed +dependencies and retained reference materials. diff --git a/SECURITY.md b/SECURITY.md new file mode 100644 index 0000000..bf3e4cd --- /dev/null +++ b/SECURITY.md @@ -0,0 +1,64 @@ +# Security Policy + +Excel Agent 0.5 is a supported private beta. Security reports are welcome from +authorized users and good-faith researchers. + +## Supported versions + +| Version | Support | +|---|---| +| `0.5.x` / `release/0.5` | Security and critical correctness fixes | +| `< 0.5` | Unsupported; upgrade before reporting unless the issue also affects `0.5.x` | +| vNext on `main` | Development only; not a supported deployment | + +The `release/0.5` line is maintained for critical and security fixes through +twelve months after v1.0 general availability. + +## Reporting a vulnerability + +Use GitHub's private vulnerability-reporting flow for this repository: +`Security` → `Advisories` → `Report a vulnerability`. If that flow is not +available to your account, contact the repository owner through the private +distribution channel. Do not open a public issue or include workbook data, +credentials, tokens, private model output, customer identities, or exploit +details in a public channel. + +Include the affected revision and deployment profile, a concise impact +statement, reproducible steps or a minimal proof, relevant logs with sensitive +content removed, and any suggested mitigation. Encrypt particularly sensitive +material using a key agreed with the maintainer before sending it. + +## Response objectives + +These are private-beta service objectives, not contractual service levels. + +| Severity | Initial acknowledgment | Triage target | Remediation target | +|---|---:|---:|---:| +| Critical | 1 business day | 2 business days | 7 calendar days or immediate compensating control | +| High | 1 business day | 3 business days | 14 calendar days | +| Medium | 3 business days | 7 business days | 45 calendar days | +| Low | 5 business days | 14 business days | Next appropriate maintenance release | + +We will coordinate validation, remediation, release timing, and disclosure. +Please allow a reasonable remediation period before public disclosure. + +## Security boundaries in scope + +Reports may cover identity and authorization, API and WebSocket sessions, +workbook or document isolation, cloud-AI egress, parser/worker isolation, +formula and OOXML handling, deletion/retention, collaboration and bridge +leasing, dependency or build-chain integrity, signed distribution artifacts, +and taskpane/browser/desktop security controls. + +Out of scope are denial-of-service tests against shared infrastructure, social +engineering, physical attacks, attacks requiring access to another person's +unlocked device, automated scanning that materially disrupts service, and +third-party services not controlled by this project. A third-party issue that +creates a concrete Excel Agent impact remains in scope as an integration report. + +## Safe-harbor intent + +Good-faith research that follows this policy, avoids privacy violations and +service disruption, and gives us a reasonable opportunity to remediate will be +treated as authorized for purposes of this policy. This statement cannot bind +third parties or override applicable law. diff --git a/SUPPORT.md b/SUPPORT.md new file mode 100644 index 0000000..a1c9ff2 --- /dev/null +++ b/SUPPORT.md @@ -0,0 +1,46 @@ +# Support Policy + +## Release posture + +Excel Agent `0.5.x` is a supported private beta, not a generally available +product. Access, support contacts, and any commercial terms are supplied through +the private distribution channel. `main` may contain clean-break vNext work and +is not a supported deployment. + +## Supported release line + +- Deploy immutable artifacts built from signed `v0.5.x` tags or an explicitly + identified private-beta release SHA. +- Critical security and correctness fixes are backported to `release/0.5`. +- `release/0.5` support ends twelve months after v1.0 general availability. +- Versions before `0.5.0` receive no fixes. + +## Supported environments + +Only combinations listed as passed in the release's signed evidence bundle are +supported. The target matrix is Windows Excel Desktop, Excel for Mac, and Excel +on the web using the signed production manifest, plus the documented on-prem +backend profile. Local developer configurations and experimental V7 routes are +not production-supported. + +## Requesting support + +Use the private distribution channel and provide the release tag/SHA, Office +host and version, operating system, deployment profile, reproduction steps, +expected and actual behavior, sanitized logs, and relevant run/plan/attempt IDs. +Never send credentials, private workbook contents, raw prompts, or customer data +unless an approved secure transfer channel has been established. + +Security issues follow `SECURITY.md`, not the ordinary support path. + +## Severity objectives + +| Severity | Example | Acknowledgment objective | +|---|---|---:| +| S1 | Data loss, unauthorized access, unsafe mutation, service unavailable | 1 business day | +| S2 | Major workflow blocked with no reasonable workaround | 2 business days | +| S3 | Degraded workflow with a workaround | 5 business days | +| S4 | Question, documentation, or enhancement | 10 business days | + +These objectives are non-contractual for the private beta. Support may require +upgrading to the latest `0.5.x` patch before investigation continues. diff --git a/VERSION b/VERSION new file mode 100644 index 0000000..8f0916f --- /dev/null +++ b/VERSION @@ -0,0 +1 @@ +0.5.0 diff --git a/backend/Dockerfile b/backend/Dockerfile index a26933b..dff76fc 100644 --- a/backend/Dockerfile +++ b/backend/Dockerfile @@ -1,4 +1,4 @@ -FROM node:22-bookworm-slim AS worker-deps +FROM node:22.23.2-bookworm-slim@sha256:d649c27dae7ba0137b3cef5dd75baa422c08dc3d9e3fc0c23dfb172dc3cc6436 AS worker-deps WORKDIR /app/workbook-worker @@ -8,7 +8,7 @@ RUN npm ci --omit=dev COPY workbook-worker/ /app/workbook-worker/ -FROM python:3.12-slim +FROM python:3.12.13-slim-trixie@sha256:229a2c5bfa27522db7815ea81f9bed70af17ccb9de9fc7ad142b1877b5830d36 WORKDIR /app @@ -24,8 +24,8 @@ RUN apt-get update \ COPY --from=worker-deps /usr/local/bin/node /usr/local/bin/node RUN node --version | grep '^v22\.' -COPY backend/requirements.txt /app/backend/requirements.txt -RUN pip install --no-cache-dir -r backend/requirements.txt \ +COPY backend/requirements.txt backend/requirements.lock.txt /app/backend/ +RUN pip install --require-hashes --no-cache-dir -r backend/requirements.lock.txt \ && useradd --create-home --shell /usr/sbin/nologin appuser \ && mkdir -p /app/data \ && chown -R appuser:appuser /app diff --git a/backend/admin_bootstrap.py b/backend/admin_bootstrap.py new file mode 100644 index 0000000..1cf2a44 --- /dev/null +++ b/backend/admin_bootstrap.py @@ -0,0 +1,133 @@ +"""Offline bootstrap for the first production enterprise administrator. + +Run this command inside the backend deployment before accepting traffic. The +generated API key is written once to a caller-selected mode-0600 file and is +never persisted in plaintext by the application. +""" + +from __future__ import annotations + +import argparse +import json +import os +import stat +import time +from pathlib import Path +from typing import Any + +from backend.storage import get_session_store +from backend.storage.protocol import SessionStoreProtocol + + +def bootstrap_admin( + store: SessionStoreProtocol, + *, + user_id: str, + email: str, + name: str, + label: str, + rotate: bool = False, +) -> dict[str, Any]: + """Create the first admin and return its one-time plaintext credential.""" + existing = store.get_enterprise_user(user_id) + if existing and not rotate: + raise RuntimeError( + f"Enterprise user {user_id!r} already exists; pass --rotate to mint a replacement credential" + ) + + metadata = dict(existing.get("metadata") or {}) if existing else {} + metadata.update({"bootstrapped_offline": True, "bootstrapped_at": time.time()}) + store.upsert_enterprise_user( + user_id=user_id, + email=email, + name=name, + role="admin", + # Offline bootstrap creates an API-key principal, not an OIDC user. + # OIDC-active users must be provisioned with issuer+subject instead. + status="api_key_only", + mfa_enabled=bool(existing.get("mfa_enabled")) if existing else False, + metadata=metadata, + ) + existing_keys = store.list_enterprise_api_keys(user_id) if rotate else [] + credential = store.create_enterprise_api_key(user_id, label) + revoked_key_ids = [ + str(item.get("key_id")) + for item in existing_keys + if item.get("active") is not False + and item.get("key_id") + and store.revoke_enterprise_api_key(str(item["key_id"])) + ] + return { + "user_id": user_id, + "email": email, + "role": "admin", + "key_id": credential["key_id"], + "api_key": credential["api_key"], + "label": label, + "revoked_key_ids": revoked_key_ids, + } + + +def write_credentials_once(output_path: Path, payload: dict[str, Any]) -> None: + """Create a secret file atomically without ever overwriting one.""" + output_path.parent.mkdir(parents=True, exist_ok=True) + descriptor = os.open(output_path, os.O_WRONLY | os.O_CREAT | os.O_EXCL, 0o600) + try: + with os.fdopen(descriptor, "w", encoding="utf-8") as handle: + json.dump(payload, handle, indent=2, sort_keys=True) + handle.write("\n") + except Exception: + output_path.unlink(missing_ok=True) + raise + if stat.S_IMODE(output_path.stat().st_mode) != 0o600: + output_path.chmod(0o600) + + +def _validate_pepper() -> None: + pepper = os.getenv("API_KEY_HMAC_PEPPER", "").strip() + if len(pepper) < 32 or "change" in pepper.lower(): + raise RuntimeError("API_KEY_HMAC_PEPPER must be a non-placeholder secret of at least 32 characters") + + +def _parser() -> argparse.ArgumentParser: + parser = argparse.ArgumentParser(description="Bootstrap the first enterprise administrator offline") + parser.add_argument("--user-id", default="bootstrap-admin") + parser.add_argument("--email", required=True) + parser.add_argument("--name", required=True) + parser.add_argument("--label", default="initial production bootstrap") + parser.add_argument("--output", required=True, type=Path, help="New mode-0600 JSON credential file") + parser.add_argument("--rotate", action="store_true", help="Mint a new key for an existing administrator") + return parser + + +def main(argv: list[str] | None = None) -> int: + args = _parser().parse_args(argv) + _validate_pepper() + if args.output.exists(): + raise RuntimeError(f"Refusing to overwrite existing credential file: {args.output}") + payload = bootstrap_admin( + get_session_store(), + user_id=args.user_id, + email=args.email, + name=args.name, + label=args.label, + rotate=args.rotate, + ) + write_credentials_once(args.output, payload) + print( + json.dumps( + { + "status": "created", + "user_id": payload["user_id"], + "key_id": payload["key_id"], + "credential_file": str(args.output), + "revoked_key_count": len(payload["revoked_key_ids"]), + }, + sort_keys=True, + ) + ) + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/backend/agents/adaptive_prompts.py b/backend/agents/adaptive_prompts.py index 95294c4..25392f9 100644 --- a/backend/agents/adaptive_prompts.py +++ b/backend/agents/adaptive_prompts.py @@ -10,7 +10,6 @@ from __future__ import annotations import json -from pathlib import Path from typing import Optional import structlog diff --git a/backend/agents/classifier.py b/backend/agents/classifier.py index 8555609..a33c4c7 100644 --- a/backend/agents/classifier.py +++ b/backend/agents/classifier.py @@ -10,9 +10,7 @@ from __future__ import annotations import hashlib -import json -from pathlib import Path -from typing import Optional +from typing import Any, Optional import structlog @@ -93,18 +91,18 @@ class SemanticClassifier: def __init__(self, model_name: str = "all-MiniLM-L6-v2"): self.model_name = model_name - self._model = None - self._canonical_embeddings = None - self._canonical_labels = None + self._model: Any | None = None + self._canonical_embeddings: Any | None = None + self._canonical_labels: list[str] | None = None self._cache: dict[str, str] = {} - self._available = None + self._available: bool | None = None @property def available(self) -> bool: """Check if sentence-transformers is available.""" if self._available is None: try: - from sentence_transformers import SentenceTransformer + from sentence_transformers import SentenceTransformer # noqa: F401 - availability probe self._available = True except ImportError: self._available = False @@ -118,7 +116,6 @@ def _load_model(self): try: from sentence_transformers import SentenceTransformer - import numpy as np self._model = SentenceTransformer(self.model_name) @@ -162,14 +159,16 @@ def classify(self, prompt: str) -> str: # Get top-3 matches top_indices = np.argsort(similarities)[-3:][::-1] top_scores = similarities[top_indices] + if self._canonical_labels is None: + return self._fallback_classify(prompt) top_labels = [self._canonical_labels[i] for i in top_indices] # Weighted vote from top-3 matches - label_weights = {} - for label, score in zip(top_labels, top_scores): + label_weights: dict[str, float] = {} + for label, score in zip(top_labels, top_scores, strict=True): label_weights[label] = label_weights.get(label, 0) + float(score) - result = max(label_weights, key=label_weights.get) + result = max(label_weights, key=lambda label: label_weights[label]) # Log the decision top_match = CANONICAL_TASKS[top_indices[0]] diff --git a/backend/agents/pipeline.py b/backend/agents/pipeline.py index 3265aa6..9ab3bd0 100644 --- a/backend/agents/pipeline.py +++ b/backend/agents/pipeline.py @@ -6,11 +6,9 @@ from __future__ import annotations -import asyncio import inspect import json -from pathlib import Path -from typing import Optional, TypedDict, Annotated +from typing import NotRequired, Optional, TypedDict import structlog @@ -34,10 +32,8 @@ from backend.agents.verifiers.logic_agent import get_logic_agent from backend.agents.verifiers.syntax_agent import get_syntax_agent from backend.agents.verifiers.performance_agent import get_performance_agent -from backend.excel.formula_tracer import trace_formula, trace_to_ui from backend.excel.citations import get_citation_tracker from backend.agents.verifiers.qa_agent import get_qa_agent -from backend.excel.multi_sheet import get_multi_sheet_analyzer logger = structlog.get_logger() @@ -53,6 +49,7 @@ class AgentState(TypedDict): provider_override: Optional[str] model_override: Optional[str] mode: str # action | plan | ask + session_id: NotRequired[str] # Planning plan: Optional[list[dict]] @@ -64,6 +61,11 @@ class AgentState(TypedDict): # Verification programmatic_issues: list[dict] + advanced_issues: NotRequired[list[dict]] + syntax_issues: NotRequired[list[dict]] + logic_issues: NotRequired[list[dict]] + performance_issues: NotRequired[list[dict]] + qa_issues: NotRequired[list[dict]] llm_verification: Optional[dict] is_verified: bool retry_count: int @@ -89,7 +91,10 @@ def _extract_json(text: str) -> dict: # Direct parse try: - return json.loads(text) + parsed = json.loads(text) + if not isinstance(parsed, dict): + raise ValueError("Model response must be a JSON object") + return parsed except json.JSONDecodeError: pass @@ -97,7 +102,10 @@ def _extract_json(text: str) -> dict: m = re.search(r"```(?:json)?\s*\n?(.*?)\n?\s*```", text, re.DOTALL) if m: try: - return json.loads(m.group(1).strip()) + parsed = json.loads(m.group(1).strip()) + if not isinstance(parsed, dict): + raise ValueError("Model response must be a JSON object") + return parsed except json.JSONDecodeError: pass @@ -105,7 +113,10 @@ def _extract_json(text: str) -> dict: m = re.search(r"\{.*\}", text, re.DOTALL) if m: try: - return json.loads(m.group(0)) + parsed = json.loads(m.group(0)) + if not isinstance(parsed, dict): + raise ValueError("Model response must be a JSON object") + return parsed except json.JSONDecodeError: pass @@ -135,7 +146,7 @@ async def run_planner(state: AgentState) -> AgentState: # v2: Generate semantic fingerprint instead of raw dump context_str = "" if state.get("workbook_context"): - wb_ctx = state["workbook_context"] + wb_ctx = state["workbook_context"] or {} snapshot = wb_ctx.get("cell_snapshot", {}) if snapshot: fp = generate_fingerprint( @@ -156,7 +167,7 @@ async def run_planner(state: AgentState) -> AgentState: planner_prompt = tuner.tune_system_prompt( base_prompt, state["user_prompt"], - model_name=state.get("model_override", ""), + model_name=state.get("model_override") or "", ) messages = [ @@ -173,33 +184,42 @@ async def run_planner(state: AgentState) -> AgentState: complexity = classify_complexity(state["user_prompt"]) - response = await router.generate( - messages=messages, - complexity=complexity, - provider_override=state.get("provider_override"), - model_override=state.get("model_override"), - temperature=0.1, - json_mode=True, - ) + try: + response = await router.generate( + messages=messages, + complexity=complexity, + provider_override=state.get("provider_override"), + model_override=state.get("model_override"), + temperature=0.1, + json_mode=True, + ) + except Exception: + logger.exception("planner_provider_failed") + state["error"] = "Planner provider failed" + state["metadata"] = {"complexity": complexity.value, "planner_status": "failed"} + return state try: result = _extract_json(response.content) - state["plan"] = result.get("plan", []) + plan = result.get("plan") + if not isinstance(plan, list) or not plan or not all(isinstance(step, dict) for step in plan): + raise ValueError("planner response must contain a non-empty plan array") + state["plan"] = plan state["plan_summary"] = result.get("summary", "") state["metadata"] = { "complexity": complexity.value, "planner_provider": response.provider, "planner_model": response.model, "planner_tokens": response.usage, - "plan_steps": len(state["plan"]), + "plan_steps": len(plan), } - logger.info("planner_complete", steps=len(state["plan"]), summary=state["plan_summary"][:80]) + logger.info("planner_complete", steps=len(plan), summary=state["plan_summary"][:80]) except ValueError as e: logger.error("planner_json_failed", error=str(e)) - # Fallback: pass the prompt directly to executor without a plan state["plan"] = None state["plan_summary"] = "" state["metadata"] = {"complexity": complexity.value, "planner_error": str(e)} + state["error"] = "Planner returned invalid JSON" return state @@ -207,6 +227,9 @@ async def run_planner(state: AgentState) -> AgentState: async def run_executor(state: AgentState) -> AgentState: """Generate Excel cell operations from the plan (or directly from prompt).""" await _emit_progress(state, "executor", "Generating formulas and operations...") + if state.get("retry_count", 0) > 0: + state["error"] = None + state["is_verified"] = False context_str = "" if state.get("workbook_context"): @@ -235,7 +258,7 @@ async def run_executor(state: AgentState) -> AgentState: # If retrying, include error feedback if state.get("retry_count", 0) > 0 and state.get("operations"): issues = state.get("programmatic_issues", []) - llm_issues = state.get("llm_verification", {}).get("issues", []) + llm_issues = (state.get("llm_verification") or {}).get("issues", []) all_issues = issues + llm_issues user_content += f""" @@ -267,18 +290,27 @@ async def run_executor(state: AgentState) -> AgentState: if state.get("retry_count", 0) > 0: complexity = max(complexity, Complexity.MEDIUM, key=lambda c: list(Complexity).index(c)) - response = await router.generate( - messages=messages, - complexity=complexity, - provider_override=state.get("provider_override"), - model_override=state.get("model_override"), - temperature=0.1, - json_mode=True, - ) + try: + response = await router.generate( + messages=messages, + complexity=complexity, + provider_override=state.get("provider_override"), + model_override=state.get("model_override"), + temperature=0.1, + json_mode=True, + ) + except Exception: + logger.exception("executor_provider_failed") + state["error"] = "Executor provider failed" + state["operations"] = [] + return state try: result = _extract_json(response.content) - state["operations"] = result.get("operations", []) + operations = result.get("operations") + if not isinstance(operations, list) or not operations or not all(isinstance(item, dict) for item in operations): + raise ValueError("executor response must contain a non-empty operations array") + state["operations"] = operations state["execution_summary"] = result.get("summary", "") state["metadata"]["executor_provider"] = response.provider state["metadata"]["executor_model"] = response.model @@ -299,7 +331,8 @@ async def run_verifier(state: AgentState) -> AgentState: operations = state.get("operations", []) if not operations: - state["is_verified"] = True + state["is_verified"] = False + state["error"] = "No operations are available for verification" return state # ── Step 1: Programmatic validation (instant, free) ────────────── @@ -321,13 +354,16 @@ async def run_verifier(state: AgentState) -> AgentState: for i in advanced_issues ] # Block on errors, warn on warnings - errors = [i for i in advanced_issues if i.severity == "error"] - if errors: - logger.warning("advanced_validation_errors", count=len(errors)) + advanced_errors = [issue for issue in advanced_issues if issue.severity == "error"] + if advanced_errors: + logger.warning("advanced_validation_errors", count=len(advanced_errors)) state["is_verified"] = False return state except Exception as e: - logger.debug("advanced_validation_skipped", error=str(e)) + logger.error("advanced_validation_failed", error=str(e)) + state["is_verified"] = False + state["error"] = "Advanced validation failed" + return state # ── Step 1c: Auto-detect chart opportunities (v2) ──────────────── try: @@ -355,13 +391,16 @@ async def run_verifier(state: AgentState) -> AgentState: "description": i.description, "suggestion": i.suggestion} for i in syntax_issues ] - errors = [i for i in syntax_issues if i.severity == "error"] - if errors: - logger.warning("syntax_validation_errors", count=len(errors)) + syntax_errors = [issue for issue in syntax_issues if issue.severity == "error"] + if syntax_errors: + logger.warning("syntax_validation_errors", count=len(syntax_errors)) state["is_verified"] = False return state except Exception as e: - logger.debug("syntax_agent_skipped", error=str(e)) + logger.error("syntax_agent_failed", error=str(e)) + state["is_verified"] = False + state["error"] = "Syntax validation failed" + return state # ── Step 3: Logic Verification Agent (v3) ──────────────────────── try: @@ -373,13 +412,16 @@ async def run_verifier(state: AgentState) -> AgentState: "description": i.description, "suggestion": i.suggestion} for i in logic_issues ] - errors = [i for i in logic_issues if i.severity == "error"] - if errors: - logger.warning("logic_verification_errors", count=len(errors)) + logic_errors = [issue for issue in logic_issues if issue.severity == "error"] + if logic_errors: + logger.warning("logic_verification_errors", count=len(logic_errors)) state["is_verified"] = False return state except Exception as e: - logger.debug("logic_agent_skipped", error=str(e)) + logger.error("logic_agent_failed", error=str(e)) + state["is_verified"] = False + state["error"] = "Logic validation failed" + return state # ── Step 4: Performance Optimization Agent (v3) ────────────────── try: @@ -397,14 +439,17 @@ async def run_verifier(state: AgentState) -> AgentState: # ── Step 5: Citation Tracking (v3) ─────────────────────────────── try: - tracker = get_citation_tracker() doc_context = state.get("metadata", {}).get("document_context") - if doc_context: + session_id = state.get("session_id", "") + if doc_context and session_id: + tracker = get_citation_tracker(session_id) operations = tracker.annotate_operations(operations, doc_context) comment_ops = tracker.generate_cell_comments(operations) if comment_ops: state["operations"].extend(comment_ops) state["metadata"]["citations_generated"] = len(comment_ops) + elif doc_context: + logger.warning("citation_tracking_requires_session") except Exception as e: logger.debug("citation_tracking_skipped", error=str(e)) @@ -419,13 +464,16 @@ async def run_verifier(state: AgentState) -> AgentState: "suggestion": i.suggestion} for i in qa_issues ] - errors = [i for i in qa_issues if i.severity == "error"] - if errors: - logger.warning("qa_verification_errors", count=len(errors)) + qa_errors = [issue for issue in qa_issues if issue.severity == "error"] + if qa_errors: + logger.warning("qa_verification_errors", count=len(qa_errors)) state["is_verified"] = False return state except Exception as e: - logger.debug("qa_agent_skipped", error=str(e)) + logger.error("qa_agent_failed", error=str(e)) + state["is_verified"] = False + state["error"] = "Quality-assurance validation failed" + return state # ── Step 7: LLM verification (for medium/complex tasks) ───────── complexity = classify_complexity(state["user_prompt"]) @@ -444,12 +492,20 @@ async def run_verifier(state: AgentState) -> AgentState: ), ] - response = await router.generate( - messages=messages, - complexity=Complexity.SIMPLE, # Verification itself is simpler - temperature=0.0, - json_mode=True, - ) + try: + response = await router.generate( + messages=messages, + complexity=Complexity.SIMPLE, # Verification itself is simpler + provider_override=state.get("provider_override"), + model_override=state.get("model_override"), + temperature=0.0, + json_mode=True, + ) + except Exception: + logger.exception("verifier_provider_failed") + state["is_verified"] = False + state["error"] = "Verifier provider failed" + return state try: verification = _extract_json(response.content) @@ -457,12 +513,16 @@ async def run_verifier(state: AgentState) -> AgentState: state["metadata"]["verifier_provider"] = response.provider state["metadata"]["verifier_model"] = response.model - if verification.get("status") == "fail": + if verification.get("status") != "pass": logger.warning("llm_verification_failed", issues=verification.get("issues", [])) state["is_verified"] = False + state["error"] = "Verifier did not explicitly pass the operations" return state except ValueError: - logger.warning("verifier_json_parse_failed — treating as pass") + logger.warning("verifier_json_parse_failed") + state["is_verified"] = False + state["error"] = "Verifier returned invalid JSON" + return state state["is_verified"] = True logger.info("verification_passed") @@ -525,6 +585,7 @@ async def run_pipeline( "provider_override": provider_override, "model_override": model_override, "mode": mode, + "session_id": str(session_id or ""), "plan": None, "plan_summary": "", "operations": [], @@ -589,9 +650,15 @@ async def run_pipeline( ) logger.info("pipeline_retry", attempt=state["retry_count"]) else: - # Max retries exceeded — post-process what we have and return with warnings - state["final_operations"] = post_process_operations(state["operations"]) - state["final_summary"] = state["execution_summary"] + " (verification warnings remain)" + # Max retries exceeded — preserve the proposal only as a labelled, + # non-executable draft. It must never be returned as apply-ready. + state["final_operations"] = [] + state["final_summary"] = "Verification failed; partial operations require fresh inspection." + state["error"] = "Operation verification failed after maximum retries" + state["metadata"]["partial_draft"] = { + "operations": state["operations"], + "requires_fresh_inspection": True, + } logger.warning("pipeline_max_retries", retries=MAX_RETRIES) return _format_output(state) @@ -599,8 +666,9 @@ async def run_pipeline( def _format_output(state: AgentState) -> dict: """Format the pipeline state into the API response format.""" + executable_operations = state.get("final_operations", []) if state.get("is_verified") else [] return { - "operations": state.get("final_operations", state.get("operations", [])), + "operations": executable_operations, "summary": state.get("final_summary", state.get("execution_summary", "")), "plan": state.get("plan"), "metadata": { diff --git a/backend/agents/prompt_manager.py b/backend/agents/prompt_manager.py index d6d91bc..6c7f223 100644 --- a/backend/agents/prompt_manager.py +++ b/backend/agents/prompt_manager.py @@ -12,7 +12,7 @@ import time from pathlib import Path from typing import Optional -from dataclasses import dataclass, field +from dataclasses import dataclass import structlog diff --git a/backend/agents/react_engine.py b/backend/agents/react_engine.py index 23a3899..4f46d73 100644 --- a/backend/agents/react_engine.py +++ b/backend/agents/react_engine.py @@ -18,12 +18,14 @@ import time from dataclasses import dataclass, field from enum import Enum -from typing import AsyncIterator, Optional, Callable +from typing import Any, Awaitable, Callable, Optional import structlog logger = structlog.get_logger() +import os + # Single LLM call must finish within this budget; if it doesn't, the loop # records a REFLECT step and bails rather than hanging the trust workspace. LLM_STEP_TIMEOUT_S = 90.0 @@ -31,6 +33,11 @@ # the prompt grows linearly with iterations and runs out of context after a # handful of steps. MAX_CONVERSATION_HISTORY_TURNS = 6 +# Token budgets — observed per-step and total. Operators can override via env; +# defaults are conservative for cost containment without throttling normal +# planning loops. +REACT_STEP_TOKEN_BUDGET = int(os.getenv("REACT_STEP_TOKEN_BUDGET", "8000")) +REACT_TOTAL_TOKEN_BUDGET = int(os.getenv("REACT_TOTAL_TOKEN_BUDGET", "40000")) class StepType(str, Enum): @@ -51,6 +58,8 @@ class ReActStep: validation_result: Optional[dict] = None duration_ms: int = 0 revised: bool = False + tool_calls: list[dict[str, Any]] = field(default_factory=list) + tool_results: list[dict[str, Any]] = field(default_factory=list) @dataclass @@ -60,10 +69,18 @@ class ReActTrace: steps: list[ReActStep] = field(default_factory=list) total_operations: list[dict] = field(default_factory=list) final_summary: str = "" - success: bool = True + success: bool = False + completed: bool = False + failure_code: str = "" + failure_message: str = "" + partial_operations: bool = False iterations: int = 0 total_duration_ms: int = 0 max_iterations: int = 10 + total_input_tokens: int = 0 + total_output_tokens: int = 0 + budget_exceeded: bool = False + budget_reason: str = "" REACT_SYSTEM_PROMPT = """You are an expert financial analyst executing an Excel task step by step. @@ -133,6 +150,7 @@ def __init__( self, llm_generate: Callable, validator: Optional[Callable] = None, + tool_executor: Optional[Callable[..., Awaitable[Any]]] = None, max_iterations: int = 10, max_operations_per_step: int = 30, ): @@ -145,6 +163,7 @@ def __init__( """ self.llm_generate = llm_generate self.validator = validator + self.tool_executor = tool_executor self.max_iterations = max_iterations self.max_ops_per_step = max_operations_per_step @@ -154,6 +173,7 @@ async def execute( workbook_context: Optional[dict] = None, rules_prompt: str = "", on_step: Optional[Callable] = None, + tool_context: Optional[dict[str, Any]] = None, ) -> ReActTrace: """Execute a task using the ReAct loop. @@ -173,6 +193,10 @@ async def execute( system_prompt = REACT_SYSTEM_PROMPT if rules_prompt: system_prompt += "\n" + rules_prompt + if self.tool_executor: + from backend.agents.tools import get_tools_prompt + + system_prompt += get_tools_prompt() # Build initial user prompt with context context_block = "" @@ -212,6 +236,8 @@ async def execute( step_type=StepType.REFLECT, content=f"LLM timeout after {LLM_STEP_TIMEOUT_S:.0f}s; aborting ReAct loop", )) + trace.failure_code = "provider_timeout" + trace.failure_message = f"LLM timeout after {LLM_STEP_TIMEOUT_S:.0f}s" break except Exception as e: logger.error("react_llm_error", iteration=iteration, error=str(e)) @@ -220,10 +246,77 @@ async def execute( step_type=StepType.REFLECT, content=f"LLM error: {str(e)}", )) + trace.failure_code = "provider_error" + trace.failure_message = "The configured model provider failed during planning." + break + + # Token accounting. Providers expose usage via response.usage or + # response.token_usage; fall back to a char/4 heuristic so the + # budget gate still works for providers that don't report. + step_input_tokens = 0 + step_output_tokens = 0 + usage = getattr(response, "usage", None) or getattr(response, "token_usage", None) or {} + if isinstance(usage, dict): + step_input_tokens = int(usage.get("input_tokens") or usage.get("prompt_tokens") or 0) + step_output_tokens = int(usage.get("output_tokens") or usage.get("completion_tokens") or 0) + if not step_input_tokens: + step_input_tokens = max(1, len(user_prompt) // 4) + if not step_output_tokens: + step_output_tokens = max(1, len(response_text) // 4) + trace.total_input_tokens += step_input_tokens + trace.total_output_tokens += step_output_tokens + step_total = step_input_tokens + step_output_tokens + grand_total = trace.total_input_tokens + trace.total_output_tokens + if step_total > REACT_STEP_TOKEN_BUDGET: + logger.error( + "react_step_token_budget_exceeded", + iteration=iteration, + step_tokens=step_total, + budget=REACT_STEP_TOKEN_BUDGET, + ) + trace.budget_exceeded = True + trace.budget_reason = f"Step {iteration} consumed {step_total} tokens (> {REACT_STEP_TOKEN_BUDGET})." + trace.failure_code = "step_token_budget_exceeded" + trace.failure_message = trace.budget_reason + trace.steps.append(ReActStep( + step_num=iteration, + step_type=StepType.REFLECT, + content=trace.budget_reason, + )) + break + if grand_total > REACT_TOTAL_TOKEN_BUDGET: + logger.error( + "react_total_token_budget_exceeded", + grand_total=grand_total, + budget=REACT_TOTAL_TOKEN_BUDGET, + ) + trace.budget_exceeded = True + trace.budget_reason = ( + f"Total token usage {grand_total} exceeded budget {REACT_TOTAL_TOKEN_BUDGET}; aborting loop." + ) + trace.steps.append(ReActStep( + step_num=iteration, + step_type=StepType.REFLECT, + content=trace.budget_reason, + )) + trace.success = False + trace.failure_code = "total_token_budget_exceeded" + trace.failure_message = trace.budget_reason break # Parse the response - step = self._parse_step(response_text, iteration) + try: + step = self._parse_step(response_text, iteration) + except ValueError as exc: + logger.error("react_response_invalid", iteration=iteration, error=str(exc)) + trace.failure_code = "invalid_model_response" + trace.failure_message = str(exc) + trace.steps.append(ReActStep( + step_num=iteration, + step_type=StepType.REFLECT, + content=f"Invalid model response: {exc}", + )) + break step.duration_ms = int((time.time() - step_start) * 1000) # ── THINK step ─────────────────────────────────────────── @@ -240,6 +333,68 @@ async def execute( # ── ACT step ───────────────────────────────────────────── if step.step_type == StepType.ACT: + if step.tool_calls: + if not self.tool_executor: + trace.failure_code = "tool_executor_unavailable" + trace.failure_message = "The model requested tools but no tool executor is configured." + trace.steps.append(step) + break + from backend.agents.tools import ToolCall + + tool_operations: list[dict[str, Any]] = [] + for tool_call in step.tool_calls: + try: + result = await self.tool_executor( + ToolCall( + tool_name=tool_call["tool"], + arguments=dict(tool_call["args"]), + call_id=tool_call["id"], + ), + tool_context or workbook_context or {}, + ) + except Exception as exc: + logger.error("react_tool_error", tool=tool_call["tool"], error=str(exc)) + trace.failure_code = "tool_error" + trace.failure_message = f"Tool {tool_call['tool']} failed." + break + observation = { + "call_id": tool_call["id"], + "tool": tool_call["tool"], + "success": bool(getattr(result, "success", False)), + "result": getattr(result, "result", None), + "error": getattr(result, "error", None), + } + step.tool_results.append(observation) + if not observation["success"]: + trace.failure_code = "tool_error" + trace.failure_message = ( + f"Tool {tool_call['tool']} failed: {observation['error'] or 'unknown error'}" + ) + break + tool_result = observation["result"] + if isinstance(tool_result, dict): + proposed = tool_result.get("operations") or [] + if not isinstance(proposed, list) or not all(isinstance(item, dict) for item in proposed): + trace.failure_code = "invalid_tool_result" + trace.failure_message = f"Tool {tool_call['tool']} returned invalid operations." + break + tool_operations.extend(proposed) + if trace.failure_code: + trace.steps.append(step) + break + step.operations.extend(tool_operations) + if len(step.operations) > self.max_ops_per_step: + trace.failure_code = "operation_limit_exceeded" + trace.failure_message = ( + f"Tool-expanded operation limit exceeded: " + f"{len(step.operations)} > {self.max_ops_per_step}." + ) + trace.steps.append(step) + break + conversation.append( + "OBSERVE (tool results):\n" + json.dumps(step.tool_results, default=str)[:6000] + ) + # Validate operations if step.operations and self.validator: try: @@ -256,13 +411,17 @@ async def execute( ) trace.steps.append(observe_step) conversation.append( - f"OBSERVE: Validation FAILED with errors:\n" + + "OBSERVE: Validation FAILED with errors:\n" + json.dumps(validation["errors"][:3], indent=2) + "\n\nPlease THINK about what went wrong, then ACT to fix the operations." ) continue except Exception as e: - logger.warning("react_validation_error", error=str(e)) + logger.error("react_validation_error", error=str(e)) + trace.failure_code = "validator_error" + trace.failure_message = "Operation validation failed unexpectedly." + trace.steps.append(step) + break # Operations passed validation — accumulate trace.total_operations.extend(step.operations) @@ -277,7 +436,9 @@ async def execute( await _call_step_callback(on_step, step, trace) # Check if complete - if not step.operations: + if not step.operations and not step.tool_calls: + trace.failure_code = "empty_act" + trace.failure_message = "The model returned an ACT step without operations or tool calls." break # Ask for next step or completion @@ -291,6 +452,12 @@ async def execute( if step.step_type == StepType.COMPLETE: trace.steps.append(step) trace.final_summary = step.content + if not trace.total_operations: + trace.failure_code = "no_operations" + trace.failure_message = "The model completed without producing validated operations." + break + trace.completed = True + trace.success = True if on_step: await _call_step_callback(on_step, step, trace) break @@ -307,7 +474,13 @@ async def execute( trace.iterations = iteration trace.total_duration_ms = int((time.time() - start_time) * 1000) - trace.success = len(trace.total_operations) > 0 + if not trace.completed and not trace.failure_code: + trace.failure_code = "max_iterations_exceeded" + trace.failure_message = "The planning loop ended before explicit completion." + trace.success = trace.completed and not trace.failure_code and bool(trace.total_operations) + trace.partial_operations = bool(trace.total_operations) and not trace.success + if not trace.success and not trace.final_summary: + trace.final_summary = trace.failure_message logger.info( "react_execution_complete", @@ -325,12 +498,8 @@ def _parse_step(self, response_text: str, step_num: int) -> ReActStep: try: # Try to extract JSON from the response data = _extract_json(response_text) - if not data: - return ReActStep( - step_num=step_num, - step_type=StepType.THINK, - content=response_text[:500], - ) + if not isinstance(data, dict): + raise ValueError("response must contain one JSON object") step_type_str = data.get("step_type", "think").lower() step_type = { @@ -342,31 +511,49 @@ def _parse_step(self, response_text: str, step_num: int) -> ReActStep: }.get(step_type_str, StepType.THINK) operations = data.get("operations", []) - # Cap operations per step. Surface the truncation in the step's - # content so the audit log records that the model attempted more - # writes than we accepted, rather than silently dropping them. - dropped = max(0, len(operations) - self.max_ops_per_step) - if dropped: - operations = operations[:self.max_ops_per_step] + if not isinstance(operations, list) or not all(isinstance(item, dict) for item in operations): + raise ValueError("operations must be an array of objects") + if len(operations) > self.max_ops_per_step: + raise ValueError( + f"operation limit exceeded: {len(operations)} > {self.max_ops_per_step}" + ) + + raw_tool_calls = data.get("tool_calls", []) + if not isinstance(raw_tool_calls, list): + raise ValueError("tool_calls must be an array") + tool_calls: list[dict[str, Any]] = [] + for index, item in enumerate(raw_tool_calls): + if not isinstance(item, dict): + raise ValueError(f"tool_calls[{index}] must be an object") + name = item.get("tool") or item.get("name") + arguments = item.get("args", item.get("arguments", {})) + if not isinstance(name, str) or not name.strip(): + raise ValueError(f"tool_calls[{index}] is missing a tool name") + if not isinstance(arguments, dict): + raise ValueError(f"tool_calls[{index}].args must be an object") + tool_calls.append( + { + "id": str(item.get("id") or item.get("call_id") or f"tool-{step_num}-{index}"), + "tool": name.strip(), + "args": arguments, + } + ) content = data.get("reasoning", "") or data.get("summary", "") or "" - if dropped: - content = f"{content}\n[truncated {dropped} additional operation(s) over per-step cap]" return ReActStep( step_num=step_num, step_type=step_type, content=content, operations=operations, + tool_calls=tool_calls, ) + except ValueError: + raise except Exception as e: logger.warning("react_parse_error", error=str(e)) - return ReActStep( - step_num=step_num, - step_type=StepType.THINK, - content=f"Parse error: {response_text[:200]}", - ) + raise ValueError("response could not be parsed") from e def _extract_json(text: str) -> Optional[dict]: diff --git a/backend/agents/rules.py b/backend/agents/rules.py index 8939b43..e5a4728 100644 --- a/backend/agents/rules.py +++ b/backend/agents/rules.py @@ -7,7 +7,7 @@ from __future__ import annotations import json -from dataclasses import dataclass, field, asdict +from dataclasses import dataclass, asdict from pathlib import Path from typing import Optional @@ -146,7 +146,7 @@ def get_rules_prompt(self) -> str: ] # Group by category - categories = {} + categories: dict[str, list[Rule]] = {} for rule in enabled: cat = categories.setdefault(rule.category.upper(), []) cat.append(rule) diff --git a/backend/agents/template_router.py b/backend/agents/template_router.py index f5f9abb..3c8e743 100644 --- a/backend/agents/template_router.py +++ b/backend/agents/template_router.py @@ -15,7 +15,7 @@ import structlog -from backend.templates.registry import list_templates, generate_template, get_template +from backend.templates.registry import generate_template, get_template from backend.excel.formatting_engine import post_process_operations logger = structlog.get_logger() @@ -258,6 +258,8 @@ def try_template_route( if not template_meta: return None + guided_workflow = template_meta.get("guided_workflow") + # Extract any user-specified parameters from the prompt params = extract_template_params(prompt, template_id) @@ -292,6 +294,8 @@ def try_template_route( "operations_count": len(operations), "verified": True, # Pre-built templates are pre-verified "retry_count": 0, + "workflow_id": template_meta.get("workflow_id"), + "guided_workflow": guided_workflow, }, "error": None, } diff --git a/backend/agents/tools.py b/backend/agents/tools.py index 8b03adb..ab3f67c 100644 --- a/backend/agents/tools.py +++ b/backend/agents/tools.py @@ -16,9 +16,8 @@ from __future__ import annotations -import json -from dataclasses import dataclass, field -from typing import Callable, Optional +from dataclasses import dataclass +from typing import Any, Callable, Optional import structlog @@ -38,7 +37,7 @@ class ToolResult: """The result of a tool execution.""" tool_name: str success: bool - result: object + result: Any error: Optional[str] = None @@ -60,14 +59,18 @@ async def tool_write_cells(args: dict) -> dict: cells: list of {cell, value?, formula?, type?, format?} """ cells = args.get("cells", []) - if not cells: + if not isinstance(cells, list) or not cells: return {"error": "No cells provided", "written": 0} # Validate cell references and normalize to typed workbook operations. import re valid = [] - for cell in cells: - ref = cell.get("cell", "") + invalid: list[str] = [] + for index, cell in enumerate(cells): + if not isinstance(cell, dict): + invalid.append(f"cells[{index}] is not an object") + continue + ref = str(cell.get("cell") or "") if re.match(r'^[A-Z]{1,3}\d{1,7}$', ref, re.IGNORECASE): valid.append({ "type": "write_cell", @@ -77,6 +80,10 @@ async def tool_write_cells(args: dict) -> dict: "format": cell.get("format"), "intent": cell.get("intent", "agent proposed write"), }) + else: + invalid.append(f"cells[{index}] has an invalid cell reference") + if invalid: + return {"error": "; ".join(invalid[:10]), "written": 0} return { "written": len(valid), @@ -86,20 +93,104 @@ async def tool_write_cells(args: dict) -> dict: async def tool_read_cells(args: dict) -> dict: - """Read current values from cells in the workbook. - - This is a stub that returns data from the workbook_context. - The actual read happens on the frontend side. + """Read a bounded range from the caller-supplied workbook snapshot. + + The backend cannot synchronously query a live Office host. This tool is + therefore explicit about its authority: it reads only the immutable + workbook context attached to the planning request and fails if that + snapshot is absent or incomplete for the requested range. """ - range_ref = args.get("range", "") - context = args.get("_context", {}) - snapshot = context.get("cell_snapshot", {}) - - result = {} - if range_ref in snapshot: - result[range_ref] = snapshot[range_ref] - - return {"cells": result, "range": range_ref} + range_ref = str(args.get("range") or "").strip() + context = args.get("_context") + if not isinstance(context, dict): + return {"error": "Workbook context is unavailable"} + snapshot = context.get("cell_snapshot") + if not isinstance(snapshot, dict) or not snapshot: + return {"error": "Workbook context has no cell snapshot"} + + try: + sheet, requested_cells = _expand_snapshot_range( + range_ref, + active_sheet=str(context.get("active_sheet") or ""), + ) + except ValueError as exc: + return {"error": str(exc)} + + cells: dict[str, Any] = {} + missing: list[str] = [] + for address in requested_cells: + qualified = f"{sheet}!{address}" if sheet else address + if qualified in snapshot: + cells[qualified] = snapshot[qualified] + elif address in snapshot: + cells[qualified] = snapshot[address] + else: + missing.append(qualified) + snapshot_complete = bool( + context.get("cell_snapshot_complete") + or context.get("complete") + or (isinstance(context.get("context_limits"), dict) and context["context_limits"].get("truncated") is False) + ) + if missing and not snapshot_complete: + return { + "error": "Workbook snapshot does not cover the entire requested range", + "missing": missing[:25], + } + + return { + "cells": cells, + "range": range_ref, + "source": "workbook_context_snapshot", + "complete": snapshot_complete or not missing, + "explicit_blanks": missing if snapshot_complete else [], + } + + +def _expand_snapshot_range(range_ref: str, *, active_sheet: str, max_cells: int = 5000) -> tuple[str, list[str]]: + import re + + match = re.fullmatch( + r"(?:(?:'((?:[^']|'')+)'|([^'!]+))!)?\$?([A-Za-z]{1,3})\$?(\d{1,7})(?::\$?([A-Za-z]{1,3})\$?(\d{1,7}))?", + range_ref, + ) + if not match: + raise ValueError("range must be a valid A1 cell or rectangular range") + quoted_sheet, plain_sheet, start_col, start_row, end_col, end_row = match.groups() + sheet = (quoted_sheet.replace("''", "'") if quoted_sheet else plain_sheet) or active_sheet + start_col_num = _column_number(start_col) + end_col_num = _column_number(end_col or start_col) + start_row_num = int(start_row) + end_row_num = int(end_row or start_row) + if not (1 <= start_col_num <= 16384 and 1 <= end_col_num <= 16384): + raise ValueError("range exceeds Excel's column limit") + if not (1 <= start_row_num <= 1048576 and 1 <= end_row_num <= 1048576): + raise ValueError("range exceeds Excel's row limit") + first_col, last_col = sorted((start_col_num, end_col_num)) + first_row, last_row = sorted((start_row_num, end_row_num)) + count = (last_col - first_col + 1) * (last_row - first_row + 1) + if count > max_cells: + raise ValueError(f"range contains {count} cells; maximum tool read is {max_cells}") + addresses = [ + f"{_column_letters(column)}{row}" + for row in range(first_row, last_row + 1) + for column in range(first_col, last_col + 1) + ] + return sheet, addresses + + +def _column_number(letters: str) -> int: + value = 0 + for character in letters.upper(): + value = value * 26 + ord(character) - 64 + return value + + +def _column_letters(number: int) -> str: + result = "" + while number: + number, remainder = divmod(number - 1, 26) + result = chr(65 + remainder) + result + return result async def tool_format_range(args: dict) -> dict: @@ -165,7 +256,7 @@ async def tool_validate_range(args: dict) -> dict: ], } except ImportError: - return {"valid": True, "issues": []} + return {"error": "Formula validator is unavailable"} async def tool_create_sheet(args: dict) -> dict: @@ -205,7 +296,7 @@ async def tool_lookup_template(args: dict) -> dict: ], } except ImportError: - return {"template_id": template_id, "outputs": []} + return {"error": "Template registry is unavailable", "template_id": template_id} async def tool_inspect_dependencies(args: dict) -> dict: @@ -267,6 +358,10 @@ async def tool_get_workbook_metadata(args: dict) -> dict: async def tool_get_cell_ranges(args: dict) -> dict: """Plan a read of one or more ranges before mutation.""" ranges = args.get("ranges") or ([args["range"]] if args.get("range") else []) + if not isinstance(ranges, list) or not ranges: + return {"error": "At least one workbook range is required"} + if any(not isinstance(range_ref, str) or not range_ref.strip() for range_ref in ranges): + return {"error": "Every workbook range must be a non-empty string"} operations = [ { "type": "get_cell_ranges", @@ -756,7 +851,10 @@ def get_tools_prompt() -> str: return "\n".join(lines) -async def execute_tool_call(call: ToolCall, context: dict = None) -> ToolResult: +async def execute_tool_call( + call: ToolCall, + context: dict[str, Any] | None = None, +) -> ToolResult: """Execute a tool call and return the result.""" tool = next((t for t in TOOLS if t.name == call.tool_name), None) if not tool: @@ -768,10 +866,24 @@ async def execute_tool_call(call: ToolCall, context: dict = None) -> ToolResult: ) try: - args = call.arguments - if context: - args["_context"] = context + validation_error = _tool_argument_error(tool, call.arguments) + if validation_error: + return ToolResult( + tool_name=call.tool_name, + success=False, + result=None, + error=validation_error, + ) + args = dict(call.arguments) + args["_context"] = context or {} result = await tool.handler(args) + if isinstance(result, dict) and result.get("error"): + return ToolResult( + tool_name=call.tool_name, + success=False, + result=result, + error=str(result["error"]), + ) return ToolResult( tool_name=call.tool_name, success=True, @@ -785,3 +897,13 @@ async def execute_tool_call(call: ToolCall, context: dict = None) -> ToolResult: result=None, error=str(e), ) + + +def _tool_argument_error(tool: ToolSpec, arguments: object) -> str | None: + if not isinstance(arguments, dict): + return "Tool arguments must be an object" + required = tool.parameters.get("required") or [] + for name in required: + if name not in arguments or arguments[name] is None: + return f"Missing required tool argument: {name}" + return None diff --git a/backend/agents/verifiers/logic_agent.py b/backend/agents/verifiers/logic_agent.py index 8862ec4..7c6fd90 100644 --- a/backend/agents/verifiers/logic_agent.py +++ b/backend/agents/verifiers/logic_agent.py @@ -16,9 +16,8 @@ from __future__ import annotations -import json import re -from dataclasses import dataclass, field +from dataclasses import dataclass from typing import Optional import structlog @@ -210,7 +209,6 @@ def _verify_growth_rates(self, operations: list[dict], cell_map: dict) -> list[L for cell, info in cell_map.items(): formula = info.get("formula", "") - label = info.get("label", "") if not formula: continue @@ -222,7 +220,6 @@ def _verify_growth_rates(self, operations: list[dict], cell_map: dict) -> list[L ) if growth_match: prev_cell = growth_match.group(1).upper() - rate_ref = growth_match.group(2) # Verify the previous cell is in the same row, adjacent column if prev_cell in cell_map: diff --git a/backend/agents/verifiers/qa_agent.py b/backend/agents/verifiers/qa_agent.py index ef750fa..e99705f 100644 --- a/backend/agents/verifiers/qa_agent.py +++ b/backend/agents/verifiers/qa_agent.py @@ -17,7 +17,7 @@ from __future__ import annotations import re -from dataclasses import dataclass, field +from dataclasses import dataclass from typing import Optional import structlog @@ -222,7 +222,6 @@ def _check_rounding_accumulation(self, operations: list[dict]) -> list[QAIssue]: issues = [] # Find SUM formulas and check if their operands are individually rounded - sum_cells = {} for op in operations: formula = op.get("formula", "") cell = op.get("cell", "") @@ -309,7 +308,6 @@ def _check_template_completeness(self, operations: list[dict]) -> list[QAIssue]: } for section_name, keywords in expected_sections.items(): - found = any(any(kw in label_text for kw in keywords) for _ in [1]) if not any(kw in label_text for kw in keywords): issues.append(QAIssue( cell="MODEL", diff --git a/backend/agents/workflows.py b/backend/agents/workflows.py index 84acc8f..d5970ed 100644 --- a/backend/agents/workflows.py +++ b/backend/agents/workflows.py @@ -11,18 +11,32 @@ from __future__ import annotations +import errno +import hashlib import json +import os +import re +import threading import time import uuid +from contextlib import contextmanager from dataclasses import dataclass, field from pathlib import Path -from typing import Optional +from typing import Any, BinaryIO, Iterator, Optional import structlog +from backend.api.security.ownership import ( + OwnershipNotFoundError, + current_principal, + require_session_access, +) + logger = structlog.get_logger() WORKFLOWS_DIR = Path(__file__).parent.parent / "storage" / "workflows" +_WORKFLOW_LOCKS_GUARD = threading.Lock() +_WORKFLOW_LOCKS: dict[str, threading.RLock] = {} @dataclass @@ -46,12 +60,16 @@ class SavedWorkflow: tags: list[str] = field(default_factory=list) execution_count: int = 0 last_executed: Optional[str] = None + owner_user_id: str = "local" + session_id: str = "" + built_in: bool = False class WorkflowEngine: """Engine for creating, saving, and executing reusable workflows.""" - def __init__(self): + def __init__(self, *, store: Any | None = None): + self._store = store self._workflows: dict[str, SavedWorkflow] = {} self._load_workflows() self._register_defaults() @@ -76,6 +94,10 @@ def _load_workflows(self): variables=data.get("variables", {}), tags=data.get("tags", []), execution_count=data.get("execution_count", 0), + last_executed=data.get("last_executed"), + owner_user_id=str(data.get("owner_user_id") or "local"), + session_id=str(data.get("session_id") or ""), + built_in=bool(data.get("built_in", False)), ) self._workflows[wf.id] = wf except Exception as e: @@ -91,8 +113,8 @@ def _register_defaults(self): steps=[ WorkflowStep( step_type="data_fetch", - config={"source": "edgar", "form": "10-Q", "count": 1}, - description="Pull latest 10-Q from EDGAR", + config={"source": "edgar", "operation": "financials", "form": "10-Q", "preview_limit": 10}, + description="Pull latest quarterly XBRL financial facts from EDGAR", ), WorkflowStep( step_type="template", @@ -165,8 +187,9 @@ def _register_defaults(self): ] for wf in defaults: - if wf.id not in self._workflows: - self._workflows[wf.id] = wf + wf.built_in = True + wf.owner_user_id = "" + self._workflows[wf.id] = wf def list_workflows(self) -> list[dict]: """List all saved workflows.""" @@ -181,31 +204,68 @@ def list_workflows(self) -> list[dict]: "execution_count": wf.execution_count, } for wf in self._workflows.values() + if self._can_access(wf) ] def get_workflow(self, workflow_id: str) -> Optional[SavedWorkflow]: - return self._workflows.get(workflow_id) + workflow = self._workflows.get(workflow_id) + return workflow if workflow and self._can_access(workflow) else None def save_workflow(self, workflow: SavedWorkflow): """Save or update a workflow.""" - workflow.updated_at = time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime()) - self._workflows[workflow.id] = workflow - - # Persist to disk - WORKFLOWS_DIR.mkdir(parents=True, exist_ok=True) - data = { - "id": workflow.id, - "name": workflow.name, - "description": workflow.description, - "steps": [{"step_type": s.step_type, "config": s.config, "description": s.description} - for s in workflow.steps], - "variables": workflow.variables, - "tags": workflow.tags, - "created_at": workflow.created_at, - "updated_at": workflow.updated_at, - "execution_count": workflow.execution_count, - } - (WORKFLOWS_DIR / f"{workflow.id}.json").write_text(json.dumps(data, indent=2)) + if workflow.built_in: + self._workflows[workflow.id] = workflow + return + principal = current_principal() + owner_user_id = principal.user_id if principal.authenticated else "local" + workflow.session_id = str(workflow.session_id or "").strip() + workflow_path = _workflow_path(workflow.id) + with _workflow_storage_lock(workflow.session_id): + if workflow.session_id: + self._require_active_session(workflow.session_id) + durable_existing = _read_workflow_record(workflow_path) + existing = self._workflows.get(workflow.id) + if durable_existing is not None: + existing_owner = str(durable_existing.get("owner_user_id") or "local") + existing_session_id = str(durable_existing.get("session_id") or "").strip() + else: + existing_owner = str(existing.owner_user_id if existing else "") or "local" + existing_session_id = str(existing.session_id if existing else "").strip() + if durable_existing is not None or existing is not None: + if existing_session_id != workflow.session_id: + workflow.session_id = existing_session_id + raise ValueError(f"Workflow '{workflow.id}' not found") + if not principal.is_admin and existing_owner != owner_user_id: + raise ValueError(f"Workflow '{workflow.id}' not found") + if not workflow.owner_user_id: + workflow.owner_user_id = owner_user_id + if not principal.is_admin and workflow.owner_user_id != owner_user_id: + raise ValueError(f"Workflow '{workflow.id}' not found") + workflow.updated_at = time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime()) + data = { + "id": workflow.id, + "name": workflow.name, + "description": workflow.description, + "steps": [ + { + "step_type": step.step_type, + "config": step.config, + "description": step.description, + } + for step in workflow.steps + ], + "variables": workflow.variables, + "tags": workflow.tags, + "created_at": workflow.created_at, + "updated_at": workflow.updated_at, + "execution_count": workflow.execution_count, + "last_executed": workflow.last_executed, + "owner_user_id": workflow.owner_user_id, + "session_id": workflow.session_id, + "built_in": False, + } + _atomic_write_workflow(workflow_path, data) + self._workflows[workflow.id] = workflow logger.info("workflow_saved", id=workflow.id) def create_workflow( @@ -213,10 +273,12 @@ def create_workflow( name: str, description: str, steps: list[dict], - variables: dict = None, - tags: list = None, + variables: Optional[dict] = None, + tags: Optional[list] = None, + session_id: str = "", ) -> SavedWorkflow: """Create a new workflow.""" + principal = current_principal() wf = SavedWorkflow( id=str(uuid.uuid4())[:8], name=name, @@ -226,23 +288,77 @@ def create_workflow( updated_at=time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime()), variables=variables or {}, tags=tags or [], + owner_user_id=principal.user_id if principal.authenticated else "local", + session_id=str(session_id or ""), ) self.save_workflow(wf) return wf def delete_workflow(self, workflow_id: str) -> bool: - wf = self._workflows.pop(workflow_id, None) - if wf: - filepath = WORKFLOWS_DIR / f"{workflow_id}.json" - if filepath.exists(): - filepath.unlink() - return True - return False + workflow_path = _workflow_path(workflow_id) + with _workflow_catalog_lock(): + durable = _read_workflow_record(workflow_path) + in_memory = self._workflows.get(workflow_id) + session_id = ( + str(durable.get("session_id") or "").strip() + if durable is not None + else str(in_memory.session_id if in_memory else "").strip() + ) + with _workflow_session_lock(session_id): + if durable is None and in_memory is None: + return False + if bool((durable or {}).get("built_in")) or (in_memory and in_memory.built_in): + return False + if session_id: + self._require_active_session(session_id) + principal = current_principal() + owner_user_id = principal.user_id if principal.authenticated else "local" + workflow_owner = ( + str(durable.get("owner_user_id") or "local") + if durable is not None + else str(in_memory.owner_user_id if in_memory else "") or "local" + ) + if not principal.is_admin and workflow_owner != owner_user_id: + return False + self._workflows.pop(workflow_id, None) + workflow_path.unlink(missing_ok=True) + _fsync_directory(WORKFLOWS_DIR) + return True + + def purge_session(self, session_id: str) -> int: + """Remove durable workflows explicitly bound to a purged session.""" + normalized_session_id = str(session_id or "").strip() + if not normalized_session_id: + raise ValueError("session_id is required") + with _workflow_storage_lock(normalized_session_id): + self._require_purging_session(normalized_session_id) + durable_records = _session_workflow_records(normalized_session_id) + workflow_ids = { + workflow.id + for workflow in self._workflows.values() + if not workflow.built_in and workflow.session_id == normalized_session_id + } + for workflow_path, workflow_id in durable_records: + workflow_ids.add(workflow_id) + workflow_path.unlink(missing_ok=True) + for workflow_id in workflow_ids: + self._workflows.pop(workflow_id, None) + if durable_records: + _fsync_directory(WORKFLOWS_DIR) + return len(workflow_ids) + + def count_session_workflows(self, session_id: str) -> int: + """Return the authoritative durable workflow count for deletion proof.""" + normalized_session_id = str(session_id or "").strip() + if not normalized_session_id: + raise ValueError("session_id is required") + with _workflow_storage_lock(normalized_session_id): + return len(_session_workflow_records(normalized_session_id)) async def execute_workflow( self, workflow_id: str, - variables: dict = None, + variables: Optional[dict] = None, pipeline_fn=None, ) -> dict: """Execute a saved workflow. @@ -252,9 +368,12 @@ async def execute_workflow( variables: Variable substitutions (e.g., {"ticker": "AAPL"}) pipeline_fn: Async function to execute prompts through the pipeline """ - wf = self._workflows.get(workflow_id) + wf = self.get_workflow(workflow_id) if not wf: raise ValueError(f"Workflow '{workflow_id}' not found") + if wf.session_id: + with _workflow_storage_lock(wf.session_id): + self._require_active_session(wf.session_id) variables = variables or {} results = [] @@ -279,11 +398,11 @@ async def execute_workflow( step_result["operations"] = len(result.get("operations", [])) else: step_result["status"] = "skipped" - step_result["prompt"] = prompt + step_result["reason"] = "pipeline_unavailable" elif step.step_type == "data_fetch": - step_result["status"] = "complete" - step_result["source"] = step.config.get("source") + fetch_result = await self._execute_data_fetch_step(step, variables) + step_result.update(fetch_result) elif step.step_type == "chart": step_result["status"] = "complete" @@ -299,9 +418,10 @@ async def execute_workflow( results.append(step_result) # Update execution stats - wf.execution_count += 1 - wf.last_executed = time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime()) - self.save_workflow(wf) + if not wf.built_in: + wf.execution_count += 1 + wf.last_executed = time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime()) + self.save_workflow(wf) return { "workflow_id": workflow_id, @@ -310,6 +430,283 @@ async def execute_workflow( "results": results, } + async def _execute_data_fetch_step(self, step: WorkflowStep, variables: dict) -> dict: + from backend.data.connectors import get_data_registry + + source = str(step.config.get("source", "")).strip() + if not source: + return {"status": "error", "error": "data_fetch step is missing source"} + + query = self._substitute(str(step.config.get("query") or ""), variables).strip() + if not query: + query = str( + variables.get("ticker") + or variables.get("symbol") + or variables.get("company") + or variables.get("query") + or "" + ).strip() + if not query: + return { + "status": "skipped", + "source": source, + "reason": "missing_query", + "required_variables": ["ticker", "symbol", "company", "query"], + } + + params = { + key: self._substitute(value, variables) + for key, value in step.config.items() + if key not in {"source", "query", "preview_limit"} + } + if "form" in params and "form_type" not in params: + params["form_type"] = params.pop("form") + + result = await get_data_registry().fetch(source, query, params) + preview_limit = int(step.config.get("preview_limit", 5) or 5) + data_preview = result.data[: max(preview_limit, 0)] + return { + "status": "complete", + "source": result.source, + "query": query, + "row_count": result.row_count, + "headers": result.headers, + "metadata": result.metadata, + "data_preview": data_preview, + "provenance_preview": self._provenance_preview(data_preview), + } + + def _provenance_preview(self, rows: list[dict]) -> list[dict]: + provenance_fields = [ + "line_item", + "concept", + "taxonomy", + "unit", + "filed", + "period_end", + "form", + "accession_number", + "source_url", + ] + preview = [] + for row in rows: + provenance = { + field: row.get(field) + for field in provenance_fields + if row.get(field) not in (None, "") + } + if provenance: + preview.append(provenance) + return preview + + def _substitute(self, value, variables: dict): + if isinstance(value, str): + resolved = value + for var_name, var_value in variables.items(): + resolved = resolved.replace(f"{{{var_name}}}", str(var_value)) + return resolved + if isinstance(value, dict): + return {key: self._substitute(item, variables) for key, item in value.items()} + if isinstance(value, list): + return [self._substitute(item, variables) for item in value] + return value + + def _can_access(self, workflow: SavedWorkflow) -> bool: + if workflow.built_in: + return True + if workflow.session_id and not self._session_is_active(workflow.session_id): + return False + principal = current_principal() + if principal.authenticated and principal.is_admin: + return True + owner_user_id = principal.user_id if principal.authenticated else "local" + return bool(owner_user_id) and workflow.owner_user_id == owner_user_id + + def _store_instance(self): + if self._store is None: + from backend.storage import get_session_store + + self._store = get_session_store() + return self._store + + def _require_active_session(self, session_id: str) -> None: + normalized_session_id = str(session_id or "").strip() + if not normalized_session_id: + raise OwnershipNotFoundError("Session not found") + try: + store = self._store_instance() + lifecycle_state = store.get_session_lifecycle_state(normalized_session_id) + if str(lifecycle_state or "") != "active": + raise OwnershipNotFoundError("Session not found") + require_session_access(store, normalized_session_id, create=False) + except OwnershipNotFoundError: + raise + except Exception as exc: + raise OwnershipNotFoundError("Session not found") from exc + + def _require_purging_session(self, session_id: str) -> None: + try: + lifecycle_state = self._store_instance().get_session_lifecycle_state(session_id) + except Exception as exc: + raise RuntimeError("Workflow purge lifecycle state is unavailable") from exc + if str(lifecycle_state or "") != "purging": + raise RuntimeError("Workflow purge requires a durable purging session") + + def _session_is_active(self, session_id: str) -> bool: + try: + self._require_active_session(session_id) + except OwnershipNotFoundError: + return False + return True + + +def _workflow_path(workflow_id: str) -> Path: + normalized = str(workflow_id or "") + if not re.fullmatch(r"[A-Za-z0-9_-]{1,64}", normalized): + raise ValueError("Invalid workflow ID") + return WORKFLOWS_DIR / f"{normalized}.json" + + +@contextmanager +def _workflow_storage_lock(session_id: str) -> Iterator[None]: + with _workflow_catalog_lock(): + with _workflow_session_lock(session_id): + yield + + +@contextmanager +def _workflow_catalog_lock() -> Iterator[None]: + with _workflow_file_lock("catalog"): + yield + + +@contextmanager +def _workflow_session_lock(session_id: str) -> Iterator[None]: + normalized_session_id = str(session_id or "").strip() or "unbound" + with _workflow_file_lock(f"session:{normalized_session_id}"): + yield + + +@contextmanager +def _workflow_file_lock(lock_identity: str) -> Iterator[None]: + lock_digest = hashlib.sha256(lock_identity.encode("utf-8")).hexdigest() + lock_dir = WORKFLOWS_DIR / ".locks" + lock_dir.mkdir(parents=True, exist_ok=True) + lock_path = lock_dir / f"{lock_digest}.lock" + lock_key = str(lock_path.resolve()) + with _WORKFLOW_LOCKS_GUARD: + process_lock = _WORKFLOW_LOCKS.setdefault(lock_key, threading.RLock()) + with process_lock: + with lock_path.open("a+b") as lock_file: + _acquire_os_lock(lock_file) + try: + yield + finally: + _release_os_lock(lock_file) + + +def _acquire_os_lock(lock_file: BinaryIO) -> None: + if os.name == "nt": + import msvcrt + + lock_file.seek(0, os.SEEK_END) + if lock_file.tell() == 0: + lock_file.write(b"\0") + lock_file.flush() + lock_file.seek(0) + while True: + try: + msvcrt.locking( # type: ignore[attr-defined] + lock_file.fileno(), + msvcrt.LK_NBLCK, # type: ignore[attr-defined] + 1, + ) + return + except OSError as exc: + if exc.errno not in {errno.EACCES, errno.EAGAIN, errno.EDEADLK}: + raise + time.sleep(0.05) + else: + import fcntl + + fcntl.flock(lock_file.fileno(), fcntl.LOCK_EX) + + +def _release_os_lock(lock_file: BinaryIO) -> None: + if os.name == "nt": + import msvcrt + + lock_file.seek(0) + msvcrt.locking( # type: ignore[attr-defined] + lock_file.fileno(), + msvcrt.LK_UNLCK, # type: ignore[attr-defined] + 1, + ) + else: + import fcntl + + fcntl.flock(lock_file.fileno(), fcntl.LOCK_UN) + + +def _read_workflow_record(workflow_path: Path) -> Optional[dict[str, Any]]: + try: + payload = json.loads(workflow_path.read_text(encoding="utf-8")) + except FileNotFoundError: + return None + except (OSError, UnicodeError, json.JSONDecodeError) as exc: + raise RuntimeError(f"Could not read durable workflow '{workflow_path.name}'") from exc + if not isinstance(payload, dict): + raise RuntimeError(f"Durable workflow '{workflow_path.name}' is not an object") + return payload + + +def _session_workflow_records(session_id: str) -> list[tuple[Path, str]]: + records: list[tuple[Path, str]] = [] + if not WORKFLOWS_DIR.exists(): + return records + candidate_paths = { + *WORKFLOWS_DIR.glob("*.json"), + *WORKFLOWS_DIR.glob(".*.tmp"), + } + for workflow_path in sorted(candidate_paths): + payload = _read_workflow_record(workflow_path) + if payload is None or bool(payload.get("built_in", False)): + continue + if str(payload.get("session_id") or "").strip() != session_id: + continue + workflow_id = str(payload.get("id") or workflow_path.stem) + _workflow_path(workflow_id) + records.append((workflow_path, workflow_id)) + return records + + +def _atomic_write_workflow(workflow_path: Path, payload: dict[str, Any]) -> None: + WORKFLOWS_DIR.mkdir(parents=True, exist_ok=True) + temporary = workflow_path.with_name(f".{workflow_path.name}.{uuid.uuid4().hex}.tmp") + try: + with temporary.open("x", encoding="utf-8") as workflow_file: + json.dump(payload, workflow_file, indent=2) + workflow_file.flush() + os.fsync(workflow_file.fileno()) + temporary.chmod(0o600) + temporary.replace(workflow_path) + _fsync_directory(WORKFLOWS_DIR) + finally: + temporary.unlink(missing_ok=True) + + +def _fsync_directory(directory: Path) -> None: + try: + directory_fd = os.open(directory, os.O_RDONLY) + except OSError: + return + try: + os.fsync(directory_fd) + except OSError: + pass + finally: + os.close(directory_fd) + # ── Singleton ──────────────────────────────────────────────────────────── diff --git a/backend/api/middleware/auth.py b/backend/api/middleware/auth.py index 6a019ad..c143350 100644 --- a/backend/api/middleware/auth.py +++ b/backend/api/middleware/auth.py @@ -1,69 +1,120 @@ """API key authentication middleware. -Validates the X-API-Key header on HTTP requests except public endpoints -(/health, /docs, /openapi.json). WebSocket authentication is handled by a -separate validator because browser WebSocket clients cannot set headers. +Validates the X-API-Key header on HTTP requests except public endpoints. +WebSocket authentication is handled by a separate validator and accepts only +header or subprotocol credentials. """ from __future__ import annotations -import hashlib import hmac -import time -from typing import Optional +from typing import Any, Optional import structlog from starlette.middleware.base import BaseHTTPMiddleware from starlette.requests import Request from starlette.responses import JSONResponse +from backend.api.security.ownership import OwnershipNotFoundError, ownership_actor_scope +from backend.api.security.local_pairing import LocalPairingError +from backend.api.security.oidc import OIDCValidationError + logger = structlog.get_logger() -# Endpoints that don't require authentication. /metrics is included so the -# Prometheus scrape can run without minting an API key; if you need to gate -# scrape access, terminate that at the proxy or set METRICS_SCRAPE_TOKEN and -# enforce it in the metrics route handler. -PUBLIC_PATHS = {"/health", "/docs", "/redoc", "/openapi.json", "/metrics"} +# Liveness/readiness stay unauthenticated so local/process supervisors can probe +# the service. Docs, OpenAPI, and Prometheus metrics are public only for +# dev-safe profiles or when auth is disabled by the caller. +ALWAYS_PUBLIC_PATHS = { + "/health", + "/health/live", + "/health/ready", + "/auth/oidc/config", + "/auth/local/pair", +} +DIAGNOSTIC_PUBLIC_PATHS = {"/docs", "/redoc", "/openapi.json", "/metrics"} + + +def normalize_public_path(path: str) -> str: + return path.rstrip("/") or "/" + + +def is_public_path(path: str, diagnostic_paths_public: bool = False) -> bool: + normalized = normalize_public_path(path) + return normalized in ALWAYS_PUBLIC_PATHS or (diagnostic_paths_public and normalized in DIAGNOSTIC_PUBLIC_PATHS) class AuthMiddleware(BaseHTTPMiddleware): """Simple API key authentication middleware for internal use.""" - def __init__(self, app, api_key: str, enabled: bool = True, static_key_admin: bool = True): + def __init__( + self, + app, + api_key: str, + enabled: bool = True, + static_key_admin: bool = True, + diagnostic_paths_public: bool = False, + oidc_authenticator: Any | None = None, + local_pairing_authenticator: Any | None = None, + ): super().__init__(app) self.api_key = api_key self.enabled = enabled self.static_key_admin = static_key_admin + self.diagnostic_paths_public = diagnostic_paths_public + self.oidc_authenticator = oidc_authenticator + self.local_pairing_authenticator = local_pairing_authenticator async def dispatch(self, request: Request, call_next): - # Skip auth if disabled (dev mode) + # Skip auth if disabled (dev mode), but still hydrate actor context + # when callers send a valid key so policy gates can be tested locally. if not self.enabled: - return await call_next(request) + bearer_token = self._bearer_token(request) + if bearer_token and ( + self.oidc_authenticator is not None or self.local_pairing_authenticator is not None + ): + try: + request.state.enterprise_user = await self._authenticate_bearer_token(bearer_token) + except (OIDCValidationError, LocalPairingError): + pass + api_key = request.headers.get("X-API-Key") + if api_key and not getattr(request.state, "enterprise_user", None): + self._attach_actor_from_api_key(request, api_key) + return await self._continue_with_actor(request, call_next) # Skip auth for public endpoints - path = request.url.path.rstrip("/") - if path in PUBLIC_PATHS: - return await call_next(request) + path = normalize_public_path(request.url.path) + if is_public_path(path, self.diagnostic_paths_public): + return await self._continue_with_actor(request, call_next) - # Skip auth for WebSocket (handled separately via query param) + # Skip auth for WebSocket (handled separately by the route). if request.scope.get("type") == "websocket": return await call_next(request) + # A presented bearer token is authoritative: an invalid token never + # falls through to an API key supplied alongside it. + authorization = request.headers.get("Authorization") + if authorization: + bearer_token = self._bearer_token(request) + if bearer_token is None: + return self._auth_failed("Invalid bearer authentication.") + try: + request.state.enterprise_user = await self._authenticate_bearer_token(bearer_token) + except (OIDCValidationError, LocalPairingError): + logger.warning( + "bearer_auth_failed", + path=path, + ip=request.client.host if request.client else "unknown", + ) + return self._auth_failed("Invalid bearer authentication.") + return await self._continue_with_actor(request, call_next) + # Query-string API keys leak into browser history, proxy logs, and - # referrers; HTTP auth only accepts the header form. + # referrers; service authentication only accepts the header form. api_key = request.headers.get("X-API-Key") if not api_key: logger.warning("auth_missing_key", path=path, ip=request.client.host if request.client else "unknown") - return JSONResponse( - status_code=401, - content={ - "error": { - "code": "AUTH_FAILED", - "message": "Missing API key. Provide X-API-Key header.", - } - }, - ) + return self._auth_failed("Missing credentials. Provide a bearer token or X-API-Key header.") # Constant-time comparison to prevent timing attacks. The static key # only auto-promotes the caller to admin when the dev-loop flag is on; @@ -77,48 +128,95 @@ async def dispatch(self, request: Request, call_next): "role": "admin", "auth_source": "static_api_key", } - return await call_next(request) + return await self._continue_with_actor(request, call_next) logger.warning( "static_key_rejected", path=path, ip=request.client.host if request.client else "unknown", ) - return JSONResponse( - status_code=401, - content={ - "error": { - "code": "AUTH_FAILED", - "message": "Static API key is not accepted; use an enterprise API key.", - } - }, - ) + return self._auth_failed("Static API key is not accepted; use an enterprise API key.") enterprise_user = None try: from backend.storage import get_session_store + enterprise_user = get_session_store().get_enterprise_user_by_api_key(api_key) except Exception as exc: logger.debug("enterprise_key_lookup_failed", error=str(exc)) if not enterprise_user: logger.warning("auth_invalid_key", path=path, ip=request.client.host if request.client else "unknown") - return JSONResponse( - status_code=401, - content={ - "error": { - "code": "AUTH_FAILED", - "message": "Invalid API key.", - } - }, - ) + return self._auth_failed("Invalid API key.") + + enterprise_user["auth_source"] = "enterprise_api_key" + request.state.enterprise_user = enterprise_user + return await self._continue_with_actor(request, call_next) + + @staticmethod + def _bearer_token(request: Request) -> str | None: + authorization = (request.headers.get("Authorization") or "").strip() + scheme, separator, credential = authorization.partition(" ") + if not separator or scheme.lower() != "bearer": + return None + token = credential.strip() + return token or None + + @staticmethod + def _auth_failed(message: str) -> JSONResponse: + return JSONResponse( + status_code=401, + content={"error": {"code": "AUTH_FAILED", "message": message}}, + headers={"WWW-Authenticate": "Bearer"}, + ) + + @staticmethod + async def _continue_with_actor(request: Request, call_next): + actor = getattr(request.state, "enterprise_user", None) or getattr(request.state, "user", None) + with ownership_actor_scope(actor): + try: + return await call_next(request) + except OwnershipNotFoundError: + return JSONResponse(status_code=404, content={"detail": "Resource not found"}) + + def _attach_actor_from_api_key(self, request: Request, api_key: str) -> bool: + if hmac.compare_digest(api_key, self.api_key): + if self.static_key_admin: + request.state.enterprise_user = { + "user_id": "admin", + "email": "admin@localhost", + "name": "Admin", + "role": "admin", + "auth_source": "static_api_key", + } + return True + return False + try: + from backend.storage import get_session_store + + enterprise_user = get_session_store().get_enterprise_user_by_api_key(api_key) + except Exception as exc: + logger.debug("enterprise_key_lookup_failed", error=str(exc)) + return False + + if not enterprise_user: + return False enterprise_user["auth_source"] = "enterprise_api_key" request.state.enterprise_user = enterprise_user - return await call_next(request) + return True + + async def _authenticate_bearer_token(self, token: str) -> Any: + if token.startswith("local."): + if self.local_pairing_authenticator is None: + raise LocalPairingError("LOCAL_TOKEN_INVALID") + return self.local_pairing_authenticator.authenticate(token) + if self.oidc_authenticator is None: + raise OIDCValidationError("OIDC bearer authentication is unavailable") + return await self.oidc_authenticator.authenticate(token) -def validate_ws_api_key(api_key_param: Optional[str], expected_key: str) -> bool: - """Validate API key for WebSocket connections (passed as query param).""" - if not api_key_param: +def validate_ws_api_key(api_key: Optional[str], expected_key: str) -> bool: + """Validate API key for WebSocket connections.""" + if not api_key: return False - return hmac.compare_digest(api_key_param, expected_key) + return hmac.compare_digest(api_key, expected_key) diff --git a/backend/api/middleware/drain.py b/backend/api/middleware/drain.py new file mode 100644 index 0000000..aa439e5 --- /dev/null +++ b/backend/api/middleware/drain.py @@ -0,0 +1,152 @@ +"""Process-wide graceful-drain admission control for HTTP requests.""" + +from __future__ import annotations + +import asyncio +import os +import threading +import time +from dataclasses import dataclass + +from starlette.responses import JSONResponse +from starlette.types import ASGIApp, Receive, Scope, Send + + +_PROBE_PATHS = frozenset({"/health", "/health/live", "/health/ready", "/metrics"}) + + +@dataclass(frozen=True) +class DrainSnapshot: + draining: bool + in_flight: int + + +class DrainController: + """Thread-safe admission and bounded in-flight join state.""" + + def __init__(self) -> None: + self._condition = threading.Condition() + self._draining = False + self._in_flight = 0 + + @staticmethod + def is_probe_path(path: str) -> bool: + return (str(path or "").rstrip("/") or "/") in _PROBE_PATHS + + def snapshot(self) -> DrainSnapshot: + with self._condition: + return DrainSnapshot(draining=self._draining, in_flight=self._in_flight) + + def start_accepting(self) -> None: + with self._condition: + self._draining = False + in_flight = self._in_flight + self._condition.notify_all() + self._publish_state(False, in_flight) + + def begin_drain(self) -> bool: + with self._condition: + if self._draining: + return False + self._draining = True + in_flight = self._in_flight + self._condition.notify_all() + self._publish_state(True, in_flight) + return True + + def try_acquire(self, path: str) -> bool: + if self.is_probe_path(path): + return True + with self._condition: + if self._draining: + return False + self._in_flight += 1 + in_flight = self._in_flight + self._publish_in_flight(in_flight) + return True + + def release(self) -> None: + with self._condition: + if self._in_flight <= 0: + raise RuntimeError("Drain in-flight counter underflow") + self._in_flight -= 1 + in_flight = self._in_flight + if in_flight == 0: + self._condition.notify_all() + self._publish_in_flight(in_flight) + + async def wait_for_idle(self, timeout_seconds: float) -> bool: + timeout = max(float(timeout_seconds), 0.0) + return await asyncio.to_thread(self._wait_for_idle_blocking, timeout) + + def _wait_for_idle_blocking(self, timeout_seconds: float) -> bool: + deadline = time.monotonic() + timeout_seconds + with self._condition: + while self._in_flight: + remaining = deadline - time.monotonic() + if remaining <= 0: + return False + self._condition.wait(remaining) + return True + + @staticmethod + def _publish_state(draining: bool, in_flight: int) -> None: + from backend.observability import set_in_flight_requests, set_service_draining + + set_service_draining(draining) + set_in_flight_requests(in_flight) + + @staticmethod + def _publish_in_flight(in_flight: int) -> None: + from backend.observability import set_in_flight_requests + + set_in_flight_requests(in_flight) + + +class DrainMiddleware: + """Reject new HTTP and WebSocket work after drain mode begins.""" + + def __init__(self, app: ASGIApp, *, controller: DrainController) -> None: + self.app = app + self.controller = controller + + async def __call__(self, scope: Scope, receive: Receive, send: Send) -> None: + scope_type = scope["type"] + if scope_type not in {"http", "websocket"}: + await self.app(scope, receive, send) + return + path = str(scope.get("path") or "") + is_probe = self.controller.is_probe_path(path) + if not self.controller.try_acquire(path): + if scope_type == "websocket": + await send({"type": "websocket.close", "code": 1013, "reason": "Service is draining"}) + else: + response = JSONResponse( + status_code=503, + content={ + "error": { + "code": "SERVICE_DRAINING", + "message": "Service is draining; retry on another instance.", + } + }, + headers={"Retry-After": "1", "Cache-Control": "no-store"}, + ) + await response(scope, receive, send) + return + try: + await self.app(scope, receive, send) + finally: + if not is_probe: + self.controller.release() + + +def graceful_shutdown_timeout_seconds() -> float: + raw = os.getenv("GRACEFUL_SHUTDOWN_TIMEOUT_SECONDS", "15") + try: + configured = float(raw) + except (TypeError, ValueError): + return 15.0 + return min(max(configured, 0.1), 300.0) + + +drain_controller = DrainController() diff --git a/backend/api/middleware/error_handler.py b/backend/api/middleware/error_handler.py index 03f0952..5ef1cc2 100644 --- a/backend/api/middleware/error_handler.py +++ b/backend/api/middleware/error_handler.py @@ -2,16 +2,18 @@ from __future__ import annotations +import re import secrets import time -import traceback import structlog +from structlog.contextvars import bound_contextvars from starlette.middleware.base import BaseHTTPMiddleware from starlette.requests import Request from starlette.responses import JSONResponse -from backend.config import PRODUCTION_MODE +from backend.observability import route_template_from_scope +from backend.operations.work_admission import WorkBudgetError, WorkExecutionTimeout logger = structlog.get_logger() @@ -23,9 +25,34 @@ 400: "Invalid request payload.", 403: "Forbidden.", 404: "Resource not found.", + 503: "Worker capacity is unavailable; retry shortly.", 504: "Upstream operation timed out.", 500: "Internal server error.", } +_REQUEST_ID_RE = re.compile(r"^[A-Za-z0-9][A-Za-z0-9._:-]{0,63}$") + + +def _request_id_from_header(value: str | None) -> str: + candidate = (value or "").strip() + if candidate and _REQUEST_ID_RE.fullmatch(candidate): + return candidate + return secrets.token_urlsafe(18) + + +def _classify_exception(exc: Exception) -> tuple[int, str]: + if isinstance(exc, WorkBudgetError): + return 503, exc.code + if isinstance(exc, WorkExecutionTimeout): + return 504, exc.code + if isinstance(exc, ValueError): + return 400, "INVALID_INPUT" + if isinstance(exc, FileNotFoundError): + return 404, "NOT_FOUND" + if isinstance(exc, PermissionError): + return 403, "FORBIDDEN" + if isinstance(exc, TimeoutError): + return 504, "TIMEOUT" + return 500, "INTERNAL_ERROR" class ErrorHandlerMiddleware(BaseHTTPMiddleware): @@ -35,66 +62,50 @@ class ErrorHandlerMiddleware(BaseHTTPMiddleware): """ async def dispatch(self, request: Request, call_next): - request_id = secrets.token_urlsafe(9) + request_id = _request_id_from_header(request.headers.get("X-Request-ID")) + request.state.request_id = request_id start_time = time.monotonic() - try: - response = await call_next(request) - elapsed = time.monotonic() - start_time - - # Inject request ID and timing into response headers - response.headers["X-Request-ID"] = request_id - response.headers["X-Response-Time"] = f"{elapsed:.3f}s" - - return response - - except Exception as exc: - elapsed = time.monotonic() - start_time - logger.error( - "unhandled_exception", - request_id=request_id, - path=request.url.path, - method=request.method, - error=str(exc), - elapsed=f"{elapsed:.3f}s", - traceback=traceback.format_exc(), - ) - - # Map common exceptions to appropriate status codes - status_code = 500 - error_code = "INTERNAL_ERROR" - - if isinstance(exc, ValueError): - status_code = 400 - error_code = "INVALID_INPUT" - elif isinstance(exc, FileNotFoundError): - status_code = 404 - error_code = "NOT_FOUND" - elif isinstance(exc, PermissionError): - status_code = 403 - error_code = "FORBIDDEN" - elif isinstance(exc, TimeoutError): - status_code = 504 - error_code = "TIMEOUT" - - # Never echo raw exception text in production; the request_id is - # sufficient for operators to look up the structured log entry. - if PRODUCTION_MODE: - message = _GENERIC_MESSAGES.get(status_code, _GENERIC_MESSAGES[500]) - else: - message = str(exc) - - return JSONResponse( - status_code=status_code, - content={ - "error": { - "code": error_code, - "message": message, - "request_id": request_id, - } - }, - headers={ - "X-Request-ID": request_id, - "X-Response-Time": f"{elapsed:.3f}s", - }, - ) + with bound_contextvars(request_id=request_id): + try: + response = await call_next(request) + elapsed = time.monotonic() - start_time + + response.headers["X-Request-ID"] = request_id + response.headers["X-Response-Time"] = f"{elapsed:.3f}s" + logger.info( + "http_request_completed", + route=route_template_from_scope(request.scope), + method=request.method, + status_code=response.status_code, + elapsed_ms=round(elapsed * 1000, 3), + ) + + return response + + except Exception as exc: + elapsed = time.monotonic() - start_time + status_code, error_code = _classify_exception(exc) + logger.error( + "unhandled_exception", + route=route_template_from_scope(request.scope), + method=request.method, + error_type=exc.__class__.__name__, + error_code=error_code, + elapsed_ms=round(elapsed * 1000, 3), + ) + + return JSONResponse( + status_code=status_code, + content={ + "error": { + "code": error_code, + "message": _GENERIC_MESSAGES.get(status_code, _GENERIC_MESSAGES[500]), + "request_id": request_id, + } + }, + headers={ + "X-Request-ID": request_id, + "X-Response-Time": f"{elapsed:.3f}s", + }, + ) diff --git a/backend/api/middleware/rate_limiter.py b/backend/api/middleware/rate_limiter.py index c9ea105..6e35ec3 100644 --- a/backend/api/middleware/rate_limiter.py +++ b/backend/api/middleware/rate_limiter.py @@ -10,6 +10,8 @@ from starlette.requests import Request from starlette.responses import JSONResponse +from backend.api.middleware.auth import is_public_path, normalize_public_path + logger = structlog.get_logger() @@ -27,13 +29,18 @@ def __init__( enabled: bool = True, endpoint_limits: dict[str, int] | None = None, trusted_proxy_headers: bool = False, + prune_interval_seconds: float = 60.0, + diagnostic_paths_public: bool = False, ): super().__init__(app) self.default_rpm = default_rpm self.enabled = enabled self.endpoint_limits = endpoint_limits or {} self.trusted_proxy_headers = trusted_proxy_headers + self.diagnostic_paths_public = diagnostic_paths_public self._requests: dict[str, list[float]] = defaultdict(list) + self._prune_interval = prune_interval_seconds + self._last_prune = 0.0 def _client_ip(self, request: Request) -> str: # X-Forwarded-For is only honored when the operator has set @@ -75,7 +82,7 @@ def _principal(self, request: Request) -> str: def _get_client_key(self, request: Request) -> str: """Build rate limit key from principal + normalized path bucket.""" principal = self._principal(request) - path = request.url.path.rstrip("/") or "/" + path = normalize_public_path(request.url.path) bucket = path for prefix in self.endpoint_limits: if path.startswith(prefix): @@ -89,13 +96,29 @@ def _clean_window(self, key: str, window_start: float): ts for ts in self._requests[key] if ts > window_start ] + def _maybe_prune_all(self, now: float, window_start: float) -> None: + """Periodically drop keys whose timestamps are entirely outside the + window. Without this, the dict grows unboundedly with unique IPs/users + even when most are long inactive (memory leak in long-running prod).""" + if now - self._last_prune < self._prune_interval: + return + self._last_prune = now + stale = [key for key, timestamps in self._requests.items() if not any(ts > window_start for ts in timestamps)] + for key in stale: + self._requests.pop(key, None) + if stale: + logger.debug("rate_limiter_pruned", removed=len(stale), remaining=len(self._requests)) + async def dispatch(self, request: Request, call_next): if not self.enabled: return await call_next(request) - # Skip rate limiting for health, docs, and Prometheus scrape so - # observability never gets starved by application traffic. - if request.url.path.rstrip("/") in {"/health", "/docs", "/redoc", "/openapi.json", "/metrics"}: + path = normalize_public_path(request.url.path) + # Keep public diagnostics reachable in dev-safe profiles, but do not + # grant production-protected diagnostics a special bypass. Local + # pairing is public for authentication purposes but remains strictly + # rate limited because it consumes a one-time credential. + if path != "/auth/local/pair" and is_public_path(path, self.diagnostic_paths_public): return await call_next(request) now = time.time() @@ -103,7 +126,6 @@ async def dispatch(self, request: Request, call_next): key = self._get_client_key(request) # Get the applicable rate limit - path = request.url.path.rstrip("/") rpm = self.default_rpm for prefix, limit in self.endpoint_limits.items(): if path.startswith(prefix): @@ -112,6 +134,7 @@ async def dispatch(self, request: Request, call_next): # Clean old entries and check self._clean_window(key, window_start) + self._maybe_prune_all(now, window_start) if len(self._requests[key]) >= rpm: retry_after = int(self._requests[key][0] - window_start) + 1 @@ -122,6 +145,9 @@ async def dispatch(self, request: Request, call_next): requests_in_window=len(self._requests[key]), limit=rpm, ) + headers = {"Retry-After": str(max(retry_after, 1))} + if path == "/auth/local/pair": + headers.update({"Cache-Control": "no-store", "Pragma": "no-cache"}) return JSONResponse( status_code=429, content={ @@ -131,7 +157,7 @@ async def dispatch(self, request: Request, call_next): "retry_after": max(retry_after, 1), } }, - headers={"Retry-After": str(max(retry_after, 1))}, + headers=headers, ) # Record this request @@ -145,7 +171,7 @@ def get_stats(self) -> dict: window_start = now - 60 active_clients = 0 total_requests = 0 - for key, timestamps in self._requests.items(): + for _key, timestamps in self._requests.items(): recent = [ts for ts in timestamps if ts > window_start] if recent: active_clients += 1 diff --git a/backend/api/routes/agent_runs.py b/backend/api/routes/agent_runs.py index fa8794e..c590b10 100644 --- a/backend/api/routes/agent_runs.py +++ b/backend/api/routes/agent_runs.py @@ -2,12 +2,16 @@ from __future__ import annotations +import hashlib +import json +import os import uuid import time -from typing import Any, Optional +from typing import Any, Literal, Optional import structlog -from fastapi import APIRouter, HTTPException +from fastapi import APIRouter, HTTPException, Query, Request +from fastapi.responses import StreamingResponse from pydantic import BaseModel, Field from backend.models.operations import AgentRun, AgentRunV3, AgentRunV4, AgentRunStepState, AuditExportV2, RepairPlan @@ -18,6 +22,11 @@ logger = structlog.get_logger() +class AgentEgressApprovalRequest(BaseModel): + approval_id: str = Field(..., min_length=1, max_length=128) + approved_digest: str = Field(..., min_length=64, max_length=64) + + class AgentRunCreateRequest(BaseModel): prompt: str = Field(..., min_length=1, max_length=10000) session_id: Optional[str] = None @@ -30,18 +39,58 @@ class AgentRunCreateRequest(BaseModel): validation_profile: Optional[str] = None accuracy_mode: str = "deep" approval_policy: str = "checkpoint_high_risk" + egress_approval: Optional[AgentEgressApprovalRequest] = None class AgentCheckpointApprovalRequest(BaseModel): checkpoint_id: str + expected_event_sequence: Optional[int] = Field(default=None, ge=0) + idempotency_key: Optional[str] = Field(default=None, min_length=1, max_length=200) + + +class AgentRunRecoveryResultRequest(BaseModel): + recovery_kind: Literal["rollback", "undo"] + status: Literal["rolled_back", "failed"] + attempt_id: Optional[str] = None + office_apply_result: dict[str, Any] = Field(default_factory=dict) + pre_apply_workbook_state: dict[str, Any] = Field(default_factory=dict) + pre_apply_workbook_fingerprint: Optional[str] = None + pre_host_snapshot: dict[str, Any] = Field(default_factory=dict) + post_apply_workbook_state: dict[str, Any] = Field(default_factory=dict) + post_apply_workbook_fingerprint: Optional[str] = None + post_host_snapshot: dict[str, Any] = Field(default_factory=dict) + error_detail: Optional[str] = Field(default=None, max_length=4000) + + +class AgentRunRecoveryResultResponse(BaseModel): + run_id: str + recovery_kind: Literal["rollback", "undo"] + attempt_id: str + status: Literal["rolled_back", "failed"] + audit_event_id: str + verification: dict[str, Any] = Field(default_factory=dict) + recorded_at: float @router.post("/agent-runs", response_model=AgentRun) -async def create_agent_run(request: AgentRunCreateRequest): +async def create_agent_run(request: AgentRunCreateRequest, http_request: Request): """Create a flagged accuracy-first agent run above the V3 plan lifecycle.""" - from backend.api.routes.tasks import _collect_plan_citations, _save_operation_plan_with_v3 + from backend.api.routes.tasks import ( + _claim_request_session, + _collect_plan_citations, + _enforce_ai_policy, + _request_actor, + _save_operation_plan_with_v3, + ) session_id = request.session_id or str(uuid.uuid4())[:12] + _enforce_ai_policy( + "preview", + request.provider, + request.source_document_ids, + _request_actor(http_request), + ) + _claim_request_session(http_request, session_id) store = get_session_store() documents = [] for document_id in request.source_document_ids: @@ -115,13 +164,101 @@ async def create_agent_run(request: AgentRunCreateRequest): return run +@router.post("/agent-runs/v4/egress-preview") +async def create_agent_run_v4_egress_preview( + request: AgentRunCreateRequest, + http_request: Request, +): + """Issue a short-lived, one-use approval preview without contacting a provider.""" + from backend.api.routes.tasks import ( + _claim_request_session, + _enforce_ai_policy, + _request_actor, + ) + from backend.api.security.ai_policy import CLOUD_PROVIDERS + from backend.services.agent_run_control import ( + build_egress_preview, + classify_egress_data, + egress_request_digest, + ) + + provider = str(request.provider or "").strip().lower() + if provider not in CLOUD_PROVIDERS: + raise HTTPException( + status_code=400, + detail={ + "code": "cloud_provider_required", + "message": "An explicit cloud provider is required for an egress preview.", + }, + ) + session_id = request.session_id or str(uuid.uuid4())[:12] + _claim_request_session(http_request, session_id) + _enforce_ai_policy( + "preview", + provider, + request.source_document_ids, + _request_actor(http_request), + ) + store = get_session_store() + documents = _load_run_documents(store, session_id, request.source_document_ids) + model = _resolved_provider_model(provider, request.model) + material = _egress_request_material(request, session_id, provider, model, documents) + request_digest = egress_request_digest(material) + data_classes, workbook_ranges, redactions, estimated_tokens = classify_egress_data( + prompt=request.prompt, + workbook_context=request.workbook_context, + documents=documents, + additional_material={"templateParams": request.template_params}, + ) + preview = build_egress_preview( + session_id=session_id, + provider=provider, + model=model, + request_digest=request_digest, + data_classes=data_classes, + workbook_ranges=workbook_ranges, + redactions=redactions, + estimated_input_tokens=estimated_tokens, + ttl_seconds=_egress_approval_ttl_seconds(), + ) + store.save_agent_egress_preview(session_id, preview) + from backend.observability import record_agent_egress_approval + + record_agent_egress_approval("cloud", "issued") + store.record_audit_event( + session_id=session_id, + event_type="agent_run_egress_preview_issued", + payload={ + "approval_id": preview["approvalId"], + "provider_scope": "cloud", + "data_class_count": len(data_classes), + "range_count": len(workbook_ranges), + "expires_at": preview["expiresAt"], + }, + ) + return preview + + @router.post("/agent-runs/v3", response_model=AgentRunV3) -async def create_agent_run_v3(request: AgentRunCreateRequest): +async def create_agent_run_v3(request: AgentRunCreateRequest, http_request: Request): """Create a V6 AgentRunV3 with persisted step-artifact and accuracy metadata.""" - from backend.api.routes.tasks import _collect_plan_citations, _save_operation_plan_with_v3 + from backend.api.routes.tasks import ( + _claim_request_session, + _collect_plan_citations, + _enforce_ai_policy, + _request_actor, + _save_operation_plan_with_v3, + ) from backend.services.agent_runner_v3 import get_agent_run_v3_service session_id = request.session_id or str(uuid.uuid4())[:12] + _enforce_ai_policy( + "preview", + request.provider, + request.source_document_ids, + _request_actor(http_request), + ) + _claim_request_session(http_request, session_id) store = get_session_store() documents = _load_run_documents(store, session_id, request.source_document_ids) try: @@ -194,24 +331,96 @@ async def create_agent_run_v3(request: AgentRunCreateRequest): @router.post("/agent-runs/v4", response_model=AgentRunV4) -async def create_agent_run_v4(request: AgentRunCreateRequest): +async def create_agent_run_v4(request: AgentRunCreateRequest, http_request: Request): """Create a V7 AgentRunV4 with execution groups, rollback snapshots, and audit export.""" - from backend.api.routes.tasks import _collect_plan_citations, _save_operation_plan_with_v3 + from backend.api.routes.tasks import ( + _claim_request_session, + _collect_plan_citations, + _enforce_ai_policy, + _request_actor, + _save_operation_plan_with_v3, + ) + from backend.api.security.ai_policy import AiPolicyViolation, CLOUD_PROVIDERS, get_effective_ai_policy + from backend.api.security.ownership import current_principal + from backend.services.agent_run_control import ( + actor_subject_digest, + build_egress_receipt, + egress_request_digest, + redact_egress_payload, + resolve_no_fallback_provider, + ) from backend.services.agent_runner_v4 import get_agent_run_v4_service session_id = request.session_id or str(uuid.uuid4())[:12] + try: + provider = resolve_no_fallback_provider(get_effective_ai_policy(), request.provider) + except AiPolicyViolation as exc: + raise HTTPException(status_code=403, detail=exc.to_detail()) from exc + _enforce_ai_policy( + "preview", + provider, + request.source_document_ids, + _request_actor(http_request), + ) + _claim_request_session(http_request, session_id) store = get_session_store() documents = _load_run_documents(store, session_id, request.source_document_ids) + model = _resolved_provider_model(provider, request.model) + request_digest = egress_request_digest( + _egress_request_material(request, session_id, provider, model, documents) + ) + approval_id = "" + approval_digest = "" + execution_context = request.workbook_context + execution_documents = documents + execution_template_params = request.template_params + if provider in CLOUD_PROVIDERS: + approval = request.egress_approval + if approval is None: + raise HTTPException( + status_code=409, + detail={ + "code": "egress_approval_required", + "message": "Create an egress preview and approve its exact digest before cloud execution.", + }, + ) + approval_id = approval.approval_id + approval_digest = approval.approved_digest + consumed = store.consume_agent_egress_approval( + session_id, + approval_id=approval_id, + approved_digest=approval_digest, + request_digest=request_digest, + consumption_id=f"agent-run-create:{uuid.uuid4().hex}", + ) + from backend.observability import record_agent_egress_approval + + record_agent_egress_approval("cloud", str(consumed.get("outcome") or "invalid")) + if consumed.get("outcome") != "consumed": + raise _egress_approval_http_error(str(consumed.get("outcome") or "invalid")) + store.record_audit_event( + session_id=session_id, + event_type="agent_run_egress_approval_consumed", + payload={ + "approval_id": approval_id, + "provider_scope": "cloud", + "request_digest": request_digest, + "fallback_allowed": False, + }, + ) + execution_context = redact_egress_payload(request.workbook_context) + execution_documents = redact_egress_payload(documents) + execution_template_params = redact_egress_payload(request.template_params) try: result = await get_agent_run_v4_service().create_run( prompt=request.prompt, session_id=session_id, - workbook_context=request.workbook_context, - documents=documents, - provider=request.provider, - model=request.model, + workbook_context=execution_context, + documents=execution_documents, + provider=provider, + model=model, template_id=request.template_id, - template_params=request.template_params, + template_params=execution_template_params, validation_profile=request.validation_profile, accuracy_mode=request.accuracy_mode, approval_policy=request.approval_policy, @@ -230,6 +439,32 @@ async def create_agent_run_v4(request: AgentRunCreateRequest): v3_plan = result["v3_plan"] truth = result["truth_model_v5"] audit = result["audit_export_v2"] + principal = current_principal() + receipt = build_egress_receipt( + provider=provider, + model=model, + request_digest=request_digest, + approval_id=approval_id, + approval_digest=approval_digest, + actor_digest=actor_subject_digest(principal.user_id), + provider_call_attempted=str((legacy_plan.metadata or {}).get("planner") or "") + == "react_tool_loop", + ) + run_payload = run.model_dump(mode="json", by_alias=True) + run_metadata = dict(run_payload.get("metadata") or {}) + run_metadata["providerRoute"] = { + "provider": provider, + "model": model, + "scope": receipt["scope"], + "fallbackAllowed": False, + "cloudFallbackAllowed": False, + } + run_payload["metadata"] = run_metadata + run_payload["artifacts"] = { + **(run_payload.get("artifacts") or {}), + "egress_receipt_v1": receipt, + } + run = AgentRunV4.model_validate(run_payload) store.add_message(session_id, "user", request.prompt) store.save_source_evidence_graph(source_graph.model_dump(mode="json", by_alias=True)) @@ -257,6 +492,22 @@ async def create_agent_run_v4(request: AgentRunCreateRequest): for snapshot in run.rollback_snapshots: store.save_agent_run_artifact(run.run_id, session_id, "rollback_snapshot_v5", snapshot.model_dump(mode="json", by_alias=True)) store.save_agent_run_artifact(run.run_id, session_id, "audit_export_v2", audit.model_dump(mode="json", by_alias=True)) + control_result = store.append_agent_run_event( + run.run_id, + expected_sequence=0, + event_type="run_created", + event_payload={ + "status": run.status, + "providerScope": receipt["scope"], + "egressReceiptId": receipt["receiptId"], + }, + updated_run=run.model_dump(mode="json", by_alias=True), + idempotency_key=f"create:{run.idempotency_key or request_digest}", + ) + _record_agent_run_control("created", control_result) + if control_result.get("outcome") not in {"appended", "replayed"}: + raise _agent_run_control_http_error(control_result) + run = AgentRunV4.model_validate(control_result["run"]) store.record_audit_event( session_id=session_id, plan_id=v3_plan.id, @@ -280,6 +531,65 @@ async def get_agent_run(run_id: str): return AgentRun.model_validate(run) +@router.get("/agent-runs/{run_id}/events") +async def stream_agent_run_events( + run_id: str, + request: Request, + after_sequence: int = Query(default=0, ge=0), + limit: int = Query(default=25, ge=1, le=100), +): + """Return one bounded SSE replay page; reconnect with Last-Event-ID for the next page.""" + store = get_session_store() + if not store.get_agent_run(run_id): + raise HTTPException(status_code=404, detail="Agent run not found") + header_cursor = str(request.headers.get("last-event-id") or "").strip() + if header_cursor: + try: + after_sequence = max(after_sequence, int(header_cursor)) + except ValueError as exc: + raise HTTPException( + status_code=400, + detail={ + "code": "agent_run_cursor_invalid", + "message": "Last-Event-ID must be a non-negative event sequence.", + }, + ) from exc + if after_sequence < 0: + raise HTTPException( + status_code=400, + detail={ + "code": "agent_run_cursor_invalid", + "message": "Last-Event-ID must be a non-negative event sequence.", + }, + ) + replay = store.list_agent_run_events( + run_id, + after_sequence=after_sequence, + limit=limit, + ) + from backend.observability import record_agent_run_sse_replay + + record_agent_run_sse_replay(len(replay["events"]), has_more=bool(replay["hasMore"])) + chunks = [_format_agent_run_sse_event(event) for event in replay["events"]] + if not chunks: + chunks = [f": replay-complete cursor={replay['nextCursor']}\n\n"] + + async def replay_page(): + for chunk in chunks: + yield chunk + + return StreamingResponse( + replay_page(), + media_type="text/event-stream", + headers={ + "Cache-Control": "no-cache, no-transform", + "X-Accel-Buffering": "no", + "X-Agent-Run-Cursor": str(replay["nextCursor"]), + "X-Agent-Run-Has-More": str(bool(replay["hasMore"])).lower(), + }, + ) + + @router.get("/agent-runs/{run_id}/artifacts") async def get_agent_run_artifacts(run_id: str): """Return durable run artifacts plus the embedded compatibility artifact map.""" @@ -357,6 +667,15 @@ async def rollback_agent_run(run_id: str): if not payload: raise HTTPException(status_code=404, detail="Agent run not found") rollback = _rollback_response_for_run(store, payload) + attempt = _begin_agent_recovery(store, payload, "rollback", rollback) + if attempt: + rollback.update( + { + "attempt_id": attempt["attemptId"], + "attemptId": attempt["attemptId"], + "recovery_status": attempt["status"], + } + ) store.record_audit_event( session_id=payload.get("sessionId") or payload.get("session_id") or "", plan_id=payload.get("planId") or payload.get("plan_id"), @@ -391,6 +710,15 @@ async def undo_agent_run(run_id: str): if not payload: raise HTTPException(status_code=404, detail="Agent run not found") rollback = _rollback_response_for_run(store, payload) + attempt = _begin_agent_recovery(store, payload, "undo", rollback) + if attempt: + rollback.update( + { + "attempt_id": attempt["attemptId"], + "attemptId": attempt["attemptId"], + "recovery_status": attempt["status"], + } + ) store.record_audit_event( session_id=payload.get("sessionId") or payload.get("session_id") or "", plan_id=payload.get("planId") or payload.get("plan_id"), @@ -400,6 +728,117 @@ async def undo_agent_run(run_id: str): return rollback +@router.post( + "/agent-runs/{run_id}/rollback-result", + response_model=AgentRunRecoveryResultResponse, +) +async def record_agent_run_recovery_result( + run_id: str, + request: AgentRunRecoveryResultRequest, +) -> AgentRunRecoveryResultResponse: + """Record the Office host result before reporting agent recovery success.""" + store = get_session_store() + payload = store.get_agent_run(run_id) + if not payload: + raise HTTPException(status_code=404, detail="Agent run not found") + metadata = dict(payload.get("metadata") or {}) + attempts = [item for item in metadata.get("recoveryAttempts") or [] if isinstance(item, dict)] + if not request.attempt_id: + raise HTTPException(status_code=409, detail="An exact agent-run recovery attempt id is required") + matching = [ + item + for item in attempts + if item.get("recoveryKind") == request.recovery_kind + and item.get("attemptId") == request.attempt_id + ] + if not matching: + raise HTTPException(status_code=409, detail="No matching agent-run recovery attempt exists") + attempt = matching[-1] + if attempt.get("status") != "rollback_pending": + if attempt.get("status") == request.status: + return AgentRunRecoveryResultResponse( + run_id=run_id, + recovery_kind=request.recovery_kind, + attempt_id=str(attempt.get("attemptId") or ""), + status=request.status, + audit_event_id=str(attempt.get("auditEventId") or ""), + verification=attempt.get("verification") or {}, + recorded_at=float(attempt.get("recordedAt") or payload.get("updatedAt") or time.time()), + ) + raise HTTPException(status_code=409, detail="Agent-run recovery attempt is already finalized") + + expected_operations = [item for item in attempt.get("approvedOperations") or [] if isinstance(item, dict)] + verification = _verify_agent_recovery_result( + expected_operations, + request.office_apply_result, + request.pre_apply_workbook_state, + request.pre_apply_workbook_fingerprint, + request.pre_host_snapshot, + request.post_apply_workbook_state, + request.post_apply_workbook_fingerprint, + request.post_host_snapshot, + attempt=attempt, + ) + if request.status == "rolled_back" and verification["status"] != "verified": + raise HTTPException( + status_code=409, + detail={ + "code": "agent_recovery_ack_incomplete", + "message": "Office did not explicitly acknowledge every recovery operation as successful.", + "verification": verification, + }, + ) + + recorded_at = time.time() + attempt.update( + { + "status": request.status, + "verification": verification, + "officeApplyResult": request.office_apply_result, + "errorDetail": request.error_detail, + "recordedAt": recorded_at, + } + ) + metadata["recoveryAttempts"] = attempts[-25:] + metadata["lastRecoveryResult"] = attempt + payload["metadata"] = metadata + payload["updatedAt"] = recorded_at + payload.setdefault("logs", []).append( + f"{request.recovery_kind} result recorded: {request.status}" + ) + store.save_agent_run(payload) + audit_id = store.record_audit_event( + session_id=payload.get("sessionId") or payload.get("session_id") or "", + plan_id=payload.get("planId") or payload.get("plan_id"), + event_type=f"agent_run_{request.recovery_kind}_{request.status}", + payload={ + "run_id": run_id, + "attempt_id": attempt.get("attemptId"), + "verification": verification, + "error_detail": request.error_detail, + }, + ) + attempt["auditEventId"] = audit_id + store.save_agent_run(payload) + response = AgentRunRecoveryResultResponse( + run_id=run_id, + recovery_kind=request.recovery_kind, + attempt_id=str(attempt.get("attemptId") or ""), + status=request.status, + audit_event_id=audit_id, + verification=verification, + recorded_at=recorded_at, + ) + from backend.services.retention import purge_terminal_session_if_immediate + + if request.status == "rolled_back" and verification.get("status") == "verified": + await purge_terminal_session_if_immediate( + str(payload.get("sessionId") or payload.get("session_id") or ""), + request.status, + ) + return response + + @router.get("/audit/agent-runs/{run_id}/export", response_model=AuditExportV2) async def export_agent_run_audit(run_id: str): store = get_session_store() @@ -421,39 +860,70 @@ async def export_agent_run_audit(run_id: str): @router.post("/agent-runs/{run_id}/pause", response_model=AgentRun) -async def pause_agent_run(run_id: str): - return _transition_run(run_id, "pause") +async def pause_agent_run(run_id: str, http_request: Request): + return _transition_run(run_id, "pause", http_request) @router.post("/agent-runs/{run_id}/resume", response_model=AgentRun) -async def resume_agent_run(run_id: str): +async def resume_agent_run(run_id: str, http_request: Request): store = get_session_store() payload = store.get_agent_run(run_id) if not payload: raise HTTPException(status_code=404, detail="Agent run not found") - pending = [ - step - for step in payload.get("steps") or [] - if isinstance(step, dict) and step.get("status") == "pending" and step.get("type") not in {"apply", "finalize"} - ] - for step in pending[:3]: - payload, _, _ = _recompute_agent_step(store, payload, str(step.get("stepId") or step.get("step_id") or "")) - payload["status"] = "checkpoint_required" if _has_unapproved_checkpoint(payload) else "ready" - payload["updatedAt"] = time.time() - payload.setdefault("logs", []).append("run resumed") - store.save_agent_run(payload) - store.record_audit_event( - session_id=payload.get("sessionId") or payload.get("session_id") or "", - plan_id=payload.get("planId") or payload.get("plan_id"), - event_type="agent_run_resumed", - payload={"run_id": run_id, "status": payload["status"], "recomputed_steps": len(pending[:3])}, + current_status = str(payload.get("status") or "") + if current_status in {"ready", "checkpoint_required", "approved"}: + return AgentRun.model_validate(payload) + if current_status != "paused": + raise HTTPException( + status_code=409, + detail={ + "code": "agent_run_resume_invalid_state", + "message": "Only a paused agent run can be resumed.", + }, + ) + expected_sequence = _expected_agent_run_sequence(payload, http_request) + updated = json.loads(json.dumps(payload)) + updated["status"] = "checkpoint_required" if _has_unapproved_checkpoint(updated) else "ready" + updated.setdefault("logs", []).append("run resumed") + idempotency_key = _agent_run_idempotency_key( + http_request, + action="resume", + run_id=run_id, + expected_sequence=expected_sequence, ) + result = store.append_agent_run_event( + run_id, + expected_sequence=expected_sequence, + event_type="run_resumed", + event_payload={"status": updated["status"], "recomputedSteps": 0}, + updated_run=updated, + idempotency_key=idempotency_key, + ) + _record_agent_run_control("resumed", result) + if result.get("outcome") not in {"appended", "replayed"}: + raise _agent_run_control_http_error(result) + payload = result["run"] + if result.get("outcome") == "appended": + store.record_audit_event( + session_id=payload.get("sessionId") or payload.get("session_id") or "", + plan_id=payload.get("planId") or payload.get("plan_id"), + event_type="agent_run_resumed", + payload={ + "run_id": run_id, + "status": payload["status"], + "event_sequence": result["event"]["sequence"], + }, + ) return AgentRun.model_validate(payload) @router.post("/agent-runs/{run_id}/cancel", response_model=AgentRun) -async def cancel_agent_run(run_id: str): - return _transition_run(run_id, "cancel") +async def cancel_agent_run(run_id: str, http_request: Request): + run = _transition_run(run_id, "cancel", http_request) + from backend.services.retention import purge_terminal_session_if_immediate + + await purge_terminal_session_if_immediate(run.session_id, run.status) + return run @router.post("/agent-runs/{run_id}/approve-checkpoint", response_model=AgentRun) @@ -466,13 +936,36 @@ async def approve_agent_run_checkpoint(run_id: str, request: AgentCheckpointAppr run = get_agent_run_service().approve_checkpoint(AgentRun.model_validate(payload), request.checkpoint_id) except ValueError as exc: raise HTTPException(status_code=404, detail=str(exc)) from exc - store.save_agent_run(run.model_dump(mode="json", by_alias=True)) - store.record_audit_event( - session_id=run.session_id, - plan_id=run.plan_id, - event_type="agent_run_checkpoint_approved", - payload={"run_id": run.run_id, "checkpoint_id": request.checkpoint_id, "status": run.status}, + expected_sequence = ( + request.expected_event_sequence + if request.expected_event_sequence is not None + else _agent_run_event_sequence(payload) ) + result = store.append_agent_run_event( + run_id, + expected_sequence=expected_sequence, + event_type="checkpoint_approved", + event_payload={"checkpointId": request.checkpoint_id, "status": run.status}, + updated_run=run.model_dump(mode="json", by_alias=True), + idempotency_key=request.idempotency_key + or f"approve-checkpoint:{request.checkpoint_id}:{expected_sequence}", + ) + _record_agent_run_control("checkpoint_approved", result) + if result.get("outcome") not in {"appended", "replayed"}: + raise _agent_run_control_http_error(result) + run = AgentRun.model_validate(result["run"]) + if result.get("outcome") == "appended": + store.record_audit_event( + session_id=run.session_id, + plan_id=run.plan_id, + event_type="agent_run_checkpoint_approved", + payload={ + "run_id": run.run_id, + "checkpoint_id": request.checkpoint_id, + "status": run.status, + "event_sequence": result["event"]["sequence"], + }, + ) return run @@ -541,11 +1034,14 @@ def _artifact_for_recomputed_step(payload: dict[str, Any], step_type: str) -> di artifacts = {} if step_type == "simulate": simulation = _recompute_v7_simulation(artifacts, payload) - if simulation: - artifacts["simulation_v6"] = simulation - payload["artifacts"] = artifacts - payload["simulationId"] = simulation.get("simulationId") or payload.get("simulationId") - return {"result": simulation, "status": simulation.get("status", "warning"), "v7GateStatus": simulation.get("v7GateStatus")} + if not simulation: + raise ValueError( + "Simulation retry has no executable operations; fresh inspection is required" + ) + artifacts["simulation_v6"] = simulation + payload["artifacts"] = artifacts + payload["simulationId"] = simulation.get("simulationId") or payload.get("simulationId") + return {"result": simulation, "status": simulation.get("status", "warning"), "v7GateStatus": simulation.get("v7GateStatus")} if step_type == "repair": run, repair = get_agent_run_service().create_repair_plan(AgentRun.model_validate(payload)) payload["artifacts"] = {**artifacts, "repair_plan": repair.model_dump(mode="json", by_alias=True)} @@ -578,14 +1074,22 @@ def _recompute_v7_simulation(artifacts: dict[str, Any], payload: dict[str, Any]) plan = artifacts.get("v3_plan") or artifacts.get("v2_plan") or {} operations_payload = plan.get("operations") if isinstance(plan, dict) else [] operations = [] + invalid_operations: list[str] = [] adapter = TypeAdapter(WorkbookOperationV2) - for item in operations_payload or []: + for index, item in enumerate(operations_payload or []): if not isinstance(item, dict): + invalid_operations.append(f"operation[{index}] must be an object") continue try: operations.append(adapter.validate_python(item)) - except Exception: + except Exception as exc: + invalid_operations.append(f"operation[{index}] failed validation: {exc}") continue + if invalid_operations: + raise ValueError( + "Stored plan contains invalid operations; retry requires fresh inspection: " + + "; ".join(invalid_operations[:5]) + ) truth_payload = artifacts.get("truth_model_v5") or {} truth = WorkbookTruthModelV5.model_validate(truth_payload) if isinstance(truth_payload, dict) and truth_payload else None if not operations: @@ -603,8 +1107,9 @@ def _retry_message(step_type: str, artifact: dict[str, Any]) -> str: if step_type == "simulate": return f"Simulation recomputed; V7 gate {artifact.get('v7GateStatus', 'unknown')}." if step_type == "repair": - result = artifact.get("result") if isinstance(artifact.get("result"), dict) else {} - return f"Repair plan recomputed; status {result.get('status', artifact.get('status', 'suggested'))}." + raw_result = artifact.get("result") + repair_result: dict[str, Any] = raw_result if isinstance(raw_result, dict) else {} + return f"Repair plan recomputed; status {repair_result.get('status', artifact.get('status', 'suggested'))}." return f"{step_type or 'Step'} refreshed from durable local artifacts." @@ -655,6 +1160,238 @@ def _rollback_response_for_run(store: Any, payload: dict[str, Any]) -> dict[str, } +def _begin_agent_recovery( + store: Any, + payload: dict[str, Any], + recovery_kind: Literal["rollback", "undo"], + rollback: dict[str, Any], +) -> dict[str, Any] | None: + """Persist a recovery attempt before Office is allowed to apply it.""" + operations = _ensure_agent_recovery_operation_ids(rollback.get("operations") or []) + if not operations: + return None + rollback["operations"] = operations + operation_digest = hashlib.sha256( + json.dumps(operations, sort_keys=True, separators=(",", ":"), default=str).encode("utf-8") + ).hexdigest() + operation_registry_hash = operation_digest + recovery_policy_hash = hashlib.sha256( + json.dumps( + { + "recoveryKind": recovery_kind, + "operationRegistryHash": operation_registry_hash, + "requiresCompletePreHostRead": True, + "requiresCompletePostHostRead": True, + }, + sort_keys=True, + separators=(",", ":"), + ).encode("utf-8") + ).hexdigest() + metadata = dict(payload.get("metadata") or {}) + attempts = [item for item in metadata.get("recoveryAttempts") or [] if isinstance(item, dict)] + existing = next( + ( + item + for item in reversed(attempts) + if item.get("recoveryKind") == recovery_kind + and item.get("operationDigest") == operation_digest + and item.get("status") == "rollback_pending" + ), + None, + ) + if existing: + return existing + + created_at = time.time() + attempt: dict[str, Any] = { + "attemptId": f"recovery_{uuid.uuid4().hex}", + "recoveryKind": recovery_kind, + "status": "rollback_pending", + "expectedOperationIds": [str(operation["id"]) for operation in operations], + "approvedOperations": operations, + "operationDigest": operation_digest, + "operationRegistryHash": operation_registry_hash, + "recoveryPolicyHash": recovery_policy_hash, + "createdAt": created_at, + } + attempts.append(attempt) + metadata["recoveryAttempts"] = attempts[-25:] + metadata["lastRecoveryAttempt"] = attempt + payload["metadata"] = metadata + payload["updatedAt"] = created_at + payload.setdefault("logs", []).append( + f"{recovery_kind} recovery attempt is pending Office acknowledgement" + ) + store.save_agent_run(payload) + audit_id = store.record_audit_event( + session_id=payload.get("sessionId") or payload.get("session_id") or "", + plan_id=payload.get("planId") or payload.get("plan_id"), + event_type=f"agent_run_{recovery_kind}_pending", + payload={ + "run_id": payload.get("runId") or payload.get("run_id"), + "attempt_id": attempt["attemptId"], + "operation_digest": operation_digest, + "operation_ids": attempt["expectedOperationIds"], + }, + ) + attempt["auditEventId"] = audit_id + store.save_agent_run(payload) + return attempt + + +def _ensure_agent_recovery_operation_ids( + operations: list[dict[str, Any]], +) -> list[dict[str, Any]]: + """Return copied recovery operations with stable, unique host-result IDs.""" + result: list[dict[str, Any]] = [] + seen: set[str] = set() + for index, operation in enumerate(operations): + copied = dict(operation) + candidate = str( + copied.get("id") + or copied.get("operationId") + or copied.get("operation_id") + or "" + ).strip() + if not candidate or candidate in seen: + canonical = json.dumps( + {"index": index, "operation": copied}, + sort_keys=True, + separators=(",", ":"), + default=str, + ) + candidate = f"agent_recovery_{hashlib.sha256(canonical.encode('utf-8')).hexdigest()[:24]}" + copied["id"] = candidate + seen.add(candidate) + result.append(copied) + return result + + +def _verify_agent_recovery_result( + expected_operations: list[dict[str, Any]], + office_apply_result: dict[str, Any], + pre_apply_workbook_state: Optional[dict[str, Any]] = None, + pre_apply_workbook_fingerprint: Optional[str] = None, + pre_host_snapshot: Optional[dict[str, Any]] = None, + post_apply_workbook_state: Optional[dict[str, Any]] = None, + post_apply_workbook_fingerprint: Optional[str] = None, + post_host_snapshot: Optional[dict[str, Any]] = None, + *, + attempt: Optional[dict[str, Any]] = None, +) -> dict[str, Any]: + """Bind Office recovery success to the exact attempt and complete host rereads.""" + from backend.api.routes.tasks import _workbook_target_binding + + failures: list[str] = [] + operations = [dict(item) for item in expected_operations if isinstance(item, dict)] + expected = [str(item.get("id") or "") for item in operations if str(item.get("id") or "")] + attempt = attempt if isinstance(attempt, dict) else {} + if not expected: + failures.append("The recovery attempt contains no expected operation IDs.") + if len(set(expected)) != len(expected): + failures.append("The recovery attempt contains duplicate operation IDs.") + + raw_results = office_apply_result.get("results") + result_rows = raw_results if isinstance(raw_results, list) else [] + if not isinstance(raw_results, list): + failures.append("Office apply result must contain a results array.") + observed: dict[str, list[dict[str, Any]]] = {} + for index, item in enumerate(result_rows): + if not isinstance(item, dict): + failures.append(f"Office result at index {index} is not an object.") + continue + operation_id = str( + item.get("operationId") + or item.get("operation_id") + or item.get("id") + or "" + ).strip() + if not operation_id: + failures.append(f"Office result at index {index} has no operation ID.") + continue + observed.setdefault(operation_id, []).append(item) + + expected_set = set(expected) + unexpected = sorted(set(observed) - expected_set) + if unexpected: + failures.append(f"Office returned unexpected operation IDs: {', '.join(unexpected)}.") + for operation_id in expected: + rows = observed.get(operation_id) or [] + if len(rows) != 1: + failures.append( + f"Operation {operation_id} has {len(rows)} result rows; exactly one is required." + ) + continue + if rows[0].get("status") != "success": + failures.append( + f"Operation {operation_id} was not acknowledged with status success." + ) + + applied = office_apply_result.get("applied") + if isinstance(applied, bool) or not isinstance(applied, int): + failures.append("Office apply result must contain an integer applied count.") + elif applied != len(expected): + failures.append( + f"Office reported {applied} applied operations; {len(expected)} were expected." + ) + warnings = office_apply_result.get("warnings") + if isinstance(warnings, list) and warnings: + failures.append("Office returned warnings for the recovery batch.") + if office_apply_result.get("error") or office_apply_result.get("errors"): + failures.append("Office returned a recovery error.") + + operation_registry_hash = hashlib.sha256( + json.dumps(operations, sort_keys=True, separators=(",", ":"), default=str).encode("utf-8") + ).hexdigest() + if operation_registry_hash != str(attempt.get("operationRegistryHash") or ""): + failures.append("The recovery operation registry does not match the durable attempt.") + recovery_policy_hash = str(attempt.get("recoveryPolicyHash") or "") + if not recovery_policy_hash: + failures.append("The durable recovery policy binding is missing.") + + pre_snapshot = pre_host_snapshot if isinstance(pre_host_snapshot, dict) else {} + post_snapshot = post_host_snapshot if isinstance(post_host_snapshot, dict) else {} + for label, snapshot in (("Pre-apply", pre_snapshot), ("Post-apply", post_snapshot)): + if ( + snapshot.get("complete") is not True + or snapshot.get("truncated") is True + or bool(snapshot.get("errors")) + ): + failures.append(f"{label} full host snapshot is incomplete.") + + pre_state = pre_apply_workbook_state if isinstance(pre_apply_workbook_state, dict) else {} + post_state = post_apply_workbook_state if isinstance(post_apply_workbook_state, dict) else {} + pre_binding = _workbook_target_binding(pre_state, operations) + post_binding = _workbook_target_binding(post_state, operations, require_post_match=True) + failures.extend(f"Pre-apply proof: {item}" for item in pre_binding["errors"][:10]) + failures.extend(f"Post-apply proof: {item}" for item in post_binding["errors"][:10]) + if not pre_apply_workbook_fingerprint: + failures.append("Pre-apply workbook fingerprint is missing.") + if not post_apply_workbook_fingerprint: + failures.append("Post-apply workbook fingerprint is missing.") + + return { + "status": "verified" if not failures else "failed", + "expectedOperationIds": expected, + "expectedOperationCount": len(expected), + "officeResultCount": len(result_rows), + "officeAppliedCount": applied if isinstance(applied, int) and not isinstance(applied, bool) else None, + "failures": failures, + "proof": { + "schemaVersion": "2026-08-agent-recovery-proof-v1", + "attemptId": str(attempt.get("attemptId") or ""), + "operationRegistryHash": operation_registry_hash, + "recoveryPolicyHash": recovery_policy_hash, + "preWorkbookFingerprint": pre_apply_workbook_fingerprint or "", + "postWorkbookFingerprint": post_apply_workbook_fingerprint or "", + "preHostFingerprint": str(pre_snapshot.get("fingerprint") or ""), + "postHostFingerprint": str(post_snapshot.get("fingerprint") or ""), + "preTargetHashes": pre_binding["target_hashes"], + "postTargetHashes": post_binding["target_hashes"], + }, + } + + def _dedupe_dicts(items: list[dict[str, Any]]) -> list[dict[str, Any]]: seen: set[str] = set() result: list[dict[str, Any]] = [] @@ -667,6 +1404,159 @@ def _dedupe_dicts(items: list[dict[str, Any]]) -> list[dict[str, Any]]: return result[:100] +def _agent_run_event_sequence(payload: dict[str, Any]) -> int: + metadata = payload.get("metadata") + metadata = metadata if isinstance(metadata, dict) else {} + control = metadata.get("controlPlane") + control = control if isinstance(control, dict) else {} + return max(int(control.get("eventSequence") or 0), 0) + + +def _expected_agent_run_sequence(payload: dict[str, Any], request: Request) -> int: + raw = str(request.headers.get("x-agent-run-expected-sequence") or "").strip() + if not raw: + return _agent_run_event_sequence(payload) + try: + sequence = int(raw) + except ValueError as exc: + raise HTTPException( + status_code=400, + detail={ + "code": "agent_run_sequence_invalid", + "message": "X-Agent-Run-Expected-Sequence must be a non-negative integer.", + }, + ) from exc + if sequence < 0: + raise HTTPException( + status_code=400, + detail={ + "code": "agent_run_sequence_invalid", + "message": "X-Agent-Run-Expected-Sequence must be a non-negative integer.", + }, + ) + return sequence + + +def _agent_run_idempotency_key( + request: Request, + *, + action: str, + run_id: str, + expected_sequence: int, +) -> str: + supplied = str(request.headers.get("idempotency-key") or "").strip() + if len(supplied) > 200: + raise HTTPException( + status_code=400, + detail={ + "code": "agent_run_idempotency_key_invalid", + "message": "Idempotency-Key must be at most 200 characters.", + }, + ) + if supplied: + return supplied + return f"{action}:{run_id}:{expected_sequence}" + + +def _agent_run_control_http_error(result: dict[str, Any]) -> HTTPException: + outcome = str(result.get("outcome") or "invalid") + if outcome == "not_found": + return HTTPException(status_code=404, detail="Agent run not found") + code = { + "sequence_conflict": "agent_run_sequence_conflict", + "terminal_fence": "agent_run_terminal_fence", + "idempotency_conflict": "agent_run_idempotency_conflict", + }.get(outcome, "agent_run_transition_rejected") + return HTTPException( + status_code=409, + detail={ + "code": code, + "message": "The agent run changed or is terminal; refresh it before retrying.", + "currentSequence": result.get("currentSequence"), + }, + ) + + +def _record_agent_run_control(event: str, result: dict[str, Any]) -> None: + from backend.observability import record_agent_run_control_event + + record_agent_run_control_event(event, str(result.get("outcome") or "rejected")) + + +def _format_agent_run_sse_event(event: dict[str, Any]) -> str: + sequence = max(int(event.get("sequence") or 0), 0) + raw_type = str(event.get("eventType") or "run_event") + event_type = "".join( + character if character.isalnum() or character in {"_", "-"} else "_" + for character in raw_type + )[:80] or "run_event" + data = json.dumps(event, sort_keys=True, separators=(",", ":"), default=str) + return f"id: {sequence}\nevent: {event_type}\ndata: {data}\nretry: 2000\n\n" + + +def _resolved_provider_model(provider: str, requested_model: Optional[str]) -> str: + if requested_model: + return str(requested_model) + from backend.config import ( + ANTHROPIC_MODEL_COMPLEX, + LMSTUDIO_MODEL, + OLLAMA_MODEL_MEDIUM, + OPENAI_MODEL, + ) + + return { + "openai": OPENAI_MODEL, + "anthropic": ANTHROPIC_MODEL_COMPLEX, + "lmstudio": LMSTUDIO_MODEL or "", + "ollama": OLLAMA_MODEL_MEDIUM, + }.get(str(provider or "").strip().lower(), "") + + +def _egress_request_material( + request: AgentRunCreateRequest, + session_id: str, + provider: str, + model: str, + documents: list[dict], +) -> dict[str, Any]: + return { + "session_id": session_id, + "prompt": request.prompt, + "provider": provider, + "model": model, + "workbook_context": request.workbook_context, + "documents": documents, + "template_id": request.template_id, + "template_params": request.template_params, + "validation_profile": request.validation_profile, + "accuracy_mode": request.accuracy_mode, + "approval_policy": request.approval_policy, + } + + +def _egress_approval_ttl_seconds() -> int: + try: + return min(max(int(os.getenv("AI_EGRESS_APPROVAL_TTL_SECONDS", "300")), 30), 900) + except ValueError: + return 300 + + +def _egress_approval_http_error(outcome: str) -> HTTPException: + code = { + "not_found": "egress_approval_not_found", + "digest_mismatch": "egress_approval_digest_mismatch", + "expired": "egress_approval_expired", + "already_consumed": "egress_approval_already_consumed", + }.get(outcome, "egress_approval_invalid") + return HTTPException( + status_code=409, + detail={ + "code": code, + "message": "The cloud egress approval is invalid, expired, or already used; create a fresh preview.", + }, + ) + + def _load_run_documents(store, session_id: str, source_document_ids: list[str]) -> list[dict]: documents = [] for document_id in source_document_ids: @@ -703,17 +1593,43 @@ async def create_agent_run_repair_plan(run_id: str): return repair -def _transition_run(run_id: str, action: str) -> AgentRun: +def _transition_run(run_id: str, action: str, http_request: Request) -> AgentRun: store = get_session_store() payload = store.get_agent_run(run_id) if not payload: raise HTTPException(status_code=404, detail="Agent run not found") run = get_agent_run_service().transition(AgentRun.model_validate(payload), action) - store.save_agent_run(run.model_dump(mode="json", by_alias=True)) - store.record_audit_event( - session_id=run.session_id, - plan_id=run.plan_id, - event_type=f"agent_run_{action}d" if action != "cancel" else "agent_run_cancelled", - payload={"run_id": run.run_id, "status": run.status}, + expected_sequence = _expected_agent_run_sequence(payload, http_request) + event_type = f"run_{action}d" if action != "cancel" else "run_cancelled" + result = store.append_agent_run_event( + run_id, + expected_sequence=expected_sequence, + event_type=event_type, + event_payload={"action": action, "status": run.status}, + updated_run=run.model_dump(mode="json", by_alias=True), + idempotency_key=_agent_run_idempotency_key( + http_request, + action=action, + run_id=run_id, + expected_sequence=expected_sequence, + ), ) + _record_agent_run_control( + "cancelled" if action == "cancel" else "paused" if action == "pause" else action, + result, + ) + if result.get("outcome") not in {"appended", "replayed"}: + raise _agent_run_control_http_error(result) + run = AgentRun.model_validate(result["run"]) + if result.get("outcome") == "appended": + store.record_audit_event( + session_id=run.session_id, + plan_id=run.plan_id, + event_type=f"agent_run_{action}d" if action != "cancel" else "agent_run_cancelled", + payload={ + "run_id": run.run_id, + "status": run.status, + "event_sequence": result["event"]["sequence"], + }, + ) return run diff --git a/backend/api/routes/benchmarks.py b/backend/api/routes/benchmarks.py index 5a2e88c..b4e81eb 100644 --- a/backend/api/routes/benchmarks.py +++ b/backend/api/routes/benchmarks.py @@ -4,7 +4,7 @@ import time import uuid -from typing import Optional +from typing import Any, Optional from fastapi import APIRouter from pydantic import BaseModel, Field @@ -107,19 +107,20 @@ async def run_benchmarks(request: BenchmarkRunRequest): if request.mode in {"workbook-aware", "workbook_aware"}: results = get_workbook_fixture_benchmark_runner().run_all() score = round(sum(result.score for result in results) / max(len(results), 1), 1) - payload = { + serialized_results = [result.model_dump(mode="json", by_alias=True) for result in results] + payload: dict[str, Any] = { "run_id": f"workbook_bench_{time.strftime('%Y%m%dT%H%M%SZ', time.gmtime())}_{uuid.uuid4().hex[:10]}", "status": "completed", "mode": "workbook-aware", "total_cases": len(results), "overall_score": score, "gate_status": "passed" if all(result.status in {"passed", "warning"} and result.score >= 70 for result in results) else "failed", - "results": [result.model_dump(mode="json", by_alias=True) for result in results], + "results": serialized_results, } if request.persist: store = get_session_store() store.save_benchmark_run(payload) - for result in payload["results"]: + for result in serialized_results: store.save_workbook_benchmark_fixture_result(payload["run_id"], result) return payload diff --git a/backend/api/routes/documents.py b/backend/api/routes/documents.py index c7d366f..86763d6 100644 --- a/backend/api/routes/documents.py +++ b/backend/api/routes/documents.py @@ -9,11 +9,13 @@ from pathlib import Path import structlog -from fastapi import APIRouter, HTTPException, UploadFile, File, Form +from fastapi import APIRouter, HTTPException, Request, UploadFile, File, Form from pydantic import BaseModel, Field from typing import Optional from backend.config import DATA_DIR +from backend.documents.pdf_worker import PDFWorkerTimeout, parse_pdf_sandboxed +from backend.operations.work_admission import WorkBudgetError, WorkKind, run_cancellable_in_thread from backend.storage import get_session_store router = APIRouter(prefix="/api/v1", tags=["documents"]) @@ -29,6 +31,13 @@ ".xlsm": (b"PK\x03\x04",), } +# ZIP bomb defenses for OOXML uploads. Excel workbooks rarely have more than +# ~2000 parts; anything beyond is suspicious. The compression-ratio cap +# bounds zip-bomb attacks where a small ZIP expands to multiple GB. +ZIP_MAX_ENTRIES = 4096 +ZIP_MAX_UNCOMPRESSED_BYTES = 250 * 1024 * 1024 +ZIP_MAX_COMPRESSION_RATIO = 200.0 + class DocumentToModelRequest(BaseModel): template_id: Optional[str] = None # Auto-detect if not specified @@ -47,10 +56,14 @@ class DocumentPlanRequest(BaseModel): @router.post("/documents/upload") async def upload_document( + http_request: Request, file: UploadFile = File(...), session_id: str = Form("default"), ): """Persist and parse a PDF/CSV/XLSX source document for later planning.""" + from backend.api.routes.tasks import _claim_request_session + + _claim_request_session(http_request, session_id) filename = file.filename or "document" ext = Path(filename).suffix.lower() if ext not in {".pdf", ".csv", ".tsv", ".xlsx", ".xlsm"}: @@ -68,27 +81,77 @@ async def upload_document( # Strip any path component the filename might carry before sanitizing. safe_name = re.sub(r"[^A-Za-z0-9_.-]", "_", Path(filename).name) file_path = data_dir / f"{document_id}_{safe_name}" - # Push the disk write to a worker thread so a 50 MB write doesn't stall - # the event loop on slow storage. - await asyncio.to_thread(file_path.write_bytes, content) try: - parsed, summary = await _parse_uploaded_content(file_path, content, filename, file.content_type or "") + from backend.services.document_cleanup import ( + cleanup_unregistered_upload, + recover_pending_upload_cleanups, + stage_managed_upload, + ) + + await asyncio.to_thread(recover_pending_upload_cleanups, DATA_DIR) + await asyncio.to_thread( + stage_managed_upload, + content, + file_path, + data_root=DATA_DIR, + session_id=session_id, + document_id=document_id, + ) + from backend.operations.work_admission import work_context + + managed_kind: WorkKind | None = "managed_xlsx" if ext in {".xlsx", ".xlsm"} else None + with work_context(session_id=session_id, kind=managed_kind): + parsed, summary = await _parse_uploaded_content(file_path, content, filename, file.content_type or "") + store = get_session_store() + store.save_document( + session_id=session_id, + document_id=document_id, + filename=filename, + content_type=file.content_type or "", + size_bytes=len(content), + file_path=str(file_path), + summary=_jsonable(summary), + parsed=_jsonable(parsed), + ) + except asyncio.CancelledError: + try: + await asyncio.to_thread( + cleanup_unregistered_upload, + file_path, + data_root=DATA_DIR, + session_id=session_id, + document_id=document_id, + reason="upload_cancelled", + ) + except Exception as cleanup_exc: + logger.critical( + "document_upload_cleanup_unavailable", + document_id=document_id, + error=str(cleanup_exc), + ) + raise except Exception as exc: - logger.error("document_upload_parse_failed", filename=filename, error=str(exc)) - raise HTTPException(status_code=500, detail=f"Failed to parse document: {exc}") + try: + await asyncio.to_thread( + cleanup_unregistered_upload, + file_path, + data_root=DATA_DIR, + session_id=session_id, + document_id=document_id, + reason="upload_failed", + ) + except Exception as cleanup_exc: + logger.critical( + "document_upload_cleanup_unavailable", + document_id=document_id, + error=str(cleanup_exc), + ) + logger.error("document_upload_failed", filename=filename, error=str(exc)) + if isinstance(exc, WorkBudgetError): + raise _work_budget_http_exception(exc) from None + raise HTTPException(status_code=500, detail=f"Failed to parse document: {exc}") from exc - store = get_session_store() - store.save_document( - session_id=session_id, - document_id=document_id, - filename=filename, - content_type=file.content_type or "", - size_bytes=len(content), - file_path=str(file_path), - summary=_jsonable(summary), - parsed=_jsonable(parsed), - ) store.record_audit_event( session_id=session_id, event_type="document_uploaded", @@ -115,11 +178,23 @@ async def get_document(document_id: str): @router.post("/documents/{document_id}/plan") -async def plan_from_document(document_id: str, request: DocumentPlanRequest): +async def plan_from_document(document_id: str, request: DocumentPlanRequest, http_request: Request): """Create a typed, cited operation plan from a stored source document.""" - from backend.api.routes.tasks import _annotate_plan_from_documents, _collect_plan_citations + from backend.api.routes.tasks import ( + _annotate_plan_from_documents, + _claim_request_session, + _collect_plan_citations, + _enforce_ai_policy, + _request_actor, + ) from backend.services.operation_planner import build_operation_plan, dump_plan + _enforce_ai_policy( + "preview", + source_document_ids=[document_id], + actor=_request_actor(http_request), + ) + _claim_request_session(http_request, request.session_id) store = get_session_store() doc = store.get_document(document_id) if not doc: @@ -160,12 +235,25 @@ async def plan_from_document(document_id: str, request: DocumentPlanRequest): @router.post("/documents/{document_id}/plan/v2") -async def plan_from_document_v2(document_id: str, request: DocumentPlanRequest): +async def plan_from_document_v2(document_id: str, request: DocumentPlanRequest, http_request: Request): """Create a source-backed V2 operation plan from a stored document.""" - from backend.api.routes.tasks import _annotate_plan_from_documents, _collect_plan_citations, _save_operation_plan_with_v2 - from backend.services.operation_planner import build_operation_plan, dump_plan + from backend.api.routes.tasks import ( + _annotate_plan_from_documents, + _claim_request_session, + _collect_plan_citations, + _enforce_ai_policy, + _request_actor, + _save_operation_plan_with_v2, + ) + from backend.services.operation_planner import build_operation_plan from backend.services.plan_intelligence import plan_to_v2_with_intelligence + _enforce_ai_policy( + "preview", + source_document_ids=[document_id], + actor=_request_actor(http_request), + ) + _claim_request_session(http_request, request.session_id) store = get_session_store() doc = store.get_document(document_id) if not doc: @@ -221,11 +309,11 @@ async def parse_document(file: UploadFile = File(...)): raise HTTPException(status_code=400, detail="File too large (max 50MB)") try: - from backend.documents.pdf_parser import parse_pdf, extract_financial_data + from backend.documents.pdf_parser import extract_financial_data content = await _read_upload_limited(file) _validate_upload_signature(".pdf", content) - parsed = parse_pdf(io.BytesIO(content), filename=file.filename) + parsed = await run_cancellable_in_thread(parse_pdf_sandboxed, content, file.filename or "document.pdf") financial = extract_financial_data(parsed) return { @@ -247,11 +335,15 @@ async def parse_document(file: UploadFile = File(...)): ], "context": parsed.to_context(), } + except PDFWorkerTimeout: + raise HTTPException(status_code=504, detail="PDF parsing timed out") from None + except WorkBudgetError as exc: + raise _work_budget_http_exception(exc) from None except ImportError: - raise HTTPException(status_code=500, detail="PyMuPDF not installed: pip install pymupdf") + raise HTTPException(status_code=500, detail="PyMuPDF not installed: pip install pymupdf") from None except Exception as e: logger.error("document_parse_failed", filename=file.filename, error=str(e)) - raise HTTPException(status_code=500, detail=f"Failed to parse PDF: {str(e)}") + raise HTTPException(status_code=500, detail=f"Failed to parse PDF: {str(e)}") from e @router.post("/documents/to-model") @@ -275,13 +367,13 @@ async def document_to_model( raise HTTPException(status_code=400, detail="File too large (max 50MB)") try: - from backend.documents.pdf_parser import parse_pdf, extract_financial_data, extract_model_params + from backend.documents.pdf_parser import extract_financial_data, extract_model_params from backend.templates.registry import get_template, generate_template from backend.excel.formatting_engine import post_process_operations content = await _read_upload_limited(file) _validate_upload_signature(".pdf", content) - parsed = parse_pdf(io.BytesIO(content), filename=file.filename) + parsed = await run_cancellable_in_thread(parse_pdf_sandboxed, content, file.filename or "document.pdf") financial = extract_financial_data(parsed) model_params = extract_model_params(parsed, financial) @@ -311,18 +403,21 @@ async def document_to_model( "source": "document_to_model", } + except PDFWorkerTimeout: + raise HTTPException(status_code=504, detail="PDF parsing timed out") from None + except WorkBudgetError as exc: + raise _work_budget_http_exception(exc) from None except ImportError as e: - raise HTTPException(status_code=500, detail=f"Missing dependency: {str(e)}") + raise HTTPException(status_code=500, detail=f"Missing dependency: {str(e)}") from e except KeyError as e: - raise HTTPException(status_code=404, detail=str(e)) + raise HTTPException(status_code=404, detail=str(e)) from e except Exception as e: logger.error("document_to_model_failed", filename=file.filename, error=str(e)) - raise HTTPException(status_code=500, detail=f"Failed to generate model: {str(e)}") + raise HTTPException(status_code=500, detail=f"Failed to generate model: {str(e)}") from e def _auto_detect_template(financial: dict, params: dict) -> str: """Auto-detect the best template based on document content.""" - doc_type = financial.get("type", "").lower() items = financial.get("items", []) item_names = [item.lower() if isinstance(item, str) else str(item).lower() for item in items] all_text = " ".join(item_names) @@ -355,9 +450,9 @@ async def _parse_uploaded_content( ) -> tuple[dict, dict]: ext = file_path.suffix.lower() if ext == ".pdf": - from backend.documents.pdf_parser import parse_pdf, extract_financial_data, extract_model_params + from backend.documents.pdf_parser import extract_financial_data, extract_model_params - parsed_doc = parse_pdf(io.BytesIO(content), filename=filename) + parsed_doc = await run_cancellable_in_thread(parse_pdf_sandboxed, content, filename) financial = extract_financial_data(parsed_doc) params = extract_model_params(parsed_doc, financial) parsed = { @@ -419,7 +514,7 @@ async def _parse_uploaded_content( if ext in {".xlsx", ".xlsm"}: from backend.excel.workbook_files import WorkbookIngestionService - summary_payload = WorkbookIngestionService().summarize(content, filename) + summary_payload = await run_cancellable_in_thread(WorkbookIngestionService().summarize, content, filename) first_sheet = (summary_payload.get("sheets") or [{}])[0] preview = first_sheet.get("preview") or [] headers = [str(value) for value in (preview[0] if preview else [])] @@ -446,7 +541,9 @@ async def _parse_uploaded_content( "sheets": summary_payload.get("sheet_names", []), "formula_count": summary_payload.get("formula_count", 0), "has_macros": summary_payload.get("has_macros", False), - "extracted_params": _extract_params_from_rows([dict(zip(headers, row)) for row in rows if headers]), + "extracted_params": _extract_params_from_rows( + [dict(zip(headers, row, strict=False)) for row in rows if headers] + ), "suggested_template": "dcf_model", } return parsed, summary @@ -454,6 +551,17 @@ async def _parse_uploaded_content( raise ValueError(f"Unsupported file type: {ext or content_type}") +def _work_budget_http_exception(exc: WorkBudgetError) -> HTTPException: + return HTTPException( + status_code=503, + detail={ + "code": exc.code, + "message": "Document worker capacity is unavailable; retry shortly.", + }, + headers={"Retry-After": "1"}, + ) + + async def _read_upload_limited(file: UploadFile) -> bytes: chunks: list[bytes] = [] total = 0 @@ -474,6 +582,57 @@ def _validate_upload_signature(ext: str, content: bytes) -> None: raise HTTPException(status_code=400, detail="Unsupported file type") if signatures and not any(content.startswith(signature) for signature in signatures): raise HTTPException(status_code=400, detail=f"File content does not match {ext.upper()} signature") + if ext.lower() in {".xlsx", ".xlsm"}: + _validate_ooxml_zip_safety(content, ext.lower()) + + +def _validate_ooxml_zip_safety(content: bytes, ext: str) -> None: + """Reject ZIP bombs and path-traversal entries before handing off to the worker. + + OOXML files are ZIPs. A malformed central directory, oversized expansion, + suspicious entry count, or ``..``/absolute-path entry is treated as a + release blocker for the upload pipeline. + """ + import zipfile + + try: + with zipfile.ZipFile(io.BytesIO(content)) as zf: + infos = zf.infolist() + if len(infos) > ZIP_MAX_ENTRIES: + raise HTTPException( + status_code=400, + detail=f"{ext.upper()} package has too many entries ({len(infos)} > {ZIP_MAX_ENTRIES})", + ) + total_uncompressed = 0 + total_compressed = max(len(content), 1) + for info in infos: + name = info.filename + if name.startswith("/") or name.startswith("\\"): + raise HTTPException(status_code=400, detail=f"{ext.upper()} entry has absolute path: {name}") + # Normalize to detect traversal patterns regardless of separator. + normalized = name.replace("\\", "/") + if ".." in normalized.split("/"): + raise HTTPException(status_code=400, detail=f"{ext.upper()} entry has traversal: {name}") + if info.file_size < 0: + raise HTTPException(status_code=400, detail=f"{ext.upper()} entry reports negative size") + total_uncompressed += info.file_size + if total_uncompressed > ZIP_MAX_UNCOMPRESSED_BYTES: + raise HTTPException( + status_code=400, + detail=f"{ext.upper()} package uncompressed size exceeds safety limit", + ) + if total_uncompressed / total_compressed > ZIP_MAX_COMPRESSION_RATIO: + raise HTTPException( + status_code=400, + detail=f"{ext.upper()} package compression ratio exceeds {ZIP_MAX_COMPRESSION_RATIO}:1; possible zip bomb", + ) + except HTTPException: + raise + except zipfile.BadZipFile as exc: + raise HTTPException( + status_code=400, + detail=f"{ext.upper()} package is not a valid ZIP: {exc}", + ) from exc def _extract_params_from_rows(rows: list[dict]) -> dict: diff --git a/backend/api/routes/enterprise.py b/backend/api/routes/enterprise.py index e8a2108..d3e9f69 100644 --- a/backend/api/routes/enterprise.py +++ b/backend/api/routes/enterprise.py @@ -7,11 +7,12 @@ from __future__ import annotations import time -from typing import Optional +from typing import Any, Optional -from fastapi import APIRouter, HTTPException +from fastapi import APIRouter, HTTPException, Request from pydantic import BaseModel, EmailStr, Field +from backend.config import API_KEY from backend.api.security.rbac import PERMISSIONS, Role, get_sso_config, get_zdr from backend.storage import get_session_store @@ -23,9 +24,11 @@ class UserRequest(BaseModel): email: EmailStr name: str = Field(..., min_length=1, max_length=200) role: Role = Role.ANALYST - status: str = "active" - mfa_enabled: bool = False + status: Optional[str] = None + mfa_enabled: Optional[bool] = None metadata: dict = Field(default_factory=dict) + oidc_issuer: Optional[str] = Field(default=None, min_length=1, max_length=2048) + oidc_subject: Optional[str] = Field(default=None, min_length=1, max_length=2048) class ApiKeyRequest(BaseModel): @@ -50,6 +53,132 @@ class RetentionPurgeRequest(BaseModel): retention_hours: Optional[int] = None +def _stored_enterprise_config(key: str) -> dict[str, Any]: + stored = get_session_store().get_enterprise_config(key) + return stored if isinstance(stored, dict) else {} + + +def _stored_config_payload(key: str) -> dict[str, Any]: + stored = _stored_enterprise_config(key) + raw_config = stored.get("config") + return raw_config if isinstance(raw_config, dict) else {} + + +def _effective_sso_config() -> dict[str, Any]: + env_config: dict[str, Any] = get_sso_config().to_dict() + stored = _stored_enterprise_config("sso") + stored_config = _stored_config_payload("sso") + effective: dict[str, Any] = { + **env_config, + **stored_config, + } + provider = str(effective.get("provider") or "") + metadata_url = str(effective.get("idp_metadata_url") or effective.get("idpMetadataUrl") or "") + effective["idp_configured"] = bool(effective.get("idp_configured") or metadata_url) + effective["enabled"] = bool(effective.get("enabled")) or bool(provider and metadata_url) + return { + "config_key": "sso", + "config": effective, + "policySource": "enterprise_config" if stored else "environment", + "storedConfig": stored_config, + "updated_at": stored.get("updated_at"), + } + + +def _effective_security_policy() -> dict[str, Any]: + stored = _stored_enterprise_config("security_policy") + stored_config = _stored_config_payload("security_policy") + sso_config = _stored_config_payload("sso") + provider_policy = _stored_config_payload("provider_policy") + retention_policy = _stored_config_payload("retention_policy") + require_mfa = any( + bool(config.get("require_mfa") or config.get("requireMfa")) + for config in (stored_config, sso_config, provider_policy) + if isinstance(config, dict) + ) + policy_config = { + **stored_config, + "require_mfa": require_mfa, + "requireMfa": require_mfa, + "api_key_mfa_gate": require_mfa, + "admin_audit_required": bool(stored_config.get("admin_audit_required", True)), + "allowed_auth_sources": stored_config.get( + "allowed_auth_sources", + ["enterprise_api_key", "static_api_key", "sso"], + ), + "retention_policy": retention_policy or get_zdr().to_dict(), + } + has_stored_policy = bool(stored_config) + return { + "config_key": "security_policy", + "config": policy_config, + "policySource": "enterprise_config" if has_stored_policy else "derived", + "derivedFrom": { + "security_policy": has_stored_policy, + "sso": bool(sso_config), + "provider_policy": bool(provider_policy), + "retention_policy": bool(retention_policy), + }, + "updated_at": stored.get("updated_at"), + } + + +def _actor_payload(request: Request) -> dict: + actor = getattr(request.state, "enterprise_user", None) or getattr(request.state, "user", None) + if isinstance(actor, dict): + return { + "user_id": actor.get("user_id") or actor.get("id") or "unknown", + "email": actor.get("email", ""), + "role": actor.get("role", ""), + "auth_source": actor.get("auth_source", ""), + } + if actor: + return { + "user_id": getattr(actor, "user_id", "unknown"), + "email": getattr(actor, "email", ""), + "role": getattr(getattr(actor, "role", ""), "value", getattr(actor, "role", "")), + "auth_source": getattr(actor, "auth_source", ""), + } + if request.headers.get("X-API-Key") == API_KEY: + return { + "user_id": "admin", + "email": "admin@localhost", + "role": "admin", + "auth_source": "static_api_key", + } + return {"user_id": "unknown", "email": "", "role": "", "auth_source": "unknown"} + + +def _audit_enterprise_admin( + request: Request, + *, + action: str, + target_type: str, + target_id: str = "", + status: str = "success", + metadata: Optional[dict] = None, +) -> str: + payload = { + "actor": _actor_payload(request), + "action": action, + "target_type": target_type, + "target_id": target_id, + "status": status, + "request": { + "method": request.method, + "path": request.url.path, + "trace_id": request.headers.get("X-Trace-Id", ""), + "client_host": request.client.host if request.client else "", + }, + "metadata": metadata or {}, + } + store = get_session_store() + return store.record_enterprise_admin_audit( + event_type=f"enterprise_admin.{action}", + payload=payload, + ) + + @router.get("/dashboard") async def enterprise_dashboard(): store = get_session_store() @@ -59,7 +188,8 @@ async def enterprise_dashboard(): "roles": [role.value for role in Role], "permissions": {role.value: sorted(perms) for role, perms in PERMISSIONS.items()}, }, - "sso": get_sso_config().to_dict(), + "sso": _effective_sso_config(), + "security_policy": _effective_security_policy(), "zero_data_retention": get_zdr().to_dict(), } @@ -70,25 +200,66 @@ async def list_users(limit: int = 100): @router.post("/users") -async def upsert_user(request: UserRequest): - user = get_session_store().upsert_enterprise_user( - user_id=request.user_id, - email=request.email, - name=request.name, - role=request.role.value, - status=request.status, - mfa_enabled=request.mfa_enabled, - metadata=request.metadata, +async def upsert_user(request: Request, payload: UserRequest): + try: + user = get_session_store().upsert_enterprise_user( + user_id=payload.user_id, + email=payload.email, + name=payload.name, + role=payload.role.value, + status=payload.status, + mfa_enabled=payload.mfa_enabled, + metadata=payload.metadata, + oidc_issuer=payload.oidc_issuer, + oidc_subject=payload.oidc_subject, + ) + except ValueError as exc: + _audit_enterprise_admin( + request, + action="upsert_user", + target_type="enterprise_user", + target_id=payload.user_id, + status="failed", + metadata={"reason": str(exc)}, + ) + raise HTTPException(status_code=409, detail=str(exc)) from exc + _audit_enterprise_admin( + request, + action="upsert_user", + target_type="enterprise_user", + target_id=payload.user_id, + metadata={ + "role": payload.role.value, + "status": user.get("status"), + "mfa_enabled": user.get("mfa_enabled"), + "oidc_bound": bool(user.get("oidc_issuer") and user.get("oidc_subject")), + }, ) return {"user": user} @router.post("/users/{user_id}/api-keys") -async def create_api_key(user_id: str, request: ApiKeyRequest): +async def create_api_key(user_id: str, request: Request, payload: ApiKeyRequest): try: - return get_session_store().create_enterprise_api_key(user_id, request.label) + key = get_session_store().create_enterprise_api_key(user_id, payload.label) except ValueError as exc: - raise HTTPException(status_code=404, detail=str(exc)) + _audit_enterprise_admin( + request, + action="create_api_key", + target_type="enterprise_api_key", + target_id=user_id, + status="failed", + metadata={"reason": str(exc)}, + ) + raise HTTPException(status_code=404, detail=str(exc)) from exc + _audit_enterprise_admin( + request, + action="create_api_key", + target_type="enterprise_api_key", + target_id=key["key_id"], + metadata={"user_id": user_id, "label": payload.label}, + ) + return key @router.get("/api-keys") @@ -96,21 +267,66 @@ async def list_api_keys(user_id: Optional[str] = None): return {"api_keys": get_session_store().list_enterprise_api_keys(user_id)} +@router.delete("/api-keys/{key_id}") +async def revoke_api_key(key_id: str, request: Request): + ok = get_session_store().revoke_enterprise_api_key(key_id) + if not ok: + _audit_enterprise_admin( + request, + action="revoke_api_key", + target_type="enterprise_api_key", + target_id=key_id, + status="failed", + metadata={"reason": "API key not found"}, + ) + raise HTTPException(status_code=404, detail="API key not found") + _audit_enterprise_admin( + request, + action="revoke_api_key", + target_type="enterprise_api_key", + target_id=key_id, + ) + return {"success": True, "key_id": key_id} + + @router.get("/teams") async def list_teams(limit: int = 100): return {"teams": get_session_store().list_enterprise_teams(limit)} @router.post("/teams") -async def create_team(request: TeamRequest): - return {"team": get_session_store().create_enterprise_team(request.name, request.metadata)} +async def create_team(request: Request, payload: TeamRequest): + team = get_session_store().create_enterprise_team(payload.name, payload.metadata) + _audit_enterprise_admin( + request, + action="create_team", + target_type="enterprise_team", + target_id=team.get("team_id", ""), + metadata={"name": payload.name}, + ) + return {"team": team} @router.post("/teams/{team_id}/members") -async def add_team_member(team_id: str, request: TeamMemberRequest): - ok = get_session_store().add_enterprise_team_member(team_id, request.user_id, request.role) +async def add_team_member(team_id: str, request: Request, payload: TeamMemberRequest): + ok = get_session_store().add_enterprise_team_member(team_id, payload.user_id, payload.role) if not ok: + _audit_enterprise_admin( + request, + action="add_team_member", + target_type="enterprise_team_member", + target_id=team_id, + status="failed", + metadata={"user_id": payload.user_id, "role": payload.role, "reason": "Team or user not found"}, + ) raise HTTPException(status_code=404, detail="Team or user not found") + _audit_enterprise_admin( + request, + action="add_team_member", + target_type="enterprise_team_member", + target_id=team_id, + metadata={"user_id": payload.user_id, "role": payload.role}, + ) return {"success": True} @@ -126,13 +342,38 @@ async def list_roles(): @router.get("/sso/config") async def get_sso(): - return get_sso_config().to_dict() + return _effective_sso_config() @router.put("/sso/config") -async def set_sso(request: ConfigRequest): - config = get_session_store().set_enterprise_config("sso", request.config) - return {"sso": config} +async def set_sso(request: Request, payload: ConfigRequest): + config = get_session_store().set_enterprise_config("sso", payload.config) + _audit_enterprise_admin( + request, + action="set_sso_config", + target_type="enterprise_config", + target_id="sso", + metadata={"config_keys": sorted(payload.config)}, + ) + return {"sso": config, "effective": _effective_sso_config()} + + +@router.get("/security-policy") +async def get_security_policy(): + return _effective_security_policy() + + +@router.put("/security-policy") +async def set_security_policy(request: Request, payload: ConfigRequest): + config = get_session_store().set_enterprise_config("security_policy", payload.config) + _audit_enterprise_admin( + request, + action="set_security_policy", + target_type="enterprise_config", + target_id="security_policy", + metadata={"config_keys": sorted(payload.config)}, + ) + return {"security_policy": config, "effective": _effective_security_policy()} @router.get("/provider-policy") @@ -150,8 +391,15 @@ async def get_provider_policy(): @router.put("/provider-policy") -async def set_provider_policy(request: ConfigRequest): - config = get_session_store().set_enterprise_config("provider_policy", request.config) +async def set_provider_policy(request: Request, payload: ConfigRequest): + config = get_session_store().set_enterprise_config("provider_policy", payload.config) + _audit_enterprise_admin( + request, + action="set_provider_policy", + target_type="enterprise_config", + target_id="provider_policy", + metadata={"config_keys": sorted(payload.config)}, + ) return {"provider_policy": config} @@ -164,32 +412,56 @@ async def get_retention_policy(): @router.put("/retention-policy") -async def set_retention_policy(request: ConfigRequest): - config = get_session_store().set_enterprise_config("retention_policy", request.config) +async def set_retention_policy(request: Request, payload: ConfigRequest): + config = get_session_store().set_enterprise_config("retention_policy", payload.config) + _audit_enterprise_admin( + request, + action="set_retention_policy", + target_type="enterprise_config", + target_id="retention_policy", + metadata={"config_keys": sorted(payload.config)}, + ) return {"retention_policy": config} @router.post("/retention/purge") -async def purge_expired_sessions(request: RetentionPurgeRequest): - hours = request.retention_hours +async def purge_expired_sessions(request: Request, payload: RetentionPurgeRequest): + hours = payload.retention_hours if hours is None: zdr = get_zdr() hours = int(zdr.retention_hours or 0) - if hours <= 0: - raise HTTPException(status_code=400, detail="retention_hours must be greater than zero") + if hours < 0: + raise HTTPException(status_code=400, detail="retention_hours cannot be negative") + from backend.services.lifecycle import get_lifecycle_coordinator + cutoff = time.time() - (hours * 3600) - deleted = get_session_store().purge_sessions_before(cutoff) - return {"deleted_sessions": deleted, "retention_hours": hours} + receipts = await get_lifecycle_coordinator().purge_before(cutoff) + deleted = sum(1 for receipt in receipts if receipt.database_deleted and not receipt.errors) + failed = sum(1 for receipt in receipts if receipt.state.value == "failed") + _audit_enterprise_admin( + request, + action="purge_expired_sessions", + target_type="retention_window", + target_id=str(hours), + metadata={ + "retention_hours": hours, + "deleted_sessions": deleted, + "failed_sessions": failed, + "cutoff_timestamp": cutoff, + }, + ) + return {"deleted_sessions": deleted, "failed_sessions": failed, "retention_hours": hours} @router.get("/security/readiness") async def security_readiness(): from backend.api.security.soc2 import get_soc2_auditor - from backend.storage.encrypted_store import get_encrypted_store + from backend.storage.encrypted_store import get_primary_storage_encryption_status return { "soc2": get_soc2_auditor().run_audit(), - "encryption": get_encrypted_store().get_encryption_status(), - "sso": get_sso_config().to_dict(), + "encryption": get_primary_storage_encryption_status(), + "sso": _effective_sso_config(), + "security_policy": _effective_security_policy(), "zero_data_retention": get_zdr().to_dict(), } diff --git a/backend/api/routes/experiments.py b/backend/api/routes/experiments.py index 5f1f851..1cc6fcd 100644 --- a/backend/api/routes/experiments.py +++ b/backend/api/routes/experiments.py @@ -6,11 +6,11 @@ import hashlib import time from pathlib import Path -from typing import Any, Literal, Optional +from typing import Any, Literal, Optional, cast -from fastapi import APIRouter, HTTPException, Query +from fastapi import APIRouter, Depends, HTTPException, Query from fastapi.responses import Response -from pydantic import BaseModel, ConfigDict, Field, TypeAdapter, ValidationError +from pydantic import BaseModel, ConfigDict, Field, TypeAdapter, ValidationError, model_validator from backend.models.operations import ( ApplyEligibility, @@ -32,34 +32,84 @@ from backend.storage import get_session_store from backend.services.operation_registry import get_operation_registry_v3 -router = APIRouter(prefix="/api/v1/experiments/v7", tags=["experiments"]) +def require_experimental_routes_enabled() -> None: + """Hide unfinished capability surfaces from production deployments.""" + from backend import config + + if config.PRODUCTION_MODE or not config.EXPERIMENTAL_ROUTES_ENABLED: + raise HTTPException(status_code=404, detail="Experimental capabilities are unavailable") + + +router = APIRouter( + prefix="/api/v1/experiments/v7", + tags=["experiments"], + dependencies=[Depends(require_experimental_routes_enabled)], +) ROOT = Path(__file__).resolve().parents[3] +def _normalize_request_aliases(data: Any, aliases: dict[str, tuple[str, ...]]) -> Any: + if not isinstance(data, dict): + return data + normalized = dict(data) + for target, candidates in aliases.items(): + if target in normalized: + for candidate in candidates: + normalized.pop(candidate, None) + continue + for candidate in candidates: + if candidate in normalized: + normalized[target] = normalized[candidate] + normalized.pop(candidate, None) + break + return normalized + + class V7WorkbookReviewRequest(BaseModel): model_config = ConfigDict(populate_by_name=True) - session_id: str = Field(alias="sessionId") - plan_id: Optional[str] = Field(default=None, alias="planId") - run_id: Optional[str] = Field(default=None, alias="runId") - workbook_context: dict[str, Any] = Field(default_factory=dict, alias="workbookContext") - workbook_summary: dict[str, Any] = Field(default_factory=dict, alias="workbookSummary") - workbook_fingerprint: str = Field(default="", alias="workbookFingerprint") + session_id: str + plan_id: Optional[str] = None + run_id: Optional[str] = None + workbook_context: dict[str, Any] = Field(default_factory=dict) + workbook_summary: dict[str, Any] = Field(default_factory=dict) + workbook_fingerprint: str = "" formulas: list[dict[str, Any]] = Field(default_factory=list) operations: list[dict[str, Any]] = Field(default_factory=list) documents: list[dict[str, Any]] = Field(default_factory=list) - downstream_outputs: list[str] = Field(default_factory=list, alias="downstreamOutputs") + downstream_outputs: list[str] = Field(default_factory=list) metadata: dict[str, Any] = Field(default_factory=dict) + @model_validator(mode="before") + @classmethod + def _normalize_aliases(cls, data: Any) -> Any: + return _normalize_request_aliases(data, { + "session_id": ("sessionId",), + "plan_id": ("planId",), + "run_id": ("runId",), + "workbook_context": ("workbookContext",), + "workbook_summary": ("workbookSummary",), + "workbook_fingerprint": ("workbookFingerprint",), + "downstream_outputs": ("downstreamOutputs",), + }) + class V7ReplayCaptureRequest(BaseModel): model_config = ConfigDict(populate_by_name=True) - session_id: str = Field(alias="sessionId") - artifact_type: str = Field(default="v7_local_replay", alias="artifactType") + session_id: str + artifact_type: str = "v7_local_replay" action: Literal["capture", "clear_session"] = "capture" payload: dict[str, Any] = Field(default_factory=dict) + @model_validator(mode="before") + @classmethod + def _normalize_aliases(cls, data: Any) -> Any: + return _normalize_request_aliases(data, { + "session_id": ("sessionId",), + "artifact_type": ("artifactType",), + }) + class V7ReplayCaptureResponse(BaseModel): model_config = ConfigDict(populate_by_name=True) @@ -111,14 +161,25 @@ class V7StackReadinessResponse(BaseModel): class BrainApplyEligibilityRequest(BaseModel): model_config = ConfigDict(populate_by_name=True) - session_id: str = Field(alias="sessionId") - plan_id: str = Field(alias="planId") - workbook_fingerprint: str = Field(default="", alias="workbookFingerprint") + session_id: str + plan_id: str + workbook_fingerprint: str = "" simulation: dict[str, Any] = Field(default_factory=dict) - source_artifact_ids: list[str] = Field(default_factory=list, alias="sourceArtifactIds") - rollback_proof_class: str = Field(default="manual", alias="rollbackProofClass") + source_artifact_ids: list[str] = Field(default_factory=list) + rollback_proof_class: str = "manual" metadata: dict[str, Any] = Field(default_factory=dict) + @model_validator(mode="before") + @classmethod + def _normalize_aliases(cls, data: Any) -> Any: + return _normalize_request_aliases(data, { + "session_id": ("sessionId",), + "plan_id": ("planId",), + "workbook_fingerprint": ("workbookFingerprint",), + "source_artifact_ids": ("sourceArtifactIds",), + "rollback_proof_class": ("rollbackProofClass",), + }) + class BrainRunListResponse(BaseModel): model_config = ConfigDict(populate_by_name=True) @@ -132,10 +193,18 @@ class LocalReplayBaselineRequest(BaseModel): model_config = ConfigDict(populate_by_name=True) name: str = "Local Replay Baseline" - session_id: str = Field(default="local-replay-baseline", alias="sessionId") - replay_limit: int = Field(default=100, alias="replayLimit", ge=0, le=1000) + session_id: str = "local-replay-baseline" + replay_limit: int = Field(default=100, ge=0, le=1000) metadata: dict[str, Any] = Field(default_factory=dict) + @model_validator(mode="before") + @classmethod + def _normalize_aliases(cls, data: Any) -> Any: + return _normalize_request_aliases(data, { + "session_id": ("sessionId",), + "replay_limit": ("replayLimit",), + }) + class LocalReplayBaselineListResponse(BaseModel): model_config = ConfigDict(populate_by_name=True) @@ -160,6 +229,7 @@ async def create_v7_workbook_review(request: V7WorkbookReviewRequest) -> V7Workb from backend.services.source_evidence_v3 import get_source_evidence_v3_service from backend.services.workbook_truth_v5 import get_workbook_truth_model_v5_service + get_session_store().get_or_create_session(request.session_id) operations, operation_gaps = _parse_operations(request.operations) workbook_summary = { **(request.workbook_summary or {}), @@ -493,6 +563,7 @@ async def create_v7_replay_baseline(request: LocalReplayBaselineRequest) -> Loca """Pin the current anonymized replay signature buckets as a local baseline.""" from backend.benchmarks.local_replay import evaluate_local_replays + get_session_store().get_or_create_session(request.session_id) evaluation = evaluate_local_replays(limit=request.replay_limit, persist=False) baseline = LocalReplayBaseline( name=request.name, @@ -517,6 +588,7 @@ async def create_v7_replay_baseline(request: LocalReplayBaselineRequest) -> Loca async def capture_v7_replay(request: V7ReplayCaptureRequest) -> V7ReplayCaptureResponse: """Capture or clear local-only anonymized replay artifacts.""" store = get_session_store() + store.get_or_create_session(request.session_id) if request.action == "clear_session": cleared = store.clear_local_replay_artifacts(request.session_id) return V7ReplayCaptureResponse(status="cleared", clearedCount=cleared) @@ -591,7 +663,8 @@ def _parse_operations(items: list[dict[str, Any]]) -> tuple[list[WorkbookOperati def _readiness_history_entry(report: dict[str, Any]) -> V7ReadinessHistoryEntry: - gates = report.get("gates") if isinstance(report.get("gates"), dict) else {} + raw_gates = report.get("gates") + gates = cast(dict[str, Any], raw_gates) if isinstance(raw_gates, dict) else {} gate_statuses = { str(name): str(value.get("status") or "unknown") for name, value in gates.items() @@ -621,6 +694,7 @@ def _stack_readiness() -> V7StackReadinessResponse: _generated_types_freshness as readiness_generated_types_freshness, _office_mock_coverage as readiness_office_mock_coverage, _registry_coverage as readiness_registry_coverage, + _rollback_readiness as readiness_rollback_readiness, _storage_parity as readiness_storage_parity, _taskpane_coverage as readiness_taskpane_coverage, _worker_hardening as readiness_worker_hardening, @@ -651,8 +725,8 @@ def _stack_readiness() -> V7StackReadinessResponse: latest_blockers = [str(item) for item in (latest_report or {}).get("blockers") or []] latest_warnings = [str(item) for item in (latest_report or {}).get("warnings") or []] - backend_blockers = [] - backend_warnings = [] + backend_blockers: list[str] = [] + backend_warnings: list[str] = [] if PLAN_ENGINE_VERSION != "v2": backend_blockers.append("PLAN_ENGINE_VERSION must remain v2 until the final default-switch change.") categories: dict[str, V7StackReadinessCategory] = { @@ -700,12 +774,15 @@ def _stack_readiness() -> V7StackReadinessResponse: taskpane = readiness_taskpane_coverage() office_mock = readiness_office_mock_coverage() + rollback_coverage = readiness_rollback_readiness() generated_types = readiness_generated_types_freshness() plugin_warnings = [] if taskpane.get("status") != "passed": plugin_warnings.append("Taskpane V7 opt-in and flow coverage is incomplete.") if office_mock.get("status") != "passed": plugin_warnings.append("Office mock coverage still has missing success or rollback signals.") + if rollback_coverage.get("status") != "passed": + plugin_warnings.append("Rollback coverage still has missing range or object restore signals.") if generated_types.get("status") != "passed": plugin_warnings.append("Generated OpenAPI taskpane types are missing required V7 schemas.") categories["plugin"] = V7StackReadinessCategory( @@ -714,6 +791,7 @@ def _stack_readiness() -> V7StackReadinessResponse: signals={ "taskpaneCoverage": taskpane, "officeMockCoverage": office_mock, + "rollbackCoverage": rollback_coverage, "generatedTypes": generated_types, }, warnings=plugin_warnings, @@ -761,7 +839,9 @@ def _stack_readiness() -> V7StackReadinessResponse: dependency = readiness_dependency_risk() worker_package = _read_json(ROOT / "workbook-worker" / "package.json") dependency_warnings = [] - if "xlsx" in (worker_package.get("dependencies") or {}): + sheetjs_present = "xlsx" in (worker_package.get("dependencies") or {}) + sheetjs_status = str(dependency.get("sheetjsAdvisoryStatus") or "warning" if sheetjs_present else "not_present") + if sheetjs_present and sheetjs_status != "accepted": dependency_warnings.append("SheetJS/xlsx remains isolated but has a high-severity npm advisory with no npm-published fix.") if dependency.get("status") != "passed": dependency_warnings.append("Third-party dependency posture has unresolved tracked items.") @@ -770,7 +850,7 @@ def _stack_readiness() -> V7StackReadinessResponse: summary="Dependency and audit posture for plugin and workbook-worker release gates.", signals={ **dependency, - "sheetjsAdvisoryStatus": "warning" if dependency_warnings else "passed", + "sheetjsAdvisoryStatus": sheetjs_status, }, warnings=_dedupe(dependency_warnings), ) @@ -1040,17 +1120,28 @@ def _executable_proofs(categories: dict[str, V7StackReadinessCategory], latest_r worker = categories.get("worker", V7StackReadinessCategory()).signals infra = categories.get("infra", V7StackReadinessCategory()).signals readiness = categories.get("readiness", V7StackReadinessCategory()).signals - generated_types = plugin.get("generatedTypes") if isinstance(plugin.get("generatedTypes"), dict) else {} - office_mock = plugin.get("officeMockCoverage") if isinstance(plugin.get("officeMockCoverage"), dict) else {} + raw_generated_types = plugin.get("generatedTypes") + raw_office_mock = plugin.get("officeMockCoverage") + raw_rollback_coverage = plugin.get("rollbackCoverage") + generated_types = cast(dict[str, Any], raw_generated_types) if isinstance(raw_generated_types, dict) else {} + office_mock = cast(dict[str, Any], raw_office_mock) if isinstance(raw_office_mock, dict) else {} + rollback_coverage = cast(dict[str, Any], raw_rollback_coverage) if isinstance(raw_rollback_coverage, dict) else {} + baseline_status = latest_report.get("baselineStatus") + baseline_status = baseline_status if isinstance(baseline_status, dict) else {} + rollback_readiness = latest_report.get("rollbackReadiness") + rollback_readiness = rollback_readiness if isinstance(rollback_readiness, dict) else {} + rollback_proof = rollback_readiness or rollback_coverage + report_metadata = latest_report.get("metadata") + report_metadata = report_metadata if isinstance(report_metadata, dict) else {} proofs = { "officeMockBehavior": _proof_status(office_mock.get("status")), "openApiTypeDrift": _proof_status(generated_types.get("status")), "postgresPersistence": _proof_status(storage.get("status")), "workerSmoke": _proof_status(worker.get("status")), "dockerSmoke": "passed" if infra.get("workerPackagedInBackendImage") and infra.get("productionUvicorn") else "warning", - "replayBaseline": "passed" if readiness.get("status") == "ready" or latest_report.get("baselineStatus", {}).get("status") == "passed" else "warning", - "rollbackRestore": "passed" if latest_report.get("rollbackReadiness", {}).get("status") == "passed" else "warning", - "signedReadinessReport": "passed" if latest_report.get("metadata", {}).get("signed") else "warning", + "replayBaseline": "passed" if readiness.get("status") == "ready" or baseline_status.get("status") == "passed" else "warning", + "rollbackRestore": "passed" if rollback_proof.get("status") == "passed" else "warning", + "signedReadinessReport": "passed" if report_metadata.get("signed") else "warning", } return { "schemaVersion": "2026-05-executable-readiness-proofs", @@ -1563,7 +1654,6 @@ def _anonymized_replay_payload(review: dict[str, Any]) -> dict[str, Any]: "source", "content", "sourcecontent", - "sourcecontent", "source_content", "sourcedocumentcontent", "source_document_content", diff --git a/backend/api/routes/sessions.py b/backend/api/routes/sessions.py index d5e0b2e..bdb09d7 100644 --- a/backend/api/routes/sessions.py +++ b/backend/api/routes/sessions.py @@ -2,18 +2,32 @@ from __future__ import annotations +import copy +import hashlib +import json +import math +import re +import secrets +import time +from decimal import Decimal +from typing import Any, Optional + import structlog from fastapi import APIRouter, HTTPException, Request from fastapi.responses import Response from pydantic import BaseModel, Field -from typing import Optional - +from backend.api.security.ownership import ( + OwnershipNotFoundError, + principal_from_actor, + require_session_access, +) from backend.excel.diff_engine import get_diff_engine -from backend.models.operations import WorkbookOperation, operation_to_legacy +from backend.services.plan_state_machine import InvalidPlanTransition, ensure_transition from backend.storage import get_session_store logger = structlog.get_logger() router = APIRouter(prefix="/api/v1", tags=["sessions"]) +ROLLBACK_CAPABILITY_TTL_SECONDS = 60 def _enforce_session_owner(request: Request, session_id: str) -> None: @@ -24,31 +38,32 @@ def _enforce_session_owner(request: Request, session_id: str) -> None: 404 — using 404 rather than 403 so we don't leak the existence of other users' sessions. """ - user = getattr(request.state, "enterprise_user", None) or getattr(request.state, "user", None) - if not user: - # Auth disabled (dev) — preserve existing single-user UX. - return - role = "" - user_id = "" - if isinstance(user, dict): - role = str(user.get("role") or "") - user_id = str(user.get("user_id") or user.get("email") or "") - else: - role = str(getattr(user, "role", "") or "") - user_id = str(getattr(user, "user_id", "") or getattr(user, "email", "") or "") - if role == "admin" or not user_id: - return + actor = getattr(request.state, "enterprise_user", None) or getattr(request.state, "user", None) + try: + require_session_access( + get_session_store(), + session_id, + actor=actor, + ) + except OwnershipNotFoundError as exc: + raise HTTPException(status_code=404, detail="Session not found") from exc + + +def _enforce_session_delete_owner(request: Request, session_id: str) -> None: + """Authorize deletion retries without restoring access to revoked data.""" store = get_session_store() - owner_fn = getattr(store, "get_session_owner", None) - if not callable(owner_fn): - # Older stores without owner tracking; soft-skip rather than break. - return + actor = getattr(request.state, "enterprise_user", None) or getattr(request.state, "user", None) + principal = principal_from_actor(actor) try: - owner = owner_fn(session_id) - except Exception: - return - if owner and owner != user_id: - raise HTTPException(status_code=404, detail="Session not found") + if not store.session_exists(session_id): + raise OwnershipNotFoundError("Session not found") + if principal.authenticated and not principal.is_admin: + if not principal.user_id or store.get_session_owner(session_id) != principal.user_id: + raise OwnershipNotFoundError("Session not found") + except OwnershipNotFoundError as exc: + raise HTTPException(status_code=404, detail="Session not found") from exc + except Exception as exc: + raise HTTPException(status_code=404, detail="Session not found") from exc # ── Models ─────────────────────────────────────────────────────────────── @@ -76,10 +91,61 @@ class RollbackResponse(BaseModel): session_id: str snapshot_id: str audit_event_id: str + status: str = "rollback_pending" + rollback_token: str + rollback_attempt_id: str + rollback_token_expires_at: float + rollback_operation_registry_hash: str operations: list[dict] = Field(default_factory=list) snapshot: dict = Field(default_factory=dict) +class RollbackAuthorizeRequest(BaseModel): + snapshot_id: str + rollback_token: str + rollback_attempt_id: str + rollback_operation_registry_hash: str + operations: list[dict] + + +class RollbackAuthorizeResponse(BaseModel): + session_id: str + snapshot_id: str + rollback_attempt_id: str + status: str = "authorized" + rollback_capability_id: str + rollback_capability_nonce: str + rollback_capability_expires_at: float + rollback_operation_registry_hash: str + + +class RollbackResultRequest(BaseModel): + snapshot_id: str + rollback_token: str + rollback_attempt_id: str + rollback_capability_id: Optional[str] = None + rollback_capability_nonce: Optional[str] = None + rollback_operation_registry_hash: Optional[str] = None + status: str + office_apply_result: dict = Field(default_factory=dict) + pre_apply_workbook_state: dict = Field(default_factory=dict) + pre_apply_workbook_fingerprint: Optional[str] = None + pre_host_snapshot: dict = Field(default_factory=dict) + post_apply_workbook_state: dict = Field(default_factory=dict) + post_apply_workbook_fingerprint: Optional[str] = None + post_host_snapshot: dict = Field(default_factory=dict) + error_detail: Optional[str] = None + + +class RollbackResultResponse(BaseModel): + session_id: str + snapshot_id: str + plan_id: str + status: str + audit_event_id: str + verification: dict = Field(default_factory=dict) + + # ── Undo/Redo ──────────────────────────────────────────────────────────── @router.post("/sessions/{session_id}/undo", response_model=UndoRedoResponse) @@ -92,9 +158,9 @@ async def undo(session_id: str, request: Request): if not changeset: raise HTTPException(status_code=404, detail="Nothing to undo") - operations = [] + operations: list[dict[str, Any]] = [] for change in changeset.changes: - op = {"cell": change.cell} + op: dict[str, Any] = {"cell": change.cell} if change.change_type == "clear": op["value"] = "" op["type"] = "clear" @@ -128,9 +194,9 @@ async def redo(session_id: str, request: Request): if not changeset: raise HTTPException(status_code=404, detail="Nothing to redo") - operations = [] + operations: list[dict[str, Any]] = [] for change in changeset.changes: - op = {"cell": change.cell} + op: dict[str, Any] = {"cell": change.cell} if change.new_formula: op["formula"] = change.new_formula op["type"] = "formula" @@ -197,6 +263,34 @@ async def list_sessions(limit: int = 20): return {"sessions": store.list_sessions(limit)} +@router.get("/sessions/{session_id}") +async def get_session_metadata(session_id: str, request: Request): + """Return the stable identity of an owned session without creating it.""" + _enforce_session_owner(request, session_id) + if not get_session_store().session_exists(session_id): + raise HTTPException(status_code=404, detail="Session not found") + return {"session_id": session_id} + + +@router.delete("/sessions/{session_id}") +async def delete_session(session_id: str, request: Request): + """Delete an owned session and every lifecycle-registered artifact.""" + _enforce_session_delete_owner(request, session_id) + + from backend.services.lifecycle import LifecycleState, get_lifecycle_coordinator + + receipt = await get_lifecycle_coordinator().purge_session(session_id) + if receipt.state is not LifecycleState.VERIFIED: + raise HTTPException( + status_code=500, + detail={ + "code": "session_deletion_unverified", + "receipt": receipt.to_dict(), + }, + ) + return receipt.to_dict() + + @router.get("/sessions/{session_id}/tokens") async def get_token_usage(session_id: str, request: Request): """Get total token usage for a session.""" @@ -254,6 +348,15 @@ async def rollback_to_snapshot(session_id: str, payload: RollbackRequest, reques snapshot = store.get_snapshot(payload.snapshot_id) if not snapshot or snapshot.get("session_id") != session_id: raise HTTPException(status_code=404, detail="Snapshot not found") + if snapshot.get("state_verified") is False: + raise HTTPException(status_code=409, detail="Snapshot integrity verification failed") + plan_id = str(snapshot.get("plan_id") or "") + plan = store.get_operation_plan(plan_id) if plan_id else None + if not plan: + raise HTTPException(status_code=409, detail="Rollback requires a durable operation plan") + if plan.get("status") != "applied": + raise HTTPException(status_code=409, detail=f"Plan is not eligible for rollback: {plan.get('status')}") + expected_plan = copy.deepcopy(plan) target_state = _extract_cell_snapshot(snapshot.get("workbook_state", {})) current_state = _extract_cell_snapshot(payload.current_workbook_state) @@ -261,25 +364,298 @@ async def rollback_to_snapshot(session_id: str, payload: RollbackRequest, reques # claims exist but the snapshot does not record are ignored to prevent a # malicious caller from steering rollback into untouched ranges. current_state = {cell: current_state[cell] for cell in current_state if cell in target_state} - operations = _build_restore_operations(target_state, current_state) - if snapshot.get("plan_id"): - store.update_operation_plan_status(snapshot["plan_id"], "rolled_back") + durable_operations = (plan.get("metadata") or {}).get("durable_rollback_operations") or [] + operations = _prepare_restore_operations(durable_operations) if durable_operations else _build_restore_operations(target_state, current_state) + if not operations: + raise HTTPException(status_code=409, detail="Snapshot contains no safe restore operations") + try: + ensure_transition("plan", str(plan.get("status") or ""), "rollback_pending", plan_id=plan_id) + except InvalidPlanTransition as exc: + raise HTTPException(status_code=409, detail=str(exc)) from exc + rollback_token = secrets.token_urlsafe(18) + rollback_attempt_id = f"rollback_{secrets.token_hex(7)}" + rollback_token_expires_at = time.time() + 600 + operations = [dict(item) for item in operations] + source_operations = { + str(item.get("id") or ""): item + for item in (plan.get("operations") or []) + if isinstance(item, dict) and str(item.get("id") or "") + } + for operation in operations: + if ( + operation.get("type") == "delete_sheet" + and operation.get("rollbackOnly") is True + and operation.get("rollbackStrategy") == "delete_created_sheet" + ): + source = source_operations.get(str(operation.get("sourceOperationId") or "")) or {} + source_target_value = source.get("target") + source_target = source_target_value if isinstance(source_target_value, dict) else {} + created_sheet = str( + source.get("sheet_name") + or source.get("sheetName") + or source.get("sheet") + or source_target.get("sheet") + or "" + ).strip() + deleted_sheet = str(operation.get("sheet") or operation.get("sheet_name") or "").strip() + if source.get("type") != "create_sheet" or not created_sheet or created_sheet.lower() != deleted_sheet.lower(): + raise HTTPException(status_code=409, detail="Delete-sheet rollback inverse is not bound to its create-sheet source") + operation["rollbackAttemptId"] = rollback_attempt_id + operation_registry_hash = _rollback_operation_registry_hash(operations) + metadata = dict(plan.get("metadata") or {}) + for key in ( + "rollback_capability_id", + "rollback_capability_nonce_hash", + "rollback_capability_operation_registry_hash", + "rollback_capability_issued_at", + "rollback_capability_expires_at", + "rollback_capability_used", + "rollback_capability_consumed_at", + "rollback_capability_result_status", + ): + metadata.pop(key, None) + metadata.update( + { + "rollback_attempt_id": rollback_attempt_id, + "rollback_snapshot_id": payload.snapshot_id, + "rollback_token_hash": _rollback_token_hash(rollback_token), + "rollback_token_expires_at": rollback_token_expires_at, + "rollback_operation_ids": [str(item.get("id") or "") for item in operations], + "rollback_operations": operations, + "rollback_operation_registry_hash": operation_registry_hash, + } + ) + plan["metadata"] = metadata + plan["status"] = "rollback_pending" + if not store.compare_and_swap_operation_plan(plan_id, expected_plan, plan): + raise HTTPException(status_code=409, detail="Operation plan changed before rollback creation") audit_id = store.record_audit_event( session_id=session_id, - event_type="rollback_requested", + event_type="rollback_pending", snapshot_id=payload.snapshot_id, - plan_id=snapshot.get("plan_id"), - payload={"operations": len(operations)}, + plan_id=plan_id, + payload={"operations": len(operations), "rollback_attempt_id": rollback_attempt_id}, ) return RollbackResponse( session_id=session_id, snapshot_id=payload.snapshot_id, audit_event_id=audit_id, - operations=[operation_to_legacy(op) for op in operations], + status="rollback_pending", + rollback_token=rollback_token, + rollback_attempt_id=rollback_attempt_id, + rollback_token_expires_at=rollback_token_expires_at, + rollback_operation_registry_hash=operation_registry_hash, + operations=operations, snapshot={k: v for k, v in snapshot.items() if k != "workbook_state"}, ) +@router.post( + "/sessions/{session_id}/rollback/authorize", + response_model=RollbackAuthorizeResponse, +) +async def authorize_rollback_attempt( + session_id: str, + payload: RollbackAuthorizeRequest, + request: Request, +): + """Issue a short-lived, one-use capability for the exact durable rollback.""" + _enforce_session_owner(request, session_id) + store = get_session_store() + snapshot = store.get_snapshot(payload.snapshot_id) + if not snapshot or snapshot.get("session_id") != session_id: + raise HTTPException(status_code=404, detail="Snapshot not found") + plan_id = str(snapshot.get("plan_id") or "") + plan = store.get_operation_plan(plan_id) if plan_id else None + if not plan or plan.get("status") != "rollback_pending": + raise HTTPException(status_code=409, detail="No pending rollback exists for this snapshot") + expected_plan = copy.deepcopy(plan) + metadata = dict(plan.get("metadata") or {}) + _validate_rollback_attempt( + metadata, + snapshot_id=payload.snapshot_id, + rollback_attempt_id=payload.rollback_attempt_id, + rollback_token=payload.rollback_token, + ) + + durable_operations = [ + dict(item) for item in (metadata.get("rollback_operations") or []) if isinstance(item, dict) + ] + durable_registry_hash = str(metadata.get("rollback_operation_registry_hash") or "") + requested_operations = [dict(item) for item in payload.operations if isinstance(item, dict)] + requested_registry_hash = _rollback_operation_registry_hash(requested_operations) + recomputed_durable_hash = _rollback_operation_registry_hash(durable_operations) + if not durable_operations or len(requested_operations) != len(payload.operations): + raise HTTPException(status_code=409, detail="Rollback operation registry is malformed") + if not ( + durable_registry_hash + and secrets.compare_digest(payload.rollback_operation_registry_hash, durable_registry_hash) + and secrets.compare_digest(requested_registry_hash, durable_registry_hash) + and secrets.compare_digest(recomputed_durable_hash, durable_registry_hash) + ): + raise HTTPException(status_code=409, detail="Rollback operation registry does not match the durable attempt") + _validate_bound_rollback_operations( + plan, + durable_operations, + rollback_attempt_id=payload.rollback_attempt_id, + ) + + now = time.time() + existing_capability_id = str(metadata.get("rollback_capability_id") or "") + existing_capability_used = metadata.get("rollback_capability_used") is True + existing_capability_expires_at = float(metadata.get("rollback_capability_expires_at") or 0) + if existing_capability_id and existing_capability_used: + raise HTTPException(status_code=409, detail="Rollback capability has already been consumed") + if existing_capability_id and now <= existing_capability_expires_at: + raise HTTPException(status_code=409, detail="An unexpired rollback capability is already outstanding") + + rollback_capability_id = f"rollback_cap_{secrets.token_hex(8)}" + rollback_capability_nonce = secrets.token_urlsafe(24) + rollback_capability_expires_at = min( + float(metadata.get("rollback_token_expires_at") or 0), + now + ROLLBACK_CAPABILITY_TTL_SECONDS, + ) + if rollback_capability_expires_at <= now: + raise HTTPException(status_code=403, detail="Rollback token expired") + metadata.update( + { + "rollback_capability_id": rollback_capability_id, + "rollback_capability_nonce_hash": _rollback_capability_hash(rollback_capability_nonce), + "rollback_capability_operation_registry_hash": durable_registry_hash, + "rollback_capability_issued_at": now, + "rollback_capability_expires_at": rollback_capability_expires_at, + "rollback_capability_used": False, + } + ) + plan["metadata"] = metadata + if not store.compare_and_swap_operation_plan(plan_id, expected_plan, plan): + raise HTTPException(status_code=409, detail="Rollback attempt changed during capability issuance") + store.record_audit_event( + session_id=session_id, + event_type="rollback_authorized", + snapshot_id=payload.snapshot_id, + plan_id=plan_id, + payload={ + "rollback_attempt_id": payload.rollback_attempt_id, + "rollback_capability_id": rollback_capability_id, + "rollback_operation_registry_hash": durable_registry_hash, + "rollback_capability_expires_at": rollback_capability_expires_at, + }, + ) + return RollbackAuthorizeResponse( + session_id=session_id, + snapshot_id=payload.snapshot_id, + rollback_attempt_id=payload.rollback_attempt_id, + rollback_capability_id=rollback_capability_id, + rollback_capability_nonce=rollback_capability_nonce, + rollback_capability_expires_at=rollback_capability_expires_at, + rollback_operation_registry_hash=durable_registry_hash, + ) + + +@router.post("/sessions/{session_id}/rollback/result", response_model=RollbackResultResponse) +async def record_rollback_result(session_id: str, payload: RollbackResultRequest, request: Request): + """Consume the Office.js rollback ACK before changing the durable plan state.""" + _enforce_session_owner(request, session_id) + if payload.status not in {"rolled_back", "failed"}: + raise HTTPException(status_code=400, detail="status must be 'rolled_back' or 'failed'") + store = get_session_store() + snapshot = store.get_snapshot(payload.snapshot_id) + if not snapshot or snapshot.get("session_id") != session_id: + raise HTTPException(status_code=404, detail="Snapshot not found") + plan_id = str(snapshot.get("plan_id") or "") + plan = store.get_operation_plan(plan_id) if plan_id else None + if not plan or plan.get("status") != "rollback_pending": + raise HTTPException(status_code=409, detail="No pending rollback exists for this snapshot") + expected_plan = copy.deepcopy(plan) + metadata = dict(plan.get("metadata") or {}) + capability_issued = bool(metadata.get("rollback_capability_id")) + _validate_rollback_attempt( + metadata, + snapshot_id=payload.snapshot_id, + rollback_attempt_id=payload.rollback_attempt_id or "", + rollback_token=payload.rollback_token, + require_unexpired=not capability_issued, + ) + + expected_operations = [ + dict(item) for item in (metadata.get("rollback_operations") or []) if isinstance(item, dict) + ] + _validate_bound_rollback_operations( + plan, + expected_operations, + rollback_attempt_id=payload.rollback_attempt_id or "", + ) + supplied_capability_fields = any( + ( + payload.rollback_capability_id, + payload.rollback_capability_nonce, + payload.rollback_operation_registry_hash, + ) + ) + if capability_issued: + _validate_rollback_capability(metadata, payload, expected_operations) + elif payload.status == "rolled_back": + raise HTTPException(status_code=409, detail="Rollback authorization is required before Office mutation") + elif supplied_capability_fields: + raise HTTPException(status_code=409, detail="Rollback capability does not match the pending attempt") + + verification = _verify_rollback_result( + expected_operations, + payload.office_apply_result, + payload.pre_apply_workbook_state, + payload.pre_apply_workbook_fingerprint, + payload.pre_host_snapshot, + payload.post_apply_workbook_state, + payload.post_apply_workbook_fingerprint, + payload.post_host_snapshot, + expected_registry_hash=str(metadata.get("rollback_operation_registry_hash") or ""), + rollback_attempt_id=str(metadata.get("rollback_attempt_id") or ""), + ) + proof_verified = payload.status == "rolled_back" and verification["status"] == "verified" + next_plan_status = "rolled_back" if proof_verified else "applied" + try: + ensure_transition("plan", "rollback_pending", next_plan_status, plan_id=plan_id) + except InvalidPlanTransition as exc: + raise HTTPException(status_code=409, detail=str(exc)) from exc + metadata["rollback_completed_at"] = time.time() + metadata["rollback_result"] = verification + metadata["rollback_error_detail"] = payload.error_detail or ( + "Rollback proof verification failed" if payload.status == "rolled_back" and not proof_verified else None + ) + if capability_issued: + metadata["rollback_capability_used"] = True + metadata["rollback_capability_consumed_at"] = time.time() + metadata["rollback_capability_result_status"] = payload.status + metadata.pop("rollback_token_hash", None) + plan["metadata"] = metadata + plan["status"] = next_plan_status + if not store.compare_and_swap_operation_plan(plan_id, expected_plan, plan): + raise HTTPException(status_code=409, detail="Rollback attempt changed before result consumption") + response_status = "rolled_back" if next_plan_status == "rolled_back" else "rollback_failed" + audit_id = store.record_audit_event( + session_id=session_id, + event_type=response_status, + snapshot_id=payload.snapshot_id, + plan_id=plan_id, + payload={ + "rollback_attempt_id": metadata.get("rollback_attempt_id"), + "rollback_capability_id": metadata.get("rollback_capability_id"), + "rollback_operation_registry_hash": metadata.get("rollback_operation_registry_hash"), + "verification": verification, + "error_detail": metadata.get("rollback_error_detail"), + }, + ) + return RollbackResultResponse( + session_id=session_id, + snapshot_id=payload.snapshot_id, + plan_id=plan_id, + status=response_status, + audit_event_id=audit_id, + verification=verification, + ) + + def _extract_cell_snapshot(payload: dict) -> dict: if not payload: return {} @@ -289,31 +665,365 @@ def _extract_cell_snapshot(payload: dict) -> dict: return state if isinstance(state, dict) else {} -def _build_restore_operations(target_state: dict, current_state: dict) -> list[WorkbookOperation]: - operations: list[WorkbookOperation] = [] - all_cells = set(target_state.keys()) | set(current_state.keys()) - for cell in sorted(all_cells): +def _build_restore_operations(target_state: dict, current_state: dict) -> list[dict]: + operations: list[dict] = [] + # Only snapshot-owned cells can be restored. Explicit blank entries are + # retained so cells filled after the snapshot are actively cleared. + for cell in sorted(target_state.keys()): target = target_state.get(cell) - if target is None: - operations.append( - WorkbookOperation( - type="clear_range", - range=cell, - intent="rollback clear cell not present in snapshot", - risk_level="high", - ) - ) - continue - formula = target.get("formula") if isinstance(target, dict) else None - value = target.get("value") if isinstance(target, dict) else target + info = target if isinstance(target, dict) else {"value": target} + parsed_sheet, address = _split_qualified_cell(str(cell), str(info.get("sheet") or "")) + formula = info.get("formula") + value = info.get("value") + number_format = info.get("format") operations.append( - WorkbookOperation( - type="write_cell", - range=cell, - formula=formula, - value=None if formula else value, - intent="rollback restore cell", - risk_level="medium", - ) + { + "id": _rollback_operation_id(parsed_sheet, address), + "type": "rollback_restore", + "sheet": parsed_sheet, + "range": address, + "values": [[value if value is not None else ""]], + "formulas": [[formula if isinstance(formula, str) and formula.startswith("=") else ""]], + "numberFormats": [[number_format if number_format is not None else "General"]], + "intent": "rollback restore cell", + } ) return operations + + +def _prepare_restore_operations(operations: list[dict]) -> list[dict]: + prepared: list[dict] = [] + for index, raw in enumerate(operations): + if not isinstance(raw, dict): + continue + operation = dict(raw) + if operation.get("type") == "delete_sheet" and not ( + operation.get("rollbackOnly") is True + and operation.get("rollbackStrategy") == "delete_created_sheet" + and operation.get("sourceOperationId") + ): + continue + operation["id"] = f"rollback_inverse_{index + 1}" + operation.setdefault("range", "A1") + prepared.append(operation) + return prepared + + +def _split_qualified_cell(cell: str, fallback_sheet: str = "") -> tuple[str, str]: + if "!" not in cell: + return fallback_sheet, cell.replace("$", "") + sheet, address = cell.rsplit("!", 1) + return sheet.strip("'"), address.replace("$", "") + + +def _rollback_operation_id(sheet: str, address: str) -> str: + suffix = re.sub(r"[^A-Za-z0-9]+", "_", f"{sheet}_{address}").strip("_") + return f"rollback_{suffix or 'cell'}" + + +def _rollback_token_hash(token: str) -> str: + return hashlib.sha256(token.encode("utf-8")).hexdigest() + + +def _rollback_capability_hash(nonce: str) -> str: + return hashlib.sha256(f"rollback-capability-v1:{nonce}".encode("utf-8")).hexdigest() + + +def _validate_rollback_attempt( + metadata: dict, + *, + snapshot_id: str, + rollback_attempt_id: str, + rollback_token: str, + require_unexpired: bool = True, +) -> None: + if metadata.get("rollback_snapshot_id") != snapshot_id: + raise HTTPException(status_code=409, detail="Rollback snapshot does not match the pending attempt") + if not rollback_attempt_id or rollback_attempt_id != metadata.get("rollback_attempt_id"): + raise HTTPException(status_code=409, detail="Rollback attempt id does not match") + expected_hash = str(metadata.get("rollback_token_hash") or "") + if not expected_hash or not secrets.compare_digest(_rollback_token_hash(rollback_token), expected_hash): + raise HTTPException(status_code=403, detail="Invalid rollback token") + if require_unexpired and time.time() > float(metadata.get("rollback_token_expires_at") or 0): + raise HTTPException(status_code=403, detail="Rollback token expired") + + +def _validate_bound_rollback_operations( + plan: dict, + operations: list[dict], + *, + rollback_attempt_id: str, +) -> None: + operation_ids = [str(item.get("id") or "") for item in operations] + if not operations or any(not operation_id for operation_id in operation_ids): + raise HTTPException(status_code=409, detail="Rollback operation registry contains a missing operation id") + if len(set(operation_ids)) != len(operation_ids): + raise HTTPException(status_code=409, detail="Rollback operation registry contains duplicate operation ids") + + source_operations = { + str(item.get("id") or ""): item + for item in (plan.get("operations") or []) + if isinstance(item, dict) and str(item.get("id") or "") + } + for operation in operations: + if operation.get("type") != "delete_sheet": + continue + source_operation_id = str(operation.get("sourceOperationId") or "") + source = source_operations.get(source_operation_id) or {} + source_target_value = source.get("target") + source_target = source_target_value if isinstance(source_target_value, dict) else {} + source_sheet = str( + source.get("sheet_name") + or source.get("sheetName") + or source.get("sheet") + or source_target.get("sheet") + or "" + ).strip() + rollback_sheet = str(operation.get("sheet") or operation.get("sheet_name") or "").strip() + if not ( + operation.get("rollbackOnly") is True + and operation.get("rollbackStrategy") == "delete_created_sheet" + and str(operation.get("rollbackAttemptId") or "") == rollback_attempt_id + and source.get("type") == "create_sheet" + and source_operation_id + and source_sheet + and rollback_sheet + and source_sheet.lower() == rollback_sheet.lower() + ): + raise HTTPException( + status_code=409, + detail="Delete-sheet rollback inverse is not bound to this attempt and its create-sheet source", + ) + + +def _validate_rollback_capability( + metadata: dict, + payload: RollbackResultRequest, + durable_operations: list[dict], +) -> None: + expected_capability_id = str(metadata.get("rollback_capability_id") or "") + expected_nonce_hash = str(metadata.get("rollback_capability_nonce_hash") or "") + expected_registry_hash = str(metadata.get("rollback_operation_registry_hash") or "") + capability_registry_hash = str(metadata.get("rollback_capability_operation_registry_hash") or "") + durable_registry_hash = _rollback_operation_registry_hash(durable_operations) + if metadata.get("rollback_capability_used") is True: + raise HTTPException(status_code=409, detail="Rollback capability has already been consumed") + if float(metadata.get("rollback_capability_expires_at") or 0) <= 0: + raise HTTPException(status_code=409, detail="Rollback capability expiry is missing") + if not payload.rollback_capability_id or not secrets.compare_digest( + payload.rollback_capability_id, + expected_capability_id, + ): + raise HTTPException(status_code=409, detail="Rollback capability id does not match") + if not payload.rollback_capability_nonce or not expected_nonce_hash or not secrets.compare_digest( + _rollback_capability_hash(payload.rollback_capability_nonce), + expected_nonce_hash, + ): + raise HTTPException(status_code=403, detail="Invalid rollback capability nonce") + if not ( + payload.rollback_operation_registry_hash + and expected_registry_hash + and secrets.compare_digest(payload.rollback_operation_registry_hash, expected_registry_hash) + and secrets.compare_digest(capability_registry_hash, expected_registry_hash) + and secrets.compare_digest(durable_registry_hash, expected_registry_hash) + ): + raise HTTPException(status_code=409, detail="Rollback capability registry does not match the durable attempt") + + +def _rollback_operation_registry_hash(operations: list[dict]) -> str: + normalized = _canonical_rollback_registry_value(operations) + canonical = json.dumps( + normalized, + ensure_ascii=False, + sort_keys=True, + separators=(",", ":"), + ) + return hashlib.sha256(canonical.encode("utf-8")).hexdigest() + + +def _canonical_rollback_registry_value(value: Any) -> list[Any]: + """Build an unambiguous JSON tree with number spelling normalized. + + Browser JSON parsing collapses ``1.0`` to ``1`` and may switch between + fixed and exponent notation. Tagging every JSON type and canonicalizing + finite numbers by decimal value keeps the registry stable across that + round trip while preserving the distinction between booleans and numbers. + """ + if value is None: + return ["null"] + if isinstance(value, bool): + return ["boolean", value] + if isinstance(value, int): + number = Decimal(value) + return ["number", format(number, "f")] + if isinstance(value, float): + if not math.isfinite(value): + raise ValueError("Rollback operation registry contains a non-finite number") + number = Decimal(str(value)) + if number == 0: + return ["number", "0"] + return ["number", format(number.normalize(), "f")] + if isinstance(value, str): + return ["string", value] + if isinstance(value, list): + return ["array", [_canonical_rollback_registry_value(item) for item in value]] + if isinstance(value, dict): + if any(not isinstance(key, str) for key in value): + raise ValueError("Rollback operation registry contains a non-string object key") + return [ + "object", + [ + [key, _canonical_rollback_registry_value(value[key])] + for key in sorted(value) + ], + ] + raise ValueError(f"Rollback operation registry contains unsupported value type: {type(value).__name__}") + + +def _verify_rollback_result( + expected_operations: list[dict], + office_apply_result: dict, + pre_apply_workbook_state: Optional[dict] = None, + pre_apply_workbook_fingerprint: Optional[str] = None, + pre_host_snapshot: Optional[dict] = None, + post_apply_workbook_state: Optional[dict] = None, + post_apply_workbook_fingerprint: Optional[str] = None, + post_host_snapshot: Optional[dict] = None, + *, + expected_registry_hash: str = "", + rollback_attempt_id: str = "", +) -> dict: + from backend.api.routes.tasks import _workbook_target_binding + + operations = [dict(item) for item in expected_operations if isinstance(item, dict)] + expected_ids = [str(item.get("id") or "") for item in operations] + expected_by_id = { + str(item.get("id") or ""): item + for item in operations + if str(item.get("id") or "") + } + results_value = office_apply_result.get("results") if isinstance(office_apply_result, dict) else [] + raw_results = results_value if isinstance(results_value, list) else [] + rows = [item for item in raw_results if isinstance(item, dict)] + by_id: dict[str, list[dict]] = {} + failures: list[str] = [] + if not isinstance(results_value, list): + failures.append("Office.js rollback results must be a list") + if len(rows) != len(raw_results): + failures.append("Office.js rollback results contain a non-object row") + if len(raw_results) != len(expected_ids): + failures.append( + f"Office.js returned {len(raw_results)} result rows; expected exactly {len(expected_ids)}" + ) + for index, item in enumerate(rows): + operation_id = str(item.get("operationId") or item.get("operation_id") or "") + if not operation_id.strip(): + failures.append(f"Office.js rollback result row {index + 1} has a missing operation ID") + by_id.setdefault(operation_id, []).append(item) + duplicate_result_ids = sorted( + operation_id + for operation_id, matches in by_id.items() + if operation_id.strip() and len(matches) > 1 + ) + if duplicate_result_ids: + failures.append( + f"Duplicate operation IDs in rollback results: {', '.join(duplicate_result_ids[:5])}" + ) + for operation_id in expected_ids: + matches = by_id.get(operation_id, []) + if len(matches) != 1: + failures.append(f"{operation_id} returned {len(matches)} results") + continue + result = matches[0] + if any(key in result for key in ("type", "operationType", "operation_type")): + expected_type = str(expected_by_id.get(operation_id, {}).get("type") or "") + result_type = str( + result.get("type") + or result.get("operationType") + or result.get("operation_type") + or "" + ) + if not expected_type or result_type != expected_type: + failures.append( + f"{operation_id} result type {result_type or ''} does not match " + f"{expected_type or ''}" + ) + if result.get("status") != "success": + failures.append(f"{operation_id} reported {result.get('status') or 'missing'}") + if not (result.get("dirtyRanges") or result.get("dirty_ranges")): + failures.append(f"{operation_id} returned no dirty range") + unexpected = [operation_id for operation_id in by_id if operation_id not in expected_ids] + if unexpected: + formatted_unexpected = [operation_id or "" for operation_id in unexpected[:5]] + failures.append(f"Unexpected rollback results: {', '.join(formatted_unexpected)}") + applied = office_apply_result.get("applied") if isinstance(office_apply_result, dict) else None + if isinstance(applied, bool) or not isinstance(applied, int): + failures.append("Office.js rollback result must include an integer applied count") + elif applied != len(expected_ids): + failures.append(f"Office.js reported {applied} applied restores; expected {len(expected_ids)}") + if isinstance(office_apply_result, dict) and office_apply_result.get("warnings"): + failures.append("Office.js returned rollback warnings") + if isinstance(office_apply_result, dict) and (office_apply_result.get("error") or office_apply_result.get("errors")): + failures.append("Office.js returned a rollback error") + if not expected_ids or any(not operation_id for operation_id in expected_ids): + failures.append("The durable rollback attempt contains missing operation IDs") + if len(set(expected_ids)) != len(expected_ids): + failures.append("The durable rollback attempt contains duplicate operation IDs") + + registry_hash = _rollback_operation_registry_hash(operations) + if not expected_registry_hash or not secrets.compare_digest(registry_hash, expected_registry_hash): + failures.append("The rollback operation registry does not match the durable attempt") + + pre_snapshot = pre_host_snapshot if isinstance(pre_host_snapshot, dict) else {} + post_snapshot = post_host_snapshot if isinstance(post_host_snapshot, dict) else {} + for label, snapshot in (("Pre-apply", pre_snapshot), ("Post-apply", post_snapshot)): + if ( + snapshot.get("schemaVersion") != "2026-08-host-snapshot-v2" + or snapshot.get("complete") is not True + or snapshot.get("truncated") is True + or bool(snapshot.get("errors")) + or not str(snapshot.get("fingerprint") or "") + ): + failures.append(f"{label} HostSnapshotV2 proof is incomplete") + + pre_state = pre_apply_workbook_state if isinstance(pre_apply_workbook_state, dict) else {} + post_state = post_apply_workbook_state if isinstance(post_apply_workbook_state, dict) else {} + pre_binding = _workbook_target_binding(pre_state, operations) + post_binding = _workbook_target_binding(post_state, operations, require_post_match=True) + failures.extend(f"Pre-apply proof: {item}" for item in pre_binding["errors"][:10]) + failures.extend(f"Post-apply proof: {item}" for item in post_binding["errors"][:10]) + if not pre_apply_workbook_fingerprint: + failures.append("Pre-apply workbook fingerprint is missing") + if not post_apply_workbook_fingerprint: + failures.append("Post-apply workbook fingerprint is missing") + + pre_sheet_names = {str(item).strip().lower() for item in (pre_state.get("sheet_names") or [])} + for operation in operations: + if operation.get("type") != "delete_sheet": + continue + expected_sheet = str(operation.get("sheet") or operation.get("sheet_name") or "").strip().lower() + if not expected_sheet or expected_sheet not in pre_sheet_names: + failures.append(f"Pre-apply proof: created sheet {expected_sheet or ''} is absent") + if not ( + operation.get("rollbackOnly") is True + and operation.get("rollbackStrategy") == "delete_created_sheet" + and str(operation.get("rollbackAttemptId") or "") == rollback_attempt_id + ): + failures.append("Delete-sheet inverse is not bound to this rollback attempt") + return { + "status": "failed" if failures else "verified", + "expected_operations": len(expected_ids), + "office_results": len(rows), + "failures": failures[:25], + "proof": { + "schemaVersion": "2026-08-plan-rollback-proof-v1", + "rollbackAttemptId": rollback_attempt_id, + "operationRegistryHash": registry_hash, + "preWorkbookFingerprint": pre_apply_workbook_fingerprint or "", + "postWorkbookFingerprint": post_apply_workbook_fingerprint or "", + "preHostFingerprint": str(pre_snapshot.get("fingerprint") or ""), + "postHostFingerprint": str(post_snapshot.get("fingerprint") or ""), + "preTargetHashes": pre_binding["target_hashes"], + "postTargetHashes": post_binding["target_hashes"], + }, + } diff --git a/backend/api/routes/tasks.py b/backend/api/routes/tasks.py index 1c4349a..a784413 100644 --- a/backend/api/routes/tasks.py +++ b/backend/api/routes/tasks.py @@ -6,23 +6,23 @@ import base64 import hashlib import json +import re import time import uuid import secrets -from typing import Any, Optional +from typing import Any, Literal, Optional import structlog -from fastapi import APIRouter, HTTPException, WebSocket, WebSocketDisconnect -from pydantic import BaseModel, Field +from fastapi import APIRouter, HTTPException, Request, Response, WebSocket, WebSocketDisconnect +from pydantic import BaseModel, ConfigDict, Field, model_validator from backend.agents.pipeline import run_pipeline, classify_complexity +from backend.api.security.ownership import require_session_access from backend.excel.diff_engine import get_diff_engine from backend.llm.router import router as llm_router from backend.models.operations import ( CitationMetadata, FormulaAnalysisResult, - FormulaGraphArtifact, - FormulaGraphArtifactV2, FormulaGraphArtifactV3, FormulaGraphArtifactV4, FormulaGraphArtifactV5, @@ -31,8 +31,6 @@ OperationPlan, OperationPlanV2, OperationPlanV3, - OperationSimulationV2, - OperationSimulationV3, OperationSimulationV4, OperationSimulationV5, OperationSimulationV6, @@ -42,17 +40,26 @@ WorkbookOperation, WorkbookOperationV2, WorkbookSemanticMap, - WorkbookTruthModel, - WorkbookTruthModelV2, WorkbookTruthModelV3, WorkbookTruthModelV4, WorkbookTruthModelV5, ) +from backend.operations.work_admission import WorkBudgetError, WorkExecutionTimeout, run_cancellable_in_thread from backend.storage import get_session_store from backend.services.operation_planner import build_operation_plan, dump_plan from backend.services.plan_intelligence import plan_to_v2_with_intelligence -from backend.services.operation_validation import refresh_plan_validation, refresh_plan_validation_v2 -from backend.config import ALLOW_LEGACY_APPLY_FINAL_STATUS, APPLY_TOKEN_TTL_SECONDS, PLAN_ENGINE_VERSION +from backend.services.operation_validation import ( + refresh_plan_validation, + refresh_plan_validation_v2, + validate_workbook_operations_v2, +) +from backend.services.plan_state_machine import InvalidPlanTransition, ensure_transition +from backend.config import ( + ALLOW_LEGACY_APPLY_FINAL_STATUS, + APPLY_TOKEN_TTL_SECONDS, + BRAIN_APPLY_ELIGIBILITY_MANDATORY, + PLAN_ENGINE_VERSION, +) logger = structlog.get_logger() router = APIRouter(prefix="/api/v1", tags=["tasks"]) @@ -67,6 +74,77 @@ } +async def _run_spreadsheet_worker(command: str, payload: dict[str, Any], operation: str) -> Any: + from backend.config import WORKBOOK_WORKER_TIMEOUT_SECONDS + from backend.excel.workbook_worker import WorkbookWorkerUnavailable, run_worker + + try: + worker = await asyncio.wait_for( + asyncio.to_thread(run_worker, command, payload), + timeout=WORKBOOK_WORKER_TIMEOUT_SECONDS, + ) + except asyncio.TimeoutError: + raise HTTPException(status_code=504, detail=f"{operation} timed out") from None + except WorkbookWorkerUnavailable as exc: + raise HTTPException(status_code=503, detail=f"{operation} worker unavailable: {exc}") from exc + except Exception as exc: + raise HTTPException(status_code=400, detail=f"{operation} failed: {exc}") from exc + if not worker.ok: + raise HTTPException(status_code=400, detail=worker.error or f"{operation} failed") + return worker.result + + +def _spreadsheet_worker_payload(request: BaseModel, fields: dict[str, Any]) -> dict[str, Any]: + payload = dict(request.model_extra or {}) + payload.update(fields) + return payload + + +def _request_actor(request: Request) -> Any | None: + return getattr(request.state, "enterprise_user", None) or getattr(request.state, "user", None) + + +def _claim_request_session(request: Request, session_id: str) -> None: + require_session_access( + get_session_store(), + session_id, + actor=_request_actor(request), + create=True, + ) + + +def _enforce_ai_policy( + mode: str, + provider: str | None = None, + source_document_ids: list[str] | None = None, + actor: Any | None = None, +) -> None: + from backend.api.security.ai_policy import AiPolicyViolation, get_effective_ai_policy + + try: + get_effective_ai_policy().require_request( + mode=mode, + provider=provider, + source_document_ids=source_document_ids, + actor=actor, + ) + except AiPolicyViolation as exc: + raise HTTPException(status_code=403, detail=exc.to_detail()) from exc + + +def _enforce_export_policy(request: "SpreadsheetExportRequest", actor: Any | None = None) -> None: + from backend.api.security.ai_policy import AiPolicyViolation, get_effective_ai_policy + + try: + get_effective_ai_policy().require_export_fallback( + allow_regenerated_fallback=request.allowRegeneratedFallback, + allow_macro_regenerated_fallback=request.allowMacroRegeneratedFallback, + actor=actor, + ) + except AiPolicyViolation as exc: + raise HTTPException(status_code=403, detail=exc.to_detail()) from exc + + # ── Models ─────────────────────────────────────────────────────────────── class WorkbookContext(BaseModel): @@ -146,7 +224,12 @@ class ApplyResultRequest(BaseModel): apply_attempt_id: Optional[str] = None status: str = "applied" office_apply_result: dict = Field(default_factory=dict) + pre_apply_workbook_state: dict = Field(default_factory=dict) + pre_apply_workbook_fingerprint: Optional[str] = None + pre_host_snapshot: dict = Field(default_factory=dict) post_apply_workbook_state: dict = Field(default_factory=dict) + post_apply_workbook_fingerprint: Optional[str] = None + post_host_snapshot: dict = Field(default_factory=dict) error_detail: Optional[str] = None @@ -237,6 +320,7 @@ class OperationSimulationV2Request(BaseModel): class SourceEvidenceV2Request(BaseModel): session_id: str documents: list[dict] = Field(default_factory=list) + document_ids: list[str] = Field(default_factory=list) workbook_context: dict = Field(default_factory=dict) downstream_outputs: list[str] = Field(default_factory=list) @@ -257,16 +341,141 @@ class WorkbookDiffRequest(BaseModel): class WorkbookJobRequest(BaseModel): - command: str = Field(..., pattern="^(ingest|export|diff|formula-analyze|semantic-map|export-qa|semantic-map-v2|formula-graph|source-extract|simulate-operations|openxml-qa|export-template|roundtrip-qa|truth-model-v4|formula-graph-v4|simulate-operations-v5|source-evidence-v2|openxml-deep-qa|fixture-generate|workbook-diff-v2|truth-model-v5|formula-graph-v5|simulate-operations-v6|source-evidence-v3|openxml-deep-qa-v2|workbook-diff-v3|fixture-validate)$") + command: str = Field(..., pattern="^(ingest|open-workbook-v1|export|export-workbook-v1|diff|formula-analyze|semantic-map|export-qa|semantic-map-v2|formula-graph|source-extract|simulate-operations|openxml-qa|export-template|roundtrip-qa|roundtrip-qa-v2|truth-model-v4|formula-graph-v4|simulate-operations-v5|source-evidence-v2|openxml-deep-qa|fixture-generate|workbook-diff-v2|truth-model-v5|formula-graph-v5|simulate-operations-v6|source-evidence-v3|openxml-deep-qa-v2|workbook-diff-v3|fixture-validate)$") payload: dict = Field(default_factory=dict) + session_id: Optional[str] = None + account_id: Optional[str] = None + + +def _normalize_request_aliases(data: Any, aliases: dict[str, tuple[str, ...]]) -> Any: + if not isinstance(data, dict): + return data + normalized = dict(data) + for target, candidates in aliases.items(): + if target in normalized: + for candidate in candidates: + normalized.pop(candidate, None) + continue + for candidate in candidates: + if candidate in normalized: + normalized[target] = normalized[candidate] + normalized.pop(candidate, None) + break + return normalized + + +class SpreadsheetOpenRequest(BaseModel): + model_config = ConfigDict(populate_by_name=True, extra="allow") + + filename: str = "workbook.xlsx" + content_base64: str + title: Optional[str] = None + workbook_id: Optional[str] = None + + @model_validator(mode="before") + @classmethod + def _normalize_aliases(cls, data: Any) -> Any: + return _normalize_request_aliases(data, { + "content_base64": ("contentBase64",), + "workbook_id": ("workbookId",), + }) + + +class SpreadsheetExportRequest(BaseModel): + model_config = ConfigDict(populate_by_name=True, extra="allow") + + session: dict = Field(default_factory=dict) + title: Optional[str] = None + filename: str = "finance-ai-workbook.xlsx" + fidelityMode: Optional[str] = None + allowRegeneratedFallback: bool = False + allowMacroRegeneratedFallback: bool = False + preferOriginal: Optional[bool] = None + + @model_validator(mode="before") + @classmethod + def _normalize_aliases(cls, data: Any) -> Any: + return _normalize_request_aliases(data, { + "fidelityMode": ("fidelity_mode",), + "allowRegeneratedFallback": ("allow_regenerated_fallback",), + "allowMacroRegeneratedFallback": ("allow_macro_regenerated_fallback",), + "preferOriginal": ("prefer_original",), + }) + + +class SpreadsheetRoundtripQaRequest(BaseModel): + model_config = ConfigDict(populate_by_name=True, extra="allow") + + session: dict = Field(default_factory=dict) + filename: str = "workbook.xlsx" + title: Optional[str] = None + content_base64: Optional[str] = None + fidelityMode: Optional[str] = None + allowRegeneratedFallback: bool = False + + @model_validator(mode="before") + @classmethod + def _normalize_aliases(cls, data: Any) -> Any: + return _normalize_request_aliases(data, { + "content_base64": ("contentBase64",), + "fidelityMode": ("fidelity_mode",), + "allowRegeneratedFallback": ("allow_regenerated_fallback",), + }) + + +class SpreadsheetOpenResponse(BaseModel): + model_config = ConfigDict(populate_by_name=True, extra="allow") + + schema_version: Optional[str] = Field(default=None, alias="schemaVersion") + workbook_id: Optional[str] = Field(default=None, alias="workbookId") + revision: Optional[int] = None + filename: Optional[str] = None + title: Optional[str] = None + active_sheet_id: Optional[str] = Field(default=None, alias="activeSheetId") + sheets: list[dict[str, Any]] = Field(default_factory=list) + unsupported_features: list[dict[str, Any]] = Field(default_factory=list, alias="unsupportedFeatures") + fidelity_report: Optional[dict[str, Any]] = Field(default=None, alias="fidelityReport") + + +class SpreadsheetRoundtripQaResponse(BaseModel): + model_config = ConfigDict(populate_by_name=True, extra="allow") + + schema_version: Optional[str] = Field(default=None, alias="schemaVersion") + status: Optional[str] = None + warnings: list[str] = Field(default_factory=list) + blockers: list[str] = Field(default_factory=list) + fidelity: dict[str, Any] = Field(default_factory=dict) + package_integrity: Optional[dict[str, Any]] = Field(default=None, alias="packageIntegrity") + trust_report: Optional[dict[str, Any]] = Field(default=None, alias="trustReport") + + +SPREADSHEET_EXPORT_RESPONSES = { + 200: { + "description": "XLSX/XLSM workbook bytes exported from a browser workbook session.", + "content": { + "application/vnd.openxmlformats-officedocument.spreadsheetml.sheet": { + "schema": {"type": "string", "format": "binary"} + } + }, + "headers": { + "Content-Disposition": {"description": "Workbook attachment filename.", "schema": {"type": "string"}}, + "X-Spreadsheet-Engine": {"description": "Workbook worker/export engine identifier.", "schema": {"type": "string"}}, + "X-Spreadsheet-Fidelity-Mode": {"description": "Export fidelity mode used by the worker.", "schema": {"type": "string"}}, + "X-Spreadsheet-Fidelity-Status": {"description": "Preservation/QA status reported by the worker.", "schema": {"type": "string"}}, + "X-Spreadsheet-Unsupported-Preserved": {"description": "Whether unsupported OOXML package parts were preserved.", "schema": {"type": "string"}}, + }, + } +} # ── POST /tasks/plan ──────────────────────────────────────────────────── @router.post("/tasks/plan", response_model=OperationPlan) -async def create_operation_plan(request: PlanRequest): +async def create_operation_plan(request: PlanRequest, http_request: Request): """Create a validated, previewable operation plan without editing Excel.""" session_id = request.session_id or str(uuid.uuid4())[:12] + _claim_request_session(http_request, session_id) + _enforce_ai_policy("preview", request.provider, request.source_document_ids, _request_actor(http_request)) try: plan = await build_operation_plan( prompt=request.prompt, @@ -303,17 +512,19 @@ async def create_operation_plan(request: PlanRequest): return plan except Exception as e: logger.error("operation_plan_failed", error=str(e), session_id=session_id) - raise HTTPException(status_code=500, detail=str(e)) + raise HTTPException(status_code=500, detail=str(e)) from e @router.post("/tasks/plan/v2", response_model=OperationPlanV2) -async def create_operation_plan_v2(request: PlanRequest): +async def create_operation_plan_v2(request: PlanRequest, http_request: Request): """Create a workbook-native V2 plan with inspection ops prepended. The legacy plan is still persisted for existing apply flows, while this endpoint exposes the richer read-before-write contract to newer clients. """ session_id = request.session_id or str(uuid.uuid4())[:12] + _claim_request_session(http_request, session_id) + _enforce_ai_policy("preview", request.provider, request.source_document_ids, _request_actor(http_request)) try: plan = await build_operation_plan( prompt=request.prompt, @@ -352,7 +563,7 @@ async def create_operation_plan_v2(request: PlanRequest): return v2_plan except Exception as e: logger.error("operation_plan_v2_failed", error=str(e), session_id=session_id) - raise HTTPException(status_code=500, detail=str(e)) + raise HTTPException(status_code=500, detail=str(e)) from e @router.post("/workbooks/map", response_model=WorkbookSemanticMap) @@ -360,6 +571,7 @@ async def create_workbook_semantic_map(request: WorkbookMapRequest): """Build and persist a workbook semantic map from live/offline evidence.""" from backend.services.semantic_map import get_semantic_map_service + get_session_store().get_or_create_session(request.session_id) semantic_map = get_semantic_map_service().build( session_id=request.session_id, workbook_context=request.workbook_context, @@ -384,12 +596,14 @@ async def create_workbook_semantic_map(request: WorkbookMapRequest): @router.post("/tasks/plan/v3", response_model=OperationPlanV3) -async def create_operation_plan_v3(request: PlanRequest): +async def create_operation_plan_v3(request: PlanRequest, http_request: Request): """Create an accuracy-first V3 plan with semantic map backed planning.""" from backend.services.plan_engine_v3 import get_plan_engine_v3 from backend.services.semantic_map import get_semantic_map_service session_id = request.session_id or str(uuid.uuid4())[:12] + _claim_request_session(http_request, session_id) + _enforce_ai_policy("preview", request.provider, request.source_document_ids, _request_actor(http_request)) try: plan = await build_operation_plan( prompt=request.prompt, @@ -499,19 +713,20 @@ async def create_operation_plan_v3(request: PlanRequest): return v3_plan except Exception as e: logger.error("operation_plan_v3_failed", error=str(e), session_id=session_id) - raise HTTPException(status_code=500, detail=str(e)) + raise HTTPException(status_code=500, detail=str(e)) from e # ── POST /tasks/apply ─────────────────────────────────────────────────── @router.post("/tasks/apply", response_model=ApplyPlanResponse) -async def approve_operation_plan(request: ApplyPlanRequest): +async def approve_operation_plan(request: ApplyPlanRequest, http_request: Request): """Approve a plan and persist the pre-apply snapshot for rollback. The backend records approval and returns the typed operations. Office.js still performs the workbook mutation client-side because only the add-in can access the workbook. """ + _enforce_ai_policy("action", actor=_request_actor(http_request)) store = get_session_store() plan = store.get_operation_plan(request.plan_id) if not plan: @@ -519,46 +734,20 @@ async def approve_operation_plan(request: ApplyPlanRequest): if plan.get("session_id") != request.session_id: raise HTTPException(status_code=400, detail="Plan/session mismatch") if request.final_status: - if not ALLOW_LEGACY_APPLY_FINAL_STATUS: - raise HTTPException( - status_code=400, - detail="Legacy final_status shortcut is disabled; approve with /tasks/apply and record Office.js results with /tasks/apply/result", - ) - status = request.final_status - if status not in {"applied", "failed", "abandoned"}: - raise HTTPException(status_code=400, detail="final_status must be 'applied', 'failed', or 'abandoned'") - # Loud deprecation warning: any operator still leaning on this path - # should see the structured warning in their logs and plan a switch to - # /tasks/apply followed by /tasks/apply/result. logger.warning( - "legacy_final_status_shortcut_used", + "legacy_final_status_shortcut_rejected", session_id=request.session_id, plan_id=request.plan_id, - status=status, - advice="ALLOW_LEGACY_APPLY_FINAL_STATUS will be removed; migrate to /tasks/apply + /tasks/apply/result", + status=request.final_status, + configured=ALLOW_LEGACY_APPLY_FINAL_STATUS, + advice="Use /tasks/apply followed by /tasks/apply/result with an exact Office host reread.", ) - audit_id = store.record_apply_result( - session_id=request.session_id, - plan_id=request.plan_id, - status=status, - payload={ - "office_apply_result": request.office_apply_result or {}, - "error_detail": request.error_detail, - }, - ) - plan_obj, v2_plan, v3_plan = _load_plan_all_views(store.get_operation_plan(request.plan_id) or plan) - # Echo the supplied apply_token rather than returning an empty string, - # and surface the deprecation in metadata so clients can detect it. - legacy_token = request.apply_token or "legacy" - return ApplyPlanResponse( - plan_id=request.plan_id, - session_id=request.session_id, - apply_token=legacy_token, - apply_attempt_id="legacy", - apply_token_expires_at=None, - rollback_snapshot_id="", - audit_event_id=audit_id, - operations=_operation_dicts(_active_apply_operations(v2_plan, v3_plan)), + raise HTTPException( + status_code=400, + detail=( + "Legacy final_status shortcut cannot prove an Office mutation and is rejected; " + "approve with /tasks/apply and record a complete host reread with /tasks/apply/result" + ), ) plan_obj, v2_plan, v3_plan = _load_plan_all_views(plan) @@ -571,6 +760,48 @@ async def approve_operation_plan(request: ApplyPlanRequest): or (v3_plan and v3_plan.validation.status == "failed") ): raise HTTPException(status_code=409, detail="Plan validation failed; refusing to approve unsafe operations") + try: + ensure_transition("plan", plan_obj.status, "approved", plan_id=request.plan_id) + if v3_plan: + ensure_transition("v3", v3_plan.lifecycle_state, "approved", plan_id=request.plan_id) + except InvalidPlanTransition as exc: + raise HTTPException( + status_code=409, + detail={ + "code": "illegal_plan_transition", + "message": str(exc), + "kind": exc.kind, + "current": exc.current, + "requested": exc.requested, + }, + ) from exc + + mutation_ops = _mutation_operations(active_operations) + apply_validation = validate_workbook_operations_v2( + active_operations, + request.workbook_state, + v2_plan.validation_profile, + ) + apply_blockers = [issue.model_dump(mode="json") for issue in apply_validation.issues if issue.severity == "error"] + if apply_blockers: + raise HTTPException( + status_code=409, + detail={ + "code": "operation_registry_apply_blocked", + "message": "Plan contains operations that are safe-gated or not apply-ready.", + "blockers": apply_blockers, + }, + ) + write_conflicts = _detect_conflicting_writes(mutation_ops) + if write_conflicts: + raise HTTPException( + status_code=409, + detail={ + "code": "conflicting_writes", + "message": "Plan contains multiple operations targeting the same (sheet, range); refusing to approve a non-atomic batch.", + "conflicts": write_conflicts, + }, + ) if request.workbook_fingerprint and plan_obj.metadata.get("approved_fingerprint"): if request.workbook_fingerprint != plan_obj.metadata.get("approved_fingerprint"): @@ -619,6 +850,17 @@ async def approve_operation_plan(request: ApplyPlanRequest): apply_token = secrets.token_urlsafe(18) token_hash = _token_hash(apply_token) apply_token_expires_at = time.time() + APPLY_TOKEN_TTL_SECONDS + mutation_payloads = _operation_dicts(mutation_ops) + pre_binding = _workbook_target_binding(request.workbook_state, mutation_payloads) + operation_registry_hash = _stable_payload_hash(mutation_payloads) + apply_policy_hash = _stable_payload_hash( + { + "schema_version": v3_plan.schema_version if v3_plan else v2_plan.schema_version, + "validation_profile": v2_plan.validation_profile, + "brain_apply_required": requires_brain_eligibility, + "operation_registry_hash": operation_registry_hash, + } + ) attempt_id = store.record_apply_attempt( session_id=request.session_id, plan_id=request.plan_id, @@ -632,6 +874,15 @@ async def approve_operation_plan(request: ApplyPlanRequest): "expires_at": apply_token_expires_at, "brain_apply_required": requires_brain_eligibility, "brain_apply_eligibility": brain_eligibility.model_dump(mode="json", by_alias=True) if brain_eligibility else None, + "plan_id": request.plan_id, + "operation_registry_hash": operation_registry_hash, + "apply_policy_hash": apply_policy_hash, + "approved_operations": mutation_payloads, + "pre_workbook_fingerprint": request.workbook_fingerprint or "", + "pre_context_hash": _stable_payload_hash(request.workbook_state), + "pre_context_complete": pre_binding["complete"], + "pre_context_errors": pre_binding["errors"], + "pre_target_hashes": pre_binding["target_hashes"], }, ) plan_obj.status = "approved" @@ -649,6 +900,8 @@ async def approve_operation_plan(request: ApplyPlanRequest): "brain_apply_operation_hash": brain_eligibility.operation_hash if brain_eligibility else "", "brain_run_id": brain_run_id, "brain_artifact_bundle_id": brain_bundle_id, + "operation_registry_hash": operation_registry_hash, + "apply_policy_hash": apply_policy_hash, } ) v2_plan.metadata.update(plan_obj.metadata) @@ -715,8 +968,9 @@ async def approve_operation_plan(request: ApplyPlanRequest): @router.post("/tasks/apply/result", response_model=ApplyResultResponse) -async def record_apply_result(request: ApplyResultRequest): +async def record_apply_result(request: ApplyResultRequest, http_request: Request): """Record the Office.js mutation result after a previously approved plan.""" + _enforce_ai_policy("action", actor=_request_actor(http_request)) if request.status not in {"applied", "failed", "abandoned"}: raise HTTPException(status_code=400, detail="status must be 'applied', 'failed', or 'abandoned'") store = get_session_store() @@ -728,6 +982,27 @@ async def record_apply_result(request: ApplyResultRequest): plan_obj, v2_plan, v3_plan = _load_plan_all_views(plan) expected_hash = plan_obj.metadata.get("apply_token_hash") active_operations = _active_apply_operations(v2_plan, v3_plan) + expected_attempt_id = plan_obj.metadata.get("apply_attempt_id") + if expected_attempt_id and request.apply_attempt_id != expected_attempt_id: + raise HTTPException(status_code=409, detail="Apply attempt id does not match the approved attempt") + latest_attempt = store.get_latest_apply_attempt(request.plan_id) or {} + if expected_hash: + if not request.apply_token or _token_hash(request.apply_token) != expected_hash: + raise HTTPException(status_code=403, detail="Invalid apply token") + expires_at = float(plan_obj.metadata.get("apply_token_expires_at") or 0) + if expires_at and time.time() > expires_at: + raise HTTPException(status_code=403, detail="Apply token expired") + # Reject reuse of a token whose attempt already completed. + if latest_attempt.get("token_hash") == expected_hash and latest_attempt.get("completed_at"): + raise HTTPException( + status_code=409, + detail={ + "code": "apply_token_consumed", + "message": "Apply token has already been used; request a new approval before retrying.", + "completed_at": latest_attempt.get("completed_at"), + "completed_status": latest_attempt.get("status"), + }, + ) verification_plan = v2_plan.model_copy(update={"operations": active_operations}) verification = _verify_apply_result( verification_plan, @@ -735,24 +1010,26 @@ async def record_apply_result(request: ApplyResultRequest): request.post_apply_workbook_state, request.status, request.error_detail, + pre_apply_workbook_state=request.pre_apply_workbook_state, + pre_apply_workbook_fingerprint=request.pre_apply_workbook_fingerprint, + pre_host_snapshot=request.pre_host_snapshot, + post_apply_workbook_fingerprint=request.post_apply_workbook_fingerprint, + post_host_snapshot=request.post_host_snapshot, + attempt_id=str(expected_attempt_id or ""), + plan_id=request.plan_id, + attempt_payload=latest_attempt.get("payload") or {}, ) if request.status == "applied" and verification.get("status") == "failed": - raise HTTPException(status_code=409, detail="Office.js result contains errors; status must be failed") - expected_attempt_id = plan_obj.metadata.get("apply_attempt_id") - if request.apply_attempt_id and expected_attempt_id and request.apply_attempt_id != expected_attempt_id: - raise HTTPException(status_code=409, detail="Apply attempt id does not match the approved attempt") + raise HTTPException(status_code=409, detail="Office.js result is incomplete; status must be failed") if expected_hash: - if not request.apply_token or _token_hash(request.apply_token) != expected_hash: - raise HTTPException(status_code=403, detail="Invalid apply token") - expires_at = float(plan_obj.metadata.get("apply_token_expires_at") or 0) - if expires_at and time.time() > expires_at: - raise HTTPException(status_code=403, detail="Apply token expired") store.complete_apply_attempt( plan_id=request.plan_id, token_hash=expected_hash, status=request.status, payload={ "office_apply_result": request.office_apply_result, + "pre_apply_workbook_fingerprint": request.pre_apply_workbook_fingerprint, + "post_apply_workbook_fingerprint": request.post_apply_workbook_fingerprint, "verification": verification, "error_detail": request.error_detail, }, @@ -772,7 +1049,25 @@ async def record_apply_result(request: ApplyResultRequest): rollback_available=bool(plan_obj.metadata.get("rollback_snapshot_id")), ) v3_plan.verification = v3_report - v3_plan.lifecycle_state = "verified" if v3_report.status == "verified" else "failed" if v3_report.status == "failed" else "applied" + requested_v3_state: Literal["verified", "failed", "applied"] = ( + "verified" if v3_report.status == "verified" + else "failed" if v3_report.status == "failed" + else "applied" + ) + try: + ensure_transition("v3", v3_plan.lifecycle_state, requested_v3_state, plan_id=request.plan_id) + except InvalidPlanTransition as exc: + raise HTTPException( + status_code=409, + detail={ + "code": "illegal_plan_transition", + "message": str(exc), + "kind": exc.kind, + "current": exc.current, + "requested": exc.requested, + }, + ) from exc + v3_plan.lifecycle_state = requested_v3_state verification["v3"] = v3_report.model_dump(mode="json", by_alias=True) v3_verification_id = store.save_verification_run( request.session_id, @@ -786,10 +1081,27 @@ async def record_apply_result(request: ApplyResultRequest): _operation_audit_rows(active_operations, request.office_apply_result, v3_report.model_dump(mode="json", by_alias=True)), ) + try: + ensure_transition("plan", plan_obj.status, request.status, plan_id=request.plan_id) + except InvalidPlanTransition as exc: + raise HTTPException( + status_code=409, + detail={ + "code": "illegal_plan_transition", + "message": str(exc), + "kind": exc.kind, + "current": exc.current, + "requested": exc.requested, + }, + ) from exc plan_obj.status = request.status # type: ignore[assignment] v2_plan.status = request.status # type: ignore[assignment] plan_obj.metadata["last_apply_verification"] = verification v2_plan.metadata["last_apply_verification"] = verification + durable_rollback_operations = verification.get("rollback_restore_operations") or [] + if request.status == "applied" and durable_rollback_operations: + plan_obj.metadata["durable_rollback_operations"] = durable_rollback_operations + v2_plan.metadata["durable_rollback_operations"] = durable_rollback_operations if v3_plan: plan_obj.metadata["last_v3_verification"] = verification.get("v3") _save_operation_plan_with_v3(store, plan_obj, v3_plan, v2_plan) @@ -851,13 +1163,18 @@ async def record_apply_result(request: ApplyResultRequest): from backend.observability import record_apply_result record_apply_result(request.status) - return ApplyResultResponse( + response = ApplyResultResponse( plan_id=request.plan_id, session_id=request.session_id, status=request.status, audit_event_id=audit_id, verification=verification, ) + from backend.services.retention import purge_terminal_session_if_immediate + + if request.status == "applied" and verification.get("status") == "verified": + await purge_terminal_session_if_immediate(request.session_id, request.status) + return response @router.post("/tasks/plan/finalize", response_model=OperationPlanV2) @@ -1008,7 +1325,25 @@ async def verify_operation_plan_v3(plan_id: str, request: VerifyPlanRequest): rollback_available=request.rollback_available, ) v3_plan.verification = report - v3_plan.lifecycle_state = "verified" if report.status == "verified" else "failed" if report.status == "failed" else "applied" + requested_v3_state: Literal["verified", "failed", "applied"] = ( + "verified" if report.status == "verified" + else "failed" if report.status == "failed" + else "applied" + ) + try: + ensure_transition("v3", v3_plan.lifecycle_state, requested_v3_state, plan_id=plan_id) + except InvalidPlanTransition as exc: + raise HTTPException( + status_code=409, + detail={ + "code": "illegal_plan_transition", + "message": str(exc), + "kind": exc.kind, + "current": exc.current, + "requested": exc.requested, + }, + ) from exc + v3_plan.lifecycle_state = requested_v3_state plan_obj.metadata["last_v3_verification"] = report.model_dump(mode="json", by_alias=True) _save_operation_plan_with_v3(store, plan_obj, v3_plan, v2_plan) verification_id = store.save_verification_run( @@ -1102,6 +1437,8 @@ async def build_formula_graph(request: FormulaGraphRequest): """Build and persist a HyperFormula-first formula graph artifact.""" from backend.services.formula_graph import get_formula_graph_service + if request.session_id: + get_session_store().get_or_create_session(request.session_id) graph = get_formula_graph_service().build( session_id=request.session_id, workbook_fingerprint=request.workbook_fingerprint, @@ -1130,6 +1467,8 @@ async def build_formula_graph_v4(request: FormulaGraphRequest): """Build and persist a V6/V4 HyperFormula-first formula graph artifact.""" from backend.services.formula_graph_v4 import get_formula_graph_v4_service + if request.session_id: + get_session_store().get_or_create_session(request.session_id) graph = get_formula_graph_v4_service().build( session_id=request.session_id, workbook_fingerprint=request.workbook_fingerprint, @@ -1159,6 +1498,8 @@ async def build_formula_graph_v5(request: FormulaGraphRequest): """Build and persist a V7 HyperFormula-first formula graph artifact.""" from backend.services.formula_graph_v5 import get_formula_graph_v5_service + if request.session_id: + get_session_store().get_or_create_session(request.session_id) graph = get_formula_graph_v5_service().build( session_id=request.session_id, workbook_fingerprint=request.workbook_fingerprint, @@ -1188,6 +1529,7 @@ async def build_workbook_truth_model(request: WorkbookTruthModelRequest): """Build and persist a WorkbookTruthModel above semantic maps and formula graphs.""" from backend.services.workbook_truth import get_workbook_truth_model_service + get_session_store().get_or_create_session(request.session_id) truth = get_workbook_truth_model_service().build( session_id=request.session_id, workbook_context=request.workbook_context, @@ -1225,6 +1567,7 @@ async def build_workbook_truth_model_v4(request: WorkbookTruthModelRequest): """Build and persist a V6 WorkbookTruthModelV4 with durable target decisions.""" from backend.services.workbook_truth_v4 import get_workbook_truth_model_v4_service + get_session_store().get_or_create_session(request.session_id) truth = get_workbook_truth_model_v4_service().build( session_id=request.session_id, workbook_context=request.workbook_context, @@ -1255,6 +1598,7 @@ async def build_workbook_truth_model_v5(request: WorkbookTruthModelRequest): """Build and persist a V7 WorkbookTruthModelV5 with typed semantic decisions.""" from backend.services.workbook_truth_v5 import get_workbook_truth_model_v5_service + get_session_store().get_or_create_session(request.session_id) truth = get_workbook_truth_model_v5_service().build( session_id=request.session_id, workbook_context=request.workbook_context, @@ -1295,6 +1639,7 @@ async def simulate_operations_v2(request: OperationSimulationV2Request): adapter = TypeAdapter(WorkbookOperationV2) operations = [adapter.validate_python(item) for item in request.operations] store = get_session_store() + store.get_or_create_session(request.session_id) truth_model = None if request.truth_model_id: stored_truth = store.get_workbook_truth_model(request.truth_model_id) @@ -1366,6 +1711,7 @@ async def simulate_operations_v5(request: OperationSimulationV2Request): adapter = TypeAdapter(WorkbookOperationV2) operations = [adapter.validate_python(item) for item in request.operations] store = get_session_store() + store.get_or_create_session(request.session_id) truth_model = None if request.truth_model_id: stored_truth = store.get_workbook_truth_model(request.truth_model_id) @@ -1418,6 +1764,7 @@ async def simulate_operations_v6(request: OperationSimulationV2Request): adapter = TypeAdapter(WorkbookOperationV2) operations = [adapter.validate_python(item) for item in request.operations] store = get_session_store() + store.get_or_create_session(request.session_id) truth_model = None if request.truth_model_id: stored_truth = store.get_workbook_truth_model(request.truth_model_id) @@ -1464,12 +1811,19 @@ async def build_source_evidence_v2(request: SourceEvidenceV2Request): """Build and persist source-to-cell provenance and citation coverage.""" from backend.services.source_evidence_v2 import get_source_evidence_v2_service + get_session_store().get_or_create_session(request.session_id) + documents = _source_evidence_documents(request) graph = get_source_evidence_v2_service().build_graph( session_id=request.session_id, - documents=request.documents, + documents=documents, workbook_context=request.workbook_context, downstream_outputs=request.downstream_outputs, ) + graph.metadata = { + **graph.metadata, + "documentCount": len(documents), + "documentSource": "request" if request.documents or request.document_ids else "session_store", + } store = get_session_store() store.save_source_evidence_graph(graph.model_dump(mode="json", by_alias=True)) store.record_audit_event( @@ -1477,6 +1831,7 @@ async def build_source_evidence_v2(request: SourceEvidenceV2Request): event_type="source_evidence_v2_built", payload={ "graph_id": graph.graph_id, + "document_count": len(documents), "evidence_count": len(graph.evidence), "cell_provenance_count": len(graph.cell_to_source), "citation_coverage": graph.citation_coverage, @@ -1490,12 +1845,19 @@ async def build_source_evidence_v3(request: SourceEvidenceV2Request): """Build and persist V7 source evidence with provenance and refresh planning.""" from backend.services.source_evidence_v3 import get_source_evidence_v3_service + get_session_store().get_or_create_session(request.session_id) + documents = _source_evidence_documents(request) graph = get_source_evidence_v3_service().build_graph( session_id=request.session_id, - documents=request.documents, + documents=documents, workbook_context=request.workbook_context, downstream_outputs=request.downstream_outputs, ) + graph.metadata = { + **graph.metadata, + "documentCount": len(documents), + "documentSource": "request" if request.documents or request.document_ids else "session_store", + } store = get_session_store() store.save_source_evidence_graph(graph.model_dump(mode="json", by_alias=True)) store.record_audit_event( @@ -1503,6 +1865,7 @@ async def build_source_evidence_v3(request: SourceEvidenceV2Request): event_type="source_evidence_v3_built", payload={ "graph_id": graph.graph_id, + "document_count": len(documents), "evidence_count": len(graph.evidence), "citation_validation": graph.citation_validation, "refresh_plan": len(graph.refresh_plan), @@ -1511,6 +1874,37 @@ async def build_source_evidence_v3(request: SourceEvidenceV2Request): return graph +def _source_evidence_documents(request: SourceEvidenceV2Request) -> list[dict]: + store = get_session_store() + documents = list(request.documents or []) + seen = { + str(document.get("document_id") or document.get("documentId") or "") + for document in documents + if isinstance(document, dict) + } + for document_id in request.document_ids: + if document_id in seen: + continue + document = store.get_document(document_id) + if document: + documents.append(document) + seen.add(document_id) + if documents: + return documents + return store.list_documents(request.session_id, limit=25) + + +def _work_budget_http_exception(exc: WorkBudgetError) -> HTTPException: + return HTTPException( + status_code=503, + detail={ + "code": exc.code, + "message": "Workbook worker capacity is unavailable; retry shortly.", + }, + headers={"Retry-After": "1"}, + ) + + @router.post("/workbook/ingest") async def ingest_workbook_file(request: WorkbookIngestRequest): """Summarize an uploaded XLSX payload for planner context and QA.""" @@ -1531,7 +1925,20 @@ async def ingest_workbook_file(request: WorkbookIngestRequest): if len(content) > WORKBOOK_WORKER_MAX_PAYLOAD_BYTES: raise HTTPException(status_code=413, detail="Workbook payload exceeds worker size limit") try: - return WorkbookIngestionService().summarize(content, request.filename) + # Worker work is synchronous (spawn + parse), so push it to a thread + # and bound the wait — a hung worker must not stall the FastAPI loop. + from backend.config import WORKBOOK_WORKER_TIMEOUT_SECONDS + + return await run_cancellable_in_thread( + WorkbookIngestionService().summarize, + content, + request.filename, + timeout_seconds=WORKBOOK_WORKER_TIMEOUT_SECONDS, + ) + except (asyncio.TimeoutError, WorkExecutionTimeout) as exc: + raise HTTPException(status_code=504, detail="Workbook ingestion timed out") from exc + except WorkBudgetError as exc: + raise _work_budget_http_exception(exc) from None except Exception as exc: raise HTTPException(status_code=400, detail=f"Workbook ingestion failed: {exc}") from exc @@ -1545,7 +1952,19 @@ async def export_workbook_file(request: WorkbookExportRequest): if not request.sheets: raise HTTPException(status_code=400, detail="sheets cannot be empty") - content = WorkbookExportService().export(request.sheets, title=request.title) + from backend.config import WORKBOOK_WORKER_TIMEOUT_SECONDS + + try: + content = await run_cancellable_in_thread( + WorkbookExportService().export, + request.sheets, + title=request.title, + timeout_seconds=WORKBOOK_WORKER_TIMEOUT_SECONDS, + ) + except (asyncio.TimeoutError, WorkExecutionTimeout) as exc: + raise HTTPException(status_code=504, detail="Workbook export timed out") from exc + except WorkBudgetError as exc: + raise _work_budget_http_exception(exc) from None return Response( content=content, media_type="application/vnd.openxmlformats-officedocument.spreadsheetml.sheet", @@ -1556,20 +1975,69 @@ async def export_workbook_file(request: WorkbookExportRequest): @router.post("/workbook/diff") async def diff_workbook_summaries(request: WorkbookDiffRequest): """Compare two workbook ingestion summaries.""" + from subprocess import TimeoutExpired + + from backend.config import WORKBOOK_WORKER_TIMEOUT_SECONDS from backend.excel.workbook_files import WorkbookDiffService - return WorkbookDiffService().diff(request.before, request.after) + try: + return await run_cancellable_in_thread( + WorkbookDiffService().diff, + request.before, + request.after, + timeout_seconds=WORKBOOK_WORKER_TIMEOUT_SECONDS + 1.0, + ) + except (asyncio.TimeoutError, TimeoutExpired) as exc: + raise HTTPException(status_code=504, detail="Workbook diff timed out") from exc + except WorkBudgetError as exc: + raise _work_budget_http_exception(exc) from None + except Exception as exc: + raise HTTPException(status_code=400, detail=f"Workbook diff failed: {exc}") from exc @router.post("/workbook/jobs") -async def submit_workbook_job(request: WorkbookJobRequest): +async def submit_workbook_job(request: WorkbookJobRequest, http_request: Request): """Submit a long-running workbook job and receive a polling run_id.""" from backend.services.workbook_jobs import get_workbook_job_manager - job = get_workbook_job_manager().submit(request.command, request.payload) # type: ignore[arg-type] + session_id = request.session_id or "" + if session_id: + _claim_request_session(http_request, session_id) + elif _request_actor(http_request) is not None: + from backend.api.security.ownership import OwnershipNotFoundError + + raise OwnershipNotFoundError("Workbook job not found") + try: + job = get_workbook_job_manager().submit( + request.command, # type: ignore[arg-type] + request.payload, + session_id=session_id, + account_id=request.account_id or "local", + ) + except RuntimeError as exc: + raise HTTPException(status_code=429, detail=str(exc)) from exc return job.to_dict(include_result=False) +@router.get("/workbook/jobs") +async def list_workbook_jobs( + limit: int = 50, + status: Optional[str] = None, + session_id: Optional[str] = None, + account_id: Optional[str] = None, +): + """List durable workbook jobs for polling dashboards or API clients.""" + from backend.services.workbook_jobs import get_workbook_job_manager + + jobs = get_workbook_job_manager().list( + limit=min(max(limit, 0), 200), + status=status, + session_id=session_id, + account_id=account_id, + ) + return {"jobs": [job.to_dict(include_result=False) for job in jobs]} + + @router.get("/workbook/jobs/{run_id}") async def get_workbook_job(run_id: str): """Poll a workbook job until completed or failed.""" @@ -1581,100 +2049,241 @@ async def get_workbook_job(run_id: str): return job.to_dict(include_result=True) +@router.get("/workbook/jobs/{run_id}/result") +async def get_workbook_job_result(run_id: str): + """Return the completed workbook job result payload.""" + from backend.services.workbook_jobs import get_workbook_job_manager + + job = get_workbook_job_manager().get(run_id) + if not job: + raise HTTPException(status_code=404, detail="Workbook job not found") + if job.status != "completed": + raise HTTPException(status_code=409, detail=f"Workbook job is {job.status}") + return {"run_id": run_id, "result": job.result} + + +@router.get("/workbook/jobs/{run_id}/download") +async def download_workbook_job_result(run_id: str): + """Download workbook bytes from a completed export-style job.""" + from backend.services.workbook_jobs import get_workbook_job_manager + + job = get_workbook_job_manager().get(run_id) + if not job: + raise HTTPException(status_code=404, detail="Workbook job not found") + if job.status != "completed": + raise HTTPException(status_code=409, detail=f"Workbook job is {job.status}") + result = job.result if isinstance(job.result, dict) else {} + content_base64 = result.get("content_base64") or result.get("contentBase64") + if not content_base64: + raise HTTPException(status_code=404, detail="Workbook job result has no downloadable content") + filename = str(result.get("filename") or f"{run_id}.xlsx") + return Response( + content=base64.b64decode(content_base64), + media_type="application/vnd.openxmlformats-officedocument.spreadsheetml.sheet", + headers={"Content-Disposition": f'attachment; filename="{filename}"'}, + ) + + +@router.post("/workbook/jobs/{run_id}/cancel") +async def cancel_workbook_job(run_id: str): + """Cancel a queued or running workbook job when possible.""" + from backend.services.workbook_jobs import get_workbook_job_manager + + job = get_workbook_job_manager().cancel(run_id) + if not job: + raise HTTPException(status_code=404, detail="Workbook job not found") + return job.to_dict(include_result=False) + + +@router.post("/spreadsheet/open", response_model=SpreadsheetOpenResponse) +async def open_spreadsheet_workbook(request: SpreadsheetOpenRequest): + """Open workbook bytes as the browser spreadsheet engine session model.""" + return await _run_spreadsheet_worker( + "open-workbook-v1", + _spreadsheet_worker_payload( + request, + { + "filename": request.filename, + "content_base64": request.content_base64, + "title": request.title, + "workbook_id": request.workbook_id, + }, + ), + "Spreadsheet open", + ) + + +@router.post("/spreadsheet/export", responses=SPREADSHEET_EXPORT_RESPONSES) +async def export_spreadsheet_workbook(request: SpreadsheetExportRequest, http_request: Request): + """Export a browser spreadsheet engine session as XLSX bytes.""" + from fastapi.responses import Response + + _enforce_export_policy(request, _request_actor(http_request)) + + result = await _run_spreadsheet_worker( + "export-workbook-v1", + _spreadsheet_worker_payload( + request, + { + "session": request.session, + "title": request.title, + "filename": request.filename, + "fidelityMode": request.fidelityMode, + "allowRegeneratedFallback": request.allowRegeneratedFallback, + "allowMacroRegeneratedFallback": request.allowMacroRegeneratedFallback, + "preferOriginal": request.preferOriginal, + }, + ), + "Spreadsheet export", + ) + content_base64 = str((result or {}).get("content_base64") or "") + try: + content = base64.b64decode(content_base64, validate=True) + except Exception as exc: + raise HTTPException(status_code=500, detail="Spreadsheet export returned invalid workbook bytes") from exc + filename = request.filename or "finance-ai-workbook.xlsx" + return Response( + content=content, + media_type="application/vnd.openxmlformats-officedocument.spreadsheetml.sheet", + headers={ + "Content-Disposition": f'attachment; filename="{filename}"', + "X-Spreadsheet-Engine": str((result or {}).get("engine") or "spreadsheet-session-export-v1"), + "X-Spreadsheet-Fidelity-Mode": str(((result or {}).get("fidelity") or {}).get("mode") or ""), + "X-Spreadsheet-Fidelity-Status": str(((((result or {}).get("fidelity") or {}).get("report") or {}).get("status")) or ""), + "X-Spreadsheet-Unsupported-Preserved": str(((result or {}).get("fidelity") or {}).get("unsupportedObjectsPreserved") or False).lower(), + }, + ) + + +@router.post("/spreadsheet/roundtrip-qa", response_model=SpreadsheetRoundtripQaResponse) +async def spreadsheet_roundtrip_qa(request: SpreadsheetRoundtripQaRequest): + """Compare browser spreadsheet session fidelity after export and reopen.""" + return await _run_spreadsheet_worker( + "roundtrip-qa-v2", + _spreadsheet_worker_payload( + request, + { + "session": request.session, + "filename": request.filename, + "title": request.title, + "content_base64": request.content_base64, + "fidelityMode": request.fidelityMode, + "allowRegeneratedFallback": request.allowRegeneratedFallback, + }, + ), + "Spreadsheet roundtrip QA", + ) + + # ── POST /tasks/ask ───────────────────────────────────────────────────── @router.post("/tasks/ask", response_model=AskResponse) -async def ask_task(request: AskRequest): +async def ask_task(request: AskRequest, http_request: Request): """Answer workbook questions without producing sheet mutations.""" session_id = request.session_id or str(uuid.uuid4())[:12] - answer = await _handle_ask_mode(request.prompt, request.context, request.provider, request.model) - store = get_session_store() - store.add_message(session_id, "user", request.prompt) - store.add_message(session_id, "assistant", answer, metadata={"mode": "ask"}) - store.record_audit_event(session_id, "ask_answered", payload={"chars": len(answer)}) - return AskResponse(session_id=session_id, answer=answer, citations=[]) + _claim_request_session(http_request, session_id) + _enforce_ai_policy("ask", request.provider, actor=_request_actor(http_request)) + from backend.services.lifecycle import get_lifecycle_coordinator + + async with get_lifecycle_coordinator().active_task(session_id): + answer = await _handle_ask_mode(request.prompt, request.context, request.provider, request.model) + store = get_session_store() + store.add_message(session_id, "user", request.prompt) + store.add_message(session_id, "assistant", answer, metadata={"mode": "ask"}) + store.record_audit_event(session_id, "ask_answered", payload={"chars": len(answer)}) + response = AskResponse(session_id=session_id, answer=answer, citations=[]) + from backend.services.retention import purge_terminal_session_if_immediate + + # Immediate retention runs in this same task. Lifecycle cancellation + # deliberately excludes the purge caller so this cannot self-deadlock. + await purge_terminal_session_if_immediate(session_id, "completed") + return response # ── POST /tasks ────────────────────────────────────────────────────────── @router.post("/tasks", response_model=TaskResponse) -async def create_task(request: TaskRequest): +async def create_task(request: TaskRequest, http_request: Request): """Process a natural language prompt through the multi-agent pipeline.""" # ~64 bits of entropy so audit-log task IDs don't collide under load. task_id = secrets.token_urlsafe(8) session_id = request.session_id or secrets.token_urlsafe(9) logger.info("task_received", task_id=task_id, session_id=session_id, mode=request.mode) + _claim_request_session(http_request, session_id) + _enforce_ai_policy(request.mode, request.provider, actor=_request_actor(http_request)) try: wb_context = request.context.model_dump() if request.context else None + from backend.services.lifecycle import get_lifecycle_coordinator - # Handle "ask" mode separately - if request.mode == "ask": - answer = await _handle_ask_mode( - request.prompt, wb_context, request.provider, request.model, - ) - return TaskResponse( - task_id=task_id, + async with get_lifecycle_coordinator().active_task(session_id): + # Handle "ask" mode separately + if request.mode == "ask": + answer = await _handle_ask_mode( + request.prompt, wb_context, request.provider, request.model, + ) + return TaskResponse( + task_id=task_id, + session_id=session_id, + summary=answer, + ) + + # Action or Plan mode → pipeline + result = await run_pipeline( + user_prompt=request.prompt, + workbook_context=wb_context, + provider_override=request.provider, + model_override=request.model, + mode=request.mode, session_id=session_id, - summary=answer, ) - # Action or Plan mode → pipeline - result = await run_pipeline( - user_prompt=request.prompt, - workbook_context=wb_context, - provider_override=request.provider, - model_override=request.model, - mode=request.mode, - session_id=session_id, - ) - - # Track changes for undo/redo - diff_engine = get_diff_engine(session_id) - if request.mode == "action" and result.get("operations"): - snapshot = wb_context.get("cell_snapshot", {}) if wb_context else {} - changeset = diff_engine.create_changeset( - operations=result["operations"], - snapshot=snapshot, + # Track changes for undo/redo + diff_engine = get_diff_engine(session_id) + if request.mode == "action" and result.get("operations"): + snapshot = wb_context.get("cell_snapshot", {}) if wb_context else {} + changeset = diff_engine.create_changeset( + operations=result["operations"], + snapshot=snapshot, + prompt=request.prompt, + summary=result.get("summary", ""), + metadata=result.get("metadata"), + ) + diff_engine.push(changeset) + + # Persist to session store + store = get_session_store() + store.add_message(session_id, "user", request.prompt) + store.add_message(session_id, "assistant", result.get("summary", ""), + metadata={"ops": len(result.get("operations", []))}) + store.record_task( + session_id=session_id, + task_id=task_id, prompt=request.prompt, + mode=request.mode, + operations=result.get("operations", []), summary=result.get("summary", ""), - metadata=result.get("metadata"), + plan=result.get("plan"), + provider=result.get("metadata", {}).get("executor_provider", ""), + model=result.get("metadata", {}).get("executor_model", ""), + tokens=result.get("metadata", {}).get("executor_tokens"), ) - diff_engine.push(changeset) - - # Persist to session store - store = get_session_store() - store.add_message(session_id, "user", request.prompt) - store.add_message(session_id, "assistant", result.get("summary", ""), - metadata={"ops": len(result.get("operations", []))}) - store.record_task( - session_id=session_id, - task_id=task_id, - prompt=request.prompt, - mode=request.mode, - operations=result.get("operations", []), - summary=result.get("summary", ""), - plan=result.get("plan"), - provider=result.get("metadata", {}).get("executor_provider", ""), - model=result.get("metadata", {}).get("executor_model", ""), - tokens=result.get("metadata", {}).get("executor_tokens"), - ) - return TaskResponse( - task_id=task_id, - session_id=session_id, - operations=result.get("operations", []), - plan=result.get("plan"), - summary=result.get("summary", ""), - metadata=result.get("metadata", {}), - can_undo=diff_engine.can_undo, - can_redo=diff_engine.can_redo, - ) + return TaskResponse( + task_id=task_id, + session_id=session_id, + operations=result.get("operations", []), + plan=result.get("plan"), + summary=result.get("summary", ""), + metadata=result.get("metadata", {}), + can_undo=diff_engine.can_undo, + can_redo=diff_engine.can_redo, + ) except Exception as e: logger.error("task_failed", task_id=task_id, error=str(e)) - raise HTTPException(status_code=500, detail=str(e)) + raise HTTPException(status_code=500, detail=str(e)) from e # ── WebSocket ──────────────────────────────────────────────────────────── @@ -1682,7 +2291,7 @@ async def create_task(request: TaskRequest): # Note: WebSocket routes must be registered on the main app, not a sub-router. # This function is called from main.py to register the WebSocket route. -async def websocket_task_handler(websocket: WebSocket, session_id: str): +async def websocket_task_handler(websocket: WebSocket, session_id: str, actor: Any | None = None): """WebSocket for real-time pipeline progress. Client sends: {"prompt": "...", "context": {...}, "mode": "action"} @@ -1712,6 +2321,7 @@ async def websocket_task_handler(websocket: WebSocket, session_id: str): mode = data.get("mode", "action") provider = data.get("provider") model = data.get("model") + source_document_ids = data.get("source_document_ids") or data.get("sourceDocumentIds") if not isinstance(prompt, str) or not prompt: await websocket.send_json({"type": "error", "detail": "Empty prompt"}) @@ -1722,6 +2332,19 @@ async def websocket_task_handler(websocket: WebSocket, session_id: str): if mode not in MAX_MODE_VALUES: await websocket.send_json({"type": "error", "detail": f"Invalid mode: {mode}"}) continue + if source_document_ids is not None and not isinstance(source_document_ids, list): + await websocket.send_json({"type": "error", "detail": "source_document_ids must be a list"}) + continue + try: + _enforce_ai_policy( + mode, + provider=provider, + source_document_ids=source_document_ids, + actor=actor, + ) + except HTTPException as exc: + await websocket.send_json({"type": "error", "detail": exc.detail}) + continue now = time.time() last_call_window[:] = [ts for ts in last_call_window if ts > now - 60] @@ -1744,54 +2367,57 @@ async def on_progress(event): pass try: - result = await run_pipeline( - user_prompt=prompt, - workbook_context=context, - provider_override=provider, - model_override=model, - mode=mode, - session_id=session_id, - on_progress=on_progress, - ) + from backend.services.lifecycle import get_lifecycle_coordinator + + async with get_lifecycle_coordinator().active_task(session_id): + result = await run_pipeline( + user_prompt=prompt, + workbook_context=context, + provider_override=provider, + model_override=model, + mode=mode, + session_id=session_id, + on_progress=on_progress, + ) - diff_engine = get_diff_engine(session_id) - if mode == "action" and result.get("operations"): - snapshot = context.get("cell_snapshot", {}) if context else {} - changeset = diff_engine.create_changeset( - operations=result["operations"], - snapshot=snapshot, + diff_engine = get_diff_engine(session_id) + if mode == "action" and result.get("operations"): + snapshot = context.get("cell_snapshot", {}) if context else {} + changeset = diff_engine.create_changeset( + operations=result["operations"], + snapshot=snapshot, + prompt=prompt, + summary=result.get("summary", ""), + ) + diff_engine.push(changeset) + + # Persist to session store + store = get_session_store() + store.add_message(session_id, "user", prompt) + store.add_message(session_id, "assistant", result.get("summary", ""), + metadata={"ops": len(result.get("operations", []))}) + store.record_task( + session_id=session_id, + task_id=secrets.token_urlsafe(8), prompt=prompt, + mode=mode, + operations=result.get("operations", []), summary=result.get("summary", ""), + plan=result.get("plan"), + provider=result.get("metadata", {}).get("executor_provider", ""), + model=result.get("metadata", {}).get("executor_model", ""), + tokens=result.get("metadata", {}).get("executor_tokens"), ) - diff_engine.push(changeset) - - # Persist to session store - store = get_session_store() - store.add_message(session_id, "user", prompt) - store.add_message(session_id, "assistant", result.get("summary", ""), - metadata={"ops": len(result.get("operations", []))}) - store.record_task( - session_id=session_id, - task_id=secrets.token_urlsafe(8), - prompt=prompt, - mode=mode, - operations=result.get("operations", []), - summary=result.get("summary", ""), - plan=result.get("plan"), - provider=result.get("metadata", {}).get("executor_provider", ""), - model=result.get("metadata", {}).get("executor_model", ""), - tokens=result.get("metadata", {}).get("executor_tokens"), - ) - await websocket.send_json({ - "type": "complete", - "operations": result.get("operations", []), - "plan": result.get("plan"), - "summary": result.get("summary", ""), - "metadata": result.get("metadata", {}), - "can_undo": diff_engine.can_undo, - "can_redo": diff_engine.can_redo, - }) + await websocket.send_json({ + "type": "complete", + "operations": result.get("operations", []), + "plan": result.get("plan"), + "summary": result.get("summary", ""), + "metadata": result.get("metadata", {}), + "can_undo": diff_engine.can_undo, + "can_redo": diff_engine.can_redo, + }) except Exception as e: logger.error("ws_task_failed", error=str(e)) @@ -2013,9 +2639,18 @@ def _requires_brain_apply_eligibility( v3_plan: Optional[OperationPlanV3], operations: list[Any], ) -> bool: - """Return true when a write path must use server-authoritative Brain eligibility.""" + """Return true when a write path must use server-authoritative Brain eligibility. + + The decision is server-authoritative: clients may *raise* the bar by setting + ``require_apply_eligibility=True`` or attaching an ``apply_eligibility`` + payload, but a client cannot *lower* it. Server policy + (``BRAIN_APPLY_ELIGIBILITY_MANDATORY`` env or plan metadata flagged as v7/ + brain) always forces eligibility regardless of what the client requests. + """ if not _mutation_operations(operations): return False + if BRAIN_APPLY_ELIGIBILITY_MANDATORY: + return True if request.require_apply_eligibility or request.apply_eligibility: return True metadata_candidates = [ @@ -2051,6 +2686,48 @@ def _active_apply_operations(v2_plan: OperationPlanV2, v3_plan: Optional[Operati return _mutation_operations(v2_plan.operations) +def _normalize_range_key(sheet: str, rng: str) -> str: + """Build a normalized (sheet, range) key for conflict detection.""" + return f"{(sheet or '').strip().lower()}::{(rng or '').strip().lower().replace(' ', '')}" + + +def _detect_conflicting_writes(operations: list[Any]) -> list[dict]: + """Find operations whose target (sheet, range) overlap by exact match. + + Returns a list of conflict descriptors. Empty list => atomic batch. + """ + by_key: dict[str, list[dict]] = {} + for index, op in enumerate(operations): + sheet = getattr(op, "sheet", "") or "" + rng = getattr(op, "range", "") or "" + if not sheet and not rng: + continue + key = _normalize_range_key(sheet, rng) + by_key.setdefault(key, []).append( + { + "operation_id": getattr(op, "id", "") or getattr(op, "operation_id", "") or f"op_{index}", + "type": getattr(op, "type", "") or "", + "sheet": sheet, + "range": rng, + "index": index, + } + ) + conflicts = [] + for key, members in by_key.items(): + if len(members) <= 1: + continue + conflicts.append( + { + "key": key, + "sheet": members[0]["sheet"], + "range": members[0]["range"], + "operations": members, + "count": len(members), + } + ) + return conflicts + + def _metadata_without_embedded_plans(metadata: dict | None) -> dict: cleaned = dict(metadata or {}) cleaned.pop("v2_plan", None) @@ -2106,7 +2783,8 @@ def _operation_target(operation: Any) -> dict: def _result_warnings(result: dict) -> list[str]: - warnings = result.get("warnings") if isinstance(result.get("warnings"), list) else [] + raw_warnings = result.get("warnings") + warnings = list(raw_warnings) if isinstance(raw_warnings, list) else [] message = result.get("message") if result.get("status") == "warning" and message: warnings = [*warnings, str(message)] @@ -2199,27 +2877,138 @@ def _verify_apply_result( post_apply_workbook_state: dict, requested_status: str, error_detail: Optional[str], + *, + pre_apply_workbook_state: Optional[dict] = None, + pre_apply_workbook_fingerprint: Optional[str] = None, + pre_host_snapshot: Optional[dict] = None, + post_apply_workbook_fingerprint: Optional[str] = None, + post_host_snapshot: Optional[dict] = None, + attempt_id: str = "", + plan_id: str = "", + attempt_payload: Optional[dict] = None, ) -> dict: """Summarize whether Office.js results and post-apply context match the plan.""" + pre_apply_workbook_state = pre_apply_workbook_state if isinstance(pre_apply_workbook_state, dict) else {} + pre_host_snapshot = pre_host_snapshot if isinstance(pre_host_snapshot, dict) else {} + post_host_snapshot = post_host_snapshot if isinstance(post_host_snapshot, dict) else {} results = office_apply_result.get("results") if isinstance(office_apply_result, dict) else [] results = results if isinstance(results, list) else [] mutation_ops = _mutation_operations(v2_plan.operations) - errors = [item for item in results if isinstance(item, dict) and item.get("status") == "error"] - warnings = [str(item.get("message")) for item in results if isinstance(item, dict) and item.get("status") == "warning" and item.get("message")] + mutation_payloads = _operation_dicts(mutation_ops) + expected_ids = [str(item.get("id") or "") for item in mutation_payloads] + expected_by_id = {str(item.get("id") or ""): item for item in mutation_payloads} + result_rows = [item for item in results if isinstance(item, dict)] + results_by_id: dict[str, list[dict]] = {} + for item in result_rows: + results_by_id.setdefault(str(item.get("operationId") or item.get("operation_id") or ""), []).append(item) + + errors = [item for item in result_rows if item.get("status") == "error"] + warnings = [ + str(item.get("message")) + for item in result_rows + if item.get("status") in {"warning", "skipped"} and item.get("message") + ] dirty_ranges = [ dirty - for item in results - if isinstance(item, dict) + for item in result_rows for dirty in (item.get("dirtyRanges") or item.get("dirty_ranges") or []) ] verification_warnings: list[str] = [] - if requested_status == "applied" and errors: - verification_warnings.append(f"{len(errors)} operation result(s) reported errors despite applied status") - if requested_status == "applied" and len(results) < len(mutation_ops): - verification_warnings.append("Office.js returned fewer operation results than the mutation plan contains") - if requested_status == "applied" and not dirty_ranges and mutation_ops: - verification_warnings.append("No dirty ranges were returned for a mutation plan") + rollback_restore_operations: list[dict] = [] + proof: dict[str, Any] = {} + attempt_payload = attempt_payload if isinstance(attempt_payload, dict) else {} + if requested_status == "applied": + registry_hash = _stable_payload_hash(mutation_payloads) + expected_registry_hash = str(attempt_payload.get("operation_registry_hash") or "") + expected_policy_hash = str(attempt_payload.get("apply_policy_hash") or "") + expected_pre_fingerprint = str(attempt_payload.get("pre_workbook_fingerprint") or "") + pre_binding = _workbook_target_binding(pre_apply_workbook_state, mutation_payloads) + post_binding = _workbook_target_binding(post_apply_workbook_state, mutation_payloads, require_post_match=True) + for label, snapshot in (("Pre-apply", pre_host_snapshot), ("Post-apply", post_host_snapshot)): + snapshot = snapshot if isinstance(snapshot, dict) else {} + if snapshot.get("complete") is not True or snapshot.get("truncated") is True or bool(snapshot.get("errors")): + verification_warnings.append(f"{label} full Office host snapshot is incomplete") + if not attempt_id: + verification_warnings.append("Applied status requires an exact apply attempt id") + if not expected_registry_hash or registry_hash != expected_registry_hash: + verification_warnings.append("Approved operation registry binding is missing or changed") + if not expected_policy_hash: + verification_warnings.append("Approved apply policy binding is missing") + if attempt_payload.get("pre_context_complete") is not True: + verification_warnings.append("Approved pre-apply workbook snapshot was incomplete") + if not pre_binding["complete"]: + verification_warnings.extend(f"Pre-apply proof: {item}" for item in pre_binding["errors"][:10]) + if expected_pre_fingerprint and pre_apply_workbook_fingerprint != expected_pre_fingerprint: + verification_warnings.append("Pre-apply workbook fingerprint does not match the approved fingerprint") + if not post_apply_workbook_fingerprint: + verification_warnings.append("Post-apply workbook fingerprint is required") + if not post_binding["complete"]: + verification_warnings.extend(f"Post-apply proof: {item}" for item in post_binding["errors"][:10]) + expected_pre_target_hashes = attempt_payload.get("pre_target_hashes") or {} + if pre_binding["target_hashes"] != expected_pre_target_hashes: + verification_warnings.append("Pre-apply exact target hashes do not match the approved attempt") + proof = { + "schema_version": "2026-08-office-apply-proof-v1", + "attempt_id": attempt_id, + "plan_id": plan_id, + "operation_registry_hash": registry_hash, + "apply_policy_hash": expected_policy_hash, + "pre_workbook_fingerprint": pre_apply_workbook_fingerprint or "", + "post_workbook_fingerprint": post_apply_workbook_fingerprint or "", + "pre_host_fingerprint": str((pre_host_snapshot or {}).get("fingerprint") or ""), + "post_host_fingerprint": str((post_host_snapshot or {}).get("fingerprint") or ""), + "pre_target_hashes": pre_binding["target_hashes"], + "post_target_hashes": post_binding["target_hashes"], + } + if errors: + verification_warnings.append(f"{len(errors)} operation result(s) reported errors despite applied status") + for operation_id in expected_ids: + matches = results_by_id.get(operation_id, []) + if len(matches) != 1: + verification_warnings.append( + f"Operation {operation_id or ''} returned {len(matches)} results; exactly one is required" + ) + continue + item = matches[0] + item_status = str(item.get("status") or "") + if item_status != "success": + verification_warnings.append( + f"Operation {operation_id} reported {item_status or 'missing'} instead of success" + ) + continue + expected_type = str(expected_by_id[operation_id].get("type") or "") + if str(item.get("type") or "") != expected_type: + verification_warnings.append(f"Operation {operation_id} result type does not match {expected_type}") + item_dirty = item.get("dirtyRanges") or item.get("dirty_ranges") or [] + if not item_dirty: + verification_warnings.append(f"Operation {operation_id} returned no dirty range") + restore_ops, restore_errors = _validated_result_restore_operations(expected_by_id[operation_id], item) + rollback_restore_operations.extend(restore_ops) + verification_warnings.extend(restore_errors) + + unexpected = [ + operation_id + for operation_id in results_by_id + if operation_id and operation_id not in expected_by_id + ] + if unexpected: + verification_warnings.append( + f"Office.js returned result(s) for unexpected operation ids: {', '.join(unexpected[:5])}" + ) + reported_applied = office_apply_result.get("applied") if isinstance(office_apply_result, dict) else None + if reported_applied is not None: + try: + applied_count = int(reported_applied) + except (TypeError, ValueError): + applied_count = -1 + if applied_count != len(mutation_ops): + verification_warnings.append( + f"Office.js reported {reported_applied} applied mutation(s); expected {len(mutation_ops)}" + ) + top_level_warnings = office_apply_result.get("warnings") if isinstance(office_apply_result, dict) else [] + if top_level_warnings: + verification_warnings.append("Office.js returned batch warnings for an applied result") formula_summary: dict[str, Any] = {} if post_apply_workbook_state: @@ -2250,7 +3039,8 @@ def _verify_apply_result( formula_summary = {"error": str(exc)} verification_warnings.append("Formula verification could not complete") - status = "failed" if errors or requested_status == "failed" else "warning" if verification_warnings or warnings else "verified" + incomplete_apply = requested_status == "applied" and bool(verification_warnings or warnings) + status = "failed" if errors or requested_status == "failed" or incomplete_apply else "warning" if verification_warnings or warnings else "verified" return { "status": status, "expected_mutations": len(mutation_ops), @@ -2258,11 +3048,406 @@ def _verify_apply_result( "dirty_ranges": dirty_ranges[:100], "office_warnings": warnings[:25], "verification_warnings": verification_warnings[:25], + "rollback_restore_operations": list(reversed(rollback_restore_operations)), "formula_analysis": formula_summary, + "proof": proof, "error_detail": error_detail, } +def _stable_payload_hash(value: Any) -> str: + payload = json.dumps(value, sort_keys=True, separators=(",", ":"), default=str, ensure_ascii=False) + return hashlib.sha256(payload.encode("utf-8")).hexdigest() + + +def _workbook_target_binding( + workbook_context: dict, + operations: list[dict], + *, + require_post_match: bool = False, +) -> dict[str, Any]: + context = workbook_context if isinstance(workbook_context, dict) else {} + errors: list[str] = [] + rollback = context.get("rollback_snapshot") or {} + if not isinstance(rollback, dict) or rollback.get("complete") is not True: + errors.append("rollback snapshot is missing or incomplete") + if isinstance(rollback, dict): + errors.extend(str(item) for item in (rollback.get("errors") or []) if item) + target_hashes: dict[str, str] = {} + for index, operation in enumerate(operations): + operation_id = str(operation.get("id") or f"operation_{index + 1}") + target_state, target_errors = _operation_target_state(context, operation) + errors.extend(f"{operation_id}: {item}" for item in target_errors) + target_hashes[operation_id] = _stable_payload_hash(target_state) + if require_post_match: + matches, reason = _operation_matches_workbook_context(context, operation, target_state) + if not matches: + errors.append(f"{operation_id}: {reason}") + return { + "complete": not errors, + "errors": list(dict.fromkeys(errors)), + "target_hashes": target_hashes, + } + + +def _operation_target_state(context: dict, operation: dict) -> tuple[Any, list[str]]: + operation_type = str(operation.get("type") or "") + target_range = _operation_range_cells(operation) + if target_range: + cells: list[dict] = [] + missing: list[str] = [] + indexed = _indexed_context_cells(context) + for key in target_range: + cell = indexed.get(key) + if cell is None: + missing.append(key) + continue + cells.append( + { + "key": key, + "value": cell.get("value"), + "formula": cell.get("formula"), + "format": cell.get("format"), + "blank": bool(cell.get("blank")), + } + ) + return cells, ([f"exact target cells are absent: {', '.join(missing[:5])}"] if missing else []) + + family = _operation_inventory_family(operation_type) + if family: + raw_objects = context.get("objects") + objects: dict[str, Any] = raw_objects if isinstance(raw_objects, dict) else {} + raw_status_map = objects.get("inventory_status") + status_map: dict[str, Any] = raw_status_map if isinstance(raw_status_map, dict) else {} + raw_status = status_map.get(family) + status: dict[str, Any] = raw_status if isinstance(raw_status, dict) else {} + errors: list[str] = [] + if status.get("complete") is not True: + errors.append(f"{family} inventory is incomplete") + errors.extend(str(item) for item in (status.get("errors") or []) if item) + return {"inventory": objects.get(family) or [], "status": status}, errors + + if operation_type in {"create_sheet", "delete_sheet", "rename_sheet"}: + raw_metadata = context.get("sheet_metadata") + metadata: list[Any] = raw_metadata if isinstance(raw_metadata, list) else [] + errors = [str(item.get("readError")) for item in metadata if isinstance(item, dict) and item.get("readError")] + names = context.get("sheet_names") or [item.get("name") for item in metadata if isinstance(item, dict)] + return {"sheet_names": names, "sheet_metadata": metadata}, errors + + if operation_type in {"set_named_range", "delete_named_range"}: + if not isinstance(context.get("named_ranges"), list): + return [], ["named-range inventory is unavailable"] + return context.get("named_ranges"), [] + + return None, [f"no exact target proof is defined for {operation_type or 'unknown operation'}"] + + +def _flatten_rectangular_operation_matrix(value: Any) -> tuple[list[Any], Optional[tuple[int, int]]]: + if not isinstance(value, list) or not value: + return [], None + if any(not isinstance(row, list) or not row for row in value): + return [], None + row_widths = {len(row) for row in value} + if len(row_widths) != 1: + return [], None + width = next(iter(row_widths)) + return [item for row in value for item in row], (len(value), width) + + +def _operation_range_matrix_shape(operation: dict) -> Optional[tuple[int, int]]: + target_value = operation.get("target") + raw_target: dict[str, Any] = target_value if isinstance(target_value, dict) else {} + reference = str( + operation.get("range") + or operation.get("cell") + or raw_target.get("range") + or "A1" + ).replace("$", "") + _, separator, address = reference.rpartition("!") + match = re.fullmatch( + r"([A-Za-z]+)(\d+)(?::([A-Za-z]+)(\d+))?", + address if separator else reference, + ) + if not match: + return None + start_col = _column_number(match.group(1)) + start_row = int(match.group(2)) + end_col = _column_number(match.group(3) or match.group(1)) + end_row = int(match.group(4) or match.group(2)) + if end_col < start_col or end_row < start_row: + return None + return end_row - start_row + 1, end_col - start_col + 1 + + +def _operation_matches_workbook_context(context: dict, operation: dict, target_state: Any) -> tuple[bool, str]: + operation_type = str(operation.get("type") or "") + cells = target_state if isinstance(target_state, list) else [] + if operation_type == "write_cell": + if len(cells) != 1: + return False, "exact post-apply cell is unavailable" + cell = cells[0] + if operation.get("formula") is not None: + return (str(cell.get("formula") or "") == str(operation.get("formula") or ""), "post-apply formula differs") + return (_same_workbook_value(cell.get("value"), operation.get("value")), "post-apply value differs") + if operation_type in {"write_range", "set_cell_range", "rollback_restore"}: + raw_formulas = operation.get("formulas") + raw_values = operation.get("values") + formulas, formula_shape = _flatten_rectangular_operation_matrix(raw_formulas) + values, value_shape = _flatten_rectangular_operation_matrix(raw_values) + range_shape = _operation_range_matrix_shape(operation) + if ( + (not formulas and not values) + or range_shape is None + or (raw_formulas and formula_shape is None) + or (raw_values and value_shape is None) + or (formulas and len(formulas) != len(cells)) + or (values and len(values) != len(cells)) + or (formula_shape is not None and formula_shape != range_shape) + or (value_shape is not None and value_shape != range_shape) + or (formula_shape is not None and value_shape is not None and formula_shape != value_shape) + ): + return False, "post-apply target matrix dimensions are incomplete" + number_formats: list[Any] = [] + if operation_type == "rollback_restore": + number_formats, number_format_shape = _flatten_rectangular_operation_matrix( + operation.get("numberFormats") + ) + if ( + number_format_shape is None + or len(number_formats) != len(cells) + or any(not isinstance(item, str) for item in number_formats) + or (formula_shape is not None and number_format_shape != formula_shape) + or (value_shape is not None and number_format_shape != value_shape) + ): + return False, "post-apply number format matrix dimensions are incomplete" + for index, cell in enumerate(cells): + formula = formulas[index] if formulas else None + value = values[index] if values else None + if isinstance(formula, str) and formula.startswith("="): + if str(cell.get("formula") or "") != formula: + return False, "post-apply formula matrix differs" + elif values and not _same_workbook_value(cell.get("value"), value): + return False, "post-apply value matrix differs" + elif formulas and str(cell.get("formula") or ""): + return False, "an unexpected post-apply formula remains" + actual_number_format = cell.get("format") + if number_formats and str( + actual_number_format if actual_number_format is not None else "" + ) != number_formats[index]: + return False, "post-apply number format matrix differs" + return True, "" + if operation_type in {"clear_range", "clear_cell_range"}: + return ( + bool(cells) and all( + cell.get("blank") is True + or (cell.get("value") in {None, ""} and not str(cell.get("formula") or "")) + for cell in cells + ), + "one or more cleared cells still contains data", + ) + if operation_type == "format_range": + expected = (operation.get("format") or {}).get("number_format") if isinstance(operation.get("format"), dict) else None + if expected is None: + return False, "format proof is not represented completely" + return (bool(cells) and all(str(cell.get("format") or "") == str(expected) for cell in cells), "post-apply number format differs") + + sheet_names = {str(item).strip().lower() for item in (context.get("sheet_names") or [])} + sheet = str(operation.get("sheet") or operation.get("sheet_name") or "").strip().lower() + if operation_type == "create_sheet": + expected = str(operation.get("sheet_name") or operation.get("name") or "").strip().lower() + return (bool(expected) and expected in sheet_names, "created sheet is absent") + if operation_type == "delete_sheet": + return (bool(sheet) and sheet not in sheet_names, "deleted sheet remains present") + if operation_type == "rename_sheet": + expected = str(operation.get("new_sheet_name") or operation.get("newSheetName") or "").strip().lower() + return (bool(sheet and expected) and sheet not in sheet_names and expected in sheet_names, "sheet rename was not observed") + + family = _operation_inventory_family(operation_type) + if family: + inventory = (target_state or {}).get("inventory") if isinstance(target_state, dict) else [] + name = str(operation.get("objectId") or operation.get("object_id") or operation.get("name") or "").strip().lower() + present = any( + str(item.get("id") or item.get("name") or "").strip().lower() == name + for item in (inventory or []) + if isinstance(item, dict) + ) + if operation_type.startswith("delete_"): + return (bool(name) and not present, "deleted object remains present") + return (bool(name) and present, "expected object is absent") + + if operation_type in {"set_named_range", "delete_named_range"}: + name = str(operation.get("name") or "").strip().lower() + found = any(str(item.get("name") or "").strip().lower() == name for item in (target_state or []) if isinstance(item, dict)) + return ((not found) if operation_type == "delete_named_range" else found, "named-range result differs") + return False, f"no deterministic post-apply proof is defined for {operation_type or 'unknown operation'}" + + +def _indexed_context_cells(context: dict) -> dict[str, dict]: + indexed: dict[str, dict] = {} + active_sheet = str(context.get("active_sheet") or "Sheet1") + for raw_key, raw_cell in (context.get("cell_snapshot") or {}).items(): + if not isinstance(raw_cell, dict): + continue + key = str(raw_key or "") + key_sheet, _, key_address = key.rpartition("!") + sheet = str(raw_cell.get("sheet") or key_sheet or active_sheet).strip("'").lower() + address = str(raw_cell.get("address") or key_address or key).replace("$", "").upper() + indexed[f"{sheet}!{address}"] = raw_cell + return indexed + + +def _operation_range_cells(operation: dict) -> list[str]: + target_value = operation.get("target") + raw_target: dict[str, Any] = target_value if isinstance(target_value, dict) else {} + reference = operation.get("range") or operation.get("cell") or raw_target.get("range") + range_types = { + "write_cell", "write_range", "set_cell_range", "rollback_restore", "clear_range", + "clear_cell_range", "format_range", "copy_to", "resize_range", "merge_range", "unmerge_range", + } + if str(operation.get("type") or "") not in range_types: + return [] + reference = str(reference or "A1").replace("$", "") + reference_sheet, separator, address = reference.rpartition("!") + sheet = str( + operation.get("sheet") + or operation.get("sheet_name") + or raw_target.get("sheet") + or (reference_sheet if separator else "") + ).strip("'").lower() + address = address if separator else reference + match = re.fullmatch(r"([A-Za-z]+)(\d+)(?::([A-Za-z]+)(\d+))?", address) + if not sheet or not match: + return [] + start_col = _column_number(match.group(1)) + start_row = int(match.group(2)) + end_col = _column_number(match.group(3) or match.group(1)) + end_row = int(match.group(4) or match.group(2)) + return [ + f"{sheet}!{_column_label(column)}{row}" + for row in range(start_row, end_row + 1) + for column in range(start_col, end_col + 1) + ] + + +def _column_number(label: str) -> int: + result = 0 + for char in label.upper(): + result = result * 26 + ord(char) - 64 + return result + + +def _column_label(number: int) -> str: + result = "" + while number > 0: + number, remainder = divmod(number - 1, 26) + result = chr(65 + remainder) + result + return result + + +def _operation_inventory_family(operation_type: str) -> Optional[str]: + if "chart" in operation_type: + return "charts" + if "pivot" in operation_type: + return "pivots" + if "table" in operation_type: + return "tables" + if "comment" in operation_type: + return "comments" + return None + + +def _same_workbook_value(actual: Any, expected: Any) -> bool: + if isinstance(actual, bool) or isinstance(expected, bool): + return isinstance(actual, bool) and isinstance(expected, bool) and actual is expected + if actual == expected: + return True + return (actual is None or actual == "") and (expected is None or expected == "") + + +def _validated_result_restore_operations(operation: dict, result: dict) -> tuple[list[dict], list[str]]: + """Accept only rollback operations that are bound to the successfully applied operation.""" + snapshot = result.get("rollbackSnapshot") or result.get("rollback_snapshot") + if snapshot is None: + # Older successful clients rely on the durable pre-apply workbook + # snapshot. New clients always send an operation-scoped proof. + return [], [] + if not isinstance(snapshot, dict): + return [], [f"Operation {operation.get('id')} returned an invalid rollback snapshot"] + proof_class = str(snapshot.get("proofClass") or snapshot.get("proof_class") or "manual") + unsupported = snapshot.get("unsupportedRollbackWarnings") or snapshot.get("unsupported_rollback_warnings") or [] + if proof_class == "manual" or unsupported: + return [], [f"Operation {operation.get('id')} requires manual rollback and cannot be recorded as applied"] + restores = snapshot.get("restoreOperations") or snapshot.get("restore_operations") or [] + if not isinstance(restores, list) or not restores: + return [], [f"Operation {operation.get('id')} rollback proof contains no inverse operation"] + + op_type = str(operation.get("type") or "") + allowed_inverse_types = { + "write_cell": {"rollback_restore"}, + "write_range": {"rollback_restore"}, + "set_cell_range": {"rollback_restore"}, + "format_range": {"rollback_restore"}, + "clear_range": {"rollback_restore"}, + "clear_cell_range": {"rollback_restore"}, + "copy_to": {"rollback_restore"}, + "create_sheet": {"delete_sheet"}, + "create_chart": {"delete_chart"}, + "update_chart": {"update_chart"}, + "delete_chart": {"create_chart"}, + "create_table": {"delete_table"}, + "update_table": {"update_table"}, + "delete_table": {"create_table"}, + "add_comment": {"delete_comment"}, + "update_comment": {"update_comment"}, + "delete_comment": {"add_comment"}, + }.get(op_type, set()) + if not allowed_inverse_types: + return [], [f"Operation {operation.get('id')} has no approved automatic inverse"] + + raw_target = operation.get("target") + target = raw_target if isinstance(raw_target, dict) else {} + expected_sheet = str( + operation.get("sheet") + or operation.get("sheet_name") + or operation.get("sheetName") + or target.get("sheet") + or "" + ) + expected_range = str(operation.get("range") or target.get("range") or "") + range_bound = op_type in { + "write_cell", "write_range", "set_cell_range", "format_range", "clear_range", + "clear_cell_range", "copy_to", "add_comment", "update_comment", "delete_comment", + } + accepted: list[dict] = [] + errors: list[str] = [] + for raw_restore in restores: + if not isinstance(raw_restore, dict): + errors.append(f"Operation {operation.get('id')} returned a non-object inverse") + continue + restore = dict(raw_restore) + if str(restore.get("type") or "") not in allowed_inverse_types: + errors.append(f"Operation {operation.get('id')} returned an unapproved inverse type") + continue + restore_sheet = str(restore.get("sheet") or "") + if expected_sheet and restore_sheet.lower() != expected_sheet.lower(): + errors.append(f"Operation {operation.get('id')} inverse targets a different sheet") + continue + if range_bound and expected_range and str(restore.get("range") or "").upper() != expected_range.upper(): + errors.append(f"Operation {operation.get('id')} inverse targets a different range") + continue + restore.pop("id", None) + if op_type == "create_sheet" and restore.get("type") == "delete_sheet": + restore.update( + { + "rollbackOnly": True, + "rollbackStrategy": "delete_created_sheet", + "sourceOperationId": str(operation.get("id") or ""), + } + ) + accepted.append(restore) + return accepted, errors + + def _broken_formula_cells(workbook_context: dict) -> list[str]: broken: list[str] = [] for address, info in (workbook_context.get("cell_snapshot") or {}).items(): diff --git a/backend/api/routes/templates.py b/backend/api/routes/templates.py index 8708c9f..2cb2b5f 100644 --- a/backend/api/routes/templates.py +++ b/backend/api/routes/templates.py @@ -2,8 +2,6 @@ from __future__ import annotations -from typing import Optional - from fastapi import APIRouter, HTTPException from pydantic import BaseModel @@ -14,7 +12,7 @@ class TemplateRequest(BaseModel): - params: Optional[dict] = None + params: dict | None = None @router.get("/templates") @@ -33,7 +31,7 @@ async def api_get_template(template_id: str): @router.post("/templates/{template_id}/generate") -async def api_generate_template(template_id: str, request: TemplateRequest = None): +async def api_generate_template(template_id: str, request: TemplateRequest | None = None): """Generate cell operations from a template with optional parameters.""" try: params = request.params if request else None @@ -48,4 +46,4 @@ async def api_generate_template(template_id: str, request: TemplateRequest = Non "count": len(operations), } except KeyError as e: - raise HTTPException(status_code=404, detail=str(e)) + raise HTTPException(status_code=404, detail=str(e)) from e diff --git a/backend/api/security/ai_policy.py b/backend/api/security/ai_policy.py new file mode 100644 index 0000000..ac66f12 --- /dev/null +++ b/backend/api/security/ai_policy.py @@ -0,0 +1,402 @@ +"""Effective admin policy for AI provider and source-data controls.""" + +from __future__ import annotations + +import os +from dataclasses import dataclass, field, replace +from typing import Any, Iterable + + +LOCAL_PROVIDERS = {"lmstudio", "ollama"} +CLOUD_PROVIDERS = {"anthropic", "openai"} +KNOWN_PROVIDERS = (*sorted(LOCAL_PROVIDERS), *sorted(CLOUD_PROVIDERS)) +KNOWN_MODES = ("ask", "plan", "preview", "action") +MODE_PERMISSIONS = { + "ask": "tasks.ask", + "plan": "tasks.plan", + "preview": "tasks.plan", + "action": "tasks.execute", +} + + +class AiPolicyViolation(ValueError): + def __init__(self, reason: str, code: str) -> None: + super().__init__(reason) + self.reason = reason + self.code = code + + def to_detail(self) -> dict[str, str]: + return {"code": self.code, "reason": self.reason} + + +@dataclass(frozen=True) +class EffectiveAiPolicy: + schema_version: str = "2026-06-effective-ai-policy-v1" + enterprise_mode: bool = False + byok_enabled: bool = False + zero_data_retention: bool = False + data_retention_hours: int = 0 + allowed_modes: tuple[str, ...] = KNOWN_MODES + allowed_providers: tuple[str, ...] = KNOWN_PROVIDERS + external_sending_allowed: bool = True + external_documents_allowed: bool = True + byok_required_for_cloud: bool = False + export_regenerated_fallback_allowed: bool = True + export_macro_regenerated_fallback_allowed: bool = False + policy_source: str = "environment" + local_providers: tuple[str, ...] = field(default_factory=lambda: tuple(sorted(LOCAL_PROVIDERS))) + cloud_providers: tuple[str, ...] = field(default_factory=lambda: tuple(sorted(CLOUD_PROVIDERS))) + + @classmethod + def from_env(cls) -> "EffectiveAiPolicy": + enterprise_mode = _bool_env("ENTERPRISE_MODE", False) + zero_data_retention = _bool_env("ZERO_DATA_RETENTION", False) + return cls( + enterprise_mode=enterprise_mode, + byok_enabled=_bool_env("BYOK_ENABLED", False), + zero_data_retention=zero_data_retention, + data_retention_hours=_int_env("AI_SOURCE_RETENTION_HOURS", _int_env("DATA_RETENTION_HOURS", 0)), + allowed_modes=tuple(_csv_env("AI_ALLOWED_MODES", KNOWN_MODES)), + allowed_providers=tuple(_csv_env("AI_ALLOWED_PROVIDERS", KNOWN_PROVIDERS)), + external_sending_allowed=_bool_env("AI_EXTERNAL_SENDING_ALLOWED", not zero_data_retention), + external_documents_allowed=_bool_env("AI_EXTERNAL_DOCUMENTS_ALLOWED", not zero_data_retention), + export_regenerated_fallback_allowed=_bool_env("AI_EXPORT_REGENERATED_FALLBACK_ALLOWED", True), + export_macro_regenerated_fallback_allowed=_bool_env("AI_EXPORT_MACRO_REGENERATED_FALLBACK_ALLOWED", False), + ) + + def require_request( + self, + *, + mode: str, + provider: str | None = None, + source_document_ids: Iterable[str] | None = None, + actor: Any | None = None, + ) -> None: + normalized_mode = (mode or "").strip().lower() + if normalized_mode not in self.allowed_modes: + raise AiPolicyViolation( + f"AI mode '{mode}' is disabled by admin policy.", + "ai_mode_disabled", + ) + if actor is not None and not _actor_can_use_mode(actor, normalized_mode): + role = _actor_role_value(actor) or "unknown" + permission = MODE_PERMISSIONS.get(normalized_mode, "tasks.execute") + raise AiPolicyViolation( + f"Role '{role}' does not have permission '{permission}' for AI mode '{normalized_mode}'.", + "ai_role_not_permitted", + ) + normalized_provider = (provider or "").strip().lower() + if normalized_provider: + self.require_provider(normalized_provider) + if source_document_ids and not self.external_documents_allowed: + raise AiPolicyViolation( + "Source documents are disabled by admin policy.", + "external_documents_disabled", + ) + + def require_provider(self, provider: str) -> str: + """Return a normalized provider only when policy permits its data boundary.""" + normalized = (provider or "").strip().lower() + if normalized not in KNOWN_PROVIDERS: + raise AiPolicyViolation( + f"AI provider '{provider}' is not recognized.", + "ai_provider_unknown", + ) + if normalized not in self.allowed_providers: + raise AiPolicyViolation( + f"AI provider '{provider}' is disabled by admin policy.", + "ai_provider_disabled", + ) + if normalized in CLOUD_PROVIDERS and not self.external_sending_allowed: + raise AiPolicyViolation( + f"AI provider '{provider}' would send data outside the local boundary.", + "external_sending_disabled", + ) + return normalized + + def approved_provider_routes( + self, + routes: Iterable[tuple[str, str]], + ) -> tuple[tuple[str, str], ...]: + """Filter automatic routes through the effective policy or fail closed.""" + approved: list[tuple[str, str]] = [] + for provider, model in routes: + try: + normalized = self.require_provider(provider) + except AiPolicyViolation: + continue + approved.append((normalized, model)) + if not approved: + raise AiPolicyViolation( + "No configured AI provider route is approved by admin policy.", + "no_approved_ai_provider", + ) + return tuple(approved) + + def require_export_fallback( + self, + *, + allow_regenerated_fallback: bool, + allow_macro_regenerated_fallback: bool, + actor: Any | None = None, + ) -> None: + if actor is not None and not _actor_can_use_permission(actor, "tasks.execute"): + role = _actor_role_value(actor) or "unknown" + raise AiPolicyViolation( + f"Role '{role}' does not have permission 'tasks.execute' for spreadsheet export.", + "ai_role_not_permitted", + ) + if allow_regenerated_fallback and not self.export_regenerated_fallback_allowed: + raise AiPolicyViolation( + "Regenerated workbook export fallback is disabled by admin policy.", + "export_regenerated_fallback_disabled", + ) + if allow_macro_regenerated_fallback and not self.export_macro_regenerated_fallback_allowed: + raise AiPolicyViolation( + "Macro workbook regenerated export fallback is disabled by admin policy.", + "export_macro_regenerated_fallback_disabled", + ) + + def provider_policy(self, provider: str) -> dict[str, object]: + normalized = provider.strip().lower() + is_cloud = normalized in CLOUD_PROVIDERS + return { + "allowed": normalized in self.allowed_providers and (not is_cloud or self.external_sending_allowed), + "scope": "cloud" if is_cloud else "local", + "externalSendingAllowed": self.external_sending_allowed if is_cloud else False, + "byokRequired": self.byok_required_for_cloud if is_cloud else False, + "source": self.policy_source, + } + + def to_dict(self, actor: Any | None = None) -> dict[str, object]: + return { + "schemaVersion": self.schema_version, + "enterpriseMode": self.enterprise_mode, + "byokEnabled": self.byok_enabled, + "zeroDataRetention": self.zero_data_retention, + "policySource": self.policy_source, + "sourceRetention": { + "hours": self.data_retention_hours, + "mode": "disabled" if self.zero_data_retention else "time_bound" if self.data_retention_hours else "not_configured", + }, + "allowedModes": list(self.allowed_modes), + "allowedProviders": list(self.allowed_providers), + "externalSendingAllowed": self.external_sending_allowed, + "externalDocumentsAllowed": self.external_documents_allowed, + "byokRequiredForCloud": self.byok_required_for_cloud, + "exportFallback": { + "regeneratedAllowed": self.export_regenerated_fallback_allowed, + "macroRegeneratedAllowed": self.export_macro_regenerated_fallback_allowed, + }, + "localProviders": list(self.local_providers), + "cloudProviders": list(self.cloud_providers), + "roleEntitlements": _role_entitlements(), + "requesterEntitlements": _actor_entitlements(actor) if actor is not None else None, + } + + +def get_effective_ai_policy() -> EffectiveAiPolicy: + policy = EffectiveAiPolicy.from_env() + config = _load_enterprise_provider_policy() + if config: + policy = _apply_enterprise_provider_policy(policy, config) + return policy + + +def _apply_enterprise_provider_policy(policy: EffectiveAiPolicy, config: dict[str, Any]) -> EffectiveAiPolicy: + allowed_providers = list(policy.allowed_providers) + stored_allowed = _list_config(config, "allowed_providers", "allowedProviders") + if stored_allowed is not None: + allowed_providers = _ordered_intersection(allowed_providers, stored_allowed) + + allow_local = _bool_config(config, True, "allow_local", "allowLocal") + allow_cloud = _bool_config(config, True, "allow_cloud", "allowCloud") + if not allow_local: + allowed_providers = [provider for provider in allowed_providers if provider not in LOCAL_PROVIDERS] + if not allow_cloud: + allowed_providers = [provider for provider in allowed_providers if provider not in CLOUD_PROVIDERS] + + require_byok = _bool_config(config, False, "require_byok_for_cloud", "requireByokForCloud") + if require_byok and not policy.byok_enabled: + allowed_providers = [provider for provider in allowed_providers if provider not in CLOUD_PROVIDERS] + + allowed_modes = list(policy.allowed_modes) + stored_modes = _list_config(config, "allowed_modes", "allowedModes") + if stored_modes is not None: + allowed_modes = _ordered_intersection(allowed_modes, stored_modes) + + external_sending_allowed = policy.external_sending_allowed and allow_cloud + stored_external_sending = _optional_bool_config(config, "external_sending_allowed", "externalSendingAllowed") + if stored_external_sending is not None: + external_sending_allowed = external_sending_allowed and stored_external_sending + if require_byok and not policy.byok_enabled: + external_sending_allowed = False + + external_documents_allowed = policy.external_documents_allowed + stored_external_documents = _optional_bool_config(config, "external_documents_allowed", "externalDocumentsAllowed") + if stored_external_documents is not None: + external_documents_allowed = external_documents_allowed and stored_external_documents + + export_regenerated_fallback_allowed = policy.export_regenerated_fallback_allowed + stored_export_fallback = _optional_bool_config( + config, + "allow_export_regenerated_fallback", + "allowExportRegeneratedFallback", + "export_regenerated_fallback_allowed", + "exportRegeneratedFallbackAllowed", + ) + if stored_export_fallback is not None: + export_regenerated_fallback_allowed = export_regenerated_fallback_allowed and stored_export_fallback + + export_macro_regenerated_fallback_allowed = policy.export_macro_regenerated_fallback_allowed + stored_macro_fallback = _optional_bool_config( + config, + "allow_macro_regenerated_fallback", + "allowMacroRegeneratedFallback", + "export_macro_regenerated_fallback_allowed", + "exportMacroRegeneratedFallbackAllowed", + ) + if stored_macro_fallback is not None: + export_macro_regenerated_fallback_allowed = export_macro_regenerated_fallback_allowed and stored_macro_fallback + + return replace( + policy, + allowed_modes=tuple(allowed_modes), + allowed_providers=tuple(allowed_providers), + external_sending_allowed=external_sending_allowed, + external_documents_allowed=external_documents_allowed, + export_regenerated_fallback_allowed=export_regenerated_fallback_allowed, + export_macro_regenerated_fallback_allowed=export_macro_regenerated_fallback_allowed, + byok_required_for_cloud=require_byok, + policy_source="enterprise_config", + ) + + +def _load_enterprise_provider_policy() -> dict[str, Any]: + try: + from backend.storage import get_session_store + + stored = get_session_store().get_enterprise_config("provider_policy") + except Exception: + return {} + config = stored.get("config") if isinstance(stored, dict) else None + return config if isinstance(config, dict) else {} + + +def _role_entitlements() -> dict[str, dict[str, object]]: + from backend.api.security.rbac import PERMISSIONS + + return { + role.value: { + "permissions": sorted(permissions), + "allowedModes": _modes_for_permissions(permissions), + } + for role, permissions in PERMISSIONS.items() + } + + +def _actor_entitlements(actor: Any) -> dict[str, object]: + from backend.api.security.rbac import PERMISSIONS + + role = _actor_role(actor) + permissions = PERMISSIONS.get(role, set()) + return { + "role": role.value, + "userId": _actor_value(actor, "user_id") or _actor_value(actor, "email") or "", + "permissions": sorted(permissions), + "allowedModes": _modes_for_permissions(permissions), + } + + +def _actor_can_use_mode(actor: Any, mode: str) -> bool: + permission = MODE_PERMISSIONS.get(mode, "tasks.execute") + return _actor_can_use_permission(actor, permission) + + +def _actor_can_use_permission(actor: Any, permission: str) -> bool: + from backend.api.security.rbac import PERMISSIONS + + return permission in PERMISSIONS.get(_actor_role(actor), set()) + + +def _actor_role(actor: Any): + from backend.api.security.rbac import Role + + value = _actor_role_value(actor) or Role.VIEWER.value + try: + return Role(value) + except ValueError: + return Role.VIEWER + + +def _actor_role_value(actor: Any) -> str: + return str(_actor_value(actor, "role") or "").strip().lower() + + +def _actor_value(actor: Any, key: str) -> Any: + if isinstance(actor, dict): + return actor.get(key) + return getattr(actor, key, None) + + +def _modes_for_permissions(permissions: set[str]) -> list[str]: + return [mode for mode, permission in MODE_PERMISSIONS.items() if permission in permissions] + + +def _ordered_intersection(current: Iterable[str], requested: Iterable[str]) -> list[str]: + requested_set = {item.strip().lower() for item in requested if item and str(item).strip()} + return [item for item in current if item in requested_set] + + +def _list_config(config: dict[str, Any], *names: str) -> list[str] | None: + for name in names: + value = config.get(name) + if value is None: + continue + if isinstance(value, str): + return [item.strip().lower() for item in value.split(",") if item.strip()] + if isinstance(value, Iterable): + return [str(item).strip().lower() for item in value if str(item).strip()] + return None + + +def _bool_config(config: dict[str, Any], default: bool, *names: str) -> bool: + value = _optional_bool_config(config, *names) + return default if value is None else value + + +def _optional_bool_config(config: dict[str, Any], *names: str) -> bool | None: + for name in names: + if name not in config: + continue + value = config.get(name) + if isinstance(value, bool): + return value + if isinstance(value, str): + return value.strip().lower() in {"1", "true", "yes", "on"} + return bool(value) + return None + + +def _csv_env(name: str, default: Iterable[str]) -> list[str]: + value = os.getenv(name) + if value is None: + return [item.strip().lower() for item in default if item.strip()] + return [item.strip().lower() for item in value.split(",") if item.strip()] + + +def _bool_env(name: str, default: bool) -> bool: + value = os.getenv(name) + if value is None: + return default + return value.strip().lower() in {"1", "true", "yes", "on"} + + +def _int_env(name: str, default: int) -> int: + value = os.getenv(name) + if value is None or value == "": + return default + try: + return max(0, int(value)) + except ValueError: + return default diff --git a/backend/api/security/local_pairing.py b/backend/api/security/local_pairing.py new file mode 100644 index 0000000..aff9313 --- /dev/null +++ b/backend/api/security/local_pairing.py @@ -0,0 +1,416 @@ +"""Loopback-only one-time pairing for the local desktop owner. + +The access token is intentionally opaque to clients. It is authenticated with +an HMAC secret held by the operating-system keychain and is also registered in +process so a backend restart revokes every outstanding local token. +""" + +from __future__ import annotations + +import base64 +import hashlib +import hmac +import ipaddress +import re +import secrets +import threading +import time +from dataclasses import dataclass +from typing import Callable + + +LOCAL_PAIRING_SCHEMA = "2026-08-local-pairing-v1" +_PAIRING_CODE_PATTERN = re.compile(r"^[A-Za-z0-9_-]{6,128}$") +_NONCE_PATTERN = re.compile(r"^[A-Za-z0-9_-]{43}$") +_TOKEN_PART_PATTERN = re.compile(r"^[A-Za-z0-9_-]+$") + + +class LocalPairingError(RuntimeError): + """A fail-closed local pairing or token validation error.""" + + def __init__( + self, + code: str, + *, + status_code: int = 401, + message: str = "Local pairing request was rejected.", + retry_after: int | None = None, + ) -> None: + super().__init__(message) + self.code = code + self.status_code = status_code + self.public_message = message + self.retry_after = retry_after + + +@dataclass(frozen=True) +class _LocalTokenRecord: + expires_at: float + client_nonce_sha256: str + + +class KeyringSecretLoader: + """Load or create the local token HMAC key in the OS credential vault.""" + + def __init__( + self, + *, + service: str = "Excel Agent Local Pairing", + account: str = "local-token-signing-v1", + ) -> None: + self.service = service + self.account = account + + def __call__(self) -> bytes: + try: + import keyring + + backend = keyring.get_keyring() + if float(getattr(backend, "priority", 0.0)) <= 0 or not _is_os_keyring_backend(backend): + raise RuntimeError("No operating-system keyring backend is available") + encoded = keyring.get_password(self.service, self.account) + if encoded: + secret = _decode_urlsafe(encoded) + if len(secret) < 32: + raise RuntimeError("Stored keyring secret is too short") + return secret + + secret = secrets.token_bytes(48) + encoded = _encode_urlsafe(secret) + keyring.set_password(self.service, self.account, encoded) + persisted = keyring.get_password(self.service, self.account) + if not persisted or not hmac.compare_digest(persisted, encoded): + raise RuntimeError("Operating-system keyring did not persist the signing secret") + return secret + except Exception as exc: + raise LocalPairingError( + "PAIRING_KEYCHAIN_UNAVAILABLE", + status_code=503, + message="Local pairing is unavailable because the OS keychain could not be opened.", + ) from exc + + +class LocalPairingManager: + """Consume one-time codes and validate short-lived local-owner tokens.""" + + def __init__( + self, + *, + enabled: bool, + production_mode: bool, + pairing_code: str = "", + code_ttl_seconds: int = 120, + token_ttl_seconds: int = 300, + max_attempts: int = 3, + lockout_seconds: int = 60, + secret_loader: Callable[[], bytes] | None = None, + clock: Callable[[], float] | None = None, + ) -> None: + self.enabled = bool(enabled) + self.production_mode = bool(production_mode) + self.code_ttl_seconds = int(code_ttl_seconds) + self.token_ttl_seconds = int(token_ttl_seconds) + self.max_attempts = int(max_attempts) + self.lockout_seconds = int(lockout_seconds) + self._secret_loader = secret_loader or KeyringSecretLoader() + self._clock = clock or time.monotonic + self._lock = threading.RLock() + self._secret: bytes | None = None + self._pending_pairing_code = str(pairing_code or "").strip() + self._code_was_configured = bool(self._pending_pairing_code) + self._pairing_code_digest: bytes | None = None + self._pairing_code_expires_at = 0.0 + self._failed_attempts = 0 + self._locked_until = 0.0 + self._tokens: dict[bytes, _LocalTokenRecord] = {} + + def ensure_ready(self) -> None: + """Initialize keychain state, failing startup if the profile is unsafe.""" + self._assert_enabled() + with self._lock: + self._load_secret_locked() + if self._pending_pairing_code: + code = self._pending_pairing_code + self._pending_pairing_code = "" + self._arm_pairing_code_locked(code, self.code_ttl_seconds) + if not self._code_was_configured: + raise LocalPairingError( + "PAIRING_CODE_UNAVAILABLE", + status_code=503, + message="Local pairing has not been armed by the desktop owner.", + ) + + def arm_pairing_code(self, pairing_code: str, *, ttl_seconds: int | None = None) -> None: + """Arm a new one-time code from the trusted desktop process boundary.""" + self._assert_enabled() + with self._lock: + self._load_secret_locked() + self._arm_pairing_code_locked(pairing_code, ttl_seconds or self.code_ttl_seconds) + + def exchange_code(self, pairing_code: str, client_nonce: str) -> dict[str, object]: + """Consume a valid code exactly once and issue an opaque bearer token.""" + self.ensure_ready() + normalized_code = str(pairing_code or "").strip() + normalized_nonce = str(client_nonce or "").strip() + if not _valid_pairing_code(normalized_code) or not _valid_client_nonce(normalized_nonce): + raise LocalPairingError( + "PAIRING_REQUEST_INVALID", + status_code=400, + message="Local pairing request is invalid.", + ) + + with self._lock: + now = self._clock() + if now < self._locked_until: + retry_after = max(int(self._locked_until - now) + 1, 1) + raise LocalPairingError( + "PAIRING_LOCKED", + status_code=429, + message="Too many local pairing attempts. Generate a new code after the lockout.", + retry_after=retry_after, + ) + if self._pairing_code_digest is None or now >= self._pairing_code_expires_at: + self._pairing_code_digest = None + raise LocalPairingError("PAIRING_REJECTED") + + actual_digest = self._code_digest_locked(normalized_code) + if not hmac.compare_digest(actual_digest, self._pairing_code_digest): + self._failed_attempts += 1 + if self._failed_attempts >= self.max_attempts: + self._pairing_code_digest = None + self._locked_until = now + self.lockout_seconds + raise LocalPairingError( + "PAIRING_LOCKED", + status_code=429, + message="Too many local pairing attempts. Generate a new code after the lockout.", + retry_after=self.lockout_seconds, + ) + raise LocalPairingError("PAIRING_REJECTED") + + # Burn the code before token creation. A failure after this point + # requires the trusted desktop process to arm a fresh code. + self._pairing_code_digest = None + self._pairing_code_expires_at = 0.0 + self._failed_attempts = 0 + self._locked_until = 0.0 + token = self._issue_token_locked(normalized_nonce, now) + + return { + "schema_version": LOCAL_PAIRING_SCHEMA, + "access_token": token, + "token_type": "Bearer", + "expires_in": self.token_ttl_seconds, + "client_nonce": normalized_nonce, + "subject": "local-owner", + "display_name": "Desktop owner", + } + + def authenticate(self, token: str) -> dict[str, str]: + """Validate a local bearer and return its user-bound owner identity.""" + self._assert_enabled(token_error=True) + normalized_token = str(token or "").strip() + parts = normalized_token.split(".") + if ( + len(parts) != 3 + or parts[0] != "local" + or not _TOKEN_PART_PATTERN.fullmatch(parts[1]) + or not _TOKEN_PART_PATTERN.fullmatch(parts[2]) + ): + raise _invalid_token() + + with self._lock: + self._load_secret_locked(token_error=True) + expected_signature = self._token_signature_locked(parts[1]) + if not hmac.compare_digest(parts[2], expected_signature): + raise _invalid_token() + now = self._clock() + self._prune_tokens_locked(now) + record = self._tokens.get(self._token_registry_key_locked(normalized_token)) + if record is None or now >= record.expires_at: + raise _invalid_token() + + return { + "user_id": "local-owner", + "email": "local-owner@localhost", + "name": "Desktop owner", + "role": "admin", + "auth_source": "local_pairing", + "client_nonce_sha256": record.client_nonce_sha256, + } + + def revoke_all(self) -> int: + """Revoke every process-local bearer, for shutdown or owner sign-out.""" + with self._lock: + count = len(self._tokens) + self._tokens.clear() + return count + + def _assert_enabled(self, *, token_error: bool = False) -> None: + if self.production_mode or not self.enabled: + if token_error: + raise _invalid_token() + raise LocalPairingError( + "LOCAL_PAIRING_DISABLED", + status_code=404, + message="Local pairing is disabled for this deployment.", + ) + + def _load_secret_locked(self, *, token_error: bool = False) -> bytes: + if self._secret is not None: + return self._secret + try: + secret = self._secret_loader() + if not isinstance(secret, bytes) or len(secret) < 32: + raise RuntimeError("Local pairing signing secret must be at least 32 bytes") + except LocalPairingError: + if token_error: + raise _invalid_token() from None + raise + except Exception as exc: + if token_error: + raise _invalid_token() from None + raise LocalPairingError( + "PAIRING_KEYCHAIN_UNAVAILABLE", + status_code=503, + message="Local pairing is unavailable because the OS keychain could not be opened.", + ) from exc + self._secret = secret + return secret + + def _arm_pairing_code_locked(self, pairing_code: str, ttl_seconds: int) -> None: + normalized = str(pairing_code or "").strip() + if not _valid_pairing_code(normalized) or not 30 <= int(ttl_seconds) <= 600: + raise LocalPairingError( + "PAIRING_CONFIGURATION_INVALID", + status_code=503, + message="Local pairing configuration is invalid.", + ) + self._pairing_code_digest = self._code_digest_locked(normalized) + self._pairing_code_expires_at = self._clock() + int(ttl_seconds) + self._code_was_configured = True + self._failed_attempts = 0 + self._locked_until = 0.0 + + def _code_digest_locked(self, pairing_code: str) -> bytes: + secret = self._load_secret_locked() + return hmac.new(secret, b"pairing-code-v1\0" + pairing_code.encode("ascii"), hashlib.sha256).digest() + + def _issue_token_locked(self, client_nonce: str, now: float) -> str: + if not 30 <= self.token_ttl_seconds <= 3600: + raise LocalPairingError( + "PAIRING_CONFIGURATION_INVALID", + status_code=503, + message="Local pairing configuration is invalid.", + ) + material = secrets.token_urlsafe(32) + token = f"local.{material}.{self._token_signature_locked(material)}" + self._prune_tokens_locked(now) + if len(self._tokens) >= 32: + oldest = min(self._tokens, key=lambda key: self._tokens[key].expires_at) + self._tokens.pop(oldest, None) + self._tokens[self._token_registry_key_locked(token)] = _LocalTokenRecord( + expires_at=now + self.token_ttl_seconds, + client_nonce_sha256=hashlib.sha256(client_nonce.encode("ascii")).hexdigest(), + ) + return token + + def _token_signature_locked(self, material: str) -> str: + secret = self._load_secret_locked() + signature = hmac.new(secret, b"local-token-v1\0" + material.encode("ascii"), hashlib.sha256).digest() + return _encode_urlsafe(signature) + + def _token_registry_key_locked(self, token: str) -> bytes: + secret = self._load_secret_locked() + return hmac.new(secret, b"local-token-record-v1\0" + token.encode("ascii"), hashlib.sha256).digest() + + def _prune_tokens_locked(self, now: float) -> None: + expired = [key for key, record in self._tokens.items() if now >= record.expires_at] + for key in expired: + self._tokens.pop(key, None) + + +def is_loopback_peer(host: str | None) -> bool: + """Validate the actual ASGI peer IP without trusting forwarded headers.""" + normalized = str(host or "").strip().removeprefix("[").removesuffix("]").split("%", 1)[0] + try: + address = ipaddress.ip_address(normalized) + except ValueError: + return False + if isinstance(address, ipaddress.IPv6Address) and address.ipv4_mapped is not None: + return address.ipv4_mapped.is_loopback + return address.is_loopback + + +def _valid_pairing_code(value: str) -> bool: + return bool(_PAIRING_CODE_PATTERN.fullmatch(value)) + + +def _valid_client_nonce(value: str) -> bool: + if not _NONCE_PATTERN.fullmatch(value): + return False + try: + return len(_decode_urlsafe(value)) == 32 + except ValueError: + return False + + +def _encode_urlsafe(value: bytes) -> str: + return base64.urlsafe_b64encode(value).rstrip(b"=").decode("ascii") + + +def _decode_urlsafe(value: str) -> bytes: + if not _TOKEN_PART_PATTERN.fullmatch(str(value or "")): + raise ValueError("Invalid base64url value") + padding = "=" * (-len(value) % 4) + return base64.urlsafe_b64decode(value + padding) + + +def _invalid_token() -> LocalPairingError: + return LocalPairingError( + "LOCAL_TOKEN_INVALID", + status_code=401, + message="Invalid or expired local bearer token.", + ) + + +def _is_os_keyring_backend(backend: object) -> bool: + module = type(backend).__module__ + secure_modules = { + "keyring.backends.SecretService", + "keyring.backends.Windows", + "keyring.backends.kwallet", + "keyring.backends.libsecret", + "keyring.backends.macOS", + } + return module in secure_modules + + +_manager: LocalPairingManager | None = None +_manager_lock = threading.Lock() + + +def get_local_pairing_manager() -> LocalPairingManager: + """Return the configured process-local pairing authority.""" + global _manager + with _manager_lock: + if _manager is None: + from backend.config import ( + LOCAL_PAIRING_CODE_TTL_SECONDS, + LOCAL_PAIRING_ENABLED, + LOCAL_PAIRING_LOCKOUT_SECONDS, + LOCAL_PAIRING_MAX_ATTEMPTS, + LOCAL_PAIRING_TOKEN_TTL_SECONDS, + PRODUCTION_MODE, + consume_local_pairing_code, + ) + + _manager = LocalPairingManager( + enabled=LOCAL_PAIRING_ENABLED, + production_mode=PRODUCTION_MODE, + pairing_code=consume_local_pairing_code(), + code_ttl_seconds=LOCAL_PAIRING_CODE_TTL_SECONDS, + token_ttl_seconds=LOCAL_PAIRING_TOKEN_TTL_SECONDS, + max_attempts=LOCAL_PAIRING_MAX_ATTEMPTS, + lockout_seconds=LOCAL_PAIRING_LOCKOUT_SECONDS, + ) + return _manager diff --git a/backend/api/security/oidc.py b/backend/api/security/oidc.py new file mode 100644 index 0000000..0c19445 --- /dev/null +++ b/backend/api/security/oidc.py @@ -0,0 +1,197 @@ +"""OIDC bearer-token validation for user-bound HTTP and WebSocket sessions. + +Tokens establish identity only. Authorization data (role, status, teams) is +always loaded from the local enterprise user store so an identity-provider +claim can never promote its own privileges. +""" + +from __future__ import annotations + +import asyncio +from collections.abc import Callable +from dataclasses import dataclass +from functools import lru_cache +from typing import Any + +import jwt +from jwt import PyJWKClient + + +class OIDCValidationError(ValueError): + """A bearer token could not establish an authorized local identity.""" + + +_ASYMMETRIC_ALGORITHMS = frozenset( + { + "RS256", + "PS256", + "ES256", + } +) + + +@dataclass(frozen=True, slots=True) +class OIDCConfig: + enabled: bool + issuer: str + audience: str + jwks_url: str + allowed_tenants: frozenset[str] = frozenset() + algorithms: tuple[str, ...] = ("RS256",) + clock_skew_seconds: int = 60 + require_mfa: bool = False + accepted_mfa_acr: frozenset[str] = frozenset() + jwks_timeout_seconds: int = 5 + jwks_cache_seconds: int = 300 + + def validate(self) -> None: + if not self.enabled: + return + if not self.issuer or not self.audience or not self.jwks_url: + raise ValueError("OIDC issuer, audience, and JWKS URL are required") + if not self.algorithms or any(algorithm not in _ASYMMETRIC_ALGORITHMS for algorithm in self.algorithms): + raise ValueError( + "OIDC algorithms must be an explicit Office-client-compatible allowlist" + ) + if self.clock_skew_seconds < 0 or self.clock_skew_seconds > 300: + raise ValueError("OIDC clock skew must be between 0 and 300 seconds") + if self.jwks_timeout_seconds < 1 or self.jwks_timeout_seconds > 30: + raise ValueError("OIDC JWKS timeout must be between 1 and 30 seconds") + if self.jwks_cache_seconds < 60: + raise ValueError("OIDC JWKS cache must be at least 60 seconds") + + +class OIDCAuthenticator: + """Validate an OIDC access token and resolve its pre-provisioned user.""" + + def __init__( + self, + config: OIDCConfig, + *, + user_loader: Callable[[str, str], dict[str, Any] | None], + signing_key_resolver: Callable[[str], Any] | None = None, + ) -> None: + config.validate() + self.config = config + self._user_loader = user_loader + self._jwk_client = None + if config.enabled and signing_key_resolver is None: + self._jwk_client = PyJWKClient( + config.jwks_url, + cache_keys=True, + timeout=config.jwks_timeout_seconds, + lifespan=config.jwks_cache_seconds, + ) + self._signing_key_resolver = signing_key_resolver or self._resolve_signing_key + + def _resolve_signing_key(self, token: str) -> Any: + if self._jwk_client is None: + raise OIDCValidationError("OIDC authentication is not configured") + return self._jwk_client.get_signing_key_from_jwt(token).key + + async def authenticate(self, token: str) -> dict[str, Any]: + """Validate off the event loop because JWKS refresh may perform I/O.""" + return await asyncio.to_thread(self.authenticate_sync, token) + + def authenticate_sync(self, token: str) -> dict[str, Any]: + if not self.config.enabled: + raise OIDCValidationError("OIDC authentication is not configured") + if not token or len(token) > 16384: + raise OIDCValidationError("identity token is missing or too large") + + try: + header = jwt.get_unverified_header(token) + algorithm = str(header.get("alg") or "") + if algorithm not in self.config.algorithms: + raise OIDCValidationError("identity token algorithm is not allowed") + signing_key = self._signing_key_resolver(token) + claims = jwt.decode( + token, + signing_key, + algorithms=list(self.config.algorithms), + audience=self.config.audience, + issuer=self.config.issuer, + leeway=self.config.clock_skew_seconds, + options={"require": ["exp", "iat", "iss", "sub"]}, + ) + except OIDCValidationError: + raise + except (jwt.PyJWTError, TypeError, ValueError, OSError): + raise OIDCValidationError("identity token validation failed") from None + + issuer = str(claims.get("iss") or "") + subject = str(claims.get("sub") or "") + tenant = str(claims.get("tid") or claims.get("tenant_id") or "") + if self.config.allowed_tenants and tenant not in self.config.allowed_tenants: + raise OIDCValidationError("identity tenant is not allowed") + + mfa_authenticated = self._has_mfa(claims) + if self.config.require_mfa and not mfa_authenticated: + raise OIDCValidationError("multi-factor authentication is required") + + user = self._user_loader(issuer, subject) + if not user: + raise OIDCValidationError("identity is not provisioned") + if str(user.get("status") or "").lower() != "active": + raise OIDCValidationError("identity is not active") + + raw_metadata = user.get("metadata") + metadata = raw_metadata if isinstance(raw_metadata, dict) else {} + bound_subject = str(user.get("oidc_subject") or metadata.get("oidc_subject") or "").strip() + bound_issuer = str(user.get("oidc_issuer") or metadata.get("oidc_issuer") or "").strip() + if not bound_subject or not bound_issuer: + raise OIDCValidationError("identity binding is incomplete") + if bound_subject != subject or bound_issuer != issuer: + raise OIDCValidationError("identity binding does not match") + + user_id = str(user.get("user_id") or "") + role = str(user.get("role") or "") + email = str(user.get("email") or "").strip().lower() + if not user_id or not role or not email: + raise OIDCValidationError("provisioned identity is incomplete") + return { + "user_id": user_id, + "email": email, + "name": str(user.get("name") or email), + "role": role, + "auth_source": "oidc", + "oidc_issuer": issuer, + "oidc_subject": subject, + "oidc_tenant": tenant, + "mfa_authenticated": mfa_authenticated, + } + + def _has_mfa(self, claims: dict[str, Any]) -> bool: + raw_amr = claims.get("amr") + amr = {str(value).lower() for value in raw_amr} if isinstance(raw_amr, list) else set() + if "mfa" in amr: + return True + acr = str(claims.get("acr") or "") + return bool(acr and acr in self.config.accepted_mfa_acr) + + +@lru_cache(maxsize=1) +def get_oidc_authenticator() -> OIDCAuthenticator: + from backend import config + from backend.storage import get_session_store + + oidc_config = OIDCConfig( + enabled=config.OIDC_ENABLED, + issuer=config.OIDC_ISSUER, + audience=config.OIDC_AUDIENCE, + jwks_url=config.OIDC_JWKS_URL, + allowed_tenants=config.OIDC_ALLOWED_TENANTS, + algorithms=config.OIDC_ALLOWED_ALGORITHMS, + clock_skew_seconds=config.OIDC_CLOCK_SKEW_SECONDS, + require_mfa=config.OIDC_REQUIRE_MFA, + accepted_mfa_acr=config.OIDC_ACCEPTED_MFA_ACR, + jwks_timeout_seconds=config.OIDC_JWKS_TIMEOUT_SECONDS, + jwks_cache_seconds=config.OIDC_JWKS_CACHE_SECONDS, + ) + return OIDCAuthenticator( + oidc_config, + user_loader=lambda issuer, subject: get_session_store().get_enterprise_user_by_oidc_identity( + issuer, + subject, + ), + ) diff --git a/backend/api/security/oidc_client.py b/backend/api/security/oidc_client.py new file mode 100644 index 0000000..2e7837a --- /dev/null +++ b/backend/api/security/oidc_client.py @@ -0,0 +1,151 @@ +"""Public OIDC authorization-code + PKCE client configuration. + +Only public OAuth metadata crosses this boundary. Client secrets, tenant +allowlists, signing policy, and JWKS validation settings remain server-side. +""" + +from __future__ import annotations + +import json +from collections.abc import Mapping +from typing import Any +from urllib.parse import parse_qs, urlsplit + + +SCHEMA_VERSION = "2026-08-oidc-client-config-v1" +_RESERVED_AUTHORIZE_PARAMS = { + "client_id", + "client_secret", + "code_challenge", + "code_challenge_method", + "nonce", + "redirect_uri", + "response_type", + "scope", + "state", +} + + +class OIDCClientConfigError(ValueError): + """Raised when public OIDC client metadata is unsafe or incomplete.""" + + +def _is_allowed_url(value: str, *, production: bool) -> bool: + parsed = urlsplit(value) + if parsed.scheme == "https" and parsed.netloc: + return True + return bool(not production and parsed.scheme == "http" and parsed.hostname in {"localhost", "127.0.0.1", "::1"}) + + +def _origin(value: str) -> str: + parsed = urlsplit(value) + if not parsed.scheme or not parsed.netloc: + return "" + return f"{parsed.scheme.lower()}://{parsed.netloc.lower()}" + + +def parse_extra_authorize_params(raw: str) -> dict[str, str]: + try: + decoded = json.loads(raw or "{}") + except json.JSONDecodeError as exc: + raise OIDCClientConfigError("OIDC_EXTRA_AUTHORIZE_PARAMS_JSON must be valid JSON") from exc + if not isinstance(decoded, dict): + raise OIDCClientConfigError("OIDC_EXTRA_AUTHORIZE_PARAMS_JSON must be a JSON object") + params: dict[str, str] = {} + for key, value in decoded.items(): + if not isinstance(key, str) or not isinstance(value, str): + raise OIDCClientConfigError("OIDC extra authorization parameters must be string pairs") + normalized_key = key.strip() + if not normalized_key or normalized_key.lower() in _RESERVED_AUTHORIZE_PARAMS: + raise OIDCClientConfigError(f"OIDC extra authorization parameter is reserved: {key!r}") + params[normalized_key] = value + return params + + +def validate_oidc_client_config(values: Mapping[str, Any], *, production: bool) -> list[str]: + errors: list[str] = [] + client_id = str(values.get("oidc_client_id") or "").strip() + issuer = str(values.get("oidc_issuer") or "").strip() + redirect_uri = str(values.get("oidc_client_redirect_uri") or "").strip() + discovery_url = str(values.get("oidc_discovery_url") or "").strip() + authorization_endpoint = str(values.get("oidc_authorization_endpoint") or "").strip() + token_endpoint = str(values.get("oidc_token_endpoint") or "").strip() + scopes = tuple(str(value).strip() for value in values.get("oidc_client_scopes") or () if str(value).strip()) + + if not client_id: + errors.append("OIDC_CLIENT_ID must identify the public PKCE client in production profiles") + if not _is_allowed_url(issuer, production=production): + errors.append("OIDC_ISSUER must be an HTTPS URL for the public PKCE client") + if not _is_allowed_url(redirect_uri, production=production): + errors.append("OIDC_CLIENT_REDIRECT_URI must be an HTTPS URL in production profiles") + else: + callback_values = parse_qs(urlsplit(redirect_uri).query).get("oidc_callback", []) + if callback_values != ["1"]: + errors.append("OIDC_CLIENT_REDIRECT_URI must include oidc_callback=1") + allowed_origins = {_origin(str(value).strip()) for value in values.get("cors_origins") or ()} + if production and _origin(redirect_uri) not in allowed_origins: + errors.append("OIDC_CLIENT_REDIRECT_URI must share an origin with CORS_ORIGINS") + if "openid" not in scopes: + errors.append("OIDC_CLIENT_SCOPES must include openid") + + has_discovery = bool(discovery_url) + has_explicit_endpoints = bool(authorization_endpoint and token_endpoint) + if not has_discovery and not has_explicit_endpoints: + errors.append("OIDC_DISCOVERY_URL or both OIDC_AUTHORIZATION_ENDPOINT and OIDC_TOKEN_ENDPOINT must be set") + for label, value in ( + ("OIDC_DISCOVERY_URL", discovery_url), + ("OIDC_AUTHORIZATION_ENDPOINT", authorization_endpoint), + ("OIDC_TOKEN_ENDPOINT", token_endpoint), + ): + if value and not _is_allowed_url(value, production=production): + errors.append(f"{label} must be an HTTPS URL in production profiles") + + try: + parse_extra_authorize_params(str(values.get("oidc_extra_authorize_params_json") or "{}")) + except OIDCClientConfigError as exc: + errors.append(str(exc)) + return errors + + +def get_public_oidc_client_config() -> dict[str, Any]: + from backend import config + + if not config.OIDC_ENABLED: + return {"schema_version": SCHEMA_VERSION, "enabled": False} + + values = { + "cors_origins": config.CORS_ORIGINS, + "oidc_issuer": config.OIDC_ISSUER, + "oidc_audience": config.OIDC_AUDIENCE, + "oidc_client_id": config.OIDC_CLIENT_ID, + "oidc_client_redirect_uri": config.OIDC_CLIENT_REDIRECT_URI, + "oidc_client_scopes": config.OIDC_CLIENT_SCOPES, + "oidc_discovery_url": config.OIDC_DISCOVERY_URL, + "oidc_authorization_endpoint": config.OIDC_AUTHORIZATION_ENDPOINT, + "oidc_token_endpoint": config.OIDC_TOKEN_ENDPOINT, + "oidc_extra_authorize_params_json": config.OIDC_EXTRA_AUTHORIZE_PARAMS_JSON, + } + errors = validate_oidc_client_config(values, production=config.PRODUCTION_MODE) + if errors: + raise OIDCClientConfigError("; ".join(errors)) + + response: dict[str, Any] = { + "schema_version": SCHEMA_VERSION, + "enabled": True, + "issuer": config.OIDC_ISSUER, + "client_id": config.OIDC_CLIENT_ID, + "redirect_uri": config.OIDC_CLIENT_REDIRECT_URI, + "scopes": list(config.OIDC_CLIENT_SCOPES), + } + optional = { + "authorization_endpoint": config.OIDC_AUTHORIZATION_ENDPOINT, + "token_endpoint": config.OIDC_TOKEN_ENDPOINT, + "discovery_url": config.OIDC_DISCOVERY_URL, + "jwks_uri": config.OIDC_JWKS_URL, + "audience": config.OIDC_AUDIENCE, + } + response.update({key: value for key, value in optional.items() if value}) + extra = parse_extra_authorize_params(config.OIDC_EXTRA_AUTHORIZE_PARAMS_JSON) + if extra: + response["extra_authorize_params"] = extra + return response diff --git a/backend/api/security/ownership.py b/backend/api/security/ownership.py new file mode 100644 index 0000000..fc11225 --- /dev/null +++ b/backend/api/security/ownership.py @@ -0,0 +1,101 @@ +"""Request-scoped ownership controls for session-bound resources. + +Authenticated non-admin users are isolated by the durable session owner. An +absent actor means authentication is disabled and preserves the local +single-user workflow; administrators intentionally bypass tenant filtering. +""" + +from __future__ import annotations + +from contextlib import contextmanager +from contextvars import ContextVar +from dataclasses import dataclass +from typing import Any, Iterator + + +class OwnershipNotFoundError(LookupError): + """Hide a missing or foreign resource behind the same not-found result.""" + + +@dataclass(frozen=True) +class OwnershipPrincipal: + authenticated: bool + user_id: str = "" + is_admin: bool = False + + +_CURRENT_ACTOR: ContextVar[Any | None] = ContextVar("ownership_actor", default=None) + + +def _actor_value(actor: Any, key: str) -> Any: + if isinstance(actor, dict): + return actor.get(key) + return getattr(actor, key, None) + + +def principal_from_actor(actor: Any | None) -> OwnershipPrincipal: + if actor is None: + return OwnershipPrincipal(authenticated=False) + user_id = str(_actor_value(actor, "user_id") or _actor_value(actor, "email") or "").strip() + role = _actor_value(actor, "role") + role = getattr(role, "value", role) + is_admin = str(role or "").strip().lower() == "admin" + return OwnershipPrincipal(authenticated=True, user_id=user_id, is_admin=is_admin) + + +def current_principal() -> OwnershipPrincipal: + return principal_from_actor(_CURRENT_ACTOR.get()) + + +@contextmanager +def ownership_actor_scope(actor: Any | None) -> Iterator[OwnershipPrincipal]: + token = _CURRENT_ACTOR.set(actor) + try: + yield principal_from_actor(actor) + finally: + _CURRENT_ACTOR.reset(token) + + +def require_session_access( + store: Any, + session_id: str, + *, + actor: Any | None = None, + create: bool = False, +) -> dict | None: + """Require or create a session for an actor without revealing foreign IDs.""" + principal = principal_from_actor(actor) if actor is not None else current_principal() + normalized_session_id = str(session_id or "").strip() + if not normalized_session_id: + raise OwnershipNotFoundError("Session not found") + from backend.services.lifecycle import get_lifecycle_coordinator + + if get_lifecycle_coordinator().reads_revoked(normalized_session_id): + raise OwnershipNotFoundError("Session not found") + + if create: + owner_user_id = principal.user_id if principal.authenticated else None + return store.get_or_create_session(normalized_session_id, owner_user_id=owner_user_id) + + if not principal.authenticated or principal.is_admin: + try: + lifecycle_getter = getattr(store, "get_session_lifecycle_state", None) + lifecycle_state = lifecycle_getter(normalized_session_id) if callable(lifecycle_getter) else None + if lifecycle_state is not None and str(lifecycle_state) != "active": + raise OwnershipNotFoundError("Session not found") + if not store.session_exists(normalized_session_id): + raise OwnershipNotFoundError("Session not found") + except OwnershipNotFoundError: + raise + except Exception as exc: + raise OwnershipNotFoundError("Session not found") from exc + return None + if not principal.user_id: + raise OwnershipNotFoundError("Session not found") + try: + owner_user_id = store.get_session_owner(normalized_session_id) + except Exception as exc: + raise OwnershipNotFoundError("Session not found") from exc + if owner_user_id != principal.user_id: + raise OwnershipNotFoundError("Session not found") + return None diff --git a/backend/api/security/rbac.py b/backend/api/security/rbac.py index e3097ce..92dada5 100644 --- a/backend/api/security/rbac.py +++ b/backend/api/security/rbac.py @@ -13,12 +13,9 @@ from __future__ import annotations -import hashlib -import hmac -import json import os import time -from dataclasses import dataclass, field +from dataclasses import dataclass from enum import Enum from typing import Optional @@ -39,6 +36,7 @@ class Role(str, Enum): @dataclass class User: """An authenticated user.""" + user_id: str email: str name: str @@ -53,19 +51,32 @@ class User: PERMISSIONS = { Role.ADMIN: { - "tasks.execute", "tasks.plan", "tasks.ask", - "documents.upload", "documents.delete", - "templates.deploy", "templates.manage", - "rules.read", "rules.write", "rules.delete", - "sessions.read", "sessions.delete", - "users.manage", "config.manage", + "tasks.execute", + "tasks.plan", + "tasks.ask", + "documents.upload", + "documents.delete", + "templates.deploy", + "templates.manage", + "rules.read", + "rules.write", + "rules.delete", + "sessions.read", + "sessions.delete", + "users.manage", + "config.manage", "compositions.deploy", - "benchmarks.run", "benchmarks.read", - "analytics.read", "teams.manage", - "enterprise.manage", "audit.export", + "benchmarks.run", + "benchmarks.read", + "analytics.read", + "teams.manage", + "enterprise.manage", + "audit.export", }, Role.ANALYST: { - "tasks.execute", "tasks.plan", "tasks.ask", + "tasks.execute", + "tasks.plan", + "tasks.ask", "documents.upload", "templates.deploy", "rules.read", @@ -142,7 +153,15 @@ async def dispatch(self, request: Request, call_next): # Skip RBAC for health, docs, metrics path = request.url.path.rstrip("/") - if path in {"/health", "/docs", "/redoc", "/openapi.json", "/metrics"}: + if path in { + "/health", + "/auth/oidc/config", + "/auth/local/pair", + "/docs", + "/redoc", + "/openapi.json", + "/metrics", + }: return await call_next(request) # Extract user from API key @@ -208,13 +227,15 @@ def __init__(self): # Create default admin user admin_key = os.getenv("ADMIN_API_KEY", "dev-key-change-in-production") - self.add_user(User( - user_id="admin", - email="admin@localhost", - name="Admin", - role=Role.ADMIN, - api_key=admin_key, - )) + self.add_user( + User( + user_id="admin", + email="admin@localhost", + name="Admin", + role=Role.ADMIN, + api_key=admin_key, + ) + ) def add_user(self, user: User): self._users[user.user_id] = user @@ -248,6 +269,7 @@ def delete_user(self, user_id: str) -> bool: # ── SSO / SAML 2.0 Stub ───────────────────────────────────────────────── + class SSOConfig: """SSO configuration for enterprise SAML 2.0 integration.""" @@ -273,6 +295,7 @@ def to_dict(self) -> dict: # ── Zero Data Retention ───────────────────────────────────────────────── + class ZeroDataRetention: """Configurable data retention policies for compliance. @@ -287,29 +310,33 @@ def __init__(self): async def purge_session(self, session_id: str): """Purge all data for a completed session.""" if not self.enabled: - return + return None try: - from backend.storage import get_session_store - store = get_session_store() - store.delete_session(session_id) - logger.info("zdr_session_purged", session_id=session_id) + from backend.services.lifecycle import get_lifecycle_coordinator + + receipt = await get_lifecycle_coordinator().purge_session(session_id) + return receipt.to_dict() except Exception as e: logger.error("zdr_purge_failed", session_id=session_id, error=str(e)) + return None async def purge_expired(self): """Purge all data older than the retention window.""" - if not self.enabled or self.retention_hours <= 0: - return + if not self.enabled: + return 0 try: - from backend.storage import get_session_store - store = get_session_store() - cutoff = time.time() - (self.retention_hours * 3600) - # The session store would need a purge_before method - logger.info("zdr_expired_check", retention_hours=self.retention_hours) + from backend.services.lifecycle import get_lifecycle_coordinator + + cutoff = time.time() - (max(self.retention_hours, 0) * 3600) + receipts = await get_lifecycle_coordinator().purge_before(cutoff) + deleted = sum(1 for receipt in receipts if receipt.database_deleted and not receipt.errors) + logger.info("zdr_expired_purged", retention_hours=self.retention_hours, deleted_sessions=deleted) + return deleted except Exception as e: logger.error("zdr_expired_purge_failed", error=str(e)) + return 0 def to_dict(self) -> dict: return { @@ -320,28 +347,34 @@ def to_dict(self) -> dict: # ── Helpers ────────────────────────────────────────────────────────────── + def _normalize_path(path: str) -> str: - """Normalize a path by removing trailing resource IDs.""" + """Normalize a path by removing unambiguous trailing resource IDs.""" import re + # Remove UUID-like segments - path = re.sub(r'/[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}', '', path) + path = re.sub(r"/[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}", "", path) # Remove numeric IDs - path = re.sub(r'/\d+$', '', path) - # Remove short hash IDs - path = re.sub(r'/[a-f0-9]{6,12}$', '', path) + path = re.sub(r"/\d+$", "", path) return path.rstrip("/") +def _path_matches_prefix(path: str, prefix: str) -> bool: + return path == prefix or path.startswith(f"{prefix}/") + + def _permission_for(method: str, normalized_path: str) -> Optional[str]: direct = ENDPOINT_PERMISSIONS.get((method, normalized_path)) if direct: return direct prefix_rules = [ ("/api/v1/tasks/ask", "tasks.ask"), + ("/api/v1/tasks/plans", "tasks.plan"), ("/api/v1/tasks/plan", "tasks.plan"), ("/api/v1/tasks/apply", "tasks.execute"), ("/api/v1/workbook", "tasks.plan" if method == "GET" else "tasks.execute"), ("/api/v1/workbooks", "tasks.plan" if method == "GET" else "tasks.execute"), + ("/api/v1/spreadsheet", "tasks.plan" if method == "GET" else "tasks.execute"), ("/api/v1/formulas", "tasks.plan"), ("/api/v1/operations", "tasks.execute" if method == "POST" else "tasks.plan"), ("/api/v1/capabilities", "tasks.ask"), @@ -365,7 +398,16 @@ def _permission_for(method: str, normalized_path: str) -> Optional[str]: ("/api/v1/documents", "documents.upload" if method == "POST" else "sessions.read"), ("/api/v1/templates", "templates.manage" if method in {"POST", "PUT", "DELETE"} else "templates.deploy"), ("/api/v1/rules", "rules.write" if method in {"POST", "PUT"} else "rules.delete" if method == "DELETE" else "rules.read"), - ("/api/v1/sessions", "audit.export" if "audit/export" in normalized_path else "sessions.read"), + ( + "/api/v1/sessions", + "audit.export" + if "audit/export" in normalized_path + else "sessions.delete" + if method == "DELETE" + else "tasks.execute" + if method in {"POST", "PUT", "PATCH"} + else "sessions.read", + ), ("/api/v1/benchmarks/run", "benchmarks.run"), ("/api/v1/benchmarks", "benchmarks.read"), ("/api/v1/analytics", "analytics.read"), @@ -375,7 +417,7 @@ def _permission_for(method: str, normalized_path: str) -> Optional[str]: ("/api/v1/security", "enterprise.manage"), ] for prefix, permission in prefix_rules: - if normalized_path.startswith(prefix): + if _path_matches_prefix(normalized_path, prefix): return permission return None @@ -409,12 +451,14 @@ def has_permission(user: Optional[User], permission: str) -> bool: _sso: Optional[SSOConfig] = None _zdr: Optional[ZeroDataRetention] = None + def get_sso_config() -> SSOConfig: global _sso if _sso is None: _sso = SSOConfig() return _sso + def get_zdr() -> ZeroDataRetention: global _zdr if _zdr is None: diff --git a/backend/api/security/soc2.py b/backend/api/security/soc2.py index 59070db..21d7574 100644 --- a/backend/api/security/soc2.py +++ b/backend/api/security/soc2.py @@ -12,7 +12,6 @@ from __future__ import annotations -import os import time from dataclasses import dataclass, field from pathlib import Path @@ -106,10 +105,11 @@ class SOC2Control: id="CC5.1", criteria="Security", category="Control Activities", title="Logical Access Controls", description="Access to systems and data is restricted to authorized users.", - implementation="API key authentication with HMAC verification. RBAC with 3 roles. Rate limiting.", + implementation="API key authentication with HMAC verification, optional required-MFA policy, RBAC with 3 roles, and rate limiting.", status="implemented", evidence=[ "backend/api/middleware/auth.py — API key auth with constant-time comparison", + "backend/session_store.py — Active-user and required-MFA enterprise API-key gates", "backend/api/security/rbac.py — 3-role RBAC with 16 permissions", "backend/api/middleware/rate_limiter.py — Request rate limiting", ], @@ -128,9 +128,16 @@ class SOC2Control: id="CC5.3", criteria="Security", category="Control Activities", title="Encryption — At Rest", description="Data is encrypted when stored.", - implementation="SQLite encryption via sqlcipher (AES-256-CBC). PBKDF2 key derivation.", - status="implemented", - evidence=["backend/storage/encrypted_store.py — AES-256 at rest"], + implementation=( + "The primary SQLite connection path can require SQLCipher with an exact " + "256-bit raw key; PostgreSQL encryption requires independent platform evidence." + ), + status="partial", + evidence=[ + "backend/storage/encrypted_store.py — fail-closed primary SQLite connection provider", + "backend/tests/test_encrypted_store_integration.py — plaintext-fallback rejection", + ], + gaps=["Signed PostgreSQL/platform encryption-at-rest attestation for the release environment"], automated_check="check_encryption", ), SOC2Control( @@ -243,6 +250,7 @@ def run_audit(self) -> dict: "category": control.category, "title": control.title, "status": control.status, + "declared_status": control.status, "evidence_count": len(control.evidence), "gap_count": len(control.gaps), "gaps": control.gaps, @@ -255,8 +263,14 @@ def run_audit(self) -> dict: try: auto_result = method() check_result["automated_check"] = auto_result - except Exception as e: - check_result["automated_check"] = {"passed": False, "error": str(e)} + if auto_result.get("passed") is not True: + check_result["status"] = "blocked" + except Exception as exc: + check_result["status"] = "blocked" + check_result["automated_check"] = { + "passed": False, + "error_type": type(exc).__name__, + } results.append(check_result) @@ -265,6 +279,7 @@ def run_audit(self) -> dict: implemented = sum(1 for r in results if r["status"] == "implemented") partial = sum(1 for r in results if r["status"] == "partial") planned = sum(1 for r in results if r["status"] == "planned") + blocked = sum(1 for r in results if r["status"] == "blocked") total_gaps = sum(r["gap_count"] for r in results) return { @@ -274,6 +289,7 @@ def run_audit(self) -> dict: "implemented": implemented, "partial": partial, "planned": planned, + "blocked": blocked, "readiness_pct": round((implemented + partial * 0.5) / total * 100, 1), "total_gaps": total_gaps, }, @@ -282,13 +298,18 @@ def run_audit(self) -> dict: } def _group_by_criteria(self, results: list[dict]) -> dict: - groups = {} + groups: dict[str, dict[str, int]] = {} for r in results: criteria = r["criteria"] - groups.setdefault(criteria, {"total": 0, "implemented": 0, "gaps": 0}) + groups.setdefault( + criteria, + {"total": 0, "implemented": 0, "blocked": 0, "gaps": 0}, + ) groups[criteria]["total"] += 1 if r["status"] == "implemented": groups[criteria]["implemented"] += 1 + if r["status"] == "blocked": + groups[criteria]["blocked"] += 1 groups[criteria]["gaps"] += r["gap_count"] return groups @@ -304,12 +325,42 @@ def check_logging(self) -> dict: def check_auth(self) -> dict: """Verify authentication middleware exists.""" - auth_path = Path(__file__).parent.parent / "api" / "middleware" / "auth.py" - rbac_path = Path(__file__).parent.parent / "api" / "security" / "rbac.py" + api_dir = Path(__file__).parents[1] + backend_dir = Path(__file__).parents[2] + auth_path = api_dir / "middleware" / "auth.py" + rbac_path = api_dir / "security" / "rbac.py" + session_store_path = backend_dir / "session_store.py" + enterprise_path = api_dir / "routes" / "enterprise.py" + storage_text = session_store_path.read_text() if session_store_path.exists() else "" + enterprise_text = enterprise_path.read_text() if enterprise_path.exists() else "" + has_required_mfa_gate = "require_mfa" in storage_text and "_enterprise_user_allows_api_key" in storage_text + required_admin_audit_actions = { + "upsert_user", + "create_api_key", + "revoke_api_key", + "create_team", + "add_team_member", + "set_sso_config", + "set_provider_policy", + "set_retention_policy", + "purge_expired_sessions", + } + audited_actions = { + action: f'action="{action}"' in enterprise_text + for action in required_admin_audit_actions + } + has_admin_audit_gate = ( + "record_enterprise_admin_audit" in enterprise_text + and "_audit_enterprise_admin" in enterprise_text + and all(audited_actions.values()) + ) return { - "passed": auth_path.exists() and rbac_path.exists(), + "passed": auth_path.exists() and rbac_path.exists() and has_required_mfa_gate and has_admin_audit_gate, "auth_middleware": auth_path.exists(), "rbac_module": rbac_path.exists(), + "required_mfa_policy_gate": has_required_mfa_gate, + "enterprise_admin_audit_gate": has_admin_audit_gate, + "enterprise_admin_audit_actions": audited_actions, } def check_tls(self) -> dict: @@ -322,12 +373,21 @@ def check_tls(self) -> dict: } def check_encryption(self) -> dict: - """Verify encryption at rest module exists.""" - enc_path = Path(__file__).parent.parent / "storage" / "encrypted_store.py" + """Require runtime proof that the primary store is encrypted.""" + from backend.storage.encrypted_store import get_primary_storage_encryption_status + + posture = get_primary_storage_encryption_status() + protected = posture.get("primaryStoreProtected") is True return { - "passed": enc_path.exists(), - "encryption_module": enc_path.exists(), - "detail": "AES-256 at-rest encryption available" if enc_path.exists() else "No encryption", + "passed": protected, + "primary_store_protected": protected, + "status": posture.get("status"), + "reason_code": posture.get("reasonCode"), + "detail": ( + "Primary data store encryption is active" + if protected + else "Primary data store encryption is not runtime-verified" + ), } def check_monitoring(self) -> dict: @@ -350,7 +410,8 @@ def check_health(self) -> dict: def _check_prometheus(self) -> bool: try: - import prometheus_client + import prometheus_client # noqa: F401 - availability probe + return True except ImportError: return False diff --git a/backend/api/teams.py b/backend/api/teams.py index 35e6618..88923b2 100644 --- a/backend/api/teams.py +++ b/backend/api/teams.py @@ -249,6 +249,25 @@ def list_team_sessions(self, team_id: str) -> list[dict]: if s.team_id == team_id ] + def purge_session(self, session_id: str) -> dict[str, int]: + """Remove one shared-session record from memory and managed storage.""" + normalized_session_id = str(session_id or "").strip() + if not normalized_session_id: + raise ValueError("session_id is required") + managed_root = TEAMS_DIR.resolve() + path = (TEAMS_DIR / f"session_{normalized_session_id}.json").resolve(strict=False) + if not path.is_relative_to(managed_root): + raise ValueError("Team session artifact is outside the managed store") + with self._lock: + removed = self._sessions.pop(normalized_session_id, None) + file_existed = path.is_file() + if file_existed: + path.unlink() + return { + "memory_sessions_evicted": int(removed is not None), + "files_deleted": int(file_existed), + } + def record_usage(self, team_id: str, user_id: str, credits: int = 1): team = self._teams.get(team_id) if not team: diff --git a/backend/api/validators.py b/backend/api/validators.py index 73e0f05..3987263 100644 --- a/backend/api/validators.py +++ b/backend/api/validators.py @@ -37,7 +37,10 @@ def validate_cell_reference(ref: str) -> bool: return False # Validate column (max XFD = column 16384) - col_str = re.match(r"[A-Z]+", ref).group() + col_match = re.match(r"[A-Z]+", ref) + if col_match is None: + return False + col_str = col_match.group() col_num = 0 for ch in col_str: col_num = col_num * 26 + (ord(ch) - ord("A") + 1) @@ -45,7 +48,10 @@ def validate_cell_reference(ref: str) -> bool: return False # Validate row (max 1048576) - row_str = re.search(r"[0-9]+", ref).group() + row_match = re.search(r"[0-9]+", ref) + if row_match is None: + return False + row_str = row_match.group() row_num = int(row_str) if row_num < 1 or row_num > 1048576: return False diff --git a/backend/benchmarks/fixtures/private-collaboration/README.md b/backend/benchmarks/fixtures/private-collaboration/README.md new file mode 100644 index 0000000..42d824f --- /dev/null +++ b/backend/benchmarks/fixtures/private-collaboration/README.md @@ -0,0 +1,51 @@ +# Private collaboration schedule corpus + +This gitignored directory holds the randomized collaboration schedules required +for vNext convergence, fencing, crash, and replay certification. Never commit +private workbook values, formulas, update bytes, actor identities, or source +metadata. Only this README and the synthetic `manifest.example.json` are +tracked. + +`manifest.schema.json` and `schedule.schema.json` document the portable JSON +contract. The Python validator remains authoritative for canonical ordering, +digest verification, containment, symlink rejection, resource budgets, and +cross-file uniqueness constraints that JSON Schema cannot express. + +The release-scale contract requires at least 100 schedules and coverage for +concurrent edits, crash/restart, network partitions, lease expiry, dual-writer +fencing, actor-scoped undo, and compaction. Each schedule stores canonical +base64 update/operation bytes only in the gitignored schedule file; the tracked +manifest and validator output expose hashed IDs, digests, counts, scenarios, +and expected convergence only. + +Validate locally: + +```bash +python -m backend.benchmarks.private_collaboration_schedules \ + --require-release-scale --json +``` + +Inventory validation is not execution evidence. The collaboration service must +replay every schedule with crash injection and sign an exact-SHA result proving +the expected final state vector, conflict journal, authority mode, and zero lost +acknowledged updates. + +After the service runner has executed the complete corpus at least twice, +validate its detached Ed25519 receipt with an out-of-repository public key: + +```bash +python scripts/validate_private_collaboration_execution_receipt.py \ + --receipt /secure/evidence/private-collaboration-execution.json \ + --signature /secure/evidence/private-collaboration-execution.ed25519 \ + --public-key /secure/keys/collaboration-runner-public.pem \ + --key-id collaboration-runner-2026-08 \ + --release-sha "$(git rev-parse HEAD)" \ + --private-root backend/benchmarks/fixtures/private-collaboration +``` + +The validator binds every schedule ID and digest to the manifest, compares the +observed state vector, conflicts, acknowledged updates, authority mode, and +bridge fence to the schedule's expected outcome, rejects lost acknowledgements, +and requires deterministic replay. Passing this validator is a vNext +collaboration promotion gate; it is not evidence that the current `0.5.x` +private-beta client exposes collaboration. diff --git a/backend/benchmarks/fixtures/private-collaboration/manifest.example.json b/backend/benchmarks/fixtures/private-collaboration/manifest.example.json new file mode 100644 index 0000000..a146a95 --- /dev/null +++ b/backend/benchmarks/fixtures/private-collaboration/manifest.example.json @@ -0,0 +1,15 @@ +{ + "schemaVersion": "2026-08-private-collaboration-schedules-v1", + "schedules": [ + { + "scheduleIdHash": "schedule_00000001", + "schedulePath": "schedules/schedule_00000001.json", + "scheduleSha256": "sha256:0000000000000000000000000000000000000000000000000000000000000000", + "provenance": { + "kind": "randomized", + "generatorHash": "sha256:1111111111111111111111111111111111111111111111111111111111111111", + "sourceReceiptHash": "sha256:2222222222222222222222222222222222222222222222222222222222222222" + } + } + ] +} diff --git a/backend/benchmarks/fixtures/private-collaboration/manifest.schema.json b/backend/benchmarks/fixtures/private-collaboration/manifest.schema.json new file mode 100644 index 0000000..5c98594 --- /dev/null +++ b/backend/benchmarks/fixtures/private-collaboration/manifest.schema.json @@ -0,0 +1,36 @@ +{ + "$schema": "https://json-schema.org/draft/2020-12/schema", + "$id": "https://excel-agent.invalid/schemas/private-collaboration-manifest-v1.json", + "title": "Private collaboration schedule manifest", + "type": "object", + "additionalProperties": false, + "required": ["schemaVersion", "schedules"], + "properties": { + "schemaVersion": { "const": "2026-08-private-collaboration-schedules-v1" }, + "schedules": { + "type": "array", + "minItems": 1, + "maxItems": 10000, + "items": { + "type": "object", + "additionalProperties": false, + "required": ["scheduleIdHash", "schedulePath", "scheduleSha256", "provenance"], + "properties": { + "scheduleIdHash": { "type": "string", "pattern": "^schedule_[0-9a-f]{8,64}$" }, + "schedulePath": { "type": "string", "pattern": "^schedules/[A-Za-z0-9._-]+\\.json$" }, + "scheduleSha256": { "type": "string", "pattern": "^sha256:[0-9a-f]{64}$" }, + "provenance": { + "type": "object", + "additionalProperties": false, + "required": ["kind", "generatorHash", "sourceReceiptHash"], + "properties": { + "kind": { "const": "randomized" }, + "generatorHash": { "type": "string", "pattern": "^sha256:[0-9a-f]{64}$" }, + "sourceReceiptHash": { "type": "string", "pattern": "^sha256:[0-9a-f]{64}$" } + } + } + } + } + } + } +} diff --git a/backend/benchmarks/fixtures/private-collaboration/schedule.schema.json b/backend/benchmarks/fixtures/private-collaboration/schedule.schema.json new file mode 100644 index 0000000..e5fd772 --- /dev/null +++ b/backend/benchmarks/fixtures/private-collaboration/schedule.schema.json @@ -0,0 +1,59 @@ +{ + "$schema": "https://json-schema.org/draft/2020-12/schema", + "$id": "https://excel-agent.invalid/schemas/private-collaboration-schedule-v1.json", + "title": "Private deterministic collaboration schedule", + "type": "object", + "additionalProperties": false, + "required": ["schemaVersion", "scheduleIdHash", "seedHash", "actors", "scenarios", "events", "expected"], + "properties": { + "schemaVersion": { "const": "2026-08-private-collaboration-schedule-v1" }, + "scheduleIdHash": { "type": "string", "pattern": "^schedule_[0-9a-f]{8,64}$" }, + "seedHash": { "type": "string", "pattern": "^sha256:[0-9a-f]{64}$" }, + "actors": { + "type": "array", + "minItems": 2, + "maxItems": 20, + "uniqueItems": true, + "items": { "type": "string", "pattern": "^actor_[0-9a-f]{8,64}$" } + }, + "scenarios": { + "type": "array", + "minItems": 1, + "uniqueItems": true, + "items": { "type": "string", "pattern": "^[a-z][a-z0-9_]{0,63}$" } + }, + "events": { + "type": "array", + "minItems": 1, + "maxItems": 100000, + "items": { + "type": "object", + "additionalProperties": false, + "required": ["sequence", "actorIdHash", "logicalTime", "kind", "payloadBase64", "payloadSha256", "expectedDisposition"], + "properties": { + "sequence": { "type": "integer", "minimum": 1 }, + "actorIdHash": { "type": "string", "pattern": "^actor_[0-9a-f]{8,64}$" }, + "logicalTime": { "type": "integer", "minimum": 0 }, + "kind": { + "enum": ["bridge_acquire", "bridge_release", "bridge_renew", "compact", "crash", "crdt_update", "mode_switch", "partition_end", "partition_start", "restart", "undo"] + }, + "payloadBase64": { "type": "string", "minLength": 4, "contentEncoding": "base64" }, + "payloadSha256": { "type": "string", "pattern": "^sha256:[0-9a-f]{64}$" }, + "expectedDisposition": { "enum": ["accepted", "conflict", "deferred", "rejected"] } + } + } + }, + "expected": { + "type": "object", + "additionalProperties": false, + "required": ["finalStateVectorHash", "conflictCount", "acknowledgedUpdateCount", "lostAcknowledgedUpdateCount", "authorityMode"], + "properties": { + "finalStateVectorHash": { "type": "string", "pattern": "^sha256:[0-9a-f]{64}$" }, + "conflictCount": { "type": "integer", "minimum": 0 }, + "acknowledgedUpdateCount": { "type": "integer", "minimum": 0 }, + "lostAcknowledgedUpdateCount": { "const": 0 }, + "authorityMode": { "enum": ["app", "office"] } + } + } + } +} diff --git a/backend/benchmarks/fixtures/private/README.md b/backend/benchmarks/fixtures/private/README.md new file mode 100644 index 0000000..11e2d81 --- /dev/null +++ b/backend/benchmarks/fixtures/private/README.md @@ -0,0 +1,75 @@ +# Private Golden Workbook Fixtures + +This folder is for local-only, private golden workbook fixtures. Keep real +customer, banker, and workbook data out of git. + +Tracked files here are limited to this README and `manifest.example.json`. +The real `manifest.json`, workbook files, extracts, exports, and audit evidence +remain ignored by `.gitignore`. + +## Local Setup + +1. Copy `manifest.example.json` to `manifest.json`. +2. Store workbook files under this folder, preferably under a local `workbooks/` + subfolder. +3. Use hashed fixture IDs such as `fx_` or `sha256:<64 hex chars>`. +4. Use workbook paths relative to this private folder. +5. Record the exact `workbookSha256` and hashed Excel-build/source-receipt + provenance. Do not put a customer name, workbook name, Excel build string, + operator identity, or source URL in the manifest. +6. Run the validator locally: + +```powershell +python -m backend.benchmarks.private_golden_fixtures +``` + +The validator is intentionally fail-closed: + +- `missing`: no private `manifest.json` exists. +- `invalid`: manifest parsing or metadata validation failed; issue messages do + not echo private workbook names, formulas, prompts, or source values. +- `ready`: all referenced workbook paths exist under this folder, are non-empty + bounded OOXML workbook packages, their bytes match the manifest digest, + their provenance is declared as Excel-authored with hashed build/source + receipts, and required expectation metadata is present. + +Release readiness is stricter than manifest validity: the private corpus must +contain at least 250 Excel-authored workbooks and cover financial models, +protected workbooks, messy formulas/data, and charts/tables/pivots/macros. A +smaller manifest remains useful for local development but reports +`insufficient` and cannot satisfy release evidence. + +This harness validates private fixture readiness only. It does not execute +private workbook benchmarks or print private workbook paths. Run the bounded, +privacy-safe worker preflight separately: + +```powershell +python scripts/run_private_corpus_preflight.py ` + --private-root backend/benchmarks/fixtures/private ` + --require-release-scale --json +``` + +The report contains hashed fixture IDs and aggregate counts only. It explicitly +sets `nativeExcelExecuted` and `releaseEvidenceEligible` to `false`: passing this +preflight cannot substitute for the signed Windows/Mac/Web Excel execution +evidence required by the release checklist. + +Native host automation must emit a detached Ed25519-signed execution receipt +covering every manifest fixture, then validate it with an out-of-repository +public key before indexing it as release evidence: + +```powershell +python scripts/validate_private_corpus_execution_receipt.py ` + --receipt C:\evidence\private-corpus-execution.json ` + --signature C:\evidence\private-corpus-execution.ed25519 ` + --public-key D:\release-keys\excel-corpus-public.pem ` + --key-id excel-corpus-windows-2026-08 ` + --release-sha ` + --private-root backend/benchmarks/fixtures/private +``` + +The signed schema binds the exact release, corpus manifest, 250 workbook +digests, Excel build, runner image, executor, per-workbook host checks, and +unique artifact digests. Any omitted workbook, repair dialog, failed +open/recalculate/save check, duplicate payload, signature mismatch, or corpus +change fails closed. diff --git a/backend/benchmarks/fixtures/private/manifest.example.json b/backend/benchmarks/fixtures/private/manifest.example.json new file mode 100644 index 0000000..408a063 --- /dev/null +++ b/backend/benchmarks/fixtures/private/manifest.example.json @@ -0,0 +1,55 @@ +{ + "schemaVersion": "2026-08-private-golden-fixtures-v2", + "fixtures": [ + { + "fixtureIdHash": "fx_3f4a8c9b1d2e5f60", + "category": "customer_like_model", + "workbookPath": "workbooks/fx_3f4a8c9b1d2e5f60.xlsx", + "workbookSha256": "sha256:0000000000000000000000000000000000000000000000000000000000000000", + "provenance": { + "kind": "excel_authored", + "excelBuildHash": "sha256:1111111111111111111111111111111111111111111111111111111111111111", + "sourceReceiptHash": "sha256:2222222222222222222222222222222222222222222222222222222222222222" + }, + "expectedTruthTargets": [ + "primary_driver_identified", + "output_metric_identified" + ], + "expectedFormulaFacts": [ + { + "sheet": "Model", + "address": "B20", + "formulaHash": "sha256:0000000000000000000000000000000000000000000000000000000000000000", + "dependencyHashes": [ + "sha256:1111111111111111111111111111111111111111111111111111111111111111" + ] + } + ], + "expectedSimulationGates": [ + "checkpoint_required" + ], + "rollbackExpectations": { + "required": true, + "dirtyRanges": [ + "Model!B20" + ], + "snapshotRequired": true + }, + "repairOutcomes": [ + { + "kind": "formula_repair", + "expectedStatus": "repair_plan_expected" + } + ], + "auditExpectations": { + "requiredEvidence": [ + "truth_model", + "formula_graph", + "simulation_gate", + "rollback_snapshot" + ], + "exportRequired": true + } + } + ] +} diff --git a/backend/benchmarks/fixtures/replays/synthetic_aggregate_replay_corpus.json b/backend/benchmarks/fixtures/replays/synthetic_aggregate_replay_corpus.json new file mode 100644 index 0000000..26ada83 --- /dev/null +++ b/backend/benchmarks/fixtures/replays/synthetic_aggregate_replay_corpus.json @@ -0,0 +1,123 @@ +{ + "schemaVersion": "2026-06-synthetic-replay-corpus-fixture-v1", + "purpose": "non_production_import_contract", + "artifacts": [ + { + "artifactType": "v7_plan", + "payload": { + "schemaVersion": "2026-05-local-replay-v1", + "artifactKind": "synthetic_contract_passed", + "operationTypes": ["write_cell"], + "operationCount": 1, + "accuracyScore": 0.99, + "confidenceGapCount": 0, + "checkpointReasonCount": 0, + "formulaFailureCount": 0, + "truthSchema": "2026-05-workbook-truth-model-v5", + "formulaSchema": "2026-05-formula-graph-artifact-v5", + "simulationSchema": "2026-05-operation-simulation-v6", + "v7GateStatus": "passed", + "dirtyRangeCount": 1, + "rollbackAvailable": true, + "fixtureIdHash": "fx_synthetic_passed", + "category": "writeback", + "sourceProject": "synthetic-replay-contract", + "sheetCount": 4, + "formulaCount": 128, + "objectCounts": { "tables": 2, "charts": 1, "pivots": 0 }, + "warningCategoryCounts": {}, + "semanticMatchCounts": { "matched": 12, "missing": 0 }, + "replaySignature": { + "schemaVersion": "2026-05-local-replay-v1", + "operationTypes": ["write_cell"], + "operationCount": 1, + "truthSchema": "2026-05-workbook-truth-model-v5", + "formulaSchema": "2026-05-formula-graph-artifact-v5", + "simulationSchema": "2026-05-operation-simulation-v6", + "v7GateStatus": "passed", + "formulaFailureCount": 0, + "confidenceGapCount": 0, + "checkpointReasonCount": 0 + } + } + }, + { + "artifactType": "v7_apply_verification", + "payload": { + "schemaVersion": "2026-05-local-replay-v1", + "artifactKind": "synthetic_contract_checkpoint", + "operationTypes": ["create_chart"], + "operationCount": 1, + "accuracyScore": 0.94, + "confidenceGapCount": 0, + "checkpointReasonCount": 1, + "formulaFailureCount": 0, + "truthSchema": "2026-05-workbook-truth-model-v5", + "formulaSchema": "2026-05-formula-graph-artifact-v5", + "simulationSchema": "2026-05-operation-simulation-v6", + "v7GateStatus": "checkpoint_required", + "dirtyRangeCount": 0, + "rollbackAvailable": false, + "fixtureIdHash": "fx_synthetic_checkpoint", + "category": "object_mutation", + "sourceProject": "synthetic-replay-contract", + "sheetCount": 2, + "formulaCount": 33, + "objectCounts": { "tables": 1, "charts": 2, "pivots": 0 }, + "warningCategoryCounts": { "checkpoint_required": 1 }, + "semanticMatchCounts": { "matched": 8, "missing": 1 }, + "replaySignature": { + "schemaVersion": "2026-05-local-replay-v1", + "operationTypes": ["create_chart"], + "operationCount": 1, + "truthSchema": "2026-05-workbook-truth-model-v5", + "formulaSchema": "2026-05-formula-graph-artifact-v5", + "simulationSchema": "2026-05-operation-simulation-v6", + "v7GateStatus": "checkpoint_required", + "formulaFailureCount": 0, + "confidenceGapCount": 0, + "checkpointReasonCount": 1 + } + } + }, + { + "artifactType": "v7_simulation", + "payload": { + "schemaVersion": "2026-05-local-replay-v1", + "artifactKind": "synthetic_contract_blocked", + "operationTypes": ["delete_sheet"], + "operationCount": 1, + "accuracyScore": 0.91, + "confidenceGapCount": 1, + "checkpointReasonCount": 1, + "formulaFailureCount": 0, + "truthSchema": "2026-05-workbook-truth-model-v5", + "formulaSchema": "2026-05-formula-graph-artifact-v5", + "simulationSchema": "2026-05-operation-simulation-v6", + "v7GateStatus": "blocked", + "dirtyRangeCount": 0, + "rollbackAvailable": false, + "fixtureIdHash": "fx_synthetic_blocked", + "category": "destructive_operation", + "sourceProject": "synthetic-replay-contract", + "sheetCount": 6, + "formulaCount": 420, + "objectCounts": { "tables": 4, "charts": 3, "pivots": 1 }, + "warningCategoryCounts": { "rollback_required": 1, "host_confirmation": 1 }, + "semanticMatchCounts": { "matched": 15, "missing": 2 }, + "replaySignature": { + "schemaVersion": "2026-05-local-replay-v1", + "operationTypes": ["delete_sheet"], + "operationCount": 1, + "truthSchema": "2026-05-workbook-truth-model-v5", + "formulaSchema": "2026-05-formula-graph-artifact-v5", + "simulationSchema": "2026-05-operation-simulation-v6", + "v7GateStatus": "blocked", + "formulaFailureCount": 0, + "confidenceGapCount": 1, + "checkpointReasonCount": 1 + } + } + } + ] +} diff --git a/backend/benchmarks/fixtures/workbooks/audit_export_flow.xlsx b/backend/benchmarks/fixtures/workbooks/audit_export_flow.xlsx index 9c1ab95..273c027 100644 Binary files a/backend/benchmarks/fixtures/workbooks/audit_export_flow.xlsx and b/backend/benchmarks/fixtures/workbooks/audit_export_flow.xlsx differ diff --git a/backend/benchmarks/fixtures/workbooks/broken_ref_formula.xlsx b/backend/benchmarks/fixtures/workbooks/broken_ref_formula.xlsx index bab7bd2..bfed626 100644 Binary files a/backend/benchmarks/fixtures/workbooks/broken_ref_formula.xlsx and b/backend/benchmarks/fixtures/workbooks/broken_ref_formula.xlsx differ diff --git a/backend/benchmarks/fixtures/workbooks/chart_source_update.xlsx b/backend/benchmarks/fixtures/workbooks/chart_source_update.xlsx index 00e0120..64064e9 100644 Binary files a/backend/benchmarks/fixtures/workbooks/chart_source_update.xlsx and b/backend/benchmarks/fixtures/workbooks/chart_source_update.xlsx differ diff --git a/backend/benchmarks/fixtures/workbooks/comps_median_multiple.xlsx b/backend/benchmarks/fixtures/workbooks/comps_median_multiple.xlsx index 818393e..756bf3f 100644 Binary files a/backend/benchmarks/fixtures/workbooks/comps_median_multiple.xlsx and b/backend/benchmarks/fixtures/workbooks/comps_median_multiple.xlsx differ diff --git a/backend/benchmarks/fixtures/workbooks/copied_formula_pattern.xlsx b/backend/benchmarks/fixtures/workbooks/copied_formula_pattern.xlsx index 7c0e455..d372c98 100644 Binary files a/backend/benchmarks/fixtures/workbooks/copied_formula_pattern.xlsx and b/backend/benchmarks/fixtures/workbooks/copied_formula_pattern.xlsx differ diff --git a/backend/benchmarks/fixtures/workbooks/dcf_terminal_value_chain.xlsx b/backend/benchmarks/fixtures/workbooks/dcf_terminal_value_chain.xlsx index 711cb43..0ecff7b 100644 Binary files a/backend/benchmarks/fixtures/workbooks/dcf_terminal_value_chain.xlsx and b/backend/benchmarks/fixtures/workbooks/dcf_terminal_value_chain.xlsx differ diff --git a/backend/benchmarks/fixtures/workbooks/debt_schedule.xlsx b/backend/benchmarks/fixtures/workbooks/debt_schedule.xlsx index 884202a..a83ab44 100644 Binary files a/backend/benchmarks/fixtures/workbooks/debt_schedule.xlsx and b/backend/benchmarks/fixtures/workbooks/debt_schedule.xlsx differ diff --git a/backend/benchmarks/fixtures/workbooks/dynamic_array_spill.xlsx b/backend/benchmarks/fixtures/workbooks/dynamic_array_spill.xlsx index 68885d3..befb97d 100644 Binary files a/backend/benchmarks/fixtures/workbooks/dynamic_array_spill.xlsx and b/backend/benchmarks/fixtures/workbooks/dynamic_array_spill.xlsx differ diff --git a/backend/benchmarks/fixtures/workbooks/external_links.xlsx b/backend/benchmarks/fixtures/workbooks/external_links.xlsx index a2da970..cf78ada 100644 Binary files a/backend/benchmarks/fixtures/workbooks/external_links.xlsx and b/backend/benchmarks/fixtures/workbooks/external_links.xlsx differ diff --git a/backend/benchmarks/fixtures/workbooks/formula_heavy_model.xlsx b/backend/benchmarks/fixtures/workbooks/formula_heavy_model.xlsx index 6b414ef..a85ce14 100644 Binary files a/backend/benchmarks/fixtures/workbooks/formula_heavy_model.xlsx and b/backend/benchmarks/fixtures/workbooks/formula_heavy_model.xlsx differ diff --git a/backend/benchmarks/fixtures/workbooks/hidden_protected_workbook.xlsx b/backend/benchmarks/fixtures/workbooks/hidden_protected_workbook.xlsx index 3ee37e9..e6c4ee4 100644 Binary files a/backend/benchmarks/fixtures/workbooks/hidden_protected_workbook.xlsx and b/backend/benchmarks/fixtures/workbooks/hidden_protected_workbook.xlsx differ diff --git a/backend/benchmarks/fixtures/workbooks/large_workbook.xlsx b/backend/benchmarks/fixtures/workbooks/large_workbook.xlsx index cc21f9e..fd01465 100644 Binary files a/backend/benchmarks/fixtures/workbooks/large_workbook.xlsx and b/backend/benchmarks/fixtures/workbooks/large_workbook.xlsx differ diff --git a/backend/benchmarks/fixtures/workbooks/lbo_cash_sweep.xlsx b/backend/benchmarks/fixtures/workbooks/lbo_cash_sweep.xlsx index e86f1e6..91b1239 100644 Binary files a/backend/benchmarks/fixtures/workbooks/lbo_cash_sweep.xlsx and b/backend/benchmarks/fixtures/workbooks/lbo_cash_sweep.xlsx differ diff --git a/backend/benchmarks/fixtures/workbooks/lbo_debt_schedule.xlsx b/backend/benchmarks/fixtures/workbooks/lbo_debt_schedule.xlsx index 17da1c2..3f226c8 100644 Binary files a/backend/benchmarks/fixtures/workbooks/lbo_debt_schedule.xlsx and b/backend/benchmarks/fixtures/workbooks/lbo_debt_schedule.xlsx differ diff --git a/backend/benchmarks/fixtures/workbooks/macro_enabled_workbook.xlsm b/backend/benchmarks/fixtures/workbooks/macro_enabled_workbook.xlsm index bcf287d..2da72f5 100644 Binary files a/backend/benchmarks/fixtures/workbooks/macro_enabled_workbook.xlsm and b/backend/benchmarks/fixtures/workbooks/macro_enabled_workbook.xlsm differ diff --git a/backend/benchmarks/fixtures/workbooks/macro_source_checkpoint.xlsm b/backend/benchmarks/fixtures/workbooks/macro_source_checkpoint.xlsm index 07b295f..e1c5e33 100644 Binary files a/backend/benchmarks/fixtures/workbooks/macro_source_checkpoint.xlsm and b/backend/benchmarks/fixtures/workbooks/macro_source_checkpoint.xlsm differ diff --git a/backend/benchmarks/fixtures/workbooks/manifest.json b/backend/benchmarks/fixtures/workbooks/manifest.json index 1e11aa4..ee7e3d7 100644 --- a/backend/benchmarks/fixtures/workbooks/manifest.json +++ b/backend/benchmarks/fixtures/workbooks/manifest.json @@ -2,6 +2,42 @@ "schemaVersion": "2026-05-workbook-golden-fixtures-v1", "generatedAt": "2026-05-01T00:00:00.000Z", "sourceProject": "ExcelJS", + "fixtureProvenance": { + "kind": "generated", + "source": "workbook-worker fixture-generate", + "authoringTool": "ExcelJS", + "excelAuthored": false, + "privateExternal": false, + "notes": "Public workbook fixtures are generated OOXML packages. Real Excel-authored/private fixture bytes belong under backend/benchmarks/fixtures/private and remain gitignored." + }, + "fixtureInventoryPolicy": { + "requiredFeatureEvidence": [ + "charts", + "tables", + "pivots", + "comments", + "macros", + "validations", + "conditionalFormats", + "protections", + "freezePanes", + "definedNames", + "drawings" + ], + "displayNameClaimsRequirePackageEvidence": true, + "privateFixtureDirectory": "backend/benchmarks/fixtures/private" + }, + "fixtureRealismPolicy": { + "scope": "public_generated_local", + "externalPrivateEvidenceRequired": false, + "hardenedExpectations": [ + "multi_operation_plans", + "multi_cell_range_mutations", + "exact_dirty_ranges", + "explicit_simulation_gates" + ], + "notes": "Public fixtures intentionally use generated OOXML but assert workbook-like multi-step mutation behavior without relying on private customer workbook bytes." + }, "fixtures": [ { "fixtureId": "messy_dcf_wacc", @@ -147,6 +183,12 @@ "activeSheet": "Model", "selectedRange": "Model!B3", "operations": [ + { + "type": "write_cell", + "sheet": "Model", + "range": "B2", + "value": 120 + }, { "type": "write_cell", "sheet": "Model", @@ -158,9 +200,27 @@ "Enterprise Value" ], "expectedDirtyRanges": [ - "B3" + "Model!B2", + "Model!B3" + ], + "expectedSimulationGates": [ + "checkpoint_required" ], "expectRollback": true, + "simulationAssertions": { + "operationCount": 2, + "exactDirtyRanges": true, + "rollbackRequired": true + }, + "realismMetadata": { + "evidenceClass": "generated_local", + "scenario": "multi-cell formula driver update", + "workbookLikeSignals": [ + "formula_chain", + "dependent_outputs", + "multi_operation_plan" + ] + }, "expectedObjects": { "charts": [], "tables": [], @@ -171,7 +231,7 @@ { "fixtureId": "macro_enabled_workbook", "filename": "macro_enabled_workbook.xlsm", - "name": "Macro Enabled Workbook", + "name": "XLSM Container Workbook", "category": "macro_enabled", "activeSheet": "Model", "selectedRange": "Model!B2", @@ -229,7 +289,7 @@ { "fixtureId": "sensitivity_table", "filename": "sensitivity_table.xlsx", - "name": "Sensitivity Table", + "name": "Sensitivity Grid", "category": "sensitivity", "activeSheet": "Model", "selectedRange": "Model!B1", @@ -664,7 +724,7 @@ { "fixtureId": "structured_reference_table", "filename": "structured_reference_table.xlsx", - "name": "Structured Reference Table", + "name": "Bracketed Reference Formula", "category": "structured_refs", "activeSheet": "Model", "selectedRange": "Model!B4", @@ -738,9 +798,27 @@ "Protected Formula" ], "expectedDirtyRanges": [ - "B3" + "Model!B3" + ], + "expectedSimulationGates": [ + "blocked" ], "expectRollback": true, + "simulationAssertions": { + "operationCount": 1, + "exactDirtyRanges": true, + "rollbackRequired": true, + "blockedBy": "formula_overwrite" + }, + "realismMetadata": { + "evidenceClass": "generated_local", + "scenario": "protected formula overwrite gate", + "workbookLikeSignals": [ + "protected_sheet", + "formula_overwrite", + "blocked_preapply_gate" + ] + }, "expectedObjects": { "charts": [], "tables": [], @@ -756,23 +834,69 @@ "activeSheet": "Model", "selectedRange": "Model!B3", "operations": [ + { + "type": "write_cell", + "sheet": "Model", + "range": "B2", + "value": 12 + }, { "type": "write_cell", "sheet": "Model", "range": "B3", "value": 13 + }, + { + "type": "write_cell", + "sheet": "Model", + "range": "B4", + "value": 14 } ], "expectedTerms": [ "Revenue" ], "expectedDirtyRanges": [ - "B3" + "Model!B2", + "Model!B3", + "Model!B4" + ], + "expectedSimulationGates": [ + "checkpoint_required" ], "expectRollback": true, + "simulationAssertions": { + "operationCount": 3, + "exactDirtyRanges": true, + "rollbackRequired": true + }, + "realismMetadata": { + "evidenceClass": "generated_local", + "scenario": "chart source series refresh", + "workbookLikeSignals": [ + "chart_source_range", + "structured_table_source", + "multi_operation_plan" + ] + }, "expectedObjects": { - "charts": [], - "tables": [], + "charts": [ + { + "sheet": "Model", + "path": "xl/charts/chart1.xml" + } + ], + "tables": [ + { + "name": "ChartSource", + "sheet": "Model", + "range": "A1:B4", + "headers": [ + "Month", + "Revenue" + ] + } + ], "pivots": [], "comments": [] } @@ -802,7 +926,12 @@ "expectedObjects": { "charts": [], "tables": [], - "pivots": [], + "pivots": [ + { + "sheet": "Model", + "path": "xl/pivotTables/pivotTable1.xml" + } + ], "comments": [] } }, @@ -848,15 +977,39 @@ "sheet": "Model", "range": "B2", "value": 11 + }, + { + "type": "write_cell", + "sheet": "Assumptions", + "range": "B2", + "value": 12 } ], "expectedTerms": [ "Linked Output" ], "expectedDirtyRanges": [ - "B2" + "Model!B2", + "Assumptions!B2" + ], + "expectedSimulationGates": [ + "checkpoint_required" ], "expectRollback": true, + "simulationAssertions": { + "operationCount": 2, + "exactDirtyRanges": true, + "rollbackRequired": true + }, + "realismMetadata": { + "evidenceClass": "generated_local", + "scenario": "cross-sheet dependency update", + "workbookLikeSignals": [ + "multi_sheet_dependency", + "dependent_formula", + "multi_operation_plan" + ] + }, "expectedObjects": { "charts": [], "tables": [], @@ -877,21 +1030,108 @@ "sheet": "Model", "range": "B2", "value": 2 + }, + { + "type": "clear_range", + "sheet": "Model", + "range": "B3:B4", + "clearType": "contents" } ], "expectedTerms": [ "Audit" ], "expectedDirtyRanges": [ - "B2" + "Model!B2", + "Model!B3:B4" + ], + "expectedSimulationGates": [ + "blocked" ], "expectRollback": true, + "simulationAssertions": { + "operationCount": 2, + "exactDirtyRanges": true, + "rollbackRequired": true, + "blockedBy": "destructive_multi_cell_clear" + }, + "realismMetadata": { + "evidenceClass": "generated_local", + "scenario": "audit flow destructive range mutation", + "workbookLikeSignals": [ + "rollback_reference", + "multi_cell_range_mutation", + "blocked_preapply_gate" + ] + }, "expectedObjects": { "charts": [], "tables": [], "pivots": [], "comments": [] } + }, + { + "fixtureId": "object_fidelity_matrix", + "filename": "object_fidelity_matrix.xlsx", + "name": "Object Fidelity Matrix", + "category": "object_fidelity", + "activeSheet": "Objects", + "selectedRange": "Objects!B2", + "operations": [ + { + "type": "write_cell", + "sheet": "Objects", + "range": "B2", + "value": 125 + } + ], + "expectedTerms": [ + "Object Fidelity" + ], + "expectedDirtyRanges": [ + "Objects!B2" + ], + "expectedSimulationGates": [ + "checkpoint_required" + ], + "expectRollback": true, + "simulationAssertions": { + "operationCount": 1, + "exactDirtyRanges": true, + "rollbackRequired": true + }, + "realismMetadata": { + "evidenceClass": "generated_local", + "scenario": "object-adjacent table mutation", + "workbookLikeSignals": [ + "protected_sheet", + "structured_table", + "cell_comment" + ] + }, + "expectedObjects": { + "charts": [], + "tables": [ + { + "name": "ObjectTable", + "sheet": "Objects", + "range": "A1:C4", + "headers": [ + "Metric", + "Value", + "Status" + ] + } + ], + "pivots": [], + "comments": [ + { + "sheet": "Objects", + "range": "B2" + } + ] + } } ] } diff --git a/backend/benchmarks/fixtures/workbooks/merger_accretion_dilution.xlsx b/backend/benchmarks/fixtures/workbooks/merger_accretion_dilution.xlsx index e8da422..bf97e69 100644 Binary files a/backend/benchmarks/fixtures/workbooks/merger_accretion_dilution.xlsx and b/backend/benchmarks/fixtures/workbooks/merger_accretion_dilution.xlsx differ diff --git a/backend/benchmarks/fixtures/workbooks/messy_dcf_wacc.xlsx b/backend/benchmarks/fixtures/workbooks/messy_dcf_wacc.xlsx index 9789b1b..f5235ff 100644 Binary files a/backend/benchmarks/fixtures/workbooks/messy_dcf_wacc.xlsx and b/backend/benchmarks/fixtures/workbooks/messy_dcf_wacc.xlsx differ diff --git a/backend/benchmarks/fixtures/workbooks/multi_sheet_dependency.xlsx b/backend/benchmarks/fixtures/workbooks/multi_sheet_dependency.xlsx index f02feed..f07e204 100644 Binary files a/backend/benchmarks/fixtures/workbooks/multi_sheet_dependency.xlsx and b/backend/benchmarks/fixtures/workbooks/multi_sheet_dependency.xlsx differ diff --git a/backend/benchmarks/fixtures/workbooks/named_range_driver.xlsx b/backend/benchmarks/fixtures/workbooks/named_range_driver.xlsx index fbab511..0abfca3 100644 Binary files a/backend/benchmarks/fixtures/workbooks/named_range_driver.xlsx and b/backend/benchmarks/fixtures/workbooks/named_range_driver.xlsx differ diff --git a/backend/benchmarks/fixtures/workbooks/object_fidelity_matrix.xlsx b/backend/benchmarks/fixtures/workbooks/object_fidelity_matrix.xlsx new file mode 100644 index 0000000..a98f421 Binary files /dev/null and b/backend/benchmarks/fixtures/workbooks/object_fidelity_matrix.xlsx differ diff --git a/backend/benchmarks/fixtures/workbooks/operating_model_kpi.xlsx b/backend/benchmarks/fixtures/workbooks/operating_model_kpi.xlsx index c03418e..11cf1c8 100644 Binary files a/backend/benchmarks/fixtures/workbooks/operating_model_kpi.xlsx and b/backend/benchmarks/fixtures/workbooks/operating_model_kpi.xlsx differ diff --git a/backend/benchmarks/fixtures/workbooks/pivot_risk_discovery.xlsx b/backend/benchmarks/fixtures/workbooks/pivot_risk_discovery.xlsx index 59c735a..50dcd6a 100644 Binary files a/backend/benchmarks/fixtures/workbooks/pivot_risk_discovery.xlsx and b/backend/benchmarks/fixtures/workbooks/pivot_risk_discovery.xlsx differ diff --git a/backend/benchmarks/fixtures/workbooks/protected_formula_overwrite.xlsx b/backend/benchmarks/fixtures/workbooks/protected_formula_overwrite.xlsx index 540372e..3fd8253 100644 Binary files a/backend/benchmarks/fixtures/workbooks/protected_formula_overwrite.xlsx and b/backend/benchmarks/fixtures/workbooks/protected_formula_overwrite.xlsx differ diff --git a/backend/benchmarks/fixtures/workbooks/repair_rollback_flow.xlsx b/backend/benchmarks/fixtures/workbooks/repair_rollback_flow.xlsx index 4ead7cc..b827677 100644 Binary files a/backend/benchmarks/fixtures/workbooks/repair_rollback_flow.xlsx and b/backend/benchmarks/fixtures/workbooks/repair_rollback_flow.xlsx differ diff --git a/backend/benchmarks/fixtures/workbooks/sensitivity_table.xlsx b/backend/benchmarks/fixtures/workbooks/sensitivity_table.xlsx index d7756a8..8c99f48 100644 Binary files a/backend/benchmarks/fixtures/workbooks/sensitivity_table.xlsx and b/backend/benchmarks/fixtures/workbooks/sensitivity_table.xlsx differ diff --git a/backend/benchmarks/fixtures/workbooks/source_backed_assumptions.xlsx b/backend/benchmarks/fixtures/workbooks/source_backed_assumptions.xlsx index 0c6ddd6..304f480 100644 Binary files a/backend/benchmarks/fixtures/workbooks/source_backed_assumptions.xlsx and b/backend/benchmarks/fixtures/workbooks/source_backed_assumptions.xlsx differ diff --git a/backend/benchmarks/fixtures/workbooks/source_tearsheet_refresh.xlsx b/backend/benchmarks/fixtures/workbooks/source_tearsheet_refresh.xlsx index 3c13529..1fc7114 100644 Binary files a/backend/benchmarks/fixtures/workbooks/source_tearsheet_refresh.xlsx and b/backend/benchmarks/fixtures/workbooks/source_tearsheet_refresh.xlsx differ diff --git a/backend/benchmarks/fixtures/workbooks/structured_reference_table.xlsx b/backend/benchmarks/fixtures/workbooks/structured_reference_table.xlsx index aec0158..6936af0 100644 Binary files a/backend/benchmarks/fixtures/workbooks/structured_reference_table.xlsx and b/backend/benchmarks/fixtures/workbooks/structured_reference_table.xlsx differ diff --git a/backend/benchmarks/fixtures/workbooks/table_source_intersection.xlsx b/backend/benchmarks/fixtures/workbooks/table_source_intersection.xlsx index ba8c158..3ca4342 100644 Binary files a/backend/benchmarks/fixtures/workbooks/table_source_intersection.xlsx and b/backend/benchmarks/fixtures/workbooks/table_source_intersection.xlsx differ diff --git a/backend/benchmarks/fixtures/workbooks/three_statement_model.xlsx b/backend/benchmarks/fixtures/workbooks/three_statement_model.xlsx index 8c1b676..9aa2820 100644 Binary files a/backend/benchmarks/fixtures/workbooks/three_statement_model.xlsx and b/backend/benchmarks/fixtures/workbooks/three_statement_model.xlsx differ diff --git a/backend/benchmarks/local_accuracy.py b/backend/benchmarks/local_accuracy.py index 56407f7..bd30454 100644 --- a/backend/benchmarks/local_accuracy.py +++ b/backend/benchmarks/local_accuracy.py @@ -9,7 +9,6 @@ import argparse import asyncio import json -import sys from typing import Any from backend.benchmarks import ( @@ -38,7 +37,7 @@ async def run_local_accuracy(*, persist: bool = False, replay_limit: int = 100) } replay_gate = _evaluate_replays(replay_limit) status = "passed" if deterministic_gate["status"] == "passed" and workbook_gate["status"] == "passed" and replay_gate["status"] == "passed" else "failed" - payload = { + payload: dict[str, Any] = { "schemaVersion": "2026-05-local-accuracy-gate-v1", "status": status, "deterministic": { @@ -53,7 +52,7 @@ async def run_local_accuracy(*, persist: bool = False, replay_limit: int = 100) } if persist: store = get_session_store() - run_payload = { + run_payload: dict[str, Any] = { "run_id": f"local_accuracy_{deterministic.run_id}", "status": status, "gate_status": status, diff --git a/backend/benchmarks/local_replay.py b/backend/benchmarks/local_replay.py index ef9816e..e7f0724 100644 --- a/backend/benchmarks/local_replay.py +++ b/backend/benchmarks/local_replay.py @@ -160,7 +160,9 @@ def evaluate_local_replays(*, limit: int = 100, persist: bool = False, baseline_ }, ) if persist: - get_session_store().save_local_replay_artifact( + store = get_session_store() + store.get_or_create_session("local-replay-evaluation") + store.save_local_replay_artifact( "local-replay-evaluation", "local_replay_evaluation", evaluation.model_dump(mode="json", by_alias=True), @@ -332,8 +334,9 @@ def main() -> int: parser.add_argument("--evaluate", action="store_true", help="Evaluate local replay artifacts.") parser.add_argument("--persist", action="store_true", help="Persist the replay evaluation as another anonymized artifact.") parser.add_argument("--limit", type=int, default=100, help="Maximum replay artifacts to evaluate.") + parser.add_argument("--baseline-id", default=None, help="Optional pinned replay baseline id to compare against.") args = parser.parse_args() - evaluation = evaluate_local_replays(limit=args.limit, persist=args.persist) + evaluation = evaluate_local_replays(limit=args.limit, persist=args.persist, baseline_id=args.baseline_id) print(json.dumps(evaluation.model_dump(mode="json", by_alias=True), indent=2)) return 1 if evaluation.status == "failed" else 0 diff --git a/backend/benchmarks/local_replay_corpus.py b/backend/benchmarks/local_replay_corpus.py new file mode 100644 index 0000000..d6fddde --- /dev/null +++ b/backend/benchmarks/local_replay_corpus.py @@ -0,0 +1,488 @@ +"""Import sanitized aggregate V7 local replay evidence. + +The importer accepts only aggregate/anonymized replay fields. It is intended to +bridge the gap between deterministic plumbing seed evidence and a real +production-like local replay corpus without allowing workbook content, formulas, +prompts, sheet text, or meaningful filenames into local replay storage. +""" + +from __future__ import annotations + +import argparse +import hashlib +import json +import math +import re +import time +from pathlib import Path +from typing import Any, Sequence + +from backend.benchmarks.local_replay import REQUIRED_REPLAY_FIELDS, evaluate_local_replays +from backend.benchmarks.local_replay_seed import LOCAL_REPLAY_SCHEMA_VERSION +from backend.models.operations import LocalReplayBaseline +from backend.storage import get_session_store + + +CORPUS_SCHEMA_VERSION = "2026-06-local-replay-corpus-v1" +DEFAULT_SESSION_ID = "v7-local-replay-corpus" +DEFAULT_BASELINE_ID = "replaybase_v7_anonymized_corpus_v1" +DEFAULT_EVIDENCE_CLASS = "anonymized_local_replay_corpus" +DEFAULT_MINIMUM_PRODUCTION_REPLAYS = 25 +PRODUCTION_CORPUS_REQUIRED_FIELDS = { + "schemaVersion", + "generatedAt", + "anonymizationPolicyVersion", + "sourceCohortHash", + "exporterVersion", + "recordCount", + "productionEvidence", +} + +ALLOWED_ARTIFACT_TYPES = { + "v7_agent_run", + "v7_apply_verification", + "v7_local_replay", + "v7_plan", + "v7_readiness", + "v7_simulation", + "v7_workbook_review", + "v7_workbook_review_client", +} + +ALLOWED_PAYLOAD_FIELDS = { + "schemaVersion", + "artifactKind", + "operationTypes", + "operationCount", + "accuracyScore", + "confidenceGapCount", + "checkpointReasonCount", + "formulaFailureCount", + "truthSchema", + "formulaSchema", + "simulationSchema", + "v7GateStatus", + "dirtyRangeCount", + "rollbackAvailable", + "replaySignature", + "payloadHash", + "fixtureIdHash", + "category", + "sourceProject", + "sheetCount", + "formulaCount", + "objectCounts", + "warningCategoryCounts", + "semanticMatchCounts", +} +ALLOWED_REPLAY_SIGNATURE_FIELDS = { + "schemaVersion", + "operationTypes", + "operationCount", + "truthSchema", + "formulaSchema", + "simulationSchema", + "v7GateStatus", + "formulaFailureCount", + "confidenceGapCount", + "checkpointReasonCount", +} +NUMERIC_SUMMARY_MAP_FIELDS = {"objectCounts", "warningCategoryCounts", "semanticMatchCounts"} +AGGREGATE_STRING_RE = re.compile(r"^[a-z0-9][a-z0-9_.:-]{0,95}$") +CELL_REFERENCE_RE = re.compile(r"\$?[A-Z]{1,3}\$?\d+(?::\$?[A-Z]{1,3}\$?\d+)?") +BANNED_PAYLOAD_KEYS = { + "address", + "addresses", + "cell", + "cells", + "cellsnapshot", + "celltext", + "content", + "document", + "documents", + "file", + "filename", + "filepath", + "formula", + "formulas", + "imagebase64", + "prompt", + "prompts", + "range", + "ranges", + "raw", + "sheetname", + "sheettext", + "source", + "text", + "value", + "values", + "workbook", + "workbookname", +} + + +def import_local_replay_corpus( + corpus_path: Path, + *, + session_id: str = DEFAULT_SESSION_ID, + baseline_id: str = DEFAULT_BASELINE_ID, + baseline_name: str = "Anonymized V7 Local Replay Corpus Baseline", + reset_session: bool = True, + production_evidence: bool = False, + evidence_class: str = DEFAULT_EVIDENCE_CLASS, + minimum_production_replays: int = DEFAULT_MINIMUM_PRODUCTION_REPLAYS, + replay_limit: int = 1000, + persist_evaluation: bool = False, +) -> dict[str, Any]: + records = load_corpus_records(corpus_path) + sanitized = [sanitize_corpus_record(record, index=index) for index, record in enumerate(records, start=1)] + production_envelope: dict[str, Any] = {} + if production_evidence: + production_envelope = production_corpus_envelope(corpus_path, expected_record_count=len(sanitized)) + if len(sanitized) < minimum_production_replays: + raise ValueError( + "Production replay evidence requires at least " + f"{minimum_production_replays} sanitized replay artifacts; received {len(sanitized)}." + ) + if evidence_class in {"", DEFAULT_EVIDENCE_CLASS, "deterministic_plumbing_seed"}: + raise ValueError("Production replay evidence requires a concrete evidence_class.") + + store = get_session_store() + store.get_or_create_session(session_id) + cleared = store.clear_local_replay_artifacts(session_id) if reset_session else 0 + replay_ids = [ + store.save_local_replay_artifact(session_id, item["artifactType"], item["payload"]) + for item in sanitized + ] + pre_baseline = evaluate_local_replays(limit=max(replay_limit, len(sanitized)), persist=False) + _assert_baselinable(pre_baseline.model_dump(mode="json", by_alias=True)) + if production_evidence and pre_baseline.capture_count != len(sanitized): + raise ValueError( + "Production replay evidence import requires an isolated local replay store: " + f"imported {len(sanitized)} artifacts but evaluated {pre_baseline.capture_count} eligible artifacts." + ) + input_hash = _file_hash(corpus_path) + baseline = LocalReplayBaseline( + baselineId=baseline_id, + name=baseline_name, + sessionId=session_id, + evaluationId=pre_baseline.evaluation_id, + replayCount=pre_baseline.capture_count, + driftSeverity=pre_baseline.drift_severity, + signatureBuckets=pre_baseline.signature_buckets, + createdAt=time.time(), + metadata={ + "source": "sanitized_local_replay_corpus", + "schemaVersion": CORPUS_SCHEMA_VERSION, + "productionEvidence": bool(production_evidence), + "evidenceClass": evidence_class, + "inputPath": str(corpus_path), + "inputSha256": input_hash, + "sessionId": session_id, + "importedReplayIds": replay_ids, + "importedArtifactCount": len(replay_ids), + "minimumProductionReplays": minimum_production_replays, + "productionCorpusEnvelope": production_envelope, + "allowedPayloadFields": sorted(ALLOWED_PAYLOAD_FIELDS), + "privacyScan": pre_baseline.privacy_scan, + "deterministicDriftClass": pre_baseline.deterministic_drift_class, + "resetSession": reset_session, + "clearedSessionArtifacts": cleared, + }, + ) + store.save_local_replay_baseline(baseline.model_dump(mode="json", by_alias=True)) + evaluation = evaluate_local_replays( + limit=max(replay_limit, len(sanitized)), + persist=persist_evaluation, + baseline_id=baseline_id, + ) + result = { + "schemaVersion": CORPUS_SCHEMA_VERSION, + "status": "passed" if evaluation.status == "passed" else "failed", + "sessionId": session_id, + "baselineId": baseline_id, + "importedReplayIds": replay_ids, + "clearedSessionArtifacts": cleared, + "productionEvidence": bool(production_evidence), + "evidenceClass": evidence_class, + "productionCorpusEnvelope": production_envelope, + "evaluation": evaluation.model_dump(mode="json", by_alias=True), + "baseline": baseline.model_dump(mode="json", by_alias=True), + } + if result["status"] != "passed": + raise RuntimeError(f"Imported local replay corpus did not pass: {json.dumps(result, sort_keys=True, default=str)}") + return result + + +def load_corpus_records(corpus_path: Path) -> list[dict[str, Any]]: + text = corpus_path.read_text(encoding="utf-8") + stripped = text.strip() + if not stripped: + raise ValueError(f"Replay corpus is empty: {corpus_path}") + if corpus_path.suffix.lower() == ".jsonl": + records = [json.loads(line) for line in stripped.splitlines() if line.strip()] + else: + raw = json.loads(stripped) + if isinstance(raw, list): + records = raw + elif isinstance(raw, dict): + payload = raw.get("artifacts") or raw.get("replays") or raw.get("items") + if not isinstance(payload, list): + raise ValueError("Replay corpus JSON must be a list or an object with artifacts/replays/items.") + records = payload + else: + raise ValueError("Replay corpus JSON must contain objects.") + if not records: + raise ValueError("Replay corpus does not contain any artifacts.") + if not all(isinstance(record, dict) for record in records): + raise ValueError("Every replay corpus record must be a JSON object.") + return records + + +def production_corpus_envelope(corpus_path: Path, *, expected_record_count: int) -> dict[str, Any]: + if corpus_path.suffix.lower() == ".jsonl": + raise ValueError("Production replay evidence requires a JSON envelope; JSONL inputs are allowed only for non-production imports.") + raw = json.loads(corpus_path.read_text(encoding="utf-8")) + if not isinstance(raw, dict): + raise ValueError("Production replay evidence requires a JSON envelope object with artifacts and provenance metadata.") + missing = sorted( + field + for field in PRODUCTION_CORPUS_REQUIRED_FIELDS + if raw.get(field) is None or raw.get(field) == "" + ) + if missing: + raise ValueError(f"Production replay evidence requires envelope metadata fields: {missing}") + if raw.get("schemaVersion") != CORPUS_SCHEMA_VERSION: + raise ValueError(f"Production replay evidence has unsupported schemaVersion: {raw.get('schemaVersion')}") + if raw.get("productionEvidence") is not True: + raise ValueError("Production replay evidence envelope must set productionEvidence: true.") + record_count = raw.get("recordCount") + if not isinstance(record_count, int) or record_count != expected_record_count: + raise ValueError( + "Production replay evidence recordCount must match sanitized artifact count: " + f"recordCount={record_count}, artifacts={expected_record_count}." + ) + source_hash = str(raw.get("sourceCohortHash") or "") + if not re.fullmatch(r"sha256:[a-f0-9]{64}", source_hash): + raise ValueError("Production replay evidence sourceCohortHash must be a sha256 digest.") + return { + "schemaVersion": raw["schemaVersion"], + "generatedAt": str(raw["generatedAt"]), + "anonymizationPolicyVersion": _sanitize_aggregate_string( + "anonymizationPolicyVersion", + str(raw["anonymizationPolicyVersion"]), + index=0, + ), + "sourceCohortHash": source_hash, + "exporterVersion": _sanitize_aggregate_string("exporterVersion", str(raw["exporterVersion"]), index=0), + "recordCount": record_count, + "productionEvidence": True, + } + + +def sanitize_corpus_record(record: dict[str, Any], *, index: int = 1) -> dict[str, Any]: + artifact_type = str(record.get("artifactType") or record.get("artifact_type") or "v7_local_replay") + if artifact_type not in ALLOWED_ARTIFACT_TYPES: + raise ValueError(f"Record {index} has unsupported artifactType: {artifact_type}") + raw_payload = record.get("payload") + payload = raw_payload if isinstance(raw_payload, dict) else { + key: value + for key, value in record.items() + if key not in {"artifactType", "artifact_type"} + } + if not isinstance(payload, dict): + raise ValueError(f"Record {index} payload must be a JSON object.") + _assert_no_restricted_keys(payload, index=index) + unknown = sorted(set(payload) - ALLOWED_PAYLOAD_FIELDS) + if unknown: + raise ValueError(f"Record {index} contains non-allowlisted aggregate fields: {unknown}") + missing = sorted(field for field in (REQUIRED_REPLAY_FIELDS | {"artifactKind", "operationTypes", "operationCount"}) if field not in payload) + if missing: + raise ValueError(f"Record {index} is missing required aggregate fields: {missing}") + if payload.get("schemaVersion") != LOCAL_REPLAY_SCHEMA_VERSION: + raise ValueError(f"Record {index} has unsupported schemaVersion: {payload.get('schemaVersion')}") + gate = str(payload.get("v7GateStatus") or "") + if gate not in {"passed", "checkpoint_required", "blocked"}: + raise ValueError(f"Record {index} has unsupported v7GateStatus: {gate}") + + sanitized = {key: _sanitize_payload_value(key, value, index=index) for key, value in payload.items() if key != "payloadHash"} + sanitized["payloadHash"] = _payload_hash(sanitized) + return {"artifactType": artifact_type, "payload": sanitized} + + +def _sanitize_payload_value(key: str, value: Any, *, index: int) -> Any: + if key == "operationTypes": + if not isinstance(value, list) or not all(isinstance(item, str) for item in value): + raise ValueError(f"Record {index} operationTypes must be a list of strings.") + return [_sanitize_aggregate_string("operationTypes", item, index=index) for item in value] + if key == "replaySignature": + if not isinstance(value, dict): + raise ValueError(f"Record {index} replaySignature must be an object.") + unknown = sorted(set(value) - ALLOWED_REPLAY_SIGNATURE_FIELDS) + if unknown: + raise ValueError(f"Record {index} replaySignature contains non-allowlisted fields: {unknown}") + _assert_no_restricted_keys(value, index=index) + return { + inner_key: _sanitize_replay_signature_value(inner_key, inner_value, index=index) + for inner_key, inner_value in value.items() + } + if key in NUMERIC_SUMMARY_MAP_FIELDS: + if not isinstance(value, dict): + raise ValueError(f"Record {index} {key} must be an object of numeric counts.") + return _numeric_summary_map(value, index=index, key=key) + if key in {"operationCount", "confidenceGapCount", "checkpointReasonCount", "formulaFailureCount", "dirtyRangeCount", "sheetCount", "formulaCount"}: + return _bounded_nonnegative_int(value, index=index, key=key) + if key == "accuracyScore": + if isinstance(value, bool) or not isinstance(value, (int, float)): + raise ValueError(f"Record {index} {key} must be a finite number.") + result = float(value) + if not math.isfinite(result) or not 0.0 <= result <= 1.0: + raise ValueError(f"Record {index} {key} must be a finite number from 0 through 1.") + return result + if key == "rollbackAvailable": + if not isinstance(value, bool): + raise ValueError(f"Record {index} {key} must be a boolean.") + return value + if isinstance(value, str): + return _sanitize_aggregate_string(key, value, index=index) + if isinstance(value, (int, float, bool)) or value is None: + return value + raise ValueError(f"Record {index} field {key} must be scalar aggregate data.") + + +def _sanitize_replay_signature_value(key: str, value: Any, *, index: int) -> Any: + if key == "operationTypes": + if not isinstance(value, list) or not all(isinstance(item, str) for item in value): + raise ValueError(f"Record {index} replaySignature.operationTypes must be a list of strings.") + return [_sanitize_aggregate_string("replaySignature.operationTypes", item, index=index) for item in value] + if key in { + "operationCount", + "formulaFailureCount", + "confidenceGapCount", + "checkpointReasonCount", + }: + return _bounded_nonnegative_int(value, index=index, key=f"replaySignature.{key}") + if isinstance(value, str): + return _sanitize_aggregate_string(f"replaySignature.{key}", value, index=index) + if isinstance(value, (int, float, bool)) or value is None: + return _json_safe(value) + raise ValueError(f"Record {index} replaySignature.{key} must be scalar aggregate data.") + + +def _sanitize_aggregate_string(key: str, value: str, *, index: int) -> str: + normalized = value.strip() + lowered = normalized.lower() + raw_markers = [".xls", ".xlsx", ".xlsm", ".xlsb", ".csv", "http://", "https://", "@", "\\", "/", "[", "]", "=", "!"] + if ( + normalized != value + or not AGGREGATE_STRING_RE.fullmatch(normalized) + or any(marker in lowered for marker in raw_markers) + or CELL_REFERENCE_RE.search(normalized) + ): + raise ValueError(f"Record {index} field {key} contains a restricted raw-content value.") + return normalized + + +def _numeric_summary_map(value: dict[str, Any], *, index: int, key: str) -> dict[str, int]: + result: dict[str, int] = {} + for raw_name, raw_count in value.items(): + name = str(raw_name) + if _normalized_key(name) in BANNED_PAYLOAD_KEYS: + raise ValueError(f"Record {index} {key} contains restricted bucket name: {name}") + if isinstance(raw_count, bool) or not isinstance(raw_count, int) or raw_count < 0: + raise ValueError(f"Record {index} {key}.{name} must be a nonnegative numeric count.") + if raw_count > 1_000_000_000: + raise ValueError(f"Record {index} {key}.{name} exceeds the numeric count budget.") + result[name] = raw_count + return result + + +def _bounded_nonnegative_int(value: Any, *, index: int, key: str) -> int: + if isinstance(value, bool) or not isinstance(value, int): + raise ValueError(f"Record {index} {key} must be a nonnegative integer.") + if not 0 <= value <= 1_000_000_000: + raise ValueError(f"Record {index} {key} exceeds the integer budget.") + return value + + +def _assert_no_restricted_keys(value: Any, *, index: int) -> None: + hits = sorted({path for path in _restricted_key_paths(value)}) + if hits: + raise ValueError(f"Record {index} contains restricted raw-content keys: {hits}") + + +def _restricted_key_paths(value: Any, prefix: str = "") -> list[str]: + if isinstance(value, dict): + hits: list[str] = [] + for key, item in value.items(): + key_text = str(key) + current = f"{prefix}.{key_text}" if prefix else key_text + if _normalized_key(key_text) in BANNED_PAYLOAD_KEYS: + hits.append(current) + hits.extend(_restricted_key_paths(item, current)) + return hits + if isinstance(value, list): + return [hit for item in value[:25] for hit in _restricted_key_paths(item, prefix)] + return [] + + +def _assert_baselinable(evaluation: dict[str, Any]) -> None: + if evaluation.get("failures"): + raise RuntimeError(f"Cannot pin replay corpus baseline with evaluation failures: {evaluation['failures']}") + if (evaluation.get("privacyScan") or {}).get("status") == "failed": + raise RuntimeError(f"Cannot pin replay corpus baseline with privacy findings: {evaluation['privacyScan']}") + if evaluation.get("driftSeverity") == "high": + raise RuntimeError(f"Cannot pin replay corpus baseline with high-severity drift: {evaluation.get('driftItems')}") + if not evaluation.get("signatureBuckets"): + raise RuntimeError("Cannot pin replay corpus baseline without deterministic signature buckets.") + + +def _payload_hash(payload: dict[str, Any]) -> str: + raw = json.dumps(_json_safe(payload), sort_keys=True, default=str) + return hashlib.sha256(raw.encode("utf-8")).hexdigest() + + +def _file_hash(path: Path) -> str: + return hashlib.sha256(path.read_bytes()).hexdigest() + + +def _json_safe(value: Any) -> Any: + return json.loads(json.dumps(value, sort_keys=True, default=str)) + + +def _normalized_key(value: str) -> str: + return value.lower().replace("_", "").replace("-", "").replace(" ", "") + + +def main(argv: Sequence[str] | None = None) -> int: + parser = argparse.ArgumentParser(description="Import a sanitized aggregate V7 local replay corpus.") + parser.add_argument("corpus", type=Path, help="JSON or JSONL corpus containing aggregate replay artifacts.") + parser.add_argument("--session-id", default=DEFAULT_SESSION_ID) + parser.add_argument("--baseline-id", default=DEFAULT_BASELINE_ID) + parser.add_argument("--baseline-name", default="Anonymized V7 Local Replay Corpus Baseline") + parser.add_argument("--keep-existing-session", action="store_true", help="Do not clear this corpus session before importing.") + parser.add_argument("--production-evidence", action="store_true", help="Mark the pinned baseline as production/anonymized evidence.") + parser.add_argument("--evidence-class", default=DEFAULT_EVIDENCE_CLASS) + parser.add_argument("--minimum-production-replays", type=int, default=DEFAULT_MINIMUM_PRODUCTION_REPLAYS) + parser.add_argument("--replay-limit", type=int, default=1000) + parser.add_argument("--persist-evaluation", action="store_true") + args = parser.parse_args(argv) + result = import_local_replay_corpus( + args.corpus, + session_id=args.session_id, + baseline_id=args.baseline_id, + baseline_name=args.baseline_name, + reset_session=not args.keep_existing_session, + production_evidence=args.production_evidence, + evidence_class=args.evidence_class, + minimum_production_replays=args.minimum_production_replays, + replay_limit=args.replay_limit, + persist_evaluation=args.persist_evaluation, + ) + print(json.dumps(result, indent=2, sort_keys=True)) + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/backend/benchmarks/local_replay_seed.py b/backend/benchmarks/local_replay_seed.py new file mode 100644 index 0000000..2f3604f --- /dev/null +++ b/backend/benchmarks/local_replay_seed.py @@ -0,0 +1,183 @@ +"""Deterministic anonymized replay seed evidence for V7 readiness. + +This module writes local-only, privacy-safe replay artifacts and pins a baseline +over the current deterministic replay signatures. It does not include workbook +cell values, formulas, prompts, source text, screenshots, or workbook payloads. +""" + +from __future__ import annotations + +import argparse +import hashlib +import json +import time +from typing import Any + +from backend.benchmarks.local_replay import evaluate_local_replays +from backend.models.operations import LocalReplayBaseline +from backend.storage import get_session_store + + +SEED_SESSION_ID = "v7-local-replay-seed" +SEED_BASELINE_ID = "replaybase_v7_local_seed_v1" +LOCAL_REPLAY_SCHEMA_VERSION = "2026-05-local-replay-v1" + + +def seed_local_replay_evidence( + *, + replay_limit: int = 100, + reset_session: bool = True, + baseline_id: str = SEED_BASELINE_ID, + persist_evaluation: bool = False, +) -> dict[str, Any]: + """Seed deterministic anonymized replays and pin a stable local baseline.""" + store = get_session_store() + store.get_or_create_session(SEED_SESSION_ID) + cleared = store.clear_local_replay_artifacts(SEED_SESSION_ID) if reset_session else 0 + replay_ids = [ + store.save_local_replay_artifact(SEED_SESSION_ID, artifact_type, _with_payload_hash(payload)) + for artifact_type, payload in _seed_payloads() + ] + pre_baseline = evaluate_local_replays(limit=replay_limit, persist=False) + _assert_seedable(pre_baseline.model_dump(mode="json", by_alias=True)) + baseline = LocalReplayBaseline( + baselineId=baseline_id, + name="Deterministic V7 Local Replay Seed Baseline", + sessionId=SEED_SESSION_ID, + evaluationId=pre_baseline.evaluation_id, + replayCount=pre_baseline.capture_count, + driftSeverity=pre_baseline.drift_severity, + signatureBuckets=pre_baseline.signature_buckets, + createdAt=time.time(), + metadata={ + "source": "deterministic_local_replay_seed", + "schemaVersion": LOCAL_REPLAY_SCHEMA_VERSION, + "productionEvidence": False, + "evidenceClass": "deterministic_plumbing_seed", + "seedReplayIds": replay_ids, + "seedArtifactCount": len(replay_ids), + "privacyScan": pre_baseline.privacy_scan, + "deterministicDriftClass": pre_baseline.deterministic_drift_class, + "resetSession": reset_session, + "clearedSeedArtifacts": cleared, + }, + ) + store.save_local_replay_baseline(baseline.model_dump(mode="json", by_alias=True)) + evaluation = evaluate_local_replays(limit=replay_limit, persist=persist_evaluation, baseline_id=baseline_id) + result = { + "schemaVersion": "2026-05-local-replay-seed-evidence", + "status": "passed" if evaluation.status == "passed" else "failed", + "sessionId": SEED_SESSION_ID, + "baselineId": baseline_id, + "seedReplayIds": replay_ids, + "clearedSeedArtifacts": cleared, + "evaluation": evaluation.model_dump(mode="json", by_alias=True), + "baseline": baseline.model_dump(mode="json", by_alias=True), + } + if result["status"] != "passed": + raise RuntimeError(f"Seeded local replay evidence did not pass: {json.dumps(result, sort_keys=True, default=str)}") + return result + + +def _seed_payloads() -> list[tuple[str, dict[str, Any]]]: + return [ + ( + "v7_plan", + { + "schemaVersion": LOCAL_REPLAY_SCHEMA_VERSION, + "artifactKind": "deterministic_seed_plan", + "operationTypes": ["write_cell"], + "operationCount": 1, + "accuracyScore": 0.99, + "confidenceGapCount": 0, + "checkpointReasonCount": 0, + "formulaFailureCount": 0, + "truthSchema": "2026-05-workbook-truth-model-v5", + "formulaSchema": "2026-05-formula-graph-artifact-v5", + "simulationSchema": "2026-05-operation-simulation-v6", + "v7GateStatus": "passed", + "dirtyRangeCount": 1, + "rollbackAvailable": True, + "replaySignature": { + "schemaVersion": LOCAL_REPLAY_SCHEMA_VERSION, + "operationTypes": ["write_cell"], + "truthSchema": "2026-05-workbook-truth-model-v5", + "formulaSchema": "2026-05-formula-graph-artifact-v5", + "simulationSchema": "2026-05-operation-simulation-v6", + "v7GateStatus": "passed", + "formulaFailureCount": 0, + "confidenceGapCount": 0, + "checkpointReasonCount": 0, + }, + }, + ), + ( + "v7_apply_verification", + { + "schemaVersion": LOCAL_REPLAY_SCHEMA_VERSION, + "artifactKind": "deterministic_seed_checkpoint", + "operationTypes": ["create_chart"], + "operationCount": 1, + "accuracyScore": 0.94, + "confidenceGapCount": 0, + "checkpointReasonCount": 1, + "formulaFailureCount": 0, + "truthSchema": "2026-05-workbook-truth-model-v5", + "formulaSchema": "2026-05-formula-graph-artifact-v5", + "simulationSchema": "2026-05-operation-simulation-v6", + "v7GateStatus": "checkpoint_required", + "dirtyRangeCount": 0, + "rollbackAvailable": False, + "replaySignature": { + "schemaVersion": LOCAL_REPLAY_SCHEMA_VERSION, + "operationTypes": ["create_chart"], + "truthSchema": "2026-05-workbook-truth-model-v5", + "formulaSchema": "2026-05-formula-graph-artifact-v5", + "simulationSchema": "2026-05-operation-simulation-v6", + "v7GateStatus": "checkpoint_required", + "formulaFailureCount": 0, + "confidenceGapCount": 0, + "checkpointReasonCount": 1, + }, + }, + ), + ] + + +def _with_payload_hash(payload: dict[str, Any]) -> dict[str, Any]: + sanitized = json.loads(json.dumps(payload, sort_keys=True, default=str)) + raw = json.dumps(sanitized, sort_keys=True, default=str) + sanitized["payloadHash"] = hashlib.sha256(raw.encode("utf-8")).hexdigest() + return sanitized + + +def _assert_seedable(evaluation: dict[str, Any]) -> None: + if evaluation.get("failures"): + raise RuntimeError(f"Cannot pin replay seed baseline with evaluation failures: {evaluation['failures']}") + if (evaluation.get("privacyScan") or {}).get("status") == "failed": + raise RuntimeError(f"Cannot pin replay seed baseline with privacy findings: {evaluation['privacyScan']}") + if evaluation.get("driftSeverity") == "high": + raise RuntimeError(f"Cannot pin replay seed baseline with high-severity drift: {evaluation.get('driftItems')}") + if not evaluation.get("signatureBuckets"): + raise RuntimeError("Cannot pin replay seed baseline without deterministic signature buckets.") + + +def main(argv: list[str] | None = None) -> int: + parser = argparse.ArgumentParser(description="Seed deterministic anonymized V7 local replay evidence.") + parser.add_argument("--replay-limit", type=int, default=100, help="Maximum local replay artifacts to baseline.") + parser.add_argument("--baseline-id", default=SEED_BASELINE_ID, help="Pinned replay baseline id.") + parser.add_argument("--keep-existing-seed-session", action="store_true", help="Do not clear prior seed-session artifacts first.") + parser.add_argument("--persist-evaluation", action="store_true", help="Persist the post-baseline evaluation artifact.") + args = parser.parse_args(argv) + result = seed_local_replay_evidence( + replay_limit=args.replay_limit, + reset_session=not args.keep_existing_seed_session, + baseline_id=args.baseline_id, + persist_evaluation=args.persist_evaluation, + ) + print(json.dumps(result, indent=2, sort_keys=True)) + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/backend/benchmarks/private_collaboration_schedules.py b/backend/benchmarks/private_collaboration_schedules.py new file mode 100644 index 0000000..1910fe1 --- /dev/null +++ b/backend/benchmarks/private_collaboration_schedules.py @@ -0,0 +1,499 @@ +"""Fail-closed validation for private randomized collaboration schedules.""" + +from __future__ import annotations + +import argparse +import base64 +import hashlib +import json +import os +import re +import stat +from dataclasses import dataclass +from pathlib import Path, PurePosixPath +from typing import Any, Literal, Sequence + + +SCHEMA_VERSION = "2026-08-private-collaboration-schedules-v1" +SCHEDULE_SCHEMA_VERSION = "2026-08-private-collaboration-schedule-v1" +MANIFEST_FILENAME = "manifest.json" +DEFAULT_PRIVATE_SCHEDULE_DIR = Path(__file__).resolve().parent / "fixtures" / "private-collaboration" +PRIVATE_RELEASE_MIN_SCHEDULE_COUNT = 100 +REQUIRED_SCENARIOS = { + "actor_undo", + "compaction", + "concurrent_edits", + "crash_recovery", + "dual_writer_fencing", + "lease_expiry", + "network_partition", +} +EVENT_KINDS = { + "bridge_acquire", + "bridge_release", + "bridge_renew", + "compact", + "crash", + "crdt_update", + "mode_switch", + "partition_end", + "partition_start", + "restart", + "undo", +} +DISPOSITIONS = {"accepted", "conflict", "deferred", "rejected"} +AUTHORITY_MODES = {"app", "office"} + +_HASH_RE = re.compile(r"^sha256:[0-9a-f]{64}$") +_ID_RE = re.compile(r"^(schedule|actor)_[0-9a-f]{8,64}$") +_TOKEN_RE = re.compile(r"^[a-z][a-z0-9_]{0,63}$") +_MAX_MANIFEST_BYTES = 4 * 1024 * 1024 +_MAX_SCHEDULE_BYTES = 64 * 1024 * 1024 +_MAX_TOTAL_BYTES = 1024 * 1024 * 1024 +_MAX_SCHEDULES = 10_000 +_MAX_ACTORS = 20 +_MAX_EVENTS = 100_000 +_MAX_EVENT_PAYLOAD_BYTES = 1024 * 1024 +_MAX_TOTAL_PAYLOAD_BYTES = 64 * 1024 * 1024 + +ValidationStatus = Literal["missing", "invalid", "ready"] + + +@dataclass(frozen=True) +class CollaborationScheduleIssue: + code: str + location: str + message: str + + def to_dict(self) -> dict[str, str]: + return {"code": self.code, "location": self.location, "message": self.message} + + +@dataclass(frozen=True) +class CollaborationScheduleSummary: + schedule_id_hash: str + seed_hash: str + expected_final_state_vector_hash: str + actor_count: int + event_count: int + scenarios: tuple[str, ...] + expected_conflict_count: int + expected_acknowledged_update_count: int + authority_mode: str + + def to_safe_dict(self) -> dict[str, Any]: + return { + "scheduleIdHash": self.schedule_id_hash, + "expectedFinalStateVectorHash": self.expected_final_state_vector_hash, + "actorCount": self.actor_count, + "eventCount": self.event_count, + "scenarios": list(self.scenarios), + "expectedConflictCount": self.expected_conflict_count, + "expectedAcknowledgedUpdateCount": self.expected_acknowledged_update_count, + "authorityMode": self.authority_mode, + } + + +@dataclass(frozen=True) +class CollaborationScheduleValidation: + status: ValidationStatus + schema_version: str | None = None + schedule_count: int = 0 + schedules: tuple[CollaborationScheduleSummary, ...] = () + issues: tuple[CollaborationScheduleIssue, ...] = () + + def to_safe_dict(self) -> dict[str, Any]: + return { + "status": self.status, + "schemaVersion": self.schema_version, + "scheduleCount": self.schedule_count, + "issueCodes": sorted({issue.code for issue in self.issues}), + "schedules": [schedule.to_safe_dict() for schedule in self.schedules], + } + + +def load_private_collaboration_schedule_manifest( + private_dir: Path | str | None = None, +) -> CollaborationScheduleValidation: + root_input = Path(private_dir) if private_dir is not None else DEFAULT_PRIVATE_SCHEDULE_DIR + if root_input.is_symlink(): + return _invalid("private_schedule_root_symlink_forbidden", "Private schedule root must not be a symlink.") + root = root_input.resolve() + manifest_path = root / MANIFEST_FILENAME + if not manifest_path.exists(): + return CollaborationScheduleValidation(status="missing") + try: + raw_manifest = _read_regular(manifest_path, _MAX_MANIFEST_BYTES) + payload = _load_json(raw_manifest) + except (OSError, ValueError, json.JSONDecodeError, UnicodeDecodeError, RecursionError): + return _invalid("private_schedule_manifest_invalid", "Private schedule manifest is invalid.") + if not isinstance(payload, dict): + return _invalid("private_schedule_manifest_invalid", "Private schedule manifest must be an object.") + + issues: list[CollaborationScheduleIssue] = [] + if set(payload) != {"schemaVersion", "schedules"}: + issues.append(_issue("manifest_fields_invalid", "manifest", "Manifest fields do not match the schema.")) + schema_version = payload.get("schemaVersion") + if schema_version != SCHEMA_VERSION: + issues.append(_issue("unsupported_schema_version", "schemaVersion", "schemaVersion is unsupported.")) + entries = payload.get("schedules") + if not isinstance(entries, list) or not entries or len(entries) > _MAX_SCHEDULES: + issues.append(_issue("schedule_inventory_invalid", "schedules", "Schedule inventory must be a bounded non-empty list.")) + entries = [] + + summaries: list[CollaborationScheduleSummary] = [] + seen_ids: set[str] = set() + seen_paths: set[str] = set() + seen_schedule_digests: set[str] = set() + seen_source_receipts: set[str] = set() + seen_seeds: set[str] = set() + total_bytes = 0 + for position, raw_entry in enumerate(entries): + location = f"schedules[{position}]" + if not isinstance(raw_entry, dict) or set(raw_entry) != { + "scheduleIdHash", + "schedulePath", + "scheduleSha256", + "provenance", + }: + issues.append(_issue("schedule_entry_invalid", location, "Schedule entry fields are invalid.")) + continue + schedule_id = raw_entry.get("scheduleIdHash") + if not isinstance(schedule_id, str) or not _ID_RE.fullmatch(schedule_id) or not schedule_id.startswith("schedule_"): + issues.append(_issue("schedule_id_hash_invalid", f"{location}.scheduleIdHash", "Schedule ID hash is invalid.")) + continue + if schedule_id in seen_ids: + issues.append(_issue("schedule_id_duplicate", f"{location}.scheduleIdHash", "Schedule ID hash is duplicated.")) + continue + seen_ids.add(schedule_id) + relative_path = _safe_schedule_path(raw_entry.get("schedulePath")) + if relative_path is None: + issues.append(_issue("schedule_path_invalid", f"{location}.schedulePath", "Schedule path is invalid.")) + continue + path_key = relative_path.casefold() + if path_key in seen_paths: + issues.append(_issue("schedule_path_duplicate", f"{location}.schedulePath", "Schedule path collides by case.")) + continue + seen_paths.add(path_key) + digest = raw_entry.get("scheduleSha256") + if not isinstance(digest, str) or not _HASH_RE.fullmatch(digest): + issues.append(_issue("schedule_sha256_invalid", f"{location}.scheduleSha256", "Schedule digest is invalid.")) + continue + if digest in seen_schedule_digests: + issues.append(_issue("schedule_sha256_duplicate", f"{location}.scheduleSha256", "Schedule digest is duplicated.")) + continue + seen_schedule_digests.add(digest) + provenance = raw_entry.get("provenance") + if not _valid_provenance(provenance): + issues.append(_issue("schedule_provenance_invalid", f"{location}.provenance", "Schedule provenance is invalid.")) + continue + assert isinstance(provenance, dict) + source_receipt = provenance["sourceReceiptHash"] + if source_receipt in seen_source_receipts: + issues.append(_issue("schedule_source_receipt_duplicate", f"{location}.provenance", "Schedule source receipt is duplicated.")) + continue + seen_source_receipts.add(source_receipt) + try: + schedule_path = _resolve_private_file(root, relative_path) + raw_schedule = _read_regular(schedule_path, _MAX_SCHEDULE_BYTES) + total_bytes += len(raw_schedule) + if total_bytes > _MAX_TOTAL_BYTES: + raise ValueError("schedule_corpus_budget_exceeded") + if f"sha256:{hashlib.sha256(raw_schedule).hexdigest()}" != digest: + raise ValueError("schedule_sha256_mismatch") + summary = _validate_schedule_payload(_load_json(raw_schedule), schedule_id) + except (OSError, ValueError, json.JSONDecodeError, UnicodeDecodeError, RecursionError) as exc: + code = str(exc) if str(exc) in { + "schedule_corpus_budget_exceeded", + "schedule_sha256_mismatch", + } else "schedule_payload_invalid" + issues.append(_issue(code, location, "Schedule payload could not be verified.")) + continue + if summary.seed_hash in seen_seeds: + issues.append(_issue("schedule_seed_duplicate", location, "Schedule seed is duplicated.")) + continue + seen_seeds.add(summary.seed_hash) + summaries.append(summary) + + if issues: + return CollaborationScheduleValidation( + status="invalid", + schema_version=schema_version if isinstance(schema_version, str) else None, + issues=tuple(issues), + ) + return CollaborationScheduleValidation( + status="ready", + schema_version=SCHEMA_VERSION, + schedule_count=len(summaries), + schedules=tuple(summaries), + ) + + +def private_collaboration_schedule_policy( + validation: CollaborationScheduleValidation, +) -> dict[str, Any]: + present_scenarios = sorted( + {scenario for schedule in validation.schedules for scenario in schedule.scenarios} + ) + missing_scenarios = sorted(REQUIRED_SCENARIOS - set(present_scenarios)) + issue_codes: list[str] = [] + if validation.status == "ready" and validation.schedule_count < PRIVATE_RELEASE_MIN_SCHEDULE_COUNT: + issue_codes.append("private_collaboration_minimum_count_missing") + if validation.status == "ready" and missing_scenarios: + issue_codes.append("private_collaboration_scenarios_missing") + return { + "ready": validation.status == "ready" and not issue_codes, + "minimumScheduleCount": PRIVATE_RELEASE_MIN_SCHEDULE_COUNT, + "requiredScenarios": sorted(REQUIRED_SCENARIOS), + "presentScenarios": present_scenarios, + "missingScenarios": missing_scenarios, + "issueCodes": issue_codes, + } + + +def private_collaboration_schedule_policy_snapshot( + private_dir: Path | str | None = None, +) -> dict[str, Any]: + validation = load_private_collaboration_schedule_manifest(private_dir) + policy = private_collaboration_schedule_policy(validation) + return { + "schemaVersion": SCHEMA_VERSION, + "status": validation.status if validation.status != "ready" or policy["ready"] else "insufficient", + "ready": policy["ready"], + "scheduleCount": validation.schedule_count, + "issueCodes": sorted({*(issue.code for issue in validation.issues), *policy["issueCodes"]}), + "releasePolicy": policy, + } + + +def _validate_schedule_payload(payload: Any, schedule_id: str) -> CollaborationScheduleSummary: + if not isinstance(payload, dict) or set(payload) != { + "schemaVersion", + "scheduleIdHash", + "seedHash", + "actors", + "scenarios", + "events", + "expected", + }: + raise ValueError("schedule_payload_invalid") + if payload["schemaVersion"] != SCHEDULE_SCHEMA_VERSION or payload["scheduleIdHash"] != schedule_id: + raise ValueError("schedule_payload_invalid") + if not isinstance(payload["seedHash"], str) or not _HASH_RE.fullmatch(payload["seedHash"]): + raise ValueError("schedule_payload_invalid") + actors = payload["actors"] + if ( + not isinstance(actors, list) + or len(actors) < 2 + or len(actors) > _MAX_ACTORS + or actors != sorted(set(actors)) + or any(not isinstance(actor, str) or not _ID_RE.fullmatch(actor) or not actor.startswith("actor_") for actor in actors) + ): + raise ValueError("schedule_payload_invalid") + scenarios = payload["scenarios"] + if ( + not isinstance(scenarios, list) + or not scenarios + or scenarios != sorted(set(scenarios)) + or any(not isinstance(value, str) or not _TOKEN_RE.fullmatch(value) for value in scenarios) + ): + raise ValueError("schedule_payload_invalid") + events = payload["events"] + if not isinstance(events, list) or not events or len(events) > _MAX_EVENTS: + raise ValueError("schedule_payload_invalid") + total_payload_bytes = 0 + prior_time = -1 + for index, event in enumerate(events, start=1): + if not isinstance(event, dict) or set(event) != { + "sequence", + "actorIdHash", + "logicalTime", + "kind", + "payloadBase64", + "payloadSha256", + "expectedDisposition", + }: + raise ValueError("schedule_payload_invalid") + logical_time = event["logicalTime"] + sequence = event["sequence"] + actor_id = event["actorIdHash"] + kind = event["kind"] + disposition = event["expectedDisposition"] + if ( + isinstance(sequence, bool) + or not isinstance(sequence, int) + or sequence != index + or not isinstance(actor_id, str) + or actor_id not in actors + or isinstance(logical_time, bool) + or not isinstance(logical_time, int) + or logical_time < prior_time + or not isinstance(kind, str) + or kind not in EVENT_KINDS + or not isinstance(disposition, str) + or disposition not in DISPOSITIONS + ): + raise ValueError("schedule_payload_invalid") + prior_time = logical_time + encoded = event["payloadBase64"] + digest = event["payloadSha256"] + if not isinstance(encoded, str) or not isinstance(digest, str) or not _HASH_RE.fullmatch(digest): + raise ValueError("schedule_payload_invalid") + try: + decoded = base64.b64decode(encoded, validate=True) + except (ValueError, TypeError) as exc: + raise ValueError("schedule_payload_invalid") from exc + if ( + not decoded + or len(decoded) > _MAX_EVENT_PAYLOAD_BYTES + or base64.b64encode(decoded).decode("ascii") != encoded + or f"sha256:{hashlib.sha256(decoded).hexdigest()}" != digest + ): + raise ValueError("schedule_payload_invalid") + total_payload_bytes += len(decoded) + if total_payload_bytes > _MAX_TOTAL_PAYLOAD_BYTES: + raise ValueError("schedule_payload_invalid") + expected = payload["expected"] + if not isinstance(expected, dict) or set(expected) != { + "finalStateVectorHash", + "conflictCount", + "acknowledgedUpdateCount", + "lostAcknowledgedUpdateCount", + "authorityMode", + }: + raise ValueError("schedule_payload_invalid") + if not isinstance(expected["finalStateVectorHash"], str) or not _HASH_RE.fullmatch(expected["finalStateVectorHash"]): + raise ValueError("schedule_payload_invalid") + conflict_count = _nonnegative_int(expected["conflictCount"]) + acknowledged_count = _nonnegative_int(expected["acknowledgedUpdateCount"]) + if ( + expected["lostAcknowledgedUpdateCount"] != 0 + or isinstance(expected["lostAcknowledgedUpdateCount"], bool) + or not isinstance(expected["authorityMode"], str) + or expected["authorityMode"] not in AUTHORITY_MODES + ): + raise ValueError("schedule_payload_invalid") + return CollaborationScheduleSummary( + schedule_id_hash=schedule_id, + seed_hash=payload["seedHash"], + expected_final_state_vector_hash=expected["finalStateVectorHash"], + actor_count=len(actors), + event_count=len(events), + scenarios=tuple(scenarios), + expected_conflict_count=conflict_count, + expected_acknowledged_update_count=acknowledged_count, + authority_mode=expected["authorityMode"], + ) + + +def _valid_provenance(value: Any) -> bool: + return ( + isinstance(value, dict) + and set(value) == {"kind", "generatorHash", "sourceReceiptHash"} + and value.get("kind") == "randomized" + and isinstance(value.get("generatorHash"), str) + and bool(_HASH_RE.fullmatch(value["generatorHash"])) + and isinstance(value.get("sourceReceiptHash"), str) + and bool(_HASH_RE.fullmatch(value["sourceReceiptHash"])) + ) + + +def _safe_schedule_path(value: Any) -> str | None: + if not isinstance(value, str) or "\\" in value: + return None + path = PurePosixPath(value) + if ( + path.is_absolute() + or path.as_posix() != value + or path.parts[:1] != ("schedules",) + or path.suffix != ".json" + ): + return None + if any(part in {"", ".", ".."} or not re.fullmatch(r"[A-Za-z0-9._-]+", part) for part in path.parts): + return None + return path.as_posix() + + +def _resolve_private_file(root: Path, relative_path: str) -> Path: + current = root + for part in PurePosixPath(relative_path).parts: + current = current / part + if current.is_symlink(): + raise ValueError("schedule_payload_invalid") + resolved = current.resolve(strict=True) + resolved.relative_to(root) + return resolved + + +def _read_regular(path: Path, maximum: int) -> bytes: + flags = os.O_RDONLY | getattr(os, "O_NOFOLLOW", 0) | getattr(os, "O_CLOEXEC", 0) + descriptor = os.open(path, flags) + try: + before = os.fstat(descriptor) + if not stat.S_ISREG(before.st_mode) or before.st_size <= 0 or before.st_size > maximum: + raise ValueError("file_boundary_invalid") + with os.fdopen(descriptor, "rb", closefd=False) as handle: + data = handle.read(maximum + 1) + after = os.fstat(descriptor) + finally: + os.close(descriptor) + if ( + len(data) != before.st_size + or len(data) > maximum + or (before.st_dev, before.st_ino, before.st_size, before.st_mtime_ns) + != (after.st_dev, after.st_ino, after.st_size, after.st_mtime_ns) + ): + raise ValueError("file_boundary_invalid") + return data + + +def _load_json(raw: bytes) -> Any: + def pairs(items: list[tuple[str, Any]]) -> dict[str, Any]: + result: dict[str, Any] = {} + for key, value in items: + if key in result: + raise ValueError("duplicate_json_key") + result[key] = value + return result + + return json.loads( + raw.decode("utf-8"), + object_pairs_hook=pairs, + parse_constant=lambda _value: (_ for _ in ()).throw(ValueError("nonstandard_json_number")), + ) + + +def _nonnegative_int(value: Any) -> int: + if isinstance(value, bool) or not isinstance(value, int) or value < 0: + raise ValueError("schedule_payload_invalid") + return value + + +def _issue(code: str, location: str, message: str) -> CollaborationScheduleIssue: + return CollaborationScheduleIssue(code=code, location=location, message=message) + + +def _invalid(code: str, message: str) -> CollaborationScheduleValidation: + return CollaborationScheduleValidation( + status="invalid", + issues=(_issue(code, "manifest", message),), + ) + + +def main(argv: Sequence[str] | None = None) -> int: + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("--private-root", type=Path, default=DEFAULT_PRIVATE_SCHEDULE_DIR) + parser.add_argument("--require-release-scale", action="store_true") + parser.add_argument("--json", action="store_true") + args = parser.parse_args(argv) + report = private_collaboration_schedule_policy_snapshot(args.private_root) + print(json.dumps(report, indent=2 if args.json else None, sort_keys=True)) + if report["status"] in {"missing", "invalid"}: + return 1 + if args.require_release_scale and report["ready"] is not True: + return 1 + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/backend/benchmarks/private_golden_fixtures.py b/backend/benchmarks/private_golden_fixtures.py new file mode 100644 index 0000000..654a207 --- /dev/null +++ b/backend/benchmarks/private_golden_fixtures.py @@ -0,0 +1,888 @@ +"""Fail-closed validation for local private golden workbook fixtures.""" + +from __future__ import annotations + +import hashlib +import json +import os +import re +import stat +import zipfile +from dataclasses import dataclass +from pathlib import Path +from typing import Any, Literal + + +SCHEMA_VERSION = "2026-08-private-golden-fixtures-v2" +MANIFEST_FILENAME = "manifest.json" +DEFAULT_PRIVATE_FIXTURE_DIR = Path(__file__).resolve().parent / "fixtures" / "private" +PRIVATE_RELEASE_MIN_FIXTURE_COUNT = 250 +PRIVATE_RELEASE_REQUIRED_CATEGORIES = { + "charts_tables_pivots_macros", + "financial_model", + "messy_formulas_data", + "protected_workbook", +} + +_HASHED_ID_RE = re.compile(r"^(fx_[a-f0-9]{8,64}|sha256:[a-f0-9]{64})$") +_HASH_RE = re.compile(r"^sha256:[a-f0-9]{64}$") +_WORKBOOK_EXTENSIONS = {".xlsx", ".xlsm", ".xlsb"} +_MAX_PACKAGE_BYTES = 2 * 1024 * 1024 * 1024 +_MAX_PACKAGE_MEMBERS = 200_000 +_MAX_MEMBER_BYTES = 512 * 1024 * 1024 +_MAX_TOTAL_UNCOMPRESSED_BYTES = 4 * 1024 * 1024 * 1024 +_MAX_COMPRESSION_RATIO = 1_000 +_MAX_REQUIRED_XML_BYTES = 16 * 1024 * 1024 +_MAX_MANIFEST_BYTES = 16 * 1024 * 1024 +_MAX_FIXTURE_COUNT = 10_000 +_SAFE_TOKEN_RE = re.compile(r"^[a-z][a-z0-9_]{0,63}$") +_FIXTURE_FIELDS = { + "fixtureIdHash", + "category", + "workbookPath", + "workbookSha256", + "provenance", + "expectedTruthTargets", + "expectedFormulaFacts", + "expectedSimulationGates", + "rollbackExpectations", + "repairOutcomes", + "auditExpectations", +} + +ValidationStatus = Literal["missing", "invalid", "ready"] + + +@dataclass(frozen=True) +class PrivateGoldenFixtureIssue: + code: str + location: str + message: str + + def to_dict(self) -> dict[str, str]: + return {"code": self.code, "location": self.location, "message": self.message} + + +@dataclass(frozen=True) +class PrivateGoldenFixture: + fixture_id_hash: str + category: str + workbook_path: str + workbook_sha256: str + workbook_extension: str + workbook_size_class: str + provenance_kind: str + expected_truth_targets: tuple[Any, ...] + expected_formula_facts: tuple[Any, ...] + expected_simulation_gates: tuple[str, ...] + rollback_expectations: dict[str, Any] + repair_outcomes: tuple[Any, ...] + audit_expectations: dict[str, Any] + + def to_summary_dict(self) -> dict[str, Any]: + return { + "fixtureIdHash": self.fixture_id_hash, + "category": self.category, + "workbookExtension": self.workbook_extension, + "workbookSizeClass": self.workbook_size_class, + "provenanceKind": self.provenance_kind, + "expectedTruthTargetCount": len(self.expected_truth_targets), + "expectedFormulaFactCount": len(self.expected_formula_facts), + "expectedSimulationGates": list(self.expected_simulation_gates), + "rollbackExpected": bool(self.rollback_expectations.get("required")), + "repairOutcomeCount": len(self.repair_outcomes), + "auditEvidenceCount": len(self.audit_expectations.get("requiredEvidence") or []), + } + + +@dataclass(frozen=True) +class PrivateGoldenFixtureValidation: + status: ValidationStatus + schema_version: str | None = None + fixture_count: int = 0 + fixtures: tuple[PrivateGoldenFixture, ...] = () + issues: tuple[PrivateGoldenFixtureIssue, ...] = () + + def to_safe_dict(self) -> dict[str, Any]: + return { + "status": self.status, + "schemaVersion": self.schema_version, + "fixtureCount": self.fixture_count, + "issues": [issue.to_dict() for issue in self.issues], + "fixtures": [fixture.to_summary_dict() for fixture in self.fixtures], + } + + +def validate_private_golden_fixture_manifest( + private_dir: Path | str | None = None, +) -> PrivateGoldenFixtureValidation: + return load_private_golden_fixture_manifest(private_dir) + + +def private_golden_fixture_policy_snapshot( + private_dir: Path | str | None = None, + *, + require_private_fixtures: bool = False, +) -> dict[str, Any]: + validation = load_private_golden_fixture_manifest(private_dir) + release_policy = private_fixture_release_policy(validation) + ready = validation.status == "ready" and release_policy["ready"] + issue_codes = sorted({issue.code for issue in validation.issues}) + required_actions: list[str] = [] + if not ready: + if validation.status == "missing": + required_actions.append( + "Populate the gitignored private golden fixture manifest before private-fixture-gated release evidence." + ) + elif validation.status == "invalid": + required_actions.append( + "Fix the gitignored private golden fixture manifest metadata before private-fixture-gated release evidence." + ) + elif validation.status == "ready" and not release_policy["ready"]: + required_actions.extend(release_policy["requiredActions"]) + if require_private_fixtures and not ready: + required_actions.append("Run private golden fixture validation until status is ready.") + return { + "schemaVersion": SCHEMA_VERSION, + "status": "insufficient" if validation.status == "ready" and not release_policy["ready"] else validation.status, + "ready": ready, + "required": require_private_fixtures, + "fixtureCount": validation.fixture_count, + "issueCodes": sorted({*issue_codes, *release_policy["issueCodes"]}), + "releasePolicy": release_policy, + "requiredActions": required_actions, + } + + +def private_fixture_release_policy(validation: PrivateGoldenFixtureValidation) -> dict[str, Any]: + if validation.status != "ready": + return { + "ready": False, + "minimumFixtureCount": PRIVATE_RELEASE_MIN_FIXTURE_COUNT, + "requiredCategories": sorted(PRIVATE_RELEASE_REQUIRED_CATEGORIES), + "presentCategories": [], + "missingCategories": sorted(PRIVATE_RELEASE_REQUIRED_CATEGORIES), + "issueCodes": [], + "requiredActions": [], + } + present_categories = sorted({fixture.category for fixture in validation.fixtures}) + missing_categories = sorted(PRIVATE_RELEASE_REQUIRED_CATEGORIES - set(present_categories)) + issue_codes: list[str] = [] + required_actions: list[str] = [] + if validation.fixture_count < PRIVATE_RELEASE_MIN_FIXTURE_COUNT: + issue_codes.append("private_fixture_minimum_count_missing") + required_actions.append( + f"Add at least {PRIVATE_RELEASE_MIN_FIXTURE_COUNT} private golden fixtures before private-fixture-gated release evidence." + ) + if missing_categories: + issue_codes.append("private_fixture_required_categories_missing") + required_actions.append( + "Add private golden fixture coverage for required categories: " + + ", ".join(missing_categories) + + "." + ) + return { + "ready": not issue_codes, + "minimumFixtureCount": PRIVATE_RELEASE_MIN_FIXTURE_COUNT, + "requiredCategories": sorted(PRIVATE_RELEASE_REQUIRED_CATEGORIES), + "presentCategories": present_categories, + "missingCategories": missing_categories, + "issueCodes": issue_codes, + "requiredActions": required_actions, + } + + +def load_private_golden_fixture_manifest( + private_dir: Path | str | None = None, +) -> PrivateGoldenFixtureValidation: + fixture_dir = Path(private_dir) if private_dir is not None else DEFAULT_PRIVATE_FIXTURE_DIR + if fixture_dir.is_symlink(): + return _invalid( + _issue( + "manifest", + "private_fixture_root_symlink_forbidden", + "Private fixture root must not be a symbolic link.", + ) + ) + fixture_root = fixture_dir.resolve() + manifest_path = fixture_root / MANIFEST_FILENAME + if not manifest_path.exists(): + return PrivateGoldenFixtureValidation(status="missing") + if manifest_path.is_symlink(): + return _invalid( + _issue( + "manifest", + "private_manifest_symlink_forbidden", + "Private fixture manifest must not be a symbolic link.", + ) + ) + + try: + payload = _read_manifest(manifest_path) + except (OSError, UnicodeDecodeError, json.JSONDecodeError, ValueError): + return _invalid(_issue("manifest", "manifest_json_invalid", "Manifest JSON could not be parsed.")) + + if not isinstance(payload, dict): + return _invalid(_issue("manifest", "manifest_must_be_object", "Manifest must be a JSON object.")) + + issues: list[PrivateGoldenFixtureIssue] = [] + schema_version = payload.get("schemaVersion") + if schema_version != SCHEMA_VERSION: + issues.append(_issue("schemaVersion", "unsupported_schema_version", "schemaVersion is not supported.")) + if set(payload) != {"schemaVersion", "fixtures"}: + issues.append( + _issue( + "manifest", + "manifest_fields_invalid", + "Manifest must contain only schemaVersion and fixtures.", + ) + ) + + fixture_items = payload.get("fixtures") + if not isinstance(fixture_items, list) or not fixture_items: + issues.append(_issue("fixtures", "fixtures_required", "fixtures must be a non-empty list.")) + fixture_items = [] + elif len(fixture_items) > _MAX_FIXTURE_COUNT: + issues.append( + _issue( + "fixtures", + "fixture_count_budget_exceeded", + f"Manifest cannot contain more than {_MAX_FIXTURE_COUNT} fixtures.", + ) + ) + fixture_items = [] + + fixtures: list[PrivateGoldenFixture] = [] + for index, item in enumerate(fixture_items): + fixture = _validate_fixture_item(item, index, fixture_root, issues) + if fixture is not None: + fixtures.append(fixture) + + seen_fixture_ids: dict[str, int] = {} + seen_workbook_paths: dict[str, int] = {} + for index, fixture in enumerate(fixtures): + fixture_key = fixture.fixture_id_hash.casefold() + if fixture_key in seen_fixture_ids: + issues.append( + _issue( + f"fixtures[{index}].fixtureIdHash", + "fixture_id_duplicate", + "Fixture IDs must be unique.", + ) + ) + else: + seen_fixture_ids[fixture_key] = index + path_key = fixture.workbook_path.casefold() + if path_key in seen_workbook_paths: + issues.append( + _issue( + f"fixtures[{index}].workbookPath", + "workbook_path_duplicate", + "Workbook paths must be unique and cannot collide by case.", + ) + ) + else: + seen_workbook_paths[path_key] = index + + if issues: + return PrivateGoldenFixtureValidation( + status="invalid", + schema_version=str(schema_version) if isinstance(schema_version, str) else None, + fixture_count=0, + issues=tuple(issues), + ) + + return PrivateGoldenFixtureValidation( + status="ready", + schema_version=SCHEMA_VERSION, + fixture_count=len(fixtures), + fixtures=tuple(fixtures), + ) + + +def _validate_fixture_item( + item: Any, + index: int, + fixture_root: Path, + issues: list[PrivateGoldenFixtureIssue], +) -> PrivateGoldenFixture | None: + location = f"fixtures[{index}]" + if not isinstance(item, dict): + issues.append(_issue(location, "fixture_must_be_object", "Fixture entry must be an object.")) + return None + if set(item) != _FIXTURE_FIELDS: + issues.append( + _issue( + location, + "fixture_fields_invalid", + "Fixture fields do not match the private corpus schema.", + ) + ) + + fixture_id_hash = _required_string(item, "fixtureIdHash", location, issues) + if fixture_id_hash and not _HASHED_ID_RE.match(fixture_id_hash): + issues.append(_issue(f"{location}.fixtureIdHash", "fixture_id_hash_required", "Fixture ID must be hashed.")) + + category = _required_string(item, "category", location, issues) + if category and not _SAFE_TOKEN_RE.fullmatch(category): + issues.append( + _issue( + f"{location}.category", + "fixture_category_invalid", + "Fixture category must be a generic lowercase token.", + ) + ) + workbook = _validate_workbook_path( + item.get("workbookPath"), + item.get("workbookSha256"), + location, + fixture_root, + issues, + ) + provenance = _validate_provenance(item.get("provenance"), location, issues) + expected_truth_targets = _required_list(item, "expectedTruthTargets", location, issues) + expected_formula_facts = _validate_formula_facts(item.get("expectedFormulaFacts"), location, issues) + expected_simulation_gates = _required_string_list(item, "expectedSimulationGates", location, issues) + rollback_expectations = _validate_rollback_expectations(item.get("rollbackExpectations"), location, issues) + repair_outcomes = _required_list(item, "repairOutcomes", location, issues) + audit_expectations = _validate_audit_expectations(item.get("auditExpectations"), location, issues) + + if ( + fixture_id_hash is None + or category is None + or workbook is None + or provenance is None + or expected_truth_targets is None + or expected_formula_facts is None + or expected_simulation_gates is None + or rollback_expectations is None + or repair_outcomes is None + or audit_expectations is None + ): + return None + if ( + not fixture_id_hash + or not category + or not workbook.relative_path + or not expected_truth_targets + or not expected_formula_facts + or not expected_simulation_gates + or not repair_outcomes + ): + return None + + return PrivateGoldenFixture( + fixture_id_hash=fixture_id_hash, + category=category, + workbook_path=workbook.relative_path, + workbook_sha256=workbook.sha256, + workbook_extension=Path(workbook.relative_path).suffix.lower(), + workbook_size_class=workbook.size_class, + provenance_kind=str(provenance["kind"]), + expected_truth_targets=tuple(expected_truth_targets), + expected_formula_facts=tuple(expected_formula_facts), + expected_simulation_gates=tuple(expected_simulation_gates), + rollback_expectations=dict(rollback_expectations), + repair_outcomes=tuple(repair_outcomes), + audit_expectations=dict(audit_expectations), + ) + + +@dataclass(frozen=True) +class _ValidatedWorkbook: + relative_path: str + sha256: str + size_class: str + + +def _validate_workbook_path( + value: Any, + expected_sha256: Any, + location: str, + fixture_root: Path, + issues: list[PrivateGoldenFixtureIssue], +) -> _ValidatedWorkbook | None: + if not isinstance(value, str) or not value.strip(): + issues.append(_issue(f"{location}.workbookPath", "workbook_path_required", "workbookPath is required.")) + return None + + raw_path = value.strip() + relative_path = Path(raw_path) + if ( + relative_path.is_absolute() + or relative_path.drive + or ".." in relative_path.parts + or "\\" in raw_path + or any(part in {"", "."} for part in relative_path.parts) + ): + issues.append( + _issue( + f"{location}.workbookPath", + "workbook_path_not_private_relative", + "workbookPath must stay under the private fixture folder.", + ) + ) + return None + if relative_path.suffix.lower() not in _WORKBOOK_EXTENSIONS: + issues.append( + _issue( + f"{location}.workbookPath", + "workbook_extension_not_supported", + "workbookPath must use a supported workbook extension.", + ) + ) + return None + + workbook_path = fixture_root / relative_path + if not _is_relative_to(workbook_path.absolute(), fixture_root): + issues.append( + _issue( + f"{location}.workbookPath", + "workbook_path_not_private_relative", + "workbookPath must stay under the private fixture folder.", + ) + ) + return None + cursor = fixture_root + try: + for part in relative_path.parts: + cursor = cursor / part + if stat.S_ISLNK(cursor.lstat().st_mode): + issues.append( + _issue( + f"{location}.workbookPath", + "workbook_path_symlink_forbidden", + "workbookPath must not contain symbolic links.", + ) + ) + return None + except OSError: + issues.append( + _issue( + f"{location}.workbookPath", + "workbook_path_missing", + "workbookPath must point to an existing private workbook.", + ) + ) + return None + if not isinstance(expected_sha256, str) or not _HASH_RE.fullmatch(expected_sha256): + issues.append( + _issue( + f"{location}.workbookSha256", + "workbook_sha256_required", + "workbookSha256 must be a lowercase sha256 digest.", + ) + ) + return None + + validation = _validate_workbook_package(workbook_path) + if isinstance(validation, str): + issues.append( + _issue( + f"{location}.workbookPath", + validation, + "workbookPath must point to a safe, bounded OOXML workbook package.", + ) + ) + return None + digest, size = validation + actual_sha256 = f"sha256:{digest}" + if actual_sha256 != expected_sha256: + issues.append( + _issue( + f"{location}.workbookSha256", + "workbook_sha256_mismatch", + "workbookSha256 does not match the private workbook bytes.", + ) + ) + return None + return _ValidatedWorkbook( + relative_path=relative_path.as_posix(), + sha256=actual_sha256, + size_class=_workbook_size_class(size), + ) + + +def _validate_workbook_package(path: Path) -> tuple[str, int] | str: + flags = os.O_RDONLY | getattr(os, "O_BINARY", 0) | getattr(os, "O_CLOEXEC", 0) + flags |= getattr(os, "O_NOFOLLOW", 0) + try: + descriptor = os.open(path, flags) + except OSError: + return "workbook_path_missing" + try: + with os.fdopen(descriptor, "rb") as workbook_file: + before = os.fstat(workbook_file.fileno()) + if ( + not stat.S_ISREG(before.st_mode) + or before.st_size <= 0 + or before.st_size > _MAX_PACKAGE_BYTES + ): + return "workbook_package_size_invalid" + signature = workbook_file.read(4) + if signature != b"PK\x03\x04": + return "workbook_package_signature_invalid" + workbook_file.seek(0) + digest = hashlib.sha256() + while chunk := workbook_file.read(1024 * 1024): + digest.update(chunk) + workbook_file.seek(0) + with zipfile.ZipFile(workbook_file) as workbook_zip: + infos = workbook_zip.infolist() + if len(infos) > _MAX_PACKAGE_MEMBERS: + return "workbook_package_member_budget_exceeded" + names: set[str] = set() + casefolded_names: set[str] = set() + total_uncompressed = 0 + for info in infos: + name = info.filename + package_path = Path(name) + if ( + not name + or "\\" in name + or package_path.is_absolute() + or package_path.drive + or ".." in package_path.parts + or any(part in {"", "."} for part in package_path.parts) + ): + return "workbook_package_member_path_invalid" + name_key = name.casefold() + if name in names or name_key in casefolded_names: + return "workbook_package_member_duplicate" + names.add(name) + casefolded_names.add(name_key) + if info.flag_bits & 0x1: + return "workbook_package_encrypted_member" + if info.file_size < 0 or info.file_size > _MAX_MEMBER_BYTES: + return "workbook_package_member_budget_exceeded" + total_uncompressed += info.file_size + if total_uncompressed > _MAX_TOTAL_UNCOMPRESSED_BYTES: + return "workbook_package_total_budget_exceeded" + if ( + info.file_size > 0 + and info.file_size / max(info.compress_size, 1) + > _MAX_COMPRESSION_RATIO + ): + return "workbook_package_compression_ratio_exceeded" + if "[Content_Types].xml" not in names: + return "workbook_package_content_types_missing" + workbook_parts = sorted( + name + for name in names + if name in {"xl/workbook.xml", "xl/workbook.bin"} + ) + if not workbook_parts: + return "workbook_package_workbook_part_missing" + for required_name in ("[Content_Types].xml", workbook_parts[0]): + with workbook_zip.open(required_name) as required_part: + if ( + len(required_part.read(_MAX_REQUIRED_XML_BYTES + 1)) + > _MAX_REQUIRED_XML_BYTES + ): + return "workbook_package_required_part_budget_exceeded" + after = os.fstat(workbook_file.fileno()) + if ( + before.st_dev != after.st_dev + or before.st_ino != after.st_ino + or before.st_size != after.st_size + or before.st_mtime_ns != after.st_mtime_ns + ): + return "workbook_package_changed_during_validation" + except ( + EOFError, + KeyError, + NotImplementedError, + OSError, + RuntimeError, + zipfile.BadZipFile, + zipfile.LargeZipFile, + ): + return "workbook_package_invalid" + return digest.hexdigest(), before.st_size + + +def _read_manifest(path: Path) -> Any: + flags = os.O_RDONLY | getattr(os, "O_BINARY", 0) | getattr(os, "O_CLOEXEC", 0) + flags |= getattr(os, "O_NOFOLLOW", 0) + descriptor = os.open(path, flags) + with os.fdopen(descriptor, "rb") as manifest_file: + before = os.fstat(manifest_file.fileno()) + if ( + not stat.S_ISREG(before.st_mode) + or before.st_size <= 0 + or before.st_size > _MAX_MANIFEST_BYTES + ): + raise ValueError("private manifest size is invalid") + raw = manifest_file.read(_MAX_MANIFEST_BYTES + 1) + after = os.fstat(manifest_file.fileno()) + if ( + len(raw) != before.st_size + or before.st_dev != after.st_dev + or before.st_ino != after.st_ino + or before.st_size != after.st_size + or before.st_mtime_ns != after.st_mtime_ns + ): + raise ValueError("private manifest changed while it was read") + return json.loads(raw.decode("utf-8"), object_pairs_hook=_unique_object) + + +def _unique_object(pairs: list[tuple[str, Any]]) -> dict[str, Any]: + result: dict[str, Any] = {} + for key, value in pairs: + if key in result: + raise ValueError("private manifest contains duplicate JSON keys") + result[key] = value + return result + + +def _workbook_size_class(size: int) -> str: + if size < 1_000_000: + return "small" + if size < 10_000_000: + return "medium" + return "large" + + +def _validate_provenance( + value: Any, + location: str, + issues: list[PrivateGoldenFixtureIssue], +) -> dict[str, str] | None: + if not isinstance(value, dict): + issues.append( + _issue( + f"{location}.provenance", + "workbook_provenance_required", + "provenance must be an object with hashed Excel-authoring evidence.", + ) + ) + return None + if set(value) != {"kind", "excelBuildHash", "sourceReceiptHash"}: + issues.append( + _issue( + f"{location}.provenance", + "workbook_provenance_fields_invalid", + "provenance must contain only kind, excelBuildHash, and sourceReceiptHash.", + ) + ) + return None + if value.get("kind") != "excel_authored": + issues.append( + _issue( + f"{location}.provenance.kind", + "workbook_provenance_kind_invalid", + "Private release workbooks must be Excel-authored.", + ) + ) + return None + for field in ("excelBuildHash", "sourceReceiptHash"): + field_value = value.get(field) + if not isinstance(field_value, str) or not _HASH_RE.fullmatch(field_value): + issues.append( + _issue( + f"{location}.provenance.{field}", + "workbook_provenance_hash_invalid", + "Provenance values must be lowercase sha256 digests.", + ) + ) + return None + return { + "kind": "excel_authored", + "excelBuildHash": str(value["excelBuildHash"]), + "sourceReceiptHash": str(value["sourceReceiptHash"]), + } + + +def _validate_rollback_expectations( + value: Any, + location: str, + issues: list[PrivateGoldenFixtureIssue], +) -> dict[str, Any] | None: + if not isinstance(value, dict): + issues.append( + _issue( + f"{location}.rollbackExpectations", + "rollback_expectations_required", + "rollbackExpectations must be an object.", + ) + ) + return None + if not isinstance(value.get("required"), bool): + issues.append( + _issue( + f"{location}.rollbackExpectations.required", + "rollback_required_flag_required", + "rollback required flag must be boolean.", + ) + ) + if not isinstance(value.get("dirtyRanges"), list): + issues.append( + _issue( + f"{location}.rollbackExpectations.dirtyRanges", + "rollback_dirty_ranges_required", + "rollback dirty ranges must be a list.", + ) + ) + return value + + +def _validate_audit_expectations( + value: Any, + location: str, + issues: list[PrivateGoldenFixtureIssue], +) -> dict[str, Any] | None: + if not isinstance(value, dict): + issues.append( + _issue( + f"{location}.auditExpectations", + "audit_expectations_required", + "auditExpectations must be an object.", + ) + ) + return None + required_evidence = value.get("requiredEvidence") + if not isinstance(required_evidence, list) or not required_evidence: + issues.append( + _issue( + f"{location}.auditExpectations.requiredEvidence", + "audit_evidence_required", + "audit required evidence must be a non-empty list.", + ) + ) + return value + + +def _validate_formula_facts( + value: Any, + location: str, + issues: list[PrivateGoldenFixtureIssue], +) -> list[dict[str, Any]] | None: + if not isinstance(value, list) or not value: + issues.append(_issue(f"{location}.expectedFormulaFacts", "field_required", "Required list field is missing.")) + return None + + formula_facts: list[dict[str, Any]] = [] + for fact_index, fact in enumerate(value): + fact_location = f"{location}.expectedFormulaFacts[{fact_index}]" + if not isinstance(fact, dict) or not fact: + issues.append( + _issue( + fact_location, + "formula_fact_must_be_object", + "Formula facts must be objects with hashed formula metadata.", + ) + ) + return None + if "formula" in fact: + issues.append( + _issue( + f"{fact_location}.formula", + "formula_fact_raw_formula_forbidden", + "Formula facts must not include raw formula text.", + ) + ) + return None + formula_hash = fact.get("formulaHash") + if not isinstance(formula_hash, str) or not _HASH_RE.match(formula_hash): + issues.append( + _issue( + f"{fact_location}.formulaHash", + "formula_fact_hash_required", + "Formula facts must include a sha256 formulaHash.", + ) + ) + return None + dependency_hashes = fact.get("dependencyHashes") + if dependency_hashes is not None and ( + not isinstance(dependency_hashes, list) + or any(not isinstance(dependency_hash, str) or not _HASH_RE.match(dependency_hash) for dependency_hash in dependency_hashes) + ): + issues.append( + _issue( + f"{fact_location}.dependencyHashes", + "formula_fact_dependency_hashes_invalid", + "Formula fact dependencyHashes must be sha256 hashes.", + ) + ) + return None + formula_facts.append(fact) + return formula_facts + + +def _required_string( + item: dict[str, Any], + field: str, + location: str, + issues: list[PrivateGoldenFixtureIssue], +) -> str | None: + value = item.get(field) + if not isinstance(value, str) or not value.strip(): + issues.append(_issue(f"{location}.{field}", "field_required", "Required string field is missing.")) + return None + return value.strip() + + +def _required_list( + item: dict[str, Any], + field: str, + location: str, + issues: list[PrivateGoldenFixtureIssue], +) -> list[Any] | None: + value = item.get(field) + if not isinstance(value, list) or not value: + issues.append(_issue(f"{location}.{field}", "field_required", "Required list field is missing.")) + return None + if any(not _valid_list_item(entry) for entry in value): + issues.append(_issue(f"{location}.{field}", "field_item_invalid", "List items must be strings or objects.")) + return None + return value + + +def _required_string_list( + item: dict[str, Any], + field: str, + location: str, + issues: list[PrivateGoldenFixtureIssue], +) -> list[str] | None: + value = _required_list(item, field, location, issues) + if value is None: + return None + if any( + not isinstance(entry, str) or not _SAFE_TOKEN_RE.fullmatch(entry.strip()) + for entry in value + ): + issues.append(_issue(f"{location}.{field}", "field_item_invalid", "List items must be non-empty strings.")) + return None + return [entry.strip() for entry in value] + + +def _valid_list_item(value: Any) -> bool: + if isinstance(value, str): + return bool(value.strip()) + return isinstance(value, dict) and bool(value) + + +def _invalid(*issues: PrivateGoldenFixtureIssue) -> PrivateGoldenFixtureValidation: + return PrivateGoldenFixtureValidation(status="invalid", issues=tuple(issues)) + + +def _issue(location: str, code: str, message: str) -> PrivateGoldenFixtureIssue: + return PrivateGoldenFixtureIssue(code=code, location=location, message=message) + + +def _is_relative_to(path: Path, root: Path) -> bool: + try: + path.relative_to(root) + except ValueError: + return False + return True + + +def main() -> int: + result = load_private_golden_fixture_manifest() + print(json.dumps(result.to_safe_dict(), indent=2, sort_keys=True)) + return 1 if result.status == "invalid" else 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/backend/benchmarks/suite.py b/backend/benchmarks/suite.py index 69a6589..02c415b 100644 --- a/backend/benchmarks/suite.py +++ b/backend/benchmarks/suite.py @@ -14,13 +14,11 @@ from __future__ import annotations import json -import math import re import time import uuid from dataclasses import asdict, dataclass, field -from pathlib import Path -from typing import Optional +from typing import Any, Optional import structlog @@ -316,8 +314,8 @@ async def run_suite( async def _run_case(self, case: BenchmarkCase, pipeline_fn) -> BenchmarkResult: """Execute a single benchmark case.""" start = time.time() - errors = [] - details = [] + errors: list[str] = [] + details: list[dict[str, Any]] = [] # Execute through the pipeline actual_operations = [] @@ -608,7 +606,8 @@ def benchmark_run_to_dict(run: BenchmarkRun) -> dict: def _op_cell(op: dict) -> str: - target = op.get("target") if isinstance(op.get("target"), dict) else {} + raw_target = op.get("target") + target = raw_target if isinstance(raw_target, dict) else {} return str(op.get("cell") or op.get("range") or target.get("range") or "") diff --git a/backend/benchmarks/v7_golden_cases.py b/backend/benchmarks/v7_golden_cases.py index 2a76850..554a5a8 100644 --- a/backend/benchmarks/v7_golden_cases.py +++ b/backend/benchmarks/v7_golden_cases.py @@ -11,6 +11,7 @@ import time from typing import Any +from backend.benchmarks.private_golden_fixtures import validate_private_golden_fixture_manifest from backend.benchmarks.workbook_fixtures import WORKBOOK_GOLDEN_FIXTURES, get_workbook_fixture_benchmark_runner from backend.models.operations import V7GoldenCase, V7GoldenCaseEvaluation from backend.storage import get_session_store @@ -28,12 +29,15 @@ def list_v7_golden_cases() -> list[V7GoldenCase]: workbookFilename=fixture.workbook_filename or "", expectedTruthTargets=fixture.expected_terms, expectedFormulaFacts=_expected_formula_facts(fixture.manifest_context), - expectedSimulationGates=_expected_simulation_gates(fixture.manifest_context), + expectedSimulationGates=fixture.expected_simulation_gates or _expected_simulation_gates(fixture.manifest_context), expectedDirtyRanges=fixture.expected_dirty_ranges, rollbackExpected=fixture.expect_rollback, repairOutcomes=_repair_outcomes(fixture.category), metadata={ "operationTypes": sorted({str(item.get("type") or "") for item in fixture.operations if isinstance(item, dict)}), + "operationCount": len(fixture.operations), + "fixtureRealism": fixture.manifest_context.get("realismMetadata") or {}, + "simulationAssertions": fixture.manifest_context.get("simulationAssertions") or {}, "privateFixture": False, "source": "public_manifest", }, @@ -42,9 +46,16 @@ def list_v7_golden_cases() -> list[V7GoldenCase]: return cases -def evaluate_v7_golden_cases(*, persist: bool = False, run_id: str | None = None) -> V7GoldenCaseEvaluation: +def evaluate_v7_golden_cases( + *, + persist: bool = False, + run_id: str | None = None, + require_private_fixtures: bool = False, +) -> V7GoldenCaseEvaluation: """Evaluate workbook-aware golden cases through the existing fixture runner.""" results = get_workbook_fixture_benchmark_runner().run_all() + private_validation = validate_private_golden_fixture_manifest() + private_summary = _private_fixture_validation_summary(private_validation.to_safe_dict()) total = len(results) passed = sum(1 for result in results if result.status == "passed") warning = sum(1 for result in results if result.status == "warning") @@ -55,6 +66,12 @@ def evaluate_v7_golden_cases(*, persist: bool = False, run_id: str | None = None for result in results for failure in result.failures ] + if require_private_fixtures and private_summary["status"] != "ready": + failed += 1 + failures.append( + "private_fixtures: private golden fixture manifest is required and not ready " + f"({private_summary['status']})." + ) status = "failed" if failed else "warning" if warning or total < 30 or score < 98 else "passed" evaluation = V7GoldenCaseEvaluation( status=status, # type: ignore[arg-type] @@ -72,6 +89,8 @@ def evaluate_v7_golden_cases(*, persist: bool = False, run_id: str | None = None "publicCaseCount": total, "privateCaseFolder": "backend/benchmarks/fixtures/private", "privateCaseFolderTracked": False, + "privateFixtureValidation": private_summary, + "requirePrivateFixtures": require_private_fixtures, "defaultReadyGate": status == "passed", "minimumScoreForReady": 98, "readinessClosureNotes": [] if status == "passed" else ["Golden cases must have zero warnings and score at least 98 for V7 readiness."], @@ -110,6 +129,9 @@ def _expected_formula_facts(manifest_context: dict[str, Any]) -> list[str]: def _expected_simulation_gates(manifest_context: dict[str, Any]) -> list[str]: + gates = manifest_context.get("expectedSimulationGates") if isinstance(manifest_context, dict) else None + if isinstance(gates, list): + return [str(gate) for gate in gates if str(gate)] gate = manifest_context.get("expectedSimulationGate") if isinstance(manifest_context, dict) else None return [str(gate)] if gate else ["passed", "checkpoint_required", "blocked"] @@ -122,11 +144,30 @@ def _repair_outcomes(category: str) -> list[str]: return ["not_required_or_optional"] -def main() -> int: +def _private_fixture_validation_summary(validation: dict[str, Any]) -> dict[str, Any]: + raw_issues = validation.get("issues") + issues: list[dict[str, Any]] = [item for item in raw_issues if isinstance(item, dict)] if isinstance(raw_issues, list) else [] + return { + "status": validation.get("status", "missing"), + "schemaVersion": validation.get("schemaVersion"), + "fixtureCount": int(validation.get("fixtureCount") or 0), + "issueCodes": sorted({str(item.get("code") or "") for item in issues if item.get("code")}), + } + + +def main(argv: list[str] | None = None) -> int: parser = argparse.ArgumentParser(description="Evaluate V7 golden workbook cases.") parser.add_argument("--persist", action="store_true", help="Persist the golden-case run in local storage.") - args = parser.parse_args() - evaluation = evaluate_v7_golden_cases(persist=args.persist) + parser.add_argument( + "--require-private-fixtures", + action="store_true", + help="Fail unless the gitignored private golden fixture manifest is ready.", + ) + args = parser.parse_args(argv) + evaluation = evaluate_v7_golden_cases( + persist=args.persist, + require_private_fixtures=args.require_private_fixtures, + ) print(json.dumps(evaluation.model_dump(mode="json", by_alias=True), indent=2)) return 1 if evaluation.status == "failed" else 0 diff --git a/backend/benchmarks/v7_readiness.py b/backend/benchmarks/v7_readiness.py index 153e2af..42da17f 100644 --- a/backend/benchmarks/v7_readiness.py +++ b/backend/benchmarks/v7_readiness.py @@ -8,14 +8,27 @@ import argparse import asyncio +import contextlib +import io import json +import sys import time +from datetime import date from pathlib import Path from typing import Any from backend.benchmarks.local_accuracy import run_local_accuracy from backend.benchmarks.local_replay import evaluate_local_replays -from backend.config import PLAN_ENGINE_VERSION +from backend.benchmarks.private_golden_fixtures import private_golden_fixture_policy_snapshot +from backend.config import ( + FORMULA_ENGINE, + HYPERFORMULA_DISTRIBUTED, + HYPERFORMULA_ENABLED, + HYPERFORMULA_LICENSE_APPROVED, + HYPERFORMULA_LICENSE_KEY, + PLAN_ENGINE_VERSION, +) +from backend.excel.hyperformula_license import is_commercial_hyperformula_key from backend.models.operations import V7DefaultReadinessReport, V7GoldenCaseEvaluation, V7PlanComparison, WriteCellOperationV2 from backend.services.formula_graph_v5 import get_formula_graph_v5_service from backend.services.operation_registry import get_operation_registry_v3 @@ -25,16 +38,25 @@ ROOT = Path(__file__).resolve().parents[2] - - -async def run_v7_readiness(*, persist: bool = False, replay_limit: int = 100) -> V7DefaultReadinessReport: +PARSER_COMPARISON_CORPUS = ROOT / "data" / "release-trust" / "parser-comparison-corpus-latest.json" + + +async def run_v7_readiness( + *, + persist: bool = False, + replay_limit: int = 100, + parser_corpus_path: Path | None = None, + replay_baseline_id: str | None = None, + require_production_replay: bool = False, + require_private_fixtures: bool = False, +) -> V7DefaultReadinessReport: gate_timing: dict[str, Any] = {} started = time.perf_counter() local_accuracy_start = time.perf_counter() local_accuracy = await run_local_accuracy(persist=False, replay_limit=replay_limit) gate_timing["localAccuracyMs"] = round((time.perf_counter() - local_accuracy_start) * 1000, 1) replay_start = time.perf_counter() - replay = evaluate_local_replays(limit=replay_limit, persist=False) + replay = evaluate_local_replays(limit=replay_limit, persist=False, baseline_id=replay_baseline_id) gate_timing["localReplayMs"] = round((time.perf_counter() - replay_start) * 1000, 1) formula_start = time.perf_counter() formula_readiness = _formula_readiness() @@ -44,11 +66,14 @@ async def run_v7_readiness(*, persist: bool = False, replay_limit: int = 100) -> storage_parity = _storage_parity() worker_hardening = _worker_hardening() office_mock_coverage = _office_mock_coverage() - dependency_risk = _dependency_risk() + dependency_risk = _dependency_risk(parser_corpus_path) taskpane_coverage = _taskpane_coverage() generated_types = _generated_types_freshness() golden_case_evaluation = _golden_case_summary(local_accuracy) - baseline_status = _baseline_status(replay) + private_fixture_policy = private_golden_fixture_policy_snapshot(require_private_fixtures=require_private_fixtures) + private_fixture_gate = _private_fixture_gate(private_fixture_policy) + baseline_metadata = _local_replay_baseline_metadata(replay.baseline_id) + baseline_status = _baseline_status(replay, baseline_metadata=baseline_metadata) history_summary = _history_summary() office_host_capability_snapshot = _office_host_capability_snapshot(registry_coverage) plan_comparison = _sample_plan_comparison(formula_readiness) @@ -67,11 +92,16 @@ async def run_v7_readiness(*, persist: bool = False, replay_limit: int = 100) -> "generatedTypes": {"status": generated_types["status"]}, "dependencyRisk": {"status": dependency_risk["status"]}, "taskpaneCoverage": {"status": taskpane_coverage["status"]}, + "privateGoldenFixtures": private_fixture_gate, } gate_timing["totalMs"] = round((time.perf_counter() - started) * 1000, 1) blockers = _blockers(gates, formula_readiness, rollback_readiness) + if require_production_replay and baseline_status.get("productionEvidence") is not True: + blockers.append("Production replay evidence is required for release/default-switch readiness.") + if private_fixture_gate["status"] == "failed": + blockers.append("Private golden fixture manifest is required for release/default-switch readiness.") warnings = _warnings(gates, replay, storage_parity, office_mock_coverage) - default_switch_blockers = _default_switch_blockers(blockers, storage_parity, office_mock_coverage, dependency_risk) + default_switch_blockers = _default_switch_blockers(blockers, storage_parity, office_mock_coverage, dependency_risk, baseline_status) score = _score(gates, warnings, blockers) status = "blocked" if blockers else "warning" if warnings else "ready" report = V7DefaultReadinessReport( @@ -111,6 +141,7 @@ async def run_v7_readiness(*, persist: bool = False, replay_limit: int = 100) -> "rawWorkbookContent": False, "rawPrompts": False, "defaultEnabledForOptIn": True, + "productionEvidence": bool(baseline_status.get("productionEvidence")), }, storageParity=storage_parity, workerHardening=worker_hardening, @@ -123,6 +154,9 @@ async def run_v7_readiness(*, persist: bool = False, replay_limit: int = 100) -> "generatedTypes": generated_types, "defaultSwitchBlockers": default_switch_blockers, "readinessClosureNotes": _readiness_closure_notes(gates, golden_case_evaluation), + "requireProductionReplay": require_production_replay, + "requirePrivateFixtures": require_private_fixtures, + "privateGoldenFixtures": private_fixture_policy, }, ) if persist: @@ -206,66 +240,46 @@ def _rollback_readiness() -> dict[str, Any]: text += rollback_path.read_text(encoding="utf-8") if rollback_path.exists() else "" has_range_snapshot = "captureRollbackSnapshot" in text and "restoreOperations" in text has_manual_warning = "unsupportedRollbackWarnings" in text - has_proof_class = "proofClass" in text and "values_formulas_formats" in text and "comment_restore" in text + has_proof_class = "proofClass" in text and "values_formulas_formats" in text and "comment_delete_restore" in text + has_table_restore = all( + signal in text + for signal in ["table_delete_restore", "table_metadata_restore", "table_recreate_restore"] + ) + has_chart_restore = all( + signal in text + for signal in ["chart_delete_restore", "chart_metadata_restore", "chart_recreate_restore"] + ) + has_comment_restore = all( + signal in text + for signal in ["comment_delete_restore", "comment_content_restore", "comment_recreate_restore"] + ) return { - "status": "passed" if has_range_snapshot and has_manual_warning and has_proof_class else "warning", + "status": "passed" if has_range_snapshot and has_manual_warning and has_proof_class and has_table_restore and has_chart_restore and has_comment_restore else "warning", "rangeSnapshots": has_range_snapshot, "manualObjectRollbackWarnings": has_manual_warning, "proofClasses": has_proof_class, - "objectRollback": "manual", + "automaticObjectRollback": { + "charts": has_chart_restore, + "tables": has_table_restore, + "comments": has_comment_restore, + "pivots": False, + }, + "objectRollback": "chart_table_comment_automatic_pivot_manual", } def _storage_parity() -> dict[str, Any]: - protocol = (ROOT / "backend" / "storage" / "protocol.py").read_text(encoding="utf-8") - postgres = (ROOT / "backend" / "storage" / "postgres.py").read_text(encoding="utf-8") - sqlite = (ROOT / "backend" / "session_store.py").read_text(encoding="utf-8") - methods = [ - "save_operation_plan", - "get_operation_plan", - "update_operation_plan_status", - "record_snapshot", - "get_snapshot", - "record_audit_event", - "export_audit_csv", - "record_apply_attempt", - "get_latest_apply_attempt", - "complete_apply_attempt", - "record_apply_result", - "get_timeline", - "delete_session", - "save_document", - "get_document", - "save_citations", - "get_citations", - "save_benchmark_run", - "save_workbook_map", - "get_workbook_map", - "save_source_evidence_graph", - "save_operation_simulation", - "save_workbook_invariant_run", - "save_agent_run", - "get_agent_run", - "save_v7_workbook_review", - "get_v7_workbook_review", - "save_v7_default_readiness_report", - "get_latest_v7_default_readiness_report", - "list_v7_default_readiness_reports", - "save_local_replay_artifact", - "list_local_replay_artifacts", - "clear_local_replay_artifacts", - "save_local_replay_baseline", - "list_local_replay_baselines", - "save_agent_run_artifact", - "list_agent_run_artifacts", - "save_rollback_snapshot_v2", - ] - missing = [ - method - for method in methods - if method not in protocol or method not in postgres or method not in sqlite - ] - return {"status": "passed" if not missing else "warning", "checkedMethods": methods, "missingMethods": missing} + from backend.storage import postgres_protocol_gap_report + from backend.storage.postgres import PostgresSessionStore + + report = postgres_protocol_gap_report(PostgresSessionStore) + return { + "status": "passed" if report["ready"] else "warning", + "checkedMethods": report["required_methods"], + "missingMethods": report["missing_methods"], + "signatureMismatches": report["signature_mismatches"], + "blockingCategories": report["blocking_categories"], + } def _worker_hardening() -> dict[str, Any]: @@ -302,8 +316,26 @@ def _office_mock_coverage() -> dict[str, Any]: "named_ranges", "table_unsupported", "chart_unsupported", + "chart_not_found", + "delete_chart", + "chart type aliases", + "named chart rollback proof", + "rolls back named chart creation", + "restores chart metadata", + "recreates deleted charts", + "rolls back created tables", + "restores table metadata", + "recreates deleted tables", + "updates and deletes comments", + "rolls back created comments", + "restores comment content", + "recreates deleted comments", "comment_unsupported", + "clear_data_validation", + "errorAlert", "host_capabilities", + "pivotTables", + "Pivot table mutations are gated", "V7ReviewSummary", "unsupported_host_feature", "create_sparkline", @@ -321,39 +353,262 @@ def _office_mock_coverage() -> dict[str, Any]: } -def _dependency_risk() -> dict[str, Any]: +def _dependency_risk(corpus_path: Path | None = None) -> dict[str, Any]: worker_pkg = _read_json(ROOT / "workbook-worker" / "package.json") + worker_lock = _read_json(ROOT / "workbook-worker" / "package-lock.json") plugin_pkg = _read_json(ROOT / "plugin" / "package.json") notices = (ROOT / "docs" / "THIRD_PARTY_NOTICES.md").read_text(encoding="utf-8") if (ROOT / "docs" / "THIRD_PARTY_NOTICES.md").exists() else "" allowlist = _read_json(ROOT / "security" / "dependency-risk-allowlist.json") + worker = (ROOT / "workbook-worker" / "src" / "cli.mjs").read_text(encoding="utf-8") if (ROOT / "workbook-worker" / "src" / "cli.mjs").exists() else "" + parser = (ROOT / "workbook-worker" / "src" / "parsers" / "workbook-parser.mjs").read_text(encoding="utf-8") if (ROOT / "workbook-worker" / "src" / "parsers" / "workbook-parser.mjs").exists() else "" worker_deps = worker_pkg.get("dependencies") or {} sheetjs_present = "xlsx" in worker_deps + hyperformula_license = _hyperformula_license_risk_status(worker_deps) + corpus = _parser_comparison_corpus_status(corpus_path or PARSER_COMPARISON_CORPUS) + xlsx_spec = str(worker_deps.get("xlsx") or "") + locked_xlsx = ((worker_lock.get("packages") or {}).get("node_modules/xlsx") or {}) + xlsx_remediated = ( + str(locked_xlsx.get("version") or "") == "0.20.3" + and "https://cdn.sheetjs.com/xlsx-0.20.3/" in str(locked_xlsx.get("resolved") or xlsx_spec) + ) + sheetjs_risk = _sheetjs_xlsx_risk_status( + allowlist, + worker, + parser, + corpus, + remediated=xlsx_remediated, + ) + sheetjs_risk_resolved = sheetjs_risk["status"] in {"accepted", "remediated"} tracked = { "sheetjsIsolated": sheetjs_present and "SheetJS" in notices, "exceljs": "exceljs" in worker_deps and "ExcelJS" in notices, - "hyperformula": "hyperformula" in worker_deps and "HyperFormula" in notices, + "hyperformula": "hyperformula" not in worker_deps or "HyperFormula" in notices, + "hyperformulaLicenseApproved": hyperformula_license["approvalEvidence"], + "hyperformulaLicenseKey": hyperformula_license["keyEvidence"], "pluginLockfile": (ROOT / "plugin" / "package-lock.json").exists(), "workerLockfile": (ROOT / "workbook-worker" / "package-lock.json").exists(), "vitePinned": bool((plugin_pkg.get("devDependencies") or {}).get("vite")), - "riskAllowlist": bool(allowlist.get("acceptedRisks")), + "riskAllowlist": isinstance(allowlist.get("acceptedRisks"), list), + "sheetjsAcceptedRisk": sheetjs_risk_resolved, + "sheetjsParserComparisonAdapter": sheetjs_risk["comparisonAdapterEvidence"], + "sheetjsParserComparisonCorpus": sheetjs_risk["comparisonCorpusEvidence"], + "sheetjsWorkerHardening": sheetjs_risk["workerHardeningEvidence"], } unresolved = [key for key, value in tracked.items() if not value] - if sheetjs_present: - unresolved.append("sheetjs_xlsx_high_severity_advisory_no_npm_fix") + if sheetjs_present and not sheetjs_risk_resolved: + unresolved.append(f"sheetjs_xlsx_high_severity_advisory_{sheetjs_risk['status']}") + if hyperformula_license["status"] != "approved": + unresolved.extend(hyperformula_license["unresolved"]) accepted_risks = [ - "SheetJS remains isolated behind worker size, extension, and OOXML checks.", - "HyperFormula license approval must be confirmed for production deployments.", + { + "package": "xlsx", + "status": sheetjs_risk["status"], + "expiresOn": sheetjs_risk["expiresOn"], + "daysUntilExpiry": sheetjs_risk["daysUntilExpiry"], + "releaseBlocker": sheetjs_risk["releaseBlocker"], + "mitigations": sheetjs_risk["mitigations"], + }, + { + "package": "hyperformula", + "status": hyperformula_license["status"], + "releaseBlocker": hyperformula_license["releaseBlocker"], + "approvalConfigured": hyperformula_license["approvalEvidence"], + "licenseKeyPresent": hyperformula_license["keyEvidence"], + "mitigations": [ + "HyperFormula license approval must be confirmed before production deployments.", + "HYPERFORMULA_LICENSE_KEY must be supplied for production/commercial deployments.", + ], + }, ] return { "status": "passed" if not unresolved else "warning", "tracked": tracked, "unresolved": unresolved, "acceptedRisks": accepted_risks, + "parserComparisonCorpus": corpus, + "sheetjsAdvisoryStatus": sheetjs_risk["status"] if sheetjs_present else "not_present", + "hyperformulaLicenseStatus": hyperformula_license, "auditPolicy": "verify-full-stack runs npm critical audit checks, pip check, and scripts/check_dependency_risks.py", "allowlist": allowlist, } +def _hyperformula_license_risk_status(worker_deps: dict[str, Any]) -> dict[str, Any]: + present = "hyperformula" in worker_deps + primary_configured = FORMULA_ENGINE == "hyperformula" and HYPERFORMULA_ENABLED + distributed = bool(HYPERFORMULA_DISTRIBUTED) + required = present or primary_configured or distributed + approval_evidence = (not required) or bool(HYPERFORMULA_LICENSE_APPROVED) + key_evidence = (not required) or is_commercial_hyperformula_key(HYPERFORMULA_LICENSE_KEY) + unresolved: list[str] = [] + if required and not (present and distributed and primary_configured): + unresolved.append("hyperformula_runtime_distribution_inconsistent") + if required and not approval_evidence: + unresolved.append("hyperformula_license_approval_missing") + if required and not key_evidence: + unresolved.append("hyperformula_commercial_license_key_missing_or_invalid") + status = "approved" if not unresolved else "blocked_unapproved" + if not required: + status = "not_required" + return { + "status": status, + "required": required, + "formulaEngine": FORMULA_ENGINE, + "hyperformulaEnabled": HYPERFORMULA_ENABLED, + "hyperformulaDistributed": distributed, + "productionDependencyPresent": present, + "approvalEvidence": approval_evidence, + "keyEvidence": key_evidence, + "releaseBlocker": bool(unresolved), + "unresolved": unresolved, + } + + +def _parser_comparison_corpus_status(path: Path) -> dict[str, Any]: + if not path.exists(): + return { + "status": "missing", + "path": _display_path(path), + "fixtureCount": 0, + "materialMismatchCount": 0, + "matched": 0, + "mismatched": 0, + "failed": 0, + } + try: + envelope = json.loads(path.read_text(encoding="utf-8")) + except (OSError, json.JSONDecodeError) as exc: + return { + "status": "invalid", + "path": _display_path(path), + "error": str(exc), + "fixtureCount": 0, + "materialMismatchCount": 0, + "matched": 0, + "mismatched": 0, + "failed": 1, + } + raw_result = envelope.get("result") if isinstance(envelope, dict) else None + result: dict[str, Any] = raw_result if isinstance(raw_result, dict) else {} + raw_summary = result.get("summary") + summary: dict[str, Any] = raw_summary if isinstance(raw_summary, dict) else {} + return { + "status": str(result.get("status") or "unknown"), + "path": _display_path(path), + "fixtureCount": int(result.get("fixtureCount") or 0), + "materialMismatchCount": int(summary.get("materialMismatchCount") or 0), + "matched": int(summary.get("matched") or 0), + "mismatched": int(summary.get("mismatched") or 0), + "failed": int(summary.get("failed") or 0), + } + + +def _sheetjs_xlsx_risk_status( + allowlist: dict[str, Any], + worker: str, + parser: str, + corpus: dict[str, Any], + today: date | None = None, + remediated: bool = False, +) -> dict[str, Any]: + today = today or date.today() + entry = next( + ( + item + for item in allowlist.get("acceptedRisks") or [] + if isinstance(item, dict) + and item.get("ecosystem") == "npm" + and item.get("package") == "xlsx" + and "workbook-worker/package-lock.json" in (item.get("paths") or []) + ), + {}, + ) + mitigations = [str(item) for item in entry.get("mitigations") or []] + advisory_ids = {str(item).upper() for item in entry.get("advisoryIds") or []} + expires_on_text = str(entry.get("expiresOn") or "") + release_blocker = str(entry.get("releaseBlocker") or "expires_on_or_before_release") + try: + expires_on = date.fromisoformat(expires_on_text) + except ValueError: + expires_on = today + days_until_expiry = (expires_on - today).days + worker_hardening = all( + signal in worker + for signal in [ + "MAX_PAYLOAD_BYTES", + "ALLOWED_EXTENSIONS", + "assertOoxmlMagic", + "MAX_OOXML_FILE_COUNT", + "ZIP_ENCRYPTED_FLAG", + "ZIP_DATA_DESCRIPTOR_FLAG", + ] + ) + comparison_adapter = all( + signal in parser + for signal in [ + "ExcelJS", + "parserComparisonSnapshot", + "parserComparisonCorpus", + "comparisonOnly", + "replacementReady: false", + ] + ) + comparison_corpus = ( + corpus.get("status") == "matched" + and int(corpus.get("fixtureCount") or 0) > 0 + and int(corpus.get("materialMismatchCount") or 0) == 0 + and int(corpus.get("failed") or 0) == 0 + ) + if remediated: + return { + "status": "remediated", + "expiresOn": "", + "daysUntilExpiry": None, + "releaseBlocker": False, + "mitigations": [ + "SheetJS is pinned to the official 0.20.3 CDN tarball with lockfile integrity.", + "npm audit reports no advisory for the resolved worker dependency tree.", + ], + "workerHardeningEvidence": worker_hardening, + "comparisonAdapterEvidence": comparison_adapter, + "comparisonCorpusEvidence": comparison_corpus, + "comparisonCorpus": corpus, + "advisoryIds": [], + } + required_ids = {"GHSA-4R6H-8V6P-XVW6", "GHSA-5PGG-2G8V-P4X9"} + has_policy = ( + bool(entry.get("owner")) + and str(entry.get("status") or "").startswith("accepted") + and advisory_ids == required_ids + and bool(mitigations) + and days_until_expiry > 0 + and release_blocker == "expires_on_or_before_release" + ) + status = "accepted" if has_policy and worker_hardening and comparison_adapter and comparison_corpus else "unaccepted" + if days_until_expiry <= 0: + status = "expired" + elif not worker_hardening: + status = "missing_worker_hardening" + elif not comparison_adapter: + status = "missing_parser_comparison_adapter" + elif not comparison_corpus: + status = "missing_or_mismatched_parser_comparison_corpus" + elif not has_policy: + status = "missing_or_invalid_allowlist" + return { + "status": status, + "expiresOn": expires_on_text, + "daysUntilExpiry": days_until_expiry, + "releaseBlocker": release_blocker, + "mitigations": mitigations, + "workerHardeningEvidence": worker_hardening, + "comparisonAdapterEvidence": comparison_adapter, + "comparisonCorpusEvidence": comparison_corpus, + "comparisonCorpus": corpus, + "advisoryIds": sorted(advisory_ids), + } + + def _taskpane_coverage() -> dict[str, Any]: package = _read_json(ROOT / "plugin" / "package.json") scripts = package.get("scripts") or {} @@ -431,9 +686,40 @@ def _golden_case_summary(local_accuracy: dict[str, Any]) -> V7GoldenCaseEvaluati ) -def _baseline_status(replay: Any) -> dict[str, Any]: +def _private_fixture_gate(snapshot: dict[str, Any]) -> dict[str, Any]: + required = snapshot.get("required") is True + ready = snapshot.get("ready") is True + return { + "status": "passed" if ready or not required else "failed", + "privateStatus": str(snapshot.get("status") or "missing"), + "ready": ready, + "required": required, + "fixtureCount": int(snapshot.get("fixtureCount") or 0), + "issueCodes": list(snapshot.get("issueCodes") or []), + } + + +def _local_replay_baseline_metadata(baseline_id: str | None) -> dict[str, Any]: + if not baseline_id: + return {} + try: + baselines = get_session_store().list_local_replay_baselines(100) + except Exception: + return {} + for baseline in baselines: + if str(baseline.get("baselineId") or baseline.get("baseline_id") or "") == str(baseline_id): + metadata = baseline.get("metadata") + return metadata if isinstance(metadata, dict) else {} + return {} + + +def _baseline_status(replay: Any, baseline_metadata: dict[str, Any] | None = None) -> dict[str, Any]: comparison = replay.baseline_comparison or {} drift_class = replay.deterministic_drift_class + metadata = baseline_metadata or {} + evidence_class = str(metadata.get("evidenceClass") or "unspecified") + production_evidence = bool(metadata.get("productionEvidence")) + evidence_tier = _replay_evidence_tier(evidence_class, production_evidence) if drift_class == "unbaselined": status = "warning" elif drift_class == "gate_drift": @@ -447,9 +733,25 @@ def _baseline_status(replay: Any) -> dict[str, Any]: "baselineId": replay.baseline_id or "", "deterministicDriftClass": drift_class, "comparison": comparison, + "productionEvidence": production_evidence, + "evidenceClass": evidence_class, + "evidenceTier": evidence_tier, } +def _replay_evidence_tier(evidence_class: str, production_evidence: bool) -> str: + normalized = evidence_class.lower().replace("-", "_").strip() + if production_evidence: + return "production_anonymized" + if normalized == "deterministic_plumbing_seed": + return "deterministic_seed" + if "synthetic" in normalized: + return "synthetic_contract" + if "corpus" in normalized or "aggregate" in normalized: + return "sanitized_non_production_corpus" + return "non_production" + + def _history_summary() -> dict[str, Any]: try: reports = get_session_store().list_v7_default_readiness_reports(20) @@ -486,6 +788,13 @@ def _read_json(path: Path) -> dict[str, Any]: return {} +def _display_path(path: Path) -> str: + try: + return str(path.relative_to(ROOT)) + except ValueError: + return str(path) + + def _readiness_closure_notes(gates: dict[str, Any], golden_case_evaluation: V7GoldenCaseEvaluation) -> list[str]: notes: list[str] = [] if gates.get("goldenCases", {}).get("status") != "passed": @@ -539,17 +848,32 @@ def _default_switch_blockers( storage_parity: dict[str, Any], office_mock_coverage: dict[str, Any], dependency_risk: dict[str, Any], + baseline_status: dict[str, Any] | None = None, ) -> list[str]: default_blockers = list(blockers) + baseline_status = baseline_status or {} if storage_parity.get("status") != "passed": default_blockers.append("Postgres/SQLite storage parity has V7 artifact gaps.") if office_mock_coverage.get("status") != "passed": default_blockers.append("Office mock coverage is missing required V7 adapter signals.") if dependency_risk.get("status") != "passed": default_blockers.append("Dependency risk posture has unresolved tracked items.") + if baseline_status.get("status") == "passed" and baseline_status.get("productionEvidence") is False: + default_blockers.append(_non_production_replay_default_blocker(baseline_status)) return sorted(set(default_blockers)) +def _non_production_replay_default_blocker(baseline_status: dict[str, Any]) -> str: + tier = str(baseline_status.get("evidenceTier") or "non_production") + if tier == "deterministic_seed": + return "Replay baseline is deterministic plumbing evidence; capture production-like anonymized local replays before switching V7 on by default." + if tier == "synthetic_contract": + return "Replay baseline is synthetic contract evidence; capture production-like anonymized local replays before switching V7 on by default." + if tier == "sanitized_non_production_corpus": + return "Replay baseline is sanitized non-production corpus evidence; capture production-like anonymized local replays before switching V7 on by default." + return "Replay baseline is non-production evidence; capture production-like anonymized local replays before switching V7 on by default." + + def _warnings( gates: dict[str, Any], replay: Any, @@ -577,13 +901,47 @@ def _score(gates: dict[str, Any], warnings: list[str], blockers: list[str]) -> f return round(max(0.0, min(1.0, base - penalty)), 3) -def main() -> int: +def main(argv: list[str] | None = None) -> int: parser = argparse.ArgumentParser(description="Run the V7 default-prep readiness gate.") parser.add_argument("--persist", action="store_true", help="Persist the readiness report in local storage.") parser.add_argument("--replay-limit", type=int, default=100, help="Maximum local replay artifacts to evaluate.") - args = parser.parse_args() - report = asyncio.run(run_v7_readiness(persist=args.persist, replay_limit=args.replay_limit)) + parser.add_argument( + "--parser-comparison-corpus", + type=Path, + default=PARSER_COMPARISON_CORPUS, + help="Parser comparison corpus JSON artifact to use for dependency-risk readiness.", + ) + parser.add_argument("--fail-on-warning", action="store_true", help="Exit non-zero when the aggregate readiness status is warning.") + parser.add_argument("--replay-baseline-id", default=None, help="Optional pinned replay baseline id to compare against.") + parser.add_argument( + "--require-production-replay", + action="store_true", + help="Block readiness unless the pinned replay baseline is marked as production/anonymized corpus evidence.", + ) + parser.add_argument( + "--require-private-fixtures", + action="store_true", + help="Block readiness unless the private golden fixture manifest is ready.", + ) + args = parser.parse_args(argv) + stdout_buffer = io.StringIO() + with contextlib.redirect_stdout(stdout_buffer): + report = asyncio.run( + run_v7_readiness( + persist=args.persist, + replay_limit=args.replay_limit, + parser_corpus_path=args.parser_comparison_corpus, + replay_baseline_id=args.replay_baseline_id, + require_production_replay=args.require_production_replay, + require_private_fixtures=args.require_private_fixtures, + ) + ) + incidental_stdout = stdout_buffer.getvalue() + if incidental_stdout: + print(incidental_stdout, end="", file=sys.stderr) print(json.dumps(report.model_dump(mode="json", by_alias=True), indent=2)) + if args.fail_on_warning and report.status == "warning": + return 1 return 1 if report.status == "blocked" else 0 diff --git a/backend/benchmarks/workbook_fixtures.py b/backend/benchmarks/workbook_fixtures.py index 0e487f9..f1efa46 100644 --- a/backend/benchmarks/workbook_fixtures.py +++ b/backend/benchmarks/workbook_fixtures.py @@ -27,6 +27,8 @@ class WorkbookGoldenFixture: operations: list[dict[str, Any]] expected_terms: list[str] = field(default_factory=list) expected_dirty_ranges: list[str] = field(default_factory=list) + expected_simulation_gates: list[str] = field(default_factory=list) + expected_operation_count: int | None = None expect_rollback: bool = False workbook_filename: str | None = None manifest_context: dict[str, Any] = field(default_factory=dict) @@ -158,6 +160,12 @@ def run_fixture(self, fixture: WorkbookGoldenFixture) -> WorkbookBenchmarkFixtur "sheetjs_ooxml": (workbook_summary or {}).get("ooxml") or {}, "semantic_diagnostics": semantic_diagnostics, "fixture_warning_category": _fixture_warning_category(failures), + "fixture_operation_count": len(fixture.operations), + "expected_operation_count": fixture.expected_operation_count, + "expected_simulation_gates": fixture.expected_simulation_gates, + "simulation_gate_report_gate": getattr(simulation.simulation_gate_report, "gate", None), + "fixture_realism_metadata": fixture.manifest_context.get("realismMetadata") or {}, + "simulation_assertions": fixture.manifest_context.get("simulationAssertions") or {}, }, ) @@ -207,6 +215,12 @@ def _load_workbook_fixtures_from_manifest( expected_objects = item.get("expectedObjects") if isinstance(item.get("expectedObjects"), dict) else {} active_sheet = str(item.get("activeSheet") or "") selected_range = str(item.get("selectedRange") or "") + operations = list(item.get("operations") or []) + expected_simulation_gates = _manifest_string_list(item.get("expectedSimulationGates")) + if not expected_simulation_gates: + expected_simulation_gates = _manifest_string_list(item.get("expectedSimulationGate")) + simulation_assertions = _manifest_dict(item.get("simulationAssertions")) + realism_metadata = _manifest_dict(item.get("realismMetadata")) fixtures.append( WorkbookGoldenFixture( fixture_id=str(item.get("fixtureId") or item.get("id") or ""), @@ -219,9 +233,11 @@ def _load_workbook_fixtures_from_manifest( "selected_range": selected_range, "objects": expected_objects, }, - operations=list(item.get("operations") or []), + operations=operations, expected_terms=[str(term) for term in item.get("expectedTerms") or []], expected_dirty_ranges=[str(range_ref) for range_ref in item.get("expectedDirtyRanges") or []], + expected_simulation_gates=expected_simulation_gates, + expected_operation_count=_optional_int(simulation_assertions.get("operationCount")), expect_rollback=bool(item.get("expectRollback")), manifest_context={ "active_sheet": active_sheet, @@ -229,6 +245,9 @@ def _load_workbook_fixtures_from_manifest( "name": str(item.get("name") or item.get("fixtureId") or ""), "category": str(item.get("category") or "workbook_accuracy"), "objects": expected_objects, + "expectedSimulationGates": expected_simulation_gates, + "simulationAssertions": simulation_assertions, + "realismMetadata": realism_metadata, "fixture_dir": str(fixture_dir), "private": private, }, @@ -237,6 +256,26 @@ def _load_workbook_fixtures_from_manifest( return tuple(fixture for fixture in fixtures if fixture.fixture_id and fixture.operations) +def _manifest_string_list(value: Any) -> list[str]: + if isinstance(value, list): + return [str(item).strip() for item in value if str(item).strip()] + if isinstance(value, str) and value.strip(): + return [value.strip()] + return [] + + +def _manifest_dict(value: Any) -> dict[str, Any]: + return dict(value) if isinstance(value, dict) else {} + + +def _optional_int(value: Any) -> int | None: + if isinstance(value, bool): + return None + if isinstance(value, int): + return value + return None + + WORKBOOK_GOLDEN_FIXTURES: tuple[WorkbookGoldenFixture, ...] = ( (_load_workbook_fixtures_from_manifest() or SYNTHETIC_WORKBOOK_GOLDEN_FIXTURES) + load_private_workbook_fixtures() @@ -346,7 +385,7 @@ def _objects_from_fixture_and_summary(fixture: WorkbookGoldenFixture, workbook_s "comments": [], **(fixture.manifest_context.get("objects") or fixture.workbook_context.get("objects") or {}), } - comments = [] + comments: list[Any] = [] for sheet in workbook_summary.get("sheets") or []: comments.extend(sheet.get("comments") or []) if comments: @@ -401,10 +440,22 @@ def _fixture_failures( for term in fixture.expected_terms: if term in diagnostics["missingTerms"]: failures.append(f"Missing expected semantic term: {term}") - dirty = {item.range for item in simulation.expected_dirty_ranges} for range_ref in fixture.expected_dirty_ranges: - if range_ref not in dirty: + if not _dirty_range_matches(str(range_ref), simulation.expected_dirty_ranges): failures.append(f"Missing expected dirty range: {range_ref}") + unexpected_dirty = _unexpected_dirty_ranges(fixture.expected_dirty_ranges, simulation.expected_dirty_ranges) + if unexpected_dirty: + failures.append(f"Unexpected dirty range(s): {', '.join(unexpected_dirty)}") + if fixture.expected_simulation_gates: + gates = set(fixture.expected_simulation_gates) + simulation_gate = str(getattr(simulation, "simulation_gate", "") or "") + report_gate = str(getattr(getattr(simulation, "simulation_gate_report", None), "gate", "") or "") + if simulation_gate not in gates: + failures.append(f"Simulation gate {simulation_gate or 'missing'} was not in expected gates: {sorted(gates)}") + if report_gate and report_gate not in gates: + failures.append(f"Simulation gate report {report_gate} was not in expected gates: {sorted(gates)}") + if fixture.expected_operation_count is not None and len(fixture.operations) != fixture.expected_operation_count: + failures.append(f"Expected {fixture.expected_operation_count} operation(s), found {len(fixture.operations)}") if fixture.expect_rollback and not simulation.rollback_required: failures.append("Rollback was expected but simulation did not require it") if simulation.schema_version != "2026-05-operation-simulation-v4": @@ -423,11 +474,44 @@ def _fixture_warning_category(failures: list[str]) -> str: return "semantic_terms" if any("dirty range" in failure for failure in failures): return "simulation_dirty_ranges" + if any("Simulation gate" in failure for failure in failures): + return "simulation_gate" if any("Rollback" in failure for failure in failures): return "rollback" return "contract" +def _dirty_range_matches(expected: str, dirty_ranges: list[Any]) -> bool: + return any(expected in _dirty_range_variants(item) for item in dirty_ranges) + + +def _unexpected_dirty_ranges(expected_ranges: list[str], dirty_ranges: list[Any]) -> list[str]: + expected = {str(item).strip() for item in expected_ranges if str(item).strip()} + if not expected: + return [] + unexpected = [ + _dirty_range_label(item) + for item in dirty_ranges + if _dirty_range_variants(item).isdisjoint(expected) + ] + return sorted(set(filter(None, unexpected))) + + +def _dirty_range_variants(item: Any) -> set[str]: + sheet = str(getattr(item, "sheet", "") or "") + range_ref = str(getattr(item, "range", "") or "") + variants = {range_ref} + if sheet and range_ref: + variants.add(f"{sheet}!{range_ref}") + return {variant for variant in variants if variant} + + +def _dirty_range_label(item: Any) -> str: + sheet = str(getattr(item, "sheet", "") or "") + range_ref = str(getattr(item, "range", "") or "") + return f"{sheet}!{range_ref}".strip("!") if range_ref else sheet + + def _semantic_term_diagnostics(fixture: WorkbookGoldenFixture, truth: Any) -> dict[str, Any]: candidates = _semantic_candidates(fixture, truth) matched_terms: list[dict[str, Any]] = [] diff --git a/backend/config.py b/backend/config.py index 2508db5..04cc725 100644 --- a/backend/config.py +++ b/backend/config.py @@ -1,5 +1,9 @@ """Application configuration — loaded from environment variables.""" +import base64 +import binascii +import ipaddress +import json import os from pathlib import Path from dotenv import load_dotenv @@ -7,14 +11,14 @@ load_dotenv() # ── Version ────────────────────────────────────────────────────────────── -VERSION = "0.4.0" +VERSION = "0.5.0" # ── Paths ──────────────────────────────────────────────────────────────── BASE_DIR = Path(__file__).resolve().parent TEMPLATES_DIR = BASE_DIR / "templates" / "finance" # ── Server ─────────────────────────────────────────────────────────────── -HOST = os.getenv("HOST", "0.0.0.0") +HOST = os.getenv("HOST", "127.0.0.1") PORT = int(os.getenv("PORT", "8000")) APP_ENV = os.getenv("APP_ENV", os.getenv("CONFIG_PROFILE", "dev")).lower() CONFIG_PROFILE = os.getenv("CONFIG_PROFILE", APP_ENV).lower() @@ -31,9 +35,16 @@ LMSTUDIO_MODEL = os.getenv("LMSTUDIO_MODEL", "") # Anthropic (cloud) +ANTHROPIC_RETIRED_MODELS = { + "claude-3-5-haiku-20241022": "claude-haiku-4-5-20251001", + "claude-sonnet-4-20250514": "claude-sonnet-4-6", + "claude-opus-4-20250514": "claude-opus-4-8", +} ANTHROPIC_API_KEY = os.getenv("ANTHROPIC_API_KEY", "") -ANTHROPIC_MODEL_MEDIUM = os.getenv("ANTHROPIC_MODEL_MEDIUM", "claude-3-5-haiku-20241022") -ANTHROPIC_MODEL_COMPLEX = os.getenv("ANTHROPIC_MODEL_COMPLEX", "claude-sonnet-4-20250514") +ANTHROPIC_MODEL_MEDIUM_RAW = os.getenv("ANTHROPIC_MODEL_MEDIUM", "claude-haiku-4-5-20251001") +ANTHROPIC_MODEL_COMPLEX_RAW = os.getenv("ANTHROPIC_MODEL_COMPLEX", "claude-sonnet-4-6") +ANTHROPIC_MODEL_MEDIUM = ANTHROPIC_RETIRED_MODELS.get(ANTHROPIC_MODEL_MEDIUM_RAW, ANTHROPIC_MODEL_MEDIUM_RAW) +ANTHROPIC_MODEL_COMPLEX = ANTHROPIC_RETIRED_MODELS.get(ANTHROPIC_MODEL_COMPLEX_RAW, ANTHROPIC_MODEL_COMPLEX_RAW) # OpenAI (fallback) OPENAI_API_KEY = os.getenv("OPENAI_API_KEY", "") @@ -46,17 +57,17 @@ IB_FORMAT = { "font_name": "Calibri", "font_size": 10, - "input_font_color": "0000FF", # Blue for hard-coded inputs - "formula_font_color": "000000", # Black for formulas - "link_font_color": "008000", # Green for cross-sheet links + "input_font_color": "0000FF", # Blue for hard-coded inputs + "formula_font_color": "000000", # Black for formulas + "link_font_color": "008000", # Green for cross-sheet links "header_bold": True, - "header_bg_color": "D9E2F3", # Light blue header background - "negative_format": "#,##0;(#,##0)", # Parentheses for negatives + "header_bg_color": "D9E2F3", # Light blue header background + "negative_format": "#,##0;(#,##0)", # Parentheses for negatives "currency_format": "$#,##0", "pct_format": "0.0%", "multiple_format": "0.0x", - "subtotal_border": "thin", # Bottom border for subtotals - "total_border": "double", # Double bottom border for totals + "subtotal_border": "thin", # Bottom border for subtotals + "total_border": "double", # Double bottom border for totals "thousands_separator": True, } @@ -68,7 +79,23 @@ # When true, the static API_KEY auto-promotes the caller to admin. This is # a developer-loop convenience — production profiles must have it off so the # enterprise API-key + RBAC pipeline is the only admin path. -STATIC_KEY_ADMIN_ENABLED = os.getenv("STATIC_KEY_ADMIN_ENABLED", "true").lower() in ("true", "1", "yes") +STATIC_KEY_ADMIN_ENABLED = os.getenv("STATIC_KEY_ADMIN_ENABLED", "false").lower() in ("true", "1", "yes") +# Local desktop pairing is explicitly opt-in and is rejected for production +# profiles. The one-time code is removed from the process environment as soon +# as configuration loads, then cleared from this module when the pairing +# manager hashes it with its OS-keychain secret. +LOCAL_PAIRING_ENABLED = os.getenv("LOCAL_PAIRING_ENABLED", "false").lower() in ("true", "1", "yes") +_LOCAL_PAIRING_CODE = os.environ.pop("LOCAL_PAIRING_CODE", "").strip() +LOCAL_PAIRING_CODE_CONFIGURED = bool(_LOCAL_PAIRING_CODE) +LOCAL_PAIRING_CODE_TTL_SECONDS = int(os.getenv("LOCAL_PAIRING_CODE_TTL_SECONDS", "120")) +LOCAL_PAIRING_TOKEN_TTL_SECONDS = int(os.getenv("LOCAL_PAIRING_TOKEN_TTL_SECONDS", "300")) +LOCAL_PAIRING_MAX_ATTEMPTS = int(os.getenv("LOCAL_PAIRING_MAX_ATTEMPTS", "3")) +LOCAL_PAIRING_LOCKOUT_SECONDS = int(os.getenv("LOCAL_PAIRING_LOCKOUT_SECONDS", "60")) +LOCAL_PAIRING_RATE_LIMIT_RPM = int(os.getenv("LOCAL_PAIRING_RATE_LIMIT_RPM", "3")) +# When true, every plan with mutating operations requires Brain apply-eligibility +# regardless of what the client sets in `require_apply_eligibility`. Off by +# default to preserve v2-plan compatibility; flip on as v7 rolls out. +BRAIN_APPLY_ELIGIBILITY_MANDATORY = os.getenv("BRAIN_APPLY_ELIGIBILITY_MANDATORY", "false").lower() in ("true", "1", "yes") # ── Rate Limiting ──────────────────────────────────────────────────────── RATE_LIMIT_RPM = int(os.getenv("RATE_LIMIT_RPM", "60")) # Default: 60 requests/minute @@ -78,6 +105,9 @@ CACHE_MAX_SIZE = int(os.getenv("CACHE_MAX_SIZE", "200")) CACHE_TTL_SIMPLE = float(os.getenv("CACHE_TTL_SIMPLE", "300")) # 5 minutes CACHE_TTL_MEDIUM = float(os.getenv("CACHE_TTL_MEDIUM", "600")) # 10 minutes +# Exact response caching remains opt-in until keys include actor, tenant, +# session, workbook, provider policy, and the complete ordered history. +LLM_EXACT_CACHE_ENABLED = os.getenv("LLM_EXACT_CACHE_ENABLED", "false").lower() in ("true", "1", "yes") # Opt-in semantic (vector) LLM cache. When enabled and sentence-transformers # is installed, "Build a DCF" and "Create a discounted cash flow model" can # share a cache entry. Defaults off so the library import stays lazy. @@ -95,21 +125,59 @@ SSO_ENABLED = os.getenv("SSO_ENABLED", "false").lower() in ("true", "1", "yes") SSO_PROVIDER = os.getenv("SSO_PROVIDER", "") # okta | azure_ad SSO_IDP_METADATA_URL = os.getenv("SSO_IDP_METADATA_URL", "") +OIDC_ENABLED = os.getenv("OIDC_ENABLED", "false").lower() in ("true", "1", "yes") +# OIDC issuer identifiers are exact, case-sensitive values. In particular, a +# provider-published trailing slash is significant and must not be normalized +# away before JWT or durable subject-binding checks. +OIDC_ISSUER = os.getenv("OIDC_ISSUER", "").strip() +OIDC_AUDIENCE = os.getenv("OIDC_AUDIENCE", "") +OIDC_JWKS_URL = os.getenv("OIDC_JWKS_URL", "") +OIDC_ALLOWED_TENANTS = frozenset(value.strip() for value in os.getenv("OIDC_ALLOWED_TENANTS", "").split(",") if value.strip()) +OIDC_ALLOWED_ALGORITHMS = tuple(value.strip() for value in os.getenv("OIDC_ALLOWED_ALGORITHMS", "RS256").split(",") if value.strip()) +OIDC_CLOCK_SKEW_SECONDS = int(os.getenv("OIDC_CLOCK_SKEW_SECONDS", "60")) +OIDC_REQUIRE_MFA = os.getenv("OIDC_REQUIRE_MFA", "true").lower() in ("true", "1", "yes") +OIDC_ACCEPTED_MFA_ACR = frozenset(value.strip() for value in os.getenv("OIDC_ACCEPTED_MFA_ACR", "").split(",") if value.strip()) +OIDC_JWKS_TIMEOUT_SECONDS = int(os.getenv("OIDC_JWKS_TIMEOUT_SECONDS", "5")) +OIDC_JWKS_CACHE_SECONDS = int(os.getenv("OIDC_JWKS_CACHE_SECONDS", "300")) +OIDC_CLIENT_ID = os.getenv("OIDC_CLIENT_ID", "") +OIDC_CLIENT_REDIRECT_URI = os.getenv("OIDC_CLIENT_REDIRECT_URI", "") +OIDC_CLIENT_SCOPES = tuple(value.strip() for value in os.getenv("OIDC_CLIENT_SCOPES", "openid,profile,email").split(",") if value.strip()) +OIDC_DISCOVERY_URL = os.getenv("OIDC_DISCOVERY_URL", "") +OIDC_AUTHORIZATION_ENDPOINT = os.getenv("OIDC_AUTHORIZATION_ENDPOINT", "") +OIDC_TOKEN_ENDPOINT = os.getenv("OIDC_TOKEN_ENDPOINT", "") +OIDC_EXTRA_AUTHORIZE_PARAMS_JSON = os.getenv("OIDC_EXTRA_AUTHORIZE_PARAMS_JSON", "{}") BYOK_ENABLED = os.getenv("BYOK_ENABLED", "false").lower() in ("true", "1", "yes") ZERO_DATA_RETENTION = os.getenv("ZERO_DATA_RETENTION", "false").lower() in ("true", "1", "yes") DATA_RETENTION_HOURS = int(os.getenv("DATA_RETENTION_HOURS", "0")) DATA_DIR = Path(os.getenv("DATA_DIR", str(BASE_DIR.parent / "data"))) PRODUCTION_MODE = CONFIG_PROFILE in {"prod", "production", "enterprise"} or APP_ENV in {"prod", "production"} or ENTERPRISE_MODE +DEV_SAFE_PUBLIC_DIAGNOSTICS = not PRODUCTION_MODE +DIAGNOSTIC_ENDPOINTS_PUBLIC = (not AUTH_ENABLED) or DEV_SAFE_PUBLIC_DIAGNOSTICS + +# Experimental planning/replay surfaces and the legacy global RAG singleton +# are development-only until their ownership and lifecycle contracts close. +EXPERIMENTAL_ROUTES_ENABLED = os.getenv( + "EXPERIMENTAL_ROUTES_ENABLED", + "false" if PRODUCTION_MODE else "true", +).lower() in ("true", "1", "yes") +RAG_ENABLED = os.getenv("RAG_ENABLED", "false" if PRODUCTION_MODE else "true").lower() in ( + "true", + "1", + "yes", +) # ── Workbook worker integrations ──────────────────────────────────────── WORKBOOK_WORKER_MODE = os.getenv("WORKBOOK_WORKER_MODE", "sheetjs_exceljs_hyperformula") WORKBOOK_WORKER_DIR = Path(os.getenv("WORKBOOK_WORKER_DIR", str(BASE_DIR.parent / "workbook-worker"))) WORKBOOK_WORKER_TIMEOUT_SECONDS = int(os.getenv("WORKBOOK_WORKER_TIMEOUT_SECONDS", "30")) +WORKBOOK_WORKER_MAX_OLD_SPACE_MB = int(os.getenv("WORKBOOK_WORKER_MAX_OLD_SPACE_MB", "512")) +WORKBOOK_WORKER_MAX_OUTPUT_BYTES = int(os.getenv("WORKBOOK_WORKER_MAX_OUTPUT_BYTES", str(16 * 1024 * 1024))) OFFICECLI_SCHEMA_MODE = os.getenv("OFFICECLI_SCHEMA_MODE", "enabled") -FORMULA_ENGINE = os.getenv("FORMULA_ENGINE", "hyperformula").lower() # hyperformula | univer | python_fallback -HYPERFORMULA_ENABLED = os.getenv("HYPERFORMULA_ENABLED", "true").lower() in ("true", "1", "yes") +FORMULA_ENGINE = os.getenv("FORMULA_ENGINE", "univer").lower() # hyperformula | univer | python_fallback +HYPERFORMULA_ENABLED = os.getenv("HYPERFORMULA_ENABLED", "false").lower() in ("true", "1", "yes") HYPERFORMULA_LICENSE_KEY = os.getenv("HYPERFORMULA_LICENSE_KEY", "") -HYPERFORMULA_LICENSE_APPROVED = os.getenv("HYPERFORMULA_LICENSE_APPROVED", "true").lower() == "true" +HYPERFORMULA_LICENSE_APPROVED = os.getenv("HYPERFORMULA_LICENSE_APPROVED", "false").lower() in ("true", "1", "yes") +HYPERFORMULA_DISTRIBUTED = os.getenv("HYPERFORMULA_DISTRIBUTED", "false").lower() in ("true", "1", "yes") # ── Plan engine rollout ───────────────────────────────────────────────── PLAN_ENGINE_VERSION = os.getenv("PLAN_ENGINE_VERSION", "v2").lower() # v2 | v3 | v4 | v5 | v6 | v7 @@ -126,11 +194,24 @@ # ── Workbook worker safety limits ─────────────────────────────────────── WORKBOOK_WORKER_MAX_PAYLOAD_BYTES = int(os.getenv("WORKBOOK_WORKER_MAX_PAYLOAD_BYTES", str(12 * 1024 * 1024))) +WORKBOOK_WORKER_MAX_COMMAND_CELLS = int(os.getenv("WORKBOOK_WORKER_MAX_COMMAND_CELLS", "50000")) +WORKBOOK_WORKER_MAX_FORMULA_REFERENCES = int(os.getenv("WORKBOOK_WORKER_MAX_FORMULA_REFERENCES", "10000")) WORKBOOK_WORKER_ALLOWED_EXTENSIONS = { ext.strip().lower() for ext in os.getenv("WORKBOOK_WORKER_ALLOWED_EXTENSIONS", ".xlsx,.xlsm,.xlsb,.xls,.csv,.tsv").split(",") if ext.strip() } +FINGERPRINT_MAX_CELLS = int(os.getenv("FINGERPRINT_MAX_CELLS", "2000")) +FINGERPRINT_MAX_TEXT_BYTES = int(os.getenv("FINGERPRINT_MAX_TEXT_BYTES", str(1024 * 1024))) +FINGERPRINT_MAX_BLOCK_COMPARISONS = int(os.getenv("FINGERPRINT_MAX_BLOCK_COMPARISONS", "128000")) + + +def consume_local_pairing_code() -> str: + """Move the bootstrapped one-time code into the pairing authority once.""" + global _LOCAL_PAIRING_CODE + pairing_code = _LOCAL_PAIRING_CODE + _LOCAL_PAIRING_CODE = "" + return pairing_code def runtime_config_errors(overrides: dict | None = None) -> list[str]: @@ -141,8 +222,17 @@ def runtime_config_errors(overrides: dict | None = None) -> list[str]: "production_mode": PRODUCTION_MODE, "cors_origins": CORS_ORIGINS, "api_key": API_KEY, + "api_key_hmac_pepper": os.getenv("API_KEY_HMAC_PEPPER", ""), "auth_enabled": AUTH_ENABLED, "static_key_admin_enabled": STATIC_KEY_ADMIN_ENABLED, + "local_pairing_enabled": LOCAL_PAIRING_ENABLED, + "local_pairing_code_configured": LOCAL_PAIRING_CODE_CONFIGURED, + "local_pairing_host": HOST, + "local_pairing_code_ttl_seconds": LOCAL_PAIRING_CODE_TTL_SECONDS, + "local_pairing_token_ttl_seconds": LOCAL_PAIRING_TOKEN_TTL_SECONDS, + "local_pairing_max_attempts": LOCAL_PAIRING_MAX_ATTEMPTS, + "local_pairing_lockout_seconds": LOCAL_PAIRING_LOCKOUT_SECONDS, + "local_pairing_rate_limit_rpm": LOCAL_PAIRING_RATE_LIMIT_RPM, "db_engine": DB_ENGINE, "database_url": DATABASE_URL, "enterprise_mode": ENTERPRISE_MODE, @@ -150,31 +240,158 @@ def runtime_config_errors(overrides: dict | None = None) -> list[str]: "sso_enabled": SSO_ENABLED, "sso_provider": SSO_PROVIDER, "sso_idp_metadata_url": SSO_IDP_METADATA_URL, + "oidc_enabled": OIDC_ENABLED, + "oidc_issuer": OIDC_ISSUER, + "oidc_audience": OIDC_AUDIENCE, + "oidc_jwks_url": OIDC_JWKS_URL, + "oidc_allowed_tenants": OIDC_ALLOWED_TENANTS, + "oidc_allowed_algorithms": OIDC_ALLOWED_ALGORITHMS, + "oidc_require_mfa": OIDC_REQUIRE_MFA, + "oidc_client_id": OIDC_CLIENT_ID, + "oidc_client_redirect_uri": OIDC_CLIENT_REDIRECT_URI, + "oidc_client_scopes": OIDC_CLIENT_SCOPES, + "oidc_discovery_url": OIDC_DISCOVERY_URL, + "oidc_authorization_endpoint": OIDC_AUTHORIZATION_ENDPOINT, + "oidc_token_endpoint": OIDC_TOKEN_ENDPOINT, + "oidc_extra_authorize_params_json": OIDC_EXTRA_AUTHORIZE_PARAMS_JSON, "byok_enabled": BYOK_ENABLED, "zero_data_retention": ZERO_DATA_RETENTION, + "lifecycle_receipt_key_id": os.getenv("EXCEL_AGENT_LIFECYCLE_RECEIPT_KEY_ID", ""), + "lifecycle_receipt_hmac_key": os.getenv("EXCEL_AGENT_LIFECYCLE_RECEIPT_HMAC_KEY", ""), + "lifecycle_receipt_previous_keys": os.getenv( + "EXCEL_AGENT_LIFECYCLE_RECEIPT_PREVIOUS_KEYS", + "{}", + ), + "ai_external_sending_allowed": os.getenv("AI_EXTERNAL_SENDING_ALLOWED"), + "ai_external_documents_allowed": os.getenv("AI_EXTERNAL_DOCUMENTS_ALLOWED"), "anthropic_api_key": ANTHROPIC_API_KEY, + "anthropic_model_medium": ANTHROPIC_MODEL_MEDIUM_RAW, + "anthropic_model_complex": ANTHROPIC_MODEL_COMPLEX_RAW, "openai_api_key": OPENAI_API_KEY, + "experimental_routes_enabled": EXPERIMENTAL_ROUTES_ENABLED, + "rag_enabled": RAG_ENABLED, + "llm_exact_cache_enabled": LLM_EXACT_CACHE_ENABLED, + "llm_vector_cache_enabled": LLM_VECTOR_CACHE_ENABLED, "workbook_worker_dir": WORKBOOK_WORKER_DIR, + "formula_engine": FORMULA_ENGINE, + "hyperformula_enabled": HYPERFORMULA_ENABLED, + "hyperformula_distributed": HYPERFORMULA_DISTRIBUTED, + "hyperformula_license_approved": HYPERFORMULA_LICENSE_APPROVED, + "hyperformula_license_key": HYPERFORMULA_LICENSE_KEY, } values.update(overrides or {}) profile = str(values["config_profile"]).lower() app_env = str(values["app_env"]).lower() - production_like = bool(values["production_mode"]) or profile in {"prod", "production", "enterprise"} or app_env in {"prod", "production"} + production_like = ( + bool(values["production_mode"]) or profile in {"prod", "production", "enterprise"} or app_env in {"prod", "production"} + ) enterprise_like = bool(values["enterprise_mode"]) or profile == "enterprise" errors: list[str] = [] if profile not in {"dev", "test", "prod", "production", "enterprise"}: errors.append(f"CONFIG_PROFILE must be one of dev, test, prod, production, enterprise; got {profile!r}") + if bool(values.get("local_pairing_enabled")): + if production_like: + errors.append("LOCAL_PAIRING_ENABLED must be false in production profiles") + if not bool(values.get("auth_enabled")): + errors.append("AUTH_ENABLED must be true when LOCAL_PAIRING_ENABLED is true") + if not _loopback_bind_host(str(values.get("local_pairing_host") or "")): + errors.append("HOST must be loopback-only when LOCAL_PAIRING_ENABLED is true") + if not bool(values.get("local_pairing_code_configured")): + errors.append("LOCAL_PAIRING_CODE must arm a one-time code when LOCAL_PAIRING_ENABLED is true") + if not 30 <= _config_int(values.get("local_pairing_code_ttl_seconds")) <= 600: + errors.append("LOCAL_PAIRING_CODE_TTL_SECONDS must be between 30 and 600") + if not 30 <= _config_int(values.get("local_pairing_token_ttl_seconds")) <= 3600: + errors.append("LOCAL_PAIRING_TOKEN_TTL_SECONDS must be between 30 and 3600") + if not 1 <= _config_int(values.get("local_pairing_max_attempts")) <= 5: + errors.append("LOCAL_PAIRING_MAX_ATTEMPTS must be between 1 and 5") + if not 30 <= _config_int(values.get("local_pairing_lockout_seconds")) <= 3600: + errors.append("LOCAL_PAIRING_LOCKOUT_SECONDS must be between 30 and 3600") + if not 1 <= _config_int(values.get("local_pairing_rate_limit_rpm")) <= 10: + errors.append("LOCAL_PAIRING_RATE_LIMIT_RPM must be between 1 and 10") + if production_like: if not bool(values["auth_enabled"]): errors.append("AUTH_ENABLED must be true in production profiles") + if not enterprise_like and not bool(values["rbac_enabled"]): + errors.append("RBAC_ENABLED must be true in production profiles") if _placeholder_secret(str(values["api_key"])) or len(str(values["api_key"])) < 24: errors.append("API_KEY must be a strong non-placeholder secret in production profiles") + pepper = str(values.get("api_key_hmac_pepper") or "") + if _placeholder_secret(pepper) or len(pepper) < 32: + errors.append("API_KEY_HMAC_PEPPER must be a strong secret of at least 32 characters in production profiles") + receipt_key_id = str(values.get("lifecycle_receipt_key_id") or "").strip() + if not _valid_receipt_key_id(receipt_key_id): + errors.append( + "EXCEL_AGENT_LIFECYCLE_RECEIPT_KEY_ID must be a nonempty 1-128 character key identifier in production profiles" + ) + if _decode_configured_receipt_key(values.get("lifecycle_receipt_hmac_key")) is None: + errors.append( + "EXCEL_AGENT_LIFECYCLE_RECEIPT_HMAC_KEY must be URL-safe base64 that decodes to at least 32 bytes in production profiles" + ) + previous_key_error = _validate_previous_receipt_keys( + values.get("lifecycle_receipt_previous_keys"), + current_key_id=receipt_key_id, + ) + if previous_key_error: + errors.append(previous_key_error) if bool(values.get("static_key_admin_enabled", STATIC_KEY_ADMIN_ENABLED)): errors.append("STATIC_KEY_ADMIN_ENABLED must be false in production profiles; provision enterprise API keys instead") - origins = [str(origin).strip() for origin in values["cors_origins"] or []] + if not bool(values.get("oidc_enabled")): + errors.append("OIDC_ENABLED must be true in production profiles") + if not str(values.get("oidc_issuer") or "").startswith("https://"): + errors.append("OIDC_ISSUER must be an HTTPS URL in production profiles") + if not str(values.get("oidc_audience") or "").strip(): + errors.append("OIDC_AUDIENCE must be set in production profiles") + if not str(values.get("oidc_jwks_url") or "").startswith("https://"): + errors.append("OIDC_JWKS_URL must be an HTTPS URL in production profiles") + if not values.get("oidc_allowed_tenants"): + errors.append("OIDC_ALLOWED_TENANTS must explicitly allow at least one tenant in production profiles") + client_compatible_algorithms = {"RS256", "PS256", "ES256"} + algorithm_values = values.get("oidc_allowed_algorithms") + algorithms = ( + {str(value) for value in algorithm_values} + if isinstance(algorithm_values, (tuple, list, set, frozenset)) + else set() + ) + if not algorithms or not algorithms.issubset(client_compatible_algorithms): + errors.append( + "OIDC_ALLOWED_ALGORITHMS must contain only Office-client-compatible algorithms: " + "RS256, PS256, or ES256" + ) + if not bool(values.get("oidc_require_mfa")): + errors.append("OIDC_REQUIRE_MFA must be true in production profiles") + from backend.api.security.oidc_client import validate_oidc_client_config + + errors.extend(validate_oidc_client_config(values, production=True)) + if bool(values.get("experimental_routes_enabled")): + errors.append("EXPERIMENTAL_ROUTES_ENABLED must be false in production profiles") + if bool(values.get("rag_enabled")): + errors.append("RAG_ENABLED must be false in production profiles until owner-scoped retrieval is available") + if bool(values.get("llm_exact_cache_enabled")): + errors.append("LLM_EXACT_CACHE_ENABLED must be false in production profiles until cache keys are owner-scoped") + if bool(values.get("llm_vector_cache_enabled")): + errors.append("LLM_VECTOR_CACHE_ENABLED must be false in production profiles") + external_sending_allowed = _optional_bool_value(values.get("ai_external_sending_allowed")) + external_documents_allowed = _optional_bool_value(values.get("ai_external_documents_allowed")) + if external_sending_allowed is None: + errors.append("AI_EXTERNAL_SENDING_ALLOWED must be explicitly set in production profiles") + elif external_sending_allowed and not bool(values["byok_enabled"]): + errors.append("AI_EXTERNAL_SENDING_ALLOWED=true requires BYOK_ENABLED=true in production profiles") + if external_documents_allowed is None: + errors.append("AI_EXTERNAL_DOCUMENTS_ALLOWED must be explicitly set in production profiles") + elif external_documents_allowed and not (bool(values["byok_enabled"]) and bool(values["zero_data_retention"])): + errors.append( + "AI_EXTERNAL_DOCUMENTS_ALLOWED=true requires BYOK_ENABLED=true and ZERO_DATA_RETENTION=true in production profiles" + ) + cors_values = values.get("cors_origins") + origins = ( + [str(origin).strip() for origin in cors_values] + if isinstance(cors_values, (tuple, list, set, frozenset)) + else [] + ) if not origins or "*" in origins or any("localhost" in origin or origin.startswith("http://") for origin in origins): errors.append("CORS_ORIGINS must be explicit HTTPS production origins") if str(values["db_engine"]).lower() != "postgres": @@ -185,9 +402,24 @@ def runtime_config_errors(overrides: dict | None = None) -> list[str]: errors.append("DATABASE_URL must be set when DB_ENGINE=postgres") elif _weak_database_url(database_url): errors.append("DATABASE_URL appears to contain weak or placeholder credentials") - worker_dir = Path(values["workbook_worker_dir"]) + worker_dir = Path(str(values["workbook_worker_dir"])) if not worker_dir.exists(): errors.append(f"WORKBOOK_WORKER_DIR must exist in production profiles: {worker_dir}") + if str(values["formula_engine"]).lower() == "hyperformula" and bool(values["hyperformula_enabled"]): + from backend.excel.hyperformula_license import is_commercial_hyperformula_key + + if not bool(values.get("hyperformula_distributed")): + errors.append( + "HYPERFORMULA_DISTRIBUTED must attest that the licensed runtime is present before production HyperFormula use" + ) + if not bool(values["hyperformula_license_approved"]): + errors.append("HYPERFORMULA_LICENSE_APPROVED must be true before production HyperFormula use") + if not is_commercial_hyperformula_key(values["hyperformula_license_key"]): + errors.append("HYPERFORMULA_LICENSE_KEY must be an approved commercial key before production HyperFormula use") + for field in ("anthropic_model_medium", "anthropic_model_complex"): + model = str(values.get(field, "")).strip() + if model in ANTHROPIC_RETIRED_MODELS: + errors.append(f"{field.upper()} uses retired Anthropic model {model}; migrate to {ANTHROPIC_RETIRED_MODELS[model]}") if enterprise_like: if not bool(values["rbac_enabled"]): @@ -221,7 +453,77 @@ def _placeholder_secret(value: str) -> bool: ) +def _valid_receipt_key_id(value: str) -> bool: + return bool(value) and len(value) <= 128 and all(char.isalnum() or char in "._-" for char in value) + + +def _decode_configured_receipt_key(value: object) -> bytes | None: + encoded = str(value or "").strip() + if encoded.startswith("base64:"): + encoded = encoded.split(":", 1)[1] + if not encoded: + return None + try: + padding = "=" * ((4 - len(encoded) % 4) % 4) + secret = base64.b64decode(encoded + padding, altchars=b"-_", validate=True) + except (binascii.Error, ValueError, TypeError): + return None + return secret if len(secret) >= 32 else None + + +def _validate_previous_receipt_keys(value: object, *, current_key_id: str) -> str | None: + try: + payload = json.loads(str(value or "{}")) + except (json.JSONDecodeError, TypeError, ValueError): + return "EXCEL_AGENT_LIFECYCLE_RECEIPT_PREVIOUS_KEYS must be a JSON object of key IDs to URL-safe base64 keys" + if not isinstance(payload, dict): + return "EXCEL_AGENT_LIFECYCLE_RECEIPT_PREVIOUS_KEYS must be a JSON object of key IDs to URL-safe base64 keys" + for raw_key_id, encoded in payload.items(): + key_id = str(raw_key_id or "").strip() + if not _valid_receipt_key_id(key_id): + return "EXCEL_AGENT_LIFECYCLE_RECEIPT_PREVIOUS_KEYS contains an invalid key ID" + if key_id == current_key_id: + return "EXCEL_AGENT_LIFECYCLE_RECEIPT_PREVIOUS_KEYS must not redefine the current key ID" + if not isinstance(encoded, str) or _decode_configured_receipt_key(encoded) is None: + return ( + "EXCEL_AGENT_LIFECYCLE_RECEIPT_PREVIOUS_KEYS values must be URL-safe base64 keys " + "that decode to at least 32 bytes" + ) + return None + + +def _optional_bool_value(value: object) -> bool | None: + if value is None: + return None + if isinstance(value, bool): + return value + normalized = str(value).strip().lower() + if not normalized: + return None + return normalized in {"true", "1", "yes", "on"} + + def _weak_database_url(value: str) -> bool: lowered = value.lower() weak_tokens = ["changeme", "password", "postgres:postgres", "financeai:financeai", "user:pass"] return any(token in lowered for token in weak_tokens) + + +def _config_int(value: object) -> int: + try: + return int(str(value)) + except (TypeError, ValueError): + return 0 + + +def _loopback_bind_host(value: str) -> bool: + normalized = str(value or "").strip().removeprefix("[").removesuffix("]").split("%", 1)[0].lower() + if normalized == "localhost": + return True + try: + address = ipaddress.ip_address(normalized) + except ValueError: + return False + if isinstance(address, ipaddress.IPv6Address) and address.ipv4_mapped is not None: + return address.ipv4_mapped.is_loopback + return address.is_loopback diff --git a/backend/data/connectors.py b/backend/data/connectors.py index 2402ed8..a6e851e 100644 --- a/backend/data/connectors.py +++ b/backend/data/connectors.py @@ -5,22 +5,281 @@ - describe() → connector metadata - health() → bool -Built-in connectors: EDGAR, CSV, XLSX, Yahoo Finance (stub) +Built-in connectors: EDGAR, CSV, XLSX, Yahoo Finance """ from __future__ import annotations +import asyncio import csv import io +import math +import os +import re +import stat +import xml.etree.ElementTree as ET from abc import ABC, abstractmethod -from dataclasses import dataclass, field +from dataclasses import asdict, dataclass, field +from decimal import Decimal, InvalidOperation +from html.parser import HTMLParser +from pathlib import Path from typing import Optional import structlog +from backend.config import DATA_DIR + logger = structlog.get_logger() +_YAHOO_SYMBOL_RE = re.compile(r"^[A-Z0-9][A-Z0-9.\-]{0,14}$") + + +def _bounded_xbrl_value( + value: str, + *, + scale: str = "", + sign: str = "", +) -> float | str: + clean_value = value.replace(",", "").replace("$", "").strip() + if sign not in {"", "-"}: + raise ValueError("XBRL numeric sign is invalid") + exponent = 0 + if scale: + if re.fullmatch(r"[+-]?\d{1,3}", scale) is None: + raise ValueError("XBRL numeric scale is invalid") + exponent = int(scale) + if not -308 <= exponent <= 308: + raise ValueError("XBRL numeric scale exceeds the supported range") + try: + decimal_value = Decimal(clean_value) + except InvalidOperation: + if scale or sign: + raise ValueError("XBRL scale and sign require a numeric fact") from None + return value + if not decimal_value.is_finite(): + raise ValueError("XBRL numeric facts must be finite") + if exponent: + decimal_value *= Decimal(10) ** exponent + if sign == "-": + decimal_value = -decimal_value + parsed = float(decimal_value) + if not math.isfinite(parsed) or (decimal_value != 0 and parsed == 0.0): + raise ValueError("XBRL numeric fact exceeds the supported range") + return parsed + + +class _BoundedInlineXBRLParser(HTMLParser): + _FACT_TAGS = {"ix:nonfraction", "ix:nonnumeric"} + + def __init__( + self, + *, + max_elements: int, + max_facts: int, + max_fact_characters: int, + labels: dict[str, str], + ) -> None: + super().__init__(convert_charrefs=True) + self._max_elements = max_elements + self._max_facts = max_facts + self._max_fact_characters = max_fact_characters + self._labels = labels + self._element_count = 0 + self._active_tag = "" + self._active_attributes: dict[str, str] = {} + self._active_text: list[str] = [] + self._active_character_count = 0 + self._exclude_depth = 0 + self.facts: list[dict] = [] + + def _count_element(self) -> None: + self._element_count += 1 + if self._element_count > self._max_elements: + raise ValueError("XBRL input exceeds the XML element budget") + + @staticmethod + def _validated_attributes(attributes: list[tuple[str, str | None]]) -> dict[str, str]: + if len(attributes) > 64: + raise ValueError("Inline XBRL fact exceeds the attribute budget") + validated: dict[str, str] = {} + for raw_name, raw_value in attributes: + name = raw_name.casefold() + if name in validated: + raise ValueError("Inline XBRL fact attributes must be unique") + if raw_value is None or len(raw_value) > 65_536: + raise ValueError("Inline XBRL fact attributes are malformed") + validated[name] = raw_value + return validated + + def handle_starttag(self, tag: str, attrs: list[tuple[str, str | None]]) -> None: + self._count_element() + normalized_tag = tag.casefold() + if normalized_tag in self._FACT_TAGS: + if self._active_tag: + raise ValueError("Inline XBRL facts must not be nested") + if len(self.facts) >= self._max_facts: + raise ValueError("XBRL input exceeds the connector fact budget") + attributes = self._validated_attributes(attrs) + if not attributes.get("name"): + raise ValueError("Inline XBRL fact is missing its concept name") + self._active_tag = normalized_tag + self._active_attributes = attributes + self._active_text = [] + self._active_character_count = 0 + self._exclude_depth = 0 + elif self._active_tag and normalized_tag == "ix:exclude": + self._exclude_depth += 1 + + def handle_startendtag(self, tag: str, attrs: list[tuple[str, str | None]]) -> None: + self.handle_starttag(tag, attrs) + self.handle_endtag(tag) + + def handle_data(self, data: str) -> None: + if not self._active_tag or self._exclude_depth: + return + self._active_character_count += len(data) + if self._active_character_count > self._max_fact_characters: + raise ValueError("XBRL fact exceeds the connector character budget") + self._active_text.append(data) + + def handle_endtag(self, tag: str) -> None: + if not self._active_tag: + return + normalized_tag = tag.casefold() + if normalized_tag == "ix:exclude" and self._exclude_depth: + self._exclude_depth -= 1 + return + if normalized_tag != self._active_tag: + return + if self._exclude_depth: + raise ValueError("Inline XBRL exclusion is unterminated") + + attributes = self._active_attributes + concept = attributes["name"] + context = attributes.get("contextref", "") + unit = attributes.get("unitref", "") + if any(len(field) > 512 for field in (concept, context, unit)): + raise ValueError("XBRL fact identity exceeds the connector character budget") + value = "".join(self._active_text).strip() + self.facts.append( + { + "concept": concept, + "label": self._labels.get(concept, concept), + "value": _bounded_xbrl_value( + value, + scale=attributes.get("scale", ""), + sign=attributes.get("sign", ""), + ), + "period": context, + "unit": unit, + } + ) + self._active_tag = "" + self._active_attributes = {} + self._active_text = [] + self._active_character_count = 0 + + def handle_decl(self, decl: str) -> None: + if " ".join(decl.split()).casefold() != "doctype html": + raise ValueError("Inline XBRL document type declarations are forbidden") + + def close(self) -> None: + super().close() + if self._active_tag: + raise ValueError("Inline XBRL fact is unterminated") + + +def _parse_xbrl_instance( + content: str, + *, + max_elements: int, + max_facts: int, + max_fact_characters: int, + labels: dict[str, str], +) -> list[dict]: + normalized_markup = content.casefold() + if " 128 or len(namespace_uri) > 2_048: + raise ValueError("XBRL namespace declaration exceeds the character budget") + namespace_prefixes.setdefault(namespace_uri, prefix) + continue + if not isinstance(payload, ET.Element): + raise ValueError("XBRL element stream is invalid") + + element_count += 1 + if element_count > max_elements: + raise ValueError("XBRL input exceeds the XML element budget") + if len(payload.attrib) > 64: + raise ValueError("XBRL fact exceeds the attribute budget") + + context = payload.attrib.get("contextRef") + if context is not None: + if len(facts) >= max_facts: + raise ValueError("XBRL input exceeds the connector fact budget") + tag = str(payload.tag) + namespace_uri = "" + local_name = tag + if tag.startswith("{") and "}" in tag: + namespace_uri, local_name = tag[1:].split("}", 1) + prefix = namespace_prefixes.get(namespace_uri, "") + concept = f"{prefix}:{local_name}" if prefix else local_name + unit = payload.attrib.get("unitRef", "") + value = "".join(payload.itertext()).strip() + if len(value) > max_fact_characters: + raise ValueError("XBRL fact exceeds the connector character budget") + if any(len(field) > 512 for field in (concept, context, unit)): + raise ValueError("XBRL fact identity exceeds the connector character budget") + facts.append( + { + "concept": concept, + "label": labels.get(concept, concept), + "value": _bounded_xbrl_value(value), + "period": context, + "unit": unit, + } + ) + payload.clear() + except ET.ParseError as exc: + raise ValueError("XBRL input is malformed") from exc + return facts + + +def _env_flag(name: str, default: bool) -> bool: + raw = os.getenv(name) + if raw is None: + return default + return raw.strip().lower() in {"1", "true", "yes", "on"} + + +def _env_float(name: str, default: float, minimum: float, maximum: float) -> float: + try: + value = float(os.getenv(name, str(default))) + except ValueError: + return default + return max(minimum, min(value, maximum)) + + +def _env_int(name: str, default: int, minimum: int, maximum: int) -> int: + try: + value = int(os.getenv(name, str(default))) + except ValueError: + return default + return max(minimum, min(value, maximum)) + + @dataclass class DataResult: """Standardized result from a data connector.""" @@ -35,7 +294,7 @@ class DataConnector(ABC): """Abstract base class for data source connectors.""" @abstractmethod - async def fetch(self, query: str, params: dict = None) -> DataResult: + async def fetch(self, query: str, params: Optional[dict] = None) -> DataResult: """Fetch data from the source.""" ... @@ -55,7 +314,7 @@ async def health(self) -> bool: class CSVConnector(DataConnector): """Parse CSV data from text or file-like objects.""" - async def fetch(self, query: str, params: dict = None) -> DataResult: + async def fetch(self, query: str, params: Optional[dict] = None) -> DataResult: """Parse CSV content. Args: @@ -63,30 +322,70 @@ async def fetch(self, query: str, params: dict = None) -> DataResult: params: Optional — {delimiter: str, has_header: bool} """ params = params or {} - delimiter = params.get("delimiter", ",") + delimiter = str(params.get("delimiter", ",")) has_header = params.get("has_header", True) + if len(delimiter) != 1 or delimiter in {"\x00", "\r", "\n", '"'}: + raise ValueError("CSV delimiter must be one safe character") + if not isinstance(has_header, bool): + raise ValueError("CSV has_header must be a boolean") + max_input_bytes = _env_int( + "CSV_CONNECTOR_MAX_INPUT_BYTES", 8 * 1024 * 1024, 1024, 64 * 1024 * 1024 + ) + max_rows = _env_int("CSV_CONNECTOR_MAX_ROWS", 100_000, 1, 1_000_000) + max_columns = _env_int("CSV_CONNECTOR_MAX_COLUMNS", 1_024, 1, 16_384) + max_cell_characters = _env_int( + "CSV_CONNECTOR_MAX_CELL_CHARACTERS", 65_536, 1, 1_048_576 + ) + max_output_cells = _env_int( + "CSV_CONNECTOR_MAX_OUTPUT_CELLS", 1_000_000, 1, 10_000_000 + ) + if len(query.encode("utf-8")) > max_input_bytes: + raise ValueError("CSV input exceeds the connector byte budget") - reader = csv.reader(io.StringIO(query), delimiter=delimiter) - rows = list(reader) + reader = csv.reader(io.StringIO(query), delimiter=delimiter, strict=True) + rows: list[list[str]] = [] + output_cells = 0 + try: + for row in reader: + if len(rows) >= max_rows + (1 if has_header else 0): + raise ValueError("CSV input exceeds the connector row budget") + if not row or len(row) > max_columns: + raise ValueError("CSV row exceeds the connector column budget") + if any(len(cell) > max_cell_characters for cell in row): + raise ValueError("CSV cell exceeds the connector character budget") + output_cells += len(row) + if output_cells > max_output_cells: + raise ValueError("CSV input exceeds the connector cell budget") + rows.append(row) + except csv.Error as exc: + raise ValueError("CSV input is malformed") from exc if not rows: return DataResult(source="csv", data=[], headers=[], row_count=0) if has_header: - headers = [h.strip() for h in rows[0]] + headers = [h.strip() or f"Column_{index + 1}" for index, h in enumerate(rows[0])] data_rows = rows[1:] else: headers = [f"Column_{i+1}" for i in range(len(rows[0]))] data_rows = rows + if len({header.casefold() for header in headers}) != len(headers): + raise ValueError("CSV headers must be unique ignoring case") + if any(len(row) != len(headers) for row in data_rows): + raise ValueError("CSV rows must have the same width as the header") - data = [dict(zip(headers, row)) for row in data_rows] + data = [dict(zip(headers, row, strict=True)) for row in data_rows] return DataResult( source="csv", data=data, headers=headers, row_count=len(data), - metadata={"delimiter": delimiter, "columns": len(headers)}, + metadata={ + "delimiter": delimiter, + "columns": len(headers), + "bounded": True, + }, ) def describe(self) -> dict: @@ -99,39 +398,65 @@ def describe(self) -> dict: class XLSXConnector(DataConnector): - """Parse Excel .xlsx files using openpyxl.""" + """Extract a bounded preview from an owned workbook via the isolated worker.""" - async def fetch(self, query: str, params: dict = None) -> DataResult: - """Parse XLSX file. + async def fetch(self, query: str, params: Optional[dict] = None) -> DataResult: + """Parse an owned XLSX document. Args: - query: File path as string + query: Owned document ID returned by the upload API params: {sheet_name: str, header_row: int} """ params = params or {} sheet_name = params.get("sheet_name") - header_row = params.get("header_row", 1) - - try: - import openpyxl - except ImportError: - logger.error("openpyxl_not_installed") - return DataResult(source="xlsx", data=[], headers=[], row_count=0) + header_row = max(int(params.get("header_row", 1) or 1), 1) + workbook_path, filename, session_id = _resolve_owned_xlsx_document(query) try: - wb = openpyxl.load_workbook(query, data_only=True, read_only=True) - ws = wb[sheet_name] if sheet_name and sheet_name in wb.sheetnames else wb.active - - rows = list(ws.iter_rows(values_only=True)) - wb.close() + from backend.excel.workbook_files import WorkbookIngestionService + from backend.operations.work_admission import run_cancellable_in_thread, work_context + + with work_context(session_id=session_id, kind="managed_xlsx"): + content = await asyncio.to_thread(_read_owned_workbook_bytes, workbook_path) + summary = await run_cancellable_in_thread(WorkbookIngestionService().summarize, content, filename) + sheets, sheet_names = _validate_connector_ingestion_summary(summary) + selected = next((item for item in sheets if sheet_name and item.get("name") == sheet_name), None) + if selected is None: + active_sheet = str(summary.get("active_sheet") or "") + selected = next((item for item in sheets if item.get("name") == active_sheet), None) + if selected is None: + selected = sheets[0] + + max_rows = _env_int("XLSX_CONNECTOR_MAX_ROWS", 100_000, 1, 1_000_000) + max_columns = _env_int("XLSX_CONNECTOR_MAX_COLUMNS", 1_000, 1, 16_384) + preview_rows = selected.get("raw_preview") or selected.get("preview") or [] + if not isinstance(preview_rows, list) or any(not isinstance(row, list) for row in preview_rows): + raise RuntimeError("Workbook worker returned an invalid ingestion result") + preview = [list(row)[:max_columns] for row in preview_rows] + header_index = header_row - 1 + max_sheet_rows = max(int(selected.get("max_row") or 0), 0) + if header_index >= len(preview) and header_row <= max_sheet_rows: + raise ValueError("header_row is outside the bounded workbook preview") + rows = preview[header_index : header_index + max_rows + 1] if not rows: - return DataResult(source="xlsx", data=[], headers=[], row_count=0) + return DataResult( + source="xlsx", + data=[], + headers=[], + row_count=0, + metadata={ + "sheet": str(selected.get("name") or ""), + "sheets_available": sheet_names, + "document_id": query, + "engine": summary.get("engine"), + "truncated": False, + }, + ) # Extract headers from the specified row - header_idx = header_row - 1 - headers = [str(h or f"Column_{i+1}").strip() for i, h in enumerate(rows[header_idx])] - data_rows = rows[header_idx + 1:] + headers = [str(h or f"Column_{i+1}").strip() for i, h in enumerate(rows[0])] + data_rows = rows[1:] data = [] for row in data_rows: @@ -142,30 +467,110 @@ async def fetch(self, query: str, params: dict = None) -> DataResult: if any(v is not None for v in row_dict.values()): data.append(row_dict) + preview_width = max((len(row) for row in preview), default=0) return DataResult( source="xlsx", data=data, headers=headers, row_count=len(data), metadata={ - "sheet": ws.title, - "sheets_available": wb.sheetnames if hasattr(wb, 'sheetnames') else [], + "sheet": str(selected.get("name") or ""), + "sheets_available": sheet_names, + "document_id": query, + "engine": summary.get("engine"), + "truncated": max_sheet_rows > header_index + len(rows) + or int(selected.get("max_column") or 0) > min(max_columns, preview_width), }, ) except Exception as e: logger.error("xlsx_parse_failed", error=str(e)) - return DataResult(source="xlsx", data=[], headers=[], row_count=0) + raise def describe(self) -> dict: return { "id": "xlsx", "name": "Excel File Parser", "description": "Parse .xlsx files and extract sheet data", - "accepts": [".xlsx", "application/vnd.openxmlformats-officedocument.spreadsheetml.sheet"], + "accepts": ["owned_document_id"], } +def _resolve_owned_xlsx_document(document_id: str) -> tuple[Path, str, str]: + """Resolve an actor-visible document ID inside the managed upload root.""" + from backend.storage import get_session_store + + normalized_id = str(document_id or "").strip() + document = get_session_store().get_document(normalized_id) if normalized_id else None + if not document: + raise ValueError("XLSX connector requires an owned document ID") + filename = str(document.get("filename") or "").lower() + if not filename.endswith((".xlsx", ".xlsm")): + raise ValueError("Owned document is not an XLSX/XLSM workbook") + raw_path = str(document.get("file_path") or "").strip() + if not raw_path: + raise ValueError("Owned workbook has no managed file") + managed_root = (DATA_DIR / "documents").resolve() + try: + resolved = Path(raw_path).resolve(strict=True) + except OSError as exc: + raise ValueError("Owned workbook file is unavailable") from exc + if not resolved.is_relative_to(managed_root) or not resolved.is_file(): + raise ValueError("Owned workbook is outside the managed document store") + return ( + resolved, + str(document.get("filename") or resolved.name), + str(document.get("session_id") or ""), + ) + + +def _validate_connector_ingestion_summary(summary: object) -> tuple[list[dict], list[str]]: + if not isinstance(summary, dict): + raise RuntimeError("Workbook worker returned an invalid ingestion result") + if not isinstance(summary.get("engine"), str) or not str(summary["engine"]).strip(): + raise RuntimeError("Workbook worker returned an invalid ingestion result") + raw_sheets = summary.get("sheets") + raw_sheet_names = summary.get("sheet_names") + if not isinstance(raw_sheets, list) or not raw_sheets or any(not isinstance(item, dict) for item in raw_sheets): + raise RuntimeError("Workbook worker returned an invalid ingestion result") + if not isinstance(raw_sheet_names, list) or any(not isinstance(item, str) for item in raw_sheet_names): + raise RuntimeError("Workbook worker returned an invalid ingestion result") + sheets: list[dict] = raw_sheets + return sheets, list(raw_sheet_names) + + +def _read_owned_workbook_bytes(path: Path) -> bytes: + """Read a regular managed file without following a last-hop symlink.""" + from backend.config import WORKBOOK_WORKER_MAX_PAYLOAD_BYTES + + flags = os.O_RDONLY + if hasattr(os, "O_CLOEXEC"): + flags |= os.O_CLOEXEC + if hasattr(os, "O_NOFOLLOW"): + flags |= os.O_NOFOLLOW + descriptor = os.open(path, flags) + try: + file_stat = os.fstat(descriptor) + if not stat.S_ISREG(file_stat.st_mode): + raise ValueError("Owned workbook is not a regular managed file") + max_bytes = max(1, ((WORKBOOK_WORKER_MAX_PAYLOAD_BYTES - 4096) * 3) // 4) + if file_stat.st_size > max_bytes: + raise ValueError("Owned workbook exceeds the bounded worker payload limit") + chunks: list[bytes] = [] + total = 0 + while True: + chunk = os.read(descriptor, min(1024 * 1024, max_bytes - total + 1)) + if not chunk: + break + total += len(chunk) + if total > max_bytes: + raise ValueError("Owned workbook exceeds the bounded worker payload limit") + chunks.append(chunk) + return b"".join(chunks) + finally: + os.close(descriptor) + + class XBRLConnector(DataConnector): """Parse XBRL financial data from SEC filings. @@ -205,61 +610,83 @@ class XBRLConnector(DataConnector): "us-gaap:DepreciationDepletionAndAmortization": "D&A", } - async def fetch(self, query: str, params: dict = None) -> DataResult: + async def fetch(self, query: str, params: Optional[dict] = None) -> DataResult: """Parse XBRL content. Args: - query: XBRL/iXBRL content as string or file path + query: XBRL/iXBRL document content as a string params: {format: "ixbrl" | "xbrl"} """ - import re + params = params or {} + format_name = str(params.get("format", "ixbrl")).strip().lower() + if format_name not in {"ixbrl", "xbrl"}: + raise ValueError("XBRL format must be ixbrl or xbrl") + max_input_bytes = _env_int( + "XBRL_CONNECTOR_MAX_INPUT_BYTES", + 16 * 1024 * 1024, + 1024, + 128 * 1024 * 1024, + ) + max_facts = _env_int("XBRL_CONNECTOR_MAX_FACTS", 100_000, 1, 1_000_000) + max_fact_characters = _env_int( + "XBRL_CONNECTOR_MAX_FACT_CHARACTERS", 1_048_576, 1, 8 * 1024 * 1024 + ) + max_xml_elements = _env_int( + "XBRL_CONNECTOR_MAX_XML_ELEMENTS", 250_000, 1, 2_000_000 + ) + if len(query.encode("utf-8")) > max_input_bytes: + raise ValueError("XBRL input exceeds the connector byte budget") + if "\x00" in query: + raise ValueError("XBRL input contains a forbidden null byte") - data = [] headers = ["concept", "label", "value", "period", "unit"] + data: list[dict] + + if format_name == "xbrl": + data = _parse_xbrl_instance( + query, + max_elements=max_xml_elements, + max_facts=max_facts, + max_fact_characters=max_fact_characters, + labels=self.GAAP_CONCEPTS, + ) + return DataResult( + source="xbrl", + data=data, + headers=headers, + row_count=len(data), + metadata={ + "concepts_found": len(data), + "format": format_name, + "bounded": True, + }, + ) - # Parse inline XBRL (HTML with embedded XBRL tags) - # Look for and tags - ix_pattern = re.compile( - r']*?)>([^<]*)', - re.IGNORECASE | re.DOTALL, + parser = _BoundedInlineXBRLParser( + max_elements=max_xml_elements, + max_facts=max_facts, + max_fact_characters=max_fact_characters, + labels=self.GAAP_CONCEPTS, ) - - for match in ix_pattern.finditer(query): - attrs_str = match.group(1) - value = match.group(2).strip() - - # Extract attributes - name_match = re.search(r'name="([^"]+)"', attrs_str) - context_match = re.search(r'contextRef="([^"]+)"', attrs_str) - unit_match = re.search(r'unitRef="([^"]+)"', attrs_str) - - if name_match: - concept = name_match.group(1) - label = self.GAAP_CONCEPTS.get(concept, concept) - context = context_match.group(1) if context_match else "" - unit = unit_match.group(1) if unit_match else "" - - # Clean numeric value - clean_val = value.replace(",", "").replace("$", "").strip() - try: - numeric_val = float(clean_val) - except ValueError: - numeric_val = value - - data.append({ - "concept": concept, - "label": label, - "value": numeric_val, - "period": context, - "unit": unit, - }) + try: + parser.feed(query) + parser.close() + except ValueError: + raise + except Exception as exc: + raise ValueError("Inline XBRL input is malformed") from exc + data = parser.facts return DataResult( source="xbrl", data=data, headers=headers, row_count=len(data), - metadata={"concepts_found": len(data)}, + metadata={ + "concepts_found": len(data), + "format": format_name, + "bounded": True, + }, ) def describe(self) -> dict: @@ -271,6 +698,381 @@ def describe(self) -> dict: } +class EDGARDataConnector(DataConnector): + """Adapter exposing SEC EDGAR through the generic data connector interface.""" + + def __init__(self, edgar=None): + self._edgar = edgar + + @property + def edgar(self): + if self._edgar is None: + from backend.data.edgar import get_edgar_connector + + self._edgar = get_edgar_connector() + return self._edgar + + async def fetch(self, query: str, params: Optional[dict] = None) -> DataResult: + """Fetch EDGAR data. + + Args: + query: Ticker symbol for company/filings/financials, or search text for search. + params: {operation: company | filings | financials | search, form_type: str, count: int} + """ + params = params or {} + operation = str(params.get("operation", params.get("action", "filings"))).lower() + form_type = params.get("form_type", "10-K") + + if operation in {"company", "company_lookup", "lookup"}: + company = await self.edgar.lookup_company(query) + rows = [company] if company else [] + return DataResult( + source="edgar", + data=rows, + headers=["ticker", "company_name", "cik"], + row_count=len(rows), + metadata=self._metadata(operation="company_lookup"), + ) + + if operation == "filings": + filings = await self.edgar.get_filings(query, form_type=form_type, count=params.get("count", 3)) + rows = [asdict(filing) for filing in filings] + return DataResult( + source="edgar", + data=rows, + headers=[ + "ticker", + "company_name", + "cik", + "form_type", + "filing_date", + "period_of_report", + "accession_number", + "primary_document", + "filing_url", + ], + row_count=len(rows), + metadata=self._metadata(operation="filings", form_type=form_type), + ) + + if operation == "financials": + statements = await self.edgar.get_xbrl_financials(query, form_type=form_type) + rows = [ + { + "period": statement.period, + "statement_type": statement.statement_type, + "line_item": line_item, + "value": value, + "currency": statement.currency, + "scale": statement.scale, + **self._line_item_metadata(statement, line_item), + } + for statement in statements + for line_item, value in statement.line_items.items() + ] + return DataResult( + source="edgar", + data=rows, + headers=[ + "period", + "statement_type", + "line_item", + "value", + "currency", + "scale", + "concept", + "taxonomy", + "unit", + "filed", + "period_end", + "fiscal_year", + "fiscal_period", + "form", + "accession_number", + "source_url", + ], + row_count=len(rows), + metadata=self._metadata(operation="financials", form_type=form_type, statements=len(statements)), + ) + + if operation == "search": + rows = await self.edgar.search_filings( + query, + form_type=form_type if params.get("form_type") else None, + date_start=params.get("date_start"), + date_end=params.get("date_end"), + ) + return DataResult( + source="edgar", + data=rows, + headers=["company", "form", "date", "description"], + row_count=len(rows), + metadata=self._metadata(operation="search", form_type=params.get("form_type")), + ) + + raise ValueError(f"Unknown EDGAR operation: {operation}") + + def describe(self) -> dict: + readiness = self.edgar.readiness() + return { + "id": "edgar", + "name": "SEC EDGAR", + "description": "Retrieve SEC company metadata, filings, filing search results, and XBRL financials", + "accepts": ["ticker", "company search text"], + "requires_credentials": False, + "requires_network": True, + "offline_testable": True, + "operations": [ + { + "id": "company_lookup", + "query": "ticker", + "description": "Resolve a ticker to SEC CIK and company name", + }, + { + "id": "filings", + "query": "ticker", + "description": "Return recent 10-K, 10-Q, 8-K, 20-F, 6-K, or S-1 filings", + }, + { + "id": "financials", + "query": "ticker", + "description": "Extract common US-GAAP XBRL companyconcept values", + }, + { + "id": "search", + "query": "search text", + "description": "Search SEC filing text with optional form/date filters", + }, + ], + "config": { + "user_agent_env": "SEC_EDGAR_USER_AGENT", + "live_requests_env": "SEC_EDGAR_ENABLE_LIVE_REQUESTS", + "timeout_env": "SEC_EDGAR_TIMEOUT_SECONDS", + "max_filings_env": "SEC_EDGAR_MAX_FILINGS", + }, + "safety": { + "rate_limit": "At least 110ms between outbound SEC requests", + "count_cap": readiness["max_filings"], + "supported_forms": readiness["supported_forms"], + "live_requests_can_be_disabled": True, + }, + "readiness": readiness, + } + + async def health(self) -> bool: + readiness = self.edgar.readiness() + return readiness["user_agent_configured"] and readiness["live_requests_enabled"] + + def _metadata(self, **overrides) -> dict: + return { + "schema_version": "2026-06-data-connector-result", + "requires_credentials": False, + "requires_network": True, + "offline_testable": True, + **overrides, + } + + def _line_item_metadata(self, statement, line_item: str) -> dict: + metadata = getattr(statement, "line_item_metadata", {}) or {} + return metadata.get(line_item, {}) + + +class YahooFinanceConnector(DataConnector): + """Optional market-data connector backed by Yahoo's public chart endpoint. + + The endpoint is useful for local enrichment and Shortcut-like "pull latest + market quote" flows, but it is not a contracted market-data feed. Live + requests therefore stay behind an explicit env flag and fail closed. + """ + + def __init__( + self, + live_requests_enabled: Optional[bool] = None, + timeout_seconds: Optional[float] = None, + max_symbols: Optional[int] = None, + ): + self.live_requests_enabled = ( + _env_flag("YAHOO_FINANCE_ENABLE_LIVE_REQUESTS", False) + if live_requests_enabled is None + else live_requests_enabled + ) + self.timeout_seconds = timeout_seconds or _env_float("YAHOO_FINANCE_TIMEOUT_SECONDS", 10.0, 1.0, 60.0) + self.max_symbols = max(1, min(int(max_symbols or _env_int("YAHOO_FINANCE_MAX_SYMBOLS", 10, 1, 25)), 25)) + + async def fetch(self, query: str, params: Optional[dict] = None) -> DataResult: + """Fetch market quote rows for one or more symbols. + + Args: + query: Symbol or comma/space separated symbols. + params: {operation: quote, symbols: list[str] | str} + """ + params = params or {} + operation = str(params.get("operation", params.get("action", "quote"))).lower() + if operation not in {"quote", "quotes", "market_quote"}: + raise ValueError(f"Unknown Yahoo Finance operation: {operation}") + + symbols = self._symbols_from(query, params) + if not symbols: + return self._result([], operation=operation, warnings=["No valid market symbols supplied."]) + + if not self.live_requests_enabled: + return self._result( + [], + operation=operation, + symbols=symbols, + warnings=["Set YAHOO_FINANCE_ENABLE_LIVE_REQUESTS=true to enable live Yahoo Finance quote fetches."], + ) + + rows = [] + warnings = [] + for symbol in symbols: + payload = await self._get_json(f"https://query1.finance.yahoo.com/v8/finance/chart/{symbol}?range=1d&interval=1d") + row = self._quote_row(symbol, payload) + if row: + rows.append(row) + else: + warnings.append(f"No quote returned for {symbol}.") + + return self._result(rows, operation=operation, symbols=symbols, warnings=warnings) + + def describe(self) -> dict: + readiness = self.readiness() + return { + "id": "yahoo_finance", + "name": "Yahoo Finance Quotes", + "description": "Fetch latest public market quote metadata for ticker symbols", + "accepts": ["ticker", "ticker list"], + "requires_credentials": False, + "requires_network": True, + "offline_testable": True, + "operations": [ + { + "id": "quote", + "query": "ticker or ticker list", + "description": "Return latest regular-market quote metadata from Yahoo Finance chart data", + } + ], + "config": { + "live_requests_env": "YAHOO_FINANCE_ENABLE_LIVE_REQUESTS", + "timeout_env": "YAHOO_FINANCE_TIMEOUT_SECONDS", + "max_symbols_env": "YAHOO_FINANCE_MAX_SYMBOLS", + }, + "safety": { + "contracted_feed": False, + "max_symbols": self.max_symbols, + "live_requests_can_be_disabled": True, + }, + "readiness": readiness, + } + + def readiness(self) -> dict: + warnings = [] + if not self.live_requests_enabled: + warnings.append("YAHOO_FINANCE_ENABLE_LIVE_REQUESTS is disabled; quote fetches return empty results.") + return { + "schema_version": "2026-06-data-connector-readiness", + "configured": self.live_requests_enabled, + "requires_credentials": False, + "requires_network": True, + "offline_testable": True, + "live_requests_enabled": self.live_requests_enabled, + "contracted_feed": False, + "max_symbols": self.max_symbols, + "timeout_seconds": self.timeout_seconds, + "warnings": warnings, + } + + async def health(self) -> bool: + return self.live_requests_enabled + + async def _get_json(self, url: str) -> Optional[dict]: + try: + import httpx + async with httpx.AsyncClient(timeout=self.timeout_seconds) as client: + response = await client.get(url) + if response.status_code == 200: + return response.json() + logger.warning("yahoo_finance_http_error", status=response.status_code, url=url) + except ImportError: + logger.error("httpx_not_installed", suggestion="pip install httpx") + except Exception as e: + logger.error("yahoo_finance_request_failed", url=url, error=str(e)) + return None + + def _symbols_from(self, query: str, params: dict) -> list[str]: + raw_symbols = params.get("symbols", query) + if isinstance(raw_symbols, str): + candidates = re.split(r"[\s,]+", raw_symbols) + elif isinstance(raw_symbols, (list, tuple, set)): + candidates = [str(symbol) for symbol in raw_symbols] + else: + candidates = [] + + symbols = [] + seen = set() + for candidate in candidates: + symbol = candidate.upper().strip() + if not _YAHOO_SYMBOL_RE.match(symbol) or symbol in seen: + continue + symbols.append(symbol) + seen.add(symbol) + if len(symbols) >= self.max_symbols: + break + return symbols + + def _quote_row(self, symbol: str, payload: Optional[dict]) -> Optional[dict]: + try: + chart = payload.get("chart", {}) if payload else {} + results = chart.get("result") or [] + meta = results[0].get("meta", {}) if results else {} + except (AttributeError, IndexError): + return None + + price = meta.get("regularMarketPrice") + previous_close = meta.get("previousClose", meta.get("chartPreviousClose")) + if price is None and previous_close is None: + return None + + return { + "symbol": meta.get("symbol", symbol), + "price": price, + "previous_close": previous_close, + "currency": meta.get("currency", ""), + "exchange": meta.get("fullExchangeName") or meta.get("exchangeName", ""), + "instrument_type": meta.get("instrumentType", ""), + "regular_market_time": meta.get("regularMarketTime"), + "timezone": meta.get("exchangeTimezoneName") or meta.get("timezone", ""), + "source": "yahoo_finance_chart", + } + + def _result(self, rows: list[dict], **metadata) -> DataResult: + return DataResult( + source="yahoo_finance", + data=rows, + headers=[ + "symbol", + "price", + "previous_close", + "currency", + "exchange", + "instrument_type", + "regular_market_time", + "timezone", + "source", + ], + row_count=len(rows), + metadata={ + "schema_version": "2026-06-data-connector-result", + "requires_credentials": False, + "requires_network": True, + "offline_testable": True, + "contracted_feed": False, + "live_requests_enabled": self.live_requests_enabled, + **metadata, + }, + ) + + # ── Connector Registry ────────────────────────────────────────────────── class DataSourceRegistry: @@ -282,6 +1084,8 @@ def __init__(self): self.register("csv", CSVConnector()) self.register("xlsx", XLSXConnector()) self.register("xbrl", XBRLConnector()) + self.register("edgar", EDGARDataConnector()) + self.register("yahoo_finance", YahooFinanceConnector()) def register(self, id: str, connector: DataConnector): """Register a data connector.""" @@ -297,7 +1101,7 @@ def list_connectors(self) -> list[dict]: for c in self._connectors.values() ] - async def fetch(self, connector_id: str, query: str, params: dict = None) -> DataResult: + async def fetch(self, connector_id: str, query: str, params: Optional[dict] = None) -> DataResult: """Fetch data from a named connector.""" connector = self._connectors.get(connector_id) if not connector: @@ -322,11 +1126,4 @@ def get_data_registry() -> DataSourceRegistry: global _registry if _registry is None: _registry = DataSourceRegistry() - # Register EDGAR connector - try: - from backend.data.edgar import get_edgar_connector - # EDGAR is async and has a different interface, register a shim - logger.info("edgar_connector_available") - except ImportError: - pass return _registry diff --git a/backend/data/edgar.py b/backend/data/edgar.py index df7cdce..6c20e6b 100644 --- a/backend/data/edgar.py +++ b/backend/data/edgar.py @@ -10,25 +10,57 @@ from __future__ import annotations import asyncio -import json +import os import re import time from dataclasses import dataclass, field -from typing import Optional +from typing import Any, Optional +from urllib.parse import quote import structlog logger = structlog.get_logger() + +def _env_flag(name: str, default: bool) -> bool: + raw = os.getenv(name) + if raw is None: + return default + return raw.strip().lower() in ("true", "1", "yes") + + +def _env_float(name: str, default: float, minimum: float, maximum: float) -> float: + try: + value = float(os.getenv(name, str(default))) + except ValueError: + return default + return max(minimum, min(value, maximum)) + + +def _env_int(name: str, default: int, minimum: int, maximum: int) -> int: + try: + value = int(os.getenv(name, str(default))) + except ValueError: + return default + return max(minimum, min(value, maximum)) + + # SEC requires identification via User-Agent -SEC_USER_AGENT = "FinanceAI/1.0 (internal-tool; contact@example.com)" +DEFAULT_SEC_USER_AGENT = "FinanceAI/1.0 (internal-tool; contact@example.com)" +SEC_USER_AGENT = os.getenv("SEC_EDGAR_USER_AGENT", DEFAULT_SEC_USER_AGENT).strip() or DEFAULT_SEC_USER_AGENT EDGAR_BASE = "https://efts.sec.gov/LATEST" EDGAR_SUBMISSIONS = "https://data.sec.gov/submissions" EDGAR_XBRL = "https://data.sec.gov/api/xbrl" +EDGAR_COMPANY_TICKERS = "https://www.sec.gov/files/company_tickers.json" +EDGAR_SUPPORTED_FORMS = ("10-K", "10-Q", "8-K", "20-F", "6-K", "S-1") +EDGAR_LIVE_REQUESTS_ENABLED = _env_flag("SEC_EDGAR_ENABLE_LIVE_REQUESTS", True) +EDGAR_TIMEOUT_SECONDS = _env_float("SEC_EDGAR_TIMEOUT_SECONDS", 15.0, 1.0, 60.0) +EDGAR_MAX_FILINGS = _env_int("SEC_EDGAR_MAX_FILINGS", 10, 1, 50) # Rate limit: SEC allows 10 requests per second max _last_request_time = 0.0 _MIN_INTERVAL = 0.11 # 110ms between requests +_TICKER_RE = re.compile(r"^[A-Z0-9][A-Z0-9.\-]{0,11}$") @dataclass @@ -52,6 +84,7 @@ class FinancialStatement: period: str statement_type: str # income_statement | balance_sheet | cash_flow line_items: dict[str, float] = field(default_factory=dict) + line_item_metadata: dict[str, dict] = field(default_factory=dict) currency: str = "USD" scale: str = "units" # units | thousands | millions @@ -66,40 +99,59 @@ class EDGARConnector: - Full-text search """ - def __init__(self): + def __init__( + self, + user_agent: Optional[str] = None, + timeout_seconds: Optional[float] = None, + max_filings: Optional[int] = None, + live_requests_enabled: Optional[bool] = None, + ): self._cik_cache: dict[str, str] = {} self._company_cache: dict[str, dict] = {} + self.user_agent = (user_agent or SEC_USER_AGENT).strip() or DEFAULT_SEC_USER_AGENT + self.timeout_seconds = timeout_seconds or EDGAR_TIMEOUT_SECONDS + self.max_filings = max(1, min(int(max_filings or EDGAR_MAX_FILINGS), 50)) + self.live_requests_enabled = EDGAR_LIVE_REQUESTS_ENABLED if live_requests_enabled is None else live_requests_enabled + + def readiness(self) -> dict: + """Return no-network EDGAR configuration/readiness metadata.""" + warnings = [] + if _placeholder_user_agent(self.user_agent): + warnings.append("Set SEC_EDGAR_USER_AGENT to a product/contact User-Agent before live SEC calls.") + if not self.live_requests_enabled: + warnings.append("SEC_EDGAR_ENABLE_LIVE_REQUESTS is disabled; fetches return empty results.") + + return { + "schema_version": "2026-06-data-connector-readiness", + "configured": not warnings or all("LIVE_REQUESTS" in warning for warning in warnings), + "requires_credentials": False, + "requires_network": True, + "offline_testable": True, + "live_requests_enabled": self.live_requests_enabled, + "user_agent_configured": not _placeholder_user_agent(self.user_agent), + "supported_forms": list(EDGAR_SUPPORTED_FORMS), + "max_filings": self.max_filings, + "timeout_seconds": self.timeout_seconds, + "warnings": warnings, + } async def lookup_company(self, ticker: str) -> Optional[dict]: """Look up a company's CIK and metadata by ticker symbol.""" - ticker = ticker.upper().strip() + ticker = _normalize_ticker(ticker) + if not ticker: + logger.warning("edgar_invalid_ticker") + return None if ticker in self._company_cache: return self._company_cache[ticker] try: - data = await self._get( - f"{EDGAR_BASE}/search-index?q=%22{ticker}%22&dateRange=custom" - f"&startdt=2020-01-01&forms=10-K", - ) - - if not data: - # Fallback: use the company tickers JSON - data = await self._get( - f"https://www.sec.gov/files/company_tickers.json" - ) - if data: - for entry in data.values(): - if entry.get("ticker", "").upper() == ticker: - cik = str(entry.get("cik_str", "")).zfill(10) - result = { - "ticker": ticker, - "company_name": entry.get("title", ""), - "cik": cik, - } - self._cik_cache[ticker] = cik - self._company_cache[ticker] = result - return result + data = await self._get(EDGAR_COMPANY_TICKERS) + result = _company_from_tickers_payload(data, ticker) + if result: + self._cik_cache[ticker] = result["cik"] + self._company_cache[ticker] = result + return result return None @@ -120,6 +172,13 @@ async def get_filings( form_type: SEC form type (10-K, 10-Q, 8-K) count: Number of recent filings to retrieve """ + ticker = _normalize_ticker(ticker) + form_type = _normalize_form_type(form_type) + if not ticker or not form_type: + logger.warning("edgar_invalid_filing_request", ticker=ticker, form_type=form_type) + return [] + count = self._bounded_count(count) + company = await self.lookup_company(ticker) if not company: logger.warning("edgar_company_not_found", ticker=ticker) @@ -139,7 +198,7 @@ async def get_filings( primary_docs = filings_data.get("primaryDocument", []) periods = filings_data.get("reportDate", []) - filings = [] + filings: list[SECFiling] = [] for i in range(len(forms)): if forms[i] == form_type and len(filings) < count: accession = accessions[i].replace("-", "") @@ -176,6 +235,12 @@ async def get_xbrl_financials( Returns income statement, balance sheet, and cash flow data. """ + ticker = _normalize_ticker(ticker) + form_type = _normalize_form_type(form_type) + if not ticker or not form_type: + logger.warning("edgar_invalid_financials_request", ticker=ticker, form_type=form_type) + return [] + company = await self.lookup_company(ticker) if not company: return [] @@ -217,32 +282,43 @@ async def get_xbrl_financials( for stmt_type, concept_list in concepts.items(): line_items = {} + line_item_metadata = {} for xbrl_concept, our_name in concept_list: try: - data = await self._get( + concept_url = ( f"{EDGAR_XBRL}/companyconcept/" f"CIK{cik}/us-gaap/{xbrl_concept}.json" ) + data = await self._get( + concept_url + ) if data and "units" in data: - # Get the most recent USD value - usd_values = data["units"].get("USD", []) - if usd_values: - # Filter for annual (10-K) or quarterly (10-Q) - filtered = [ - v for v in usd_values - if v.get("form") == form_type - ] - if filtered: - latest = filtered[-1] - line_items[our_name] = latest.get("val", 0) + latest = _latest_xbrl_fact(data["units"], form_type) + if latest: + unit, fact = latest + line_items[our_name] = fact.get("val", 0) + line_item_metadata[our_name] = { + "concept": xbrl_concept, + "taxonomy": "us-gaap", + "unit": unit, + "filed": fact.get("filed", ""), + "period_end": fact.get("end", ""), + "fiscal_year": fact.get("fy"), + "fiscal_period": fact.get("fp", ""), + "form": fact.get("form", form_type), + "accession_number": fact.get("accn", ""), + "source_url": concept_url, + } except Exception: continue if line_items: + period = _statement_period(line_item_metadata, form_type) statements.append(FinancialStatement( - period=form_type, + period=period, statement_type=stmt_type, line_items=line_items, + line_item_metadata=line_item_metadata, )) logger.info( @@ -258,7 +334,7 @@ def financials_to_operations( statements: list[FinancialStatement], ticker: str, start_row: int = 1, - ) -> list[dict]: + ) -> list[dict[str, Any]]: """Convert extracted financial data to Excel operations. Generates a formatted financial statement layout with: @@ -266,7 +342,7 @@ def financials_to_operations( - Line items with proper formatting - Blue font for hard-coded data (IB convention) """ - operations = [] + operations: list[dict[str, Any]] = [] current_row = start_row for stmt in statements: @@ -360,8 +436,15 @@ async def search_filings( date_end: Optional[str] = None, ) -> list[dict]: """Full-text search across SEC filings.""" - params = f"q=%22{query.replace(' ', '%20')}%22" + query = (query or "").strip() + if not query: + return [] + query = query[:200] + params = f"q=%22{quote(query)}%22" if form_type: + form_type = _normalize_form_type(form_type) + if not form_type: + return [] params += f"&forms={form_type}" if date_start: params += f"&dateRange=custom&startdt={date_start}" @@ -391,6 +474,13 @@ async def _get(self, url: str) -> Optional[dict]: """Rate-limited GET request to SEC EDGAR.""" global _last_request_time + if not self.live_requests_enabled: + logger.info("edgar_live_requests_disabled", url=url) + return None + if _placeholder_user_agent(self.user_agent): + logger.warning("edgar_user_agent_unconfigured", env="SEC_EDGAR_USER_AGENT") + return None + # Enforce rate limit now = time.time() elapsed = now - _last_request_time @@ -402,10 +492,10 @@ async def _get(self, url: str) -> Optional[dict]: import httpx async with httpx.AsyncClient( headers={ - "User-Agent": SEC_USER_AGENT, + "User-Agent": self.user_agent, "Accept": "application/json", }, - timeout=15.0, + timeout=self.timeout_seconds, ) as client: response = await client.get(url) if response.status_code == 200: @@ -420,6 +510,81 @@ async def _get(self, url: str) -> Optional[dict]: logger.error("edgar_request_failed", url=url, error=str(e)) return None + def _bounded_count(self, count: int) -> int: + try: + requested = int(count) + except (TypeError, ValueError): + requested = 3 + return max(1, min(requested, self.max_filings)) + + +def _normalize_ticker(ticker: str) -> str: + normalized = (ticker or "").upper().strip() + return normalized if _TICKER_RE.match(normalized) else "" + + +def _normalize_form_type(form_type: str) -> str: + normalized = (form_type or "").upper().strip() + return normalized if normalized in EDGAR_SUPPORTED_FORMS else "" + + +def _company_from_tickers_payload(data: Optional[dict], ticker: str) -> Optional[dict]: + if not data: + return None + for entry in data.values(): + if entry.get("ticker", "").upper() == ticker: + cik = str(entry.get("cik_str", "")).zfill(10) + return { + "ticker": ticker, + "company_name": entry.get("title", ""), + "cik": cik, + } + return None + + +def _latest_xbrl_fact(units: dict, form_type: str) -> Optional[tuple[str, dict]]: + preferred_units = ("USD", "shares", "pure", "USD/shares") + candidates: list[tuple[str, dict]] = [] + for unit in preferred_units: + for fact in units.get(unit, []) or []: + if fact.get("form") == form_type and "val" in fact: + candidates.append((unit, fact)) + for unit, facts in units.items(): + if unit in preferred_units: + continue + for fact in facts or []: + if fact.get("form") == form_type and "val" in fact: + candidates.append((unit, fact)) + + if not candidates: + return None + + def sort_key(candidate: tuple[str, dict]) -> tuple[str, str, str]: + _unit, fact = candidate + return ( + str(fact.get("filed") or ""), + str(fact.get("end") or ""), + str(fact.get("accn") or ""), + ) + + return sorted(candidates, key=sort_key)[-1] + + +def _statement_period(line_item_metadata: dict[str, dict], fallback: str) -> str: + periods = [ + str(metadata.get("period_end") or "") + for metadata in line_item_metadata.values() + if metadata.get("period_end") + ] + if not periods: + return fallback + return sorted(periods)[-1] + + +def _placeholder_user_agent(user_agent: str) -> bool: + lowered = (user_agent or "").lower() + return not lowered or "contact@example.com" in lowered or "internal-tool" in lowered + # ── Singleton ──────────────────────────────────────────────────────────── diff --git a/backend/documents/pdf_parser.py b/backend/documents/pdf_parser.py index 1dfb097..d455c22 100644 --- a/backend/documents/pdf_parser.py +++ b/backend/documents/pdf_parser.py @@ -8,17 +8,40 @@ from __future__ import annotations -import io import re +import os from dataclasses import dataclass, field from pathlib import Path -from typing import Optional, BinaryIO +from typing import Any, BinaryIO import structlog logger = structlog.get_logger() +class PDFBudgetExceeded(ValueError): + """Raised when a PDF exceeds a deterministic structural parse budget.""" + + +@dataclass(frozen=True) +class PDFParseLimits: + max_pages: int = 500 + max_text_chars: int = 2_000_000 + max_tables: int = 500 + max_table_rows: int = 20_000 + max_table_cells: int = 250_000 + + @classmethod + def from_env(cls) -> "PDFParseLimits": + return cls( + max_pages=_bounded_env_int("PDF_MAX_PAGES", cls.max_pages, 1, 5_000), + max_text_chars=_bounded_env_int("PDF_MAX_TEXT_CHARS", cls.max_text_chars, 1_000, 20_000_000), + max_tables=_bounded_env_int("PDF_MAX_TABLES", cls.max_tables, 0, 5_000), + max_table_rows=_bounded_env_int("PDF_MAX_TABLE_ROWS", cls.max_table_rows, 0, 500_000), + max_table_cells=_bounded_env_int("PDF_MAX_TABLE_CELLS", cls.max_table_cells, 0, 5_000_000), + ) + + @dataclass class ExtractedTable: """A table extracted from a PDF page.""" @@ -39,7 +62,7 @@ class ParsedDocument: def to_context(self, max_chars: int = 8000) -> dict: """Convert to a context dict suitable for LLM consumption.""" - context = { + context: dict[str, Any] = { "source": self.filename, "pages": self.page_count, "metadata": self.metadata, @@ -67,7 +90,12 @@ def to_context(self, max_chars: int = 8000) -> dict: return context -def parse_pdf(source: str | Path | BinaryIO, filename: str = "document.pdf") -> ParsedDocument: +def parse_pdf( + source: str | Path | BinaryIO, + filename: str = "document.pdf", + *, + limits: PDFParseLimits | None = None, +) -> ParsedDocument: """Parse a PDF and extract text + tables. Args: @@ -79,9 +107,10 @@ def parse_pdf(source: str | Path | BinaryIO, filename: str = "document.pdf") -> """ try: import pymupdf # PyMuPDF - except ImportError: - raise ImportError("PyMuPDF is required: pip install pymupdf") + except ImportError as exc: + raise ImportError("PyMuPDF is required: pip install pymupdf") from exc + limits = limits or PDFParseLimits.from_env() # Open document if isinstance(source, (str, Path)): doc = pymupdf.open(str(source)) @@ -89,71 +118,94 @@ def parse_pdf(source: str | Path | BinaryIO, filename: str = "document.pdf") -> data = source.read() if hasattr(source, "read") else source doc = pymupdf.open(stream=data, filetype="pdf") - all_text = [] - all_tables = [] + page_count = len(doc) + if page_count > limits.max_pages: + doc.close() + raise PDFBudgetExceeded(f"PDF page count exceeds limit ({page_count} > {limits.max_pages})") + all_text: list[str] = [] + all_tables: list[ExtractedTable] = [] + text_chars = 0 + table_rows = 0 + table_cells = 0 - for page_num in range(len(doc)): - page = doc[page_num] - - # ── Text extraction ────────────────────────────────────── - text = page.get_text("text") - if text.strip(): - all_text.append(f"--- Page {page_num + 1} ---\n{text.strip()}") - - # ── Table extraction ───────────────────────────────────── - try: - tables = page.find_tables() - for table in tables: - extracted = table.extract() - if not extracted or len(extracted) < 2: - continue - - # First row as headers, rest as data - headers = [_clean_cell(c) for c in extracted[0]] - rows = [ - [_clean_cell(c) for c in row] - for row in extracted[1:] - if any(c and str(c).strip() for c in row) # skip empty rows - ] - - if rows: - # Attempt to detect table title from preceding text - title = _detect_table_title(text, headers) - all_tables.append(ExtractedTable( - page=page_num + 1, - rows=rows, - headers=headers, - title=title, - )) - except Exception as e: - logger.debug("table_extraction_failed", page=page_num + 1, error=str(e)) - - # Extract document metadata - meta = doc.metadata or {} - doc_metadata = { - "title": meta.get("title", ""), - "author": meta.get("author", ""), - "subject": meta.get("subject", ""), - "creation_date": meta.get("creationDate", ""), - } - - doc.close() + try: + for page_num in range(page_count): + page = doc[page_num] + + # ── Text extraction ────────────────────────────────── + text = page.get_text("text") + if text.strip(): + rendered_text = f"--- Page {page_num + 1} ---\n{text.strip()}" + text_chars += len(rendered_text) + if text_chars > limits.max_text_chars: + raise PDFBudgetExceeded("PDF extracted text exceeds safety limit") + all_text.append(rendered_text) + + # ── Table extraction ───────────────────────────────── + try: + tables = page.find_tables() + for table in tables: + if len(all_tables) >= limits.max_tables: + raise PDFBudgetExceeded("PDF table count exceeds safety limit") + extracted = table.extract() + if not extracted or len(extracted) < 2: + continue + + # First row as headers, rest as data + headers = [_clean_cell(c) for c in extracted[0]] + rows = [ + [_clean_cell(c) for c in row] + for row in extracted[1:] + if any(c and str(c).strip() for c in row) # skip empty rows + ] + table_rows += len(rows) + table_cells += len(headers) + sum(len(row) for row in rows) + if table_rows > limits.max_table_rows or table_cells > limits.max_table_cells: + raise PDFBudgetExceeded("PDF table data exceeds safety limit") + + if rows: + title = _detect_table_title(text, headers) + all_tables.append(ExtractedTable( + page=page_num + 1, + rows=rows, + headers=headers, + title=title, + )) + except PDFBudgetExceeded: + raise + except Exception as e: + logger.debug("table_extraction_failed", page=page_num + 1, error=str(e)) + + meta = doc.metadata or {} + doc_metadata = { + "title": meta.get("title", ""), + "author": meta.get("author", ""), + "subject": meta.get("subject", ""), + "creation_date": meta.get("creationDate", ""), + } + finally: + doc.close() result = ParsedDocument( filename=filename, - page_count=len(doc) if hasattr(doc, '__len__') else 0, + page_count=page_count, text="\n\n".join(all_text), tables=all_tables, metadata={k: v for k, v in doc_metadata.items() if v}, ) - # Fix page_count after close - result.page_count = len(all_text) or 1 - logger.info("pdf_parsed", filename=filename, pages=result.page_count, tables=len(all_tables), text_chars=len(result.text)) return result +def _bounded_env_int(name: str, default: int, minimum: int, maximum: int) -> int: + try: + value = int(os.getenv(name, str(default))) + except ValueError: + value = default + return max(minimum, min(value, maximum)) + + def extract_financial_data(parsed: ParsedDocument) -> dict: """Post-process parsed document to identify financial data. @@ -162,7 +214,7 @@ def extract_financial_data(parsed: ParsedDocument) -> dict: - Balance sheet items (Total Assets, Total Liabilities) - Key metrics and ratios """ - financial = { + financial: dict[str, Any] = { "type": "unknown", "items": {}, "tables_with_numbers": [], diff --git a/backend/documents/pdf_worker.py b/backend/documents/pdf_worker.py new file mode 100644 index 0000000..11c8747 --- /dev/null +++ b/backend/documents/pdf_worker.py @@ -0,0 +1,379 @@ +"""Resource-bounded subprocess boundary for untrusted PDF parsing.""" + +from __future__ import annotations + +import ctypes +import json +import multiprocessing +import os +import tempfile +import time +from pathlib import Path +from typing import Any + +from backend.documents.pdf_parser import ExtractedTable, ParsedDocument +from backend.operations.work_admission import ( + CancellationEvent, + WorkAdmissionCancelled, + WorkExecutionTimeout, + current_work_context, + work_admission_controller, +) + + +class PDFWorkerError(RuntimeError): + """The isolated PDF worker rejected or failed to parse a document.""" + + +class PDFWorkerTimeout(PDFWorkerError, WorkExecutionTimeout): + """The isolated PDF worker exceeded its wall-clock budget.""" + + +def _env_int(name: str, default: int, minimum: int, maximum: int) -> int: + try: + value = int(os.getenv(name, str(default))) + except ValueError: + value = default + return max(minimum, min(value, maximum)) + + +PDF_PARSE_TIMEOUT_SECONDS = _env_int("PDF_PARSE_TIMEOUT_SECONDS", 30, 1, 300) +PDF_PARSE_CPU_SECONDS = _env_int("PDF_PARSE_CPU_SECONDS", 20, 1, 240) +PDF_PARSE_MEMORY_BYTES = _env_int("PDF_PARSE_MEMORY_BYTES", 768 * 1024 * 1024, 128 * 1024 * 1024, 4 * 1024**3) +PDF_PARSE_MAX_OUTPUT_BYTES = _env_int("PDF_PARSE_MAX_OUTPUT_BYTES", 16 * 1024 * 1024, 1024, 128 * 1024 * 1024) +PDF_PARSE_MAX_CONCURRENCY = _env_int("PDF_PARSE_MAX_CONCURRENCY", 2, 1, 16) +PDF_PARSE_QUEUE_TIMEOUT_SECONDS = _env_int("PDF_PARSE_QUEUE_TIMEOUT_SECONDS", 5, 1, 60) + +_WINDOWS_JOB_OBJECT_EXTENDED_LIMIT_INFORMATION = 9 +_WINDOWS_JOB_OBJECT_LIMIT_PROCESS_TIME = 0x00000002 +_WINDOWS_JOB_OBJECT_LIMIT_JOB_TIME = 0x00000004 +_WINDOWS_JOB_OBJECT_LIMIT_PROCESS_MEMORY = 0x00000100 +_WINDOWS_JOB_OBJECT_LIMIT_JOB_MEMORY = 0x00000200 +_WINDOWS_JOB_OBJECT_LIMIT_KILL_ON_JOB_CLOSE = 0x00002000 +_WINDOWS_JOB_TERMINATE_EXIT_CODE = 1 +_WINDOWS_TICKS_PER_SECOND = 10_000_000 +_WORKER_START_GATE_TIMEOUT_SECONDS = 10 + + +class _JobObjectBasicLimitInformation(ctypes.Structure): + _fields_ = [ + ("PerProcessUserTimeLimit", ctypes.c_longlong), + ("PerJobUserTimeLimit", ctypes.c_longlong), + ("LimitFlags", ctypes.c_uint32), + ("MinimumWorkingSetSize", ctypes.c_size_t), + ("MaximumWorkingSetSize", ctypes.c_size_t), + ("ActiveProcessLimit", ctypes.c_uint32), + ("Affinity", ctypes.c_size_t), + ("PriorityClass", ctypes.c_uint32), + ("SchedulingClass", ctypes.c_uint32), + ] + + +class _IoCounters(ctypes.Structure): + _fields_ = [ + ("ReadOperationCount", ctypes.c_uint64), + ("WriteOperationCount", ctypes.c_uint64), + ("OtherOperationCount", ctypes.c_uint64), + ("ReadTransferCount", ctypes.c_uint64), + ("WriteTransferCount", ctypes.c_uint64), + ("OtherTransferCount", ctypes.c_uint64), + ] + + +class _JobObjectExtendedLimitInformation(ctypes.Structure): + _fields_ = [ + ("BasicLimitInformation", _JobObjectBasicLimitInformation), + ("IoInfo", _IoCounters), + ("ProcessMemoryLimit", ctypes.c_size_t), + ("JobMemoryLimit", ctypes.c_size_t), + ("PeakProcessMemoryUsed", ctypes.c_size_t), + ("PeakJobMemoryUsed", ctypes.c_size_t), + ] + + +class _WindowsJob: + """Owning handle for a per-parse Windows Job Object.""" + + def __init__(self, kernel32: Any, handle: Any) -> None: + self._kernel32 = kernel32 + self._handle = handle + + def terminate(self) -> None: + if self._handle is None: + return + if not self._kernel32.TerminateJobObject(self._handle, _WINDOWS_JOB_TERMINATE_EXIT_CODE): + raise _windows_api_error("TerminateJobObject") + + def close(self) -> None: + handle = self._handle + if handle is not None and not self._kernel32.CloseHandle(handle): + raise _windows_api_error("CloseHandle(job)") + self._handle = None + + +def _is_windows() -> bool: + return os.name == "nt" + + +def _windows_api_error(api_name: str) -> OSError: + get_last_error = getattr(ctypes, "get_last_error", None) + error_code = int(get_last_error()) if callable(get_last_error) else 0 + return OSError(error_code, f"{api_name} failed with Win32 error {error_code}") + + +def _load_kernel32() -> Any: + loader = getattr(ctypes, "WinDLL", None) + if loader is None: + raise OSError("Windows Job Object APIs are unavailable") + kernel32 = loader("kernel32", use_last_error=True) + kernel32.CreateJobObjectW.argtypes = [ctypes.c_void_p, ctypes.c_wchar_p] + kernel32.CreateJobObjectW.restype = ctypes.c_void_p + kernel32.SetInformationJobObject.argtypes = [ctypes.c_void_p, ctypes.c_int, ctypes.c_void_p, ctypes.c_uint32] + kernel32.SetInformationJobObject.restype = ctypes.c_int + kernel32.AssignProcessToJobObject.argtypes = [ctypes.c_void_p, ctypes.c_void_p] + kernel32.AssignProcessToJobObject.restype = ctypes.c_int + kernel32.TerminateJobObject.argtypes = [ctypes.c_void_p, ctypes.c_uint32] + kernel32.TerminateJobObject.restype = ctypes.c_int + kernel32.CloseHandle.argtypes = [ctypes.c_void_p] + kernel32.CloseHandle.restype = ctypes.c_int + return kernel32 + + +def _install_windows_job(process_handle: int) -> _WindowsJob: + """Create, configure, and assign the waiting parser process to a Job Object.""" + kernel32 = _load_kernel32() + job_handle = kernel32.CreateJobObjectW(None, None) + if not job_handle: + raise _windows_api_error("CreateJobObjectW") + + try: + limits = _JobObjectExtendedLimitInformation() + limits.BasicLimitInformation.PerProcessUserTimeLimit = PDF_PARSE_CPU_SECONDS * _WINDOWS_TICKS_PER_SECOND + limits.BasicLimitInformation.PerJobUserTimeLimit = PDF_PARSE_CPU_SECONDS * _WINDOWS_TICKS_PER_SECOND + limits.BasicLimitInformation.LimitFlags = ( + _WINDOWS_JOB_OBJECT_LIMIT_PROCESS_TIME + | _WINDOWS_JOB_OBJECT_LIMIT_JOB_TIME + | _WINDOWS_JOB_OBJECT_LIMIT_PROCESS_MEMORY + | _WINDOWS_JOB_OBJECT_LIMIT_JOB_MEMORY + | _WINDOWS_JOB_OBJECT_LIMIT_KILL_ON_JOB_CLOSE + ) + limits.ProcessMemoryLimit = PDF_PARSE_MEMORY_BYTES + limits.JobMemoryLimit = PDF_PARSE_MEMORY_BYTES + if not kernel32.SetInformationJobObject( + job_handle, + _WINDOWS_JOB_OBJECT_EXTENDED_LIMIT_INFORMATION, + ctypes.byref(limits), + ctypes.sizeof(limits), + ): + raise _windows_api_error("SetInformationJobObject") + + if not kernel32.AssignProcessToJobObject(job_handle, process_handle): + raise _windows_api_error("AssignProcessToJobObject") + except BaseException: + kernel32.CloseHandle(job_handle) + raise + return _WindowsJob(kernel32, job_handle) + + +def _stop_worker_process(process: Any, windows_job: _WindowsJob | None = None) -> None: + job_error: OSError | None = None + if windows_job is not None: + try: + windows_job.terminate() + except OSError as exc: + job_error = exc + elif process.is_alive(): + process.terminate() + + process.join(timeout=2) + if process.is_alive(): + process.terminate() + process.join(timeout=2) + if process.is_alive() and hasattr(process, "kill"): + process.kill() + process.join(timeout=2) + if process.is_alive(): + raise OSError("PDF worker process did not exit after termination") + if job_error is not None: + raise job_error + + +def parse_pdf_sandboxed( + content: bytes, + filename: str = "document.pdf", + *, + timeout_seconds: int | None = None, +) -> ParsedDocument: + """Parse bytes in a disposable process with time, memory, CPU, and output caps.""" + context = current_work_context() + with work_admission_controller.admit( + "pdf", + session_id=context.session_id, + cancellation_event=context.cancellation_event, + queue_timeout_seconds=PDF_PARSE_QUEUE_TIMEOUT_SECONDS, + ) as lease: + return _run_isolated_parse( + content, + filename, + timeout_seconds or PDF_PARSE_TIMEOUT_SECONDS, + cancellation_event=lease.cancellation, + ) + + +def _run_isolated_parse( + content: bytes, + filename: str, + timeout_seconds: int, + *, + cancellation_event: CancellationEvent | None = None, +) -> ParsedDocument: + with tempfile.TemporaryDirectory(prefix="excel-agent-pdf-") as temp_dir: + input_path = Path(temp_dir) / "input.pdf" + output_path = Path(temp_dir) / "result.json" + input_path.write_bytes(content) + context = multiprocessing.get_context("spawn") + start_gate = context.Event() + process = context.Process( + target=_parse_worker, + args=(str(input_path), str(output_path), filename, start_gate), + daemon=True, + ) + process_started = False + windows_job: _WindowsJob | None = None + try: + process.start() + process_started = True + if _is_windows(): + try: + process_handle = int(process.sentinel) + if process_handle <= 0: + raise OSError("spawned PDF worker has no process handle") + windows_job = _install_windows_job(process_handle) + except (OSError, ValueError, TypeError) as exc: + raise PDFWorkerError("PDF worker Windows resource limits could not be installed") from exc + start_gate.set() + execution_timeout = max(int(timeout_seconds), 1) + if cancellation_event is None: + process.join(timeout=execution_timeout) + else: + deadline = time.monotonic() + execution_timeout + while process.is_alive(): + if cancellation_event.is_set(): + try: + _stop_worker_process(process, windows_job) + except OSError as exc: + raise PDFWorkerError("PDF worker process tree could not be terminated") from exc + reason = str(getattr(cancellation_event, "reason", "request") or "request") + raise WorkAdmissionCancelled("pdf", reason) + remaining = deadline - time.monotonic() + if remaining <= 0: + break + process.join(timeout=min(0.05, remaining)) + if process.is_alive(): + try: + _stop_worker_process(process, windows_job) + except OSError as exc: + raise PDFWorkerError("PDF worker process tree could not be terminated") from exc + raise PDFWorkerTimeout("PDF parsing timed out") + if not output_path.exists(): + raise PDFWorkerError(f"PDF worker exited without a result (exit code {process.exitcode})") + if output_path.stat().st_size > PDF_PARSE_MAX_OUTPUT_BYTES: + raise PDFWorkerError("PDF worker output exceeded safety limit") + try: + payload = json.loads(output_path.read_text(encoding="utf-8")) + except (OSError, json.JSONDecodeError) as exc: + raise PDFWorkerError("PDF worker returned an invalid result") from exc + if not payload.get("ok"): + error_type = str(payload.get("error_type") or "PDFWorkerError") + message = str(payload.get("error") or "PDF parsing failed") + raise PDFWorkerError(f"{error_type}: {message}") + return _parsed_document_from_dict(payload["document"]) + finally: + if process_started and process.is_alive(): + try: + _stop_worker_process(process, windows_job) + except OSError as exc: + raise PDFWorkerError("PDF worker process tree could not be terminated") from exc + if windows_job is not None: + try: + windows_job.close() + except OSError as exc: + raise PDFWorkerError("PDF worker Windows Job Object could not be closed") from exc + + +def _parse_worker(input_path: str, output_path: str, filename: str, start_gate: Any) -> None: + try: + if not start_gate.wait(timeout=_WORKER_START_GATE_TIMEOUT_SECONDS): + raise PDFWorkerError("PDF worker start gate was not released") + _apply_process_limits() + from backend.documents.pdf_parser import parse_pdf + + parsed = parse_pdf(input_path, filename=filename) + payload = {"ok": True, "document": _parsed_document_to_dict(parsed)} + except BaseException as exc: + payload = { + "ok": False, + "error_type": exc.__class__.__name__, + "error": str(exc)[:1_000], + } + encoded = json.dumps(payload, ensure_ascii=False, separators=(",", ":")).encode("utf-8") + if len(encoded) > PDF_PARSE_MAX_OUTPUT_BYTES: + encoded = json.dumps( + {"ok": False, "error_type": "PDFBudgetExceeded", "error": "PDF output exceeds safety limit"}, + separators=(",", ":"), + ).encode("utf-8") + Path(output_path).write_bytes(encoded) + + +def _apply_process_limits() -> None: + if _is_windows(): + # The parent releases the start gate only after assigning this process + # to a configured Job Object. + return + try: + import resource + + resource.setrlimit(resource.RLIMIT_CPU, (PDF_PARSE_CPU_SECONDS, PDF_PARSE_CPU_SECONDS)) + resource.setrlimit(resource.RLIMIT_FSIZE, (PDF_PARSE_MAX_OUTPUT_BYTES, PDF_PARSE_MAX_OUTPUT_BYTES)) + if hasattr(resource, "RLIMIT_AS"): + resource.setrlimit(resource.RLIMIT_AS, (PDF_PARSE_MEMORY_BYTES, PDF_PARSE_MEMORY_BYTES)) + except (ImportError, OSError, ValueError): + # Preserve the existing cross-POSIX behavior: macOS can reject an + # address-space ceiling below the interpreter's current reservation. + return + + +def _parsed_document_to_dict(parsed: ParsedDocument) -> dict[str, Any]: + return { + "filename": parsed.filename, + "page_count": parsed.page_count, + "text": parsed.text, + "tables": [ + { + "page": table.page, + "rows": table.rows, + "headers": table.headers, + "title": table.title, + } + for table in parsed.tables + ], + "metadata": parsed.metadata, + } + + +def _parsed_document_from_dict(payload: dict[str, Any]) -> ParsedDocument: + return ParsedDocument( + filename=str(payload.get("filename") or "document.pdf"), + page_count=int(payload.get("page_count") or 0), + text=str(payload.get("text") or ""), + tables=[ + ExtractedTable( + page=int(table.get("page") or 0), + rows=[[str(cell) for cell in row] for row in table.get("rows") or []], + headers=[str(cell) for cell in table.get("headers") or []], + title=str(table.get("title") or ""), + ) + for table in payload.get("tables") or [] + ], + metadata=dict(payload.get("metadata") or {}), + ) diff --git a/backend/documents/rag.py b/backend/documents/rag.py index 380cfab..9ed351e 100644 --- a/backend/documents/rag.py +++ b/backend/documents/rag.py @@ -10,16 +10,18 @@ import hashlib import re -import time from dataclasses import dataclass, field -from pathlib import Path -from typing import Optional +from typing import Any, Optional import structlog logger = structlog.get_logger() +class RAGDisabledError(RuntimeError): + """Raised when the legacy unscoped RAG surface is disabled by policy.""" + + @dataclass class DocumentChunk: """A chunk of text from a parsed document.""" @@ -92,18 +94,19 @@ class RAGPipeline: def __init__(self, model_name: str = "all-MiniLM-L6-v2"): self.model_name = model_name - self._model = None - self._available = None + self._model: Any | None = None + self._available: bool | None = None self._documents: dict[str, list[DocumentChunk]] = {} self._all_chunks: list[DocumentChunk] = [] - self._chunk_embeddings = None # numpy matrix + self._chunk_embeddings: Any | None = None # numpy matrix @property def available(self) -> bool: if self._available is None: try: - from sentence_transformers import SentenceTransformer - import numpy as np + from sentence_transformers import SentenceTransformer # noqa: F401 - availability probe + import numpy as np # noqa: F401 - availability probe + self._available = True except ImportError: self._available = False @@ -219,7 +222,6 @@ def _rebuild_index(self): self._load_model() if self._model: try: - import numpy as np texts = [c.text for c in self._all_chunks] self._chunk_embeddings = self._model.encode(texts, convert_to_numpy=True) for i, chunk in enumerate(self._all_chunks): @@ -295,7 +297,10 @@ def retrieve( import numpy as np self._load_model() - query_emb = self._model.encode([query], convert_to_numpy=True)[0] + model = self._model + if model is None: + raise RuntimeError("RAG embedding model failed to load") + query_emb = model.encode([query], convert_to_numpy=True)[0] # Filter embeddings for candidates if doc_id: @@ -461,7 +466,7 @@ def build_context_prompt( lines = ["\n## DOCUMENT CONTEXT (from uploaded files):\n"] total_chars = 0 - for i, ctx in enumerate(results, 1): + for ctx in results: text = ctx.text[:500] # Truncate per result if total_chars + len(text) > max_chars: break @@ -483,6 +488,27 @@ def get_stats(self) -> dict: "model": self.model_name if self.available else "keyword_fallback", } + def purge_session(self, session_id: str) -> dict[str, int]: + """Evict document chunks explicitly owned by a deleted session.""" + owned_document_ids = [ + document_id + for document_id, chunks in self._documents.items() + if any( + str(chunk.metadata.get("session_id") or chunk.metadata.get("sessionId") or "") + == session_id + for chunk in chunks + ) + ] + removed_chunks = sum(len(self._documents.get(document_id) or []) for document_id in owned_document_ids) + for document_id in owned_document_ids: + self._documents.pop(document_id, None) + if owned_document_ids: + self._rebuild_index() + return { + "documents_evicted": len(owned_document_ids), + "chunks_evicted": removed_chunks, + } + # ── Singleton ──────────────────────────────────────────────────────────── @@ -490,7 +516,16 @@ def get_stats(self) -> dict: def get_rag_pipeline() -> RAGPipeline: + from backend import config + + if config.PRODUCTION_MODE or not config.RAG_ENABLED: + raise RAGDisabledError( + "Legacy RAG is unavailable until retrieval is owner-scoped and lifecycle-managed" + ) global _pipeline if _pipeline is None: _pipeline = RAGPipeline() + from backend.services.lifecycle import get_lifecycle_coordinator + + get_lifecycle_coordinator().register_purger("legacy_rag", _pipeline.purge_session) return _pipeline diff --git a/backend/excel/chart_engine.py b/backend/excel/chart_engine.py index 17d33e9..5b60d02 100644 --- a/backend/excel/chart_engine.py +++ b/backend/excel/chart_engine.py @@ -147,7 +147,6 @@ def detect_chart_opportunities(operations: list[dict], summary: str = "") -> lis first_label = labels[0]["cell"] last_data = adjacent_data[-1]["cell"] data_range = f"{first_label}:{last_data}" - max_row = max(_parse_row(d["cell"]) for d in adjacent_data) chart_position = f"D{_parse_row(labels[0]['cell'])}" charts.append(ChartOperation( diff --git a/backend/excel/citations.py b/backend/excel/citations.py index 808ff87..14d7964 100644 --- a/backend/excel/citations.py +++ b/backend/excel/citations.py @@ -12,9 +12,10 @@ from __future__ import annotations -import re +import copy from dataclasses import dataclass, field from datetime import datetime +from threading import RLock from typing import Optional import structlog @@ -76,14 +77,39 @@ class CitationTracker: cell values back to their source documents. """ - def __init__(self): + def __init__(self, *, session_id: str = ""): + self._session_id = str(session_id or "").strip() + self._lock = RLock() self._index = CitationIndex() self._active_documents: dict[str, dict] = {} + def _ensure_active(self) -> None: + if not self._session_id: + return + from backend.services.lifecycle import get_lifecycle_coordinator + + if get_lifecycle_coordinator().writes_revoked(self._session_id): + raise RuntimeError("Session lifecycle is not active") + + def _purge_for_lifecycle(self) -> dict[str, int]: + with self._lock: + citations_evicted = len(self._index.citations) + documents_evicted = len(self._index.documents) + self._index.citations.clear() + self._index.documents.clear() + self._active_documents.clear() + return { + "citations_evicted": citations_evicted, + "documents_evicted": documents_evicted, + } + def register_document(self, doc_id: str, metadata: dict): """Register a parsed document for citation tracking.""" - self._index.documents[doc_id] = metadata - self._active_documents[doc_id] = metadata + self._ensure_active() + with self._lock: + self._ensure_active() + self._index.documents[doc_id] = metadata + self._active_documents[doc_id] = metadata logger.info("citation_document_registered", doc_id=doc_id) def track_from_retrieval( @@ -100,6 +126,8 @@ def track_from_retrieval( if not retrieval_results: return None + self._ensure_active() + # Use the highest-relevance result best = max(retrieval_results, key=lambda r: r.get("score", 0)) @@ -114,7 +142,9 @@ def track_from_retrieval( citation_type="document", ) - self._index.add(citation) + with self._lock: + self._ensure_active() + self._index.add(citation) return citation def track_from_document( @@ -127,6 +157,7 @@ def track_from_document( confidence: float = 1.0, ) -> Citation: """Directly create a citation from a known document source.""" + self._ensure_active() citation = Citation( cell=cell.upper(), source_document=doc_id, @@ -137,7 +168,9 @@ def track_from_document( timestamp=datetime.now().isoformat(), citation_type="document", ) - self._index.add(citation) + with self._lock: + self._ensure_active() + self._index.add(citation) return citation def track_assumption( @@ -146,6 +179,7 @@ def track_assumption( description: str, ) -> Citation: """Track a cell that contains a user assumption (not from a source).""" + self._ensure_active() citation = Citation( cell=cell.upper(), source_document="User Assumption", @@ -153,7 +187,9 @@ def track_assumption( timestamp=datetime.now().isoformat(), citation_type="assumption", ) - self._index.add(citation) + with self._lock: + self._ensure_active() + self._index.add(citation) return citation def track_calculation( @@ -163,6 +199,7 @@ def track_calculation( description: str = "", ) -> Citation: """Track a cell that is a calculated result.""" + self._ensure_active() citation = Citation( cell=cell.upper(), source_document="Calculated", @@ -170,7 +207,9 @@ def track_calculation( timestamp=datetime.now().isoformat(), citation_type="calculation", ) - self._index.add(citation) + with self._lock: + self._ensure_active() + self._index.add(citation) return citation def generate_cell_comments( @@ -181,17 +220,20 @@ def generate_cell_comments( Returns a list of comment operations to apply via Office.js. """ + self._ensure_active() comment_ops = [] - for citation in self._index.citations: - comment_text = self._format_comment(citation) - if comment_text: - comment_ops.append({ - "type": "comment", - "cell": citation.cell, - "comment": comment_text, - "author": "Finance AI", - }) + with self._lock: + self._ensure_active() + for citation in self._index.citations: + comment_text = self._format_comment(citation) + if comment_text: + comment_ops.append({ + "type": "comment", + "cell": citation.cell, + "comment": comment_text, + "author": "Finance AI", + }) logger.info("citation_comments_generated", count=len(comment_ops)) return comment_ops @@ -206,61 +248,71 @@ def annotate_operations( Adds citation_source to each operation that can be traced back to a document. """ + self._ensure_active() if not parsed_doc: return operations doc_source = parsed_doc.get("filename", "Unknown Document") tables = parsed_doc.get("tables", []) - for op in operations: - cell = op.get("cell", "") - value = op.get("value") - op_type = op.get("type", "") - - # Input values that might come from a document - if op_type == "input" and value is not None: - # Try to find this value in the document tables - match = self._find_value_in_tables(value, tables) - if match: - citation = self.track_from_document( - cell=cell, - doc_id=doc_source, - page=match.get("page"), - table_title=match.get("title"), - source_text=match.get("context", ""), - ) - op["citation"] = { - "source": doc_source, - "page": match.get("page"), - "table": match.get("title"), - } - - # Formulas are calculations - elif op_type == "formula" and op.get("formula"): - label = op.get("label", "") - self.track_calculation(cell, op["formula"], label) + with self._lock: + self._ensure_active() + for op in operations: + cell = op.get("cell", "") + value = op.get("value") + op_type = op.get("type", "") + + # Input values that might come from a document + if op_type == "input" and value is not None: + # Try to find this value in the document tables + match = self._find_value_in_tables(value, tables) + if match: + self.track_from_document( + cell=cell, + doc_id=doc_source, + page=match.get("page"), + table_title=match.get("title"), + source_text=match.get("context", ""), + ) + op["citation"] = { + "source": doc_source, + "page": match.get("page"), + "table": match.get("title"), + } + + # Formulas are calculations + elif op_type == "formula" and op.get("formula"): + label = op.get("label", "") + self.track_calculation(cell, op["formula"], label) return operations def get_index(self) -> CitationIndex: - return self._index + """Return a detached snapshot so callers cannot bypass lifecycle fencing.""" + self._ensure_active() + with self._lock: + self._ensure_active() + return copy.deepcopy(self._index) def get_stats(self) -> dict: - return { - "total_citations": len(self._index.citations), - "documents": len(self._index.documents), - "by_type": { - "document": sum(1 for c in self._index.citations if c.citation_type == "document"), - "calculation": sum(1 for c in self._index.citations if c.citation_type == "calculation"), - "assumption": sum(1 for c in self._index.citations if c.citation_type == "assumption"), - }, - } + self._ensure_active() + with self._lock: + self._ensure_active() + return { + "total_citations": len(self._index.citations), + "documents": len(self._index.documents), + "by_type": { + "document": sum(1 for c in self._index.citations if c.citation_type == "document"), + "calculation": sum(1 for c in self._index.citations if c.citation_type == "calculation"), + "assumption": sum(1 for c in self._index.citations if c.citation_type == "assumption"), + }, + } # ── Internal ───────────────────────────────────────────────────── def _format_comment(self, citation: Citation) -> str: """Format a citation as an Excel cell comment.""" - lines = [f"[Finance AI Citation]"] + lines = ["[Finance AI Citation]"] if citation.citation_type == "document": lines.append(f"Source: {citation.source_document}") @@ -271,11 +323,11 @@ def _format_comment(self, citation: Citation) -> str: if citation.confidence < 1.0: lines.append(f"Confidence: {citation.confidence:.0%}") elif citation.citation_type == "assumption": - lines.append(f"Type: User Assumption") + lines.append("Type: User Assumption") if citation.source_text: lines.append(f"Note: {citation.source_text}") elif citation.citation_type == "calculation": - lines.append(f"Type: Calculated Value") + lines.append("Type: Calculated Value") if citation.timestamp: lines.append(f"Generated: {citation.timestamp[:19]}") @@ -305,12 +357,42 @@ def _find_value_in_tables(self, value, tables: list) -> Optional[dict]: return None -# ── Singleton ──────────────────────────────────────────────────────────── - -_tracker: Optional[CitationTracker] = None - -def get_citation_tracker() -> CitationTracker: - global _tracker - if _tracker is None: - _tracker = CitationTracker() - return _tracker +# ── Session-scoped registry ────────────────────────────────────────────── + +_trackers: dict[str, CitationTracker] = {} +_trackers_lock = RLock() + + +def get_citation_tracker(session_id: str) -> CitationTracker: + """Return the tracker owned by one session and enlist it in lifecycle purge.""" + normalized_session_id = str(session_id or "").strip() + if not normalized_session_id: + raise ValueError("session_id is required for citation tracking") + from backend.services.lifecycle import get_lifecycle_coordinator + + coordinator = get_lifecycle_coordinator() + if coordinator.writes_revoked(normalized_session_id): + raise RuntimeError("Session lifecycle is not active") + with _trackers_lock: + if coordinator.writes_revoked(normalized_session_id): + raise RuntimeError("Session lifecycle is not active") + tracker = _trackers.get(normalized_session_id) + if tracker is None: + tracker = CitationTracker(session_id=normalized_session_id) + _trackers[normalized_session_id] = tracker + coordinator.register_purger("legacy_citations", purge_citation_tracker) + return tracker + + +def purge_citation_tracker(session_id: str) -> dict[str, int]: + """Evict citation content belonging to one deleted session.""" + normalized_session_id = str(session_id or "").strip() + with _trackers_lock: + tracker = _trackers.pop(normalized_session_id, None) + if tracker is None: + return {"trackers_evicted": 0, "citations_evicted": 0, "documents_evicted": 0} + evicted = tracker._purge_for_lifecycle() + return { + "trackers_evicted": 1, + **evicted, + } diff --git a/backend/excel/context_extractor.py b/backend/excel/context_extractor.py index 67acaf7..cd17224 100644 --- a/backend/excel/context_extractor.py +++ b/backend/excel/context_extractor.py @@ -98,8 +98,10 @@ def extract_context(raw_context: Optional[dict]) -> Optional[dict]: # Include formulas that differ from values (these are actual formulas) if sampled_formulas: formula_cells = {} - for r_idx, (val_row, fml_row) in enumerate(zip(sampled_values, sampled_formulas)): - for c_idx, (val, fml) in enumerate(zip(val_row, fml_row)): + for r_idx, (val_row, fml_row) in enumerate( + zip(sampled_values, sampled_formulas, strict=False) + ): + for c_idx, (val, fml) in enumerate(zip(val_row, fml_row, strict=False)): if fml and str(fml) != str(val) and str(fml).startswith("="): col_letter = _col_idx_to_letter(c_idx) cell_ref = f"{col_letter}{r_idx + 1}" @@ -167,7 +169,7 @@ def _detect_data_patterns(context: dict) -> list[str]: - "Contains date columns (likely time series data)" - "Looks like a loan/debt schedule" """ - patterns = [] + patterns: list[str] = [] headers_all = [] for sheet_headers in context.get("headers", {}).values(): headers_all.extend([h.lower() for h in sheet_headers if h]) diff --git a/backend/excel/diff_engine.py b/backend/excel/diff_engine.py index 9eb40f3..53fe8f6 100644 --- a/backend/excel/diff_engine.py +++ b/backend/excel/diff_engine.py @@ -7,10 +7,11 @@ from __future__ import annotations import time +import threading import uuid +from collections import deque from dataclasses import dataclass, field from typing import Optional -from collections import deque @dataclass @@ -65,10 +66,25 @@ def to_dict(self) -> dict: class DiffEngine: """Manages change tracking and undo/redo for a session.""" - def __init__(self, max_history: int = 50): + def __init__(self, max_history: int = 50, *, session_id: str = ""): + self._session_id = str(session_id or "").strip() + self._lock = threading.RLock() self.undo_stack: deque[ChangeSet] = deque(maxlen=max_history) self.redo_stack: deque[ChangeSet] = deque(maxlen=max_history) + def _ensure_active(self) -> None: + if not self._session_id: + return + from backend.services.lifecycle import get_lifecycle_coordinator + + if get_lifecycle_coordinator().writes_revoked(self._session_id): + raise RuntimeError("Session lifecycle is not active") + + def _purge_for_lifecycle(self) -> None: + with self._lock: + self.undo_stack.clear() + self.redo_stack.clear() + def create_changeset( self, operations: list[dict], @@ -91,6 +107,7 @@ def create_changeset( Returns: A ChangeSet ready to be applied """ + self._ensure_active() snapshot = snapshot or {} changes = [] @@ -120,79 +137,119 @@ def create_changeset( metadata=metadata or {}, ) + self._ensure_active() return changeset def push(self, changeset: ChangeSet): """Push a changeset to the undo stack (called after applying changes).""" - self.undo_stack.append(changeset) - # Clear redo stack when new changes are made - self.redo_stack.clear() + self._ensure_active() + with self._lock: + self._ensure_active() + self.undo_stack.append(changeset) + # Clear redo stack when new changes are made + self.redo_stack.clear() def get_undo_operations(self) -> Optional[ChangeSet]: """ Pop the last changeset and return reverse operations for undo. Returns None if nothing to undo. """ - if not self.undo_stack: - return None - - changeset = self.undo_stack.pop() - self.redo_stack.append(changeset) - - # Create reverse changeset - reverse_changes = [] - for change in changeset.changes: - reverse = CellChange( - cell=change.cell, - old_value=change.new_value, - old_formula=change.new_formula, - old_format=change.new_format, - new_value=change.old_value, - new_formula=change.old_formula, - new_format=change.old_format, - change_type="clear" if change.change_type == "create" else "modify", + self._ensure_active() + with self._lock: + self._ensure_active() + if not self.undo_stack: + return None + + changeset = self.undo_stack.pop() + self.redo_stack.append(changeset) + + # Create reverse changeset + reverse_changes = [] + for change in changeset.changes: + reverse = CellChange( + cell=change.cell, + old_value=change.new_value, + old_formula=change.new_formula, + old_format=change.new_format, + new_value=change.old_value, + new_formula=change.old_formula, + new_format=change.old_format, + change_type="clear" if change.change_type == "create" else "modify", + ) + reverse_changes.append(reverse) + + return ChangeSet( + prompt=f"Undo: {changeset.prompt}", + summary=f"Undid: {changeset.summary}", + changes=reverse_changes, ) - reverse_changes.append(reverse) - - return ChangeSet( - prompt=f"Undo: {changeset.prompt}", - summary=f"Undid: {changeset.summary}", - changes=reverse_changes, - ) def get_redo_operations(self) -> Optional[ChangeSet]: """ Pop from redo stack and return operations to re-apply. Returns None if nothing to redo. """ - if not self.redo_stack: - return None + self._ensure_active() + with self._lock: + self._ensure_active() + if not self.redo_stack: + return None - changeset = self.redo_stack.pop() - self.undo_stack.append(changeset) - return changeset + changeset = self.redo_stack.pop() + self.undo_stack.append(changeset) + return changeset def get_history(self, limit: int = 20) -> list[dict]: """Return recent change history for the audit trail UI.""" - history = list(self.undo_stack)[-limit:] - return [cs.to_dict() for cs in reversed(history)] + self._ensure_active() + with self._lock: + self._ensure_active() + history = list(self.undo_stack)[-limit:] + return [cs.to_dict() for cs in reversed(history)] @property def can_undo(self) -> bool: - return len(self.undo_stack) > 0 + self._ensure_active() + with self._lock: + self._ensure_active() + return len(self.undo_stack) > 0 @property def can_redo(self) -> bool: - return len(self.redo_stack) > 0 + self._ensure_active() + with self._lock: + self._ensure_active() + return len(self.redo_stack) > 0 # ── Session storage ────────────────────────────────────────────────────── # Session-id → DiffEngine mapping _sessions: dict[str, DiffEngine] = {} +_sessions_lock = threading.RLock() def get_diff_engine(session_id: str) -> DiffEngine: """Get or create a DiffEngine for a session.""" - if session_id not in _sessions: - _sessions[session_id] = DiffEngine() - return _sessions[session_id] + normalized_session_id = str(session_id or "").strip() + if not normalized_session_id: + raise ValueError("session_id is required for diff history") + from backend.services.lifecycle import get_lifecycle_coordinator + + coordinator = get_lifecycle_coordinator() + if coordinator.writes_revoked(normalized_session_id): + raise RuntimeError("Session lifecycle is not active") + with _sessions_lock: + if coordinator.writes_revoked(normalized_session_id): + raise RuntimeError("Session lifecycle is not active") + if normalized_session_id not in _sessions: + _sessions[normalized_session_id] = DiffEngine(session_id=normalized_session_id) + return _sessions[normalized_session_id] + + +def purge_diff_engine(session_id: str) -> dict[str, int]: + """Evict one session's in-memory audit and undo history.""" + with _sessions_lock: + removed = _sessions.pop(str(session_id), None) + if removed is not None: + removed._purge_for_lifecycle() + return {"engines_evicted": int(removed is not None)} diff --git a/backend/excel/fingerprint.py b/backend/excel/fingerprint.py index b72d8ac..346b528 100644 --- a/backend/excel/fingerprint.py +++ b/backend/excel/fingerprint.py @@ -13,11 +13,17 @@ import re from collections import Counter, defaultdict -from dataclasses import dataclass, field -from typing import Optional +from dataclasses import dataclass +from typing import Any, Optional import structlog +from backend.config import ( + FINGERPRINT_MAX_BLOCK_COMPARISONS, + FINGERPRINT_MAX_CELLS, + FINGERPRINT_MAX_TEXT_BYTES, +) + logger = structlog.get_logger() @@ -78,6 +84,10 @@ class SheetFingerprint: CELL_REF_PATTERN = re.compile(r'(? None: + if not isinstance(cell_snapshot, dict): + raise FingerprintBudgetExceeded("Fingerprint snapshot must be an object") + cell_count = len(cell_snapshot) + if cell_count > FINGERPRINT_MAX_CELLS: + raise FingerprintBudgetExceeded( + f"Fingerprint cell budget exceeded: {cell_count} > {FINGERPRINT_MAX_CELLS}" + ) + + text_bytes = 0 + keyword_labels = 0 + block_keywords = tuple(keyword for keywords in BLOCK_KEYWORDS.values() for keyword in keywords) + for ref, info in cell_snapshot.items(): + if not isinstance(info, dict): + raise FingerprintBudgetExceeded(f"Fingerprint cell {ref!s} must be an object") + text_bytes += len(str(ref).encode("utf-8", errors="replace")) + for key in ("value", "formula"): + value = info.get(key) + if isinstance(value, str): + text_bytes += len(value.encode("utf-8", errors="replace")) + if text_bytes > FINGERPRINT_MAX_TEXT_BYTES: + raise FingerprintBudgetExceeded( + f"Fingerprint text budget exceeded: {text_bytes} > {FINGERPRINT_MAX_TEXT_BYTES}" + ) + value = info.get("value") + if isinstance(value, str): + lowered = value.strip().lower() + if any(keyword in lowered for keyword in block_keywords): + keyword_labels += 1 + attempted = keyword_labels * cell_count + if attempted > FINGERPRINT_MAX_BLOCK_COMPARISONS: + raise FingerprintBudgetExceeded( + "Fingerprint block-detection complexity budget exceeded: " + f"{attempted} > {FINGERPRINT_MAX_BLOCK_COMPARISONS}" + ) + + def fingerprint_to_prompt_context(fp: SheetFingerprint) -> str: """Convert a fingerprint to a concise LLM prompt context block.""" lines = [ f"\n## CURRENT SHEET CONTEXT: \"{fp.sheet_name}\"", - f"", + "", f"**Summary**: {fp.summary}", - f"", + "", f"**Statistics**: {fp.total_cells} cells ({fp.input_cells} inputs, {fp.formula_cells} formulas, {fp.label_cells} labels)", "", ] @@ -333,16 +382,19 @@ def _infer_column_types(cells: dict[str, CellInfo]) -> dict[str, str]: return result -def _find_key_inputs(cells: dict[str, CellInfo], deps: dict) -> list[dict]: +def _find_key_inputs( + cells: dict[str, CellInfo], + deps: dict[str, list[str]], +) -> list[dict[str, Any]]: """Find the most referenced input cells (assumptions).""" # Count how many formulas reference each cell - ref_counts = Counter() + ref_counts: Counter[str] = Counter() for cell_refs in deps.values(): for ref in cell_refs: ref_counts[ref] += 1 # Get input cells sorted by reference count - inputs = [] + inputs: list[dict[str, Any]] = [] for ref, cell in cells.items(): if cell.cell_type == "input": count = ref_counts.get(ref, 0) @@ -356,7 +408,7 @@ def _find_key_inputs(cells: dict[str, CellInfo], deps: dict) -> list[dict]: "data_type": cell.data_type, }) - inputs.sort(key=lambda x: x["references"], reverse=True) + inputs.sort(key=lambda item: int(item["references"]), reverse=True) return inputs[:10] diff --git a/backend/excel/formatting_engine.py b/backend/excel/formatting_engine.py index 34db948..3ae2b5f 100644 --- a/backend/excel/formatting_engine.py +++ b/backend/excel/formatting_engine.py @@ -13,11 +13,14 @@ from __future__ import annotations from dataclasses import dataclass -from typing import Optional +from typing import Optional, cast from copy import deepcopy from backend.config import IB_FORMAT +IB_FONT_NAME = cast(str, IB_FORMAT["font_name"]) +IB_FONT_SIZE = cast(int, IB_FORMAT["font_size"]) + # ── Formatting Profiles ────────────────────────────────────────────────── @@ -46,20 +49,20 @@ class FormatProfile: "input": FormatProfile( name="Hard-coded Input", font_color=f"#{IB_FORMAT['input_font_color']}", - font_name=IB_FORMAT["font_name"], - font_size=IB_FORMAT["font_size"], + font_name=IB_FONT_NAME, + font_size=IB_FONT_SIZE, ), "formula": FormatProfile( name="Formula Cell", font_color=f"#{IB_FORMAT['formula_font_color']}", - font_name=IB_FORMAT["font_name"], - font_size=IB_FORMAT["font_size"], + font_name=IB_FONT_NAME, + font_size=IB_FONT_SIZE, ), "link": FormatProfile( name="Cross-Sheet Link", font_color=f"#{IB_FORMAT['link_font_color']}", - font_name=IB_FORMAT["font_name"], - font_size=IB_FORMAT["font_size"], + font_name=IB_FONT_NAME, + font_size=IB_FONT_SIZE, ), # ── Headers ────────────────────────────────────────────────────── @@ -67,28 +70,28 @@ class FormatProfile: name="Column Header", bold=True, bg_color=f"#{IB_FORMAT['header_bg_color']}", - font_name=IB_FORMAT["font_name"], - font_size=IB_FORMAT["font_size"], + font_name=IB_FONT_NAME, + font_size=IB_FONT_SIZE, bottom_border="thin", alignment="center", ), "section_header": FormatProfile( name="Section Header", bold=True, - font_name=IB_FORMAT["font_name"], + font_name=IB_FONT_NAME, font_size=11, bottom_border="thin", ), "row_label": FormatProfile( name="Row Label", - font_name=IB_FORMAT["font_name"], - font_size=IB_FORMAT["font_size"], + font_name=IB_FONT_NAME, + font_size=IB_FONT_SIZE, alignment="left", ), "row_label_indented": FormatProfile( name="Indented Row Label", - font_name=IB_FORMAT["font_name"], - font_size=IB_FORMAT["font_size"], + font_name=IB_FONT_NAME, + font_size=IB_FONT_SIZE, alignment="left", indent=1, ), @@ -97,22 +100,22 @@ class FormatProfile: "subtotal": FormatProfile( name="Subtotal Row", bold=True, - font_name=IB_FORMAT["font_name"], - font_size=IB_FORMAT["font_size"], + font_name=IB_FONT_NAME, + font_size=IB_FONT_SIZE, bottom_border="thin", ), "total": FormatProfile( name="Total Row", bold=True, - font_name=IB_FORMAT["font_name"], - font_size=IB_FORMAT["font_size"], + font_name=IB_FONT_NAME, + font_size=IB_FONT_SIZE, bottom_border="double", ), "grand_total": FormatProfile( name="Grand Total Row", bold=True, - font_name=IB_FORMAT["font_name"], - font_size=IB_FORMAT["font_size"], + font_name=IB_FONT_NAME, + font_size=IB_FONT_SIZE, top_border="thin", bottom_border="double", ), diff --git a/backend/excel/formula_analyzer.py b/backend/excel/formula_analyzer.py index 12a1ab5..668563c 100644 --- a/backend/excel/formula_analyzer.py +++ b/backend/excel/formula_analyzer.py @@ -7,16 +7,23 @@ from __future__ import annotations +import json import re from typing import Any from backend.excel.formula_parser import ( FUNCTION_SPECS, + FormulaResourceLimitError, build_dependency_graph, detect_circular_references, - extract_cell_references, ) -from backend.config import FORMULA_ENGINE +from backend.config import ( + FORMULA_ENGINE, + WORKBOOK_WORKER_MAX_COMMAND_CELLS, + WORKBOOK_WORKER_MAX_FORMULA_REFERENCES, + WORKBOOK_WORKER_MAX_OUTPUT_BYTES, + WORKBOOK_WORKER_MAX_PAYLOAD_BYTES, +) from backend.models.operations import FormulaAnalysisResult, FormulaCellAnalysis @@ -29,6 +36,10 @@ STRUCTURED_REF_RE = re.compile(r"\b([A-Za-z_][A-Za-z0-9_]*)\[[^\]]+\]") +class FormulaAnalysisUnavailable(RuntimeError): + """The configured isolated formula engine failed or rejected its input.""" + + def analyze_formula_payload( *, formulas: list[dict[str, Any]] | None = None, @@ -36,9 +47,17 @@ def analyze_formula_payload( workbook_context: dict[str, Any] | None = None, ) -> FormulaAnalysisResult: """Analyze formulas collected from explicit formulas, ops, and context.""" + _validate_python_fallback_input(formulas, operations, workbook_context) formula_items = _collect_formula_items(formulas, operations, workbook_context) + if len(formula_items) > WORKBOOK_WORKER_MAX_COMMAND_CELLS: + raise FormulaResourceLimitError( + f"Formula cell budget exceeded: {len(formula_items)} > {WORKBOOK_WORKER_MAX_COMMAND_CELLS}" + ) graph_ops = [{"cell": item["address"], "formula": item["formula"]} for item in formula_items] - graph = build_dependency_graph(graph_ops) + graph = build_dependency_graph( + graph_ops, + max_reference_edges=WORKBOOK_WORKER_MAX_FORMULA_REFERENCES, + ) dependents = _reverse_graph(graph) cycles = detect_circular_references(graph) @@ -88,7 +107,7 @@ def analyze_formula_payload( finance_warnings.extend(_finance_model_context_warnings(formula_items, workbook_context or {})) - return FormulaAnalysisResult( + result = FormulaAnalysisResult( engine="python_fallback", hyperformula_available=False, univer_available=False, @@ -98,6 +117,15 @@ def analyze_formula_payload( financeWarnings=finance_warnings, source_project="Python fallback analyzer", ) + encoded_result = json.dumps( + {"ok": True, "result": result.model_dump(by_alias=True)}, + separators=(",", ":"), + ).encode("utf-8") + if len(encoded_result) > WORKBOOK_WORKER_MAX_OUTPUT_BYTES: + raise FormulaResourceLimitError( + f"Formula analysis output budget exceeded: {len(encoded_result)} > {WORKBOOK_WORKER_MAX_OUTPUT_BYTES}" + ) + return result def analyze_formula_payload_best( @@ -106,26 +134,73 @@ def analyze_formula_payload_best( operations: list[dict[str, Any]] | None = None, workbook_context: dict[str, Any] | None = None, ) -> FormulaAnalysisResult: - """Use the configured worker formula engine first, then Python fallback.""" + """Use the configured engine without crossing isolation boundaries on failure.""" if FORMULA_ENGINE == "python_fallback": return analyze_formula_payload(formulas=formulas, operations=operations, workbook_context=workbook_context) + from backend.excel.workbook_worker import run_worker + + worker = run_worker( + "formula-analyze", + { + "formulas": formulas or [], + "operations": operations or [], + "workbook_context": workbook_context or {}, + "formula_engine": FORMULA_ENGINE, + }, + ) + if not worker.ok: + raise FormulaAnalysisUnavailable(worker.error or "Workbook worker rejected formula analysis") + try: + return FormulaAnalysisResult.model_validate(worker.result) + except (TypeError, ValueError) as exc: + raise FormulaAnalysisUnavailable("Workbook worker returned an invalid formula analysis result") from exc + + +def _validate_python_fallback_input( + formulas: list[dict[str, Any]] | None, + operations: list[dict[str, Any]] | None, + workbook_context: dict[str, Any] | None, +) -> None: + payload = { + "formulas": formulas or [], + "operations": operations or [], + "workbook_context": workbook_context or {}, + "formula_engine": "python_fallback", + } try: - from backend.excel.workbook_worker import run_worker - - worker = run_worker( - "formula-analyze", - { - "formulas": formulas or [], - "operations": operations or [], - "workbook_context": workbook_context or {}, - "formula_engine": FORMULA_ENGINE, - }, + encoded = json.dumps(payload, separators=(",", ":")).encode("utf-8") + except (TypeError, ValueError) as exc: + raise FormulaResourceLimitError("Formula analysis input is not JSON serializable") from exc + if len(encoded) > WORKBOOK_WORKER_MAX_PAYLOAD_BYTES: + raise FormulaResourceLimitError( + f"Formula analysis input budget exceeded: {len(encoded)} > {WORKBOOK_WORKER_MAX_PAYLOAD_BYTES}" + ) + + command_cells = len(formulas or []) + len((workbook_context or {}).get("cell_snapshot") or {}) + command_cells += len((workbook_context or {}).get("named_ranges") or []) + for row in (workbook_context or {}).get("sample_values") or []: + if isinstance(row, list): + command_cells += len(row) + for snapshot in (workbook_context or {}).get("range_snapshots") or []: + if not isinstance(snapshot, dict): + continue + for key in ("values", "text", "formulas"): + matrix = snapshot.get(key) + if isinstance(matrix, list): + command_cells += sum(len(row) for row in matrix if isinstance(row, list)) + if command_cells > WORKBOOK_WORKER_MAX_COMMAND_CELLS: + break + for operation in operations or []: + command_cells += 1 + matrix = operation.get("formulas") if isinstance(operation, dict) else None + if isinstance(matrix, list): + command_cells += sum(len(row) for row in matrix if isinstance(row, list)) + if command_cells > WORKBOOK_WORKER_MAX_COMMAND_CELLS: + break + if command_cells > WORKBOOK_WORKER_MAX_COMMAND_CELLS: + raise FormulaResourceLimitError( + f"Formula cell budget exceeded: {command_cells} > {WORKBOOK_WORKER_MAX_COMMAND_CELLS}" ) - if worker.ok: - return FormulaAnalysisResult.model_validate(worker.result) - except Exception: - pass - return analyze_formula_payload(formulas=formulas, operations=operations, workbook_context=workbook_context) def _collect_formula_items( @@ -144,9 +219,9 @@ def _collect_formula_items( for op in operations or []: sheet = str(op.get("sheet") or "") address = str(op.get("cell") or op.get("range") or "") - formula = op.get("formula") - if isinstance(formula, str) and formula.startswith("=") and address: - items.append({"sheet": sheet, "address": _clean_address(address), "formula": formula}) + operation_formula = op.get("formula") + if isinstance(operation_formula, str) and operation_formula.startswith("=") and address: + items.append({"sheet": sheet, "address": _clean_address(address), "formula": operation_formula}) formulas_matrix = op.get("formulas") if isinstance(formulas_matrix, list) and address: for r_idx, row in enumerate(formulas_matrix): @@ -165,8 +240,8 @@ def _collect_formula_items( snapshot = (workbook_context or {}).get("cell_snapshot") or {} for address, info in snapshot.items(): if isinstance(info, dict): - formula = info.get("formula") - if isinstance(formula, str) and formula.startswith("="): + snapshot_formula = info.get("formula") + if isinstance(snapshot_formula, str) and snapshot_formula.startswith("="): sheet = str((workbook_context or {}).get("active_sheet") or "") address_text = str(address) if "!" in address_text: @@ -175,7 +250,7 @@ def _collect_formula_items( { "sheet": sheet, "address": _clean_address(address_text), - "formula": formula, + "formula": snapshot_formula, } ) diff --git a/backend/excel/formula_engine_adapters.py b/backend/excel/formula_engine_adapters.py index 35e519a..e29d27c 100644 --- a/backend/excel/formula_engine_adapters.py +++ b/backend/excel/formula_engine_adapters.py @@ -1,8 +1,8 @@ """Formula-engine adapter status boundaries. Production analysis is handled by the workbook worker. HyperFormula is approved -as the primary formula engine when enabled; Univer and the Python analyzer remain -fallback paths. +only with an explicitly injected license; Univer is a worker-confined alternative +and Python is an explicitly selected bounded compatibility profile. """ from __future__ import annotations @@ -12,10 +12,12 @@ from backend.config import ( FORMULA_ENGINE, + HYPERFORMULA_DISTRIBUTED, HYPERFORMULA_ENABLED, HYPERFORMULA_LICENSE_APPROVED, HYPERFORMULA_LICENSE_KEY, ) +from backend.excel.hyperformula_license import is_commercial_hyperformula_key @dataclass(frozen=True) @@ -59,18 +61,26 @@ class HyperFormulaAdapter: name = "hyperformula" def __init__(self) -> None: - enabled = FORMULA_ENGINE == "hyperformula" and HYPERFORMULA_ENABLED and HYPERFORMULA_LICENSE_APPROVED + enabled = ( + FORMULA_ENGINE == "hyperformula" + and HYPERFORMULA_DISTRIBUTED + and HYPERFORMULA_ENABLED + and HYPERFORMULA_LICENSE_APPROVED + and is_commercial_hyperformula_key(HYPERFORMULA_LICENSE_KEY) + ) reason = ( - "Approved primary worker formula engine." + "Approved primary worker formula engine with an explicitly injected license key." if enabled - else "Disabled until FORMULA_ENGINE=hyperformula, HYPERFORMULA_ENABLED=true, and license approval are configured." + else ( + "Disabled until FORMULA_ENGINE=hyperformula, HYPERFORMULA_DISTRIBUTED=true, " + "HYPERFORMULA_ENABLED=true, HYPERFORMULA_LICENSE_APPROVED=true, and an approved commercial " + "HYPERFORMULA_LICENSE_KEY are configured." + ) ) - if enabled and not HYPERFORMULA_LICENSE_KEY: - reason = "Approved primary worker formula engine; local/dev runs use HyperFormula's gpl-v3 license key unless HYPERFORMULA_LICENSE_KEY is supplied." self._status = FormulaEngineAdapterStatus( name="hyperformula", source_project="HyperFormula", - license_mode="GPL-3.0/commercial", + license_mode="explicit-approved-license", enabled=enabled, reason=reason, ) diff --git a/backend/excel/formula_parser.py b/backend/excel/formula_parser.py index 4bad9ed..8979ab8 100644 --- a/backend/excel/formula_parser.py +++ b/backend/excel/formula_parser.py @@ -12,11 +12,13 @@ from __future__ import annotations import re -from dataclasses import dataclass, field -from typing import Optional +from dataclasses import dataclass +from typing import TypedDict import structlog +from backend.config import WORKBOOK_WORKER_MAX_FORMULA_REFERENCES + logger = structlog.get_logger() @@ -182,7 +184,14 @@ # ── Financial Sanity Bounds ────────────────────────────────────────────── -SANITY_CHECKS = { +class SanityBounds(TypedDict): + min: float + max: float + label: str + format: str + + +SANITY_CHECKS: dict[str, SanityBounds] = { "wacc": {"min": 0.03, "max": 0.25, "label": "WACC", "format": "%"}, "discount_rate": {"min": 0.03, "max": 0.30, "label": "Discount Rate", "format": "%"}, "tax_rate": {"min": 0.0, "max": 0.50, "label": "Tax Rate", "format": "%"}, @@ -224,13 +233,25 @@ class FormulaIssue: re.IGNORECASE, ) +EXCEL_MAX_ROWS = 1_048_576 +EXCEL_MAX_COLUMNS = 16_384 + -def extract_cell_references(formula: str) -> list[str]: +class FormulaResourceLimitError(ValueError): + """Formula parsing exceeded an Excel boundary or configured worker budget.""" + + +def extract_cell_references( + formula: str, + *, + max_references: int = WORKBOOK_WORKER_MAX_FORMULA_REFERENCES, +) -> list[str]: """Extract all cell references from a formula, expanding ranges.""" if not formula or not formula.startswith("="): return [] - refs = set() + _require_positive_budget(max_references) + refs: set[str] = set() formula_body = formula[1:] # Strip leading '=' # Remove string literals to avoid false matches @@ -240,35 +261,45 @@ def extract_cell_references(formula: str) -> list[str]: for match in RANGE_REF_PATTERN.finditer(cleaned): start_ref = match.group(1).replace('$', '').upper() end_ref = match.group(2).replace('$', '').upper() - expanded = expand_range(start_ref, end_ref) + expanded = expand_range(start_ref, end_ref, max_cells=max_references) refs.update(expanded) + _check_reference_budget(len(refs), max_references) - # Extract individual cell references (not part of ranges) - range_spans = [(m.start(), m.end()) for m in RANGE_REF_PATTERN.finditer(cleaned)] - for match in CELL_REF_PATTERN.finditer(cleaned): - # Check if this ref is part of a range - pos = match.start() - in_range = any(start <= pos < end for start, end in range_spans) - if not in_range: - ref = match.group(1).replace('$', '').upper() - refs.add(ref) + # Remove ranges before scanning scalar references. This avoids a + # range-count × reference-count containment loop on compact inputs. + scalar_reference_text = RANGE_REF_PATTERN.sub(" ", cleaned) + for match in CELL_REF_PATTERN.finditer(scalar_reference_text): + ref = match.group(1).replace('$', '').upper() + refs.add(_require_excel_cell(ref)) + _check_reference_budget(len(refs), max_references) return sorted(refs) -def expand_range(start: str, end: str) -> list[str]: +def expand_range( + start: str, + end: str, + *, + max_cells: int = WORKBOOK_WORKER_MAX_FORMULA_REFERENCES, +) -> list[str]: """Expand a range like A1:C3 into individual cell references.""" - start_match = re.match(r'^([A-Z]+)(\d+)$', start) - end_match = re.match(r'^([A-Z]+)(\d+)$', end) - if not start_match or not end_match: - return [start, end] + _require_positive_budget(max_cells) + normalized_start = _require_excel_cell(start) + normalized_end = _require_excel_cell(end) + start_match = re.match(r'^([A-Z]+)(\d+)$', normalized_start) + end_match = re.match(r'^([A-Z]+)(\d+)$', normalized_end) + if not start_match or not end_match: # pragma: no cover - guarded above + raise FormulaResourceLimitError("Invalid Excel range") start_col = col_to_num(start_match.group(1)) start_row = int(start_match.group(2)) end_col = col_to_num(end_match.group(1)) end_row = int(end_match.group(2)) - cells = [] + cell_count = (abs(end_row - start_row) + 1) * (abs(end_col - start_col) + 1) + _check_reference_budget(cell_count, max_cells) + + cells: list[str] = [] for r in range(min(start_row, end_row), max(start_row, end_row) + 1): for c in range(min(start_col, end_col), max(start_col, end_col) + 1): cells.append(f"{num_to_col(c)}{r}") @@ -276,16 +307,47 @@ def expand_range(start: str, end: str) -> list[str]: return cells +def _require_excel_cell(address: str) -> str: + normalized = str(address or "").replace("$", "").strip().upper() + match = re.fullmatch(r"([A-Z]+)(\d+)", normalized) + if not match: + raise FormulaResourceLimitError(f"Invalid Excel cell address: {address}") + col = col_to_num(match.group(1)) + row = int(match.group(2)) + if row < 1 or row > EXCEL_MAX_ROWS or col < 1 or col > EXCEL_MAX_COLUMNS: + raise FormulaResourceLimitError(f"Cell address is outside Excel worksheet bounds: {normalized}") + return f"{num_to_col(col)}{row}" + + +def _require_positive_budget(limit: int) -> None: + if not isinstance(limit, int) or isinstance(limit, bool) or limit < 1: + raise FormulaResourceLimitError("Formula reference budget must be a positive integer") + + +def _check_reference_budget(attempted: int, limit: int) -> None: + if attempted > limit: + raise FormulaResourceLimitError(f"Formula reference budget exceeded: {attempted} > {limit}") + + # ── Dependency Graph ───────────────────────────────────────────────────── -def build_dependency_graph(operations: list[dict]) -> dict[str, list[str]]: +def build_dependency_graph( + operations: list[dict], + *, + max_reference_edges: int = WORKBOOK_WORKER_MAX_FORMULA_REFERENCES, +) -> dict[str, list[str]]: """Build a directed graph: cell → [cells it depends on].""" - graph = {} + _require_positive_budget(max_reference_edges) + graph: dict[str, list[str]] = {} + reference_edges: set[tuple[str, str]] = set() for op in operations: cell = op.get("cell", "").upper() formula = op.get("formula", "") if cell and formula: - deps = extract_cell_references(formula) + deps = extract_cell_references(formula, max_references=max_reference_edges) + for dependency in deps: + reference_edges.add((cell, dependency)) + _check_reference_budget(len(reference_edges), max_reference_edges) graph[cell] = deps elif cell: graph[cell] = [] diff --git a/backend/excel/formula_tracer.py b/backend/excel/formula_tracer.py index dc48c42..7cd568f 100644 --- a/backend/excel/formula_tracer.py +++ b/backend/excel/formula_tracer.py @@ -12,13 +12,17 @@ import re from collections import defaultdict from dataclasses import dataclass, field -from typing import Optional +from typing import Any, Optional import structlog logger = structlog.get_logger() -CELL_REF_PATTERN = re.compile(r"(? label for op in operations: - cell = op.get("cell", "").upper() + sheet = _op_sheet(op) + cell = _normalize_cell(op.get("cell", "")) if cell: - cell_ops[cell] = op + key = _qualified_cell(sheet, cell) + cell_ops[key] = op if op.get("type") == "label" and op.get("value"): row_match = re.search(r'\d+', cell) if row_match: - label_map[int(row_match.group())] = str(op["value"]).strip() + label_map[(sheet, int(row_match.group()))] = str(op["value"]).strip() - target = target_cell.upper() + target = _resolve_target_key(target_cell, cell_ops) if target not in cell_ops: return FormulaTrace( root_cell=target, @@ -83,7 +90,7 @@ def trace_formula( root_op = cell_ops[target] trace = FormulaTrace( root_cell=target, - root_label=_get_label(target, root_op, label_map), + root_label=_get_label(target, _op_sheet(root_op), root_op, label_map), root_formula=root_op.get("formula", ""), ) @@ -97,8 +104,8 @@ def trace_formula( continue visited.add(cell) - op = cell_ops.get(cell) - if not op: + current_op = cell_ops.get(cell) + if not current_op: # External reference — not in current operations trace.nodes[cell] = TraceNode( cell=cell, @@ -107,25 +114,26 @@ def trace_formula( formula=None, node_type="external", depth=depth, + sheet=_split_reference(cell)[0], ) trace.input_cells.append(cell) continue - formula = op.get("formula", "") - node_type = op.get("type", "input") + formula = current_op.get("formula", "") + sheet = _op_sheet(current_op) + node_type = current_op.get("type", "input") if formula: node_type = "formula" elif node_type == "label": continue # Labels don't participate in data flow - label = _get_label(cell, op, label_map) + label = _get_label(cell, sheet, current_op, label_map) children = [] # Extract referenced cells from formula if formula: - refs = CELL_REF_PATTERN.findall(formula) - for col, row in refs: - ref = f"{col.upper()}{row}" + refs = _formula_references(formula, sheet) + for ref in refs: if ref != cell: # Skip self-references children.append(ref) if ref not in visited: @@ -134,10 +142,11 @@ def trace_formula( node = TraceNode( cell=cell, label=label, - value=op.get("value"), + value=current_op.get("value"), formula=formula if formula else None, node_type=node_type, depth=depth, + sheet=sheet, children=children, ) trace.nodes[cell] = node @@ -163,7 +172,11 @@ def trace_to_ui(trace: FormulaTrace) -> dict: Returns a tree structure suitable for rendering as an interactive dependency graph or collapsible tree view. """ - def build_node(cell: str, depth: int = 0, visited: set = None) -> dict: + def build_node( + cell: str, + depth: int = 0, + visited: set[str] | None = None, + ) -> dict[str, Any]: if visited is None: visited = set() if cell in visited or depth > 10: @@ -178,13 +191,14 @@ def build_node(cell: str, depth: int = 0, visited: set = None) -> dict: "label": cell, } - result = { + result: dict[str, Any] = { "cell": node.cell, "type": node.node_type, "label": node.label, "value": node.value, "formula": node.formula, "depth": node.depth, + "sheet": node.sheet, } if node.children: @@ -215,19 +229,25 @@ def build_reverse_trace( This answers: "If I change cell B5, what else changes?" """ - target = target_cell.upper() + cell_ops: dict[str, dict] = {} + for op in operations: + cell = _normalize_cell(op.get("cell", "")) + if cell: + cell_ops[_qualified_cell(_op_sheet(op), cell)] = op + + target = _resolve_target_key(target_cell, cell_ops) dependents = defaultdict(list) for op in operations: formula = op.get("formula", "") - cell = op.get("cell", "").upper() + sheet = _op_sheet(op) + cell = _normalize_cell(op.get("cell", "")) if not formula or not cell: continue + cell_key = _qualified_cell(sheet, cell) - refs = CELL_REF_PATTERN.findall(formula) - for col, row in refs: - ref = f"{col.upper()}{row}" - dependents[ref].append(cell) + for ref in _formula_references(formula, sheet): + dependents[ref].append(cell_key) # BFS from target affected = [] @@ -245,9 +265,10 @@ def build_reverse_trace( "cell": dep, "depends_on": current, "formula": next( - (op.get("formula", "") for op in operations if op.get("cell", "").upper() == dep), + (op.get("formula", "") for key, op in cell_ops.items() if key == dep), "", ), + "sheet": _split_reference(dep)[0], }) if dep not in visited: queue.append(dep) @@ -261,15 +282,63 @@ def build_reverse_trace( # ── Helpers ────────────────────────────────────────────────────────────── -def _get_label(cell: str, op: dict, label_map: dict[int, str]) -> str: +def _get_label(cell: str, sheet: str, op: dict, label_map: dict[tuple[str, int], str]) -> str: """Get a human-readable label for a cell.""" if op.get("type") == "label": return str(op.get("value", cell)) - row_match = re.search(r'\d+', cell) + _sheet, address = _split_reference(cell) + row_match = re.search(r'\d+', address) if row_match: row = int(row_match.group()) - if row in label_map: - return label_map[row] + if (sheet, row) in label_map: + return label_map[(sheet, row)] + if ("", row) in label_map: + return label_map[("", row)] return cell + + +def _op_sheet(op: dict) -> str: + return str(op.get("sheet") or op.get("sheetName") or op.get("worksheet") or "").strip() + + +def _normalize_cell(cell: object) -> str: + return str(cell or "").replace("$", "").upper() + + +def _qualified_cell(sheet: str, cell: str) -> str: + normalized = _normalize_cell(cell) + return f"{sheet}!{normalized}" if sheet else normalized + + +def _split_reference(reference: str) -> tuple[str, str]: + text = str(reference or "").strip().replace("$", "") + if "!" not in text: + return "", _normalize_cell(text) + sheet, cell = text.rsplit("!", 1) + return sheet.strip().strip("'"), _normalize_cell(cell) + + +def _resolve_target_key(target_cell: str, cell_ops: dict[str, dict]) -> str: + sheet, cell = _split_reference(target_cell) + target = _qualified_cell(sheet, cell) + if target in cell_ops or sheet: + return target + + matches = [key for key in cell_ops if _split_reference(key)[1] == cell] + if len(matches) == 1: + return matches[0] + return target + + +def _formula_references(formula: str, default_sheet: str) -> list[str]: + refs: list[str] = [] + for match in CELL_REF_PATTERN.finditer(str(formula or "").replace("$", "")): + sheet = match.group(1) or match.group(2) or default_sheet + col = match.group(3) or match.group(5) + row = match.group(4) or match.group(6) + if not col or not row: + continue + refs.append(_qualified_cell(str(sheet or "").strip().strip("'"), f"{col}{row}")) + return list(dict.fromkeys(refs)) diff --git a/backend/excel/formula_validator.py b/backend/excel/formula_validator.py index baf315b..fcfdf43 100644 --- a/backend/excel/formula_validator.py +++ b/backend/excel/formula_validator.py @@ -131,14 +131,14 @@ def validate_formula(formula: str, cell: str = "?") -> list[ValidationIssue]: cell=cell, issue_type="invalid_ref", description=f"Row {row_num} is out of Excel range (1-1048576)", - suggestion=f"Use a valid row number", + suggestion="Use a valid row number", )) if len(col_str) > 3 or (len(col_str) == 3 and col_str > "XFD"): issues.append(ValidationIssue( cell=cell, issue_type="invalid_ref", description=f"Column {col_str} is beyond Excel's maximum (XFD)", - suggestion=f"Use a valid column reference", + suggestion="Use a valid column reference", )) # ── Check for double operators ─────────────────────────────────── @@ -201,7 +201,7 @@ def validate_operations( f"Cell {cell_key} is written twice — " f"first as '{prev_op.get('type', '?')}', then as '{op.get('type', '?')}'" ), - suggestion=f"Remove duplicate operation or use different cells", + suggestion="Remove duplicate operation or use different cells", )) created_cells.add(cell_key) diff --git a/backend/excel/hyperformula_license.py b/backend/excel/hyperformula_license.py new file mode 100644 index 0000000..e211443 --- /dev/null +++ b/backend/excel/hyperformula_license.py @@ -0,0 +1,20 @@ +"""Shared proprietary HyperFormula license-policy helpers.""" + +from __future__ import annotations + +from typing import Any + + +NONCOMMERCIAL_HYPERFORMULA_KEYS = frozenset( + { + "gpl-v3", + "internal-use-in-handsontable", + "non-commercial-and-evaluation", + } +) + + +def is_commercial_hyperformula_key(value: Any) -> bool: + """Return whether an opaque key is present and is not a known free sentinel.""" + normalized = str(value or "").strip().lower() + return bool(normalized) and normalized not in NONCOMMERCIAL_HYPERFORMULA_KEYS diff --git a/backend/excel/macro_awareness.py b/backend/excel/macro_awareness.py index 1d6bcfb..6ab8da9 100644 --- a/backend/excel/macro_awareness.py +++ b/backend/excel/macro_awareness.py @@ -47,15 +47,36 @@ class MacroAwarenessEngine: def scan_workbook_context(self, context: dict) -> MacroInfo: """Analyze workbook context for macro presence. - - The context comes from the frontend Office.js API which can - detect VBA modules, named ranges, and protection status. + + Detection layers (most authoritative first): + 1. OOXML inventory from the worker — ``ooxml.vba_project_present`` + is set when the worker has parsed the package and seen + ``xl/vbaProject.bin`` in the ZIP. This is the only spoof-resistant + signal. + 2. Explicit ``has_macros`` / ``macro_present`` flags from the host. + 3. Filename extension. Cheapest, but spoofable — an attacker can + rename ``.xlsm`` → ``.xlsx`` and slip macros past extension + checks, so this is a hint, not the source of truth. """ info = MacroInfo() - # Check for VBA indicator + ooxml = context.get("ooxml") or context.get("workbook_ooxml") or {} + if isinstance(ooxml, dict): + if ooxml.get("vba_project_present") or ooxml.get("macroProjectPresent"): + info.has_macros = True + for name in ooxml.get("vba_modules") or ooxml.get("vbaModules") or []: + if name and name not in info.macro_modules: + info.macro_modules.append(str(name)) + + if context.get("has_macros") or context.get("macro_present"): + info.has_macros = True + macro_manifest = context.get("macro_manifest") or context.get("macroManifest") or {} + if isinstance(macro_manifest, dict) and macro_manifest.get("present"): + info.has_macros = True + + # Filename hint (least authoritative — spoofable). filename = context.get("filename", "") - if filename.lower().endswith((".xlsm", ".xlsb")): + if not info.has_macros and filename.lower().endswith((".xlsm", ".xlsb")): info.has_macros = True # Named ranges @@ -195,7 +216,7 @@ def _extract_formulas(self, context: dict) -> list[str]: """Extract all formulas from workbook context.""" formulas = [] snapshot = context.get("cell_snapshot", {}) - for ref, info in snapshot.items(): + for _ref, info in snapshot.items(): formula = info.get("formula", "") if formula and formula.startswith("="): formulas.append(formula) diff --git a/backend/excel/multi_sheet.py b/backend/excel/multi_sheet.py index 386c59e..0fe7024 100644 --- a/backend/excel/multi_sheet.py +++ b/backend/excel/multi_sheet.py @@ -78,7 +78,7 @@ def analyze(self, workbook_context: dict) -> WorkbookMap: labels = [] data_count = 0 - for ref, cell_info in cells.items(): + for _ref, cell_info in cells.items(): if isinstance(cell_info, dict): formula = cell_info.get("formula", "") value = cell_info.get("value", "") diff --git a/backend/excel/workbook_files.py b/backend/excel/workbook_files.py index 62764ee..72158bb 100644 --- a/backend/excel/workbook_files.py +++ b/backend/excel/workbook_files.py @@ -1,13 +1,14 @@ """Workbook file ingestion, export, and summary diff services. -The FastAPI backend delegates to the Node workbook worker for SheetJS, ExcelJS, -HyperFormula, and Univer-backed spreadsheet intelligence, then falls back to -openpyxl when the worker is disabled or unavailable. +The FastAPI backend delegates untrusted workbook processing to the bounded Node +worker. Private Python helpers remain for trusted compatibility tooling, but a +worker rejection is never retried in-process. """ from __future__ import annotations import base64 +import binascii from io import BytesIO from typing import Any from zipfile import BadZipFile, ZipFile @@ -21,16 +22,18 @@ class WorkbookIngestionService: """Parse XLSX bytes into metadata, preview rows, formulas, and macro signals.""" def summarize(self, content: bytes, filename: str = "workbook.xlsx") -> dict[str, Any]: - try: - from backend.excel.workbook_worker import run_worker - - worker = run_worker("ingest", {"filename": filename, "content_base64": base64.b64encode(content).decode("ascii")}) - if worker.ok: - return worker.result - except Exception: - pass - - return self._summarize_python(content, filename) + _validate_workbook_archive(content, filename) + from backend.excel.workbook_worker import run_worker + + worker = run_worker( + "ingest", + {"filename": filename, "content_base64": base64.b64encode(content).decode("ascii")}, + ) + if not worker.ok: + raise RuntimeError(worker.error or "Workbook worker rejected the workbook") + if not isinstance(worker.result, dict): + raise RuntimeError("Workbook worker returned an invalid ingestion result") + return worker.result def _summarize_python(self, content: bytes, filename: str = "workbook.xlsx") -> dict[str, Any]: has_macros = _looks_macro_enabled(filename, content) @@ -83,16 +86,20 @@ class WorkbookExportService: """Generate styled XLSX artifacts from simple sheet payloads.""" def export(self, sheets: dict[str, Any], *, title: str = "Finance AI Export") -> bytes: + from backend.excel.workbook_worker import run_worker + + worker = run_worker("export", {"title": title, "sheets": sheets}) + if not worker.ok: + raise RuntimeError(worker.error or "Workbook worker rejected export") + if not isinstance(worker.result, dict): + raise RuntimeError("Workbook worker returned an invalid export result") + encoded = worker.result.get("content_base64") + if not isinstance(encoded, str) or not encoded: + raise RuntimeError("Workbook worker returned an empty export result") try: - from backend.excel.workbook_worker import run_worker - - worker = run_worker("export", {"title": title, "sheets": sheets}) - if worker.ok and worker.result and worker.result.get("content_base64"): - return base64.b64decode(worker.result["content_base64"]) - except Exception: - pass - - return self._export_python(sheets, title=title) + return base64.b64decode(encoded, validate=True) + except (binascii.Error, ValueError, TypeError) as exc: + raise RuntimeError("Workbook worker returned invalid export bytes") from exc def _export_python(self, sheets: dict[str, Any], *, title: str = "Finance AI Export") -> bytes: workbook = Workbook() @@ -136,51 +143,46 @@ class WorkbookDiffService: """Compare two workbook summaries at a sheet/formula/object inventory level.""" def diff(self, before: dict[str, Any], after: dict[str, Any]) -> dict[str, Any]: - try: - from backend.excel.workbook_worker import run_worker - - worker = run_worker("diff", {"before": before, "after": after}) - if worker.ok: - return worker.result - except Exception: - pass - - before_sheets = {sheet["name"]: sheet for sheet in before.get("sheets", [])} - after_sheets = {sheet["name"]: sheet for sheet in after.get("sheets", [])} - before_formulas = {(item.get("sheet"), item.get("address")): item.get("formula") for item in before.get("formulas", [])} - after_formulas = {(item.get("sheet"), item.get("address")): item.get("formula") for item in after.get("formulas", [])} - before_names = {item.get("name"): item.get("ref") for item in before.get("defined_names", []) if isinstance(item, dict)} - after_names = {item.get("name"): item.get("ref") for item in after.get("defined_names", []) if isinstance(item, dict)} - - return { - "engine": "python_fallback", - "added_sheets": sorted(set(after_sheets) - set(before_sheets)), - "removed_sheets": sorted(set(before_sheets) - set(after_sheets)), - "changed_dimensions": [ - { - "sheet": name, - "before": before_sheets[name].get("used_range"), - "after": after_sheets[name].get("used_range"), - } - for name in sorted(set(before_sheets) & set(after_sheets)) - if before_sheets[name].get("used_range") != after_sheets[name].get("used_range") - ], - "added_formulas": sorted([f"{sheet}!{address}" for sheet, address in set(after_formulas) - set(before_formulas)]), - "removed_formulas": sorted([f"{sheet}!{address}" for sheet, address in set(before_formulas) - set(after_formulas)]), - "changed_formulas": sorted( - [ - f"{sheet}!{address}" - for sheet, address in set(before_formulas) & set(after_formulas) - if before_formulas[(sheet, address)] != after_formulas[(sheet, address)] - ] - ), - "added_named_ranges": sorted(set(after_names) - set(before_names)), - "removed_named_ranges": sorted(set(before_names) - set(after_names)), - "changed_named_ranges": sorted( - name for name in set(before_names) & set(after_names) if before_names[name] != after_names[name] - ), - "macro_flag_changed": bool(before.get("has_macros")) != bool(after.get("has_macros")), - } + from backend.excel.workbook_worker import run_worker + + worker = run_worker("diff", {"before": before, "after": after}) + if not worker.ok: + raise RuntimeError(worker.error or "Workbook worker rejected the diff") + return _validate_workbook_diff_result(worker.result) + + +def _validate_workbook_diff_result(result: Any) -> dict[str, Any]: + required_list_fields = ( + "added_sheets", + "removed_sheets", + "changed_dimensions", + "added_formulas", + "removed_formulas", + "changed_formulas", + "added_cells", + "removed_cells", + "changed_cells", + "added_named_ranges", + "removed_named_ranges", + "changed_named_ranges", + "changed_merges", + "changed_formula_counts", + "changed_comment_counts", + "changed_table_counts", + "changed_validation_counts", + "changed_conditional_format_counts", + "changed_freeze_panes", + "changed_protection", + ) + if not isinstance(result, dict): + raise RuntimeError("Workbook worker returned an invalid diff result") + if not isinstance(result.get("engine"), str) or not result["engine"].strip(): + raise RuntimeError("Workbook worker returned an invalid diff result") + if not isinstance(result.get("macro_flag_changed"), bool): + raise RuntimeError("Workbook worker returned an invalid diff result") + if any(not isinstance(result.get(field), list) for field in required_list_fields): + raise RuntimeError("Workbook worker returned an invalid diff result") + return result def _looks_macro_enabled(filename: str, content: bytes) -> bool: @@ -193,6 +195,36 @@ def _looks_macro_enabled(filename: str, content: bytes) -> bool: return False +def _validate_workbook_archive(content: bytes, filename: str) -> None: + """Apply the upload archive budgets before any workbook parser runs.""" + max_entries = 4096 + max_uncompressed_bytes = 250 * 1024 * 1024 + max_compression_ratio = 200.0 + extension = ".xlsm" if filename.lower().endswith(".xlsm") else ".xlsx" + try: + with ZipFile(BytesIO(content)) as archive: + infos = archive.infolist() + if len(infos) > max_entries: + raise ValueError(f"{extension.upper()} package has too many entries") + total_uncompressed = 0 + for info in infos: + normalized = info.filename.replace("\\", "/") + if normalized.startswith("/") or ".." in normalized.split("/"): + raise ValueError(f"{extension.upper()} package contains an unsafe entry path") + if info.file_size < 0: + raise ValueError(f"{extension.upper()} package contains an invalid entry size") + total_uncompressed += info.file_size + if total_uncompressed > max_uncompressed_bytes: + raise ValueError(f"{extension.upper()} package exceeds the uncompressed size limit") + if total_uncompressed / max(len(content), 1) > max_compression_ratio: + raise ValueError(f"{extension.upper()} package exceeds the compression-ratio limit") + names = set(archive.namelist()) + if "[Content_Types].xml" not in names or "xl/workbook.xml" not in names: + raise ValueError(f"{extension.upper()} package is missing required OOXML parts") + except BadZipFile as exc: + raise ValueError(f"{extension.upper()} package is not a valid ZIP archive") from exc + + def _safe_sheet_name(name: str) -> str: cleaned = "".join(ch for ch in str(name or "Sheet") if ch not in r"[]:*?/\\").strip() or "Sheet" return cleaned[:31] diff --git a/backend/excel/workbook_worker.py b/backend/excel/workbook_worker.py index 832a340..a767ef3 100644 --- a/backend/excel/workbook_worker.py +++ b/backend/excel/workbook_worker.py @@ -1,14 +1,17 @@ """Bridge to the Node workbook worker. FastAPI remains the orchestrator, while SheetJS, ExcelJS, HyperFormula, and -Univer run in a small package-managed Node CLI. Every call falls back to the Python -implementation when the worker is disabled or unavailable. +Univer run in a small package-managed Node CLI. Security-sensitive callers +propagate worker rejection instead of crossing into an in-process parser. """ from __future__ import annotations import json +import os +import signal import subprocess +import threading import time from dataclasses import dataclass from pathlib import Path @@ -22,14 +25,26 @@ WORKBOOK_WORKER_ALLOWED_EXTENSIONS, WORKBOOK_WORKER_DIR, WORKBOOK_WORKER_MAX_PAYLOAD_BYTES, + WORKBOOK_WORKER_MAX_OLD_SPACE_MB, + WORKBOOK_WORKER_MAX_OUTPUT_BYTES, WORKBOOK_WORKER_MODE, WORKBOOK_WORKER_TIMEOUT_SECONDS, ) +from backend.operations.work_admission import ( + CancellationEvent, + WorkAdmissionCancelled, + WorkExecutionTimeout, + WorkKind, + current_work_context, + work_admission_controller, +) WORKER_COMMANDS = { "ingest", + "open-workbook-v1", "export", + "export-workbook-v1", "diff", "formula-analyze", "semantic-map", @@ -41,6 +56,7 @@ "openxml-qa", "export-template", "roundtrip-qa", + "roundtrip-qa-v2", "truth-model-v4", "formula-graph-v4", "simulate-operations-v5", @@ -69,6 +85,14 @@ class WorkbookWorkerUnavailable(RuntimeError): pass +class WorkbookWorkerTimeout(subprocess.TimeoutExpired, WorkExecutionTimeout): + """Workbook subprocess wall-clock timeout with shared budget taxonomy.""" + + +_IO_CHUNK_BYTES = 64 * 1024 +_PROCESS_TERMINATION_GRACE_SECONDS = 0.2 + + def worker_enabled() -> bool: return WORKBOOK_WORKER_MODE in {"sheetjs_exceljs_hyperformula", "sheetjs_exceljs_univer"} and _worker_cli().exists() @@ -81,11 +105,50 @@ def run_worker(command: str, payload: dict[str, Any]) -> WorkerResult: _validate_payload(command, payload) encoded_payload = json.dumps(payload) - env = None - if command in {"formula-analyze", "formula-graph", "formula-graph-v4", "formula-graph-v5", "semantic-map-v2", "truth-model-v4", "truth-model-v5", "simulate-operations", "simulate-operations-v5", "simulate-operations-v6", "roundtrip-qa"}: - import os + return _run_admitted_worker(command, encoded_payload) + + +def _run_admitted_worker(command: str, encoded_payload: str) -> WorkerResult: + context = current_work_context() + kind: WorkKind = context.kind if context.kind in {"workbook", "managed_xlsx"} else "workbook" + with work_admission_controller.admit( + kind, + session_id=context.session_id, + cancellation_event=context.cancellation_event, + ) as lease: + result = _invoke_worker_process(command, encoded_payload, kind=kind, cancellation_event=lease.cancellation) + if not result.ok: + lease.release("failed") + return result + - env = os.environ.copy() +def _invoke_worker_process( + command: str, + encoded_payload: str, + *, + kind: WorkKind, + cancellation_event: CancellationEvent, +) -> WorkerResult: + + env = os.environ.copy() + node_options = env.get("NODE_OPTIONS", "").strip() + heap_limit = f"--max-old-space-size={max(64, WORKBOOK_WORKER_MAX_OLD_SPACE_MB)}" + if "--max-old-space-size" not in node_options: + env["NODE_OPTIONS"] = " ".join(part for part in (node_options, heap_limit) if part) + if command in { + "formula-analyze", + "formula-graph", + "formula-graph-v4", + "formula-graph-v5", + "semantic-map-v2", + "truth-model-v4", + "truth-model-v5", + "simulate-operations", + "simulate-operations-v5", + "simulate-operations-v6", + "roundtrip-qa", + "roundtrip-qa-v2", + }: env.setdefault("FORMULA_ENGINE", FORMULA_ENGINE) env.setdefault("HYPERFORMULA_ENABLED", "true" if HYPERFORMULA_ENABLED else "false") env.setdefault("HYPERFORMULA_LICENSE_APPROVED", "true" if HYPERFORMULA_LICENSE_APPROVED else "false") @@ -93,16 +156,15 @@ def run_worker(command: str, payload: dict[str, Any]) -> WorkerResult: env.setdefault("HYPERFORMULA_LICENSE_KEY", HYPERFORMULA_LICENSE_KEY) started = time.perf_counter() - process = subprocess.run( + process = _run_bounded_process( ["node", str(_worker_cli()), command], - input=encoded_payload, - text=True, - stdout=subprocess.PIPE, - stderr=subprocess.PIPE, + input_text=encoded_payload, cwd=str(WORKBOOK_WORKER_DIR), env=env, - timeout=WORKBOOK_WORKER_TIMEOUT_SECONDS, - check=False, + timeout_seconds=WORKBOOK_WORKER_TIMEOUT_SECONDS, + max_output_bytes=WORKBOOK_WORKER_MAX_OUTPUT_BYTES, + cancellation_event=cancellation_event, + cancellation_kind=kind, ) duration_seconds = time.perf_counter() - started stdout = process.stdout.strip() @@ -120,16 +182,216 @@ def run_worker(command: str, payload: dict[str, Any]) -> WorkerResult: result_metadata = {} from backend.observability import record_worker_execution + fallback_metadata = metadata.get("fallback") + if isinstance(fallback_metadata, dict): + fallback_used = bool(fallback_metadata.get("used")) + else: + fallback_used = bool(fallback_metadata) + fallback_used = bool(metadata.get("fallbackUsed", fallback_used)) + result_fallback = result_metadata.get("fallback") + if isinstance(result_fallback, dict): + fallback_used = fallback_used or bool(result_fallback.get("used")) + else: + fallback_used = fallback_used or bool(result_fallback) + record_worker_execution( command, duration_seconds, - engine=str(metadata.get("engine") or result_metadata.get("engine") or "workbook-worker"), - fallback=bool(metadata.get("fallback") or result_metadata.get("fallback")), - warning_count=int(metadata.get("warningCount") or result_metadata.get("warningCount") or len(result_metadata.get("warnings") or [])), + engine=str(metadata.get("resultEngine") or metadata.get("engine") or result_metadata.get("engine") or "workbook-worker"), + fallback=fallback_used, + warning_count=int( + metadata.get("warningCount") or result_metadata.get("warningCount") or len(result_metadata.get("warnings") or []) + ), ) return WorkerResult(ok=True, result=decoded.get("result")) +def _run_bounded_process( + args: list[str], + *, + input_text: str, + cwd: str, + env: dict[str, str], + timeout_seconds: float, + max_output_bytes: int, + cancellation_event: CancellationEvent | None = None, + cancellation_kind: WorkKind = "workbook", +) -> subprocess.CompletedProcess[str]: + """Run a worker with bounded output and process-tree lifecycle control.""" + popen_options: dict[str, Any] + if os.name == "nt": + popen_options = { + "creationflags": getattr(subprocess, "CREATE_NEW_PROCESS_GROUP", 0), + } + else: + popen_options = {"start_new_session": True} + + process = subprocess.Popen( + args, + stdin=subprocess.PIPE, + stdout=subprocess.PIPE, + stderr=subprocess.PIPE, + cwd=cwd, + env=env, + bufsize=0, + **popen_options, + ) + stdin = process.stdin + stdout_pipe = process.stdout + stderr_pipe = process.stderr + if stdin is None or stdout_pipe is None or stderr_pipe is None: + _terminate_process_tree(process) + raise WorkbookWorkerUnavailable("Workbook worker pipes could not be created") + + limit = max(0, int(max_output_bytes)) + read_size = max(1, min(_IO_CHUNK_BYTES, limit + 1)) + stdout_buffer = bytearray() + stderr_buffer = bytearray() + overflow = threading.Event() + stdout_done = threading.Event() + stderr_done = threading.Event() + writer_done = threading.Event() + stream_errors: list[BaseException] = [] + + def drain_stream(stream, sink: bytearray, done: threading.Event) -> None: + try: + while not overflow.is_set(): + chunk = stream.read(read_size) + if not chunk: + break + remaining = limit - len(sink) + if len(chunk) > remaining: + if remaining > 0: + sink.extend(chunk[:remaining]) + overflow.set() + break + sink.extend(chunk) + except (OSError, ValueError) as exc: + stream_errors.append(exc) + finally: + done.set() + + def write_stdin() -> None: + try: + stdin.write(input_text.encode("utf-8")) + stdin.flush() + except BrokenPipeError: + pass + except OSError as exc: + if process.poll() is None: + stream_errors.append(exc) + finally: + try: + stdin.close() + except OSError: + pass + writer_done.set() + + threads = [ + threading.Thread(target=drain_stream, args=(stdout_pipe, stdout_buffer, stdout_done), daemon=True), + threading.Thread(target=drain_stream, args=(stderr_pipe, stderr_buffer, stderr_done), daemon=True), + threading.Thread(target=write_stdin, daemon=True), + ] + for thread in threads: + thread.start() + + deadline = time.monotonic() + max(0.0, float(timeout_seconds)) + timed_out = False + cancelled = False + transport_failed = False + while True: + if overflow.is_set(): + _terminate_process_tree(process) + break + if stream_errors: + transport_failed = True + _terminate_process_tree(process) + break + if cancellation_event is not None and cancellation_event.is_set(): + cancelled = True + _terminate_process_tree(process) + break + now = time.monotonic() + if now >= deadline: + timed_out = True + _terminate_process_tree(process) + break + if process.poll() is not None and stdout_done.is_set() and stderr_done.is_set(): + break + overflow.wait(min(0.05, deadline - now)) + + for stream in (stdin, stdout_pipe, stderr_pipe): + try: + stream.close() + except OSError: + pass + for thread in threads: + thread.join(timeout=1) + if process.poll() is None: + _terminate_process_tree(process) + + stdout_bytes = bytes(stdout_buffer) + stderr_bytes = bytes(stderr_buffer) + if overflow.is_set(): + raise WorkbookWorkerUnavailable(f"Workbook worker output exceeds configured {max_output_bytes} byte safety limit") + if cancelled: + cancelled_kind: WorkKind = "managed_xlsx" if cancellation_kind == "managed_xlsx" else "workbook" + reason = str(getattr(cancellation_event, "reason", "request") or "request") + raise WorkAdmissionCancelled(cancelled_kind, reason) + if timed_out: + raise WorkbookWorkerTimeout(args, timeout_seconds, output=stdout_bytes, stderr=stderr_bytes) + if transport_failed: + raise WorkbookWorkerUnavailable(f"Workbook worker I/O failed: {stream_errors[0]}") + if not writer_done.is_set(): + raise WorkbookWorkerUnavailable("Workbook worker input pipe did not close") + stdout = stdout_bytes.decode("utf-8", errors="replace") + stderr = stderr_bytes.decode("utf-8", errors="replace") + return subprocess.CompletedProcess( + args=args, + returncode=int(process.returncode or 0), + stdout=stdout, + stderr=stderr, + ) + + +def _terminate_process_tree(process: subprocess.Popen[bytes]) -> None: + """Terminate the worker and descendants, escalating to a forced tree kill.""" + if os.name == "nt": + try: + process.send_signal(signal.CTRL_BREAK_EVENT) # type: ignore[attr-defined] + except (AttributeError, OSError, ValueError): + pass + time.sleep(_PROCESS_TERMINATION_GRACE_SECONDS) + try: + subprocess.run( + ["taskkill", "/PID", str(process.pid), "/T", "/F"], + stdout=subprocess.DEVNULL, + stderr=subprocess.DEVNULL, + timeout=2, + check=False, + ) + except (OSError, subprocess.SubprocessError): + if process.poll() is None: + process.kill() + else: + try: + os.killpg(process.pid, signal.SIGTERM) + except (ProcessLookupError, PermissionError): + if process.poll() is None: + process.terminate() + time.sleep(_PROCESS_TERMINATION_GRACE_SECONDS) + try: + os.killpg(process.pid, signal.SIGKILL) + except (ProcessLookupError, PermissionError): + if process.poll() is None: + process.kill() + try: + process.wait(timeout=2) + except subprocess.TimeoutExpired: + process.kill() + process.wait(timeout=2) + + def _worker_cli() -> Path: return Path(WORKBOOK_WORKER_DIR) / "src" / "cli.mjs" @@ -143,7 +405,19 @@ def _validate_payload(command: str, payload: dict[str, Any]) -> None: suffix = Path(filename).suffix.lower() if suffix and suffix not in WORKBOOK_WORKER_ALLOWED_EXTENSIONS: raise WorkbookWorkerUnavailable(f"Workbook worker file extension is not allowed: {suffix}") - if command in {"ingest", "semantic-map", "semantic-map-v2", "truth-model-v4", "truth-model-v5", "openxml-qa", "openxml-deep-qa", "openxml-deep-qa-v2", "roundtrip-qa"}: + if command in { + "ingest", + "open-workbook-v1", + "semantic-map", + "semantic-map-v2", + "truth-model-v4", + "truth-model-v5", + "openxml-qa", + "openxml-deep-qa", + "openxml-deep-qa-v2", + "roundtrip-qa", + "roundtrip-qa-v2", + }: content = str(payload.get("content_base64") or payload.get("contentBase64") or "") if content and len(content.encode("ascii", errors="ignore")) > WORKBOOK_WORKER_MAX_PAYLOAD_BYTES: raise WorkbookWorkerUnavailable("Workbook worker file content exceeds configured safety limit") diff --git a/backend/llm/cache.py b/backend/llm/cache.py index 54d843a..f3e17cb 100644 --- a/backend/llm/cache.py +++ b/backend/llm/cache.py @@ -9,18 +9,44 @@ import hashlib import time from collections import OrderedDict -from dataclasses import dataclass, field -from typing import Optional +from dataclasses import dataclass +from typing import Optional, Protocol, cast import structlog +from backend.llm.base import LLMResponse + logger = structlog.get_logger() +class LLMCacheProtocol(Protocol): + def get( + self, + system_prompt: str, + user_prompt: str, + model: str = "", + temperature: float = 0.1, + ) -> Optional[LLMResponse]: ... + + def put( + self, + system_prompt: str, + user_prompt: str, + response: LLMResponse, + model: str = "", + temperature: float = 0.1, + ttl: float = 300.0, + ) -> None: ... + + def invalidate(self) -> None: ... + + def get_stats(self) -> dict: ... + + @dataclass class CacheEntry: """A cached LLM response.""" - response: object # LLMResponse + response: LLMResponse created_at: float ttl: float # seconds hit_count: int = 0 @@ -65,7 +91,7 @@ def get( user_prompt: str, model: str = "", temperature: float = 0.1, - ) -> Optional[object]: + ) -> Optional[LLMResponse]: """Look up a cached response. Returns None on miss or expiry.""" key = self._make_key(system_prompt, user_prompt, model, temperature) @@ -91,7 +117,7 @@ def put( self, system_prompt: str, user_prompt: str, - response: object, + response: LLMResponse, model: str = "", temperature: float = 0.1, ttl: float = 300.0, # 5 minutes default @@ -121,7 +147,6 @@ def get_stats(self) -> dict: hit_rate = (self._hits / total * 100) if total > 0 else 0.0 # Count expired entries - now = time.time() expired = sum(1 for e in self._cache.values() if e.is_expired) return { @@ -136,10 +161,10 @@ def get_stats(self) -> dict: # ── Global singleton ──────────────────────────────────────────────────── -_cache: Optional[object] = None +_cache: Optional[LLMCacheProtocol] = None -def get_llm_cache() -> object: +def get_llm_cache() -> LLMCacheProtocol: """Return the active LLM cache singleton. When ``LLM_VECTOR_CACHE_ENABLED=true`` and ``sentence-transformers`` is @@ -158,10 +183,25 @@ def get_llm_cache() -> object: if LLM_VECTOR_CACHE_ENABLED: try: from backend.llm.cache_vector import SemanticLLMCache - _cache = SemanticLLMCache(similarity_threshold=LLM_VECTOR_CACHE_THRESHOLD) + _cache = cast( + LLMCacheProtocol, + SemanticLLMCache(similarity_threshold=LLM_VECTOR_CACHE_THRESHOLD), + ) logger.info("llm_cache_mode", mode="semantic", threshold=LLM_VECTOR_CACHE_THRESHOLD) return _cache except Exception as exc: logger.warning("llm_cache_vector_unavailable", error=str(exc)) _cache = LLMCache() return _cache + + +def purge_llm_cache_for_session(_session_id: str) -> dict[str, object]: + """Fail-safe eviction for an unscoped legacy cache. + + Until cache entries carry a durable owner/session scope, deleting any + session clears the complete cache rather than risk retaining its content. + """ + cache = get_llm_cache() + stats = cache.get_stats() + cache.invalidate() + return {"scope": "global", "entries_evicted": int(stats.get("size") or 0)} diff --git a/backend/llm/cache_vector.py b/backend/llm/cache_vector.py index cedcd69..214d41f 100644 --- a/backend/llm/cache_vector.py +++ b/backend/llm/cache_vector.py @@ -14,8 +14,8 @@ import hashlib import time from collections import OrderedDict -from dataclasses import dataclass, field -from typing import Optional +from dataclasses import dataclass +from typing import Any, Optional import structlog @@ -56,8 +56,8 @@ def __init__( self._entries: OrderedDict[str, VectorCacheEntry] = OrderedDict() self._exact_cache: OrderedDict[str, VectorCacheEntry] = OrderedDict() - self._model = None - self._available = None + self._model: Any | None = None + self._available: bool | None = None self._hits_semantic = 0 self._hits_exact = 0 @@ -68,8 +68,9 @@ def available(self) -> bool: """Check if vector search is available.""" if self._available is None: try: - from sentence_transformers import SentenceTransformer - import numpy as np + from sentence_transformers import SentenceTransformer # noqa: F401 - availability probe + import numpy as np # noqa: F401 - availability probe + self._available = True except ImportError: self._available = False @@ -96,7 +97,6 @@ def _embed(self, text: str): if self._model is None: return None try: - import numpy as np return self._model.encode([text], convert_to_numpy=True)[0] except Exception: return None @@ -225,7 +225,6 @@ def get_stats(self) -> dict: total = self._hits_semantic + self._hits_exact + self._misses hit_rate = ((self._hits_semantic + self._hits_exact) / total * 100) if total > 0 else 0.0 - now = time.time() expired_exact = sum(1 for e in self._exact_cache.values() if e.is_expired) expired_semantic = sum(1 for e in self._entries.values() if e.is_expired) diff --git a/backend/llm/providers/openai.py b/backend/llm/providers/openai.py index 6747301..48998f2 100644 --- a/backend/llm/providers/openai.py +++ b/backend/llm/providers/openai.py @@ -1,8 +1,9 @@ """OpenAI provider — GPT-4o fallback for when Ollama/Anthropic aren't available.""" from __future__ import annotations + import time -from typing import Optional +from typing import Any, Optional import structlog @@ -33,12 +34,15 @@ async def generate( model = model or OPENAI_MODEL client = AsyncOpenAI(api_key=self.api_key) - kwargs = dict( - model=model, - temperature=temperature, - max_tokens=max_tokens, - messages=[{"role": m.role, "content": m.content} for m in messages], - ) + kwargs: dict[str, Any] = { + "model": model, + "temperature": temperature, + "max_tokens": max_tokens, + "messages": [{"role": message.role, "content": message.content} for message in messages], + # OpenAI storage is opt-out. Closure requests must never be retained + # implicitly by the cloud provider. + "store": False, + } if json_mode: kwargs["response_format"] = {"type": "json_object"} diff --git a/backend/llm/router.py b/backend/llm/router.py index 4c3e7d0..5511c4c 100644 --- a/backend/llm/router.py +++ b/backend/llm/router.py @@ -6,7 +6,7 @@ from __future__ import annotations import time from enum import Enum -from typing import Optional, AsyncGenerator +from typing import Optional import structlog @@ -17,13 +17,13 @@ from backend.llm.providers.openai import OpenAIProvider from backend.llm.providers.lmstudio import LMStudioProvider from backend.config import ( - DEFAULT_PROVIDER, OLLAMA_MODEL_SIMPLE, OLLAMA_MODEL_MEDIUM, LMSTUDIO_MODEL, ANTHROPIC_MODEL_MEDIUM, ANTHROPIC_MODEL_COMPLEX, OPENAI_MODEL, + LLM_EXACT_CACHE_ENABLED, ) logger = structlog.get_logger() @@ -84,6 +84,28 @@ def __init__(self): ], } + def _policy_approved_routes( + self, + complexity: Complexity, + provider_override: Optional[str], + model_override: Optional[str], + ) -> list[tuple[str, str]]: + """Resolve only routes allowed by the current effective AI policy.""" + from backend.api.security.ai_policy import AiPolicyViolation, get_effective_ai_policy + + policy = get_effective_ai_policy() + if provider_override: + normalized = policy.require_provider(provider_override) + if normalized not in self.providers: + raise AiPolicyViolation( + f"AI provider '{provider_override}' is not configured.", + "ai_provider_unknown", + ) + return [(normalized, model_override or "")] + + routes = self.routing_table.get(complexity, self.routing_table[Complexity.SIMPLE]) + return list(policy.approved_provider_routes(routes)) + async def generate( self, messages: list[LLMMessage], @@ -100,36 +122,42 @@ async def generate( If provider_override is set, uses only that provider (no fallback). """ + routes = self._policy_approved_routes(complexity, provider_override, model_override) + approved_provider_names = {provider_name for provider_name, _model in routes} + # ── Cache lookup (skip for complex or explicit overrides) ──── cache = get_llm_cache() - use_cache = complexity != Complexity.COMPLEX and not provider_override + use_cache = ( + LLM_EXACT_CACHE_ENABLED + and complexity != Complexity.COMPLEX + and not provider_override + and len(messages) == 2 + ) cache_ttl = {Complexity.SIMPLE: 300.0, Complexity.MEDIUM: 600.0}.get(complexity, 300.0) - if use_cache and len(messages) >= 2: + if use_cache: sys_prompt = messages[0].content if messages[0].role == "system" else "" user_prompt = messages[-1].content cached = cache.get(sys_prompt, user_prompt, model_override or "", temperature) - if cached: + if cached and getattr(cached, "provider", "") in approved_provider_names: logger.info("cache_hit", complexity=complexity.value) return cached # If user explicitly chose a provider, use it directly - if provider_override and provider_override in self.providers: - provider = self.providers[provider_override] + if provider_override: + provider_name, _route_model = routes[0] + provider = self.providers[provider_name] response = await provider.generate(messages, model_override, temperature, max_tokens, json_mode) return response - # Otherwise, try the routing table in order of preference - routes = self.routing_table.get(complexity, self.routing_table[Complexity.SIMPLE]) - last_error = None for provider_name, model in routes: - provider = self.providers.get(provider_name) - if not provider: + candidate_provider = self.providers.get(provider_name) + if not candidate_provider: continue try: - if not await provider.health_check(): + if not await candidate_provider.health_check(): logger.debug("provider_unavailable", provider=provider_name) continue @@ -141,10 +169,10 @@ async def generate( provider=provider_name, model=final_model, ) - response = await provider.generate(messages, final_model, temperature, max_tokens, json_mode) + response = await candidate_provider.generate(messages, final_model, temperature, max_tokens, json_mode) # ── Cache the response ──────────────────────────── - if use_cache and len(messages) >= 2: + if use_cache: sys_prompt = messages[0].content if messages[0].role == "system" else "" user_prompt = messages[-1].content cache.put(sys_prompt, user_prompt, response, final_model, temperature, cache_ttl) @@ -175,24 +203,26 @@ async def generate_stream( Falls back to non-streaming generate() if streaming fails. """ + routes = self._policy_approved_routes(complexity, provider_override, model_override) + approved_provider_names = {provider_name for provider_name, _model in routes} + # Check cache first — if cached, emit all at once cache = get_llm_cache() - use_cache = complexity != Complexity.COMPLEX and not provider_override - if use_cache and len(messages) >= 2: + use_cache = ( + LLM_EXACT_CACHE_ENABLED + and complexity != Complexity.COMPLEX + and not provider_override + and len(messages) == 2 + ) + if use_cache: sys_prompt = messages[0].content if messages[0].role == "system" else "" user_prompt = messages[-1].content cached = cache.get(sys_prompt, user_prompt, model_override or "", temperature) - if cached: + if cached and getattr(cached, "provider", "") in approved_provider_names: if on_token and callable(on_token): on_token(cached.content) return cached - # Resolve provider - if provider_override and provider_override in self.providers: - routes = [(provider_override, model_override or "")] - else: - routes = self.routing_table.get(complexity, self.routing_table[Complexity.SIMPLE]) - last_error = None for provider_name, model in routes: provider = self.providers.get(provider_name) @@ -223,7 +253,7 @@ async def generate_stream( ) # Cache the assembled response - if use_cache and len(messages) >= 2: + if use_cache: sys_prompt = messages[0].content if messages[0].role == "system" else "" user_prompt = messages[-1].content cache_ttl = {Complexity.SIMPLE: 300.0, Complexity.MEDIUM: 600.0}.get(complexity, 300.0) @@ -270,4 +300,3 @@ def invalidate_health_cache(self): # Singleton router = LLMRouter() - diff --git a/backend/logging_config.py b/backend/logging_config.py new file mode 100644 index 0000000..ef88265 --- /dev/null +++ b/backend/logging_config.py @@ -0,0 +1,135 @@ +"""Central structured-log configuration and fail-closed field redaction.""" + +from __future__ import annotations + +import os +import re +from collections.abc import Mapping +from typing import Any + +import structlog + + +REDACTED = "[REDACTED]" + +_SENSITIVE_KEYS = frozenset( + { + "access_token", + "api_key", + "authorization", + "client_ip", + "client_secret", + "content", + "cookie", + "credential", + "database_url", + "db", + "document_id", + "email", + "file_path", + "filename", + "formula", + "formula_text", + "id_token", + "ip", + "owner_user_id", + "pairing_code", + "password", + "path", + "plan_id", + "prompt", + "proxy_authorization", + "refresh_token", + "secret", + "session_id", + "set_cookie", + "source_content", + "source_ref", + "source_text", + "token", + "user_id", + "workbook_id", + "workbook_name", + "x_api_key", + } +) +_CREDENTIAL_VALUE_RE = re.compile( + r"(?i)\b(authorization|api[-_]?key|password|secret|token|cookie)\b" + r"(\s*[:=]\s*|\s+)(?:bearer\s+)?([^\s&,;]+)" +) +_BEARER_RE = re.compile(r"(?i)\bbearer\s+[A-Za-z0-9._~+/=-]+") +_EMAIL_RE = re.compile(r"(? str: + return re.sub(r"[^a-z0-9]+", "_", str(key).strip().lower()).strip("_") + + +def _sanitize_text(value: str) -> str: + sanitized = _CREDENTIAL_VALUE_RE.sub(lambda match: f"{match.group(1)}={REDACTED}", value) + sanitized = _BEARER_RE.sub(f"Bearer {REDACTED}", sanitized) + sanitized = _EMAIL_RE.sub(REDACTED, sanitized) + return _LOCAL_PATH_RE.sub(REDACTED, sanitized) + + +def sanitize_log_value(value: Any, *, depth: int = 0) -> Any: + """Return a JSON-safe value with sensitive nested content removed.""" + if depth >= _MAX_REDACTION_DEPTH: + return "[MAX_DEPTH]" + if value is None or isinstance(value, (bool, int, float)): + return value + if isinstance(value, str): + return _sanitize_text(value) + if isinstance(value, Mapping): + sanitized: dict[str, Any] = {} + for key, item in value.items(): + key_text = str(key) + if _normalized_key(key) in _SENSITIVE_KEYS: + sanitized[key_text] = REDACTED + else: + sanitized[key_text] = sanitize_log_value(item, depth=depth + 1) + return sanitized + if isinstance(value, (list, tuple, set, frozenset)): + return [sanitize_log_value(item, depth=depth + 1) for item in value] + if isinstance(value, BaseException): + return value.__class__.__name__ + return f"<{value.__class__.__name__}>" + + +def redact_structured_event( + _logger: Any, + _method_name: str, + event_dict: dict[str, Any], +) -> dict[str, Any]: + """Structlog processor that redacts secrets, content, and durable identifiers.""" + sanitized = sanitize_log_value(event_dict) + if not isinstance(sanitized, dict): + return {"event": "invalid_log_event"} + return sanitized + + +def configure_structured_logging() -> None: + """Configure JSON structured logging once for the process.""" + global _configured + if _configured: + return + processors: list[Any] = [ + structlog.contextvars.merge_contextvars, + structlog.processors.add_log_level, + structlog.processors.TimeStamper(fmt="iso", utc=True), + redact_structured_event, + ] + if os.getenv("STRUCTURED_LOG_FORMAT", "json").strip().lower() == "console": + processors.append(structlog.dev.ConsoleRenderer()) + else: + processors.append(structlog.processors.JSONRenderer(sort_keys=True)) + structlog.configure( + processors=processors, + cache_logger_on_first_use=True, + ) + _configured = True diff --git a/backend/main.py b/backend/main.py index f5f66c1..2c1d924 100644 --- a/backend/main.py +++ b/backend/main.py @@ -6,25 +6,52 @@ from __future__ import annotations -import io +import asyncio +import json import time from contextlib import asynccontextmanager -from typing import Optional +from typing import Any, Optional import structlog -from fastapi import FastAPI, Request, WebSocket +from fastapi import Body, FastAPI, Request, WebSocket from fastapi.middleware.cors import CORSMiddleware -from pydantic import BaseModel +from fastapi.responses import JSONResponse +from pydantic import BaseModel, Field + +from backend.logging_config import configure_structured_logging + +configure_structured_logging() from backend.config import ( - HOST, PORT, CORS_ORIGINS, API_KEY, AUTH_ENABLED, VERSION, RATE_LIMIT_RPM, - TRUSTED_PROXY_HEADERS, STATIC_KEY_ADMIN_ENABLED, - ENTERPRISE_MODE, RBAC_ENABLED, SSO_ENABLED, SSO_PROVIDER, SSO_IDP_METADATA_URL, - BYOK_ENABLED, ZERO_DATA_RETENTION, ANTHROPIC_API_KEY, OPENAI_API_KEY, + HOST, + PORT, + CORS_ORIGINS, + API_KEY, + AUTH_ENABLED, + VERSION, + RATE_LIMIT_RPM, + TRUSTED_PROXY_HEADERS, + STATIC_KEY_ADMIN_ENABLED, + DIAGNOSTIC_ENDPOINTS_PUBLIC, + PRODUCTION_MODE, + EXPERIMENTAL_ROUTES_ENABLED, + RAG_ENABLED, + ENTERPRISE_MODE, + RBAC_ENABLED, + SSO_ENABLED, + SSO_PROVIDER, + SSO_IDP_METADATA_URL, + BYOK_ENABLED, + ZERO_DATA_RETENTION, + DATA_RETENTION_HOURS, + LOCAL_PAIRING_ENABLED, + LOCAL_PAIRING_RATE_LIMIT_RPM, + ANTHROPIC_API_KEY, + OPENAI_API_KEY, validate_runtime_config, ) from backend.llm.router import router as llm_router -from backend.llm.cache import get_llm_cache +from backend.llm.cache import get_llm_cache, purge_llm_cache_for_session # Route modules from backend.api.routes.tasks import router as tasks_router, websocket_task_handler @@ -38,26 +65,80 @@ # Middleware from backend.api.middleware.auth import AuthMiddleware, validate_ws_api_key +from backend.api.middleware.drain import ( + DrainMiddleware, + drain_controller, + graceful_shutdown_timeout_seconds, +) from backend.api.middleware.error_handler import ErrorHandlerMiddleware +from backend.operations.work_admission import work_admission_controller from backend.api.middleware.rate_limiter import RateLimiterMiddleware +from backend.api.security.ownership import ( + OwnershipNotFoundError, + ownership_actor_scope, + require_session_access, +) +from backend.api.security.oidc import OIDCValidationError, get_oidc_authenticator +from backend.api.security.oidc_client import OIDCClientConfigError, get_public_oidc_client_config +from backend.api.security.local_pairing import ( + LOCAL_PAIRING_SCHEMA, + LocalPairingError, + get_local_pairing_manager, + is_loopback_peer, +) from backend.api.security.rbac import RBACMiddleware logger = structlog.get_logger() +oidc_authenticator = get_oidc_authenticator() +local_pairing_manager = get_local_pairing_manager() + + +def _request_actor(request: Request): + return getattr(request.state, "enterprise_user", None) or getattr(request.state, "user", None) + + +async def _retention_loop(interval_seconds: float) -> None: + from backend.api.security.rbac import get_zdr + + while True: + try: + await get_zdr().purge_expired() + except asyncio.CancelledError: + raise + except Exception as exc: + logger.error("retention_loop_iteration_failed", error=exc.__class__.__name__) + await asyncio.sleep(interval_seconds) # ── Lifespan ───────────────────────────────────────────────────────────── @asynccontextmanager async def lifespan(app: FastAPI): + drain_controller.start_accepting() + work_admission_controller.start_accepting() validate_runtime_config() + if LOCAL_PAIRING_ENABLED: + local_pairing_manager.ensure_ready() + from backend.services.lifecycle import get_lifecycle_coordinator + + get_lifecycle_coordinator().register_purger("llm_cache", purge_llm_cache_for_session) logger.info("server_starting", host=HOST, port=PORT, version=VERSION) health = await llm_router.health() logger.info("llm_providers_status", **health) # Log loaded rules from backend.agents.rules import get_rules_engine + rules = get_rules_engine() logger.info("ai_rules_loaded", count=len(rules.get_enabled_rules())) + retention_task: asyncio.Task[None] | None = None + if ZERO_DATA_RETENTION and DATA_RETENTION_HOURS > 0: + interval_seconds = min(max(DATA_RETENTION_HOURS * 900, 60), 3600) + retention_task = asyncio.create_task( + _retention_loop(interval_seconds), + name="session-retention-loop", + ) + if ENTERPRISE_MODE: missing = [] if SSO_ENABLED and not (SSO_PROVIDER and SSO_IDP_METADATA_URL): @@ -71,6 +152,44 @@ async def lifespan(app: FastAPI): yield + drain_controller.begin_drain() + work_admission_controller.begin_drain() + drain_snapshot = drain_controller.snapshot() + worker_snapshot = work_admission_controller.snapshot() + logger.info( + "server_draining", + in_flight=drain_snapshot.in_flight, + worker_active=worker_snapshot.active, + worker_queued=worker_snapshot.queued, + ) + drain_timeout = graceful_shutdown_timeout_seconds() + request_drained, worker_drained = await asyncio.gather( + drain_controller.wait_for_idle(drain_timeout), + work_admission_controller.wait_for_idle(drain_timeout), + ) + if not request_drained or not worker_drained: + cancelled_workers = work_admission_controller.cancel_running("shutdown") + forced_worker_drain = await work_admission_controller.wait_for_idle(min(max(drain_timeout, 0.5), 3.0)) + logger.warning( + "server_drain_timeout", + in_flight=drain_controller.snapshot().in_flight, + worker_active=work_admission_controller.snapshot().active, + cancelled_workers=cancelled_workers, + worker_termination_complete=forced_worker_drain, + timeout_seconds=drain_timeout, + ) + else: + logger.info("server_drain_complete") + + if retention_task is not None: + retention_task.cancel() + try: + await retention_task + except asyncio.CancelledError: + pass + + local_pairing_manager.revoke_all() + # Graceful shutdown — close persistent HTTP clients for name, provider in llm_router.providers.items(): if hasattr(provider, "close"): @@ -80,7 +199,11 @@ async def lifespan(app: FastAPI): except Exception: pass logger.info("server_stopping") - + # Production exits after lifespan shutdown. Resetting here keeps repeated + # in-process app lifespans deterministic for native-host and contract tests. + drain_controller.start_accepting() + if work_admission_controller.wait_for_idle_blocking(0): + work_admission_controller.start_accepting() # ── App ────────────────────────────────────────────────────────────────── @@ -102,27 +225,32 @@ async def request_metrics_middleware(request: Request, call_next): status_code = response.status_code return response finally: - from backend.observability import record_request_metrics + from backend.observability import record_request_metrics, route_template_from_scope record_request_metrics( request.method, - request.url.path, + route_template_from_scope(request.scope), status_code, time.perf_counter() - started, ) + # ── Middleware ─────────────────────────────────────────────────────────── # Starlette inserts new middleware at the head of the chain, so the LAST # call to add_middleware runs OUTERMOST. We want this execution order: -# request → CORS → ErrorHandler → Auth → RBAC → RateLimiter → app +# request → CORS → ErrorHandler → Drain → Auth → RBAC → RateLimiter → app # That means the insertion order below must be the reverse of the desired # execution order (innermost first, outermost last). app.add_middleware( RateLimiterMiddleware, default_rpm=RATE_LIMIT_RPM, enabled=True, - endpoint_limits={"/api/v1/documents": 10}, + endpoint_limits={ + "/api/v1/documents": 10, + "/auth/local/pair": LOCAL_PAIRING_RATE_LIMIT_RPM, + }, trusted_proxy_headers=TRUSTED_PROXY_HEADERS, + diagnostic_paths_public=DIAGNOSTIC_ENDPOINTS_PUBLIC, ) app.add_middleware( @@ -135,16 +263,52 @@ async def request_metrics_middleware(request: Request, call_next): api_key=API_KEY, enabled=AUTH_ENABLED, static_key_admin=STATIC_KEY_ADMIN_ENABLED, + diagnostic_paths_public=DIAGNOSTIC_ENDPOINTS_PUBLIC, + oidc_authenticator=oidc_authenticator, + local_pairing_authenticator=local_pairing_manager, ) +app.add_middleware(DrainMiddleware, controller=drain_controller) + app.add_middleware(ErrorHandlerMiddleware) app.add_middleware( CORSMiddleware, allow_origins=CORS_ORIGINS, allow_credentials=True, - allow_methods=["*"], - allow_headers=["*"], + # Pin to the verbs the routes actually use. Wildcard "*" + credentials is + # a credential-leak shape; pinning here is the defense-in-depth backstop + # for misconfigured CORS_ORIGINS lists. + allow_methods=["GET", "POST", "PUT", "PATCH", "DELETE", "OPTIONS"], + # Pin headers explicitly. Office.js and our taskpane/spreadsheet clients + # send the small set below; anything else is rejected. + allow_headers=[ + "Accept", + "Accept-Language", + "Authorization", + "Content-Type", + "Content-Disposition", + "If-Match", + "If-None-Match", + "Origin", + "X-API-Key", + "X-Apply-Token", + "X-Idempotency-Key", + "X-Requested-With", + "X-Request-ID", + "X-Session-Id", + "X-Trace-Id", + ], + expose_headers=[ + "Content-Disposition", + "X-Spreadsheet-Engine", + "X-Spreadsheet-Fidelity-Mode", + "X-Spreadsheet-Fidelity-Status", + "X-Spreadsheet-Unsupported-Preserved", + "Retry-After", + "X-Request-ID", + "X-Response-Time", + ], ) # ── Mount Routers ──────────────────────────────────────────────────────── @@ -155,21 +319,227 @@ async def request_metrics_middleware(request: Request, call_next): app.include_router(benchmarks_router) app.include_router(enterprise_router) app.include_router(agent_runs_router) -app.include_router(experiments_router) +if EXPERIMENTAL_ROUTES_ENABLED and not PRODUCTION_MODE: + app.include_router(experiments_router) + + +@app.get("/auth/oidc/config", include_in_schema=False) +async def oidc_client_config(): + """Return public-only authorization-code + PKCE metadata to clients.""" + try: + payload = get_public_oidc_client_config() + except OIDCClientConfigError: + logger.error("oidc_client_config_invalid") + return JSONResponse( + status_code=503, + content={"error": {"code": "OIDC_CLIENT_CONFIG_INVALID", "message": "Sign-in is unavailable."}}, + headers={"Cache-Control": "no-store", "Pragma": "no-cache"}, + ) + return JSONResponse( + content=payload, + headers={"Cache-Control": "no-store", "Pragma": "no-cache"}, + ) + + +_LOCAL_PAIRING_MAX_BODY_BYTES = 4096 +_LOCAL_PAIRING_NO_STORE_HEADERS = {"Cache-Control": "no-store", "Pragma": "no-cache"} + + +@app.post("/auth/local/pair", include_in_schema=False) +async def pair_local_client(request: Request): + """Consume a desktop-issued one-time code from an actual loopback peer.""" + if not is_loopback_peer(request.client.host if request.client else None): + return _local_pairing_error_response( + LocalPairingError( + "LOOPBACK_REQUIRED", + status_code=403, + message="Local pairing is restricted to a loopback connection.", + ) + ) + + content_type = (request.headers.get("content-type") or "").split(";", 1)[0].strip().lower() + if content_type != "application/json": + return _local_pairing_error_response( + LocalPairingError( + "PAIRING_REQUEST_INVALID", + status_code=415, + message="Local pairing requires an application/json request.", + ) + ) + try: + declared_size = int(request.headers.get("content-length") or "0") + except ValueError: + declared_size = _LOCAL_PAIRING_MAX_BODY_BYTES + 1 + if declared_size < 0 or declared_size > _LOCAL_PAIRING_MAX_BODY_BYTES: + return _local_pairing_error_response( + LocalPairingError( + "PAIRING_REQUEST_TOO_LARGE", + status_code=413, + message="Local pairing request is too large.", + ) + ) + + body = bytearray() + async for chunk in request.stream(): + body.extend(chunk) + if len(body) > _LOCAL_PAIRING_MAX_BODY_BYTES: + return _local_pairing_error_response( + LocalPairingError( + "PAIRING_REQUEST_TOO_LARGE", + status_code=413, + message="Local pairing request is too large.", + ) + ) + try: + payload = json.loads(bytes(body)) + except (TypeError, ValueError): + payload = None + expected_fields = {"schema_version", "pairing_code", "client_nonce"} + if ( + not isinstance(payload, dict) + or set(payload) != expected_fields + or payload.get("schema_version") != LOCAL_PAIRING_SCHEMA + or not isinstance(payload.get("pairing_code"), str) + or not isinstance(payload.get("client_nonce"), str) + ): + return _local_pairing_error_response( + LocalPairingError( + "PAIRING_REQUEST_INVALID", + status_code=400, + message="Local pairing request is invalid.", + ) + ) + try: + grant = local_pairing_manager.exchange_code( + str(payload.get("pairing_code") or ""), + str(payload.get("client_nonce") or ""), + ) + except LocalPairingError as exc: + return _local_pairing_error_response(exc) + return JSONResponse(content=grant, headers=_LOCAL_PAIRING_NO_STORE_HEADERS) + + +def _local_pairing_error_response(error: LocalPairingError) -> JSONResponse: + headers = dict(_LOCAL_PAIRING_NO_STORE_HEADERS) + if error.retry_after is not None: + headers["Retry-After"] = str(max(error.retry_after, 1)) + return JSONResponse( + status_code=error.status_code, + content={ + "detail": error.public_message, + "error": {"code": error.code, "message": error.public_message}, + }, + headers=headers, + ) # ── WebSocket (must be on main app, not a sub-router) ──────────────────── +def _extract_ws_credential(websocket: WebSocket) -> tuple[str, str] | None: + """Pull one authoritative credential from a header or subprotocol. + + Query-string credentials are intentionally unsupported because URLs appear in + proxy/CDN access logs, browser history, and Referer headers. + """ + authorization = (websocket.headers.get("authorization") or "").strip() + if authorization: + scheme, separator, credential = authorization.partition(" ") + if separator and scheme.lower() == "bearer" and credential.strip(): + return "bearer", credential.strip() + return "invalid", "" + + api_key = (websocket.headers.get("x-api-key") or "").strip() + if api_key: + return "api_key", api_key + + subprotocol = websocket.headers.get("sec-websocket-protocol", "") + for token in subprotocol.split(","): + token = token.strip() + if token.startswith("bearer.") and len(token) > len("bearer."): + return "bearer", token[len("bearer.") :] + if token.startswith("api-key.") and len(token) > len("api-key."): + return "api_key", token[len("api-key.") :] + return None + + +def _extract_ws_api_key(websocket: WebSocket) -> Optional[str]: + """Compatibility helper that extracts service API keys only.""" + credential = _extract_ws_credential(websocket) + return credential[1] if credential and credential[0] == "api_key" else None + + +def _resolve_ws_actor(api_key: Optional[str]) -> dict[str, Any] | None: + """Resolve a WebSocket credential through the same actor model as HTTP auth.""" + if not api_key: + return None + if validate_ws_api_key(api_key, API_KEY): + if not STATIC_KEY_ADMIN_ENABLED: + return None + return { + "user_id": "admin", + "email": "admin@localhost", + "name": "Admin", + "role": "admin", + "auth_source": "static_api_key", + } + try: + from backend.storage import get_session_store + + enterprise_user = get_session_store().get_enterprise_user_by_api_key(api_key) + except Exception as exc: + logger.debug("enterprise_ws_key_lookup_failed", error=str(exc)) + return None + if not enterprise_user: + return None + enterprise_user["auth_source"] = "enterprise_api_key" + return enterprise_user + + +async def _resolve_ws_credential_actor(credential: tuple[str, str] | None) -> dict[str, Any] | None: + if not credential: + return None + scheme, value = credential + if scheme == "bearer": + if value.startswith("local."): + try: + return local_pairing_manager.authenticate(value) + except LocalPairingError: + return None + try: + return await oidc_authenticator.authenticate(value) + except OIDCValidationError: + return None + if scheme == "api_key": + return _resolve_ws_actor(value) + return None + + @app.websocket("/ws/tasks/{session_id}") async def websocket_task(websocket: WebSocket, session_id: str): """WebSocket for real-time pipeline progress.""" - if AUTH_ENABLED and not validate_ws_api_key(websocket.query_params.get("api_key"), API_KEY): + credential = _extract_ws_credential(websocket) + actor = await _resolve_ws_credential_actor(credential) + if AUTH_ENABLED and actor is None: await websocket.close(code=1008) return - await websocket_task_handler(websocket, session_id) + with ownership_actor_scope(actor): + try: + from backend.storage import get_session_store + + require_session_access( + get_session_store(), + session_id, + actor=actor, + create=True, + ) + except OwnershipNotFoundError: + await websocket.close(code=1008) + return + await websocket_task_handler(websocket, session_id, actor=actor) # ── Health — enhanced with diagnostics ─────────────────────────────────── + class HealthResponse(BaseModel): status: str version: str @@ -179,9 +549,68 @@ class HealthResponse(BaseModel): diagnostics: Optional[dict] = None -@app.get("/health", response_model=HealthResponse) -async def health_check(): - """Server and provider health check with deep diagnostics.""" +class LivenessResponse(BaseModel): + status: str + version: str + + +class ReadinessResponse(BaseModel): + status: str + version: str + checks: dict[str, str] + + +def _storage_readiness() -> tuple[bool, dict[str, str], dict]: + checks = {"storage": "error", "migrations": "unknown"} + try: + from backend.storage import get_session_store + + store = get_session_store() + _ = store.list_sessions(1) + migration_info = store.get_migration_status() + except Exception as exc: + logger.error("readiness_storage_failed", error_type=exc.__class__.__name__) + return False, checks, {} + + checks["storage"] = "ok" + pending = migration_info.get("pending", []) if isinstance(migration_info, dict) else ["invalid"] + checks["migrations"] = "pending" if pending else "ok" + return not pending, checks, migration_info if isinstance(migration_info, dict) else {} + + +@app.get("/health/live", response_model=LivenessResponse, include_in_schema=False) +@app.get("/health", response_model=LivenessResponse) +async def health_check() -> LivenessResponse: + """Process liveness probe with no optional-provider dependencies.""" + return LivenessResponse(status="alive", version=VERSION) + + +@app.get("/health/ready", response_model=ReadinessResponse) +async def readiness_check() -> JSONResponse: + """Required storage and migration readiness probe.""" + if drain_controller.snapshot().draining: + payload = ReadinessResponse( + status="not_ready", + version=VERSION, + checks={ + "drain": "draining", + "storage": "not_checked", + "migrations": "not_checked", + }, + ) + return JSONResponse(status_code=503, content=payload.model_dump()) + ready, checks, _migration_info = _storage_readiness() + payload = ReadinessResponse( + status="ready" if ready else "not_ready", + version=VERSION, + checks=checks, + ) + return JSONResponse(status_code=200 if ready else 503, content=payload.model_dump()) + + +@app.get("/health/diagnostics", response_model=HealthResponse) +async def health_diagnostics() -> HealthResponse: + """Authenticated provider, cache, rules, and storage diagnostics.""" providers = await llm_router.health() # Cache stats @@ -190,22 +619,14 @@ async def health_check(): # Rules stats from backend.agents.rules import get_rules_engine + rules_engine = get_rules_engine() rules_info = { "total": len(rules_engine.rules), "enabled": len(rules_engine.get_enabled_rules()), } - # SQLite connectivity - db_ok = True - migration_info = {} - try: - from backend.storage import get_session_store - store = get_session_store() - _ = store.list_sessions(1) - migration_info = store.get_migration_status() - except Exception: - db_ok = False + db_ok, _checks, migration_info = _storage_readiness() diagnostics = { "database": "ok" if db_ok else "error", @@ -225,22 +646,27 @@ async def health_check(): # ── Metrics (Prometheus) ───────────────────────────────────────────────── + @app.get("/metrics") async def prometheus_metrics(): """Prometheus metrics endpoint.""" from backend.observability import get_metrics_response, init_app_info + init_app_info(VERSION) content, content_type = get_metrics_response() from fastapi.responses import Response + return Response(content=content, media_type=content_type) # ── AI Rules API ───────────────────────────────────────────────────────── + @app.get("/api/v1/rules") async def list_rules(category: Optional[str] = None): """List all AI rules, optionally filtered by category.""" from backend.agents.rules import get_rules_engine + engine = get_rules_engine() return {"rules": engine.list_rules(category)} @@ -249,10 +675,12 @@ async def list_rules(category: Optional[str] = None): async def update_rule(rule_id: str, enabled: bool): """Enable or disable an AI rule.""" from backend.agents.rules import get_rules_engine + engine = get_rules_engine() if engine.set_enabled(rule_id, enabled): return {"rule_id": rule_id, "enabled": enabled} from fastapi import HTTPException + raise HTTPException(status_code=404, detail=f"Rule '{rule_id}' not found") @@ -270,32 +698,38 @@ async def create_rule(request: CreateRuleRequest): """Add a custom AI rule.""" from backend.agents.rules import get_rules_engine from dataclasses import asdict + engine = get_rules_engine() try: rule = engine.add_rule(request.model_dump()) return {"rule": asdict(rule)} except ValueError as e: from fastapi import HTTPException - raise HTTPException(status_code=400, detail=str(e)) + + raise HTTPException(status_code=400, detail=str(e)) from e @app.delete("/api/v1/rules/{rule_id}") async def delete_rule(rule_id: str): """Delete a custom AI rule.""" from backend.agents.rules import get_rules_engine + engine = get_rules_engine() if engine.delete_rule(rule_id): return {"deleted": rule_id} from fastapi import HTTPException + raise HTTPException(status_code=404, detail=f"Rule '{rule_id}' not found") # ── Prompt Management ──────────────────────────────────────────────────── + @app.get("/api/v1/prompts/metrics") async def prompt_metrics(): """Get prompt performance metrics (call count, success rate).""" from backend.agents.prompt_manager import get_prompt_manager + return {"prompts": get_prompt_manager().get_metrics()} @@ -303,22 +737,30 @@ async def prompt_metrics(): async def reload_prompts(name: Optional[str] = None): """Hot-reload prompts from disk without server restart.""" from backend.agents.prompt_manager import get_prompt_manager + get_prompt_manager().reload(name) return {"status": "reloaded", "name": name or "all"} # ── Providers ──────────────────────────────────────────────────────────── + @app.get("/api/v1/providers") -async def list_providers(): +async def list_providers(request: Request): """List available LLM providers and their models.""" + from backend.api.security.ai_policy import get_effective_ai_policy from backend.config import ( - OLLAMA_MODEL_SIMPLE, OLLAMA_MODEL_MEDIUM, + OLLAMA_MODEL_SIMPLE, + OLLAMA_MODEL_MEDIUM, LMSTUDIO_MODEL, - ANTHROPIC_MODEL_MEDIUM, ANTHROPIC_MODEL_COMPLEX, + ANTHROPIC_MODEL_MEDIUM, + ANTHROPIC_MODEL_COMPLEX, OPENAI_MODEL, ) + health = await llm_router.health() + ai_policy = get_effective_ai_policy() + actor = _request_actor(request) # Auto-detect LM Studio loaded models lmstudio_models = [] @@ -339,32 +781,39 @@ async def list_providers(): "models": lmstudio_models, "cost": "free (local)", "note": "OpenAI-compatible API — serves whatever model is loaded", + "policy": ai_policy.provider_policy("lmstudio"), }, "ollama": { "available": health.get("ollama", False), "models": [OLLAMA_MODEL_SIMPLE, OLLAMA_MODEL_MEDIUM], "cost": "free (local)", + "policy": ai_policy.provider_policy("ollama"), }, "anthropic": { "available": health.get("anthropic", False), "models": [ANTHROPIC_MODEL_MEDIUM, ANTHROPIC_MODEL_COMPLEX], "cost": "paid (cloud API)", + "policy": ai_policy.provider_policy("anthropic"), }, "openai": { "available": health.get("openai", False), "models": [OPENAI_MODEL], "cost": "paid (cloud API)", + "policy": ai_policy.provider_policy("openai"), }, - } + }, + "aiPolicy": ai_policy.to_dict(actor), } # ── v2: Template Compositions ──────────────────────────────────────────── + @app.get("/api/v1/compositions") async def list_compositions(): """List available template compositions (e.g., Full Valuation, PE Pitch).""" from backend.templates.composer import get_composer + return {"compositions": get_composer().list_compositions()} @@ -372,6 +821,7 @@ async def list_compositions(): async def deploy_composition(composition_id: str): """Deploy a template composition to the workbook.""" from backend.templates.composer import get_composer + try: operations = get_composer().compose(composition_id) return { @@ -381,33 +831,39 @@ async def deploy_composition(composition_id: str): } except ValueError as e: from fastapi import HTTPException - raise HTTPException(status_code=404, detail=str(e)) + + raise HTTPException(status_code=404, detail=str(e)) from e # ── v2: Intelligence Diagnostics ───────────────────────────────────────── + @app.get("/api/v1/intelligence/stats") async def intelligence_stats(): """Get diagnostics for all v2 intelligence modules.""" from backend.agents.classifier import get_classifier from backend.agents.adaptive_prompts import get_prompt_tuner - from backend.documents.rag import get_rag_pipeline classifier = get_classifier() - rag = get_rag_pipeline() stats = { "classifier": classifier.get_stats(), - "rag": rag.get_stats(), "prompt_tuner": { "examples": len(get_prompt_tuner()._examples), "model_profiles": list(get_prompt_tuner()._model_profiles.keys()), }, } + if RAG_ENABLED and not PRODUCTION_MODE: + from backend.documents.rag import get_rag_pipeline + + stats["rag"] = get_rag_pipeline().get_stats() + else: + stats["rag"] = {"enabled": False, "reason": "owner_scoped_retrieval_unavailable"} # Vector cache stats (if available) try: from backend.llm.cache_vector import get_semantic_cache + stats["vector_cache"] = get_semantic_cache().get_stats() except Exception: stats["vector_cache"] = {"type": "not_initialized"} @@ -417,27 +873,29 @@ async def intelligence_stats(): # ── v2: RAG Pipeline API ──────────────────────────────────────────────── -@app.get("/api/v1/rag/search") -async def rag_search(query: str, top_k: int = 5): - """Search indexed documents using semantic retrieval.""" - from backend.documents.rag import get_rag_pipeline - pipeline = get_rag_pipeline() - results = pipeline.retrieve(query, top_k=top_k) - return { - "query": query, - "results": [ - {"text": r.text[:300], "source": r.source, "score": r.relevance_score} - for r in results - ], - } +if RAG_ENABLED and not PRODUCTION_MODE: + + @app.get("/api/v1/rag/search") + async def rag_search(query: str, top_k: int = 5): + """Search indexed documents using semantic retrieval.""" + from backend.documents.rag import get_rag_pipeline + + pipeline = get_rag_pipeline() + results = pipeline.retrieve(query, top_k=top_k) + return { + "query": query, + "results": [{"text": r.text[:300], "source": r.source, "score": r.relevance_score} for r in results], + } # ── v3: EDGAR / SEC Integration ────────────────────────────────────────── + @app.get("/api/v1/edgar/filings/{ticker}") async def edgar_filings(ticker: str, form_type: str = "10-K", count: int = 3): """Retrieve SEC filings for a company by ticker.""" from backend.data.edgar import get_edgar_connector + connector = get_edgar_connector() filings = await connector.get_filings(ticker, form_type, count) return { @@ -459,6 +917,7 @@ async def edgar_filings(ticker: str, form_type: str = "10-K", count: int = 3): async def edgar_financials(ticker: str, form_type: str = "10-K"): """Extract structured financial data from EDGAR XBRL.""" from backend.data.edgar import get_edgar_connector + connector = get_edgar_connector() statements = await connector.get_xbrl_financials(ticker, form_type) operations = connector.financials_to_operations(statements, ticker.upper()) @@ -472,10 +931,12 @@ async def edgar_financials(ticker: str, form_type: str = "10-K"): # ── v3: Saved Workflows ───────────────────────────────────────────────── + @app.get("/api/v1/workflows") async def list_workflows(): """List all saved workflows.""" from backend.agents.workflows import get_workflow_engine + return {"workflows": get_workflow_engine().list_workflows()} @@ -483,14 +944,20 @@ class CreateWorkflowRequest(BaseModel): name: str description: str steps: list[dict] - variables: dict = {} - tags: list[str] = [] + variables: dict = Field(default_factory=dict) + tags: list[str] = Field(default_factory=list) + session_id: str = "" @app.post("/api/v1/workflows") -async def create_workflow(request: CreateWorkflowRequest): +async def create_workflow(request: CreateWorkflowRequest, http_request: Request): """Create a new saved workflow.""" from backend.agents.workflows import get_workflow_engine + + if request.session_id: + from backend.api.routes.tasks import _claim_request_session + + _claim_request_session(http_request, request.session_id) engine = get_workflow_engine() wf = engine.create_workflow( name=request.name, @@ -498,34 +965,67 @@ async def create_workflow(request: CreateWorkflowRequest): steps=request.steps, variables=request.variables, tags=request.tags, + session_id=request.session_id, ) return {"workflow_id": wf.id, "name": wf.name} @app.post("/api/v1/workflows/{workflow_id}/execute") -async def execute_workflow(workflow_id: str, variables: dict = {}): +async def execute_workflow(workflow_id: str, variables: dict = Body(default_factory=dict)): """Execute a saved workflow.""" from backend.agents.workflows import get_workflow_engine + engine = get_workflow_engine() try: result = await engine.execute_workflow(workflow_id, variables) return result except ValueError as e: from fastapi import HTTPException - raise HTTPException(status_code=404, detail=str(e)) + + raise HTTPException(status_code=404, detail=str(e)) from e # ── v3: Data Connectors ───────────────────────────────────────────────── + @app.get("/api/v1/data/connectors") async def list_data_connectors(): """List available data source connectors.""" from backend.data.connectors import get_data_registry + return {"connectors": get_data_registry().list_connectors()} +class DataConnectorFetchRequest(BaseModel): + query: str + params: dict = Field(default_factory=dict) + + +@app.post("/api/v1/data/connectors/{connector_id}/fetch") +async def fetch_data_connector(connector_id: str, request: DataConnectorFetchRequest): + """Fetch structured data through a registered connector.""" + from fastapi import HTTPException + + from backend.data.connectors import get_data_registry + + try: + result = await get_data_registry().fetch(connector_id, request.query, request.params) + except ValueError as exc: + raise HTTPException(status_code=404, detail=str(exc)) from exc + + return { + "connector_id": connector_id, + "source": result.source, + "headers": result.headers, + "row_count": result.row_count, + "metadata": result.metadata, + "data": result.data, + } + + # ── v3: Formula Tracing ───────────────────────────────────────────────── + class TraceRequest(BaseModel): cell: str operations: list[dict] @@ -535,6 +1035,7 @@ class TraceRequest(BaseModel): async def trace_cell(request: TraceRequest): """Trace formula dependencies for a cell.""" from backend.excel.formula_tracer import trace_formula, trace_to_ui, build_reverse_trace + trace = trace_formula(request.cell, request.operations) return { "forward_trace": trace_to_ui(trace), @@ -544,21 +1045,34 @@ async def trace_cell(request: TraceRequest): # ── v3: Security / RBAC Config ────────────────────────────────────────── + @app.get("/api/v1/security/config") async def security_config(): """Get current security configuration.""" - from backend.api.security.rbac import get_sso_config, get_zdr + from backend.api.routes.enterprise import _effective_security_policy, _effective_sso_config + from backend.api.security.rbac import get_zdr from backend.api.security.tls import get_tls_config + return { "tls": get_tls_config(), - "sso": get_sso_config().to_dict(), + "sso": _effective_sso_config(), + "security_policy": _effective_security_policy(), "zero_data_retention": get_zdr().to_dict(), "rbac_enabled": False, # Set from env } +@app.get("/api/v1/security/ai-policy") +async def security_ai_policy(request: Request): + """Return the effective admin policy for AI modes, providers, and source data.""" + from backend.api.security.ai_policy import get_effective_ai_policy + + return get_effective_ai_policy().to_dict(_request_actor(request)) + + # ── v3: Macro Awareness ───────────────────────────────────────────────── + class MacroCheckRequest(BaseModel): operations: list[dict] workbook_context: dict = {} @@ -568,6 +1082,7 @@ class MacroCheckRequest(BaseModel): async def check_macro_conflicts(request: MacroCheckRequest): """Check operations for macro/VBA conflicts.""" from backend.excel.macro_awareness import get_macro_engine + engine = get_macro_engine() info = engine.scan_workbook_context(request.workbook_context) conflicts = engine.check_conflicts(request.operations, info) @@ -577,8 +1092,7 @@ async def check_macro_conflicts(request: MacroCheckRequest): "custom_functions": info.custom_functions, "protected_sheets": info.protected_sheets, "conflicts": [ - {"cell": c.cell, "type": c.conflict_type, "severity": c.severity, - "description": c.description, "suggestion": c.suggestion} + {"cell": c.cell, "type": c.conflict_type, "severity": c.severity, "description": c.description, "suggestion": c.suggestion} for c in conflicts ], } @@ -586,6 +1100,7 @@ async def check_macro_conflicts(request: MacroCheckRequest): # ── v3: QA Agent ──────────────────────────────────────────────────────── + class QAReviewRequest(BaseModel): operations: list[dict] workbook_context: dict = {} @@ -595,6 +1110,7 @@ class QAReviewRequest(BaseModel): async def qa_review(request: QAReviewRequest): """Run quality assurance review on operations.""" from backend.agents.verifiers.qa_agent import get_qa_agent + agent = get_qa_agent() issues = agent.review(request.operations, request.workbook_context) return { @@ -603,9 +1119,14 @@ async def qa_review(request: QAReviewRequest): "warnings": sum(1 for i in issues if i.severity == "warning"), "info": sum(1 for i in issues if i.severity == "info"), "issues": [ - {"cell": i.cell, "rule": i.rule, "severity": i.severity, - "category": i.category, "description": i.description, - "suggestion": i.suggestion} + { + "cell": i.cell, + "rule": i.rule, + "severity": i.severity, + "category": i.category, + "description": i.description, + "suggestion": i.suggestion, + } for i in issues ], } @@ -613,6 +1134,7 @@ async def qa_review(request: QAReviewRequest): # ── v3: Teams / Multi-User ────────────────────────────────────────────── + class CreateTeamRequest(BaseModel): name: str owner_id: str @@ -637,6 +1159,7 @@ class CreateSessionRequest(BaseModel): async def list_teams(): """List all teams.""" from backend.api.teams import get_team_manager + return {"teams": get_team_manager().list_teams()} @@ -644,9 +1167,8 @@ async def list_teams(): async def create_team(request: CreateTeamRequest): """Create a new team.""" from backend.api.teams import get_team_manager - team = get_team_manager().create_team( - request.name, request.owner_id, request.owner_email, request.owner_name - ) + + team = get_team_manager().create_team(request.name, request.owner_id, request.owner_email, request.owner_name) return {"team_id": team.team_id, "name": team.name} @@ -654,9 +1176,8 @@ async def create_team(request: CreateTeamRequest): async def add_team_member(team_id: str, request: AddMemberRequest): """Add member to a team.""" from backend.api.teams import get_team_manager - ok = get_team_manager().add_member( - team_id, request.user_id, request.email, request.name, request.role - ) + + ok = get_team_manager().add_member(team_id, request.user_id, request.email, request.name, request.role) return {"success": ok} @@ -664,6 +1185,7 @@ async def add_team_member(team_id: str, request: AddMemberRequest): async def remove_team_member(team_id: str, user_id: str): """Remove member from a team.""" from backend.api.teams import get_team_manager + ok = get_team_manager().remove_member(team_id, user_id) return {"success": ok} @@ -672,6 +1194,7 @@ async def remove_team_member(team_id: str, user_id: str): async def list_team_sessions(team_id: str): """List all sessions for a team.""" from backend.api.teams import get_team_manager + return {"sessions": get_team_manager().list_team_sessions(team_id)} @@ -679,9 +1202,8 @@ async def list_team_sessions(team_id: str): async def create_team_session(request: CreateSessionRequest): """Create a shared team session.""" from backend.api.teams import get_team_manager - session = get_team_manager().create_shared_session( - request.team_id, request.owner_id, request.title - ) + + session = get_team_manager().create_shared_session(request.team_id, request.owner_id, request.title) if not session: return {"error": "Team not found"} return {"session_id": session.session_id, "title": session.title} @@ -691,15 +1213,18 @@ async def create_team_session(request: CreateSessionRequest): async def team_analytics(team_id: str): """Get team usage analytics.""" from backend.api.teams import get_team_manager + return get_team_manager().get_team_analytics(team_id) # ── v3: Usage Analytics Dashboard ──────────────────────────────────────── + @app.get("/api/v1/analytics/dashboard") async def analytics_dashboard(days: int = 30): """Get admin analytics dashboard.""" from backend.api.analytics import get_analytics + return get_analytics().get_dashboard(days) @@ -707,11 +1232,13 @@ async def analytics_dashboard(days: int = 30): async def user_analytics(user_id: str): """Get per-user analytics.""" from backend.api.analytics import get_analytics + return get_analytics().get_user_stats(user_id) # ── v3: Multi-Sheet Analysis ──────────────────────────────────────────── + class WorkbookAnalyzeRequest(BaseModel): sheets: dict = {} @@ -720,6 +1247,7 @@ class WorkbookAnalyzeRequest(BaseModel): async def analyze_workbook(request: WorkbookAnalyzeRequest): """Analyze multi-sheet workbook structure.""" from backend.excel.multi_sheet import get_multi_sheet_analyzer + analyzer = get_multi_sheet_analyzer() wmap = analyzer.analyze({"sheets": request.sheets}) return { @@ -728,9 +1256,14 @@ async def analyze_workbook(request: WorkbookAnalyzeRequest): "total_formulas": wmap.total_formulas, "total_data_cells": wmap.total_data_cells, "sheets": [ - {"name": s.name, "type": s.sheet_type, "rows": s.data_rows, - "formulas": s.formula_count, "references": s.references, - "referenced_by": s.referenced_by} + { + "name": s.name, + "type": s.sheet_type, + "rows": s.data_rows, + "formulas": s.formula_count, + "references": s.references, + "referenced_by": s.referenced_by, + } for s in wmap.sheets ], "cross_sheet_links": wmap.cross_sheet_links, @@ -740,10 +1273,12 @@ async def analyze_workbook(request: WorkbookAnalyzeRequest): # ── v3: SOC 2 Compliance ──────────────────────────────────────────────── + @app.get("/api/v1/compliance/soc2") async def soc2_audit(): """Run SOC 2 readiness audit.""" from backend.api.security.soc2 import get_soc2_auditor + return get_soc2_auditor().run_audit() @@ -751,20 +1286,24 @@ async def soc2_audit(): async def soc2_report(): """Get human-readable SOC 2 readiness report.""" from backend.api.security.soc2 import get_soc2_auditor + return {"report": get_soc2_auditor().generate_readiness_report()} # ── v3: Encryption Status ─────────────────────────────────────────────── + @app.get("/api/v1/security/encryption") async def encryption_status(): """Get database encryption status.""" - from backend.storage.encrypted_store import get_encrypted_store - return get_encrypted_store().get_encryption_status() + from backend.storage.encrypted_store import get_primary_storage_encryption_status + + return get_primary_storage_encryption_status() # ── Entry Point ────────────────────────────────────────────────────────── if __name__ == "__main__": import uvicorn + uvicorn.run("backend.main:app", host=HOST, port=int(PORT), reload=True) diff --git a/backend/migration/__init__.py b/backend/migration/__init__.py new file mode 100644 index 0000000..4f55000 --- /dev/null +++ b/backend/migration/__init__.py @@ -0,0 +1,20 @@ +"""One-way release migration tooling. + +This package is deliberately outside the HTTP API and normal runtime. It +exports neutral, reviewable records instead of exposing the v0.5 database or +model classes as a compatibility surface. +""" + +from backend.migration.v05_bundle import ( + ExportLimits, + ExportResult, + MigrationExportError, + export_v05_bundle, +) + +__all__ = [ + "ExportLimits", + "ExportResult", + "MigrationExportError", + "export_v05_bundle", +] diff --git a/backend/migration/v05_bundle.py b/backend/migration/v05_bundle.py new file mode 100644 index 0000000..743bd88 --- /dev/null +++ b/backend/migration/v05_bundle.py @@ -0,0 +1,628 @@ +"""Deterministic, owner-scoped v0.5 export bundle. + +The export boundary intentionally includes only portable session descriptors, +raw managed document blobs, and saved workflow recipes requiring review. It +does not serialize the SQLite/PostgreSQL schema, HTTP contracts, model-version +objects, job/replay/cache state, credentials, or parsed document derivatives. +""" + +from __future__ import annotations + +from contextlib import nullcontext +from dataclasses import dataclass +import hashlib +import io +import json +import math +import os +from pathlib import Path +import re +import stat +import tempfile +from typing import Any, ContextManager +import zipfile + +from vnext.migration.portable_secret_policy import ( + allowed_step_parameter_keys, + is_secret_key, + redact_secret_text, +) + + +SCHEMA_VERSION = "com.excelagent.migration.v05.v1" +SOURCE_PRODUCT = "Excel Agent" +SOURCE_RELEASE = "0.5.0" +MIGRATION_MODE = "one_way_clean_break" + +EXCLUDED_CATEGORIES = ( + "api_credentials", + "audit_history", + "conversation_history", + "database_schema", + "document_derivatives", + "enterprise_configuration", + "legacy_api_contracts", + "legacy_model_artifacts_v2_v7", + "llm_caches", + "rag_indexes", + "replay_data", + "task_history", + "worker_jobs_and_cli_state", +) + +_HEX_SHA256 = re.compile(r"[0-9a-f]{64}") + + +class MigrationExportError(RuntimeError): + """The source state cannot be exported without weakening the boundary.""" + + +@dataclass(frozen=True) +class ExportLimits: + max_sessions: int = 256 + max_documents_per_session: int = 4096 + max_workflows: int = 4096 + max_workflow_steps: int = 256 + max_record_bytes: int = 4 * 1024 * 1024 + max_blob_bytes: int = 50 * 1024 * 1024 + max_total_payload_bytes: int = 512 * 1024 * 1024 + read_chunk_bytes: int = 1024 * 1024 + + +@dataclass(frozen=True) +class ExportResult: + archive_path: Path + sha256_path: Path + archive_sha256: str + content_sha256: str + counts: dict[str, int] + + +def export_v05_bundle( + *, + store: Any, + workflow_engine: Any, + owner_id: str, + session_ids: list[str] | tuple[str, ...], + managed_documents_root: Path, + output_path: Path, + limits: ExportLimits | None = None, +) -> ExportResult: + """Export portable v0.5 state through owner-aware service interfaces. + + ``managed_documents_root`` is a configured application root, not a caller + selected source path. Every recorded document path must resolve beneath it. + The companion ``.sha256`` file authenticates the exact archive bytes when + it is transported over a trusted channel. + """ + + limits = limits or ExportLimits() + _validate_limits(limits) + normalized_owner = _normalize_identifier(owner_id, label="owner ID", max_length=512) + normalized_sessions = [ + _normalize_identifier(value, label="session ID", max_length=512) for value in session_ids + ] + if not normalized_sessions: + raise MigrationExportError("At least one session ID is required") + if len(normalized_sessions) > limits.max_sessions: + raise MigrationExportError("Session count exceeds the export limit") + if len(set(normalized_sessions)) != len(normalized_sessions): + raise MigrationExportError("Duplicate session IDs are not allowed") + normalized_sessions.sort() + + managed_root = _validated_managed_root(managed_documents_root) + output = Path(output_path).expanduser().resolve(strict=False) + sha256_path = output.with_name(f"{output.name}.sha256") + if output.exists() or sha256_path.exists(): + raise MigrationExportError("Output archive or SHA-256 sidecar already exists") + + owner_scope = _sha256_text(normalized_owner) + payloads: dict[str, tuple[str, bytes]] = {} + counts = {"sessions": 0, "documents": 0, "workflows": 0} + seen_document_ids: set[str] = set() + + with _ownership_scope(normalized_owner): + for session_id in normalized_sessions: + _require_owned_session(store, session_id, normalized_owner) + session_record = { + "kind": "session", + "legacy_session_id": session_id, + "owner_scope_sha256": owner_scope, + "record_version": 1, + } + _add_json_payload( + payloads, + kind="session", + path=f"records/sessions/{_sha256_text(session_id)}.json", + value=session_record, + limits=limits, + ) + counts["sessions"] += 1 + + documents = list(store.list_documents(session_id, limit=limits.max_documents_per_session + 1)) + if len(documents) > limits.max_documents_per_session: + raise MigrationExportError(f"Document count exceeds the export limit for session {session_id!r}") + documents.sort(key=lambda item: str(item.get("document_id") or "")) + for document in documents: + document_id = _normalize_identifier( + document.get("document_id"), label="document ID", max_length=512 + ) + if document_id in seen_document_ids: + raise MigrationExportError(f"Duplicate document ID in source state: {document_id!r}") + seen_document_ids.add(document_id) + if str(document.get("session_id") or "") != session_id: + raise MigrationExportError(f"Document {document_id!r} is outside the requested session scope") + + stable_before = _stable_document_authority(document) + authoritative = store.get_document(document_id) + if not authoritative or _stable_document_authority(authoritative) != stable_before: + raise MigrationExportError(f"Document {document_id!r} changed while the export was being prepared") + blob = _read_managed_blob( + raw_path=str(authoritative.get("file_path") or ""), + managed_root=managed_root, + limits=limits, + ) + declared_size = _nonnegative_int(authoritative.get("size_bytes"), label="document size") + if declared_size and declared_size != len(blob): + raise MigrationExportError(f"Document {document_id!r} size does not match its managed blob") + blob_sha256 = hashlib.sha256(blob).hexdigest() + blob_path = f"blobs/sha256/{blob_sha256}" + _add_payload(payloads, kind="blob", path=blob_path, content=blob) + + redactions: list[str] = [] + filename = _sanitize_portable_value( + str(authoritative.get("filename") or "document"), + pointer="/filename", + redactions=redactions, + ) + content_type = _sanitize_portable_value( + str(authoritative.get("content_type") or "application/octet-stream"), + pointer="/content_type", + redactions=redactions, + ) + record = { + "blob": { + "path": blob_path, + "sha256": blob_sha256, + "size_bytes": len(blob), + }, + "content_type": content_type, + "filename": filename, + "kind": "document", + "legacy_document_id": document_id, + "legacy_session_id": session_id, + "record_version": 1, + "redactions": sorted(set(redactions)), + } + _add_json_payload( + payloads, + kind="document", + path=f"records/documents/{_sha256_text(document_id)}.json", + value=record, + limits=limits, + ) + counts["documents"] += 1 + + authoritative_after = store.get_document(document_id) + if not authoritative_after or _stable_document_authority(authoritative_after) != stable_before: + raise MigrationExportError(f"Document {document_id!r} changed during export") + + _require_owned_session(store, session_id, normalized_owner) + + workflow_summaries = list(workflow_engine.list_workflows()) + if len(workflow_summaries) > limits.max_workflows: + raise MigrationExportError("Workflow count exceeds the export limit") + workflow_ids = sorted( + _normalize_identifier(item.get("id"), label="workflow ID", max_length=512) + for item in workflow_summaries + ) + if len(set(workflow_ids)) != len(workflow_ids): + raise MigrationExportError("Duplicate workflow IDs are not allowed") + for workflow_id in workflow_ids: + workflow = workflow_engine.get_workflow(workflow_id) + if workflow is None or bool(getattr(workflow, "built_in", False)): + continue + workflow_owner = str(getattr(workflow, "owner_user_id", "") or "local") + if workflow_owner != normalized_owner: + raise MigrationExportError(f"Workflow {workflow_id!r} is outside the requested owner scope") + workflow_session = str(getattr(workflow, "session_id", "") or "") + if workflow_session and workflow_session not in normalized_sessions: + continue + record = _workflow_record( + workflow, + owner_scope=owner_scope, + limits=limits, + ) + record_bytes = _canonical_json(record) + workflow_after = workflow_engine.get_workflow(workflow_id) + if workflow_after is None: + raise MigrationExportError(f"Workflow {workflow_id!r} changed during export") + if _canonical_json(_workflow_record(workflow_after, owner_scope=owner_scope, limits=limits)) != record_bytes: + raise MigrationExportError(f"Workflow {workflow_id!r} changed during export") + _add_payload( + payloads, + kind="workflow_recipe", + path=f"records/workflows/{_sha256_text(workflow_id)}.json", + content=record_bytes, + ) + counts["workflows"] += 1 + + total_payload_bytes = sum(len(content) for _, content in payloads.values()) + if total_payload_bytes > limits.max_total_payload_bytes: + raise MigrationExportError("Bundle payload exceeds the total export size limit") + + descriptors = [ + { + "kind": kind, + "path": path, + "sha256": hashlib.sha256(content).hexdigest(), + "size_bytes": len(content), + } + for path, (kind, content) in sorted(payloads.items()) + ] + content_sha256 = hashlib.sha256(_canonical_json(descriptors)).hexdigest() + manifest = { + "content_sha256": content_sha256, + "counts": counts, + "excluded_categories": list(EXCLUDED_CATEGORIES), + "migration_mode": MIGRATION_MODE, + "owner_scope_sha256": owner_scope, + "payloads": descriptors, + "schema_version": SCHEMA_VERSION, + "source_product": SOURCE_PRODUCT, + "source_release": SOURCE_RELEASE, + } + archive_bytes = _build_archive(manifest, payloads) + archive_sha256 = hashlib.sha256(archive_bytes).hexdigest() + if not _HEX_SHA256.fullmatch(archive_sha256): + raise MigrationExportError("Internal archive digest failure") + sidecar_bytes = f"{archive_sha256} {output.name}\n".encode("ascii") + _publish_pair(output, archive_bytes, sha256_path, sidecar_bytes) + return ExportResult( + archive_path=output, + sha256_path=sha256_path, + archive_sha256=archive_sha256, + content_sha256=content_sha256, + counts=dict(counts), + ) + + +def _validate_limits(limits: ExportLimits) -> None: + for name, value in vars(limits).items(): + if not isinstance(value, int) or isinstance(value, bool) or value <= 0: + raise MigrationExportError(f"Export limit {name} must be a positive integer") + + +def _normalize_identifier(value: Any, *, label: str, max_length: int) -> str: + normalized = str(value or "").strip() + if not normalized or len(normalized) > max_length or any(ord(character) < 32 for character in normalized): + raise MigrationExportError(f"Invalid {label}") + return normalized + + +def _validated_managed_root(path: Path) -> Path: + root = Path(path).expanduser() + try: + metadata = root.lstat() + resolved = root.resolve(strict=True) + except OSError as exc: + raise MigrationExportError("Managed document root is unavailable") from exc + if stat.S_ISLNK(metadata.st_mode) or not stat.S_ISDIR(metadata.st_mode): + raise MigrationExportError("Managed document root must be a real directory") + return resolved + + +def _require_owned_session(store: Any, session_id: str, owner_id: str) -> None: + try: + exists = bool(store.session_exists(session_id)) + stored_owner = store.get_session_owner(session_id) + except Exception as exc: + raise MigrationExportError(f"Unable to verify ownership for session {session_id!r}") from exc + if not exists: + raise MigrationExportError(f"Session {session_id!r} does not exist") + effective_owner = str(stored_owner or "local") + if effective_owner != owner_id: + raise MigrationExportError(f"Session {session_id!r} is not owned by the requested owner") + + +def _ownership_scope(owner_id: str) -> ContextManager[Any]: + try: + from backend.api.security.ownership import ownership_actor_scope + except ImportError: + return nullcontext() + if owner_id == "local": + return ownership_actor_scope(None) + return ownership_actor_scope({"user_id": owner_id, "role": "analyst"}) + + +def _stable_document_authority(document: dict[str, Any]) -> tuple[str, str, str, str, int]: + return ( + str(document.get("document_id") or ""), + str(document.get("session_id") or ""), + str(document.get("file_path") or ""), + str(document.get("content_type") or ""), + _nonnegative_int(document.get("size_bytes"), label="document size"), + ) + + +def _nonnegative_int(value: Any, *, label: str) -> int: + if isinstance(value, bool): + raise MigrationExportError(f"Invalid {label}") + try: + result = int(value or 0) + except (TypeError, ValueError) as exc: + raise MigrationExportError(f"Invalid {label}") from exc + if result < 0: + raise MigrationExportError(f"Invalid {label}") + return result + + +def _read_managed_blob(*, raw_path: str, managed_root: Path, limits: ExportLimits) -> bytes: + if not raw_path: + raise MigrationExportError("Document has no managed blob") + candidate = Path(raw_path).expanduser() + try: + metadata = candidate.lstat() + resolved = candidate.resolve(strict=True) + except OSError as exc: + raise MigrationExportError("Managed document blob is unavailable") from exc + if stat.S_ISLNK(metadata.st_mode) or not stat.S_ISREG(metadata.st_mode): + raise MigrationExportError("Managed document blob must be a real regular file") + if not resolved.is_relative_to(managed_root): + raise MigrationExportError("Recorded document path is outside the managed document root") + + flags = os.O_RDONLY + if hasattr(os, "O_NOFOLLOW"): + flags |= os.O_NOFOLLOW + try: + descriptor = os.open(resolved, flags) + except OSError as exc: + raise MigrationExportError("Managed document blob could not be opened safely") from exc + try: + before = os.fstat(descriptor) + if not stat.S_ISREG(before.st_mode): + raise MigrationExportError("Managed document blob must be a regular file") + if before.st_size > limits.max_blob_bytes: + raise MigrationExportError("Managed document exceeds the blob size limit") + chunks: list[bytes] = [] + total = 0 + while True: + chunk = os.read(descriptor, limits.read_chunk_bytes) + if not chunk: + break + total += len(chunk) + if total > limits.max_blob_bytes: + raise MigrationExportError("Managed document exceeds the blob size limit") + chunks.append(chunk) + after = os.fstat(descriptor) + stable_fields = ("st_dev", "st_ino", "st_size", "st_mtime_ns") + if any(getattr(before, field) != getattr(after, field) for field in stable_fields): + raise MigrationExportError("Managed document changed while it was read") + return b"".join(chunks) + finally: + os.close(descriptor) + + +def _workflow_record(workflow: Any, *, owner_scope: str, limits: ExportLimits) -> dict[str, Any]: + workflow_id = _normalize_identifier(getattr(workflow, "id", ""), label="workflow ID", max_length=512) + steps = list(getattr(workflow, "steps", []) or []) + if len(steps) > limits.max_workflow_steps: + raise MigrationExportError(f"Workflow {workflow_id!r} exceeds the step limit") + redactions: list[str] = [] + portable_steps = [] + for index, step in enumerate(steps): + pointer = f"/steps/{index}" + legacy_kind = str(getattr(step, "step_type", "") or "") + raw_parameters = getattr(step, "config", {}) or {} + if not isinstance(raw_parameters, dict): + raise MigrationExportError("Workflow step parameters must be an object") + allowed_keys = allowed_step_parameter_keys(legacy_kind) + portable_parameters: dict[str, Any] = {} + for raw_key in sorted(raw_parameters, key=lambda item: str(item)): + key = str(raw_key) + child_pointer = f"{pointer}/parameters/{_json_pointer_token(key)}" + if key not in allowed_keys: + redactions.append(child_pointer) + continue + portable_parameters[key] = _sanitize_portable_value( + raw_parameters[raw_key], + pointer=child_pointer, + redactions=redactions, + ) + portable_steps.append( + { + "intent": _sanitize_portable_value( + str(getattr(step, "description", "") or ""), + pointer=f"{pointer}/intent", + redactions=redactions, + ), + "legacy_kind": _sanitize_portable_value( + legacy_kind, + pointer=f"{pointer}/legacy_kind", + redactions=redactions, + ), + "parameters": portable_parameters, + } + ) + session_id = str(getattr(workflow, "session_id", "") or "") or None + record = { + "description": _sanitize_portable_value( + str(getattr(workflow, "description", "") or ""), + pointer="/description", + redactions=redactions, + ), + "kind": "workflow_recipe", + "legacy_session_id": session_id, + "legacy_workflow_id": workflow_id, + "name": _sanitize_portable_value( + str(getattr(workflow, "name", "") or ""), + pointer="/name", + redactions=redactions, + ), + "owner_scope_sha256": owner_scope, + "record_version": 1, + "status": "requires_review", + "steps": portable_steps, + "tags": _sanitize_portable_value( + list(getattr(workflow, "tags", []) or []), + pointer="/tags", + redactions=redactions, + ), + "variables": _sanitize_portable_value( + getattr(workflow, "variables", {}) or {}, + pointer="/variables", + redactions=redactions, + ), + } + record["redactions"] = sorted(set(redactions)) + return record + + +def _sanitize_portable_value(value: Any, *, pointer: str, redactions: list[str], depth: int = 0) -> Any: + if depth > 32: + raise MigrationExportError("Portable workflow data exceeds the nesting limit") + if value is None or isinstance(value, (bool, int)): + return value + if isinstance(value, float): + if not math.isfinite(value): + raise MigrationExportError("Portable workflow data contains a non-finite number") + return value + if isinstance(value, str): + if len(value) > 1_000_000: + raise MigrationExportError("Portable workflow string exceeds the size limit") + sanitized, changed = redact_secret_text(value) + if changed: + redactions.append(pointer) + return sanitized + if isinstance(value, (list, tuple)): + if len(value) > 10_000: + raise MigrationExportError("Portable workflow list exceeds the item limit") + return [ + _sanitize_portable_value(item, pointer=f"{pointer}/{index}", redactions=redactions, depth=depth + 1) + for index, item in enumerate(value) + ] + if isinstance(value, dict): + if len(value) > 10_000: + raise MigrationExportError("Portable workflow object exceeds the field limit") + result: dict[str, Any] = {} + for raw_key in sorted(value, key=lambda item: str(item)): + key = str(raw_key) + child_pointer = f"{pointer}/{_json_pointer_token(key)}" + if is_secret_key(key): + redactions.append(child_pointer) + continue + result[key] = _sanitize_portable_value( + value[raw_key], + pointer=child_pointer, + redactions=redactions, + depth=depth + 1, + ) + return result + raise MigrationExportError(f"Portable workflow data contains unsupported type {type(value).__name__}") + + +def _json_pointer_token(value: str) -> str: + return value.replace("~", "~0").replace("/", "~1") + + +def _add_json_payload( + payloads: dict[str, tuple[str, bytes]], + *, + kind: str, + path: str, + value: Any, + limits: ExportLimits, +) -> None: + content = _canonical_json(value) + if len(content) > limits.max_record_bytes: + raise MigrationExportError(f"Portable {kind} record exceeds the record size limit") + _add_payload(payloads, kind=kind, path=path, content=content) + + +def _add_payload(payloads: dict[str, tuple[str, bytes]], *, kind: str, path: str, content: bytes) -> None: + existing = payloads.get(path) + if existing is not None: + if existing != (kind, content): + raise MigrationExportError(f"Conflicting content-addressed payload at {path!r}") + return + payloads[path] = (kind, content) + + +def _canonical_json(value: Any) -> bytes: + try: + return json.dumps( + value, + ensure_ascii=False, + allow_nan=False, + separators=(",", ":"), + sort_keys=True, + ).encode("utf-8") + b"\n" + except (TypeError, ValueError) as exc: + raise MigrationExportError("Portable record is not canonical JSON") from exc + + +def _build_archive(manifest: dict[str, Any], payloads: dict[str, tuple[str, bytes]]) -> bytes: + buffer = io.BytesIO() + with zipfile.ZipFile(buffer, mode="w", compression=zipfile.ZIP_STORED, strict_timestamps=True) as archive: + entries = [("manifest.json", _canonical_json(manifest))] + entries.extend((path, content) for path, (_, content) in sorted(payloads.items())) + for path, content in entries: + info = zipfile.ZipInfo(path, date_time=(1980, 1, 1, 0, 0, 0)) + info.create_system = 3 + info.compress_type = zipfile.ZIP_STORED + info.external_attr = (stat.S_IFREG | 0o600) << 16 + info.extra = b"" + info.comment = b"" + archive.writestr(info, content) + return buffer.getvalue() + + +def _publish_pair(archive_path: Path, archive: bytes, sidecar_path: Path, sidecar: bytes) -> None: + archive_path.parent.mkdir(parents=True, exist_ok=True) + if archive_path.parent.is_symlink() or not archive_path.parent.is_dir(): + raise MigrationExportError("Output parent must be a real directory") + temporary_paths: list[Path] = [] + published_archive = False + try: + archive_temp = _write_temporary(archive_path.parent, archive) + temporary_paths.append(archive_temp) + sidecar_temp = _write_temporary(sidecar_path.parent, sidecar) + temporary_paths.append(sidecar_temp) + os.link(archive_temp, archive_path) + published_archive = True + os.link(sidecar_temp, sidecar_path) + except FileExistsError as exc: + if published_archive: + archive_path.unlink(missing_ok=True) + raise MigrationExportError("Output archive or SHA-256 sidecar already exists") from exc + except OSError as exc: + if published_archive: + archive_path.unlink(missing_ok=True) + sidecar_path.unlink(missing_ok=True) + raise MigrationExportError("Unable to publish the migration bundle atomically") from exc + finally: + for temporary in temporary_paths: + temporary.unlink(missing_ok=True) + + +def _write_temporary(parent: Path, content: bytes) -> Path: + descriptor, raw_path = tempfile.mkstemp(prefix=".migration-", dir=parent) + path = Path(raw_path) + try: + os.fchmod(descriptor, 0o600) + with os.fdopen(descriptor, "wb", closefd=True) as handle: + handle.write(content) + handle.flush() + os.fsync(handle.fileno()) + except Exception: + try: + os.close(descriptor) + except OSError: + pass + path.unlink(missing_ok=True) + raise + return path + + +def _sha256_text(value: str) -> str: + return hashlib.sha256(value.encode("utf-8")).hexdigest() diff --git a/backend/models/operations.py b/backend/models/operations.py index 41cf4eb..21bc9b7 100644 --- a/backend/models/operations.py +++ b/backend/models/operations.py @@ -367,7 +367,7 @@ class OperationPlan(BaseModel): request_id: Optional[str] = None prompt: str summary: str = "" - status: Literal["draft", "approved", "applied", "failed", "rolled_back", "abandoned"] = "draft" + status: Literal["draft", "approved", "applied", "failed", "rollback_pending", "rolled_back", "abandoned"] = "draft" mode: Literal["preview", "action", "template", "react"] = "preview" operations: list[WorkbookOperation] = Field(default_factory=list) validation: ValidationReport = Field(default_factory=ValidationReport) @@ -584,7 +584,7 @@ class OperationPlanV2(BaseModel): request_id: Optional[str] = None prompt: str summary: str = "" - status: Literal["draft", "approved", "applied", "failed", "rolled_back", "abandoned"] = "draft" + status: Literal["draft", "approved", "applied", "failed", "rollback_pending", "rolled_back", "abandoned"] = "draft" mode: Literal["preview", "action", "template", "react"] = "preview" operations: list[WorkbookOperationV2] = Field(default_factory=list) validation: ValidationReport = Field(default_factory=ValidationReport) diff --git a/backend/observability.py b/backend/observability.py index 3177317..ff4266e 100644 --- a/backend/observability.py +++ b/backend/observability.py @@ -19,9 +19,10 @@ from __future__ import annotations import time +from collections.abc import Mapping from contextlib import contextmanager from functools import wraps -from typing import Optional +from typing import Any import structlog @@ -34,7 +35,7 @@ from prometheus_client import ( Counter, Histogram, Gauge, Info, generate_latest, CONTENT_TYPE_LATEST, - CollectorRegistry, REGISTRY, + REGISTRY, ) PROMETHEUS_AVAILABLE = True @@ -70,7 +71,7 @@ LLM_TOKENS = Counter( "financeai_llm_tokens_total", "Total LLM tokens used", - ["provider", "type"], # type: "prompt" or "completion" + ["provider", "type"], # token category: "prompt" or "completion" ) # Pipeline metrics @@ -93,6 +94,59 @@ "Number of active WebSocket connections", ) + SERVICE_DRAINING = Gauge( + "financeai_service_draining", + "Whether this backend instance is rejecting new non-probe work", + ) + + IN_FLIGHT_REQUESTS = Gauge( + "financeai_in_flight_requests", + "Number of admitted non-probe HTTP requests still in flight", + ) + + WORK_ADMISSION_ACTIVE = Gauge( + "financeai_work_admission_active", + "Active isolated worker operations by bounded work kind", + ["kind"], + ) + + WORK_ADMISSION_QUEUED = Gauge( + "financeai_work_admission_queued", + "Queued isolated worker operations by bounded work kind", + ["kind"], + ) + + WORK_ADMISSION_EVENTS = Counter( + "financeai_work_admission_events_total", + "Isolated worker admission and completion outcomes", + ["kind", "outcome"], + ) + + WORK_ADMISSION_WAIT = Histogram( + "financeai_work_admission_wait_seconds", + "Time spent waiting for an isolated worker slot", + ["kind"], + buckets=[0.001, 0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1.0, 2.5, 5.0, 10.0, 30.0], + ) + + AGENT_RUN_CONTROL_EVENTS = Counter( + "financeai_agent_run_control_events_total", + "Durable agent-run control-plane outcomes with finite labels", + ["event", "outcome"], + ) + + AGENT_RUN_SSE_REPLAY = Counter( + "financeai_agent_run_sse_replay_events_total", + "Agent-run events returned by bounded SSE replay pages", + ["result"], + ) + + AGENT_EGRESS_APPROVALS = Counter( + "financeai_agent_egress_approvals_total", + "Agent-run provider-egress approval outcomes", + ["scope", "outcome"], + ) + # Template metrics TEMPLATE_DEPLOYMENTS = Counter( "financeai_template_deployments_total", @@ -259,7 +313,7 @@ async def wrapper(*args, **kwargs): span.set_attribute("duration_seconds", duration) return result - except Exception as e: + except Exception: if span: span.set_attribute("error", True) raise @@ -294,18 +348,136 @@ def ws_connection_closed(): ACTIVE_WEBSOCKETS.dec() +def set_service_draining(draining: bool) -> None: + """Publish the process drain state as a 0/1 gauge.""" + if PROMETHEUS_AVAILABLE: + SERVICE_DRAINING.set(1 if draining else 0) + + +def set_in_flight_requests(count: int) -> None: + """Publish the exact admitted non-probe request count.""" + if PROMETHEUS_AVAILABLE: + IN_FLIGHT_REQUESTS.set(max(int(count), 0)) + + +def set_work_admission_state(kind: str, active: int, queued: int) -> None: + """Publish bounded worker state without session or document labels.""" + if PROMETHEUS_AVAILABLE: + kind_label = _bounded_work_kind(kind) + WORK_ADMISSION_ACTIVE.labels(kind=kind_label).set(max(int(active), 0)) + WORK_ADMISSION_QUEUED.labels(kind=kind_label).set(max(int(queued), 0)) + + +def record_work_admission_event(kind: str, outcome: str, wait_seconds: float) -> None: + """Record a finite admission outcome and its non-negative queue wait.""" + if PROMETHEUS_AVAILABLE: + kind_label = _bounded_work_kind(kind) + outcome_label = _bounded_work_outcome(outcome) + WORK_ADMISSION_EVENTS.labels(kind=kind_label, outcome=outcome_label).inc() + if outcome_label == "admitted": + WORK_ADMISSION_WAIT.labels(kind=kind_label).observe(max(float(wait_seconds), 0.0)) + + +def _bounded_work_kind(kind: object) -> str: + normalized = str(kind or "").strip().lower() + return normalized if normalized in {"workbook", "pdf", "managed_xlsx"} else "unknown" + + +def _bounded_work_outcome(outcome: object) -> str: + normalized = str(outcome or "").strip().lower() + allowed = { + "admitted", + "completed", + "failed", + "queue_full", + "queue_timeout", + "cancelled", + "draining", + "execution_timeout", + } + return normalized if normalized in allowed else "failed" + + +def record_agent_run_control_event(event: str, outcome: str) -> None: + """Record run-control state without run, session, tenant, or owner labels.""" + if PROMETHEUS_AVAILABLE: + event_label = str(event or "").strip().lower() + if event_label not in { + "created", + "paused", + "resumed", + "cancelled", + "checkpoint_approved", + "recovery", + "step_retry", + }: + event_label = "other" + outcome_label = str(outcome or "").strip().lower() + if outcome_label not in { + "appended", + "replayed", + "sequence_conflict", + "terminal_fence", + "idempotency_conflict", + "not_found", + }: + outcome_label = "rejected" + AGENT_RUN_CONTROL_EVENTS.labels(event=event_label, outcome=outcome_label).inc() + + +def record_agent_run_sse_replay(event_count: int, *, has_more: bool) -> None: + """Record aggregate replay volume without resource identifiers.""" + if PROMETHEUS_AVAILABLE: + result = "continued" if has_more else "complete" + AGENT_RUN_SSE_REPLAY.labels(result=result).inc(max(int(event_count), 0)) + + +def record_agent_egress_approval(scope: str, outcome: str) -> None: + """Record finite egress outcomes without provider model or actor labels.""" + if PROMETHEUS_AVAILABLE: + scope_label = str(scope or "").strip().lower() + if scope_label not in {"local", "cloud"}: + scope_label = "unknown" + outcome_label = str(outcome or "").strip().lower() + if outcome_label not in { + "issued", + "consumed", + "digest_mismatch", + "expired", + "already_consumed", + "not_found", + }: + outcome_label = "rejected" + AGENT_EGRESS_APPROVALS.labels(scope=scope_label, outcome=outcome_label).inc() + + def record_template_deployment(template_id: str): """Track template deployment.""" if PROMETHEUS_AVAILABLE: TEMPLATE_DEPLOYMENTS.labels(template_id=template_id).inc() +def route_template_from_scope(scope: Mapping[str, Any]) -> str: + """Return the matched route template or a bounded unmatched label. + + Raw request paths are deliberately never used because resource identifiers + would create unbounded Prometheus cardinality and leak durable identifiers. + Starlette populates ``scope["route"]`` after routing has completed. + """ + route = scope.get("route") + template = getattr(route, "path_format", None) or getattr(route, "path", None) + if not isinstance(template, str) or not template.startswith("/") or len(template) > 256: + return "__unmatched__" + return template + + def record_request_metrics(method: str, endpoint: str, status: int | str, duration_seconds: float): """Track HTTP request count and latency.""" if PROMETHEUS_AVAILABLE: status_text = str(status) - REQUEST_LATENCY.labels(method=method, endpoint=endpoint, status=status_text).observe(duration_seconds) - REQUEST_COUNT.labels(method=method, endpoint=endpoint, status=status_text).inc() + endpoint_label = endpoint if endpoint.startswith("/") and len(endpoint) <= 256 else "__unmatched__" + REQUEST_LATENCY.labels(method=method, endpoint=endpoint_label, status=status_text).observe(duration_seconds) + REQUEST_COUNT.labels(method=method, endpoint=endpoint_label, status=status_text).inc() def record_v7_readiness_report(status: str, score: float): diff --git a/backend/operations/__init__.py b/backend/operations/__init__.py new file mode 100644 index 0000000..6aa352b --- /dev/null +++ b/backend/operations/__init__.py @@ -0,0 +1 @@ +"""Operator-facing readiness and disposable rehearsal utilities.""" diff --git a/backend/operations/managed_object_store_operator.py b/backend/operations/managed_object_store_operator.py new file mode 100644 index 0000000..93d6f5d --- /dev/null +++ b/backend/operations/managed_object_store_operator.py @@ -0,0 +1,251 @@ +"""Privacy-safe operator status and disposable managed-object rehearsal.""" + +from __future__ import annotations + +import argparse +import base64 +import binascii +import hashlib +import hmac +import json +import os +import re +import tempfile +from collections.abc import Mapping, Sequence +from pathlib import Path +from typing import Any + +from backend.storage.managed_object_store import ( + EncryptedManagedObjectStore, + ManagedObjectIntegrityError, + ManagedObjectKind, + ManagedObjectMetadataRepository, + ManagedObjectScope, + ManagedObjectStoreConfig, + PostgresManagedObjectMetadataRepository, + SQLiteManagedObjectMetadataRepository, +) + + +OPERATOR_SCHEMA_VERSION = "2026-08-managed-object-operator-v1" + + +def build_operator_report( + command: str, + environ: Mapping[str, str] | None = None, +) -> dict[str, Any]: + env = os.environ if environ is None else environ + normalized_command = str(command or "").strip().lower() + if normalized_command == "rehearse": + return _disposable_rehearsal(env) + if normalized_command not in {"status", "recover", "rotate"}: + raise ValueError("Unsupported managed-object operator command") + + config = ManagedObjectStoreConfig.from_environment(env) + if not config.enabled: + return { + "schema_version": OPERATOR_SCHEMA_VERSION, + "command": normalized_command, + "status": "disabled", + "enabled": False, + "production_encryption_claim": False, + "cloud_provider_contacted": False, + } + store = _configured_store(config, env) + if normalized_command == "status": + storage_status = store.operator_status() + report: dict[str, Any] = { + "schema_version": OPERATOR_SCHEMA_VERSION, + "command": "status", + "status": ("ready" if storage_status.get("integrity_status") == "passed" else "failed"), + "storage": storage_status, + } + elif normalized_command == "recover": + recovery = store.recover() + storage_status = store.operator_status() + recovery_passed = ( + int(recovery.get("active_files_missing", -1)) == 0 + and int(recovery.get("recovery_failures", -1)) == 0 + and storage_status.get("integrity_status") == "passed" + ) + report = { + "schema_version": OPERATOR_SCHEMA_VERSION, + "command": "recover", + "status": "passed" if recovery_passed else "failed", + "recovery": recovery, + "storage": storage_status, + } + else: + rotation_id = str(env.get("MANAGED_OBJECT_STORE_ROTATION_ID", "")).strip() + new_key_id = str(env.get("MANAGED_OBJECT_STORE_NEW_KEY_ID", "")).strip() + new_key = _decode_key(env.get("MANAGED_OBJECT_STORE_NEW_KEY", "")) + receipt = store.rotate( + rotation_id=rotation_id, + new_key_id=new_key_id, + new_key=new_key, + ) + storage_status = store.operator_status() + report = { + "schema_version": OPERATOR_SCHEMA_VERSION, + "command": "rotate", + "status": ( + "passed" + if receipt.get("status") == "completed" + and storage_status.get("integrity_status") == "passed" + else "failed" + ), + "rotation": receipt, + "storage": storage_status, + } + report["production_encryption_claim"] = False + report["cloud_provider_contacted"] = False + return report + + +def _configured_store( + config: ManagedObjectStoreConfig, + env: Mapping[str, str], +) -> EncryptedManagedObjectStore: + engine = str(env.get("DB_ENGINE", "sqlite") or "sqlite").strip().lower() + if engine == "sqlite": + db_path = str(env.get("DB_PATH", "") or "").strip() + if not db_path: + raise RuntimeError("Managed-object SQLite metadata database is not configured") + repository: ManagedObjectMetadataRepository = SQLiteManagedObjectMetadataRepository(Path(db_path)) + elif engine == "postgres": + dsn = str(env.get("DATABASE_URL", "") or "").strip() + if not dsn: + raise RuntimeError("Managed-object PostgreSQL metadata database is not configured") + repository = PostgresManagedObjectMetadataRepository(dsn) + else: + raise RuntimeError("Managed-object metadata provider is unsupported") + return EncryptedManagedObjectStore(config, repository) + + +def _disposable_rehearsal(env: Mapping[str, str]) -> dict[str, Any]: + from backend.session_store import SessionStore + + temporary_parent = str(env.get("TMPDIR", "") or "").strip() or None + with tempfile.TemporaryDirectory(prefix="managed-object-rehearsal-", dir=temporary_parent) as raw: + root = Path(raw) + sessions = SessionStore(root / "sessions.db") + session_id = "rehearsal-session" + owner_id = "rehearsal-owner" + sessions.get_or_create_session(session_id, owner_id) + config = ManagedObjectStoreConfig( + enabled=True, + root=root / "objects", + current_key_id="rehearsal-current", + current_key=os.urandom(32), + previous_keys={}, + address_key=os.urandom(32), + max_object_bytes=1024 * 1024, + total_bytes=4 * 1024 * 1024, + chunk_bytes=4096, + ) + store = EncryptedManagedObjectStore( + config, + SQLiteManagedObjectMetadataRepository(sessions.db_path), + ) + scope = ManagedObjectScope("rehearsal-tenant", owner_id, session_id) + payload = os.urandom(64 * 1024) + b"managed-object-rehearsal" + receipt = store.put_bytes(scope, ManagedObjectKind.EVIDENCE, payload) + round_trip_verified = store.get_bytes(scope, receipt.object_id) == payload + + object_file = next((root / "objects" / "objects").rglob("*.mobj")) + original_ciphertext = object_file.read_bytes() + tampered = bytearray(original_ciphertext) + tampered[-20] ^= 0x01 + object_file.write_bytes(tampered) + object_file.chmod(0o600) + try: + store.get_bytes(scope, receipt.object_id) + except ManagedObjectIntegrityError: + tamper_rejected = True + else: + tamper_rejected = False + object_file.write_bytes(original_ciphertext) + object_file.chmod(0o600) + + rotation = store.rotate( + rotation_id="disposable-rehearsal", + new_key_id="rehearsal-next", + new_key=os.urandom(32), + ) + rotation_verified = rotation.get("status") == "completed" and store.get_bytes(scope, receipt.object_id) == payload + sessions.set_session_lifecycle_state( + session_id, + "purging", + receipt_id="disposable-rehearsal-purge", + started_at=1.0, + ) + purge = store.purge_session(session_id) + purge_verified = int(purge.get("residual_files", -1)) == 0 + + release_sha = _verified_release_sha(env.get("RELEASE_SHA", "")) + report: dict[str, Any] = { + "schema_version": OPERATOR_SCHEMA_VERSION, + "command": "rehearse", + "status": "passed" if all((round_trip_verified, tamper_rejected, rotation_verified, purge_verified)) else "failed", + "provider": "disposable_sqlite", + "round_trip_verified": round_trip_verified, + "tamper_rejected": tamper_rejected, + "rotation_verified": rotation_verified, + "purge_verified": purge_verified, + "release_sha": release_sha, + "production_encryption_claim": False, + "cloud_provider_contacted": False, + } + canonical = json.dumps(report, sort_keys=True, separators=(",", ":")).encode() + report["evidence_sha256"] = hashlib.sha256(canonical).hexdigest() + return report + + +def _verified_release_sha(raw: Any) -> str: + normalized = str(raw or "").strip().lower() + if re.fullmatch(r"(?:[0-9a-f]{40}|[0-9a-f]{64})", normalized): + return normalized + return "unavailable" + + +def _decode_key(raw: Any) -> bytes: + encoded = str(raw or "").strip() + if re.fullmatch(r"[0-9a-fA-F]{64}", encoded): + return bytes.fromhex(encoded) + if not re.fullmatch(r"[A-Za-z0-9_-]{43}=?", encoded): + raise ValueError("Managed-object rotation key encoding is invalid") + try: + padding = "=" * (-len(encoded) % 4) + decoded = base64.b64decode(encoded + padding, altchars=b"-_", validate=True) + except (ValueError, binascii.Error) as exc: + raise ValueError("Managed-object rotation key encoding is invalid") from exc + canonical = base64.urlsafe_b64encode(decoded).rstrip(b"=").decode("ascii") + if len(decoded) != 32 or not hmac.compare_digest(canonical, encoded.rstrip("=")): + raise ValueError("Managed-object rotation key encoding is invalid") + return decoded + + +def main(argv: Sequence[str] | None = None) -> int: + parser = argparse.ArgumentParser(description="Managed encrypted object-store operator") + parser.add_argument("command", choices=("status", "rehearse", "recover", "rotate")) + args = parser.parse_args(list(argv) if argv is not None else None) + try: + report = build_operator_report(args.command) + except Exception as exc: + report = { + "schema_version": OPERATOR_SCHEMA_VERSION, + "status": "failed", + "error_code": "managed_object_operator_failed", + "error_type": exc.__class__.__name__, + } + print(json.dumps(report, sort_keys=True, separators=(",", ":"))) + return 1 + print(json.dumps(report, sort_keys=True, separators=(",", ":"))) + return 0 if report.get("status") in {"ready", "passed", "disabled"} else 1 + + +if __name__ == "__main__": + raise SystemExit(main()) + + +__all__ = ["OPERATOR_SCHEMA_VERSION", "build_operator_report", "main"] diff --git a/backend/operations/operator_cli.py b/backend/operations/operator_cli.py new file mode 100644 index 0000000..11fc354 --- /dev/null +++ b/backend/operations/operator_cli.py @@ -0,0 +1,297 @@ +"""Provider-free, machine-readable operator readiness command.""" + +from __future__ import annotations + +import argparse +import io +import json +import re +import sys +from contextlib import redirect_stdout +from datetime import UTC, datetime +from pathlib import Path +from types import ModuleType +from typing import Any, Callable + +from backend.operations.postgres_release_rehearsal import ( + REPORT_SCHEMA_VERSION as POSTGRES_REHEARSAL_SCHEMA_VERSION, + PostgresRehearsalError, + build_target_authorization, + run_postgres_release_rehearsal, +) +from backend.operations.sqlite_rehearsal import run_sqlite_backup_restore_rehearsal +from backend.operations.sqlcipher_rehearsal import run_disposable_sqlcipher_rehearsal +from backend.operations.storage_encryption_attestation import ( + REPORT_SCHEMA_VERSION as STORAGE_ATTESTATION_REPORT_SCHEMA_VERSION, + validate_storage_encryption_attestation, +) +from backend.operations.work_admission import work_admission_controller + + +_LIFECYCLE_CAPABILITIES = ( + "get_session_lifecycle_state", + "get_session_lifecycle_record", + "session_deletion_verified", +) + + +def _bounded_choice(value: object, allowed: frozenset[str]) -> str: + normalized = str(value or "").strip().lower() + return normalized if normalized in allowed else "unknown" + + +def _default_store_factory(): + from backend.storage import get_session_store + + return get_session_store() + + +def build_operator_report( + *, + config_module: ModuleType | Any | None = None, + store_factory: Callable[[], Any] | None = None, +) -> dict[str, Any]: + """Build a redacted readiness report without importing model providers.""" + config_values = config_module + if config_values is None: + from backend import config as default_config + + config_values = default_config + + engine = _bounded_choice(getattr(config_values, "DB_ENGINE", "unknown"), frozenset({"sqlite", "postgres"})) + profile = _bounded_choice( + getattr(config_values, "CONFIG_PROFILE", "unknown"), + frozenset({"dev", "development", "test", "prod", "production", "enterprise"}), + ) + report: dict[str, Any] = { + "schema_version": "operator-readiness-v1", + "generated_at": datetime.now(UTC).isoformat(), + "status": "not_ready", + "version": str(getattr(config_values, "VERSION", "unknown"))[:32], + "model_providers_contacted": False, + "configuration": { + "profile": profile, + "production_mode": bool(getattr(config_values, "PRODUCTION_MODE", False)), + "listen_loopback": str(getattr(config_values, "HOST", "")) in {"127.0.0.1", "::1", "localhost"}, + "port_configured": isinstance(getattr(config_values, "PORT", None), int), + "auth_enabled": bool(getattr(config_values, "AUTH_ENABLED", False)), + "rbac_enabled": bool(getattr(config_values, "RBAC_ENABLED", False)), + "oidc_enabled": bool(getattr(config_values, "OIDC_ENABLED", False)), + "static_key_admin_enabled": bool(getattr(config_values, "STATIC_KEY_ADMIN_ENABLED", False)), + "local_pairing_enabled": bool(getattr(config_values, "LOCAL_PAIRING_ENABLED", False)), + }, + "work_admission": work_admission_controller.redacted_status(), + } + config_validator = getattr(config_values, "runtime_config_errors", None) + if callable(config_validator): + try: + config_errors = config_validator() + except Exception as exc: + report["configuration"]["validation"] = "error" + report["configuration"]["validation_error_count"] = 1 + report["configuration"]["validation_error_type"] = exc.__class__.__name__ + else: + report["configuration"]["validation"] = "error" if config_errors else "ok" + report["configuration"]["validation_error_count"] = len(config_errors) + else: + report["configuration"]["validation"] = "not_available" + report["configuration"]["validation_error_count"] = 0 + + if engine == "postgres": + report["storage"] = { + "configured_engine": "postgres", + "connectivity": "not_probed", + "migration_status": "external_evidence_required", + "pending_migration_count": None, + "applied_migration_count": None, + "external_evidence": False, + } + report["lifecycle"] = { + "status": "not_probed", + "required_capabilities": len(_LIFECYCLE_CAPABILITIES), + "available_capabilities": 0, + } + return report + + if engine != "sqlite": + report["storage"] = { + "configured_engine": "unknown", + "connectivity": "not_probed", + "migration_status": "unknown", + "pending_migration_count": None, + "applied_migration_count": None, + } + report["lifecycle"] = { + "status": "not_probed", + "required_capabilities": len(_LIFECYCLE_CAPABILITIES), + "available_capabilities": 0, + } + return report + + factory = store_factory or _default_store_factory + try: + # Existing store initialization can use structlog's stdout-oriented + # default in command contexts. Discard it so no path or identifier can + # contaminate the machine-readable channel or the operator's stderr. + with redirect_stdout(io.StringIO()): + store = factory() + _ = store.list_sessions(1) + migration = store.get_migration_status() + except Exception as exc: + report["storage"] = { + "configured_engine": "sqlite", + "connectivity": "error", + "migration_status": "unknown", + "pending_migration_count": None, + "applied_migration_count": None, + "error_type": exc.__class__.__name__, + } + report["lifecycle"] = { + "status": "not_probed", + "required_capabilities": len(_LIFECYCLE_CAPABILITIES), + "available_capabilities": 0, + } + return report + + migration = migration if isinstance(migration, dict) else {} + pending_value = migration.get("pending") + applied_value = migration.get("applied") + pending: list[Any] = pending_value if isinstance(pending_value, list) else ["invalid"] + applied: list[Any] = applied_value if isinstance(applied_value, list) else [] + available = sum(callable(getattr(store, name, None)) for name in _LIFECYCLE_CAPABILITIES) + lifecycle_ready = available == len(_LIFECYCLE_CAPABILITIES) + report["storage"] = { + "configured_engine": "sqlite", + "connectivity": "ok", + "migration_status": "pending" if pending else "ok", + "pending_migration_count": len(pending), + "applied_migration_count": len(applied), + } + report["lifecycle"] = { + "status": "ready" if lifecycle_ready else "blocked", + "required_capabilities": len(_LIFECYCLE_CAPABILITIES), + "available_capabilities": available, + } + if not pending and lifecycle_ready and report["configuration"]["validation"] != "error": + report["status"] = "ready" + return report + + +def _parser() -> argparse.ArgumentParser: + parser = argparse.ArgumentParser(description="Excel Agent operator readiness utilities") + parser.add_argument( + "command", + nargs="?", + choices=( + "report", + "sqlite-backup-rehearsal", + "sqlcipher-rehearsal", + "storage-encryption-attestation", + "postgres-release-rehearsal", + ), + default="report", + ) + parser.add_argument("--source-dsn", default="") + parser.add_argument("--restore-dsn", default="") + parser.add_argument("--target-marker", default="") + parser.add_argument("--release-sha", default="") + parser.add_argument("--target-authorization-digest", default="") + parser.add_argument("--timeout-seconds", type=int, default=300) + parser.add_argument("--attestation", default="") + parser.add_argument("--attestation-signature", default="") + parser.add_argument("--attestation-public-key", default="") + parser.add_argument("--attestation-key-id", default="") + parser.add_argument("--attestation-deployment-sha256", default="") + parser.add_argument("--attestation-runner-sha256", default="") + parser.add_argument("--attestation-key-identity-sha256", default="") + parser.add_argument("--attestation-public-key-sha256", default="") + parser.add_argument( + "--attestation-maximum-age-seconds", + type=int, + default=7 * 24 * 60 * 60, + ) + parser.add_argument( + "--authorize-target-only", + action="store_true", + help="Generate the second-step digest for a production-looking disposable target.", + ) + return parser + + +def main(argv: list[str] | None = None) -> int: + args = _parser().parse_args(argv) + if args.command == "postgres-release-rehearsal": + try: + if args.authorize_target_only: + authorization = build_target_authorization( + restore_dsn=args.restore_dsn, + target_marker=args.target_marker, + release_sha=args.release_sha, + ) + report = { + "authorization": authorization, + "liveDatabase": False, + "modelProvidersContacted": False, + "schemaVersion": POSTGRES_REHEARSAL_SCHEMA_VERSION, + "status": "authorization_generated", + } + else: + report = run_postgres_release_rehearsal( + source_dsn=args.source_dsn, + restore_dsn=args.restore_dsn, + target_marker=args.target_marker, + release_sha=args.release_sha, + target_authorization_digest=args.target_authorization_digest, + timeout_seconds=args.timeout_seconds, + ) + except PostgresRehearsalError as exc: + candidate_release_sha = str(args.release_sha or "").strip() + report = { + "errorCode": exc.code, + "liveDatabase": False, + "modelProvidersContacted": False, + "releaseSha": candidate_release_sha + if re.fullmatch(r"[0-9a-f]{40}", candidate_release_sha) + else "", + "schemaVersion": POSTGRES_REHEARSAL_SCHEMA_VERSION, + "status": "failed", + } + elif args.command == "storage-encryption-attestation": + try: + report = validate_storage_encryption_attestation( + receipt_path=Path(args.attestation), + signature_path=Path(args.attestation_signature), + trusted_public_key_path=Path(args.attestation_public_key), + trusted_signing_key_id=args.attestation_key_id, + expected_release_sha=args.release_sha, + expected_deployment_sha256=args.attestation_deployment_sha256, + expected_runner_sha256=args.attestation_runner_sha256, + expected_key_identity_sha256=args.attestation_key_identity_sha256, + expected_public_key_sha256=args.attestation_public_key_sha256, + maximum_age_seconds=args.attestation_maximum_age_seconds, + ) + except Exception: + report = { + "authenticatedExternalAttestation": False, + "errorCode": "storage_encryption_attestation_invalid", + "modelProvidersContacted": False, + "primaryStoreProtected": False, + "releaseSha": args.release_sha + if re.fullmatch(r"[0-9a-f]{40}", str(args.release_sha or "")) + else "", + "schemaVersion": STORAGE_ATTESTATION_REPORT_SCHEMA_VERSION, + "status": "blocked", + } + elif args.command == "sqlcipher-rehearsal": + with redirect_stdout(io.StringIO()): + report = run_disposable_sqlcipher_rehearsal(release_sha=args.release_sha) + elif args.command == "sqlite-backup-rehearsal": + report = run_sqlite_backup_restore_rehearsal() + else: + report = build_operator_report() + sys.stdout.write(json.dumps(report, sort_keys=True, separators=(",", ":")) + "\n") + return 0 if report.get("status") in {"authorization_generated", "ready", "passed"} else 1 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/backend/operations/postgres_release_rehearsal.py b/backend/operations/postgres_release_rehearsal.py new file mode 100644 index 0000000..41cb4c7 --- /dev/null +++ b/backend/operations/postgres_release_rehearsal.py @@ -0,0 +1,1542 @@ +"""Provider-free live PostgreSQL release rehearsal. + +The workflow is deliberately strict: both databases are explicit, the restore +database must be a dedicated marker-named target, every subprocess is invoked +without a shell, and the public report contains only bounded status values, +counts, and digests. It never turns a configured DSN into a claim of live +evidence; PostgreSQL 16 server, transaction, tool, restore, and logical probes +must all pass first. +""" + +from __future__ import annotations + +import asyncio +import hashlib +import hmac +import json +import os +import re +import shutil +import signal +import subprocess +import tempfile +import time +from collections.abc import Mapping +from dataclasses import dataclass +from datetime import UTC, datetime +from pathlib import Path +from typing import Protocol +from urllib.parse import parse_qsl, unquote, urlsplit + +from backend.storage.postgres import ( + POSTGRES_SESSION_LIFECYCLE_TABLES, + POSTGRES_TYPED_PROJECTION_MIGRATIONS, + POSTGRES_TYPED_PROJECTION_RECONCILIATION, + POSTGRES_TYPED_PROJECTION_TABLES, + PostgresSessionStore, +) + + +REPORT_SCHEMA_VERSION = "2026-08-postgres-release-rehearsal-v1" +MANIFEST_SCHEMA_VERSION = "2026-08-postgres-backup-manifest-v1" +AUTHORIZATION_SCHEMA_VERSION = "2026-08-postgres-target-authorization-v1" +SUPPORTED_POSTGRES_MAJOR = 16 +TARGET_DATABASE_PREFIX = "excel_agent_rehearsal_" +_SHA256_RE = re.compile(r"^[0-9a-f]{64}$") +_RELEASE_SHA_RE = re.compile(r"^[0-9a-f]{40}$") +_MARKER_RE = re.compile(r"^[a-z0-9][a-z0-9_-]{5,47}$") +_TOOL_VERSION_EXACT_RE = re.compile(r"^[0-9]{1,3}\.[0-9]{1,3}$") +_VERSION_RE = re.compile(r"PostgreSQL\)?\s+(\d+)\.(\d+)", re.IGNORECASE) +_PRODUCTION_TOKENS = frozenset( + {"main", "prd", "primary", "prod", "production", "live", "customer"} +) +_LIBPQ_QUERY_KEYS = frozenset( + { + "application_name", + "connect_timeout", + "sslcert", + "sslkey", + "sslmode", + "sslrootcert", + "target_session_attrs", + } +) +_ROLLBACK_COMPATIBILITY_COLUMNS: dict[str, frozenset[str]] = { + "sessions": frozenset( + {"session_id", "owner_user_id", "created_at", "updated_at", "metadata"} + ), + "session_lifecycle": frozenset( + { + "session_id", + "state", + "receipt_id", + "started_at", + "completed_at", + "errors", + "updated_at", + } + ), + "flagship_artifacts": frozenset( + { + "artifact_id", + "artifact_type", + "session_id", + "run_id", + "plan_id", + "payload", + "created_at", + "updated_at", + } + ), + "postgres_schema_migrations": frozenset({"migration_id", "applied_at", "details"}), +} +_EXPECTED_PUBLIC_TABLES = frozenset( + { + "conversations", + "enterprise_admin_audits", + "flagship_artifacts", + "postgres_schema_migrations", + "session_lifecycle", + "sessions", + "task_history", + *POSTGRES_TYPED_PROJECTION_TABLES, + } +) +_EXPECTED_RELEASE_CONTRACTS: dict[str, object] = { + "auditPosture": "encrypted_or_redacted", + "lifecycleFencing": "verified", + "lifecycleTombstones": "verified", + "migrationLedger": "verified", + "ownership": "verified", + "typedProjectionBackfills": "verified", +} +_AUDIT_PAYLOAD_COLUMNS = ( + ("enterprise_admin_audits", "payload"), + ("operation_audits", "payload_json"), +) +_SENSITIVE_AUDIT_KEY_PATTERN = ( + "api[_-]?key|authorization|password|secret|database[_-]?url|dsn|" + "connection[_-]?string|access[_-]?token|refresh[_-]?token" +) +_SAFE_AUDIT_VALUE_PATTERN = ( + r"(?:\[REDACTED\]||redacted|enc:v[0-9]+:[^\"]+|sha256:[0-9a-f]{64})" +) + + +class PostgresRehearsalError(RuntimeError): + """A privacy-safe, stable failure from the rehearsal workflow.""" + + def __init__(self, code: str) -> None: + normalized = str(code or "postgres_rehearsal_failed").strip().lower() + if not re.fullmatch(r"[a-z0-9_]{3,80}", normalized): + normalized = "postgres_rehearsal_failed" + self.code = normalized + super().__init__(normalized) + + +@dataclass(frozen=True) +class DatabaseProbe: + database_name: str + database_fingerprint: str + server_major: int + server_version_sha256: str + transaction_probe: bool + live_postgres: bool + + +@dataclass(frozen=True) +class DatabaseSnapshot: + table_counts: dict[str, int] + counts_sha256: str + schema_sha256: str + logical_sha256: str + sequence_sha256: str + lifecycle_tombstone_sha256: str + lifecycle_state_counts: dict[str, int] + + +@dataclass(frozen=True) +class ToolVersions: + pg_dump: str + pg_restore: str + pg_dump_sha256: str + pg_restore_sha256: str + + +class RehearsalDatabase(Protocol): + def probe(self, dsn: str, *, timeout_seconds: int) -> DatabaseProbe: ... + + def assert_disposable_target_empty( + self, + dsn: str, + *, + expected_database: str, + timeout_seconds: int, + ) -> None: ... + + def run_current_migrations(self, dsn: str, *, timeout_seconds: int) -> None: ... + + def prepare_disposable_target( + self, + dsn: str, + *, + expected_database: str, + timeout_seconds: int, + ) -> None: ... + + def capture_snapshot(self, dsn: str, *, timeout_seconds: int) -> DatabaseSnapshot: ... + + def validate_release_contracts( + self, + dsn: str, + *, + timeout_seconds: int, + ) -> dict[str, object]: ... + + def cleanup_disposable_target( + self, + dsn: str, + *, + expected_database: str, + timeout_seconds: int, + ) -> None: ... + + +class RehearsalTools(Protocol): + def versions(self, *, timeout_seconds: int) -> ToolVersions: ... + + def dump( + self, + dsn: str, + artifact_path: Path, + *, + timeout_seconds: int, + workspace: Path, + ) -> None: ... + + def restore( + self, + dsn: str, + artifact_path: Path, + *, + expected_sha256: str, + timeout_seconds: int, + workspace: Path, + ) -> None: ... + + +@dataclass(frozen=True) +class _ParsedDsn: + hostname: str + port: int + database: str + username: str + password: str + query: tuple[tuple[str, str], ...] + + @property + def fingerprint(self) -> str: + identity = f"{self.hostname.lower()}:{self.port}/{self.database.lower()}" + return _sha256_bytes(identity.encode("utf-8")) + + +class _Deadline: + def __init__(self, timeout_seconds: int) -> None: + if not 30 <= timeout_seconds <= 3600: + raise PostgresRehearsalError("timeout_seconds_invalid") + self._deadline = time.monotonic() + timeout_seconds + + def remaining(self) -> int: + remaining = int(self._deadline - time.monotonic()) + if remaining < 1: + raise PostgresRehearsalError("rehearsal_timeout") + return remaining + + +def _canonical_json(payload: object) -> bytes: + return json.dumps( + payload, + sort_keys=True, + separators=(",", ":"), + ensure_ascii=True, + ).encode("utf-8") + + +def _json_safe(value: object) -> object: + if value is None or isinstance(value, (bool, float, int, str)): + return value + if isinstance(value, bytes): + return value.hex() + if isinstance(value, Mapping): + return {str(key): _json_safe(child) for key, child in value.items()} + if isinstance(value, (list, tuple)): + return [_json_safe(child) for child in value] + return str(value) + + +def _sha256_bytes(payload: bytes) -> str: + return hashlib.sha256(payload).hexdigest() + + +def _sha256_file(path: Path) -> str: + if not path.is_file() or path.is_symlink(): + raise PostgresRehearsalError("backup_artifact_invalid") + digest = hashlib.sha256() + try: + with path.open("rb") as handle: + while chunk := handle.read(1024 * 1024): + digest.update(chunk) + except OSError as exc: + raise PostgresRehearsalError("backup_artifact_unreadable") from exc + return digest.hexdigest() + + +def _parse_dsn(value: str, *, missing_code: str) -> _ParsedDsn: + raw = str(value or "").strip() + if not raw: + raise PostgresRehearsalError(missing_code) + try: + parsed = urlsplit(raw) + port = parsed.port or 5432 + except ValueError as exc: + raise PostgresRehearsalError("postgres_dsn_invalid") from exc + if parsed.scheme not in {"postgres", "postgresql"}: + raise PostgresRehearsalError("postgres_dsn_invalid") + hostname = str(parsed.hostname or "").strip() + username = unquote(parsed.username or "").strip() + database = unquote(parsed.path.lstrip("/")).strip() + if not hostname or not username or not database or parsed.fragment: + raise PostgresRehearsalError("postgres_dsn_invalid") + query = tuple(sorted(parse_qsl(parsed.query, keep_blank_values=True))) + if any(key not in _LIBPQ_QUERY_KEYS for key, _ in query): + raise PostgresRehearsalError("postgres_dsn_option_unsupported") + for field in (hostname, username, database, unquote(parsed.password or "")): + if "\n" in field or "\r" in field or "\x00" in field: + raise PostgresRehearsalError("postgres_dsn_invalid") + return _ParsedDsn( + hostname=hostname, + port=port, + database=database, + username=username, + password=unquote(parsed.password or ""), + query=query, + ) + + +def _normalized_marker(marker: str) -> str: + normalized = str(marker or "").strip().lower() + if not _MARKER_RE.fullmatch(normalized): + raise PostgresRehearsalError("target_marker_invalid") + return normalized + + +def _expected_target_database(marker: str) -> str: + return TARGET_DATABASE_PREFIX + marker.replace("-", "_") + + +def _validate_release_sha(release_sha: str) -> str: + normalized = str(release_sha or "").strip() + if not _RELEASE_SHA_RE.fullmatch(normalized): + raise PostgresRehearsalError("release_sha_invalid") + return normalized + + +def _production_looking_target(parsed: _ParsedDsn) -> bool: + tokens = { + token + for token in re.split(r"[^a-z0-9]+", f"{parsed.hostname.lower()} {parsed.database.lower()}") + if token + } + return bool(tokens & _PRODUCTION_TOKENS) or any( + token.startswith(("customer", "live", "primary", "prod")) for token in tokens + ) + + +def build_target_authorization( + *, + restore_dsn: str, + target_marker: str, + release_sha: str, +) -> dict[str, str]: + """Build the privacy-safe second-step confirmation for a risky target name.""" + parsed = _parse_dsn(restore_dsn, missing_code="restore_dsn_required") + marker = _normalized_marker(target_marker) + exact_sha = _validate_release_sha(release_sha) + expected_database = _expected_target_database(marker) + if parsed.database != expected_database: + raise PostgresRehearsalError("restore_database_marker_mismatch") + payload = { + "action": "authorize_disposable_postgres_restore_and_cleanup", + "releaseSha": exact_sha, + "schemaVersion": AUTHORIZATION_SCHEMA_VERSION, + "targetFingerprint": parsed.fingerprint, + "targetMarker": marker, + } + return { + "authorizationDigest": _sha256_bytes(_canonical_json(payload)), + "releaseSha": exact_sha, + "targetFingerprint": parsed.fingerprint, + "targetMarker": marker, + } + + +def _validate_request( + *, + source_dsn: str, + restore_dsn: str, + target_marker: str, + release_sha: str, + target_authorization_digest: str, +) -> tuple[_ParsedDsn, _ParsedDsn, str, str]: + source = _parse_dsn(source_dsn, missing_code="source_dsn_required") + restore = _parse_dsn(restore_dsn, missing_code="restore_dsn_required") + marker = _normalized_marker(target_marker) + exact_sha = _validate_release_sha(release_sha) + if source.fingerprint == restore.fingerprint: + raise PostgresRehearsalError("source_restore_not_separate") + expected_database = _expected_target_database(marker) + if restore.database != expected_database: + raise PostgresRehearsalError("restore_database_marker_mismatch") + if _production_looking_target(restore): + expected_authorization = build_target_authorization( + restore_dsn=restore_dsn, + target_marker=marker, + release_sha=exact_sha, + )["authorizationDigest"] + supplied = str(target_authorization_digest or "").strip() + if not _SHA256_RE.fullmatch(supplied) or not hmac.compare_digest( + supplied, + expected_authorization, + ): + raise PostgresRehearsalError("production_target_authorization_required") + return source, restore, marker, exact_sha + + +def create_backup_manifest( + *, + artifact_path: Path, + manifest_path: Path, + release_sha: str, + source_snapshot: DatabaseSnapshot, + source_probe: DatabaseProbe, + tool_versions: ToolVersions, + created_at: str, +) -> dict[str, object]: + """Bind a custom-format dump to the exact release and source snapshot.""" + if manifest_path.exists() or manifest_path.is_symlink(): + raise PostgresRehearsalError("backup_manifest_target_exists") + artifact_sha256 = _sha256_file(artifact_path) + try: + artifact_size = artifact_path.stat().st_size + with artifact_path.open("rb") as handle: + magic = handle.read(5) + except OSError as exc: + raise PostgresRehearsalError("backup_artifact_unreadable") from exc + if artifact_size < 6 or magic != b"PGDMP": + raise PostgresRehearsalError("backup_artifact_format_invalid") + manifest: dict[str, object] = { + "artifactBytes": artifact_size, + "artifactSha256": artifact_sha256, + "countsSha256": source_snapshot.counts_sha256, + "createdAt": str(created_at), + "logicalSha256": source_snapshot.logical_sha256, + "pgDumpSha256": tool_versions.pg_dump_sha256, + "pgDumpVersion": tool_versions.pg_dump, + "releaseSha": _validate_release_sha(release_sha), + "schemaSha256": source_snapshot.schema_sha256, + "schemaVersion": MANIFEST_SCHEMA_VERSION, + "sequenceSha256": source_snapshot.sequence_sha256, + "sourceDatabaseFingerprint": source_probe.database_fingerprint, + "sourcePostgresMajor": source_probe.server_major, + } + envelope: dict[str, object] = { + "manifest": manifest, + "manifestSha256": _sha256_bytes(_canonical_json(manifest)), + } + try: + manifest_path.write_bytes(_canonical_json(envelope) + b"\n") + manifest_path.chmod(0o600) + except OSError as exc: + raise PostgresRehearsalError("backup_manifest_write_failed") from exc + return envelope + + +def verify_backup_manifest( + *, + artifact_path: Path, + manifest_path: Path, +) -> dict[str, object]: + if ( + not manifest_path.is_file() + or manifest_path.is_symlink() + or manifest_path.stat().st_size > 64 * 1024 + ): + raise PostgresRehearsalError("backup_manifest_invalid") + try: + envelope = json.loads(manifest_path.read_text(encoding="utf-8")) + except (OSError, UnicodeError, json.JSONDecodeError) as exc: + raise PostgresRehearsalError("backup_manifest_invalid") from exc + if not isinstance(envelope, dict) or set(envelope) != {"manifest", "manifestSha256"}: + raise PostgresRehearsalError("backup_manifest_invalid") + manifest = envelope.get("manifest") + if not isinstance(manifest, dict): + raise PostgresRehearsalError("backup_manifest_invalid") + expected_fields = { + "artifactBytes", + "artifactSha256", + "countsSha256", + "createdAt", + "logicalSha256", + "pgDumpSha256", + "pgDumpVersion", + "releaseSha", + "schemaSha256", + "schemaVersion", + "sequenceSha256", + "sourceDatabaseFingerprint", + "sourcePostgresMajor", + } + if set(manifest) != expected_fields or manifest.get("schemaVersion") != MANIFEST_SCHEMA_VERSION: + raise PostgresRehearsalError("backup_manifest_invalid") + digest_fields = ( + "artifactSha256", + "countsSha256", + "logicalSha256", + "pgDumpSha256", + "schemaSha256", + "sequenceSha256", + "sourceDatabaseFingerprint", + ) + if ( + any(not _SHA256_RE.fullmatch(str(manifest.get(field) or "")) for field in digest_fields) + or not _RELEASE_SHA_RE.fullmatch(str(manifest.get("releaseSha") or "")) + or manifest.get("sourcePostgresMajor") != SUPPORTED_POSTGRES_MAJOR + or _tool_major(str(manifest.get("pgDumpVersion") or "")) != SUPPORTED_POSTGRES_MAJOR + or not isinstance(manifest.get("artifactBytes"), int) + or int(manifest["artifactBytes"]) < 6 + ): + raise PostgresRehearsalError("backup_manifest_invalid") + expected_manifest_sha = _sha256_bytes(_canonical_json(manifest)) + if not hmac.compare_digest(str(envelope.get("manifestSha256") or ""), expected_manifest_sha): + raise PostgresRehearsalError("backup_manifest_digest_mismatch") + artifact_sha = _sha256_file(artifact_path) + if not hmac.compare_digest(str(manifest.get("artifactSha256") or ""), artifact_sha): + raise PostgresRehearsalError("backup_artifact_digest_mismatch") + try: + artifact_size = artifact_path.stat().st_size + except OSError as exc: + raise PostgresRehearsalError("backup_artifact_unreadable") from exc + if manifest.get("artifactBytes") != artifact_size: + raise PostgresRehearsalError("backup_artifact_size_mismatch") + try: + with artifact_path.open("rb") as handle: + magic = handle.read(5) + except OSError as exc: + raise PostgresRehearsalError("backup_artifact_unreadable") from exc + if magic != b"PGDMP": + raise PostgresRehearsalError("backup_artifact_format_invalid") + return envelope + + +def _assert_probe(probe: DatabaseProbe, *, expected_fingerprint: str) -> None: + if ( + not probe.live_postgres + or not probe.transaction_probe + or probe.database_fingerprint != expected_fingerprint + or not _SHA256_RE.fullmatch(probe.server_version_sha256) + ): + code = ( + "postgres_transaction_probe_failed" + if not probe.transaction_probe + else "postgres_live_probe_failed" + ) + raise PostgresRehearsalError(code) + if probe.server_major != SUPPORTED_POSTGRES_MAJOR: + raise PostgresRehearsalError("postgres_version_unsupported") + + +def _tool_major(version: str) -> int: + try: + return int(str(version).split(".", 1)[0]) + except (TypeError, ValueError) as exc: + raise PostgresRehearsalError("postgres_tool_version_invalid") from exc + + +def _assert_tool_versions(versions: ToolVersions) -> None: + if ( + not _TOOL_VERSION_EXACT_RE.fullmatch(versions.pg_dump) + or not _TOOL_VERSION_EXACT_RE.fullmatch(versions.pg_restore) + or + _tool_major(versions.pg_dump) != SUPPORTED_POSTGRES_MAJOR + or _tool_major(versions.pg_restore) != SUPPORTED_POSTGRES_MAJOR + ): + raise PostgresRehearsalError("postgres_tool_version_mismatch") + if not _SHA256_RE.fullmatch(versions.pg_dump_sha256) or not _SHA256_RE.fullmatch( + versions.pg_restore_sha256 + ): + raise PostgresRehearsalError("postgres_tool_digest_invalid") + + +def _assert_snapshot_shape(snapshot: DatabaseSnapshot) -> None: + if ( + not set(snapshot.table_counts).issubset(_EXPECTED_PUBLIC_TABLES) + or any(not isinstance(count, int) or count < 0 for count in snapshot.table_counts.values()) + or any( + not _SHA256_RE.fullmatch(value) + for value in ( + snapshot.counts_sha256, + snapshot.lifecycle_tombstone_sha256, + snapshot.logical_sha256, + snapshot.schema_sha256, + snapshot.sequence_sha256, + ) + ) + or not set(snapshot.lifecycle_state_counts).issubset( + {"active", "failed", "purging", "verified"} + ) + or any( + not isinstance(count, int) or count < 0 + for count in snapshot.lifecycle_state_counts.values() + ) + ): + raise PostgresRehearsalError("postgres_snapshot_invalid") + + +def _assert_release_contracts(contracts: dict[str, object]) -> None: + if contracts != _EXPECTED_RELEASE_CONTRACTS: + raise PostgresRehearsalError("postgres_contract_validation_failed") + + +def _normalized_generated_at(value: str | None) -> str: + if value is None: + return datetime.now(UTC).isoformat() + try: + parsed = datetime.fromisoformat(str(value)) + except ValueError as exc: + raise PostgresRehearsalError("generated_at_invalid") from exc + if parsed.tzinfo is None: + raise PostgresRehearsalError("generated_at_invalid") + return parsed.astimezone(UTC).isoformat() + + +def _assert_snapshot_equal( + expected: DatabaseSnapshot, + actual: DatabaseSnapshot, + *, + code: str, +) -> None: + if expected != actual: + raise PostgresRehearsalError(code) + + +def _public_probe(probe: DatabaseProbe) -> dict[str, object]: + return { + "databaseFingerprint": probe.database_fingerprint, + "livePostgres": probe.live_postgres, + "postgresMajor": probe.server_major, + "serverVersionSha256": probe.server_version_sha256, + "transactionProbe": probe.transaction_probe, + } + + +def _snapshot_digests(snapshot: DatabaseSnapshot) -> dict[str, str]: + return { + "countsSha256": snapshot.counts_sha256, + "lifecycleTombstoneSha256": snapshot.lifecycle_tombstone_sha256, + "logicalSha256": snapshot.logical_sha256, + "schemaSha256": snapshot.schema_sha256, + "sequenceSha256": snapshot.sequence_sha256, + } + + +def run_postgres_release_rehearsal( + *, + source_dsn: str, + restore_dsn: str, + target_marker: str, + release_sha: str, + target_authorization_digest: str = "", + timeout_seconds: int = 300, + database: RehearsalDatabase | None = None, + tools: RehearsalTools | None = None, + generated_at: str | None = None, +) -> dict[str, object]: + """Run a bounded migration/dump/restore/rollback-compatibility rehearsal.""" + source_config, restore_config, marker, exact_sha = _validate_request( + source_dsn=source_dsn, + restore_dsn=restore_dsn, + target_marker=target_marker, + release_sha=release_sha, + target_authorization_digest=target_authorization_digest, + ) + deadline = _Deadline(timeout_seconds) + database_adapter = database or AsyncpgRehearsalDatabase() + tool_adapter = tools or LibpqToolRunner() + expected_target_database = _expected_target_database(marker) + target_preflighted = False + generated = _normalized_generated_at(generated_at) + + try: + source_probe = database_adapter.probe(source_dsn, timeout_seconds=deadline.remaining()) + restore_probe = database_adapter.probe(restore_dsn, timeout_seconds=deadline.remaining()) + _assert_probe(source_probe, expected_fingerprint=source_config.fingerprint) + _assert_probe(restore_probe, expected_fingerprint=restore_config.fingerprint) + if source_probe.database_fingerprint == restore_probe.database_fingerprint: + raise PostgresRehearsalError("source_restore_not_separate") + if restore_probe.database_name != expected_target_database: + raise PostgresRehearsalError("restore_database_marker_mismatch") + + database_adapter.assert_disposable_target_empty( + restore_dsn, + expected_database=expected_target_database, + timeout_seconds=deadline.remaining(), + ) + target_preflighted = True + versions = tool_adapter.versions(timeout_seconds=deadline.remaining()) + _assert_tool_versions(versions) + + pre_migration = database_adapter.capture_snapshot( + source_dsn, + timeout_seconds=deadline.remaining(), + ) + _assert_snapshot_shape(pre_migration) + database_adapter.run_current_migrations( + source_dsn, + timeout_seconds=deadline.remaining(), + ) + post_migration = database_adapter.capture_snapshot( + source_dsn, + timeout_seconds=deadline.remaining(), + ) + _assert_snapshot_shape(post_migration) + source_contracts = database_adapter.validate_release_contracts( + source_dsn, + timeout_seconds=deadline.remaining(), + ) + _assert_release_contracts(source_contracts) + database_adapter.run_current_migrations( + source_dsn, + timeout_seconds=deadline.remaining(), + ) + idempotent_snapshot = database_adapter.capture_snapshot( + source_dsn, + timeout_seconds=deadline.remaining(), + ) + _assert_snapshot_shape(idempotent_snapshot) + _assert_snapshot_equal( + post_migration, + idempotent_snapshot, + code="source_migration_not_idempotent", + ) + + with tempfile.TemporaryDirectory(prefix="excel-agent-postgres-rehearsal-") as temp_dir: + workspace = Path(temp_dir) + artifact_path = workspace / "source.dump" + manifest_path = workspace / "source.manifest.json" + tool_adapter.dump( + source_dsn, + artifact_path, + timeout_seconds=deadline.remaining(), + workspace=workspace, + ) + source_after_dump = database_adapter.capture_snapshot( + source_dsn, + timeout_seconds=deadline.remaining(), + ) + _assert_snapshot_shape(source_after_dump) + _assert_snapshot_equal(post_migration, source_after_dump, code="source_drift_during_dump") + envelope = create_backup_manifest( + artifact_path=artifact_path, + manifest_path=manifest_path, + release_sha=exact_sha, + source_snapshot=post_migration, + source_probe=source_probe, + tool_versions=versions, + created_at=generated, + ) + verify_backup_manifest(artifact_path=artifact_path, manifest_path=manifest_path) + manifest = envelope["manifest"] + if not isinstance(manifest, dict): + raise PostgresRehearsalError("backup_manifest_invalid") + artifact_sha256 = str(manifest["artifactSha256"]) + database_adapter.prepare_disposable_target( + restore_dsn, + expected_database=expected_target_database, + timeout_seconds=deadline.remaining(), + ) + tool_adapter.restore( + restore_dsn, + artifact_path, + expected_sha256=artifact_sha256, + timeout_seconds=deadline.remaining(), + workspace=workspace, + ) + restored_snapshot = database_adapter.capture_snapshot( + restore_dsn, + timeout_seconds=deadline.remaining(), + ) + _assert_snapshot_shape(restored_snapshot) + _assert_snapshot_equal(post_migration, restored_snapshot, code="restore_logical_drift") + restore_contracts = database_adapter.validate_release_contracts( + restore_dsn, + timeout_seconds=deadline.remaining(), + ) + _assert_release_contracts(restore_contracts) + if source_contracts != restore_contracts: + raise PostgresRehearsalError("restore_contract_drift") + + # Re-running the current application migration path proves that a + # restored backup remains boot-compatible. No DROP, migration + # reversal, or schema downgrade is attempted. + database_adapter.run_current_migrations( + restore_dsn, + timeout_seconds=deadline.remaining(), + ) + rollback_compatibility = database_adapter.capture_snapshot( + restore_dsn, + timeout_seconds=deadline.remaining(), + ) + _assert_snapshot_shape(rollback_compatibility) + _assert_snapshot_equal( + restored_snapshot, + rollback_compatibility, + code="rollback_compatibility_drift", + ) + + artifact_bytes = int(manifest["artifactBytes"]) + manifest_sha256 = str(envelope["manifestSha256"]) + + database_adapter.cleanup_disposable_target( + restore_dsn, + expected_database=expected_target_database, + timeout_seconds=deadline.remaining(), + ) + target_preflighted = False + except KeyboardInterrupt as exc: + if target_preflighted: + _cleanup_after_failure( + database_adapter, + restore_dsn, + expected_target_database, + deadline, + ) + raise PostgresRehearsalError("rehearsal_cancelled") from exc + except PostgresRehearsalError: + if target_preflighted: + _cleanup_after_failure( + database_adapter, + restore_dsn, + expected_target_database, + deadline, + ) + raise + except Exception as exc: + if target_preflighted: + _cleanup_after_failure( + database_adapter, + restore_dsn, + expected_target_database, + deadline, + ) + raise PostgresRehearsalError("postgres_rehearsal_failed") from exc + + checks = { + "applicationRollbackCompatibility": "verified_without_schema_downgrade", + "auditPosture": str(source_contracts["auditPosture"]), + "backupDigest": "verified", + "lifecycleFencing": str(source_contracts["lifecycleFencing"]), + "lifecycleTombstones": str(source_contracts["lifecycleTombstones"]), + "logicalEquality": "verified", + "manifestDigest": "verified", + "migrationLedger": str(source_contracts["migrationLedger"]), + "ownership": str(source_contracts["ownership"]), + "restoreCleanup": "verified", + "schemaEquality": "verified", + "transactionProbes": "verified", + "typedProjectionBackfills": str(source_contracts["typedProjectionBackfills"]), + } + candidate_details = { + "applicationRollbackCompatibility": True, + "auditPosture": checks["auditPosture"], + "backupSha256": artifact_sha256, + "liveDatabase": True, + "manifestSha256": manifest_sha256, + "postgresMajor": SUPPORTED_POSTGRES_MAJOR, + "restoreVerified": True, + "schemaSha256": post_migration.schema_sha256, + "successful": True, + } + return { + "artifact": { + "bytes": artifact_bytes, + "manifestSha256": manifest_sha256, + "sha256": artifact_sha256, + }, + "canonicalCounts": { + "postMigration": post_migration.table_counts, + "preMigration": pre_migration.table_counts, + "restored": restored_snapshot.table_counts, + }, + "checks": checks, + "generatedAt": generated, + "liveDatabase": True, + "modelProvidersContacted": False, + "releaseEvidenceCandidate": { + "details": candidate_details, + "id": "postgres_backup_restore", + "releaseSha": exact_sha, + "status": "passed", + }, + "releaseSha": exact_sha, + "restore": _public_probe(restore_probe), + "schemaVersion": REPORT_SCHEMA_VERSION, + "snapshotDigests": { + "postMigration": _snapshot_digests(post_migration), + "preMigration": _snapshot_digests(pre_migration), + "restored": _snapshot_digests(restored_snapshot), + }, + "source": _public_probe(source_probe), + "status": "passed", + "tools": { + "pgDumpSha256": versions.pg_dump_sha256, + "pgDumpVersion": versions.pg_dump, + "pgRestoreSha256": versions.pg_restore_sha256, + "pgRestoreVersion": versions.pg_restore, + }, + } + + +def _cleanup_after_failure( + database: RehearsalDatabase, + restore_dsn: str, + expected_database: str, + deadline: _Deadline, +) -> None: + try: + database.cleanup_disposable_target( + restore_dsn, + expected_database=expected_database, + timeout_seconds=deadline.remaining(), + ) + except Exception as exc: + raise PostgresRehearsalError("target_cleanup_failed") from exc + + +def _quote_ident(identifier: str) -> str: + return '"' + identifier.replace('"', '""') + '"' + + +class AsyncpgRehearsalDatabase: + """Live PostgreSQL implementation; imports asyncpg only when invoked.""" + + def probe(self, dsn: str, *, timeout_seconds: int) -> DatabaseProbe: + return self._run(self._probe(dsn), timeout_seconds, "postgres_live_probe_failed") + + def assert_disposable_target_empty( + self, + dsn: str, + *, + expected_database: str, + timeout_seconds: int, + ) -> None: + self._run( + self._assert_disposable_target_empty(dsn, expected_database), + timeout_seconds, + "restore_target_preflight_failed", + ) + + def run_current_migrations(self, dsn: str, *, timeout_seconds: int) -> None: + self._run( + self._run_current_migrations(dsn), + timeout_seconds, + "postgres_migration_failed", + ) + + def prepare_disposable_target( + self, + dsn: str, + *, + expected_database: str, + timeout_seconds: int, + ) -> None: + self._run( + self._prepare_disposable_target(dsn, expected_database), + timeout_seconds, + "restore_target_prepare_failed", + ) + + def capture_snapshot(self, dsn: str, *, timeout_seconds: int) -> DatabaseSnapshot: + return self._run( + self._capture_snapshot(dsn), + timeout_seconds, + "postgres_snapshot_failed", + ) + + def validate_release_contracts( + self, + dsn: str, + *, + timeout_seconds: int, + ) -> dict[str, object]: + return self._run( + self._validate_release_contracts(dsn), + timeout_seconds, + "postgres_contract_validation_failed", + ) + + def cleanup_disposable_target( + self, + dsn: str, + *, + expected_database: str, + timeout_seconds: int, + ) -> None: + self._run( + self._cleanup_disposable_target(dsn, expected_database), + timeout_seconds, + "target_cleanup_failed", + ) + + @staticmethod + def _run(coroutine, timeout_seconds: int, error_code: str): + try: + return asyncio.run(asyncio.wait_for(coroutine, timeout=max(1, timeout_seconds))) + except PostgresRehearsalError: + raise + except TimeoutError as exc: + raise PostgresRehearsalError("postgres_database_timeout") from exc + except KeyboardInterrupt: + raise + except Exception as exc: + raise PostgresRehearsalError(error_code) from exc + + @staticmethod + async def _connect(dsn: str): + try: + import asyncpg + except ImportError as exc: # pragma: no cover - optional runtime dependency + raise PostgresRehearsalError("asyncpg_unavailable") from exc + return await asyncpg.connect(dsn=dsn, timeout=15, command_timeout=60) + + async def _probe(self, dsn: str) -> DatabaseProbe: + parsed = _parse_dsn(dsn, missing_code="postgres_dsn_required") + conn = await self._connect(dsn) + transaction_probe = False + try: + row = await conn.fetchrow( + "SELECT current_database() AS database_name, " + "current_setting('server_version_num') AS version_num, version() AS version_text" + ) + transaction = conn.transaction(readonly=True) + await transaction.start() + try: + transaction_probe = bool(await conn.fetchval("SELECT 1")) + finally: + await transaction.rollback() + finally: + await conn.close() + if row is None: + raise PostgresRehearsalError("postgres_live_probe_failed") + version_text = str(row["version_text"] or "") + version_num = int(str(row["version_num"] or "0")) + return DatabaseProbe( + database_name=str(row["database_name"] or ""), + database_fingerprint=parsed.fingerprint, + server_major=version_num // 10000, + server_version_sha256=_sha256_bytes(version_text.encode("utf-8")), + transaction_probe=transaction_probe, + live_postgres=version_text.startswith("PostgreSQL "), + ) + + async def _assert_disposable_target_empty(self, dsn: str, expected_database: str) -> None: + conn = await self._connect(dsn) + try: + database_name = str(await conn.fetchval("SELECT current_database()") or "") + if database_name != expected_database: + raise PostgresRehearsalError("restore_database_marker_mismatch") + relation_count = int( + await conn.fetchval( + "SELECT COUNT(*)::int FROM pg_class AS class " + "JOIN pg_namespace AS namespace ON namespace.oid = class.relnamespace " + "WHERE class.relkind IN ('r', 'p', 'S', 'v', 'm') " + "AND namespace.nspname NOT LIKE 'pg_%' " + "AND namespace.nspname <> 'information_schema'" + ) + or 0 + ) + if relation_count: + raise PostgresRehearsalError("restore_target_not_empty") + finally: + await conn.close() + + async def _prepare_disposable_target(self, dsn: str, expected_database: str) -> None: + conn = await self._connect(dsn) + try: + database_name = str(await conn.fetchval("SELECT current_database()") or "") + if database_name != expected_database: + raise PostgresRehearsalError("restore_database_marker_mismatch") + await conn.execute("DROP SCHEMA IF EXISTS public CASCADE") + finally: + await conn.close() + + async def _run_current_migrations(self, dsn: str) -> None: + try: + import asyncpg + except ImportError as exc: # pragma: no cover - optional runtime dependency + raise PostgresRehearsalError("asyncpg_unavailable") from exc + pool = await asyncpg.create_pool( + dsn=dsn, + min_size=1, + max_size=2, + command_timeout=60, + ) + try: + store = PostgresSessionStore(dsn) + store._pool = pool + store._pool_loop = asyncio.get_running_loop() + await store._init_schema() + finally: + await pool.close() + + async def _capture_snapshot(self, dsn: str) -> DatabaseSnapshot: + conn = await self._connect(dsn) + try: + table_rows = await conn.fetch( + "SELECT namespace.nspname AS schema_name, class.relname AS table_name " + "FROM pg_class AS class " + "JOIN pg_namespace AS namespace ON namespace.oid = class.relnamespace " + "WHERE class.relkind IN ('r', 'p') " + "AND namespace.nspname = 'public' " + "ORDER BY class.relname" + ) + table_names = [str(row["table_name"]) for row in table_rows] + table_counts: dict[str, int] = {} + logical_digest = hashlib.sha256() + for table_name in table_names: + quoted = _quote_ident(table_name) + count = int(await conn.fetchval(f"SELECT COUNT(*)::bigint FROM public.{quoted}") or 0) + table_counts[table_name] = count + logical_digest.update(f"{table_name}:{count}:".encode()) + async with conn.transaction(readonly=True): + cursor = conn.cursor( + f"SELECT md5(to_jsonb(row_value)::text) AS row_digest " + f"FROM public.{quoted} AS row_value ORDER BY 1", + prefetch=1000, + ) + async for record in cursor: + logical_digest.update(str(record["row_digest"] or "").encode("ascii")) + + schema_rows = await conn.fetch( + "SELECT columns.table_schema, columns.table_name, columns.column_name, " + "columns.ordinal_position, columns.data_type, columns.udt_name, " + "columns.is_nullable, COALESCE(columns.column_default, '') AS column_default " + "FROM information_schema.columns AS columns " + "WHERE columns.table_schema = 'public' " + "ORDER BY columns.table_name, columns.ordinal_position" + ) + constraint_rows = await conn.fetch( + "SELECT class.relname AS table_name, con.contype, con.conname, " + "pg_get_constraintdef(con.oid, true) AS definition " + "FROM pg_constraint AS con " + "JOIN pg_class AS class ON class.oid = con.conrelid " + "JOIN pg_namespace AS namespace ON namespace.oid = class.relnamespace " + "WHERE namespace.nspname = 'public' " + "ORDER BY class.relname, con.conname" + ) + index_rows = await conn.fetch( + "SELECT tablename, indexname, indexdef FROM pg_indexes " + "WHERE schemaname = 'public' ORDER BY tablename, indexname" + ) + trigger_rows = await conn.fetch( + "SELECT class.relname AS table_name, trigger.tgname, " + "pg_get_triggerdef(trigger.oid, true) AS definition " + "FROM pg_trigger AS trigger " + "JOIN pg_class AS class ON class.oid = trigger.tgrelid " + "JOIN pg_namespace AS namespace ON namespace.oid = class.relnamespace " + "WHERE namespace.nspname = 'public' AND NOT trigger.tgisinternal " + "ORDER BY class.relname, trigger.tgname" + ) + schema_payload = { + "columns": _json_safe([dict(row) for row in schema_rows]), + "constraints": _json_safe([dict(row) for row in constraint_rows]), + "indexes": _json_safe([dict(row) for row in index_rows]), + "triggers": _json_safe([dict(row) for row in trigger_rows]), + } + + sequence_rows = await conn.fetch( + "SELECT schemaname, sequencename, start_value, min_value, max_value, " + "increment_by, cycle, cache_size, last_value " + "FROM pg_sequences WHERE schemaname = 'public' ORDER BY sequencename" + ) + lifecycle_state_counts: dict[str, int] = {} + tombstone_digest = hashlib.sha256() + if "session_lifecycle" in table_names: + state_rows = await conn.fetch( + "SELECT state, COUNT(*)::int AS count FROM public.session_lifecycle " + "GROUP BY state ORDER BY state" + ) + lifecycle_state_counts = { + str(row["state"]): int(row["count"]) for row in state_rows + } + tombstone_rows = await conn.fetch( + "SELECT md5(concat_ws('|', session_id, state, receipt_id, " + "started_at::text, completed_at::text, errors::text)) AS row_digest " + "FROM public.session_lifecycle WHERE state <> 'active' ORDER BY 1" + ) + for row in tombstone_rows: + tombstone_digest.update(str(row["row_digest"] or "").encode("ascii")) + + return DatabaseSnapshot( + table_counts=table_counts, + counts_sha256=_sha256_bytes(_canonical_json(table_counts)), + schema_sha256=_sha256_bytes(_canonical_json(schema_payload)), + logical_sha256=logical_digest.hexdigest(), + sequence_sha256=_sha256_bytes( + _canonical_json(_json_safe([dict(row) for row in sequence_rows])) + ), + lifecycle_tombstone_sha256=tombstone_digest.hexdigest(), + lifecycle_state_counts=lifecycle_state_counts, + ) + finally: + await conn.close() + + async def _validate_release_contracts(self, dsn: str) -> dict[str, object]: + conn = await self._connect(dsn) + try: + applied_rows = await conn.fetch( + "SELECT migration_id FROM public.postgres_schema_migrations ORDER BY migration_id" + ) + applied = {str(row["migration_id"]) for row in applied_rows} + expected_migrations = set(POSTGRES_TYPED_PROJECTION_MIGRATIONS) | { + "postgres_artifact_parity_v1", + "postgres_artifact_identity_v2", + "postgres_enterprise_oidc_identity_v1", + "postgres_session_lifecycle_fencing_v1", + } + if expected_migrations - applied: + raise PostgresRehearsalError("migration_ledger_incomplete") + + existing_tables = { + str(row["table_name"]) + for row in await conn.fetch( + "SELECT table_name FROM information_schema.tables " + "WHERE table_schema = 'public' AND table_type = 'BASE TABLE'" + ) + } + if set(POSTGRES_TYPED_PROJECTION_TABLES) - existing_tables: + raise PostgresRehearsalError("typed_projection_table_missing") + if existing_tables - _EXPECTED_PUBLIC_TABLES: + raise PostgresRehearsalError("unexpected_public_table_detected") + if _EXPECTED_PUBLIC_TABLES - existing_tables: + raise PostgresRehearsalError("application_table_missing") + for table_name, (id_column, artifact_predicate) in ( + POSTGRES_TYPED_PROJECTION_RECONCILIATION.items() + ): + quoted_table = _quote_ident(table_name) + quoted_id = _quote_ident(id_column) + drift_count = int( + await conn.fetchval( + "SELECT COUNT(*)::int FROM public.flagship_artifacts AS artifact " + f"WHERE ({artifact_predicate}) " # noqa: S608 -- static registry + "AND NOT EXISTS (SELECT 1 " + f"FROM public.{quoted_table} AS projection " # noqa: S608 + f"WHERE projection.{quoted_id} = artifact.artifact_id)" # noqa: S608 + ) + or 0 + ) + if drift_count: + raise PostgresRehearsalError("typed_projection_backfill_drift") + + trigger_tables = { + str(row["table_name"]) + for row in await conn.fetch( + "SELECT class.relname AS table_name FROM pg_trigger AS trigger " + "JOIN pg_class AS class ON class.oid = trigger.tgrelid " + "JOIN pg_namespace AS namespace ON namespace.oid = class.relnamespace " + "WHERE namespace.nspname = 'public' " + "AND trigger.tgname = 'trg_session_lifecycle_write' " + "AND NOT trigger.tgisinternal" + ) + } + expected_trigger_tables = set(POSTGRES_SESSION_LIFECYCLE_TABLES) + if expected_trigger_tables - trigger_tables: + raise PostgresRehearsalError("lifecycle_trigger_missing") + + orphan_lifecycle = int( + await conn.fetchval( + "SELECT COUNT(*)::int FROM public.sessions AS session " + "LEFT JOIN public.session_lifecycle AS lifecycle " + "ON lifecycle.session_id = session.session_id " + "WHERE lifecycle.session_id IS NULL" + ) + or 0 + ) + if orphan_lifecycle: + raise PostgresRehearsalError("lifecycle_orphan_detected") + for table_name in POSTGRES_SESSION_LIFECYCLE_TABLES: + if table_name in {"session_lifecycle"}: + continue + quoted_table = _quote_ident(table_name) + leaked = int( + await conn.fetchval( + f"SELECT COUNT(*)::int FROM public.{quoted_table} AS durable " # noqa: S608 + "JOIN public.session_lifecycle AS lifecycle " + "ON lifecycle.session_id = durable.session_id " + "WHERE lifecycle.state <> 'active'" + ) + or 0 + ) + if leaked: + raise PostgresRehearsalError("lifecycle_tombstone_data_leak") + + ownerless = int( + await conn.fetchval( + "SELECT COUNT(*)::int FROM public.sessions " + "WHERE owner_user_id IS NULL OR BTRIM(owner_user_id) = ''" + ) + or 0 + ) + if ownerless: + raise PostgresRehearsalError("ownership_validation_failed") + + safe_value = _SAFE_AUDIT_VALUE_PATTERN.replace("'", "''") + sensitive_key = _SENSITIVE_AUDIT_KEY_PATTERN.replace("'", "''") + for table_name, payload_column in _AUDIT_PAYLOAD_COLUMNS: + quoted_table = _quote_ident(table_name) + quoted_column = _quote_ident(payload_column) + unsafe_audits = int( + await conn.fetchval( + "SELECT COUNT(*)::int " + f"FROM public.{quoted_table} " # noqa: S608 -- static registry + "WHERE regexp_replace(" + f"{quoted_column}::text, " # noqa: S608 + f"'\"({sensitive_key})\"\\s*:\\s*\"{safe_value}\"', " + "'', 'gi') " + f"~* '\"({sensitive_key})\"\\s*:'" + ) + or 0 + ) + if unsafe_audits: + raise PostgresRehearsalError("audit_redaction_validation_failed") + + column_rows = await conn.fetch( + "SELECT table_name, column_name FROM information_schema.columns " + "WHERE table_schema = 'public'" + ) + columns: dict[str, set[str]] = {} + for row in column_rows: + columns.setdefault(str(row["table_name"]), set()).add(str(row["column_name"])) + for table_name, required_columns in _ROLLBACK_COMPATIBILITY_COLUMNS.items(): + if not required_columns.issubset(columns.get(table_name, set())): + raise PostgresRehearsalError("rollback_compatibility_contract_failed") + + return { + "auditPosture": "encrypted_or_redacted", + "lifecycleFencing": "verified", + "lifecycleTombstones": "verified", + "migrationLedger": "verified", + "ownership": "verified", + "typedProjectionBackfills": "verified", + } + finally: + await conn.close() + + async def _cleanup_disposable_target(self, dsn: str, expected_database: str) -> None: + conn = await self._connect(dsn) + try: + database_name = str(await conn.fetchval("SELECT current_database()") or "") + if database_name != expected_database: + raise PostgresRehearsalError("restore_database_marker_mismatch") + await conn.execute("DROP SCHEMA IF EXISTS public CASCADE") + await conn.execute("CREATE SCHEMA public") + remaining = int( + await conn.fetchval( + "SELECT COUNT(*)::int FROM pg_class AS class " + "JOIN pg_namespace AS namespace ON namespace.oid = class.relnamespace " + "WHERE namespace.nspname = 'public' " + "AND class.relkind IN ('r', 'p', 'S', 'v', 'm')" + ) + or 0 + ) + if remaining: + raise PostgresRehearsalError("target_cleanup_failed") + finally: + await conn.close() + + +class LibpqToolRunner: + """Run pg_dump/pg_restore with service files and no shell or DSN argv.""" + + def __init__(self) -> None: + self._pg_dump = self._resolve_tool("pg_dump") + self._pg_restore = self._resolve_tool("pg_restore") + + @staticmethod + def _resolve_tool(name: str) -> Path: + resolved = shutil.which(name) + if not resolved: + raise PostgresRehearsalError("postgres_tool_unavailable") + path = Path(resolved).resolve() + if not path.is_file() or not os.access(path, os.X_OK): + raise PostgresRehearsalError("postgres_tool_unavailable") + return path + + def versions(self, *, timeout_seconds: int) -> ToolVersions: + dump_version = self._tool_version(self._pg_dump, timeout_seconds) + restore_version = self._tool_version(self._pg_restore, timeout_seconds) + return ToolVersions( + pg_dump=dump_version, + pg_restore=restore_version, + pg_dump_sha256=_sha256_file(self._pg_dump), + pg_restore_sha256=_sha256_file(self._pg_restore), + ) + + def dump( + self, + dsn: str, + artifact_path: Path, + *, + timeout_seconds: int, + workspace: Path, + ) -> None: + service_file = self._write_service_file(workspace, "source", dsn) + self._run_tool( + [ + str(self._pg_dump), + "--dbname=service=source", + "--format=custom", + "--compress=6", + "--no-owner", + "--no-acl", + "--schema=public", + "--file", + str(artifact_path), + ], + service_file=service_file, + timeout_seconds=timeout_seconds, + failure_code="pg_dump_failed", + ) + if _sha256_file(artifact_path) == _sha256_bytes(b""): + raise PostgresRehearsalError("backup_artifact_invalid") + + def restore( + self, + dsn: str, + artifact_path: Path, + *, + expected_sha256: str, + timeout_seconds: int, + workspace: Path, + ) -> None: + actual_sha256 = _sha256_file(artifact_path) + if not hmac.compare_digest(actual_sha256, expected_sha256): + raise PostgresRehearsalError("backup_artifact_digest_mismatch") + service_file = self._write_service_file(workspace, "restore", dsn) + self._run_tool( + [ + str(self._pg_restore), + "--dbname=service=restore", + "--exit-on-error", + "--no-owner", + "--no-acl", + str(artifact_path), + ], + service_file=service_file, + timeout_seconds=timeout_seconds, + failure_code="pg_restore_failed", + ) + + @staticmethod + def _tool_version(tool: Path, timeout_seconds: int) -> str: + stdout = LibpqToolRunner._run_tool( + [str(tool), "--version"], + service_file=None, + timeout_seconds=timeout_seconds, + failure_code="postgres_tool_version_failed", + ) + match = _VERSION_RE.search(stdout.decode("utf-8", errors="replace")) + if match is None: + raise PostgresRehearsalError("postgres_tool_version_invalid") + return f"{match.group(1)}.{match.group(2)}" + + @staticmethod + def _write_service_file(workspace: Path, service_name: str, dsn: str) -> Path: + parsed = _parse_dsn(dsn, missing_code="postgres_dsn_required") + service_file = workspace / f".{service_name}.conf" + if service_file.exists() or service_file.is_symlink(): + raise PostgresRehearsalError("postgres_service_file_collision") + fields = { + "dbname": parsed.database, + "host": parsed.hostname, + "port": str(parsed.port), + "user": parsed.username, + } + if parsed.password: + fields["password"] = parsed.password + fields.update(dict(parsed.query)) + if any("\n" in value or "\r" in value for value in fields.values()): + raise PostgresRehearsalError("postgres_dsn_invalid") + content = [f"[{service_name}]"] + content.extend(f"{key}={_escape_service_value(value)}" for key, value in sorted(fields.items())) + try: + service_file.write_text("\n".join(content) + "\n", encoding="utf-8") + service_file.chmod(0o600) + except OSError as exc: + raise PostgresRehearsalError("postgres_service_file_failed") from exc + return service_file + + @staticmethod + def _run_tool( + args: list[str], + *, + service_file: Path | None, + timeout_seconds: int, + failure_code: str, + ) -> bytes: + environment = { + "LANG": "C", + "LC_ALL": "C", + "PATH": os.environ.get("PATH", ""), + "PGCONNECT_TIMEOUT": str(max(1, min(timeout_seconds, 30))), + } + if service_file is not None: + environment["PGSERVICEFILE"] = str(service_file) + process = subprocess.Popen( # noqa: S603 -- fixed executables and structured args + args, + stdin=subprocess.DEVNULL, + stdout=subprocess.PIPE, + stderr=subprocess.PIPE, + env=environment, + shell=False, + start_new_session=True, + ) + try: + stdout, _stderr = process.communicate(timeout=max(1, timeout_seconds)) + except subprocess.TimeoutExpired as exc: + _terminate_process_group(process) + raise PostgresRehearsalError("postgres_tool_timeout") from exc + except KeyboardInterrupt: + _terminate_process_group(process) + raise + if process.returncode != 0: + raise PostgresRehearsalError(failure_code) + return stdout[:4096] + + +def _escape_service_value(value: str) -> str: + return "'" + str(value).replace("\\", "\\\\").replace("'", "\\'") + "'" + + +def _terminate_process_group(process: subprocess.Popen[bytes]) -> None: + try: + os.killpg(process.pid, signal.SIGTERM) + process.wait(timeout=3) + except (OSError, subprocess.TimeoutExpired): + try: + os.killpg(process.pid, signal.SIGKILL) + except OSError: + pass + try: + process.wait(timeout=3) + except (OSError, subprocess.TimeoutExpired): + pass diff --git a/backend/operations/sqlcipher_rehearsal.py b/backend/operations/sqlcipher_rehearsal.py new file mode 100644 index 0000000..9db7aac --- /dev/null +++ b/backend/operations/sqlcipher_rehearsal.py @@ -0,0 +1,145 @@ +"""Provider-free disposable SQLCipher encryption and key-rotation rehearsal.""" + +from __future__ import annotations + +import re +import secrets +import tempfile +from collections.abc import Callable +from pathlib import Path +from typing import Any, Protocol + +from backend.storage.encrypted_store import EncryptedSessionStore, RotationError + + +REHEARSAL_SCHEMA_VERSION = "2026-08-disposable-sqlcipher-rehearsal-v1" +_RELEASE_SHA_RE = re.compile(r"[0-9a-f]{40}") + + +class _Store(Protocol): + def get_encryption_status(self) -> dict[str, Any]: ... + + +StoreFactory = Callable[[Path, str, str], _Store] + + +def _default_store_factory(database: Path, key: str, key_id: str) -> EncryptedSessionStore: + return EncryptedSessionStore( + db_path=str(database), + encrypt_key=key, + key_id=key_id, + encrypt_enabled=True, + ) + + +def _base_report(release_sha: str) -> dict[str, Any]: + return { + "schemaVersion": REHEARSAL_SCHEMA_VERSION, + "releaseSha": release_sha, + "liveSqlcipherTested": False, + "externalEvidence": False, + "releaseEvidenceEligible": False, + "modelProvidersContacted": False, + } + + +def run_disposable_sqlcipher_rehearsal( + *, + release_sha: str, + store_factory: StoreFactory = _default_store_factory, +) -> dict[str, Any]: + """Exercise a disposable database, or report honest driver unavailability. + + The workflow performs no network or model-provider calls. Its temporary + directory, raw keys, and driver errors are intentionally absent from the + returned machine-readable report. + """ + report = _base_report(release_sha) + if _RELEASE_SHA_RE.fullmatch(release_sha) is None: + return { + **report, + "status": "failed", + "reasonCode": "release_sha_invalid", + } + + old_key = secrets.token_hex(32) + new_key = secrets.token_hex(32) + try: + with tempfile.TemporaryDirectory(prefix="sqlcipher-rehearsal-") as directory: + database = Path(directory) / "disposable.db" + store = store_factory(database, old_key, "disposable-old-key") + initial_status = store.get_encryption_status() + if initial_status.get("driverAvailable") is not True: + return { + **report, + "status": "not_available", + "reasonCode": "sqlcipher_driver_unavailable", + } + if not isinstance(store, EncryptedSessionStore): + return { + **report, + "status": "failed", + "reasonCode": "sqlcipher_rehearsal_store_invalid", + } + + with store.connection() as connection: + connection.execute( + "CREATE TABLE rehearsal_marker (id INTEGER PRIMARY KEY, value TEXT NOT NULL)" + ) + connection.execute( + "INSERT INTO rehearsal_marker (id, value) VALUES (?, ?)", + (1, "verified"), + ) + + before = store.get_encryption_status() + if before.get("active") is not True: + return { + **report, + "status": "failed", + "reasonCode": "sqlcipher_pre_rotation_verification_failed", + } + rotation_result = store.rotate_key( + rotation_id="disposable-sqlcipher-rotation", + old_key_id="disposable-old-key", + new_key_id="disposable-new-key", + new_key=new_key, + release_sha=release_sha, + lock_timeout_seconds=10, + ) + reopened = store_factory(database, new_key, "disposable-new-key") + if not isinstance(reopened, EncryptedSessionStore): + raise RotationError("sqlcipher_rehearsal_store_invalid") + after = reopened.get_encryption_status() + with reopened.connection() as connection: + marker = connection.execute( + "SELECT value FROM rehearsal_marker WHERE id = 1" + ).fetchone() + if after.get("active") is not True or marker is None or marker[0] != "verified": + raise RotationError("sqlcipher_restore_verification_failed") + + receipt_envelope = rotation_result.get("receipt") + receipt_sha = ( + receipt_envelope.get("receiptSha256") + if isinstance(receipt_envelope, dict) + else None + ) + return { + **report, + "status": "passed", + "reasonCode": None, + "liveSqlcipherTested": True, + "releaseEvidenceEligible": True, + "rotationReceiptSha256": receipt_sha, + "primaryStoreProtected": True, + "rotationVerified": True, + "reopenVerified": True, + } + except Exception: + return { + **report, + "status": "failed", + "reasonCode": "sqlcipher_rehearsal_failed", + } + + +__all__ = ["REHEARSAL_SCHEMA_VERSION", "run_disposable_sqlcipher_rehearsal"] diff --git a/backend/operations/sqlite_rehearsal.py b/backend/operations/sqlite_rehearsal.py new file mode 100644 index 0000000..b04169d --- /dev/null +++ b/backend/operations/sqlite_rehearsal.py @@ -0,0 +1,198 @@ +"""Disposable SQLite backup/restore integrity rehearsal. + +This module proves only the local SQLite mechanics exercised against a generated +fixture. It deliberately does not inspect or make claims about live PostgreSQL, +production data, external signing, or disaster-recovery evidence. +""" + +from __future__ import annotations + +import hashlib +import hmac +import json +import sqlite3 +import tempfile +from datetime import UTC, datetime +from pathlib import Path +from typing import Any + + +class BackupIntegrityError(RuntimeError): + """Raised when a backup artifact, manifest, or restored database is invalid.""" + + +def _canonical_json(payload: Any) -> bytes: + return json.dumps(payload, sort_keys=True, separators=(",", ":"), ensure_ascii=True).encode("utf-8") + + +def _sha256_bytes(payload: bytes) -> str: + return hashlib.sha256(payload).hexdigest() + + +def _sha256_file(path: Path) -> str: + digest = hashlib.sha256() + with path.open("rb") as handle: + while chunk := handle.read(1024 * 1024): + digest.update(chunk) + return digest.hexdigest() + + +def _sqlite_integrity(path: Path) -> str: + try: + with sqlite3.connect(f"file:{path}?mode=ro", uri=True) as connection: + row = connection.execute("PRAGMA integrity_check").fetchone() + except sqlite3.Error as exc: + raise BackupIntegrityError("SQLite integrity check could not run") from exc + if not row or str(row[0]).lower() != "ok": + raise BackupIntegrityError("SQLite integrity check failed") + return "ok" + + +def _logical_digest(path: Path) -> str: + try: + with sqlite3.connect(f"file:{path}?mode=ro", uri=True) as connection: + dump = "\n".join(connection.iterdump()).encode("utf-8") + except sqlite3.Error as exc: + raise BackupIntegrityError("SQLite logical digest could not be created") from exc + return _sha256_bytes(dump) + + +def create_disposable_sqlite_fixture(path: Path) -> None: + if path.exists(): + raise BackupIntegrityError("Disposable fixture target already exists") + path.parent.mkdir(parents=True, exist_ok=True) + with sqlite3.connect(path) as connection: + connection.executescript( + """ + PRAGMA journal_mode=DELETE; + PRAGMA user_version=1; + CREATE TABLE sessions ( + session_id TEXT PRIMARY KEY, + owner_user_id TEXT NOT NULL, + created_at REAL NOT NULL + ); + CREATE TABLE session_lifecycle ( + session_id TEXT PRIMARY KEY, + state TEXT NOT NULL, + generation INTEGER NOT NULL, + FOREIGN KEY (session_id) REFERENCES sessions(session_id) + ); + INSERT INTO sessions(session_id, owner_user_id, created_at) + VALUES ('disposable-session', 'disposable-owner', 1.0); + INSERT INTO session_lifecycle(session_id, state, generation) + VALUES ('disposable-session', 'active', 1); + """ + ) + connection.commit() + _sqlite_integrity(path) + + +def backup_sqlite_database(source: Path, backup: Path, manifest_path: Path) -> dict[str, Any]: + if not source.is_file() or source.is_symlink(): + raise BackupIntegrityError("SQLite backup source is not a regular file") + if backup.exists() or manifest_path.exists(): + raise BackupIntegrityError("SQLite backup output already exists") + _sqlite_integrity(source) + with sqlite3.connect(f"file:{source}?mode=ro", uri=True) as source_connection: + with sqlite3.connect(backup) as backup_connection: + source_connection.backup(backup_connection) + _sqlite_integrity(backup) + + manifest = { + "schema_version": "sqlite-backup-manifest-v1", + "engine": "sqlite", + "scope": "disposable_fixture_only", + "created_at": datetime.now(UTC).isoformat(), + "artifact_size": backup.stat().st_size, + "artifact_sha256": _sha256_file(backup), + "logical_sha256": _logical_digest(source), + } + envelope = { + "manifest": manifest, + "manifest_sha256": _sha256_bytes(_canonical_json(manifest)), + } + manifest_path.write_bytes(_canonical_json(envelope) + b"\n") + return envelope + + +def verify_backup_manifest(backup: Path, manifest_path: Path) -> dict[str, Any]: + if not backup.is_file() or backup.is_symlink() or not manifest_path.is_file() or manifest_path.is_symlink(): + raise BackupIntegrityError("SQLite backup or manifest is not a regular file") + try: + envelope = json.loads(manifest_path.read_text(encoding="utf-8")) + except (OSError, UnicodeError, json.JSONDecodeError) as exc: + raise BackupIntegrityError("SQLite backup manifest is unreadable") from exc + if not isinstance(envelope, dict) or set(envelope) != {"manifest", "manifest_sha256"}: + raise BackupIntegrityError("SQLite backup manifest envelope is invalid") + manifest = envelope.get("manifest") + if not isinstance(manifest, dict) or set(manifest) != { + "schema_version", + "engine", + "scope", + "created_at", + "artifact_size", + "artifact_sha256", + "logical_sha256", + }: + raise BackupIntegrityError("SQLite backup manifest fields are invalid") + expected_manifest_digest = _sha256_bytes(_canonical_json(manifest)) + if not hmac.compare_digest(str(envelope.get("manifest_sha256") or ""), expected_manifest_digest): + raise BackupIntegrityError("SQLite backup manifest digest mismatch") + if ( + manifest.get("schema_version") != "sqlite-backup-manifest-v1" + or manifest.get("engine") != "sqlite" + or manifest.get("scope") != "disposable_fixture_only" + ): + raise BackupIntegrityError("SQLite backup manifest identity is invalid") + if manifest.get("artifact_size") != backup.stat().st_size: + raise BackupIntegrityError("SQLite backup artifact size mismatch") + if not hmac.compare_digest(str(manifest.get("artifact_sha256") or ""), _sha256_file(backup)): + raise BackupIntegrityError("SQLite backup artifact digest mismatch") + _sqlite_integrity(backup) + return envelope + + +def restore_and_verify_sqlite_backup(backup: Path, manifest_path: Path, restored: Path) -> dict[str, str]: + envelope = verify_backup_manifest(backup, manifest_path) + if restored.exists(): + raise BackupIntegrityError("SQLite restore target already exists") + with sqlite3.connect(f"file:{backup}?mode=ro", uri=True) as backup_connection: + with sqlite3.connect(restored) as restored_connection: + backup_connection.backup(restored_connection) + integrity = _sqlite_integrity(restored) + logical_digest = _logical_digest(restored) + expected_logical_digest = str(envelope["manifest"].get("logical_sha256") or "") + if not hmac.compare_digest(logical_digest, expected_logical_digest): + raise BackupIntegrityError("SQLite restored logical digest mismatch") + return {"sqlite_integrity": integrity, "logical_restore": "verified"} + + +def run_sqlite_backup_restore_rehearsal() -> dict[str, Any]: + with tempfile.TemporaryDirectory(prefix="excel-agent-sqlite-rehearsal-") as temp_dir: + workspace = Path(temp_dir) + source = workspace / "source.db" + backup = workspace / "backup.db" + restored = workspace / "restored.db" + manifest_path = workspace / "backup.manifest.json" + create_disposable_sqlite_fixture(source) + envelope = backup_sqlite_database(source, backup, manifest_path) + verify_backup_manifest(backup, manifest_path) + restore_checks = restore_and_verify_sqlite_backup(backup, manifest_path, restored) + return { + "schema_version": "sqlite-backup-rehearsal-v1", + "status": "passed", + "engine": "sqlite", + "scope": "disposable_fixture_only", + "live_postgres_tested": False, + "external_evidence": False, + "checks": { + "manifest_digest": "verified", + "backup_digest": "verified", + **restore_checks, + }, + "artifact": { + "bytes": envelope["manifest"]["artifact_size"], + "sha256": envelope["manifest"]["artifact_sha256"], + "manifest_sha256": envelope["manifest_sha256"], + }, + } diff --git a/backend/operations/storage_encryption_attestation.py b/backend/operations/storage_encryption_attestation.py new file mode 100644 index 0000000..45f6f0c --- /dev/null +++ b/backend/operations/storage_encryption_attestation.py @@ -0,0 +1,643 @@ +"""Detached Ed25519 validation for external storage-encryption evidence.""" + +from __future__ import annotations + +import argparse +import hashlib +import json +import os +import re +import stat +import sys +import time +from pathlib import Path +from typing import Any + +from cryptography.exceptions import InvalidSignature +from cryptography.hazmat.primitives import serialization +from cryptography.hazmat.primitives.asymmetric.ed25519 import Ed25519PublicKey + + +ATTESTATION_SCHEMA_VERSION = "2026-08-storage-encryption-attestation-v2" +REPORT_SCHEMA_VERSION = "2026-08-storage-encryption-attestation-report-v2" +_RELEASE_SHA_RE = re.compile(r"^[0-9a-f]{40}$") +_SHA256_RE = re.compile(r"^[0-9a-f]{64}$") +_KEY_ID_RE = re.compile(r"^[A-Za-z0-9][A-Za-z0-9._-]{2,127}$") +_ALGORITHMS = frozenset({"AES-256", "AES-256-GCM", "AES-256-XTS"}) +_PROVIDERS = frozenset( + { + "aws-kms", + "azure-key-vault", + "gcp-cloud-kms", + "hashicorp-vault", + "on-prem-hsm", + } +) +_PLATFORMS = frozenset( + { + "aws-rds-postgresql", + "azure-database-postgresql", + "gcp-cloud-sql-postgresql", + "on-prem-postgresql", + "self-managed-postgresql", + } +) +_MAX_RECEIPT_BYTES = 64 * 1024 +_MAX_PUBLIC_KEY_BYTES = 16 * 1024 +_SIGNATURE_BYTES = 64 +_DEFAULT_MAXIMUM_AGE_SECONDS = 7 * 24 * 60 * 60 +_MAXIMUM_AGE_LIMIT_SECONDS = 30 * 24 * 60 * 60 +_CLOCK_SKEW_SECONDS = 300 +_RUNTIME_ENV = { + "receipt": "STORAGE_ENCRYPTION_ATTESTATION_PATH", + "signature": "STORAGE_ENCRYPTION_ATTESTATION_SIGNATURE_PATH", + "public_key": "STORAGE_ENCRYPTION_ATTESTATION_PUBLIC_KEY_PATH", + "key_id": "STORAGE_ENCRYPTION_ATTESTATION_KEY_ID", + "release_sha": "STORAGE_ENCRYPTION_ATTESTATION_RELEASE_SHA", + "deployment_sha256": "STORAGE_ENCRYPTION_ATTESTATION_DEPLOYMENT_SHA256", + "runner_sha256": "STORAGE_ENCRYPTION_ATTESTATION_RUNNER_SHA256", + "key_identity_sha256": "STORAGE_ENCRYPTION_ATTESTATION_KEY_IDENTITY_SHA256", + "public_key_sha256": "STORAGE_ENCRYPTION_ATTESTATION_PUBLIC_KEY_SHA256", +} + + +class StorageEncryptionAttestationError(ValueError): + """A bounded external-attestation validation failure.""" + + +def _canonical_json(payload: object) -> bytes: + return json.dumps( + payload, + sort_keys=True, + separators=(",", ":"), + ensure_ascii=True, + ).encode("utf-8") + + +def _sha256(payload: bytes) -> str: + return hashlib.sha256(payload).hexdigest() + + +def _reject_duplicates(pairs: list[tuple[str, Any]]) -> dict[str, Any]: + result: dict[str, Any] = {} + for key, value in pairs: + if key in result: + raise StorageEncryptionAttestationError("Attestation contains duplicate fields.") + result[key] = value + return result + + +def _read_regular(path: Path, maximum: int, label: str) -> bytes: + candidate = Path(path) + file_descriptor = -1 + try: + if candidate.is_symlink(): + raise StorageEncryptionAttestationError(f"{label} must be a regular non-symlink file.") + resolved = candidate.resolve(strict=True) + flags = os.O_RDONLY + if hasattr(os, "O_NOFOLLOW"): + flags |= os.O_NOFOLLOW + file_descriptor = os.open(resolved, flags) + file_stat = os.fstat(file_descriptor) + if not stat.S_ISREG(file_stat.st_mode): + raise StorageEncryptionAttestationError(f"{label} must be a regular non-symlink file.") + if file_stat.st_nlink != 1: + raise StorageEncryptionAttestationError(f"{label} must not be hard-linked.") + if os.name != "nt" and ( + file_stat.st_mode & 0o022 or file_stat.st_uid not in {os.geteuid(), 0} + ): + raise StorageEncryptionAttestationError( + f"{label} ownership or permissions are unsafe." + ) + path_stat = os.lstat(resolved) + if ( + path_stat.st_dev != file_stat.st_dev + or path_stat.st_ino != file_stat.st_ino + ): + raise StorageEncryptionAttestationError(f"{label} changed while being opened.") + if file_stat.st_size < 1 or file_stat.st_size > maximum: + raise StorageEncryptionAttestationError(f"{label} has an invalid bounded size.") + with os.fdopen(file_descriptor, "rb", closefd=False) as handle: + payload = handle.read(maximum + 1) + if len(payload) != file_stat.st_size or len(payload) > maximum: + raise StorageEncryptionAttestationError(f"{label} changed while being read.") + return payload + except Exception: + raise + except OSError as exc: + raise StorageEncryptionAttestationError(f"{label} could not be read.") from exc + finally: + if file_descriptor >= 0: + os.close(file_descriptor) + + +def _trusted_public_key(path: Path, repository_root: Path) -> tuple[Ed25519PublicKey, str]: + candidate = Path(path) + try: + resolved = candidate.resolve(strict=True) + repository = Path(repository_root).resolve(strict=True) + except OSError as exc: + raise StorageEncryptionAttestationError("Trusted public key location is invalid.") from exc + if resolved == repository or resolved.is_relative_to(repository): + raise StorageEncryptionAttestationError( + "Trusted public key must remain outside the repository." + ) + raw_key = _read_regular(resolved, _MAX_PUBLIC_KEY_BYTES, "Trusted public key") + try: + loaded = serialization.load_pem_public_key(raw_key) + except (TypeError, ValueError) as exc: + raise StorageEncryptionAttestationError("Trusted public key is invalid.") from exc + if not isinstance(loaded, Ed25519PublicKey): + raise StorageEncryptionAttestationError("Trusted public key must be Ed25519.") + key_bytes = loaded.public_bytes( + encoding=serialization.Encoding.Raw, + format=serialization.PublicFormat.Raw, + ) + return loaded, _sha256(key_bytes) + + +def _exact_object( + value: object, + *, + fields: frozenset[str], + label: str, +) -> dict[str, Any]: + if not isinstance(value, dict) or set(value) != fields: + raise StorageEncryptionAttestationError(f"{label} fields are invalid.") + return value + + +def _exact_sha(value: object, *, label: str) -> str: + normalized = value if isinstance(value, str) else "" + if not _SHA256_RE.fullmatch(normalized): + raise StorageEncryptionAttestationError(f"{label} must be an exact SHA-256 digest.") + return normalized + + +def _release_sha(value: object) -> str: + normalized = value if isinstance(value, str) else "" + if not _RELEASE_SHA_RE.fullmatch(normalized): + raise StorageEncryptionAttestationError( + "Attestation release SHA must be exact lowercase 40-character hexadecimal." + ) + return normalized + + +def _key_id(value: object) -> str: + normalized = value if isinstance(value, str) else "" + if not _KEY_ID_RE.fullmatch(normalized): + raise StorageEncryptionAttestationError("Attestation signing key id is invalid.") + return normalized + + +def _timestamp(value: object, *, label: str) -> int: + if not isinstance(value, int) or isinstance(value, bool) or value < 1: + raise StorageEncryptionAttestationError(f"{label} must be a positive Unix timestamp.") + return value + + +def _maximum_age(value: int) -> int: + if ( + not isinstance(value, int) + or isinstance(value, bool) + or value < _CLOCK_SKEW_SECONDS + or value > _MAXIMUM_AGE_LIMIT_SECONDS + ): + raise StorageEncryptionAttestationError( + "Attestation maximum age is outside the allowed range." + ) + return value + + +def validate_storage_encryption_attestation( + *, + receipt_path: Path, + signature_path: Path, + trusted_public_key_path: Path, + trusted_signing_key_id: str, + expected_release_sha: str, + expected_deployment_sha256: str | None = None, + expected_runner_sha256: str | None = None, + expected_key_identity_sha256: str | None = None, + expected_public_key_sha256: str | None = None, + validation_time_unix_seconds: int | None = None, + maximum_age_seconds: int = _DEFAULT_MAXIMUM_AGE_SECONDS, + repository_root: Path | None = None, +) -> dict[str, Any]: + """Authenticate and validate one strict external PostgreSQL attestation.""" + release_sha = _release_sha(expected_release_sha) + signing_key_id = _key_id(trusted_signing_key_id) + maximum_age = _maximum_age(maximum_age_seconds) + validation_time = ( + int(time.time()) + if validation_time_unix_seconds is None + else _timestamp(validation_time_unix_seconds, label="Validation time") + ) + repository = repository_root or Path(__file__).resolve().parents[2] + raw_receipt = _read_regular( + receipt_path, + _MAX_RECEIPT_BYTES, + "Storage encryption attestation", + ) + raw_signature = _read_regular( + signature_path, + _SIGNATURE_BYTES, + "Storage encryption attestation signature", + ) + if len(raw_signature) != _SIGNATURE_BYTES: + raise StorageEncryptionAttestationError( + "Storage encryption attestation signature must be exactly 64 bytes." + ) + public_key, public_key_sha = _trusted_public_key(trusted_public_key_path, repository) + if expected_public_key_sha256 is not None and public_key_sha != _exact_sha( + expected_public_key_sha256, + label="Trusted public key digest", + ): + raise StorageEncryptionAttestationError( + "Trusted public key digest does not match the configured trust root." + ) + try: + public_key.verify(raw_signature, raw_receipt) + except InvalidSignature as exc: + raise StorageEncryptionAttestationError( + "Storage encryption attestation signature verification failed." + ) from exc + try: + payload = json.loads(raw_receipt, object_pairs_hook=_reject_duplicates) + except (UnicodeError, json.JSONDecodeError) as exc: + raise StorageEncryptionAttestationError( + "Storage encryption attestation JSON is invalid." + ) from exc + if not isinstance(payload, dict) or raw_receipt != _canonical_json(payload) + b"\n": + raise StorageEncryptionAttestationError( + "Storage encryption attestation must use canonical JSON." + ) + top = _exact_object( + payload, + fields=frozenset( + { + "backupRestore", + "deployment", + "encryption", + "expiresAtUnixSeconds", + "issuedAtUnixSeconds", + "privacy", + "releaseSha", + "rotation", + "schemaVersion", + "signingKeyId", + } + ), + label="Attestation", + ) + if top["schemaVersion"] != ATTESTATION_SCHEMA_VERSION: + raise StorageEncryptionAttestationError("Attestation schema version is invalid.") + if _release_sha(top["releaseSha"]) != release_sha: + raise StorageEncryptionAttestationError("Attestation release SHA does not match.") + if _key_id(top["signingKeyId"]) != signing_key_id: + raise StorageEncryptionAttestationError("Attestation signing key id does not match.") + issued_at = _timestamp(top["issuedAtUnixSeconds"], label="Attestation issue time") + expires_at = _timestamp(top["expiresAtUnixSeconds"], label="Attestation expiry time") + if expires_at <= validation_time: + raise StorageEncryptionAttestationError("Storage encryption attestation has expired.") + if issued_at > validation_time + _CLOCK_SKEW_SECONDS: + raise StorageEncryptionAttestationError( + "Storage encryption attestation issue time is in the future." + ) + if validation_time - issued_at > maximum_age: + raise StorageEncryptionAttestationError("Storage encryption attestation is stale.") + if expires_at <= issued_at or expires_at - issued_at > maximum_age: + raise StorageEncryptionAttestationError( + "Storage encryption attestation validity window is invalid." + ) + + deployment = _exact_object( + top["deployment"], + fields=frozenset( + {"databaseEngine", "deploymentSha256", "platform", "runnerSha256"} + ), + label="Deployment", + ) + deployment_sha = _exact_sha( + deployment["deploymentSha256"], + label="Deployment digest", + ) + runner_sha = _exact_sha(deployment["runnerSha256"], label="Runner digest") + if expected_deployment_sha256 is not None and deployment_sha != _exact_sha( + expected_deployment_sha256, + label="Expected deployment digest", + ): + raise StorageEncryptionAttestationError( + "Attestation deployment digest does not match runtime deployment." + ) + if expected_runner_sha256 is not None and runner_sha != _exact_sha( + expected_runner_sha256, + label="Expected runner digest", + ): + raise StorageEncryptionAttestationError( + "Attestation runner digest does not match runtime runner." + ) + if deployment["databaseEngine"] != "postgresql": + raise StorageEncryptionAttestationError( + "External storage attestation database engine must be PostgreSQL." + ) + platform = str(deployment["platform"] or "") + if platform not in _PLATFORMS: + raise StorageEncryptionAttestationError("Attestation platform is not allowlisted.") + + encryption = _exact_object( + top["encryption"], + fields=frozenset( + {"algorithm", "atRestVerified", "hardwareBacked", "keyIdentitySha256", "provider"} + ), + label="Encryption", + ) + if encryption["atRestVerified"] is not True: + raise StorageEncryptionAttestationError( + "Primary database at-rest encryption is not verified." + ) + if encryption["hardwareBacked"] is not True: + raise StorageEncryptionAttestationError( + "KMS or HSM hardware-backed key protection is not verified." + ) + algorithm = str(encryption["algorithm"] or "") + provider = str(encryption["provider"] or "") + if algorithm not in _ALGORITHMS: + raise StorageEncryptionAttestationError("Encryption algorithm is not allowlisted.") + if provider not in _PROVIDERS: + raise StorageEncryptionAttestationError("Encryption provider is not allowlisted.") + key_identity_sha = _exact_sha( + encryption["keyIdentitySha256"], + label="KMS or HSM key identity digest", + ) + if expected_key_identity_sha256 is not None and key_identity_sha != _exact_sha( + expected_key_identity_sha256, + label="Expected KMS or HSM key identity digest", + ): + raise StorageEncryptionAttestationError( + "Attestation active key identity does not match runtime configuration." + ) + + rotation = _exact_object( + top["rotation"], + fields=frozenset( + { + "newKeyIdentitySha256", + "oldKeyIdentitySha256", + "completedAtUnixSeconds", + "currentKeyVerified", + "previousKeyDisabled", + "rehearsalVerified", + "rotationReceiptSha256", + } + ), + label="Rotation rehearsal", + ) + if rotation["rehearsalVerified"] is not True: + raise StorageEncryptionAttestationError("Rotation rehearsal is not verified.") + if rotation["currentKeyVerified"] is not True: + raise StorageEncryptionAttestationError("Rotation active key is not verified.") + if rotation["previousKeyDisabled"] is not True: + raise StorageEncryptionAttestationError("Rotation previous key is not disabled.") + rotation_completed_at = _timestamp( + rotation["completedAtUnixSeconds"], + label="Rotation completion time", + ) + if rotation_completed_at > issued_at or issued_at - rotation_completed_at > maximum_age: + raise StorageEncryptionAttestationError("Rotation rehearsal evidence is stale.") + old_key_sha = _exact_sha( + rotation["oldKeyIdentitySha256"], + label="Old key identity digest", + ) + new_key_sha = _exact_sha( + rotation["newKeyIdentitySha256"], + label="New key identity digest", + ) + rotation_receipt_sha = _exact_sha( + rotation["rotationReceiptSha256"], + label="Rotation receipt digest", + ) + if old_key_sha == new_key_sha: + raise StorageEncryptionAttestationError( + "Rotation rehearsal key identities must be distinct." + ) + if key_identity_sha != new_key_sha: + raise StorageEncryptionAttestationError( + "Attestation active key must match the verified new rotation key." + ) + + backup = _exact_object( + top["backupRestore"], + fields=frozenset( + { + "backupArtifactSha256", + "completedAtUnixSeconds", + "encryptedBackupVerified", + "restoreVerified", + "restoredAtRestVerified", + "restoredLogicalSha256", + } + ), + label="Encrypted backup and restore", + ) + if ( + backup["encryptedBackupVerified"] is not True + or backup["restoreVerified"] is not True + or backup["restoredAtRestVerified"] is not True + ): + raise StorageEncryptionAttestationError( + "Encrypted backup and restore proof is incomplete." + ) + backup_completed_at = _timestamp( + backup["completedAtUnixSeconds"], + label="Backup and restore completion time", + ) + if backup_completed_at > issued_at or issued_at - backup_completed_at > maximum_age: + raise StorageEncryptionAttestationError( + "Encrypted backup and restore evidence is stale." + ) + backup_artifact_sha = _exact_sha( + backup["backupArtifactSha256"], + label="Backup artifact digest", + ) + restored_logical_sha = _exact_sha( + backup["restoredLogicalSha256"], + label="Restored logical digest", + ) + + privacy = _exact_object( + top["privacy"], + fields=frozenset( + {"containsCustomerData", "containsDsn", "containsFilesystemPaths"} + ), + label="Attestation privacy", + ) + if any(value is not False for value in privacy.values()): + raise StorageEncryptionAttestationError( + "Attestation privacy declaration must exclude DSNs, paths, and customer data." + ) + + receipt_sha = _sha256(raw_receipt) + candidate_details = { + "algorithm": algorithm, + "authenticatedExternalAttestation": True, + "backupArtifactSha256": backup_artifact_sha, + "backupRestoreVerified": True, + "deploymentSha256": deployment_sha, + "keyIdentitySha256": key_identity_sha, + "primaryStoreProtected": True, + "previousKeyDisabled": True, + "provider": provider, + "currentKeyVerified": True, + "rotationRehearsalVerified": True, + "rotationReceiptSha256": rotation_receipt_sha, + "runnerSha256": runner_sha, + "restoredLogicalSha256": restored_logical_sha, + "successful": True, + } + return { + "algorithm": algorithm, + "authenticatedExternalAttestation": True, + "databaseEngine": "postgresql", + "deploymentSha256": deployment_sha, + "keyIdentitySha256": key_identity_sha, + "expiresAtUnixSeconds": expires_at, + "issuedAtUnixSeconds": issued_at, + "modelProvidersContacted": False, + "platform": platform, + "primaryStoreProtected": True, + "provider": provider, + "publicKeySha256": public_key_sha, + "receiptSha256": receipt_sha, + "releaseEvidenceCandidate": { + "details": candidate_details, + "id": "storage_encryption_attestation", + "releaseSha": release_sha, + "status": "passed", + }, + "releaseSha": release_sha, + "runnerSha256": runner_sha, + "schemaVersion": REPORT_SCHEMA_VERSION, + "signatureVerified": True, + "signingKeyIdSha256": _sha256(signing_key_id.encode()), + "status": "passed", + } + + +def runtime_external_storage_encryption_status() -> dict[str, Any]: + """Validate the optional out-of-band runtime attestation without trust-key defaults.""" + values = {name: str(os.getenv(env_name) or "").strip() for name, env_name in _RUNTIME_ENV.items()} + configured = [bool(value) for value in values.values()] + if not any(configured): + return { + "authenticatedExternalAttestation": False, + "primaryStoreProtected": False, + "reasonCode": "external_attestation_not_configured", + "status": "not_configured", + } + if not all(configured): + return { + "authenticatedExternalAttestation": False, + "primaryStoreProtected": False, + "reasonCode": "external_attestation_configuration_incomplete", + "status": "blocked", + } + maximum_age_raw = str( + os.getenv("STORAGE_ENCRYPTION_ATTESTATION_MAX_AGE_SECONDS") + or _DEFAULT_MAXIMUM_AGE_SECONDS + ).strip() + try: + maximum_age = _maximum_age(int(maximum_age_raw)) + except (TypeError, ValueError, StorageEncryptionAttestationError): + return { + "authenticatedExternalAttestation": False, + "primaryStoreProtected": False, + "reasonCode": "external_attestation_maximum_age_invalid", + "status": "blocked", + } + try: + report = validate_storage_encryption_attestation( + receipt_path=Path(values["receipt"]), + signature_path=Path(values["signature"]), + trusted_public_key_path=Path(values["public_key"]), + trusted_signing_key_id=values["key_id"], + expected_release_sha=values["release_sha"], + expected_deployment_sha256=values["deployment_sha256"], + expected_runner_sha256=values["runner_sha256"], + expected_key_identity_sha256=values["key_identity_sha256"], + expected_public_key_sha256=values["public_key_sha256"], + maximum_age_seconds=maximum_age, + repository_root=Path(__file__).resolve().parents[2], + ) + except StorageEncryptionAttestationError: + return { + "authenticatedExternalAttestation": False, + "primaryStoreProtected": False, + "reasonCode": "external_attestation_invalid", + "status": "blocked", + } + return { + "algorithm": report["algorithm"], + "authenticatedExternalAttestation": True, + "keyIdentitySha256": report["keyIdentitySha256"], + "expiresAtUnixSeconds": report["expiresAtUnixSeconds"], + "primaryStoreProtected": True, + "provider": report["provider"], + "reasonCode": None, + "receiptSha256": report["receiptSha256"], + "releaseSha": report["releaseSha"], + "status": "authenticated", + } + + +def main(argv: list[str] | None = None) -> int: + parser = argparse.ArgumentParser( + description="Validate a detached external PostgreSQL storage-encryption attestation." + ) + parser.add_argument("--receipt", type=Path, required=True) + parser.add_argument("--signature", type=Path, required=True) + parser.add_argument("--public-key", type=Path, required=True) + parser.add_argument("--key-id", required=True) + parser.add_argument("--release-sha", required=True) + parser.add_argument("--deployment-sha256", required=True) + parser.add_argument("--runner-sha256", required=True) + parser.add_argument("--key-identity-sha256", required=True) + parser.add_argument("--public-key-sha256", required=True) + parser.add_argument( + "--maximum-age-seconds", + type=int, + default=_DEFAULT_MAXIMUM_AGE_SECONDS, + ) + args = parser.parse_args(argv) + try: + report = validate_storage_encryption_attestation( + receipt_path=args.receipt, + signature_path=args.signature, + trusted_public_key_path=args.public_key, + trusted_signing_key_id=args.key_id, + expected_release_sha=args.release_sha, + expected_deployment_sha256=args.deployment_sha256, + expected_runner_sha256=args.runner_sha256, + expected_key_identity_sha256=args.key_identity_sha256, + expected_public_key_sha256=args.public_key_sha256, + maximum_age_seconds=args.maximum_age_seconds, + ) + exit_code = 0 + except Exception: + report = { + "authenticatedExternalAttestation": False, + "errorCode": "storage_encryption_attestation_invalid", + "primaryStoreProtected": False, + "releaseSha": args.release_sha + if _RELEASE_SHA_RE.fullmatch(str(args.release_sha or "")) + else "", + "schemaVersion": REPORT_SCHEMA_VERSION, + "status": "blocked", + } + exit_code = 1 + sys.stdout.write(json.dumps(report, sort_keys=True, separators=(",", ":")) + "\n") + return exit_code + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/backend/operations/v05_import_consumer.py b/backend/operations/v05_import_consumer.py new file mode 100644 index 0000000..5931204 --- /dev/null +++ b/backend/operations/v05_import_consumer.py @@ -0,0 +1,574 @@ +"""Consume an authenticated v0.5 stage into neutral lifecycle-owned storage. + +The boundary is intentionally one-way and narrow: exact staged bytes are +persisted for a future reviewed importer, while current workbook, document, +workflow, and legacy stores remain untouched. +""" + +from __future__ import annotations + +import argparse +from contextlib import redirect_stderr, redirect_stdout +from dataclasses import dataclass +import io +import json +import os +from pathlib import Path, PurePosixPath +import stat +import sys +from typing import Any, Sequence + +from backend.storage.protocol import SessionStoreProtocol +from backend.storage.v05_import_store import ( + NeutralV05ImportError, + STAGE_SCHEMA_VERSION, + binding_sha256, + canonical_json, + neutral_import_id, + neutral_session_id, + normalize_owner_id, + normalize_sha256, + sha256_bytes, + sha256_text, +) +from vnext.migration.v05_import import MigrationImportError, _validate_records + + +_MANIFEST_KEYS = frozenset( + { + "content_sha256", + "counts", + "excluded_categories", + "migration_mode", + "owner_scope_sha256", + "payloads", + "schema_version", + "source_product", + "source_release", + } +) +_DESCRIPTOR_KEYS = frozenset({"kind", "path", "sha256", "size_bytes"}) +_COUNT_KEYS = frozenset({"documents", "sessions", "workflows"}) +_PAYLOAD_KINDS = frozenset({"blob", "document", "session", "workflow_recipe"}) +_REQUIRED_EXCLUSIONS = frozenset( + { + "api_credentials", + "database_schema", + "enterprise_configuration", + "legacy_api_contracts", + "legacy_model_artifacts_v2_v7", + "llm_caches", + "rag_indexes", + "replay_data", + "worker_jobs_and_cli_state", + } +) + + +@dataclass(frozen=True) +class ConsumptionLimits: + max_files: int = 8194 + max_manifest_bytes: int = 4 * 1024 * 1024 + max_receipt_bytes: int = 64 * 1024 + max_entry_bytes: int = 64 * 1024 * 1024 + max_total_payload_bytes: int = 512 * 1024 * 1024 + + +def consume_validated_v05_stage( + *, + store: SessionStoreProtocol, + staging_path: Path, + expected_archive_sha256: str, + expected_content_sha256: str, + target_owner_id: str, + limits: ConsumptionLimits | None = None, +) -> dict[str, Any]: + """Persist an immutable, already-validated stage without domain conversion.""" + + request, raw_payloads = _prepare_validated_v05_stage( + staging_path=staging_path, + expected_archive_sha256=expected_archive_sha256, + expected_content_sha256=expected_content_sha256, + target_owner_id=target_owner_id, + limits=limits, + ) + return _store_prepared_v05_import(store, request, raw_payloads) + + +def _prepare_validated_v05_stage( + *, + staging_path: Path, + expected_archive_sha256: str, + expected_content_sha256: str, + target_owner_id: str, + limits: ConsumptionLimits | None = None, +) -> tuple[dict[str, Any], list[dict[str, Any]]]: + + normalized_archive = normalize_sha256( + expected_archive_sha256, + code="invalid_expected_archive_digest", + ) + normalized_content = normalize_sha256( + expected_content_sha256, + code="invalid_expected_content_digest", + ) + normalized_owner = normalize_owner_id(target_owner_id) + manifest, stage_receipt, raw_payloads, manifest_bytes, receipt_bytes = _read_stage( + Path(staging_path), + expected_archive_sha256=normalized_archive, + expected_content_sha256=normalized_content, + target_owner_id=normalized_owner, + limits=limits or ConsumptionLimits(), + ) + del stage_receipt + + request: dict[str, Any] = { + "archiveSha256": normalized_archive, + "contentSha256": normalized_content, + "counts": dict(manifest["counts"]), + "importId": neutral_import_id(normalized_content), + "manifestSha256": sha256_bytes(manifest_bytes), + "ownerId": normalized_owner, + "ownerScopeSha256": sha256_text(normalized_owner), + "schemaVersion": STAGE_SCHEMA_VERSION, + "stageReceiptSha256": sha256_bytes(receipt_bytes), + "targetSessionId": neutral_session_id(normalized_content), + } + request["bindingSha256"] = binding_sha256(request) + return request, raw_payloads + + +def _store_prepared_v05_import( + store: SessionStoreProtocol, + request: dict[str, Any], + raw_payloads: list[dict[str, Any]], +) -> dict[str, Any]: + try: + return store.consume_v05_import(request, raw_payloads) + except NeutralV05ImportError: + raise + except Exception as exc: + raise NeutralV05ImportError("storage_write_failed") from exc + + +def _read_stage( + staging_path: Path, + *, + expected_archive_sha256: str, + expected_content_sha256: str, + target_owner_id: str, + limits: ConsumptionLimits, +) -> tuple[dict[str, Any], dict[str, Any], list[dict[str, Any]], bytes, bytes]: + _validate_limits(limits) + root_fd = _open_secure_directory(staging_path, code="stage_unavailable") + try: + root_stat = os.fstat(root_fd) + if staging_path.name != expected_content_sha256: + raise NeutralV05ImportError("stage_content_mismatch") + manifest_bytes = _read_secure_file( + root_fd, + "manifest.json", + maximum_bytes=limits.max_manifest_bytes, + ) + receipt_bytes = _read_secure_file( + root_fd, + "receipt.json", + maximum_bytes=limits.max_receipt_bytes, + ) + manifest = _load_canonical_object(manifest_bytes, code="invalid_stage_manifest") + stage_receipt = _load_canonical_object(receipt_bytes, code="invalid_stage_receipt") + descriptors = _validate_manifest( + manifest, + expected_content_sha256=expected_content_sha256, + target_owner_id=target_owner_id, + ) + _validate_stage_receipt( + stage_receipt, + expected_archive_sha256=expected_archive_sha256, + expected_content_sha256=expected_content_sha256, + target_owner_id=target_owner_id, + ) + + expected_files = {"manifest.json", "receipt.json", *(item["path"] for item in descriptors)} + actual_files = _enumerate_secure_files(root_fd, maximum_files=limits.max_files) + if actual_files != expected_files: + raise NeutralV05ImportError("invalid_stage_file_ledger") + + payloads: list[dict[str, Any]] = [] + record_payloads: dict[str, bytes] = {} + total_bytes = 0 + for descriptor in descriptors: + content = _read_secure_file( + root_fd, + descriptor["path"], + maximum_bytes=min(limits.max_entry_bytes, descriptor["size_bytes"]), + ) + total_bytes += len(content) + if total_bytes > limits.max_total_payload_bytes: + raise NeutralV05ImportError("stage_payload_limit_exceeded") + if ( + len(content) != descriptor["size_bytes"] + or sha256_bytes(content) != descriptor["sha256"] + ): + raise NeutralV05ImportError("stage_payload_digest_mismatch") + record_payloads[descriptor["path"]] = content + payloads.append( + { + "content": content, + "kind": descriptor["kind"], + "path": descriptor["path"], + "sha256": descriptor["sha256"], + "sizeBytes": descriptor["size_bytes"], + } + ) + try: + _validate_records(manifest, record_payloads) + except MigrationImportError as exc: + raise NeutralV05ImportError("invalid_stage_records") from exc + + current_root_stat = os.fstat(root_fd) + try: + path_stat = os.stat(staging_path, follow_symlinks=False) + except OSError as exc: + raise NeutralV05ImportError("stage_changed_during_read") from exc + if _file_identity(current_root_stat) != _file_identity(root_stat) or _file_identity( + path_stat + ) != _file_identity(root_stat): + raise NeutralV05ImportError("stage_changed_during_read") + return manifest, stage_receipt, payloads, manifest_bytes, receipt_bytes + finally: + os.close(root_fd) + + +def _validate_limits(limits: ConsumptionLimits) -> None: + values = ( + limits.max_files, + limits.max_manifest_bytes, + limits.max_receipt_bytes, + limits.max_entry_bytes, + limits.max_total_payload_bytes, + ) + if any(not isinstance(value, int) or isinstance(value, bool) or value <= 0 for value in values): + raise NeutralV05ImportError("invalid_consumer_limits") + + +def _validate_manifest( + manifest: dict[str, Any], + *, + expected_content_sha256: str, + target_owner_id: str, +) -> list[dict[str, Any]]: + if frozenset(manifest) != _MANIFEST_KEYS: + raise NeutralV05ImportError("invalid_stage_manifest") + if ( + manifest.get("schema_version") != STAGE_SCHEMA_VERSION + or manifest.get("source_product") != "Excel Agent" + or manifest.get("source_release") != "0.5.0" + or manifest.get("migration_mode") != "one_way_clean_break" + or manifest.get("content_sha256") != expected_content_sha256 + or manifest.get("owner_scope_sha256") != sha256_text(target_owner_id) + ): + raise NeutralV05ImportError("invalid_stage_manifest") + exclusions = manifest.get("excluded_categories") + if ( + not isinstance(exclusions, list) + or not all(isinstance(item, str) for item in exclusions) + or exclusions != sorted(set(exclusions)) + or not _REQUIRED_EXCLUSIONS.issubset(exclusions) + ): + raise NeutralV05ImportError("invalid_stage_manifest") + counts = manifest.get("counts") + if not isinstance(counts, dict) or frozenset(counts) != _COUNT_KEYS: + raise NeutralV05ImportError("invalid_stage_manifest") + if any( + not isinstance(value, int) or isinstance(value, bool) or value < 0 + for value in counts.values() + ): + raise NeutralV05ImportError("invalid_stage_manifest") + + raw_descriptors = manifest.get("payloads") + if not isinstance(raw_descriptors, list) or not raw_descriptors: + raise NeutralV05ImportError("invalid_stage_manifest") + descriptors: list[dict[str, Any]] = [] + seen_paths: set[str] = set() + for raw in raw_descriptors: + if not isinstance(raw, dict) or frozenset(raw) != _DESCRIPTOR_KEYS: + raise NeutralV05ImportError("invalid_stage_manifest") + path = _canonical_relative_path(raw.get("path")) + kind = raw.get("kind") + size = raw.get("size_bytes") + if ( + path in seen_paths + or path in {"manifest.json", "receipt.json"} + or kind not in _PAYLOAD_KINDS + or not isinstance(size, int) + or isinstance(size, bool) + or size < 0 + ): + raise NeutralV05ImportError("invalid_stage_manifest") + seen_paths.add(path) + descriptors.append( + { + "kind": str(kind), + "path": path, + "sha256": normalize_sha256(raw.get("sha256"), code="invalid_stage_manifest"), + "size_bytes": size, + } + ) + if descriptors != sorted(descriptors, key=lambda item: item["path"]): + raise NeutralV05ImportError("invalid_stage_manifest") + if sha256_bytes(canonical_json(descriptors, newline=True)) != expected_content_sha256: + raise NeutralV05ImportError("invalid_stage_manifest") + actual_counts = { + "documents": sum(item["kind"] == "document" for item in descriptors), + "sessions": sum(item["kind"] == "session" for item in descriptors), + "workflows": sum(item["kind"] == "workflow_recipe" for item in descriptors), + } + if actual_counts != counts: + raise NeutralV05ImportError("invalid_stage_manifest") + return descriptors + + +def _validate_stage_receipt( + receipt: dict[str, Any], + *, + expected_archive_sha256: str, + expected_content_sha256: str, + target_owner_id: str, +) -> None: + expected = { + "archive_sha256": expected_archive_sha256, + "compatibility_layer_installed": False, + "content_sha256": expected_content_sha256, + "owner_scope_sha256": sha256_text(target_owner_id), + "runtime_storage_mutated": False, + "schema_version": STAGE_SCHEMA_VERSION, + "status": "validated_pending_import", + } + if receipt != expected: + raise NeutralV05ImportError("invalid_stage_receipt") + + +def _load_canonical_object(content: bytes, *, code: str) -> dict[str, Any]: + try: + value = json.loads(content, object_pairs_hook=_reject_duplicate_keys) + except (UnicodeDecodeError, ValueError) as exc: + raise NeutralV05ImportError(code) from exc + if not isinstance(value, dict) or canonical_json(value, newline=True) != content: + raise NeutralV05ImportError(code) + return value + + +def _reject_duplicate_keys(pairs: list[tuple[str, Any]]) -> dict[str, Any]: + value: dict[str, Any] = {} + for key, item in pairs: + if key in value: + raise ValueError("duplicate key") + value[key] = item + return value + + +def _canonical_relative_path(raw: Any) -> str: + if not isinstance(raw, str) or not raw or "\\" in raw or "\x00" in raw or raw.startswith("/"): + raise NeutralV05ImportError("invalid_stage_path") + path = PurePosixPath(raw) + if raw != path.as_posix() or any(part in {"", ".", ".."} for part in path.parts): + raise NeutralV05ImportError("invalid_stage_path") + if not path.parts or ":" in path.parts[0]: + raise NeutralV05ImportError("invalid_stage_path") + return raw + + +def _secure_metadata(metadata: os.stat_result, *, directory: bool) -> bool: + expected_type = stat.S_ISDIR if directory else stat.S_ISREG + if not expected_type(metadata.st_mode): + return False + if not directory and metadata.st_nlink != 1: + return False + if os.name != "nt" and ( + metadata.st_uid not in {os.geteuid(), 0} or metadata.st_mode & 0o022 + ): + return False + return True + + +def _file_identity(metadata: os.stat_result) -> tuple[int, int]: + return metadata.st_dev, metadata.st_ino + + +def _directory_flags() -> int: + return ( + os.O_RDONLY + | getattr(os, "O_CLOEXEC", 0) + | getattr(os, "O_DIRECTORY", 0) + | getattr(os, "O_NOFOLLOW", 0) + ) + + +def _open_secure_directory(path: Path, *, code: str = "unsafe_stage") -> int: + try: + before = os.stat(path, follow_symlinks=False) + except OSError as exc: + raise NeutralV05ImportError(code) from exc + if not _secure_metadata(before, directory=True): + raise NeutralV05ImportError("unsafe_stage") + try: + descriptor = os.open(path, _directory_flags()) + except OSError as exc: + raise NeutralV05ImportError("unsafe_stage") from exc + try: + opened = os.fstat(descriptor) + if not _secure_metadata(opened, directory=True) or _file_identity(opened) != _file_identity( + before + ): + raise NeutralV05ImportError("unsafe_stage") + return descriptor + except Exception: + os.close(descriptor) + raise + + +def _open_child_directory(parent_fd: int, name: str) -> int: + try: + before = os.stat(name, dir_fd=parent_fd, follow_symlinks=False) + if not _secure_metadata(before, directory=True): + raise NeutralV05ImportError("unsafe_stage") + descriptor = os.open(name, _directory_flags(), dir_fd=parent_fd) + except NeutralV05ImportError: + raise + except OSError as exc: + raise NeutralV05ImportError("unsafe_stage") from exc + try: + opened = os.fstat(descriptor) + if not _secure_metadata(opened, directory=True) or _file_identity(opened) != _file_identity( + before + ): + raise NeutralV05ImportError("unsafe_stage") + return descriptor + except Exception: + os.close(descriptor) + raise + + +def _read_secure_file(root_fd: int, relative: str, *, maximum_bytes: int) -> bytes: + path = PurePosixPath(_canonical_relative_path(relative)) + directory_fd = os.dup(root_fd) + try: + for component in path.parts[:-1]: + next_fd = _open_child_directory(directory_fd, component) + os.close(directory_fd) + directory_fd = next_fd + name = path.parts[-1] + try: + before = os.stat(name, dir_fd=directory_fd, follow_symlinks=False) + flags = os.O_RDONLY | getattr(os, "O_CLOEXEC", 0) | getattr(os, "O_NOFOLLOW", 0) + file_fd = os.open(name, flags, dir_fd=directory_fd) + except OSError as exc: + raise NeutralV05ImportError("unsafe_stage") from exc + try: + opened = os.fstat(file_fd) + if ( + not _secure_metadata(before, directory=False) + or not _secure_metadata(opened, directory=False) + or _file_identity(opened) != _file_identity(before) + or opened.st_size < 0 + or opened.st_size > maximum_bytes + ): + raise NeutralV05ImportError("unsafe_stage") + chunks: list[bytes] = [] + total = 0 + while chunk := os.read(file_fd, min(1024 * 1024, maximum_bytes + 1 - total)): + chunks.append(chunk) + total += len(chunk) + if total > maximum_bytes: + raise NeutralV05ImportError("stage_payload_limit_exceeded") + after = os.fstat(file_fd) + current = os.stat(name, dir_fd=directory_fd, follow_symlinks=False) + if ( + _file_identity(after) != _file_identity(opened) + or _file_identity(current) != _file_identity(opened) + or after.st_size != total + ): + raise NeutralV05ImportError("stage_changed_during_read") + return b"".join(chunks) + finally: + os.close(file_fd) + finally: + os.close(directory_fd) + + +def _enumerate_secure_files(root_fd: int, *, maximum_files: int) -> set[str]: + files: set[str] = set() + + def visit(directory_fd: int, prefix: tuple[str, ...]) -> None: + try: + names = sorted(os.listdir(directory_fd)) + except OSError as exc: + raise NeutralV05ImportError("unsafe_stage") from exc + for name in names: + if not isinstance(name, str) or name in {"", ".", ".."} or "/" in name or "\x00" in name: + raise NeutralV05ImportError("unsafe_stage") + try: + metadata = os.stat(name, dir_fd=directory_fd, follow_symlinks=False) + except OSError as exc: + raise NeutralV05ImportError("unsafe_stage") from exc + relative = "/".join((*prefix, name)) + if stat.S_ISDIR(metadata.st_mode): + child_fd = _open_child_directory(directory_fd, name) + try: + visit(child_fd, (*prefix, name)) + finally: + os.close(child_fd) + elif _secure_metadata(metadata, directory=False): + files.add(relative) + if len(files) > maximum_files: + raise NeutralV05ImportError("stage_payload_limit_exceeded") + else: + raise NeutralV05ImportError("unsafe_stage") + + visit(root_fd, ()) + return files + + +def _parser() -> argparse.ArgumentParser: + parser = argparse.ArgumentParser(description="Consume a validated v0.5 neutral migration stage") + parser.add_argument("--staging-path", type=Path, required=True) + parser.add_argument("--expected-archive-sha256", required=True) + parser.add_argument("--expected-content-sha256", required=True) + parser.add_argument("--target-owner-id", required=True) + return parser + + +def main(argv: Sequence[str] | None = None) -> int: + args = _parser().parse_args(argv) + try: + request, raw_payloads = _prepare_validated_v05_stage( + staging_path=args.staging_path, + expected_archive_sha256=args.expected_archive_sha256, + expected_content_sha256=args.expected_content_sha256, + target_owner_id=args.target_owner_id, + ) + from backend.storage import get_session_store + + # The CLI contract is one JSON document. Application initialization + # may emit ordinary structured logs, which must not corrupt or disclose + # into this machine-readable operator channel. + captured_logs = io.StringIO() + with redirect_stdout(captured_logs), redirect_stderr(captured_logs): + result = _store_prepared_v05_import( + get_session_store(), + request, + raw_payloads, + ) + except NeutralV05ImportError as exc: + print(json.dumps({"errorCode": exc.code, "status": "blocked"}, sort_keys=True)) + return 2 + except Exception: + print(json.dumps({"errorCode": "neutral_import_failed", "status": "blocked"}, sort_keys=True)) + return 2 + print(canonical_json(result).decode("utf-8")) + return 0 + + +if __name__ == "__main__": # pragma: no cover + sys.exit(main()) diff --git a/backend/operations/v05_managed_materializer.py b/backend/operations/v05_managed_materializer.py new file mode 100644 index 0000000..30c1c3a --- /dev/null +++ b/backend/operations/v05_managed_materializer.py @@ -0,0 +1,546 @@ +"""One-way neutral v0.5 import materialization into encrypted managed objects. + +The operator deliberately creates no runtime workbook, document, workflow, or +legacy state. It copies exact, lifecycle-owned neutral bytes into the +explicitly configured managed-object substrate and publishes one deterministic +content-addressed manifest only after every object has been reread exactly. +""" + +from __future__ import annotations + +import argparse +import asyncio +from contextvars import ContextVar +import hmac +import json +import os +import re +from collections.abc import Mapping, Sequence +from typing import Any, Protocol + +from backend.storage.managed_object_store import ( + EncryptedManagedObjectStore, + ManagedObjectKind, + ManagedObjectScope, + ManagedObjectStoreConfig, + ManagedObjectStoreError, + ManagedObjectStoreLimitError, +) +from backend.storage.v05_import_store import ( + MATERIALIZATION_MAX_ENTRY_BYTES, + MATERIALIZATION_MAX_PAYLOAD_BYTES, + MATERIALIZATION_MAX_PAYLOADS, + NeutralV05ImportError, + canonical_json, + neutral_import_id, + neutral_session_id, + normalize_import_bundle, + normalize_owner_id, + normalize_sha256, + sha256_bytes, + sha256_text, + validate_import_receipt, +) + + +MATERIALIZATION_MANIFEST_SCHEMA_VERSION = ( + "com.excelagent.migration.v05.managed-materialization.v1" +) +MATERIALIZATION_RECEIPT_SCHEMA_VERSION = ( + "com.excelagent.migration.v05.managed-materialization.receipt.v1" +) + +_ERROR_CODE = re.compile(r"[a-z][a-z0-9_]{0,63}") +_SOURCE_BUNDLE_KEYS = frozenset({"payloads", "receipt", "request"}) +_KIND_MAP = { + "blob": ManagedObjectKind.DOCUMENT, + "document": ManagedObjectKind.EVIDENCE, + "session": ManagedObjectKind.EVIDENCE, + "workflow_recipe": ManagedObjectKind.EVIDENCE, +} +_ACTIVE_MATERIALIZATION: ContextVar[tuple[int, str] | None] = ContextVar( + "v05_managed_materialization_active", + default=None, +) + + +class V05ManagedMaterializationError(RuntimeError): + """Categorical materialization failure safe for logs and operator JSON.""" + + def __init__(self, code: str) -> None: + normalized = str(code or "").strip().lower() + if _ERROR_CODE.fullmatch(normalized) is None: + normalized = "v05_materialization_failed" + self.code = normalized + super().__init__(normalized) + + +class V05ImportBundleSource(Protocol): + """Narrow materializer dependency implemented by both session stores.""" + + def read_v05_import_bundle( + self, + import_id: str, + *, + expected_binding_sha256: str, + expected_content_sha256: str, + expected_owner_id: str, + expected_owner_scope_sha256: str, + expected_target_session_id: str, + ) -> dict | None: ... + + +def materialize_v05_import( + *, + source_store: V05ImportBundleSource, + managed_store: EncryptedManagedObjectStore, + import_id: str, + expected_binding_sha256: str, + tenant_id: str, + owner_id: str, + lifecycle_coordinator: Any, +) -> dict[str, Any]: + """Materialize exact neutral bytes and publish a deterministic manifest last.""" + + normalized_import_id, content_sha256, binding, owner, target_session_id = ( + _normalize_materialization_request( + import_id=import_id, + expected_binding_sha256=expected_binding_sha256, + owner_id=owner_id, + ) + ) + lifecycle_marker = (id(lifecycle_coordinator), target_session_id) + if _ACTIVE_MATERIALIZATION.get() != lifecycle_marker: + return _run_with_lifecycle_fence( + lifecycle_coordinator=lifecycle_coordinator, + target_session_id=target_session_id, + callback=lambda: materialize_v05_import( + source_store=source_store, + managed_store=managed_store, + import_id=normalized_import_id, + expected_binding_sha256=binding, + tenant_id=tenant_id, + owner_id=owner, + lifecycle_coordinator=lifecycle_coordinator, + ), + managed_store=managed_store, + ) + try: + scope = ManagedObjectScope(tenant_id, owner, target_session_id) + except (TypeError, ValueError) as exc: + raise V05ManagedMaterializationError("invalid_materialization_request") from exc + + expected_owner_scope = sha256_text(owner) + source = _read_validated_source( + source_store, + import_id=normalized_import_id, + expected_binding_sha256=binding, + expected_content_sha256=content_sha256, + expected_owner_id=owner, + expected_owner_scope_sha256=expected_owner_scope, + expected_target_session_id=target_session_id, + ) + + entries: list[dict[str, Any]] = [] + try: + for payload in source["payloads"]: + managed_kind = _KIND_MAP[payload["kind"]] + receipt = managed_store.put_bytes(scope, managed_kind, payload["content"]) + if ( + receipt.object_kind != managed_kind.value + or receipt.plaintext_size != payload["sizeBytes"] + ): + raise V05ManagedMaterializationError("managed_object_receipt_mismatch") + _verify_managed_reread( + managed_store, + scope, + receipt.object_id, + expected=payload["content"], + ) + entries.append( + { + "managedKind": managed_kind.value, + "objectId": receipt.object_id, + "path": payload["path"], + "sha256": payload["sha256"], + "sizeBytes": payload["sizeBytes"], + "sourceKind": payload["kind"], + } + ) + except V05ManagedMaterializationError: + raise + except ManagedObjectStoreLimitError as exc: + raise V05ManagedMaterializationError("managed_object_budget_exhausted") from exc + except (ManagedObjectStoreError, OSError, TypeError, ValueError) as exc: + raise V05ManagedMaterializationError("managed_object_write_failed") from exc + + manifest = { + "bindingSha256": source["request"]["bindingSha256"], + "contentSha256": source["request"]["contentSha256"], + "importId": source["request"]["importId"], + "objectCount": len(entries), + "objects": entries, + "ownerScopeSha256": source["request"]["ownerScopeSha256"], + "schemaVersion": MATERIALIZATION_MANIFEST_SCHEMA_VERSION, + "sourceReceiptSha256": source["receipt"]["receiptSha256"], + "targetSessionSha256": sha256_text(target_session_id), + "tenantScopeSha256": sha256_text(scope.tenant_id), + "uniquePayloadObjects": len({entry["objectId"] for entry in entries}), + } + manifest_bytes = canonical_json(manifest, newline=True) + + # The initial read is not durable authorization for publication. Re-read + # the complete source under its lifecycle/owner snapshot immediately before + # the manifest write, while the managed write itself independently checks + # the same active lifecycle row and owner in its metadata transaction. + refreshed = _read_validated_source( + source_store, + import_id=normalized_import_id, + expected_binding_sha256=binding, + expected_content_sha256=content_sha256, + expected_owner_id=owner, + expected_owner_scope_sha256=expected_owner_scope, + expected_target_session_id=target_session_id, + ) + if refreshed != source: + raise V05ManagedMaterializationError("neutral_import_changed") + + try: + manifest_receipt = managed_store.put_bytes( + scope, + ManagedObjectKind.EVIDENCE, + manifest_bytes, + ) + except ManagedObjectStoreLimitError as exc: + raise V05ManagedMaterializationError("managed_object_budget_exhausted") from exc + except (ManagedObjectStoreError, OSError, TypeError, ValueError) as exc: + raise V05ManagedMaterializationError("managed_object_write_failed") from exc + if ( + manifest_receipt.object_kind != ManagedObjectKind.EVIDENCE.value + or manifest_receipt.plaintext_size != len(manifest_bytes) + ): + raise V05ManagedMaterializationError("managed_object_receipt_mismatch") + _verify_managed_reread( + managed_store, + scope, + manifest_receipt.object_id, + expected=manifest_bytes, + ) + + # Close the publication window: a purge or source replacement that wins + # immediately after publication prevents a success receipt. + final_source = _read_validated_source( + source_store, + import_id=normalized_import_id, + expected_binding_sha256=binding, + expected_content_sha256=content_sha256, + expected_owner_id=owner, + expected_owner_scope_sha256=expected_owner_scope, + expected_target_session_id=target_session_id, + ) + if final_source != source: + raise V05ManagedMaterializationError("neutral_import_changed") + _verify_managed_reread( + managed_store, + scope, + manifest_receipt.object_id, + expected=manifest_bytes, + ) + + report: dict[str, Any] = { + "binding_sha256": binding, + "cloud_provider_contacted": False, + "content_sha256": content_sha256, + "manifest_object_id": manifest_receipt.object_id, + "manifest_sha256": sha256_bytes(manifest_bytes), + "materialization_max_entry_bytes": MATERIALIZATION_MAX_ENTRY_BYTES, + "materialization_max_payload_bytes": MATERIALIZATION_MAX_PAYLOAD_BYTES, + "materialization_max_payloads": MATERIALIZATION_MAX_PAYLOADS, + "object_count": len(entries), + "owner_scope_sha256": expected_owner_scope, + "production_encryption_claim": False, + "provider": managed_store.provider_name, + "reread_verified": True, + "schema_version": MATERIALIZATION_RECEIPT_SCHEMA_VERSION, + "source_receipt_sha256": source["receipt"]["receiptSha256"], + "status": "materialized", + "target_session_sha256": sha256_text(target_session_id), + "tenant_scope_sha256": sha256_text(scope.tenant_id), + "unique_payload_objects": len({entry["objectId"] for entry in entries}), + } + report["receipt_sha256"] = sha256_bytes(canonical_json(report)) + return report + + +def _run_with_lifecycle_fence( + *, + lifecycle_coordinator: Any, + target_session_id: str, + callback: Any, + managed_store: EncryptedManagedObjectStore, +) -> dict[str, Any]: + active_task = getattr(lifecycle_coordinator, "active_task", None) + if not callable(active_task): + raise V05ManagedMaterializationError("lifecycle_coordinator_invalid") + try: + # Register before entering the active-task boundary so a purge can + # never inventory this task without also owning its managed bytes. + managed_store.register_lifecycle_purger(lifecycle_coordinator) + except (TypeError, ValueError) as exc: + raise V05ManagedMaterializationError("lifecycle_purger_unavailable") from exc + + async def run() -> dict[str, Any]: + try: + async with active_task(target_session_id): + token = _ACTIVE_MATERIALIZATION.set( + (id(lifecycle_coordinator), target_session_id) + ) + try: + # Deliberately execute each bounded synchronous storage + # mutation on this task. Cancellation cannot detach a + # background writer that outlives the lifecycle fence. + return callback() + finally: + _ACTIVE_MATERIALIZATION.reset(token) + except V05ManagedMaterializationError: + raise + except asyncio.CancelledError: + raise V05ManagedMaterializationError("session_fenced") from None + except RuntimeError as exc: + raise V05ManagedMaterializationError("session_fenced") from exc + + try: + asyncio.get_running_loop() + except RuntimeError: + return asyncio.run(run()) + raise V05ManagedMaterializationError("async_materialization_required") + + +async def materialize_v05_import_async( + *, + source_store: V05ImportBundleSource, + managed_store: EncryptedManagedObjectStore, + import_id: str, + expected_binding_sha256: str, + tenant_id: str, + owner_id: str, + lifecycle_coordinator: Any, +) -> dict[str, Any]: + """Async entrypoint that keeps publication inside one active-task fence.""" + + normalized_import_id, _content, binding, owner, target_session_id = ( + _normalize_materialization_request( + import_id=import_id, + expected_binding_sha256=expected_binding_sha256, + owner_id=owner_id, + ) + ) + active_task = getattr(lifecycle_coordinator, "active_task", None) + if not callable(active_task): + raise V05ManagedMaterializationError("lifecycle_coordinator_invalid") + try: + managed_store.register_lifecycle_purger(lifecycle_coordinator) + async with active_task(target_session_id): + token = _ACTIVE_MATERIALIZATION.set( + (id(lifecycle_coordinator), target_session_id) + ) + try: + return materialize_v05_import( + source_store=source_store, + managed_store=managed_store, + import_id=normalized_import_id, + expected_binding_sha256=binding, + tenant_id=tenant_id, + owner_id=owner, + lifecycle_coordinator=lifecycle_coordinator, + ) + finally: + _ACTIVE_MATERIALIZATION.reset(token) + except V05ManagedMaterializationError: + raise + except asyncio.CancelledError: + raise V05ManagedMaterializationError("session_fenced") from None + except (RuntimeError, TypeError, ValueError) as exc: + raise V05ManagedMaterializationError("session_fenced") from exc + + +def _normalize_materialization_request( + *, + import_id: Any, + expected_binding_sha256: Any, + owner_id: Any, +) -> tuple[str, str, str, str, str]: + try: + normalized_import_id = str(import_id or "").strip() + prefix = "v05-import-" + if not normalized_import_id.startswith(prefix): + raise NeutralV05ImportError("invalid_import_contract") + content_sha256 = normalize_sha256(normalized_import_id.removeprefix(prefix)) + if neutral_import_id(content_sha256) != normalized_import_id: + raise NeutralV05ImportError("invalid_import_contract") + binding = normalize_sha256(expected_binding_sha256) + owner = normalize_owner_id(owner_id) + target_session_id = neutral_session_id(content_sha256) + except NeutralV05ImportError as exc: + raise V05ManagedMaterializationError("invalid_materialization_request") from exc + return normalized_import_id, content_sha256, binding, owner, target_session_id + + +def _read_validated_source( + source_store: V05ImportBundleSource, + *, + import_id: str, + expected_binding_sha256: str, + expected_content_sha256: str, + expected_owner_id: str, + expected_owner_scope_sha256: str, + expected_target_session_id: str, +) -> dict[str, Any]: + try: + raw = source_store.read_v05_import_bundle( + import_id, + expected_binding_sha256=expected_binding_sha256, + expected_content_sha256=expected_content_sha256, + expected_owner_id=expected_owner_id, + expected_owner_scope_sha256=expected_owner_scope_sha256, + expected_target_session_id=expected_target_session_id, + ) + except NeutralV05ImportError as exc: + code = ( + "neutral_import_budget_exhausted" + if exc.code == "materialization_budget_exceeded" + else "neutral_import_invalid" + ) + raise V05ManagedMaterializationError(code) from exc + except Exception as exc: + raise V05ManagedMaterializationError("neutral_import_read_failed") from exc + if raw is None: + raise V05ManagedMaterializationError("neutral_import_unavailable") + if not isinstance(raw, dict) or frozenset(raw) != _SOURCE_BUNDLE_KEYS: + raise V05ManagedMaterializationError("neutral_import_invalid") + request = raw.get("request") + payloads = raw.get("payloads") + receipt = raw.get("receipt") + if not isinstance(request, Mapping) or not isinstance(payloads, (tuple, list)): + raise V05ManagedMaterializationError("neutral_import_invalid") + try: + normalized_request, normalized_payloads = normalize_import_bundle( + request, + list(payloads), + ) + normalized_receipt = validate_import_receipt( + receipt, + normalized_request, + stored_payloads=len(normalized_payloads), + ) + except NeutralV05ImportError as exc: + raise V05ManagedMaterializationError("neutral_import_invalid") from exc + if ( + normalized_request["importId"] != import_id + or not hmac.compare_digest( + normalized_request["bindingSha256"], + expected_binding_sha256, + ) + or not hmac.compare_digest( + normalized_request["contentSha256"], + expected_content_sha256, + ) + or normalized_request["ownerId"] != expected_owner_id + or not hmac.compare_digest( + normalized_request["ownerScopeSha256"], + expected_owner_scope_sha256, + ) + or normalized_request["targetSessionId"] != expected_target_session_id + ): + raise V05ManagedMaterializationError("neutral_import_invalid") + return { + "payloads": normalized_payloads, + "receipt": normalized_receipt, + "request": normalized_request, + } + + +def _verify_managed_reread( + managed_store: EncryptedManagedObjectStore, + scope: ManagedObjectScope, + object_id: str, + *, + expected: bytes, +) -> None: + try: + actual = managed_store.get_bytes(scope, object_id) + except (ManagedObjectStoreError, OSError, TypeError, ValueError) as exc: + raise V05ManagedMaterializationError("managed_object_reread_failed") from exc + if len(actual) != len(expected) or not hmac.compare_digest(actual, expected): + raise V05ManagedMaterializationError("managed_object_reread_failed") + + +def _build_configured_managed_store( + environ: Mapping[str, str] | None = None, +) -> EncryptedManagedObjectStore: + from backend.operations.managed_object_store_operator import _configured_store + + env = os.environ if environ is None else environ + config = ManagedObjectStoreConfig.from_environment(env) + return _configured_store(config, env) + + +def main(argv: Sequence[str] | None = None) -> int: + parser = argparse.ArgumentParser( + description="Materialize an authenticated neutral v0.5 import into managed objects" + ) + parser.add_argument("--import-id", required=True) + parser.add_argument("--expected-binding-sha256", required=True) + parser.add_argument("--tenant-id", required=True) + parser.add_argument("--owner-id", required=True) + args = parser.parse_args(list(argv) if argv is not None else None) + try: + from backend.storage import get_session_store + from backend.services.lifecycle import get_lifecycle_coordinator + + report = materialize_v05_import( + source_store=get_session_store(), + managed_store=_build_configured_managed_store(), + import_id=args.import_id, + expected_binding_sha256=args.expected_binding_sha256, + tenant_id=args.tenant_id, + owner_id=args.owner_id, + lifecycle_coordinator=get_lifecycle_coordinator(), + ) + except V05ManagedMaterializationError as exc: + report = { + "schema_version": MATERIALIZATION_RECEIPT_SCHEMA_VERSION, + "status": "blocked", + "error_code": exc.code, + "production_encryption_claim": False, + "cloud_provider_contacted": False, + } + print(json.dumps(report, sort_keys=True, separators=(",", ":"))) + return 2 + except Exception: + report = { + "schema_version": MATERIALIZATION_RECEIPT_SCHEMA_VERSION, + "status": "failed", + "error_code": "v05_materialization_failed", + "production_encryption_claim": False, + "cloud_provider_contacted": False, + } + print(json.dumps(report, sort_keys=True, separators=(",", ":"))) + return 1 + print(json.dumps(report, sort_keys=True, separators=(",", ":"))) + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) + + +__all__ = [ + "MATERIALIZATION_MANIFEST_SCHEMA_VERSION", + "MATERIALIZATION_RECEIPT_SCHEMA_VERSION", + "V05ManagedMaterializationError", + "V05ImportBundleSource", + "main", + "materialize_v05_import", + "materialize_v05_import_async", +] diff --git a/backend/operations/work_admission.py b/backend/operations/work_admission.py new file mode 100644 index 0000000..c1c1593 --- /dev/null +++ b/backend/operations/work_admission.py @@ -0,0 +1,683 @@ +"""Bounded, fair admission control for isolated document workers. + +The controller is deliberately synchronous because workbook and PDF work runs +in dedicated threads before crossing a subprocess boundary. It provides one +process-wide budget, per-worker-pool budgets, round-robin session fairness, and +cooperative cancellation signals that the subprocess loops can poll. +""" + +from __future__ import annotations + +import asyncio +import hashlib +import os +import threading +import time +from collections import Counter, deque +from collections.abc import Callable, Iterator, Mapping +from contextlib import contextmanager +from contextvars import ContextVar +from dataclasses import dataclass +from typing import Any, Literal, Protocol, TypeVar + + +WorkKind = Literal["workbook", "pdf", "managed_xlsx"] +_WORK_KINDS: tuple[WorkKind, ...] = ("workbook", "pdf", "managed_xlsx") +_POOL_BY_KIND: Mapping[WorkKind, str] = { + "workbook": "workbook", + "managed_xlsx": "workbook", + "pdf": "pdf", +} +_CANCELLATION_POLL_SECONDS = 0.05 +_THREAD_TERMINATION_TIMEOUT_SECONDS = 3.0 +_ResultT = TypeVar("_ResultT") + + +class CancellationEvent(Protocol): + def is_set(self) -> bool: ... + + +class WorkBudgetError(RuntimeError): + """Base class for stable work-budget failure taxonomy.""" + + code = "WORK_BUDGET_ERROR" + + def __init__(self, kind: WorkKind, message: str) -> None: + self.kind = kind + super().__init__(message) + + +class WorkQueueFull(WorkBudgetError): + code = "WORK_QUEUE_FULL" + + def __init__(self, kind: WorkKind) -> None: + super().__init__(kind, f"{kind} work queue is full") + + +class WorkQueueTimeout(WorkBudgetError): + code = "WORK_QUEUE_TIMEOUT" + + def __init__(self, kind: WorkKind, timeout_seconds: float) -> None: + self.timeout_seconds = max(float(timeout_seconds), 0.0) + super().__init__(kind, f"{kind} work queue wait timed out") + + +class WorkAdmissionCancelled(WorkBudgetError): + code = "WORK_ADMISSION_CANCELLED" + + def __init__(self, kind: WorkKind, reason: str = "request") -> None: + self.reason = _bounded_reason(reason) + super().__init__(kind, f"{kind} work was cancelled") + + +class WorkAdmissionDraining(WorkBudgetError): + code = "WORK_ADMISSION_DRAINING" + + def __init__(self, kind: WorkKind) -> None: + super().__init__(kind, f"{kind} work rejected while service is draining") + + +class WorkExecutionTimeout(RuntimeError): + """Marker base for a subprocess wall-clock execution timeout.""" + + code = "WORK_EXECUTION_TIMEOUT" + + +@dataclass(frozen=True) +class WorkContext: + session_id: str = "" + kind: WorkKind | None = None + cancellation_event: CancellationEvent | None = None + + +_WORK_CONTEXT: ContextVar[WorkContext | None] = ContextVar("work_admission_context", default=None) + + +def current_work_context() -> WorkContext: + return _WORK_CONTEXT.get() or WorkContext() + + +@contextmanager +def work_context( + *, + session_id: str | None = None, + kind: WorkKind | None = None, + cancellation_event: CancellationEvent | None = None, +) -> Iterator[WorkContext]: + """Bind non-public fairness/cancellation metadata across thread handoffs.""" + current = current_work_context() + bound = WorkContext( + session_id=current.session_id if session_id is None else str(session_id or ""), + kind=current.kind if kind is None else _normalize_kind(kind), + cancellation_event=current.cancellation_event if cancellation_event is None else cancellation_event, + ) + token = _WORK_CONTEXT.set(bound) + try: + yield bound + finally: + _WORK_CONTEXT.reset(token) + + +async def run_cancellable_in_thread( + function: Callable[..., _ResultT], + *args: Any, + timeout_seconds: float | None = None, + session_id: str | None = None, + kind: WorkKind | None = None, + cancellation_event: CancellationEvent | None = None, + **kwargs: Any, +) -> _ResultT: + """Run blocking work while forwarding async cancellation into its context.""" + current = current_work_context() + signal = CancellationSignal(cancellation_event or current.cancellation_event) + with work_context( + session_id=session_id, + kind=kind, + cancellation_event=signal, + ): + task = asyncio.create_task(asyncio.to_thread(function, *args, **kwargs)) + try: + if timeout_seconds is None: + return await asyncio.shield(task) + timeout = max(float(timeout_seconds), 0.0) + return await asyncio.wait_for(asyncio.shield(task), timeout=timeout) + except TimeoutError: + signal.cancel("request") + work_admission_controller.wake() + await _settle_cancelled_thread(task) + raise + except asyncio.CancelledError: + signal.cancel("request") + work_admission_controller.wake() + await _settle_cancelled_thread(task) + raise + + +async def _settle_cancelled_thread(task: asyncio.Task[Any]) -> None: + try: + await asyncio.wait_for(asyncio.shield(task), timeout=_THREAD_TERMINATION_TIMEOUT_SECONDS) + except TimeoutError: + task.add_done_callback(_consume_task_result) + except (Exception, asyncio.CancelledError): + # The caller receives the original timeout/cancellation. The isolated + # worker owns its bounded process-tree teardown and reports separately. + pass + + +def _consume_task_result(task: asyncio.Task[Any]) -> None: + try: + task.exception() + except (Exception, asyncio.CancelledError): + pass + + +class CancellationSignal: + """Composite signal for caller cancellation plus controller shutdown.""" + + def __init__(self, external: CancellationEvent | None = None) -> None: + self._external = external + self._internal = threading.Event() + self._reason = "request" + + @property + def reason(self) -> str: + if not self._internal.is_set() and self._external is not None and self._external.is_set(): + return _bounded_reason(getattr(self._external, "reason", "request")) + return self._reason + + def is_set(self) -> bool: + return self._internal.is_set() or bool(self._external and self._external.is_set()) + + def wait(self, timeout: float | None = None) -> bool: + deadline = None if timeout is None else time.monotonic() + max(float(timeout), 0.0) + while not self.is_set(): + remaining = None if deadline is None else deadline - time.monotonic() + if remaining is not None and remaining <= 0: + return False + self._internal.wait( + _CANCELLATION_POLL_SECONDS if remaining is None else min(_CANCELLATION_POLL_SECONDS, remaining) + ) + return True + + def cancel(self, reason: str = "shutdown") -> None: + self._reason = _bounded_reason(reason) + self._internal.set() + + +@dataclass(frozen=True) +class WorkAdmissionSnapshot: + draining: bool + active: int + queued: int + max_concurrency: int + max_queue: int + max_queue_per_session: int + queue_timeout_seconds: float + pool_concurrency: dict[str, int] + pool_queue: dict[str, int] + active_by_kind: dict[str, int] + queued_by_kind: dict[str, int] + + +@dataclass +class _Waiter: + identifier: int + kind: WorkKind + pool: str + session_key: str + signal: CancellationSignal + enqueued_at: float + deadline: float + granted: bool = False + failure: WorkBudgetError | None = None + + +@dataclass +class _ActiveWork: + kind: WorkKind + pool: str + session_key: str + signal: CancellationSignal + + +class WorkLease: + """An admitted unit of work; releasing it opens the next fair queue slot.""" + + def __init__( + self, + controller: WorkAdmissionController, + identifier: int, + kind: WorkKind, + signal: CancellationSignal, + ) -> None: + self._controller = controller + self._identifier = identifier + self.kind = kind + self.cancellation = signal + self._released = False + + def __enter__(self) -> WorkLease: + return self + + def __exit__(self, exc_type, exc, _traceback) -> None: + if exc is None: + outcome = "completed" + elif isinstance(exc, WorkExecutionTimeout): + outcome = "execution_timeout" + elif isinstance(exc, WorkAdmissionCancelled): + outcome = "cancelled" + else: + outcome = "failed" + self.release(outcome) + + def release(self, outcome: str = "completed") -> None: + if self._released: + return + self._released = True + self._controller._release(self._identifier, outcome) + + +class WorkAdmissionController: + """Thread-safe bounded scheduler with round-robin session selection.""" + + def __init__( + self, + *, + max_concurrency: int, + max_queue: int, + max_queue_per_session: int | None = None, + queue_timeout_seconds: float, + pool_concurrency: Mapping[str, int] | None = None, + pool_queue: Mapping[str, int] | None = None, + ) -> None: + self.max_concurrency = _bounded_int(max_concurrency, 1, 64) + self.max_queue = _bounded_int(max_queue, 0, 4096) + default_session_queue = min(self.max_queue, max(self.max_queue // 4, 1)) if self.max_queue else 0 + self.max_queue_per_session = _bounded_int( + default_session_queue if max_queue_per_session is None else max_queue_per_session, + 0, + self.max_queue, + ) + self.queue_timeout_seconds = _bounded_float(queue_timeout_seconds, 0.01, 300.0) + raw_pool_concurrency = dict(pool_concurrency or {}) + raw_pool_queue = dict(pool_queue or {}) + self.pool_concurrency = { + pool: _bounded_int(raw_pool_concurrency.get(pool, self.max_concurrency), 1, self.max_concurrency) + for pool in frozenset(_POOL_BY_KIND.values()) + } + self.pool_queue = { + pool: _bounded_int(raw_pool_queue.get(pool, self.max_queue), 0, self.max_queue) + for pool in frozenset(_POOL_BY_KIND.values()) + } + self._condition = threading.Condition() + self._draining = False + self._next_identifier = 1 + self._queues: dict[str, deque[_Waiter]] = {} + self._session_ring: deque[str] = deque() + self._active: dict[int, _ActiveWork] = {} + self._queued_by_kind: Counter[str] = Counter() + self._active_by_kind: Counter[str] = Counter() + self._queued_by_pool: Counter[str] = Counter() + self._active_by_pool: Counter[str] = Counter() + + def admit( + self, + kind: WorkKind, + *, + session_id: str | None = None, + cancellation_event: CancellationEvent | None = None, + queue_timeout_seconds: float | None = None, + ) -> WorkLease: + """Block for a fair slot or raise a stable queue/admission error.""" + normalized_kind = _normalize_kind(kind) + pool = _POOL_BY_KIND[normalized_kind] + context = current_work_context() + effective_session = context.session_id if session_id is None else str(session_id or "") + effective_cancellation = cancellation_event or context.cancellation_event + signal = CancellationSignal(effective_cancellation) + timeout = self.queue_timeout_seconds if queue_timeout_seconds is None else _bounded_float( + queue_timeout_seconds, 0.0, 300.0 + ) + started = time.monotonic() + + with self._condition: + if signal.is_set(): + self._record_event(normalized_kind, "cancelled", 0.0) + raise WorkAdmissionCancelled(normalized_kind, signal.reason) + if self._draining: + self._record_event(normalized_kind, "draining", 0.0) + raise WorkAdmissionDraining(normalized_kind) + + identifier = self._next_identifier + self._next_identifier += 1 + session_key = _session_key(effective_session) + if not self._queues and self._can_run_locked(pool): + self._activate_locked(identifier, normalized_kind, pool, session_key, signal) + self._publish_state_locked() + self._record_event(normalized_kind, "admitted", 0.0) + return WorkLease(self, identifier, normalized_kind, signal) + + session_queue = self._queues.get(session_key) + if ( + self._queued_count_locked() >= self.max_queue + or self._queued_by_pool[pool] >= self.pool_queue[pool] + or len(session_queue or ()) >= self.max_queue_per_session + ): + self._record_event(normalized_kind, "queue_full", 0.0) + raise WorkQueueFull(normalized_kind) + + waiter = _Waiter( + identifier=identifier, + kind=normalized_kind, + pool=pool, + session_key=session_key, + signal=signal, + enqueued_at=started, + deadline=started + timeout, + ) + queue = self._queues.get(session_key) + if queue is None: + queue = self._queues[session_key] = deque() + self._session_ring.append(session_key) + queue.append(waiter) + self._queued_by_kind[normalized_kind] += 1 + self._queued_by_pool[pool] += 1 + self._grant_waiters_locked() + self._publish_state_locked() + + while not waiter.granted and waiter.failure is None: + if signal.is_set(): + self._remove_waiter_locked(waiter) + waiter.failure = WorkAdmissionCancelled(normalized_kind, signal.reason) + break + remaining = waiter.deadline - time.monotonic() + if remaining <= 0: + self._remove_waiter_locked(waiter) + waiter.failure = WorkQueueTimeout(normalized_kind, timeout) + break + self._condition.wait(min(remaining, _CANCELLATION_POLL_SECONDS)) + + waited = max(time.monotonic() - started, 0.0) + self._grant_waiters_locked() + self._publish_state_locked() + if waiter.failure is not None: + outcome = { + WorkAdmissionCancelled: "cancelled", + WorkAdmissionDraining: "draining", + WorkQueueTimeout: "queue_timeout", + }.get(type(waiter.failure), "failed") + self._record_event(normalized_kind, outcome, waited) + raise waiter.failure + self._record_event(normalized_kind, "admitted", waited) + return WorkLease(self, identifier, normalized_kind, signal) + + def snapshot(self) -> WorkAdmissionSnapshot: + with self._condition: + return self._snapshot_locked() + + def redacted_status(self) -> dict[str, object]: + """Return bounded aggregate state without any session or document key.""" + snapshot = self.snapshot() + status = "draining" if snapshot.draining else ("busy" if snapshot.active or snapshot.queued else "ready") + return { + "status": status, + "draining": snapshot.draining, + "active": snapshot.active, + "queued": snapshot.queued, + "budgets": { + "max_concurrency": snapshot.max_concurrency, + "max_queue": snapshot.max_queue, + "max_queue_per_session": snapshot.max_queue_per_session, + "queue_timeout_seconds": snapshot.queue_timeout_seconds, + "pool_concurrency": snapshot.pool_concurrency, + "pool_queue": snapshot.pool_queue, + }, + "active_by_kind": snapshot.active_by_kind, + "queued_by_kind": snapshot.queued_by_kind, + } + + def start_accepting(self) -> None: + with self._condition: + self._draining = False + self._grant_waiters_locked() + self._condition.notify_all() + self._publish_state_locked() + + def begin_drain(self) -> bool: + """Reject new work and fail every queued waiter; active work may finish.""" + with self._condition: + if self._draining: + return False + self._draining = True + for queue in self._queues.values(): + for waiter in queue: + waiter.failure = WorkAdmissionDraining(waiter.kind) + self._queues.clear() + self._session_ring.clear() + self._queued_by_kind.clear() + self._queued_by_pool.clear() + self._condition.notify_all() + self._publish_state_locked() + return True + + def cancel_running(self, reason: str = "shutdown") -> int: + with self._condition: + active = tuple(self._active.values()) + for work in active: + work.signal.cancel(reason) + self._condition.notify_all() + return len(active) + + def wake(self) -> None: + with self._condition: + self._condition.notify_all() + + async def wait_for_idle(self, timeout_seconds: float) -> bool: + return await asyncio.to_thread(self.wait_for_idle_blocking, timeout_seconds) + + def wait_for_idle_blocking(self, timeout_seconds: float) -> bool: + deadline = time.monotonic() + max(float(timeout_seconds), 0.0) + with self._condition: + while self._active or self._queued_count_locked(): + remaining = deadline - time.monotonic() + if remaining <= 0: + return False + self._condition.wait(remaining) + return True + + def _release(self, identifier: int, outcome: str) -> None: + with self._condition: + active = self._active.pop(identifier, None) + if active is None: + return + self._active_by_kind[active.kind] -= 1 + self._active_by_pool[active.pool] -= 1 + self._prune_counter(self._active_by_kind, active.kind) + self._prune_counter(self._active_by_pool, active.pool) + if not self._draining: + self._grant_waiters_locked() + self._condition.notify_all() + self._publish_state_locked() + self._record_event(active.kind, _bounded_outcome(outcome), 0.0) + + def _grant_waiters_locked(self) -> None: + while self._session_ring and len(self._active) < self.max_concurrency and not self._draining: + granted = False + sessions_to_scan = len(self._session_ring) + for _ in range(sessions_to_scan): + session_key = self._session_ring.popleft() + queue = self._queues.get(session_key) + if not queue: + self._queues.pop(session_key, None) + continue + + while queue and queue[0].signal.is_set(): + cancelled = queue.popleft() + self._decrement_queued_locked(cancelled) + cancelled.failure = WorkAdmissionCancelled(cancelled.kind, cancelled.signal.reason) + if not queue: + self._queues.pop(session_key, None) + continue + + waiter = queue[0] + if self._can_run_locked(waiter.pool): + queue.popleft() + self._decrement_queued_locked(waiter) + if queue: + self._session_ring.append(session_key) + else: + self._queues.pop(session_key, None) + waiter.granted = True + self._activate_locked( + waiter.identifier, + waiter.kind, + waiter.pool, + waiter.session_key, + waiter.signal, + ) + granted = True + self._condition.notify_all() + break + self._session_ring.append(session_key) + if not granted: + break + + def _activate_locked( + self, + identifier: int, + kind: WorkKind, + pool: str, + session_key: str, + signal: CancellationSignal, + ) -> None: + self._active[identifier] = _ActiveWork(kind=kind, pool=pool, session_key=session_key, signal=signal) + self._active_by_kind[kind] += 1 + self._active_by_pool[pool] += 1 + + def _remove_waiter_locked(self, waiter: _Waiter) -> None: + queue = self._queues.get(waiter.session_key) + if queue is None: + return + try: + queue.remove(waiter) + except ValueError: + return + self._decrement_queued_locked(waiter) + if not queue: + self._queues.pop(waiter.session_key, None) + try: + self._session_ring.remove(waiter.session_key) + except ValueError: + pass + + def _decrement_queued_locked(self, waiter: _Waiter) -> None: + self._queued_by_kind[waiter.kind] -= 1 + self._queued_by_pool[waiter.pool] -= 1 + self._prune_counter(self._queued_by_kind, waiter.kind) + self._prune_counter(self._queued_by_pool, waiter.pool) + + def _can_run_locked(self, pool: str) -> bool: + return len(self._active) < self.max_concurrency and self._active_by_pool[pool] < self.pool_concurrency[pool] + + def _queued_count_locked(self) -> int: + return sum(self._queued_by_kind.values()) + + def _snapshot_locked(self) -> WorkAdmissionSnapshot: + return WorkAdmissionSnapshot( + draining=self._draining, + active=len(self._active), + queued=self._queued_count_locked(), + max_concurrency=self.max_concurrency, + max_queue=self.max_queue, + max_queue_per_session=self.max_queue_per_session, + queue_timeout_seconds=self.queue_timeout_seconds, + pool_concurrency=dict(sorted(self.pool_concurrency.items())), + pool_queue=dict(sorted(self.pool_queue.items())), + active_by_kind={kind: int(self._active_by_kind[kind]) for kind in _WORK_KINDS}, + queued_by_kind={kind: int(self._queued_by_kind[kind]) for kind in _WORK_KINDS}, + ) + + def _publish_state_locked(self) -> None: + from backend.observability import set_work_admission_state + + for kind in _WORK_KINDS: + set_work_admission_state(kind, self._active_by_kind[kind], self._queued_by_kind[kind]) + + @staticmethod + def _record_event(kind: WorkKind, outcome: str, wait_seconds: float) -> None: + from backend.observability import record_work_admission_event + + record_work_admission_event(kind, outcome, wait_seconds) + + @staticmethod + def _prune_counter(counter: Counter[str], key: str) -> None: + if counter[key] <= 0: + counter.pop(key, None) + + +def _normalize_kind(kind: object) -> WorkKind: + normalized = str(kind or "").strip().lower() + if normalized not in _WORK_KINDS: + raise ValueError("Unsupported work admission kind") + return normalized # type: ignore[return-value] + + +def _session_key(session_id: object) -> str: + normalized = str(session_id or "").strip() + if not normalized: + return "anonymous" + return hashlib.blake2s(normalized.encode("utf-8", errors="ignore"), digest_size=16).hexdigest() + + +def _bounded_reason(reason: object) -> str: + normalized = str(reason or "request").strip().lower() + return normalized if normalized in {"request", "shutdown", "session_purge"} else "request" + + +def _bounded_outcome(outcome: object) -> str: + normalized = str(outcome or "failed").strip().lower() + allowed = {"completed", "failed", "cancelled", "execution_timeout"} + return normalized if normalized in allowed else "failed" + + +def _bounded_int(value: object, minimum: int, maximum: int) -> int: + try: + parsed = int(str(value).strip()) + except (TypeError, ValueError): + parsed = minimum + return min(max(parsed, minimum), maximum) + + +def _bounded_float(value: object, minimum: float, maximum: float) -> float: + try: + parsed = float(str(value).strip()) + except (TypeError, ValueError): + parsed = minimum + return min(max(parsed, minimum), maximum) + + +def _env_int(name: str, default: int, minimum: int, maximum: int) -> int: + return _bounded_int(os.getenv(name, str(default)), minimum, maximum) + + +def _env_float(name: str, default: float, minimum: float, maximum: float) -> float: + return _bounded_float(os.getenv(name, str(default)), minimum, maximum) + + +_GLOBAL_MAX_CONCURRENCY = _env_int("WORK_ADMISSION_MAX_CONCURRENCY", 4, 1, 64) +_GLOBAL_MAX_QUEUE = _env_int("WORK_ADMISSION_MAX_QUEUE", 64, 0, 4096) + +work_admission_controller = WorkAdmissionController( + max_concurrency=_GLOBAL_MAX_CONCURRENCY, + max_queue=_GLOBAL_MAX_QUEUE, + max_queue_per_session=_env_int("WORK_ADMISSION_MAX_QUEUE_PER_SESSION", 8, 0, _GLOBAL_MAX_QUEUE), + queue_timeout_seconds=_env_float("WORK_ADMISSION_QUEUE_TIMEOUT_SECONDS", 5.0, 0.01, 300.0), + pool_concurrency={ + "workbook": _env_int("WORKBOOK_WORKER_MAX_CONCURRENCY", 2, 1, _GLOBAL_MAX_CONCURRENCY), + "pdf": _env_int("PDF_PARSE_MAX_CONCURRENCY", 2, 1, _GLOBAL_MAX_CONCURRENCY), + }, + pool_queue={ + "workbook": _env_int("WORKBOOK_WORKER_MAX_QUEUE", 32, 0, _GLOBAL_MAX_QUEUE), + "pdf": _env_int("PDF_PARSE_MAX_QUEUE", 16, 0, _GLOBAL_MAX_QUEUE), + }, +) diff --git a/backend/platforms/google_sheets.py b/backend/platforms/google_sheets.py index c7bc262..02e3544 100644 --- a/backend/platforms/google_sheets.py +++ b/backend/platforms/google_sheets.py @@ -12,9 +12,8 @@ from __future__ import annotations -import json import re -from dataclasses import dataclass, field +from dataclasses import dataclass from typing import Optional import structlog diff --git a/backend/requirements.lock.txt b/backend/requirements.lock.txt new file mode 100644 index 0000000..0361451 --- /dev/null +++ b/backend/requirements.lock.txt @@ -0,0 +1,2036 @@ +# Universal Python 3.12 dependency lock with artifact hashes for pip +# --require-hashes. Regenerate only through an intentional dependency update: +# uv pip compile backend/requirements.txt --universal --generate-hashes \ +# --no-annotate --no-header --output-file backend/requirements.lock.txt +annotated-doc==0.0.5 \ + --hash=sha256:117bac03a25ede5df5440e855b32d556049ca169ead221505badf432fed4b101 \ + --hash=sha256:c7e58ce09192557605d8bbd92836d7e1d520ac9580096042c0bfd197efacf1bb +annotated-types==0.8.0 \ + --hash=sha256:13b2beaad985e05e2d6407ee4c4f35590b11f8d693a258a561055cac8f64cab7 \ + --hash=sha256:f072f4d804ea359e4eaf198b1af7a8b0943881a87f31bb764f8bf219bb9419e0 +anthropic==0.121.0 \ + --hash=sha256:6048713fa441e59e1cba8363171cd2a86273b25bd213e9c7ac70a523af88b011 \ + --hash=sha256:e79d6e08ab3376602fc9a70d4d5ea3540817c76cf7e16658bed790834e1833d6 +anyio==4.14.2 \ + --hash=sha256:9f505dda5ac9f0c8309b5e8bd445a8c2bf7246f3ce950121e45ea15bc41d1494 \ + --hash=sha256:cfa139f3ed1a23ee8f88a145ddb5ac7605b8bbfd8592baacd7ce3d8bb4313c7f +ast-serialize==0.8.0 \ + --hash=sha256:0485a25ef519c62e749ee3c1ad8070e591b380d67226349eb5a70b228dc1ac4a \ + --hash=sha256:057769b5921336eb2d9124f2a731b42ed05ffdac559b840dbdf6f3937cf153dc \ + --hash=sha256:1f9caa63fad8241257ae401b5ff0a64026c6adb36b8e86cbe8782d9ea505daf6 \ + --hash=sha256:252f883290d1cdb728eb7fe1d9a7221b88af5a329aae0bc91ddee4dafb820331 \ + --hash=sha256:2880350b13d3eae69a0d70bc1fb6c9bfaca4dbd0e20ba8cd1aa483080b56ff06 \ + --hash=sha256:293cc1c5bfa741f8e3fbe8175b9c07beee487c9a6fdbb25a5acad9f1df2d30a9 \ + --hash=sha256:2d39a56282cfcc0d8eeea37267c754be59c98d48505c23b1dae5c6011f3813dd \ + --hash=sha256:2efa40b068197d5efb62655b43baadb842ed71c4958cccd3e8b86a35726f0119 \ + --hash=sha256:31883542dd6c94d178f5db3d32fbd69c5eb88b3a7c018e7ac8cc0c45195ddbed \ + --hash=sha256:3926fa117b5e65019853a2969966d11c7175af377a3425991f3fe73784412405 \ + --hash=sha256:39e92ff8e8cb45947fe9007174b2950e1fb098e6abd00266a13cd3bcf6675068 \ + --hash=sha256:3a8660fe66667b76a6e9dccd1d33e66b229fde3b308db991c041609226c005b6 \ + --hash=sha256:3ccebbed24f1281062d5852353c72c47502955926cfcb8345ffb3a44d87ff3d3 \ + --hash=sha256:3d822605fa7bb326ef868d25fafced7fc660fa46d9b90c02ea86d5e2f5d325f7 \ + --hash=sha256:40a57b73731be45da4fa41430c4d5dc94a24b3a4faba7b9e069978c0402064ea \ + --hash=sha256:43dd6d596879bb1cb8a12cc9dae7bb10090a39a35883026c24f82488a195619a \ + --hash=sha256:485f1113af805e9e170b95ef993ca3fbd4f89c04bab25c58b4fc632d854801ab \ + --hash=sha256:4c38b915511e32bc718c49dbce98ff9af36bac0ad6a604f58000cd5e3aecdba7 \ + --hash=sha256:4ca7e6fd1ad845d1cc649dc2ecd499db2f8f46af5bf8da7b70dd858774cc038b \ + --hash=sha256:5075b9da3ef807eda752502446dfecea3b381c4900b7e27a5d5f4f899eb39951 \ + --hash=sha256:54f95b486018d262bcb387a9afd96f0da74508b442762b80c769454a6fbb3ee3 \ + --hash=sha256:6102f2f985c2e542be85cd857678ec9356fefa792b93cadfadd31139f5696f27 \ + --hash=sha256:6479d9722a4cd21b578f5478074c41e6169f04811996ec881655560f703a5bba \ + --hash=sha256:6c37c43e4004dfb42d321ddedc569dc17ff4259296f3af577c9ea46a809bc010 \ + --hash=sha256:6f18048fe9f6dd266bd577cdec48bdcecb74faaa01fe941324435483b013ed2a \ + --hash=sha256:77308ae6c5cf5264cc0f01a7c556ec77a9e68eb1f61b093534d698139fdc3b14 \ + --hash=sha256:861794565b06337005c1447ef23103a3d5a627d08bdc827870d00d0b28ef5f51 \ + --hash=sha256:86b8a1e6d90467345356098b040150e82fbc26d24a7a202224b13dc1f6264ca0 \ + --hash=sha256:8c9d537f59e936392cfd3597789d1390304dd659efc3c486ce7f40fb6b8a9f53 \ + --hash=sha256:8d53a23f27e1ed3a36b2d26fd2a1a6228c8e85a1ed62ff7cdb44bd610769f20a \ + --hash=sha256:9118ad3e369727060b2696fc4078f250ecffca4248ba87f537f55cea9f9dce06 \ + --hash=sha256:96abc072ad29db8d02194afd47d68987322622787daceae82398d7b69f3ba2e6 \ + --hash=sha256:9830ff7e764f74d9eefb01170c61a9f0fd2c027dac5fcb72e064decd57d56371 \ + --hash=sha256:9a2ef9cf12f2de4f1028c42c1dd7d775255e0fb3e5bb48896c97e35ef52366fe \ + --hash=sha256:9d187197d234aa45d6cfa2b096be5f666e8cc2e7eb3722d0ab8926293cf5720c \ + --hash=sha256:9da7330f3e235bf7da89b8d39205c6350fc0c08a85379743f2df9fff87d6d980 \ + --hash=sha256:a02cbed7d8bfdcdee88edaac12bd50d53d9953aaa2e1852ef078625be5f1c0b5 \ + --hash=sha256:a63bed264e818cd83eec11feed0f50aa162542b91132ef58afebc857182763a5 \ + --hash=sha256:ab0f9a59f7d63d0d441b56b9a818b273705264352d5115cfee12e940e816d958 \ + --hash=sha256:ac4f0a83c55a9b782f79ad55a5247b7db123c1db405959791c2ef886e9710c9f \ + --hash=sha256:b2a5978662fd4db463dfb4b974d2b10ac6430b98f5333aabc7051909df3561d0 \ + --hash=sha256:bd84d60bca7079e741be4ac5dbe237751a59d7f6f9f0126b11880d63822cbe16 \ + --hash=sha256:c85d8d18db5b2dfcb3b7e38a4d600ca35504c0ed8a6f75cd1c811e4ffe248a15 \ + --hash=sha256:d8b3c8eee4c1baef9d4e84d2a59a805501617127be42615cb48970b15b0892b6 \ + --hash=sha256:db1b957291bca08c7e72f43a12357b2948e20775d970e3fc3dac0aa3160ab725 \ + --hash=sha256:ddd3b61f45c132da66c5476b281891e08c1fd87fbdabe8a6973e1622efc85f06 \ + --hash=sha256:e0910c3442a75216dde0f102d854ba2aaa71d2482e0ee213630b9bf29584fba3 \ + --hash=sha256:e1bd223df0f6c96b396975fa604cb33bce53d9b4a0185490be4c4a289f7c9c87 \ + --hash=sha256:e7266307e5fba39836edb79def8608887af48820508bff3c5f2941e1e04d1534 \ + --hash=sha256:e94f9121d13fa36cbf21314783c77d05ae3a0868decd18cf5233fdcc6de49ac8 \ + --hash=sha256:f0190a33d7f97c65e9069f7a7f40499eea6b5cbe260c558378109caf20ce934b \ + --hash=sha256:f3186969ee66a9863b00acc6523ace44c56974eecb348a7ea4b228d9f0b80e19 \ + --hash=sha256:f359df4bd921918af8bebd142a376c77511d7151cc8ba852760b587b5a4a54f3 \ + --hash=sha256:f7cc5f10386994c0f4844f1e6d6a97127e9b478660eb6dec2b257644f0acab64 \ + --hash=sha256:fa70ed4dea0bb18b30a1789c77baa701d0ef30c474f2ccabdea61e25623a8827 \ + --hash=sha256:fdc0d5b18ff8fb364e87923e47c0a91d0d69dbcaeaa274591f7fd26892cc3a3a \ + --hash=sha256:ffa5e7cb08f96fed9121f77b224151e41caf88feab9d652bb46c78202b6fbeda +asyncpg==0.31.0 \ + --hash=sha256:027eaa61361ec735926566f995d959ade4796f6a49d3bde17e5134b9964f9ba8 \ + --hash=sha256:04d19392716af6b029411a0264d92093b6e5e8285ae97a39957b9a9c14ea72be \ + --hash=sha256:0b17c89312c2f4ccea222a3a6571f7df65d4ba2c0e803339bfc7bed46a96d3be \ + --hash=sha256:0bfbcc5b7ffcd9b75ab1558f00db2ae07db9c80637ad1b2469c43df79d7a5ae2 \ + --hash=sha256:0c89ccf741c067614c9b5fc7f1fc6f3b61ab05ae4aaa966e6fd6b93097c7d20d \ + --hash=sha256:12b3b2e39dc5470abd5e98c8d3373e4b1d1234d9fbdedf538798b2c13c64460a \ + --hash=sha256:18c83b03bc0d1b23e6230f5bf8d4f217dc9bc08644ce0502a9d91dc9e634a9c7 \ + --hash=sha256:19857a358fc811d82227449b7ca40afb46e75b33eb8897240c3839dd8b744218 \ + --hash=sha256:1b41f1afb1033f2b44f3234993b15096ddc9cd71b21a42dbd87fc6a57b43d65d \ + --hash=sha256:22bc525ebbdc24d1261ecbf6f504998244d4e3be1721784b5f64664d61fbe602 \ + --hash=sha256:22be6e02381bab3101cd502d9297ac71e2f966c86e20e78caead9934c98a8af6 \ + --hash=sha256:2657204552b75f8288de08ca60faf4a99a65deef3a71d1467454123205a88fab \ + --hash=sha256:2d076d42eb583601179efa246c5d7ae44614b4144bc1c7a683ad1222814ed095 \ + --hash=sha256:334dec28cf20d7f5bb9e45b39546ddf247f8042a690bff9b9573d00086e69cb5 \ + --hash=sha256:37a58919cfef2448a920df00d1b2f821762d17194d0dbf355d6dde8d952c04f9 \ + --hash=sha256:37fc6c00a814e18eef51833545d1891cac9aa69140598bb076b4cd29b3e010b9 \ + --hash=sha256:3b1fbcb0e396a5ca435a8826a87e5c2c2cc0c8c68eb6fadf82168056b0e53a8c \ + --hash=sha256:3df118d94f46d85b2e434fd62c84cb66d5834d5a890725fe625f498e72e4d5ec \ + --hash=sha256:3faa62f997db0c9add34504a68ac2c342cfee4d57a0c3062fcf0d86c7f9cb1e8 \ + --hash=sha256:480c4befbdf079c14c9ca43c8c5e1fe8b6296c96f1f927158d4f1e750aacc047 \ + --hash=sha256:54a64f91839ba59008eccf7aad2e93d6e3de688d796f35803235ea1c4898ae1e \ + --hash=sha256:5a4af56edf82a701aece93190cc4e094d2df7d33f6e915c222fb09efbb5afc24 \ + --hash=sha256:6d11b198111a72f47154fa03b85799f9be63701e068b43f84ac25da0bda9cb31 \ + --hash=sha256:72d6bdcbc93d608a1158f17932de2321f68b1a967a13e014998db87a72ed3186 \ + --hash=sha256:795416369c3d284e1837461909f58418ad22b305f955e625a4b3a2521d80a5f3 \ + --hash=sha256:831712dd3cf117eec68575a9b50da711893fd63ebe277fc155ecae1c6c9f0f61 \ + --hash=sha256:8df714dba348efcc162d2adf02d213e5fab1bd9f557e1305633e851a61814a7a \ + --hash=sha256:8ea599d45c361dfbf398cb67da7fd052affa556a401482d3ff1ee99bd68808a1 \ + --hash=sha256:9322b563e2661a52e3cdbc93eed3be7748b289f792e0011cb2720d278b366ce2 \ + --hash=sha256:98cc158c53f46de7bb677fd20c417e264fc02b36d901cc2a43bd6cb0dc6dbfd2 \ + --hash=sha256:9ea33213ac044171f4cac23740bed9a3805abae10e7025314cfbd725ec670540 \ + --hash=sha256:a429e842a3a4b4ea240ea52d7fe3f82d5149853249306f7ff166cb9948faa46c \ + --hash=sha256:a8d758dac9d2e723e173d286ef5e574f0b350ec00e9186fce84d0fc5f6a8e6b8 \ + --hash=sha256:aad7a33913fb8bcb5454313377cc330fbb19a0cd5faa7272407d8a0c4257b671 \ + --hash=sha256:b44c31e1efc1c15188ef183f287c728e2046abb1d26af4d20858215d50d91fad \ + --hash=sha256:ba5f8886e850882ff2c2ace5732300e99193823e8107e2c53ef01c1ebfa1e85d \ + --hash=sha256:bb223567dea5f47c45d347f2bde5486be8d9f40339f27217adb3fb1c3be51298 \ + --hash=sha256:bc2b685f400ceae428f79f78b58110470d7b4466929a7f78d455964b17ad1008 \ + --hash=sha256:bd4107bb7cdd0e9e65fae66a62afd3a249663b844fa34d479f6d5b3bef9c04c3 \ + --hash=sha256:bd5b6efff3c17c3202d4b37189969acf8927438a238c6257f66be3c426beba20 \ + --hash=sha256:bdb957706da132e982cc6856bb2f7b740603472b54c3ebc77fe60ea3e57e1bd2 \ + --hash=sha256:bef056aa502ee34204c161c72ca1f3c274917596877f825968368b2c33f585f4 \ + --hash=sha256:c0807be46c32c963ae40d329b3a686356e417f674c976c07fa49f1b30303f109 \ + --hash=sha256:c0e0822b1038dc7253b337b0f3f676cadc4ac31b126c5d42691c39691962e403 \ + --hash=sha256:c1a9c5b71d2371a2290bc93336cd05ba4ec781683cab292adbddc084f89443c6 \ + --hash=sha256:c1e1ab5bc65373d92dd749d7308c5b26fb2dc0fbe5d3bf68a32b676aa3bcd24a \ + --hash=sha256:c204fab1b91e08b0f47e90a75d1b3c62174dab21f670ad6c5d0f243a228f015b \ + --hash=sha256:c989386c83940bfbd787180f2b1519415e2d3d6277a70d9d0f0145ac73500735 \ + --hash=sha256:cea3a0b2a14f95834cee29432e4ddc399b95700eb1d51bbc5bfee8f31fa07b2b \ + --hash=sha256:dc5f2fa9916f292e5c5c8b2ac2813763bcd7f58e130055b4ad8a0531314201ab \ + --hash=sha256:e009abc333464ff18b8f6fd146addffd9aaf63e79aa3bb40ab7a4c332d0c5e9e \ + --hash=sha256:e5d5098f63beeae93512ee513d4c0c53dc12e9aa2b7a1af5a81cddf93fe4e4da \ + --hash=sha256:e6974f36eb9a224d8fb428bcf66bd411aa12cf57c2967463178149e73d4de366 \ + --hash=sha256:ebb3cde58321a1f89ce41812be3f2a98dddedc1e76d0838aba1d724f1e4e1a95 \ + --hash=sha256:eee690960e8ab85063ba93af2ce128c0f52fd655fdff9fdb1a28df01329f031d \ + --hash=sha256:f6b56b91bb0ffc328c4e3ed113136cddd9deefdf5f79ab448598b9772831df44 \ + --hash=sha256:f890de5e1e4f7e14023619399a471ce4b71f5418cd67a51853b9910fdfa73696 +certifi==2026.7.22 \ + --hash=sha256:62f22742b58a1a33014a2b6b706588a8d7e2a88ae7bd1a6ebe8c992928483775 \ + --hash=sha256:741e2c3b351ddf169a738da9f2c048608ff7f2c5cc02f1ebc6b118bb090d5d55 +cffi==2.1.1 \ + --hash=sha256:046bfc24911b37851ee1b51aab8bffe713d89c68c6a057b09484ce9fd5f69b4e \ + --hash=sha256:06c72bb76605a4b0cd0aad6930b69d4baf7dd5d806cfc409b824191099700e66 \ + --hash=sha256:0beceaabe56af686895136a2de78db54ecd8e4046b236b8fd6d6cb61389e9bf2 \ + --hash=sha256:154852545011f779917b11c78db2358d095da62a9a172b78ad0a583ee5adc0d0 \ + --hash=sha256:194cffa889098ced9976c3fc6340305e43f6303657d298da55366907c05c22d6 \ + --hash=sha256:19ee6127ee34de7d83ce3d371ebc5ed91addbdcc39f9ab15ce4eb35a4e534971 \ + --hash=sha256:1a18a57b58cfb21fc28d72e876acf10eaed67a1ed96226f92af4df681d571c4c \ + --hash=sha256:1aa5645c30469b09530c4ebca77ebf8f17618293c58f8549cb1a543a50236e7d \ + --hash=sha256:1dea0e4d7d4f11f619fe8c1d76caf49e24405b4b5743c0e3be16a500ecd930c9 \ + --hash=sha256:208f941bb9d18e768138677f0a6d2ce01f590df56043dda1df1535ac57c88517 \ + --hash=sha256:210019b6c7cf07f081b4c54635c8cf744377001350e29cc0f81c4377b4797735 \ + --hash=sha256:246fa40ce8645a614ff682e0b70f37134e460eaf93a775e0cbe3cca585a67a80 \ + --hash=sha256:25792eac27877609e7bb06d42ff88278a6624fff2ba9bbb523c09616b117e80f \ + --hash=sha256:27350daa11d4f10c540e6e89dada4c54feb7256ad03e9a4dc075ebad7ba360d1 \ + --hash=sha256:28907ab9bfb6aa13184cfc17c6b8e1023c5ab6fd7076d8c20a35e59fe04f8f29 \ + --hash=sha256:2ae64be792b8966f2c69538199728b290e34726562896df1e5dc8ffd8d8188e8 \ + --hash=sha256:31348097ff5bbe827ccc41795d4dd099d9f0625e7def00ee653c137a490c2a6c \ + --hash=sha256:3143d81e29e1e20a9ce10901ec369012947876596f75a222235965f2b7ae832e \ + --hash=sha256:3222ba5d678f80a030e6afbcc33dc1ae5cb45facabb61cee2c7016b8432fde48 \ + --hash=sha256:3311ed60d36f83378794e1009ac6258bafbf81f7888b4caa7b35a521e3f95813 \ + --hash=sha256:334644fbac4eff73d985a17a91226df55d0f394160c4cfb880e084c8f7161cac \ + --hash=sha256:34e261f78cb6ceaaa36f42f2613f4380d94d9c759a9c73c769ee6e0247364632 \ + --hash=sha256:363e05fa78e15116c3c32c210ee36884fd6b9afa6d440e47112c3bd511d64cb6 \ + --hash=sha256:398aff33cee2767e3e781d2554c54bd0dff386bb437581e0d8011fde1a942ec1 \ + --hash=sha256:3d22a20b1fb1632cc72c22f95f7b0d2961c3e1c235f245ba4c606c4771035659 \ + --hash=sha256:42a494cee34437f05546455144f2b5d9ac09b1face62bcfce597d2e521066688 \ + --hash=sha256:42e2f76b9455f5a9a844f770bf3e200ed3da0e15f5df3db9c31fe80b04b3d004 \ + --hash=sha256:42f6930c31dc7f50732c9ae793c2786c7b6b044195967bbdde40bb9be81c4cc0 \ + --hash=sha256:456a61fa52d579ebf9df2e9552ead5129855dbaff6c1e5a9b1bc408809bdc062 \ + --hash=sha256:471cee653ae88de62096552e6d24ccb4a5adb8c8c9f10b5054d0122c15bf2779 \ + --hash=sha256:49cbc70e6542d4ccccb936558d1064a8012541e78f821f955cff24e357776c94 \ + --hash=sha256:4a7c934f7360e8cd64fe9efadcbd10c7c6364f531e432b9a4bf5ccbc9e0e8b50 \ + --hash=sha256:4be96343e422f2dfcd12ab5c9f5aebe03f82f737c6bffeca6830b3875cb44aab \ + --hash=sha256:4f42141fc14250de6dde5ee7ea4432be017252d91f19c5ad043c084cea629cac \ + --hash=sha256:507a24c282e0f42f8ed737cf048572cbf580468da5555764a8331735e9c736b6 \ + --hash=sha256:51b31d1c98274844cfd7838ce00bfc27c7423a4dc00fc0772fc3331c2cc90676 \ + --hash=sha256:58acb8ab8e295e6c5ea12f888cbb13cf21511ef2a3303a23f4325c29d17fe5c1 \ + --hash=sha256:5a59cc1c4442bc3d5c703bf720b51138d0bfc173618807c9ee2490a7541dd3d9 \ + --hash=sha256:5bb4e7ea95dcd6a014a6fef62e62467d67d8e582326443f3d68e71d6320a9fcf \ + --hash=sha256:5c58fe613dc5e5336357eff555824a314d8e43282600435c8d1cb6a7a2fedd13 \ + --hash=sha256:5e7cecbaadb83884793e05828cee59b210b24583b9c7425d0ba6a754fe22eb4e \ + --hash=sha256:616f097f2fe415bc92a247f02e11f634e1f9e9a83d327e3c915c15089c87869e \ + --hash=sha256:63bbfd5ded17c4840ac07cd8f1c21ba9d9708141f840b324f422f41b207e3973 \ + --hash=sha256:64faea20f4e2613363a1a9b9c7dd73058f3ecd00133a511e72ad7c511658f527 \ + --hash=sha256:661c298b4821edebead0c91edd2b00374d67ad7c5a1f7a91d4442633b79d6a72 \ + --hash=sha256:68e62fe11f30d5ca8289242866f0a5291402d8529ca2178ab8afc5c9694ae890 \ + --hash=sha256:6a8dddef476fab96d066d578fc88526767b836ab5ab21754e1d5bf3879c31c7c \ + --hash=sha256:6e192623c49c94421616a5778fba35cf0d5a8d000650c1967ef4448ee5cdd990 \ + --hash=sha256:7225e4514edb64eb6740324353e0da0711954fd8d7da4576755b1c6e09b697cd \ + --hash=sha256:75f80557d1389eddbd0de2681f6a390a0c5338c31ddaa821381c203fc3fd50d9 \ + --hash=sha256:770de9db11e84213beec501cfcaa013b019820ca881e03344dea5844f7876d94 \ + --hash=sha256:7750c6449dff7864bb9bb27ddfb0267756189201a3afc911d82b3caacd70dfc3 \ + --hash=sha256:7bde5e4cc5c10140859842b9d383af292b22639a4dffb725314baf45968cef80 \ + --hash=sha256:7ce713ace7c0e4520535b42b77eaa742c16dab813978064913e5a3cf82973b41 \ + --hash=sha256:7da0c5eff80f0197f3b3d1232ec5a682a9325f4ae9016a78f5f5ca35f9ced1f5 \ + --hash=sha256:7dbb61fe3a7699468030f71bbe5f8a0e326a151daa91beb11a6fc1f980c55e1c \ + --hash=sha256:811bd1e21d32de12efca32393a0ab3f5133b54fce9bd44b8bd77ab07da14bf6a \ + --hash=sha256:8ef53b2de9bcb9197d31854256575d59dbac0cba72ac627bb291ef5eceb74be4 \ + --hash=sha256:937c0052c05a31ca1daf18de3158eed4dbfcb9cc107adbea227728d647be701e \ + --hash=sha256:9d2055050ea716bd38b7f7f1579c275386646b4894c155a3e2f3cd62ed41b7c6 \ + --hash=sha256:9f8d177621de5cb38ee3e731eda45d421db093ec0739f46a5594babda7987a98 \ + --hash=sha256:a2d7755bef5a12ed488f4ef1f1b69ee9191d7396083b755a5d2295f6edb4768b \ + --hash=sha256:a48d62ab9d6f4f98c983223a547af44be6ca3691074c31cecced6facd3ba2dc1 \ + --hash=sha256:a4f00aa42f75d6e4595e8866e748cc1705adc0cddfeb2ca86d0d03993d63ba03 \ + --hash=sha256:a6e721d4b0e45d5b65e87534470e67b18dcd092c83f68fba09f152b9cbc061af \ + --hash=sha256:a730a083190634c65cca36ba5f489531576ebd79bcd5c8e172130f6453127231 \ + --hash=sha256:a931079504ecc49efed7744c476a5c343a92fabf66dec2db95edb1b2fdc770e2 \ + --hash=sha256:aa9511c62d14da7aacc9b4bf51f3f697a621e83b2d6919008243c3aad168eea3 \ + --hash=sha256:ab36d55f9ed2d067327667c2fea18dda018eb628dd6347aa01dda6cf1f5d3836 \ + --hash=sha256:ad2c86c495b899d862ea0f4b42891b8713a3bd45dd4105c7fd51c2a72f39f3a5 \ + --hash=sha256:aeae0e330c9f6acd681f647d46cefd30c29f93e3392882e792e82080c9691399 \ + --hash=sha256:b0431303acaea1089ad4b3e9ce4e6518193def1118d4073ca848635ee4ea2e96 \ + --hash=sha256:b5bdfd1c873d4e093aabc0ca84c4ca6dbc4f752afb5c86f146d9742580c9da2e \ + --hash=sha256:baed1e86cc735622097354b9d1281406caf42ff42a886d29faa8e8d1630333be \ + --hash=sha256:c1453022f490d2459a11819d83ad1d586e9ff65a12ac3e705ffebd46d3685dcf \ + --hash=sha256:c26608d2222fb1e94487e4a387d85f13eb55d5ed725cb25a0c589ac4ee60e7bc \ + --hash=sha256:c7659f22557c5a0bc4855cd635f55edec690cc008a40768527762cb9fb263455 \ + --hash=sha256:c8c69575568085ba0b1b10c0249d779a214aea6f6522e949a0fc9fb0fcb449d0 \ + --hash=sha256:c8d2c9fd1f2d16f780d15127abb050d13d1a76c03a4bd87d7e4980e45e511e12 \ + --hash=sha256:ca82be1a1d406ecfe1d25dc16cb33488e5a16bf4438c9fb590484ea29d92478b \ + --hash=sha256:cc572dace3f60ef98d7b12ff411d20f5362feb31a0439eab0085bbfd349982d7 \ + --hash=sha256:d18e5ac0f2f03f4f518d3e23db0f0cad7faa1da8620e9c09461d443bbf6e6692 \ + --hash=sha256:d28630f5854ab07ab1fd4aba756de52326c82e6be15d414b12793f1975048b54 \ + --hash=sha256:d9c275eaacd24aa73f94ffd6de08fc3f932424d8b6c376f4bed7cde376fe7bc3 \ + --hash=sha256:da0e573f9f97159390c89d9f1a9e41908b66d408cc5b58d08cf3847d844c531b \ + --hash=sha256:dd31f52ea1086513bb9df30f8fcee9b8918323ae067a3d5b78bc826a000712be \ + --hash=sha256:dddad92b554513a31f272570678ba307fb9f618f05e3d4a5eacafff9eae03e1d \ + --hash=sha256:df423d40ee8654634421812bc3b196da3f9bd7d32929da813f8394c4348a5358 \ + --hash=sha256:df913725b79db7bcf03448f36b7bf8815363417d5b58deecf9305e3e30f0f21a \ + --hash=sha256:e0bcb7e0f677f543555d2adff3bf19c05f66cdb4796e5ff602442ab2fe3c4ef7 \ + --hash=sha256:e2d65b31f36619cda3999b78b2aa9632e76b78448e7a56fc4240824200e7c4fc \ + --hash=sha256:e6e8cff14d6fb0be70a09c0bdc58096f501952d04624ebf867e0e56da2df8960 \ + --hash=sha256:f16c709686a78c727bbbf059f92b0bf41c6fc60deec706d2dc19f529175a6125 \ + --hash=sha256:f24fb43132a4c6b4cb4eb029492919b2db645be6808d738f244fd146c03c32cb \ + --hash=sha256:f53e442b08449d42821fa4a4fba000095af9f62742a500f978a9f557ec44339a \ + --hash=sha256:f5cfbc5fe74540d335175b656c725d74d90e3730c626d92575eea35029d9afaa \ + --hash=sha256:f81b3b8f3d4e343550fa4baa0e479bba9f2d29ce9c2e9b51d1ce1718d7442fcf \ + --hash=sha256:f8ec5e643a9a937f64e1999eb9f75d072263751912dc5cd06d3c85f8f44be7c3 \ + --hash=sha256:fb92203a88b3d3053034db775110081c49d28be6551923805e039924093761e4 \ + --hash=sha256:fcd22650c908d7b7da162bbfaab594a1227a15d1643a98c68b122ac642fa2264 +charset-normalizer==3.4.9 \ + --hash=sha256:0327fcd59a935777d83410750c50600ee9571af2846f71ce40f25b13da1ef380 \ + --hash=sha256:03d07803992c6c7bbc976327f34b18b6160327fc81cb82c9d504720ac0be3b62 \ + --hash=sha256:04ce310cb89c15df659582aee80a0603788732a5e017d5bd5c81158106ce249c \ + --hash=sha256:0d861473f743244d349b50f850d10eb87aeb22bbdcc8e64f79273c94af5a8226 \ + --hash=sha256:0e94703ec9684807f20cfb5eed95c70f67f2a8f21ad620146d7b5a13677b93e5 \ + --hash=sha256:0fa1aec2d32bcc03c8fa0f6f1712caad1adc38509f31142112e5c9daf5b9c833 \ + --hash=sha256:16b65ea0f2465b6fb52aa22de5eca612aa964ddfec00a912e26f4656cbef890b \ + --hash=sha256:16d10d789dd9bcca1173c95af82c58433122564b7bc39385124be735a35cbe99 \ + --hash=sha256:19ac87f93086ce37b86e098888555c4b4bc48102279bae3350098c0ed664b501 \ + --hash=sha256:1d22856ffbe153a602df38e4a5464f0b748a54002e0d69ac6d2ad0a197cc99ec \ + --hash=sha256:21e764fd1e70b6a3e205a0e46f3051701f98a8cb3fad66eeb80e48bb502f8698 \ + --hash=sha256:231ddcbb35e2ff8973e1365db41fe0572662893b99a05deb183b68ad4c0c8bd4 \ + --hash=sha256:253a4a220747e8b5faf57ec320c4f5efb0cef05f647420bf267143ec15dba10a \ + --hash=sha256:280081916dc341820640489a66e4696049401ef1cf6dd672f672e70ad915aca3 \ + --hash=sha256:2a441ea71902098ffe78c5abe6c494f44160b4af614ed16c3d9a3b1d17fd8ee2 \ + --hash=sha256:304b13570067b2547562e308af560b3963857b1fa90bd6afd978130130fe2d6a \ + --hash=sha256:32286a2c8d167e897177b673176c1e3e00d4057caf5d2b64eef9a3666b03018e \ + --hash=sha256:33bdcc2a32c0a0e861f60841a512c8acc658c87c2ac59d89e3a46dacf7d866e4 \ + --hash=sha256:375b83ed0aecfce76c16d198fbc21f3b11b337d68662bea0a995046682a11419 \ + --hash=sha256:3c09a49d6cde137258beb3d551994a2927fd35ad5cf96aed573f61bbd67c5f84 \ + --hash=sha256:3d92613ec25e43b05f042302531ec0f00b8445190e43325880cbd6ab7c2581da \ + --hash=sha256:40a126142a56b2dfc0aacbad1de8310cbf60da7656db0e6b16eebd48e3e93519 \ + --hash=sha256:416c229f77e5ea25b3dfd4b582f8d73d7e43c22320302b9ab128a2d3a0b38efe \ + --hash=sha256:432786d3561e69aeeae6c7e8648964ce0ad05736120135601f87ac26b9c83381 \ + --hash=sha256:43b9e366a31fdd1c87d0eb08f579b4a82b723ea54338f040d6b4e518a026ea29 \ + --hash=sha256:440eede837960000d74978f0eba527be106b5b9aee0daf779d395276ed0b0614 \ + --hash=sha256:45b0cc4e3556cd875e09102988d1ab8356c998b596c9fced84547c8138b487a0 \ + --hash=sha256:476743fe6dfe14a2da12e3ac79125dc84a3b2cf8094369a47a1529b0cd8549fe \ + --hash=sha256:4773092f8019072343a7447203308b176e10199920eb02d6195e81bbb3274c29 \ + --hash=sha256:4b3dac63058cc36820b0dd072f89898604e2d39686fe05321729d00d8ac185a0 \ + --hash=sha256:4d1c96a7a18b9690a4d46df09e3e3382406ae3213727cd1019ebade1c4a81917 \ + --hash=sha256:51307f5c71007673a2bf8232ad973483d281e74cb99c8c5a990af1eefa6277d9 \ + --hash=sha256:51447e9aa2684679af07ca5021c3db526e0284347ebf4ffcec1154c3350cfe32 \ + --hash=sha256:58150c9f9b9a552505912d182ccdf26f6396fb6094816ceebcbb20eecabaed94 \ + --hash=sha256:5b10cd92fc5c498b35a8635df6d5a100207f88b63a4dc1de7ef9a548e1e2cd63 \ + --hash=sha256:5e226f6218febc71f6c1fc2fafb91c226f75bdc1d8fb12d66823716e891608fd \ + --hash=sha256:609b3ba8fcc0fb5ab7af00719d0fb6ad0cb518e48e7712d12fd68f1327951198 \ + --hash=sha256:60f44ade2cf573dad7a277e6f8ca9a51a21dda572b13bd7d8539bb3cd5dbedde \ + --hash=sha256:611057cc5d5c0afc743ba8be6bd828c17e0aaa8643f9d0a9b9bb7dea80eb8012 \ + --hash=sha256:6366a16e1a25018694d6a5d784d09b046edc9eac40ea2b54065c3052672516a1 \ + --hash=sha256:65a7ff3f705e57d392f7261b6d0550fe137c3019477431f1c355e0db0a7d3e15 \ + --hash=sha256:673611bbd43f0810bec0b0f028ddeaaa501190339cac411f347ac76917c3ae7b \ + --hash=sha256:67830fc78e67501f47bb950471b2dcb9b35b140084429318e862895a8e89c993 \ + --hash=sha256:68ce9f4d6b26d5ccbf7fd4459bf75f74a0a146677ebba80597df60cbdb20e6f4 \ + --hash=sha256:68e5f26a1ad57ded6d1cfb85331d1c1a195314756471d97758c48498bb4dcdf5 \ + --hash=sha256:69b157c5d3292bcd443faca052f3096f637f1e074b98212a933c074ae23dc3b8 \ + --hash=sha256:75286256590a6320cf106a0d28970d3560aad9ee09aa7b34fb40524792436d35 \ + --hash=sha256:78841cccf1af7b40f6f716338d50c0902dbe88d9f800b3c973b7a9a0a693a642 \ + --hash=sha256:78fa18e436a1a0e58dbd7e02fc4473f3f32cceb12df9dfca542d075961c307d2 \ + --hash=sha256:79580094b00d1789d1f93ea55bc43cb2f611910c72235b7657f3482ddcc1b22d \ + --hash=sha256:7b86a2b16095d250c6f58b3d9b2eee6f4147754344f3dab0922f7c9bf7d226c9 \ + --hash=sha256:83aed2c10721ddd90f68140685391b50811a880af20654c59af6b6c66c40513c \ + --hash=sha256:84fd18bcc17526fc2b3c1af7d2b9217d32c9c04448c16ec693b9b4f1985c3d33 \ + --hash=sha256:871ff67ea1aad4dfd91736464934d56b32dac49f9fbe16cddba36198a7b3a0db \ + --hash=sha256:898f0e9068ca27d37f8e83a5b962821df851532e6c4a7d615c1c033f9da6eedf \ + --hash=sha256:8a79d9f4d8001473a30c163556b3c3bfebec837495a412dde78b51672f6134f9 \ + --hash=sha256:8c041122946b7ba21bb32c45b1aa57b1be35527690aeb3c5c234521085632eee \ + --hash=sha256:90c44bc373b7687f6948b693cceaea1348ae0975d7474746559494468e3c1d84 \ + --hash=sha256:9104ed0bd76a429d46f9ec0dbc9b08ad1d2dcdf2b00a5a0daa1c145329b35b44 \ + --hash=sha256:920079c3f7456fa213e0829ed2073aaa727fd39d889ead5b4f35d0de5460d04f \ + --hash=sha256:93d59d504b230e83c7a843251681959a0b6a9cd76f6e146ce1b8a80eb8739af9 \ + --hash=sha256:9b2aff1c7b3884512b9512c3eaadd9bab39fb45042ffaaa1dd08ff2b9f8109d9 \ + --hash=sha256:9b8e0f3107e2200b76f6054de99016eac3ee6762713587b36baaa7e4bd2ae177 \ + --hash=sha256:9bb41182d93ea91f60b4bc8fbf4c820c69ef8a12ab2d917f3f1834f1acad07e8 \ + --hash=sha256:9cdef90ae47919cae358d8ab15797a800ed41da7aba5d72419fb510729e2ed4b \ + --hash=sha256:a1786910334ed46ab1dd73222f2cd1e05c2c3bb39f6dddb4f8b36fc382058a39 \ + --hash=sha256:a4cfde78a9f2880208d16a93b795726a3017d5977e08d1e162a7a31322479c41 \ + --hash=sha256:a4fbdde9dd4a9ce5fd52c2b3a347bb50cc89483ef783f1cb00d408c13f7a96c0 \ + --hash=sha256:aa99adc8f081b475a12843953db36831eaf83ec33eb46a90629ca6a5de45a616 \ + --hash=sha256:ac351b3b8014eead140e77e9717e2992c6bbe30b63bc3422422eb84865412e3d \ + --hash=sha256:ad41ba96094304aa090f5a30cb6e4fb3b3f1c264c523394b4c39bbacc4dc92ba \ + --hash=sha256:b5314963fce9b0b12743891de876e724997864ee22aa496f903f426c7e2fa5b2 \ + --hash=sha256:bcf74c1df76758a395bf0af608c04c82257523f55c9868b334f06270d0f2112b \ + --hash=sha256:bd47ba7fc3ca94896759ea0109775132d3e7ab921fbf54038e1bab2e46c313c9 \ + --hash=sha256:c0323c9daef75ef2e5083624b4585018a0c9d5e3b40f607eed81a311270b934b \ + --hash=sha256:c1225416b463483160e4af85d5fc3a9690ccb53fd4b1865a6437825f5ede3209 \ + --hash=sha256:c1c948747b03be832dceed96ca815cef7360de9aa19d37c730f8e3f6101aca48 \ + --hash=sha256:c25fe15c70c59eb7c5ce8c06a1f3fa1da0ecc5ea1e7a5922c40fd2fa9b0d5046 \ + --hash=sha256:cc1b0fff8ead343dae06305f954eb8468ba0ec1a97881f42489d198e4ce3c632 \ + --hash=sha256:cd6280cf040f233bd7d3407b743b4b4c74f70e8e1c4199cb112a62c941c0772a \ + --hash=sha256:cd6c3d4b783c556fa00bf540854e42f135e2f256abd29669fcd0da0f2dec79c2 \ + --hash=sha256:d4d6fcde76f94f5cb9e43e9e9a61f16dacefd228cbbf6f1a09bd9b219a92f1a1 \ + --hash=sha256:ddf4af30b417d9fe16481e9b81c27ab2a7cde1ff7ba3e85653b02db7d145dc7b \ + --hash=sha256:df115d4d83168fdf2cae48ef1ff6d1cb4c466364e30861b37121de0f3bf1b990 \ + --hash=sha256:df7276909358e5635ae203673ab7e509ddd224225a8d6b0790bf13eb2bde1cc5 \ + --hash=sha256:e4fd89cc178bced6ad29cb3e6dd4aa63fa5017c3524dbd0b25998fb64a87cc8b \ + --hash=sha256:e9701d0049d92c16703a42771b98d560b95248949f23f8cf7b4eddd201814fb9 \ + --hash=sha256:ee2f2a527e3c1a6e6411eb4209642e138b544a2d72fe5d0d76daf77b24063534 \ + --hash=sha256:f7fb7d750cfa0a070d2c24e831fd3481019a60dd317ea2b39acbcebc08b6ed81 \ + --hash=sha256:f840ed6d8ecba8255df8c42b87fadeda98ddfc6eeec05e2dc66e26d46dd6f58a \ + --hash=sha256:f86c6358749bd4fda175388691e3ba8c46e24c5347d0afd20f9b7edfc9faf07d \ + --hash=sha256:fa36ec09ef71d158186bc79e359ff5fdd6e7996fe8ab638f00d6b93139ba4fcf \ + --hash=sha256:fe2c7201c642b7c308f1675355ad7ff7b66acfe3541625efe5a3ad38f29d6115 +click==8.4.2 \ + --hash=sha256:9a6cea6e60b17ebe0a44c5cc636d94f09bd66142c1cd7d8b4cd731c4917a15f6 \ + --hash=sha256:e6f9f66136c816745b9d65817da91d61d957fb16e02e4dcd0552553c5a197b76 +colorama==0.4.6 ; sys_platform == 'win32' \ + --hash=sha256:08695f5cb7ed6e0531a20572697297273c47b8cae5a63ffc6d6ed5c201be6e44 \ + --hash=sha256:4f1d9991f5acc0ca119f9d443620b77f9d6b33703e51011c16baf57afb285fc6 +cryptography==50.0.0 \ + --hash=sha256:031e2d5dd4bb9caa3ca9c82e5a197fd8ae680232cee62603d1a813f3f07e3d03 \ + --hash=sha256:06a32a980526a6ab9a4b9bf8f7385800791e2bb960903cb6b530e4817509a3b7 \ + --hash=sha256:07479a1cb08219ab719147e742e76090c9c773321959bb94946fffdd397a6437 \ + --hash=sha256:07949c449a1abcf60d1ee6e88956d89404c7df3c8258f46589e912988e551987 \ + --hash=sha256:105110f43a471dbd0060b9c9516cb8a6a79233631a04cc2ba16f28323ac6e025 \ + --hash=sha256:11b74db56cdbe3cdee6e3f6982ecb70334fa10dce99ed58bf7894aaaa3b2a037 \ + --hash=sha256:12b9c6996425c76ea6c457ace4f3073e715b8c545add07cd1a8f3a4f90691269 \ + --hash=sha256:1489e263a8048bb8b6a8bac662eb2d402ea5d2b7b4699b72f385f1e2772db105 \ + --hash=sha256:19736989797678c6af1e55cd49055cdbcb55d8f6b5583ac5335f933aba9101dc \ + --hash=sha256:1b4a266766514614f8aa60416e71f2fc6e575d36e7bdc90f644fadb2f4b75b95 \ + --hash=sha256:2a8183b489dc1f7f80f135780fadc1108f14b31b8a40411c7a5b17425f65f28b \ + --hash=sha256:37fdb0d0111f1e2ff07139dfb79f1b49531f8e213c46f1163dd7642979b58c47 \ + --hash=sha256:3f5735ffe4996d28b809371756219f5354864902a3b9e7c0b9ee87041209fc9c \ + --hash=sha256:49e7d93abdbd2990caced757e5fade25302f719c3c8fb6e6fff2dde98999fc41 \ + --hash=sha256:5e34edd123674534acd70147f0ca331eaa2c74e6325fb2028c886aa26ba0b68c \ + --hash=sha256:62598a8a57f815db4c6259a4e97d857dab56697e7de8e8ab02352ab74da1995d \ + --hash=sha256:65c2c3add92b45fd0709db8594536aea39c2a67af0e27ffcf049c498501140b7 \ + --hash=sha256:6ba6a53445bd3cfa809ef3ef5f1589aa6ba08784a1d962bf47d0940e871dab1c \ + --hash=sha256:6e7d61120573a7f2cd94cc095f9e81f6967c61ccdf194285aa143ecec8e0b708 \ + --hash=sha256:7cec5b856506da6defb290f30c9ee687d5f5e8cb0bd3f6459dde43b0b4fa40ef \ + --hash=sha256:80b63928fa35083b33966ce1efb70e5b9607181e49dcd1c22c8c005e319f667f \ + --hash=sha256:82148ec5bddac30b51a5b3c1945075f896fa022cb93f8e4a01e9f6ee95292c5f \ + --hash=sha256:828743d939e9629bc267b8e2d08d8bb67cd4319c771a33d4b18b22dd8fb7440a \ + --hash=sha256:8d89f3976b10b4ce31118de72329025f70d2c6ead14a8217c5514dd2c6d5a78f \ + --hash=sha256:8eb5e1172eb569ea8a872796576e6a67c276351728b6455d5beb01242b027c6a \ + --hash=sha256:900131fafd8aead39ac7dd3a7e833be754c17a95cfd91221636949fe4eb0aa8a \ + --hash=sha256:910d11e1a385c654bf738bf3e6b8e6ed5de0f5610fcae2be9e5b398d8081d20e \ + --hash=sha256:910e1d2668e7de9648f2bcee30e180db2a6b15c30f887d7c4c93ddf96e3992e3 \ + --hash=sha256:9aa87839c383bdbab6ef865787a1fb877af8dd03464c4400322726feaaadfc6d \ + --hash=sha256:a1b30560f2acc95aa8b2e06e716a13dbfc97314747b80d9707e307f77b40d6b3 \ + --hash=sha256:a91296cb61e8df6f86d0c19cc4068228da256bf59bf86049fbd821084565327f \ + --hash=sha256:b42a28c1844fd9de8f3f7d540e36b66f3a9c83fceac7170ebc7a6a19edd9dcae \ + --hash=sha256:bd1c592e4d5974f0d08d4888e432157adba757c66da0246918e43677fafa2d30 \ + --hash=sha256:c87f62a3d3b9888ed0fdde100ec06aa61ca9cd44bad9057d1dff9a516b5f5bb9 \ + --hash=sha256:c99c003e088647b8a5b7c145d6f78c335f6348332b62e142d411c4b63d1460b9 \ + --hash=sha256:ccdc4a71a4dabae05de219404f9f4abc38e3b58422177ff93d0da05967dafa07 \ + --hash=sha256:d24fead1d4d076e1bfb006dcec392074a3cd8d7b4fc8a595aa64073b2b7a96ba \ + --hash=sha256:d58c3db7cd6eed54e6c06744db55456b65ebd7492ddeae9c1e93cfca7aa857d3 \ + --hash=sha256:d764dcf130c428ef66786f866dd750f53182bc608813489915e9fc106bb0c82f \ + --hash=sha256:df2a58a472f332225671c35b0a830208b86d004f82baa8530fa3782c85646533 \ + --hash=sha256:e722f16708d854fe924790e051061f6704a472c3bac347b6fd88033ea8dd0dc5 \ + --hash=sha256:ecfed7367f965a0328cfbdd70da860f15441f002f613185668c6e6ebf5a0ac11 \ + --hash=sha256:eeac2acb5a20ed25e0ad6d1df9891a520b78b404266b6d11778f25d5d691a6c9 \ + --hash=sha256:f59e38625469987d7ef6d495323c55e7db6c212eaf6112267e0d3b565a2e9c9f \ + --hash=sha256:f89831ef99dd7dd169ab06d63a831adb9e20a87aac6d380266bbda5823349169 \ + --hash=sha256:fd9192b7b70c573d7f214eb1ae35e00d359f6f5e4b27c7e21e30de1fc6204645 +distro==1.9.0 \ + --hash=sha256:2fa77c6fd8940f116ee1d6b94a2f90b13b5ea8d019b98bc8bafdcabcdd9bdbed \ + --hash=sha256:7bffd925d65168f85027d8da9af6bddab658135b840670a223589bc0c8ef02b2 +dnspython==2.8.0 \ + --hash=sha256:01d9bbc4a2d76bf0db7c1f729812ded6d912bd318d3b1cf81d30c0f845dbf3af \ + --hash=sha256:181d3c6996452cb1189c4046c61599b84a5a86e099562ffde77d26984ff26d0f +docstring-parser==0.18.0 \ + --hash=sha256:292510982205c12b1248696f44959db3cdd1740237a968ea1e2e7a900eeb2015 \ + --hash=sha256:b3fcbed555c47d8479be0796ef7e19c2670d428d72e96da63f3a40122860374b +email-validator==2.3.0 \ + --hash=sha256:80f13f623413e6b197ae73bb10bf4eb0908faf509ad8362c5edeb0be7fd450b4 \ + --hash=sha256:9fc05c37f2f6cf439ff414f8fc46d917929974a82244c20eb10231ba60c54426 +et-xmlfile==2.0.0 \ + --hash=sha256:7a91720bc756843502c3b7504c77b8fe44217c85c537d85037f0f536151b2caa \ + --hash=sha256:dab3f4764309081ce75662649be815c4c9081e88f0837825f90fd28317d4da54 +fastapi==0.141.1 \ + --hash=sha256:bfb91aa2d334c61cb35ba9a116fc123b3d3df31640b801cf57a7a78ec3f603b3 \ + --hash=sha256:e8822fc40db1e1858054d7a949a888695bc9bdce70139178e33bd2871a453ca1 +h11==0.16.0 \ + --hash=sha256:4e35b956cf45792e4caa5885e69fba00bdbc6ffafbfa020300e549b208ee5ff1 \ + --hash=sha256:63cf8bbe7522de3bf65932fda1d9c2772064ffb3dae62d55932da54b31cb6c86 +httpcore==1.0.9 \ + --hash=sha256:2d400746a40668fc9dec9810239072b40b4484b640a8c38fd654a024c7a1bf55 \ + --hash=sha256:6e34463af53fd2ab5d807f399a9b45ea31c3dfa2276f15a2c3f00afff6e176e8 +httptools==0.8.0 \ + --hash=sha256:0770728beb05094c809b98e814edff5fef69d26ad7d21185f2f6d5884a0ba683 \ + --hash=sha256:0ea897f0c729581ebf72131a438a7932d9b14efef72d75ada966700cac3caaeb \ + --hash=sha256:159e9ab5f701ccd42e555a12f1ad8ff69702910fc1c996cf2bb66e5fcb7a231b \ + --hash=sha256:19d1ee275bb59ba2643ba9a3a1e51cc0c788caf2b8df506368e03f56fdd08527 \ + --hash=sha256:20b4aac66ff65f7db06a375808b78f42a94970aa22e826b3cb2b43eb09174124 \ + --hash=sha256:2a021c3a8e65cc125390d72f59b968afca3bdcaff25bd67965e0a055a14946ca \ + --hash=sha256:2c032fa028f46871ec7e1fc59fc15e8023eab3e6bbe6ece786a1611719a5d081 \ + --hash=sha256:2d689918c15a013c65ef52d9fd495d766893ab831a2c8d89f2ac5940a5df847c \ + --hash=sha256:384c17174464c8e873398b7af24f0b1f44d992c820328413951a625323155d77 \ + --hash=sha256:425f83884fd6343828d8c565f046cb72b6d19063f6924093e11bcd8e1548cd09 \ + --hash=sha256:48774d39cbb70e2b1f71f88852a3087ae1d3a1eb80482bb48c13067ab080c14f \ + --hash=sha256:52dd695b865fe96d9d2b16b64a895f3f57bf3cb064e8383cd3b5713a069e8085 \ + --hash=sha256:57278e6fa0424c42a8a3e454828ab4f0aff27b40cddf9679579b98c6dce6a376 \ + --hash=sha256:5931891fb7b441b8a3853cf1b85c82c903defce084dd5f6771ca46e31bf862c5 \ + --hash=sha256:5d7fa4ba7292c1139c0526f0b5aad507c6263c948206ea1b1cbca015c8af1b62 \ + --hash=sha256:5eb911c515b96ee44bbd861e42cbefc488681d450545b1d02127f6136e3a86f5 \ + --hash=sha256:614ceea8ea606848bece2338ac03b3ce5324bcb4be8dc7d377ed708012fa4db8 \ + --hash=sha256:6a43c9dd399758ccc0531acb0a3c4a6c299ee893ee9400e9c893b7bdcfae0681 \ + --hash=sha256:6b2a32f18d97e16e90827d7a819ffa8dbd8cc245fc4e1fa9d1095b54ef4bd999 \ + --hash=sha256:7685df791fad561384bfb139e77fde27a1ffd93134e016f95a0db424ffbf77b1 \ + --hash=sha256:7b71e7d7031928c650e1006e6c03e911bf967f7c69c011d37d541c3e7bf55005 \ + --hash=sha256:880490234c10f70a9830743097e8958d6e4b9f5a0ffc24515023afeef984054d \ + --hash=sha256:88bdd940f2b5d487b4d032c6afa5489a7dc4694410d43de3c38c4fb3af0dc45d \ + --hash=sha256:88eead8ec8680a9f146c655bc88445a325bd7921cfd8194c7337e9467282427d \ + --hash=sha256:9518c406d7b310f05adb1a37f80acabac40504a575d7c0da6d3e365c695ac20d \ + --hash=sha256:9878eb2785ba5eb70631ad269b37976f73d647955e26c91d490eb8a4edfda4ba \ + --hash=sha256:9fc1644f415372cec4f8a5be3a64183737398f10dbb1263602a036427fe75247 \ + --hash=sha256:a1afd7c9fbff0d9f5d489c4ce2768bd09c84a46ddefc7161e6aa82ae35c85745 \ + --hash=sha256:a1b4c8e7a489a0d750d91894e9a8cdc295838f1924c0ca903ae993456fddec07 \ + --hash=sha256:a3b7387147361c3fd47a0bde763c5c91b5b4cd4dc9989b8ece84ff436c99843b \ + --hash=sha256:a6f21e2a3b0067bbe7f67e34cfd16276af556e5e52f4c7503be0cb5f90e905e4 \ + --hash=sha256:b15fc622b0f869d19207c4089a501d9bcc63ca5e071ffdd2f03f922df882dcb2 \ + --hash=sha256:b205e5f5523fa039679da0dfe5a10132b2a4abeae6a86fdd1ddc035f7f836557 \ + --hash=sha256:bbb8caadb2b742d293169d2b458b5c001ef70e3158704aa3d3ef9597624c5d1d \ + --hash=sha256:bf3b6f807c8541503cecfbb8a8dffb385640d0d96102f3d112aa8740f9b7c826 \ + --hash=sha256:c08ffe3e79756e0963cbc8fe410139f38a5884874b6f2e17761bef6563fdcd9b \ + --hash=sha256:c0d726cc107fceb7d45f978483b4b70dd8caa836f5914d3434bb18628eb73813 \ + --hash=sha256:c4a9f1707e4823d54dfec6c33fa3697d302aed536ed352a7ebb5a061ddb869d0 \ + --hash=sha256:cd96f29b4bab1d42fa6e3d008711c75e0f79e94e06827330160e3a304227f150 \ + --hash=sha256:d76ad7b951387e3632c8716a9bb03ac5b45c5f16119aa409db0459520887944e \ + --hash=sha256:da684f2e1aa2ee9bdcb083f3f3a68c5956750b375bc5df864d3a5f0c42a40b77 \ + --hash=sha256:de1ed58a974e75d56560acc7e7fed01a454994429456f65209789992e41f2568 \ + --hash=sha256:de242a49b5d18e0a8776e654e9f6bf6d89f3875a5c35b425a0e7ce940feb3fd6 \ + --hash=sha256:df31ef5494f406ab6cf827b7e64a22841c6e2d654100e6a116ea15b46d02d5e8 \ + --hash=sha256:e93c227b595c6926c1acee96891dd9da4be338cfbe82e5cd3bb9d8dd7dc4ac0b \ + --hash=sha256:eb3028cca2fc0a6d720e52ef61d8ebb62fcbfeb1de56874546d858d3f25a26b7 \ + --hash=sha256:ed377e64805bdba4943c82717333f8f8603a13b09aff9cead2717c6c817fb168 \ + --hash=sha256:ef7c3c97f4311c7be57e2986629df89d49cb434dbff78eafcd48c2bff986b15a \ + --hash=sha256:f256d6ce930c52ca1cb2a960b7da03548c454e7d28b06059ad41bfe789036ce0 \ + --hash=sha256:fe2a4c95aeba2209434e7b31172da572846cae8ca0bf1e7013e61b99fbbf5e72 +httpx==0.28.1 \ + --hash=sha256:75e98c5f16b0f35b567856f597f06ff2270a374470a5c2392242528e3e3e42fc \ + --hash=sha256:d909fcccc110f8c7faf814ca82a9a4d816bc5a6dbfea25d6591d6985b8ba59ad +idna==3.18 \ + --hash=sha256:7f952cbe720b688055e3f87de14f5c3e5fdaa8bc3928985c4077ca689de849a2 \ + --hash=sha256:ffb385a7e039654cef1ab9ef32c6fafe283c0c0467bba1d9029738ce4a14a848 +iniconfig==2.3.0 \ + --hash=sha256:c76315c77db068650d49c5b56314774a7804df16fee4402c1f19d6d15d8c4730 \ + --hash=sha256:f631c04d2c48c52b84d0d0549c99ff3859c98df65b3101406327ecc7d53fbf12 +jaraco-classes==3.4.0 \ + --hash=sha256:47a024b51d0239c0dd8c8540c6c7f484be3b8fcf0b2d85c13825780d3b3f3acd \ + --hash=sha256:f662826b6bed8cace05e7ff873ce0f9283b5c924470fe664fff1c2f00f581790 +jaraco-context==6.1.2 \ + --hash=sha256:bf8150b79a2d5d91ae48629d8b427a8f7ba0e1097dd6202a9059f29a36379535 \ + --hash=sha256:f1a6c9d391e661cc5b8d39861ff077a7dc24dc23833ccee564b234b81c82dfe3 +jaraco-functools==4.6.0 \ + --hash=sha256:880c577ec9720b3a052d5bc611fb9f2269b3d87902ef42440df443b88e443280 \ + --hash=sha256:99e3dc0060c5cbe8fcd1cdb36258e2a65ca40f1566b2033b12abb1bb44dd3c30 +jeepney==0.9.0 ; sys_platform == 'linux' \ + --hash=sha256:97e5714520c16fc0a45695e5365a2e11b81ea79bba796e26f9f1d178cb182683 \ + --hash=sha256:cf0e9e845622b81e4a28df94c40345400256ec608d0e55bb8a3feaa9163f5732 +jiter==0.16.0 \ + --hash=sha256:0542a7189c26920778658fc8fcf2af8bae05bae9924577f71804acef37996536 \ + --hash=sha256:0758ab7747a984797cf048e8eedea1d8ef39d7994b25611daf5b48fc903e8873 \ + --hash=sha256:0df91907609837f33341b8e6fe73b95991fdaa57caf1a0fbd343dffe826f386f \ + --hash=sha256:0e2e9efbe042210df657bade597f66d6d75723e3d8f45a12ea6d8167ff8bbce3 \ + --hash=sha256:0fa25b09b13075c46f5bc174f2690525a925a4fc2f7c82969a2bbabff22386ce \ + --hash=sha256:10438939205546132189c8e74a2d536a707841f3a25cd7c74ee91fe503407a26 \ + --hash=sha256:12288303c9844e61e1651d02a9a6f6633e47d39f897d6991d1427161ce6b746e \ + --hash=sha256:17ca7fae79f6d99cd9a042b75f917eaada7b895cfc7dd2ee3a16089dcaec7a85 \ + --hash=sha256:1878349266f8ee36ecb1375cc5ba2f115f35fd9f0a1a4119e725e379126647f7 \ + --hash=sha256:1a0f05e229edb29e68cdd0ccb83cea13b64263416120cf943767a6fd72e6787f \ + --hash=sha256:2577196f4474ef3fc4779a088a23b0897bbf86f9ea3679c372d45b8383b43207 \ + --hash=sha256:26798522707abb47d767db536e4148ceac1b14446bf028ee85e579a2e043cfe5 \ + --hash=sha256:26d122613ada2b708eb714695446f40fce5bdf2edb4b02116dec62faa62dfab3 \ + --hash=sha256:2a0d46ef67cc58d906a6132dd3040ca70ae4f0b0d7c9c052fe432c658a69b3f6 \ + --hash=sha256:2a8e9e39cf083016137aa5cadafe3188adc2ba6ba1fbf1e5d18889ad3e9ad056 \ + --hash=sha256:2c842cbf374a8daf50b2c04212995bee34ca2ac2cdc29a901b4cdb072c9c4131 \ + --hash=sha256:2ed5738ae4af18271a51a528b8811b0cbfa4a1858de9d83359e4169855d6a331 \ + --hash=sha256:3126a5dbad56401989ac769aca0cb56005bfb3e2366eea0ca99d1a91c3c1ee03 \ + --hash=sha256:31aaee5b80f672c1dc21272bcfb9cbdcfc1ea04ff50f00ed5af500b80c44fa93 \ + --hash=sha256:3a23f0e4f957e1be65752d2dfac9a5a06b1917af8dc85deb639c3b9d02e31290 \ + --hash=sha256:3b21b412b899fd8bd51a3046934b59a3bb068b79f70a5c6010053ac77cc53f0c \ + --hash=sha256:3c1fd2dbe1b0af19e987f03fe66c5f5bd105a2229c1aff4ab14890b24f41d21a \ + --hash=sha256:3f4d9e473a5ce7d27fef8b848df4dc16e283893d3f53b4a585e72c9595f3c284 \ + --hash=sha256:3fb7bc819187b56dc48aa5c833aaf92257da8e07efdb9306156667bd2eeb491c \ + --hash=sha256:41977aa5654023948c2dae2a81cbf9c43343954bef1cd59a154dd15a4d84c195 \ + --hash=sha256:44d019fa8cdaf89bf29c71b39e3712143fdd0ac76725c6ef954f9957a5ea8730 \ + --hash=sha256:46add52f4ad47a08bfb1219f3e673da972191489a33016edefdb5ea55bfa8c48 \ + --hash=sha256:49060fd70737fad59d33ba9dcc0d83247dc9e77187de26053a19c16c9f32bd69 \ + --hash=sha256:491e7d072a253b156fff46b78bceac4652a697aa8d7082c9c18c03d7b7917d24 \ + --hash=sha256:4a1d68ff7ca1d3b5dee20a97a3decda7d5f15003823bf6d140c81f8561d3bc5c \ + --hash=sha256:51d7b836acb0108d7c77df1742332cac2a1fa04a74d6dacec46e7091f0e91274 \ + --hash=sha256:560b2cf3fb03240cd34f27409a238547488708f05b7c3924f571a60422251ec7 \ + --hash=sha256:561926e0573ffe4a32498420a76d64b16c513e1ab413b9d28158a8764ac701e5 \ + --hash=sha256:57b37fc887a32d44798e4d8ebfa7c9683ff3da1d5bf38f08d1bb3573ccb39106 \ + --hash=sha256:5a71d0d2014c3275043e1170bf3d4e771493cb0dcf07be54c567155f4d8ee64b \ + --hash=sha256:5ab4f50ff971b611d656554ea10b75f80097392c827bc32923c6eeb6386c8b00 \ + --hash=sha256:5af7780e4a26bd7d0d989592bf9ef12ebf806b74ab709223ecca37c749872ea9 \ + --hash=sha256:5cf109d010b4b05a105afb3d43be36a21322d345ad3111e13d15f680afef0e5b \ + --hash=sha256:5ed466aee31294d7cdcd4d37dfe5c42c97bc29d9a5f00eacf24504358309cb9b \ + --hash=sha256:616e89e008a93c01104161c75b4988e58716b01d62307ebfe161e52a56d2a818 \ + --hash=sha256:62c1b7fe1f77925acf5af68b6140b8810fa87dfd4dc0a9c8568ec2fa2a10429c \ + --hash=sha256:63efadc657488f45db1c676d81e704cac2abf3fdb892def1faea61db053127e2 \ + --hash=sha256:63ffdbdae7d4499f4cda14eadc12ddcabef0fc0c081191bdc2247489cb698077 \ + --hash=sha256:64c0203212098470032aabcde9356fc168f377aade3e43def61dfe17e92f2037 \ + --hash=sha256:64d613743df53199b1aa256a7d328340da6d7078aac7705a7db9d7a791e9cfd2 \ + --hash=sha256:659039cc50b5addcc35fcc87ae2c1833b7c0a8e5326ef631a75e4478447bcf84 \ + --hash=sha256:6722bcef4ffc86c835574b1b2fac6b33b9fb4a889c781e67950e891591f3c55a \ + --hash=sha256:67c3bc1760f8c99d805dcab4e644027142a53b1d5d861f18780ebdbd5d40b72a \ + --hash=sha256:67fddeda1688f0cce2d2ae83ccf8a80f79936f2d2997d6cc2261f82fdb54a4d3 \ + --hash=sha256:6edb63a46e65a82c26800a868e49b2cac30dd5a4218b88d74bc2c848c8ad60bb \ + --hash=sha256:70a490b55634dc0d2606ce8a8e01b1d62459011beb368d15d76e1eaf62460e3d \ + --hash=sha256:710cc51d4ebdcd3c1f70b232c1db1ea1344a075770422bbd4bede5708335acbe \ + --hash=sha256:741eed508c233a76313a1c7b001f8f21b82f14327e9196ae8bd29a2cc164ae84 \ + --hash=sha256:7b24c3492c5f4f84a37946ad9cf504910cf6a782d6a4e0689b6673c5894b4a1c \ + --hash=sha256:7c9610fd25ebccb43fca584136f5c2fbb26802447eccd430dfdbab95a0fd5126 \ + --hash=sha256:850ccb1d7eedb4200f4014b1c0e8a577de114fc3cd88faad646dcc9bc4bb12ad \ + --hash=sha256:8597d23c87f59294f83bcb6229b9ed1fccee13dbba967b46930d2f1759466fee \ + --hash=sha256:8d031aabecc4f1b6276adfb42e3aabb77c89d468bf616600e8d3a11328929053 \ + --hash=sha256:8d30a4a1c87713060c8d1cc59a7b6c8fb6b8ef0a6900368014c76c87922a2929 \ + --hash=sha256:8fb8de1e23a0cb2a7f53c335049c7b72b6db41aa6227cdcc0972a1de5cb39450 \ + --hash=sha256:8fc4d94713c4697347e38faf7d6ef91547c142219bdcfc7220c4870879974244 \ + --hash=sha256:96e38eea538c8ddf853a35727c7be0741c76c13f04148ac5c116222f50ece3b3 \ + --hash=sha256:9acf1b2faec82d998811ecce7ae84d9005e53410773e9d37d61cdc424ba4581b \ + --hash=sha256:9c8a956fd72c2cf1e730d01ea080341f13aa0a97a4a33b51abebe725b7ae9ca9 \ + --hash=sha256:9ec553a99b0987efd7a3645a1a825cf29c224e494db267a83369fcc8da9aeda5 \ + --hash=sha256:a111256a7193bea0759267b10385e5870949c239ed7b6ddbaaf57573edb38734 \ + --hash=sha256:a32d2027a9fa67f109ff245a3252ece3ccc32cc56703e1deab6cc846a59e0585 \ + --hash=sha256:a5c54ef4ff776d9675837ef535b3308d6e31c208d43ebc44a0f7ab8a208c68f7 \ + --hash=sha256:adbb8edeadd431bc4477879d5d371ece7cb1334486584e0f252656dd7ffada29 \ + --hash=sha256:adff21bc78edfe086c15eb495b900306076de378dc2337c132401fc39bd79c91 \ + --hash=sha256:b1e7923093a376d93c6eb507c77045ae258d689ba577392846a1b3f10d0b09a9 \ + --hash=sha256:b2c61484666ad42726029af0c00ef4541f0f3b5cdc550221f56c2343208018ee \ + --hash=sha256:b42e9ff5376819c053da25809a8d4b6fa6e473b4856ebe42e298ac958be3d7f9 \ + --hash=sha256:b72d0b2990ca754a9102779ac98d8597b7cb31678958562214a007f909eab78e \ + --hash=sha256:b7dc85ea77d4abbae8bad0d3538678aedee75bceec4e2f6c8dfb1c74772e5aa5 \ + --hash=sha256:b8a3f9a6008048fe9def7bf465180564a6e458047d2ce499149cfbe73c3ae9db \ + --hash=sha256:baad945ed47f163ad833314f8e3288c396118934f94e7bbb9e243ce4b341a4fd \ + --hash=sha256:bae96332410f866e5900d809298b1ed82735932986c672495f9701daacd80620 \ + --hash=sha256:bc837c1b9631be10abfe0191537fe8009838204cec7e44827401ace390ddb567 \ + --hash=sha256:bf3ea07d9bc8e7d03a9fbc051295462e6dbc295b894fd72457c3136e3e43d898 \ + --hash=sha256:c22a488f7b9218e245a0025a9ba6b100e2e54700831cf4cf16833a27fba3ad01 \ + --hash=sha256:c4b4717bdb35ae456f831a6b08d01880fff399887a6bbc526a583a406e484eea \ + --hash=sha256:c5fc4f8def331036a7b8e981b4347ebe409981edbc8308a5ea842b8c3614fa6c \ + --hash=sha256:c682bea068a90b764577bdb78a60a4c1d1606daf9cd4c893832a37c7cc9d9026 \ + --hash=sha256:c90c0f63df322be920eda6ce622e3083d8906ba267f8220fe7873213b8b4430e \ + --hash=sha256:c9c53be232c2e206ef9cdbad81a48bfa74c3d3f08bcf8124630a8a748aad993e \ + --hash=sha256:cbd18dd5e2df96b580487b5745adf57ef64ad89ba2d9662fc3c19386acce7db8 \ + --hash=sha256:cf0d73f50e7b6935677854f6e8e31d499ca7064dd24734f703e060f5b237d883 \ + --hash=sha256:d284fb8d94d5855d60c44fefcab4bf966f1da6fada73992b01f6f0c9bc0c6702 \ + --hash=sha256:d28bb3c26762358dadf3e5bf0bccd29ae987d65e6988d2e6f49829c76b003c09 \ + --hash=sha256:d5bf78d0e05e45cfdd66558893938d59afe3d1b1a824a202039b20e607d25a72 \ + --hash=sha256:d5f91b1c27fc22a57993d5a5cb8a627cb8ed4b10502716fac1ffbfe1d19d84e8 \ + --hash=sha256:d8f80521644426d451e70f00c7974240cab8f6ee088aedaa9af2697153ab7805 \ + --hash=sha256:da3d7ec75dc83bb18bca888b5edfae0656a26849056c59e05a7728badd17e7af \ + --hash=sha256:dab907db06fc593645e73109acf4581ba5b548897d28b9348dc41ddc8343b2d3 \ + --hash=sha256:de5ba8763e56b793561f43bed197c9ea55776daa5e9a6b91eed68a909bc9cdbf \ + --hash=sha256:e03a5f21a5ce96a9441b8cb32719a8b88ed5388f53e0f339c5bcf54f1317f9d0 \ + --hash=sha256:e34e97bda77eb63242a410243c071e28ac7e0d8c0948c5ee658498690a4b2f2f \ + --hash=sha256:e431cfc9caf44c1d5459ff77d4e64cbf85fddb6a35dad836a15c6a9ec23087c1 \ + --hash=sha256:eab2cd170150e70153de16896a1774e3a1dca80154c56b54d7a812c479a7165e \ + --hash=sha256:ee6162b77d49a9939229df666dfa8af3e656b6701b54c4c84966d740e189264e \ + --hash=sha256:f17d61a28b4b3e0e3e2ba98490c70501403b4d196f78732439160e7fd3678127 \ + --hash=sha256:f3bd327cdfa118bc1ce69c214c2678571d5bd39b8ccd0ebf43a54db00541ba9a \ + --hash=sha256:f4444a83f946605990c98f625cdd3d2725bfb818158760c5748c653170a20e0e \ + --hash=sha256:fb08c276dd02dac3a284acdd02cacc630d2e3cd6572a4b85519f35cbd133c3de +jsonpatch==1.33 \ + --hash=sha256:0ae28c0cd062bbd8b8ecc26d7d164fbbea9652a1a3693f3b956c1eae5145dade \ + --hash=sha256:9fcd4009c41e6d12348b4a0ff2563ba56a2923a7dfee731d004e212e1ee5030c +jsonpointer==3.1.1 \ + --hash=sha256:0b801c7db33a904024f6004d526dcc53bbb8a4a0f4e32bfd10beadf60adf1900 \ + --hash=sha256:8ff8b95779d071ba472cf5bc913028df06031797532f08a7d5b602d8b2a488ca +keyring==25.7.0 \ + --hash=sha256:be4a0b195f149690c166e850609a477c532ddbfbaed96a404d4e43f8d5e2689f \ + --hash=sha256:fe01bd85eb3f8fb3dd0405defdeac9a5b4f6f0439edbb3149577f244a2e8245b +langchain==1.3.14 \ + --hash=sha256:1b6696c72ba3bbbce54d745e0180742c9f6ece8bbc59ed5a46c3e20b9a435929 \ + --hash=sha256:4d10dbe91005952cddd56d0dc77aa108964da6bae90ab20063653957e901f782 +langchain-anthropic==1.5.4 \ + --hash=sha256:113cb9bdac3169f2da65ea395dedb290e30dc6f899d01c205ba88aa88a2a02c2 \ + --hash=sha256:730a9cb1ad384c9f1642840469c1ebbf20237066b1635f5d4fa9876e365fceaf +langchain-core==1.5.3 \ + --hash=sha256:48b56fa580277209594dd7baf837f5b9a2a3651613f34ff9fb1728b429df015f \ + --hash=sha256:a56457ac444fef41e9404443c187f0ecea708d36e816ea4ba9573c027f7d1a2d +langchain-ollama==1.1.0 \ + --hash=sha256:43ac83a6eacb0f43855810739794dd55019e0d9b17bdcf3ecb3b1991ac3b59dd \ + --hash=sha256:f776f56f6782ae4da7692579b94a6575906118318d1023b455d7207f9d059811 +langchain-openai==1.4.2 \ + --hash=sha256:4ece4ba6ed662e903d7deb9bf864b22687a2b4934d84e4acde90eb446f2d5052 \ + --hash=sha256:88208c5bc32dd95d21129a0d6f0ac3f6af380eed46b9fa0944c06c18ea325a49 +langchain-protocol==0.0.18 \ + --hash=sha256:70b53a86fbf9cedc863555effe44da192ab02d556ddbf2cf95b8873adcf41b5a \ + --hash=sha256:ec3e11782f1ed0c9db38e5a9ed01b0e7a0d3fba406faa8aef6594b73c56a63e6 +langgraph==1.2.10 \ + --hash=sha256:05a183a746ed570a06c7c1b879920163509a75df9e44e92dd2238218d677fd37 \ + --hash=sha256:52c48bd42fa31a1de0e1c0f0ebfe342e11ca2957b8b3563f83dbd60d8e30f921 +langgraph-checkpoint==4.2.0 \ + --hash=sha256:0547fd228935a0b758865de3a3d6d7a2537c308895d0f9ab092ce9151b5da942 \ + --hash=sha256:51a593b6bee684b0818e5d6e58e28ab340c6db7794575056ce7bd1b746a84ed7 +langgraph-prebuilt==1.1.0 \ + --hash=sha256:3c579cf6eed2d17f9c157c2d0fcaddcd8688524e7022d3b22b37a3bf4589d528 \ + --hash=sha256:51e311747d755b751d5c6b39b0c1446124d3a7643d2515017e6714b323508fc9 +langgraph-sdk==0.4.2 \ + --hash=sha256:75fa5096c1177ce39c847096a8fe3745ffd480ddb412995f836e9f5f884c43dd \ + --hash=sha256:b88f0f5f6328ac0680d6790614a905b2bcfa257f2276dba4e38f0e86db0aa738 +langsmith==0.10.17 \ + --hash=sha256:2a1242a0500147ed846ecbd5fe2f41c21b1652f6833b1af0f209231006f67e5e \ + --hash=sha256:2bf0e69246ef77282ea93e78eb22dad1ec8d8a002d31cbd607d0d6448f9c6458 +librt==0.15.0 ; platform_python_implementation != 'PyPy' \ + --hash=sha256:04d5387b908676c0b8d5d2f5fb58373b4ea382d81f7a6f0fab8ea2a462bb4738 \ + --hash=sha256:077471b3182db4e17c36ae91555f36a4d2c00080b267f749bcad34a478a9a302 \ + --hash=sha256:0a15cb554761247d84a3ec0cbdf4078d70725384f0e4662c0fa3b26266eb60ad \ + --hash=sha256:0da0d94cb802f32a0524653e7201f2cef72d5f700a5407678f5290483d4fcd08 \ + --hash=sha256:0e2d0c0acf5b0ada7d045912b7cf787c21315c95b38b1fa939ef72d45d366b3d \ + --hash=sha256:0f0ee3644d951f31055ad07d77d92520e84505dd7a432cc4cd501dd70ee06785 \ + --hash=sha256:1172c6ad2a88b646e7fe3b480e3fac4ab4418b3443fd8a4061fdd531e0622fc7 \ + --hash=sha256:1256589e0b0adb31751d685a68bce29d73407ddf4ef05d4188f49d5dcf9566d9 \ + --hash=sha256:1a1a8cd430c7dd0c083f455cb1b328d7fc682b05c31b940906f7845bdff80881 \ + --hash=sha256:1cd3b721f24c206398b9e26da3c3a9c011e6e89d06f318ba8ebefc30f1003890 \ + --hash=sha256:1e47b8ba865d7ede071a91a7163073bbaeb72541f1ef8a07d512c45c7b5007f2 \ + --hash=sha256:1f4ef2e71db33df4309167ed7f1520c4fae5e611226e159fa9cf33f93e6ddb3d \ + --hash=sha256:2067ff438048cead9d223ca5675bae2a25e520a7c3e6c1498bf9c6892d22caab \ + --hash=sha256:22d6263b9d39d7bbb286fa791945646e3218f1be2d693e36fb630f1d0e59cd13 \ + --hash=sha256:234d8d394721fa0d786af15ebf1f3fb7f3ed82fd1cd0cde45c2f247b5d4281d2 \ + --hash=sha256:256237037a3ab001ae8d9803b2d43562a4c3aa38739843694349e4d5ebb0fd56 \ + --hash=sha256:291bf73caf78b9e88d6fae9bfd693207ff7d832e2fdbe2cf8e746bc13f5f892b \ + --hash=sha256:29c4cab9df457b19672c39be7f384ebb2bc925c4e2684b8780c222b43eb36389 \ + --hash=sha256:2cfd1a81a648806e6a7717be4cc4d1bb392fa229752bf8444ba365e381e984d6 \ + --hash=sha256:2fde98cf1fc4bac144ce23c2c4c017b924ba714509ea9334977b0b27050c837d \ + --hash=sha256:32896a0af72508ea979e0acb4e4c04cbeeae04938167950d535c83c45597167d \ + --hash=sha256:355e3a4c725225a14262004fc1872a552b9d3634b4f791a0dfc80804aafbfd55 \ + --hash=sha256:3722a099730704c9a3d70c879fc0f51daec25fe5f1555672d97bc595abeafb95 \ + --hash=sha256:38c0c7d4b6fc06c3324b3f9162c8391bfc4fd9dde53afe1033ce7edb48d5a714 \ + --hash=sha256:39ffd14646190c454f0d86e0d256b33f00a87a26ab410e619773b841d0e41416 \ + --hash=sha256:3de789c82752730f94782a5ee518baf9c05edf85733aeaf73bb6e518755cdf54 \ + --hash=sha256:3e79f05e4a08b4d880342673312bbc895b56df7765605796f15902eb5367d3ae \ + --hash=sha256:3ff5893a2c23d886aa9ce786de5ac6ddc74aeeaf90743682b74d920e117d2e28 \ + --hash=sha256:411ca4d1b905b860ceba7570dd6717a71dedaddcc4b0f77ece710aa41ee11f8d \ + --hash=sha256:4388184646efe2054911c5b00a1077d6d1ee86a95b7e8ba96dc7850a809f3f40 \ + --hash=sha256:4a6369168d371207339b1e50d4532b06a7121586141f82599505a3f315751d47 \ + --hash=sha256:4bbcc257e3babea20a91715c361b24554ec4e8f51aa578568afc230799fe1a19 \ + --hash=sha256:4e66cbe84437497d951b799d3e1551291b6fb3d643820a7014b3655d57a59162 \ + --hash=sha256:4e6ee93fc3cf848dcbf0cce2eca73d8e7dcd0cc2b6df3a529d57750b30a4c55c \ + --hash=sha256:4eafbaff06b9563f8b1c850621ce51605de05208e09d4d71ce490bc972b7b9e8 \ + --hash=sha256:52e8db01f603f5da0ca30987479acff98769382efc8e142fa3962395dcf3ffdb \ + --hash=sha256:5500eeae393a184d14e1f35645962c27129d20c81afa4069e6ef826ebc2b3aaa \ + --hash=sha256:55456ea87d8df21808446d03817be2f65e20391c1c615d9187440dff28cd08dc \ + --hash=sha256:5563302a8359bc2295bb7084d1a8ed1519df96afb30eb2aa4e0bff7b54228988 \ + --hash=sha256:567b1c430f8bd560e689421468278ac5941bab4a05303b5d95b6ae10db03f451 \ + --hash=sha256:57f5eeb6ad4c180de583b1038e61fe5fbd9796bb69a8a1c1a0c7ddbec4c8c60f \ + --hash=sha256:59fe030d8ae4a57e3fb7756bf35a858de74e04066fc8555c53d0af979132af81 \ + --hash=sha256:5a6526a2a956bbb1e4ae3568c82e650fc99119c66bb011ea60715744955a2b4d \ + --hash=sha256:5a86a5a08c2235316bdb359d5dbb6ce0abfca7fac06363103e2c5af571d92f95 \ + --hash=sha256:5d2a91724463bfed4f573cd7a9fdc856d2e230d0c0e5a61416a93481dccd8605 \ + --hash=sha256:64b0c8c35aa4c4ed79896359f3e0b285cbe4e610042106500da4811c322cc108 \ + --hash=sha256:68242379c9b65a582b6e97318a1e9fbd6d445e58954f2d437991c4804ab11578 \ + --hash=sha256:6912fa5e635d74529ac7cdb1bdf6ca3af4453da8d1edbe0110ee1cb4ad407ebf \ + --hash=sha256:6c013cd3a1721e69e14380ada97eaa4b7b0cdf1c6b96fa765d4ea47c875088db \ + --hash=sha256:6c0eb900c0e91f4aebe680845242e614f1864edfd44106380d0752ac29522bf8 \ + --hash=sha256:6c6624fe268625869485553dd7cc1daf30d22558215bb2a4ff16f67a9801a31a \ + --hash=sha256:6d15a29033c57490cfe2069097c6fc4049e4e65ffbb749be7dc453b7c4c68965 \ + --hash=sha256:6d28a05796b99f749bf8794f17ba9ba1612d0076b802e9cfc62c554634e9ce3b \ + --hash=sha256:6d5225ef8801e4ea5e482fa9b5dfb891dd9ef6f6d870f1f25d449ca2c70ac218 \ + --hash=sha256:6da110e5f314c19ab8478464d02ae18808ae73d522c15260fa4918acdcd64da9 \ + --hash=sha256:6ecfc32dfb46fb7b565bcd6abf9412acf978775a998273d22888a6d7953730dd \ + --hash=sha256:713bd7df21170b982e729e46870f31d6b437bd1a9b4648cffb529bd3c2ec5c4b \ + --hash=sha256:71599e011ac880e8e45d46047d714871894c7d4ab6f25626f8d4f89da21f368d \ + --hash=sha256:7220697efaa6e5348fc3d18ee7f8563d4bfecd9872b37ffb915bfc1d08840622 \ + --hash=sha256:73b30cfa976659b3917c8f6153bdb0591c6a9ec6583599fd24a689b690622022 \ + --hash=sha256:779a6e7c894737e5983e7790a9c78c4000c30e23c9aada08081bdbea53b0fa60 \ + --hash=sha256:80811e1c42386ea95c6fb30571d3250ad43d7863f883f787f70517f441150e59 \ + --hash=sha256:814ff83a25b5fce8b9c80c4dd803153fb5c5599fc74db9e022466938368957ef \ + --hash=sha256:81a398f45b45a59200e13cd5ad1ae1d3f44334de98b148331afe2cdfee701c52 \ + --hash=sha256:823b92cf3c18ecd08afc70c42473888b41b6e8ef5046f3b82c05c154a2fa3d22 \ + --hash=sha256:82909c8f7eb9952656b65d3147afde4cf8e6d5a991eebc86418b5e65843b0ab8 \ + --hash=sha256:83380ffde38062a2e9bb55d83e74474f6614665528b98a6928720fc006dfffbb \ + --hash=sha256:8443e38dcfcfdbcf5add5118c623efd788d65ac2e25756d6251a54a06a4d0aca \ + --hash=sha256:84d244b00604d17df3fc7736c327892d6bba66181254aa4087be807b6c342bdc \ + --hash=sha256:856f743ae607f2c1380eccb566c0038a9fb3eabf0fc2be2704d76d9f73557239 \ + --hash=sha256:85ea21ec6730194d67156b0e0b5430ccb1d61f8b8b907e39b37f9812b74a13f0 \ + --hash=sha256:86a21a7bd3fe3a419512ef424cc1c020f6771d0b29cfddff36d1635a855e63f0 \ + --hash=sha256:88c2a17815c266e6d8180204ff62cb739ab869ada4a746d4c505331526ac58f1 \ + --hash=sha256:89cc46cfd15022e35084355478c9ac809d90b1152222706ac9a7655ec21df6fa \ + --hash=sha256:8ae493ed5f659a7761c43d42f183db514536073ded9bcf671d2d1df47e29a07e \ + --hash=sha256:8b2fdd7ead3c995c37940a790690660d0ca006c302db26cc51933f6766866fc3 \ + --hash=sha256:8b62076030baa2d8b1501a46bf0e19c27a489aa90671c55665bff7887f7660b0 \ + --hash=sha256:8e11699ed745931c395acd3621b07062e0f840efa6935aad87a64ed0995f0915 \ + --hash=sha256:92bfed8deec93df30286b9fe9e3b1dd17329cc076a192b4ee5ec223841d54953 \ + --hash=sha256:96bb17dbe8bab3c0954fbebfc69ed395599de75b6bbc35e3270a878e15d4dd65 \ + --hash=sha256:97335f59082f9fe2ce6c2a9cc6433a0114bbb6cd4d5c09dd76c95c68b9f9a8b0 \ + --hash=sha256:a417149c0cba4d50b61e992e5a15e69eaf96746609b461cc4ed168aeef6b79dd \ + --hash=sha256:a5207ec414d1c4a2a7231b2086970dc036f94293cdf338190984958a013a42f1 \ + --hash=sha256:a54cf9e0ef47b96af580849db5471142200568ce1e02cbf416addab551369570 \ + --hash=sha256:a56a1d4f859a82ca5b99fc4b82c9b027b15e3c455c5cd99e7d0719f27bb20b6c \ + --hash=sha256:a5fa8f1f916988d0bf1afea005bda37f56ac41a18016e813ccf0097a8d460ca4 \ + --hash=sha256:a6cd22c9da0d866558e46a041f1cc0c2bbb26b61b137b2347fa834c332e1d101 \ + --hash=sha256:aa1f1995789dca3698bc550aaceb09a51bd5df0a057ff84ff15296cd1975b801 \ + --hash=sha256:b0411b4066db926b80258c60dcb0e6db4c9cee312eab45b7e8866b17ddf9ada1 \ + --hash=sha256:b230acc1c3bfe2d6f2627ba2b95dc92e58aa494600e9722d0e6ccbc931e59702 \ + --hash=sha256:b30e600e8f337b9bd7f39b86d9fdfedc73cc46e3d0f745931a23a234220bb7e2 \ + --hash=sha256:b845b8d48088fad0cadc84be4b8fda63203be7e9237b71015b3925443c1f35ab \ + --hash=sha256:b87d67e33afaf265262f2a66db578284b88ee2e6fcd224579cb5c15518677ad8 \ + --hash=sha256:bac89069bc496ebdf4f79ebb57bbd10d0b214c8454225deb672d91002bd17e18 \ + --hash=sha256:bc25fb356d0c7810bb49ff3df908ad1fda6995d660ab099ded69244ed7ab6053 \ + --hash=sha256:bccbd8e5b0bffb7106cf18eb1baa3d7194b1cebb3b4b1cdbd4bdb19382a6ee6c \ + --hash=sha256:c16d15ee371643ab48dc8248a3e680ebbeca573a13af2c3dd0c985b142d77162 \ + --hash=sha256:c434e072557ade9cbc642d052c89d031efe47d5c9614523619d0d74a02378e81 \ + --hash=sha256:c47318cd3a61401452de11282242937e3e057c4fd3dbaf601e269d0928a06c0a \ + --hash=sha256:c70bc1b602cf59917e8f0c7a2cbc8bcc6fbc14d5486136b00707a79619121d63 \ + --hash=sha256:c7eec6a42018bc1d45763b1c162d3d2bf7c3b9a1b0ed30d3e91dcba390efefcc \ + --hash=sha256:c802434092b769b1d613ed2e13fac15fbfce1934a74bd10283b03c0fae231cd1 \ + --hash=sha256:cc30523e3f1a23fb7511cc659834a0d01a1042bb9de359bc1c131cc4ec6c9656 \ + --hash=sha256:d00d20d1818e82a07a0ee0aa89a98b17ed7916b92441090b683719cb20a59b6d \ + --hash=sha256:d2813ba2503764f0450680c533d13df7cff9b49df1411062eded5f67db4195b9 \ + --hash=sha256:d2c05c729b589e734c09578bf5964be48a911765484840d017bbc84f49d4c4ad \ + --hash=sha256:d4c7bacb70930f3d0a56f4ecf1be474a1f0d941b01dd73b756f3c256d42cb879 \ + --hash=sha256:d5f51401d102c885b9ca509e62c79b1dbff286e1b9b047fde6f763780789356d \ + --hash=sha256:d8363d7accb0286ac3a0e633f396e93800dafb8150494505daf9515bbda591f3 \ + --hash=sha256:d8bc24219b24c0af375718942ab75e3544b2763085f40f965be4326734ae8328 \ + --hash=sha256:d8edcf6f550e918dca779c069b9e156385c60b406f99fc7641f32c52f7193659 \ + --hash=sha256:da7a94d6a3411f579d72aa3e3bc5fbca7ed4549f3dbd7e5de3aa567333374285 \ + --hash=sha256:db13ca398005abcbe538deda87b686d9bd08b7001cf40c4c06b444960ae10a26 \ + --hash=sha256:dbab647e88d90b3167b91efe7091e248653688ed4337e4f90907a722c7361bb9 \ + --hash=sha256:dbd605739f228912dc49027cb764456b9757750bdc2b6b7773164db7096c6fd1 \ + --hash=sha256:e0b5deec9a8664eb722c797241970fd4aa1894d25fda36a1ddac0f7407606bd6 \ + --hash=sha256:e0d00c708fb2f5822b152429b1ac80a58dbbbc3f6c232c4d13a3f7fcf2ea5b4c \ + --hash=sha256:e1a49adf16a7c9d9646816c2946135527197b6fcf4347c7b8b761cf1bfbf4489 \ + --hash=sha256:e3b461183c5fa7681b48560f91515f53a953122fb30c71e07abc67d7ddf58c38 \ + --hash=sha256:e4c911f15a1652ca94ae9f1abd92e74cbb1b3597d2d92fdd556202f94e8cd455 \ + --hash=sha256:e56b6a368529bed262da40ce13f8fef590db0479819cca84f16a1f01ac356d0b \ + --hash=sha256:e87bc679f86a99aa3b26e3c78eeb821a247c9a28eae48eaafcc32c3bf4c3bb9e \ + --hash=sha256:e8c9a650a188e38bac005048cbe6342e81407782944d01934540ab75e417df21 \ + --hash=sha256:eab9208b00ca55bf75983ec99f7bf13acc746a36102e98953addaad7f7ea1e1b \ + --hash=sha256:ec3ba415afaf951f6951b1dd16d3c8e4f540065fc382d7e70b823a79567ca374 \ + --hash=sha256:ec4b19788f835711a2072f9dbe6b03b3bf32ed1f0fb30cf399bdd59d9f0c33fa \ + --hash=sha256:f395a4a9a03ac062dbe9a9f82e0c720502e590a38feee6a757bc82e9c63afbd8 \ + --hash=sha256:f42b74a53e5f26a0ba0007411a7455b66c67ce4022a39cc1f56fc4efd65bcbab \ + --hash=sha256:f54598964d357b1c5ab77cf5d92f21e598fe0e23cdbe9618480807f81b4eba15 \ + --hash=sha256:f56b397858a23dacf35ede366ed2212fdc03a6a57a1ad36468ad6e9dc5fac091 \ + --hash=sha256:f5de7feedc56337a088eb15cd9fafa9938367362221d8cc62c642b7f94821993 \ + --hash=sha256:f75720477ee05d509a310e856cacc8d909adc182f7b91193c207bcc26d7ee6db \ + --hash=sha256:f779070399f991400fc451719e0ea388eb7de313388bada2c127a35de05f798a \ + --hash=sha256:f9ca190fe9edc0eb08eec558a509a16d28d91c35667b8f043cba40ed5e77a959 \ + --hash=sha256:fa60887537e1d0cd2d9982269d33a709bf54b195cd2b9364fc0a758022af5bd9 \ + --hash=sha256:fc1ed11c4ad0b91af24def2050f2840ea4567828e3dd058fbe608d982f6e5465 \ + --hash=sha256:febb1ce6cac545a54e6b769982824e955a700fdd9fbf3a08a3d82c990968b57d +more-itertools==11.1.0 \ + --hash=sha256:48e8f4d9e7e5878571ecf6f2b4e57634f93cd474cc8cfbd2376f2d11b396e30d \ + --hash=sha256:4b65538ae22f6fed0ce4874efd317463a7489796a0939fa66824dd542125a192 +mypy==2.3.0 \ + --hash=sha256:04e617030eca5221909c8b7d8d7fd1c637948199aa2100b2ad9813feb07e1491 \ + --hash=sha256:094af99f92638aa92852326188b85a89e50f4a472f44827c03362228482f0762 \ + --hash=sha256:09abd66d8685e73f8f7d17b847c3e104d9a7b164a8706ea87d6c96a3d45816d5 \ + --hash=sha256:0b025a93cffb9781d231f232be07a17912f35f10a313c24f301c81e842870654 \ + --hash=sha256:13b1b16e2fa39f3b2e33fb1c468abc7a69369fa2e886b4b87b5afc81472325cd \ + --hash=sha256:1c6c6bf687b17f90dbfcad95b960d32eaa0154c00da45f03ab50bf8952e047fe \ + --hash=sha256:1fa8d916ac3b705af733c4c1e6c9ebe38fd0d52beb15b105c3e8355b55e6ecdc \ + --hash=sha256:28e1e2af8cd8fff551fd30f2fe4b03fb76764ac8b1ba6c6a1bd00ad32b412db3 \ + --hash=sha256:2d53fc67b9d28a43c6199077f49fea0f05839e36cf6158500331c9549225e5a5 \ + --hash=sha256:3419d00717afbc5265b50dd14b1278f29ea4884dd398ab67873489ac093fd329 \ + --hash=sha256:37fa4de896a84e2dc9200d91e614c22563b43d1a266789d4bbac7b22ebe6192b \ + --hash=sha256:3961a4a34b05f7c74b0f05aa51fbfe99a2d1e126038df40318d15c8f558b7ef3 \ + --hash=sha256:3dd0bed92c4bdec57c42505b96416fb9e6a5aa7be84d2809bcd5f2ecec2860d7 \ + --hash=sha256:3e77244df3843048c3f927182916730e40c124cbaa43905c1fb86cb382aa0805 \ + --hash=sha256:4359424140d985192c778c1ce2c114a10c1ca58a381ed79cfa70d37df94b299f \ + --hash=sha256:465965d41cd9a2726694e983e8ce7113259327bec798115d1e1dfa2a52fb666e \ + --hash=sha256:56c184d2c20ca6b6378d58d1960270a767f41f5e44acbbd27f05effef4f4e1d7 \ + --hash=sha256:5e91adad1ca81742ac7ef9893959911df867752206b37135185e88dfb3c89494 \ + --hash=sha256:691fdc37132b1ae628d834f672e74de83462d9fb4aff621835767fb43a8dd373 \ + --hash=sha256:6b1cdb579446b60432432b2b2403a6201b4b475a004d7f488511c9ba177c9e88 \ + --hash=sha256:6f99ec626e3c3a2f7c0b22c5b90ddb5dabb1c18729c971e9bdaca1f1766d2cee \ + --hash=sha256:7247eb2824f996722a949530183394921ca71deb9680052a338cf53cff7925c2 \ + --hash=sha256:75b0984bb3cbd76bb5c9291a8671f7ae66ca3b51c7584c358fc2e923259f0757 \ + --hash=sha256:75cbb4b9ef04a0c84a957f07abc4504fbf64b8dcc145675101f2d3a78a4b1d6a \ + --hash=sha256:7da939dd335cfd2ad788bdfd081c9f4e47634ab995e5a45eb15fd1e5bc052f8b \ + --hash=sha256:85c5385b93012ffa3b31479ab579aef5415f4f3a32c6cf1ae07a984d2a0ff461 \ + --hash=sha256:91ad22a52ae2c7e621c2f67c94d5a17f66b3209a4cff5cf8a573579835c69e97 \ + --hash=sha256:944c665d984157cb96a679dfb7a4a81dd1d36b24b9c284b699514e6e626b82d4 \ + --hash=sha256:9559ab18a9c9957dfa3004ab57cd4bac5f26a724329a9584e583367f0c2e1117 \ + --hash=sha256:982e3d53dd23d0a4cef67dd66791fdbede0cf38f9eb617bf47663554c51e1e36 \ + --hash=sha256:99ac767cc5d3b64c8d0ae226ead10c96694f94e4e7da1668642225dcd4e75aac \ + --hash=sha256:adebc76aab4f3495a88b41d48aa4aff0c03f2822501da76625afcca5975f19e5 \ + --hash=sha256:aec15d465d477558fd842757b487849007311cf3897849cdda0e3162ac0ac556 \ + --hash=sha256:b1942b9314d4c784b8ea1dbab4972603290e5dd5630f06675f13aec97526bc4c \ + --hash=sha256:b352b7e49f5e6576009e8df730e1ff4f915cb565b851b396d2ffe2f5a6f5da88 \ + --hash=sha256:b5cd2f027a972a4a5f2278a11fac9747f5f81a53a30b714d74950b6807e55568 \ + --hash=sha256:be51653d7669d7d7955d613b8d0bb57d5b652eaf71a873ddf65ac87254dd2595 \ + --hash=sha256:cfca8ee88544090f86b6dcce05ec55d66eb48a762412ac2507810ba4bd793b6f \ + --hash=sha256:d78fcf900b59cb7e82cb7e3a235e31b462d9333d92285bd1e4952d355b8ffba1 \ + --hash=sha256:de121747278144fc9ae7caa2e978cf5df12aebc82933182f5b3b86081a30baef \ + --hash=sha256:de6d2c484742a4d7b0ed6d07b143375624d3b899c5749c7b3c947f56261f48a6 \ + --hash=sha256:ea317b060ce83e26050f8f9e4d7d6bf44ed7597c8ff9990bccffbb9d1d8522db \ + --hash=sha256:f1b3a98dfd21058bc759bb3337d5d1f61d0fdf9f3cf9c00f4291790fb5427bff \ + --hash=sha256:f4ed18f111bfe2d599bca7468e7f9251042c1c2118f762c8de2766a56d773c60 \ + --hash=sha256:fbc00cee7bdbb9291979ddc9d08034a29dfcda4932628c9bbc28c1edd589df0c +mypy-extensions==1.1.0 \ + --hash=sha256:1be4cccdb0f2482337c4743e60421de3a356cd97508abadd57d47403e94f5505 \ + --hash=sha256:52e68efc3284861e772bbcd66823fde5ae21fd2fdb51c62a211403730b916558 +ollama==0.6.2 \ + --hash=sha256:3ad7daab28e5a973445c36a73882a3ef698c2ebb00e21e308652741577509f7d \ + --hash=sha256:936d55daa684f474364c098611c933626f8d6c7d67065c5b7ae0c477b508b07f +openai==2.53.0 \ + --hash=sha256:baf5802ad08980e1d9d561e1b996e800c8bcd14af5847c6d0e7a5cc59e4d4116 \ + --hash=sha256:c694ffc747a3c4d1663ef2b07b811315a476164ee5efa3a993967349ebca7618 +openpyxl==3.1.5 \ + --hash=sha256:5282c12b107bffeef825f4617dc029afaf41d0ea60823bbb665ef3079dc79de2 \ + --hash=sha256:cf0e3cf56142039133628b5acffe8ef0c12bc902d2aadd3e0fe5878dc08d1050 +orjson==3.11.9 \ + --hash=sha256:011382e2a60fda9d46f1cdee31068cfc52ffe952b587d683ec0463002802a0f4 \ + --hash=sha256:03db380e3780fa0015ed776a90f20e8e20bb11dde13b216ce19e5718e3dfba62 \ + --hash=sha256:051b102c93b4f634e89f3866b07b9a9a98915ada541f4ec30f177067b2694979 \ + --hash=sha256:08f4d8ebb44925c794e535b2bebc507cebf32209df81de22ae285fb0d8d66de0 \ + --hash=sha256:0b34789fa0da61cf7bef0546b09c738fb195331e017e477096d129e9105ab03d \ + --hash=sha256:0e4eed3b200023042814d2fc8a5d2e880f13b52e1ed2485e83da4f3962f7dc1a \ + --hash=sha256:115ab5f5f4a0f203cc2a5f0fb09aee503a3f771aa08392949ab5ca230c4fbdbd \ + --hash=sha256:135869ef917b8704ea0a94e01620e0c05021c15c52036e4663baffe75e72f8ce \ + --hash=sha256:147302878da387104b66bb4a8b0227d1d487e976ce41a8501916161072ed87b1 \ + --hash=sha256:14ed654580c1ed2bc217352ec82f91b047aef82951aa71c7f64e0dcb03c0e180 \ + --hash=sha256:16969c9d369c98eb084889c6e4d2d39b77c7eb38ceccf8da2a9fff62ae908980 \ + --hash=sha256:19b72ed11572a2ee51a67a903afbe5af504f84ed6f529c0fe44b0ab3fb5cc697 \ + --hash=sha256:231742b4a11dad8d5380a435962c57e91b7c37b79be858f4ef1c0df1a259897e \ + --hash=sha256:25e4aed0312d292c09f61af25bba34e0b2c88546041472b09088c39a4d828af1 \ + --hash=sha256:26a473dbb4162108b27901492546f83c76fdcea3d0eadff00ae7a07e18dcce09 \ + --hash=sha256:277fefe9d76ee17eb14debf399e3533d4d63b5f677a4d3719eb763536af1f4bd \ + --hash=sha256:2d057a602cdd19a0ad680417527c45b6961a095081c0f46fe0e03e304aac6470 \ + --hash=sha256:32ef5f4283a3be81913947d19608eacb7c6608026851123790cd9cc8982af34b \ + --hash=sha256:33d7d766701847dc6729846362dc27895d2f2d2251264f9d10e7cb9878194877 \ + --hash=sha256:34fd2317602587321faab75ab76c623a0117e80841a6413654f04e47f339a8fb \ + --hash=sha256:3513550321f8c8c811a7c3297b8a630e82dc08e4c10216d07703c997776236cd \ + --hash=sha256:380cdce7ba24989af81d0a7013d0aaec5d0e2a21734c0e2681b1bc4f141957fe \ + --hash=sha256:3a81d52442a7c99b3662333235b3adf96a1715864658b35bb797212be7bddb97 \ + --hash=sha256:3ebca4179031ee716ed076ffadc29428e900512f6fccee8614c9983157fcf19c \ + --hash=sha256:48ee05097750de0ff69ed5b7bbcf0732182fd57a24043dcc2a1da780a5ead3a5 \ + --hash=sha256:4bab1b2d6141fe7b32ae71dac905666ece4f94936efbfb13d55bb7739a3a6021 \ + --hash=sha256:4d4e98d6f3b8afed8bc8cd9718ec0cdf46661826beefb53fe8eafb37f2bf0362 \ + --hash=sha256:4d7fde5501b944f83b3e665e1b31343ff6e154b15560a16b7130ea1e594a4206 \ + --hash=sha256:4da3c38a2083ca4aaf9c2a36776cce3e9328e6647b10d118948f3cfb4913ffe4 \ + --hash=sha256:4e39364e726a8fff737309aff059ff67d8a8c8d5b677be7bb49a8b3e84b7e218 \ + --hash=sha256:4fd66214623f1b17501df9f0543bef0b833979ab5b6ded1e1d123222866aa8c9 \ + --hash=sha256:4fef17e1f8722c11587a6ef18e35902450221da0028e65dbaaa543619e68e48f \ + --hash=sha256:53b50b0e14084b8f7e29c5ce84c5af0f1160169b30d8a6914231d97d2fe297d4 \ + --hash=sha256:57ea77fb70a448ce87d18fca050193202a3da5e54598f6501ca5476fb66cfe02 \ + --hash=sha256:59e403b1cc5a676da8eaf31f6254801b7341b3e29efa85f92b48d272637e77be \ + --hash=sha256:5b192c6cf397e4455b11523c5cf2b18ed084c1bbd61b6c0926344d2129481972 \ + --hash=sha256:5f63aaf97afd9f6dec5b1a68e1b8da12bfccb4cb9a9a65c3e0b6c847849e7586 \ + --hash=sha256:63e0efbc991250c0b3143488fa57d95affcabbfc63c99c48d625dd37779aafe2 \ + --hash=sha256:6cc7923789694fd58f001cbcac7e47abc13af4d560ebbfcf3b41a8b1a0748124 \ + --hash=sha256:71e63adb0e1f1ed5d9e168f50a91ceb93ae6420731d222dc7da5c69409aa47aa \ + --hash=sha256:71f3db16e69b667b132e0f305a833d5497da302d801508cbb051ed9a9819da47 \ + --hash=sha256:844417969855fc7a41be124aafe83dc424592a7f77cd4501900c67307122b92c \ + --hash=sha256:8697ab6a080a5c46edaad50e2bc5bd8c7ca5c66442d24104fa44ec74910a8244 \ + --hash=sha256:87e4d4ab280b0c87424d47695bec2182caf8cfc17879ea78dab76680194abc13 \ + --hash=sha256:8aff7da9952a5ad1cef8e68017724d96c7b9a66e99e91d6252e1b133d67a7b10 \ + --hash=sha256:8ecc30f10465fa1e0ce13fd01d9e22c316e5053a719a8d915d4545a09a5ff677 \ + --hash=sha256:97d0d932803c1b164fde11cb542a9efcb1e0f63b184537cca65887147906ff48 \ + --hash=sha256:97db4c94a7db398a5bd636273324f0b3fd58b350bbbac8bb380ceb825a9b40f4 \ + --hash=sha256:9af678d6488357948f1f84c6cd1c1d397c014e1ae2f98ae082a44eb48f602624 \ + --hash=sha256:9ef6fe90aadef185c7b128859f40beb24720b4ecea95379fc9000931179c3a49 \ + --hash=sha256:9f78cf8fec5bd627f4082b8dfeac7871b43d7f3274904492a43dab39f18a19a0 \ + --hash=sha256:a028425d1b440c5d92a6be1e1a020739dfe67ea87d96c6dbe828c1b30041728b \ + --hash=sha256:a6082706765a95a6680d812e1daf1c0cfe8adec7831b3ff3b625693f3b461b1c \ + --hash=sha256:a8f5f8bc7ce7d59f08d9f99fa510c06496164a24cb5f3d34537dbd9ca30132e2 \ + --hash=sha256:aaea64f3f467d22e70eeed68bdccb3bc4f83f650446c4a03c59f2cba28a108db \ + --hash=sha256:ace6c58523302d3b97b6ac5c38a5298a54b473762b6be82726b4265c41029f92 \ + --hash=sha256:b3afcf569c15577a9fe64627292daa3e6b3a70f4fb77a5df246a87ec21681b94 \ + --hash=sha256:b6ef1979adc4bc243523f1a2ba91418030a8e29b0a99cbe7e0e2d6807d4dce6e \ + --hash=sha256:be4fa4f0af7fa18951f7ab3fc2148e223af211bf03f59e1c6034ec3f97f21d61 \ + --hash=sha256:c2d3dc759490128c5c1711a53eeaa8ee1d437fd0038ffd2b6008abf46db3f882 \ + --hash=sha256:c5d001196b89fa9cf0a4ab79766cd835b991a166e4b621ba95089edc50c429ff \ + --hash=sha256:cce9127885941bd28f080cecf1f1d288336b7e0d812c345b08be88b572796254 \ + --hash=sha256:cde1a448023ba7d5bb4c01c5afb48894380b5e4956e0627266526587ef4e535f \ + --hash=sha256:d4087e5c0209a0a8efe4de3303c234b9c44d1174161dcd851e8eea07c7560b32 \ + --hash=sha256:d8ea516b3726d190e1b4297e6f4e7a8650347ae053868a18163b4dd3641d1fff \ + --hash=sha256:e30ab17845bb9fa54ccf67fa4f9f5282652d54faa6d17452f47d0f369d038673 \ + --hash=sha256:e5c9b8f28e726e97d97696c826bc7bea5d71cecd63576dba92924a32c1961291 \ + --hash=sha256:ea407d4ccf5891d667d045fecae97a7a1e5e87b3b97f97ae1803c2e741130be0 \ + --hash=sha256:ea5c46eb2d3af39e806b986f4b09d5c2706a1f5afde3cbf7544ce6616127173c \ + --hash=sha256:eebdbdeef0094e4f5aefa20dcd4eb2368ab5e7a3b4edea27f1e7b2892e009cf9 \ + --hash=sha256:f01c4818b3fc9b0da8e096722a84318071eaa118df35f6ed2344da0e73a5444f \ + --hash=sha256:f36b7f32c7c0db4a719f1fc5824db4a9c6f8bd1a354debb91faf26ebf3a4c71e \ + --hash=sha256:f5d89a2ed90731df3be64bab0aa44f78bff39fdc9d71c291f4a8023aa46425b7 \ + --hash=sha256:ffe02797b5e9f3a9d8292ddcd289b474ad13e81ad83cd1891a240811f1d2cb81 +ormsgpack==1.12.2 \ + --hash=sha256:0b39e629fd2e1c5b2f46f99778450b59454d1f901bc507963168985e79f09c5d \ + --hash=sha256:118576ea6006893aea811b17429bfc561b4778fad393f5f538c84af70b01260c \ + --hash=sha256:18867233df592c997154ff942a6503df274b5ac1765215bceba7a231bea2745d \ + --hash=sha256:1dcc17d92b6390d4f18f937cf0b99054824a7815818012ddca925d6e01c2e49e \ + --hash=sha256:21f4276caca5c03a818041d637e4019bc84f9d6ca8baa5ea03e5cc8bf56140e9 \ + --hash=sha256:292410a7d23de9b40444636b9b8f1e4e4b814af7f1ef476e44887e52a123f09d \ + --hash=sha256:29a9f17a3dac6054c0dce7925e0f4995c727f7c41859adf9b5572180f640d172 \ + --hash=sha256:34d5b28b3570e9fed9a5a76528fc7230c3c76333bc214798958e58e9b79cc18a \ + --hash=sha256:3708693412c28f3538fb5a65da93787b6bbab3484f6bc6e935bfb77a62400ae5 \ + --hash=sha256:39c1bd2092880e413902910388be8715f70b9f15f20779d44e673033a6146f2d \ + --hash=sha256:43013a3f3e2e902e1d05e72c0f1aeb5bedbb8e09240b51e26792a3c89267e181 \ + --hash=sha256:458e4568be13d311ef7d8877275e7ccbe06c0e01b39baaac874caaa0f46d826c \ + --hash=sha256:46d084427b4132553940070ad95107266656cb646ea9da4975f85cb1a6676553 \ + --hash=sha256:4ee766d2e78251b7a63daf1cddfac36a73562d3ddef68cacfb41b2af64698033 \ + --hash=sha256:50b7249244382209877deedeee838aef1542f3d0fc28b8fe71ca9d7e1896a0d7 \ + --hash=sha256:58d379d72b6c5e964851c77cfedfb386e474adee4fd39791c2c5d9efb53505cc \ + --hash=sha256:59f5da97000c12bc2d50e988bdc8576b21f6ab4e608489879d35b2c07a8ab51a \ + --hash=sha256:5af04800d844451cf102a59c74a841324868d3f1625c296a06cc655c542a6685 \ + --hash=sha256:5ea60cb5f210b1cfbad8c002948d73447508e629ec375acb82910e3efa8ff355 \ + --hash=sha256:5f13034dc6c84a6280c6c33db7ac420253852ea233fc3ee27c8875f8dd651163 \ + --hash=sha256:7121b3d355d3858781dc40dafe25a32ff8a8242b9d80c692fd548a4b1f7fd3c8 \ + --hash=sha256:73670ed0375ecc303858e3613f407628dd1fca18fe6ac57b7b7ce66cc7bb006c \ + --hash=sha256:7a1c460655d7288407ffa09065e322a7231997c0d62ce914bf3a96ad2dc6dedd \ + --hash=sha256:7a29d09b64b9694b588ff2f80e9826bdceb3a2b91523c5beae1fab27d5c940e7 \ + --hash=sha256:7c8b1667a72cbba74f0ae7ecf3105a5e01304620ed14528b2cb4320679d2869b \ + --hash=sha256:837dd316584485b72ef451d08dd3e96c4a11d12e4963aedb40e08f89685d8ec2 \ + --hash=sha256:8463a3fc5f09832e67bdb0e2fda6d518dc4281b133166146a67f54c08496442e \ + --hash=sha256:8cde5eaa6c6cbc8622db71e4a23de56828e3d876aeb6460ffbcb5b8aff91093b \ + --hash=sha256:944a2233640273bee67521795a73cf1e959538e0dfb7ac635505010455e53b33 \ + --hash=sha256:958dcb270d30a7cb633a45ee62b9444433fa571a752d2ca484efdac07480876e \ + --hash=sha256:9e4459c3f27066beadb2b81ea48a076a417aafffff7df1d3c11c519190ed44f2 \ + --hash=sha256:ae3aba7eed4ca7cb79fd3436eddd29140f17ea254b91604aa1eb19bfcedb990f \ + --hash=sha256:b009049086ddc6b8f80c76b3955df1aa22a5fbd7673c525cd63bf91f23122ede \ + --hash=sha256:baca4b6773d20a82e36d6fd25f341064244f9f86a13dead95dd7d7f996f51709 \ + --hash=sha256:bc68dd5915f4acf66ff2010ee47c8906dc1cf07399b16f4089f8c71733f6e36c \ + --hash=sha256:bd5f4bf04c37888e864f08e740c5a573c4017f6fd6e99fa944c5c935fabf2dd9 \ + --hash=sha256:c010da16235806cf1d7bc4c96bf286bfa91c686853395a299b3ddb49499a3e13 \ + --hash=sha256:c1429217f8f4d7fcb053523bbbac6bed5e981af0b85ba616e6df7cce53c19657 \ + --hash=sha256:c2be829954434e33601ae5da328cccce3266b098927ca7a30246a0baec2ce7bd \ + --hash=sha256:c6a4c34ddef109647c769d69be65fa1de7a6022b02ad45546a69b3216573eb4a \ + --hash=sha256:cec70477d4371cd524534cd16472d8b9cc187e0e3043a8790545a9a9b296c258 \ + --hash=sha256:d024b40828f1dde5654faebd0d824f9cc29ad46891f626272dd5bfd7af2333a4 \ + --hash=sha256:da538c542bac7d1c8f3f2a937863dba36f013108ce63e55745941dda4b75dbb6 \ + --hash=sha256:dc7a33be14c347893edbb1ceda89afbf14c467d593a5ee92c11de4f1666b4d4f \ + --hash=sha256:df6961442140193e517303d0b5d7bc2e20e69a879c2d774316125350c4a76b92 \ + --hash=sha256:eddffb77eff0bad4e67547d67a130604e7e2dfbb7b0cde0796045be4090f35c6 \ + --hash=sha256:f04b5e896d510b07c0ad733d7fce2d44b260c5e6c402d272128f8941984e4285 \ + --hash=sha256:f3601f19afdbea273ed70b06495e5794606a8b690a568d6c996a90d7255e51c1 \ + --hash=sha256:fcd55e5f6ba0dbce624942adf9f152062135f991a0126064889f68eb850de0dd +packaging==26.3 \ + --hash=sha256:94edc256424af38762eb31306eed28beb9f0efc50a8837492c9d6fd6004aed79 \ + --hash=sha256:d7193f7c8e4e93f444fde0262bf90af30e16fa0ad0ad44cb553c87339b23cd1c +pathspec==1.1.1 \ + --hash=sha256:17db5ecd524104a120e173814c90367a96a98d07c45b2e10c2f3919fff91bf5a \ + --hash=sha256:a00ce642f577bf7f473932318056212bc4f8bfdf53128c78bbd5af0b9b20b189 +pluggy==1.6.0 \ + --hash=sha256:7dcc130b76258d33b90f61b658791dede3486c3e6bfb003ee5c9bfb396dd22f3 \ + --hash=sha256:e920276dd6813095e9377c0bc5566d94c932c33b27a3e3945d8389c374dd4746 +prometheus-client==0.26.0 \ + --hash=sha256:04a91bcf94e2cf74a44a1a874d651a2e853ed354b6e822f3b7487751465d5c2b \ + --hash=sha256:fa93d06737aa02bacd05794768508bb97d2fbee28cb3bca04eaae92f0ca953d6 +pycparser==3.0 \ + --hash=sha256:600f49d217304a5902ac3c37e1281c9fe94e4d0489de643a9504c5cdfdfc6b29 \ + --hash=sha256:b727414169a36b7d524c1c3e31839a521725078d7b2ff038656844266160a992 +pydantic==2.13.4 \ + --hash=sha256:45a282cde31d808236fd7ea9d919b128653c8b38b393d1c4ab335c62924d9aba \ + --hash=sha256:c40756b57adaa8b1efeeced5c196f3f3b7c435f90e84ea7f443901bec8099ef6 +pydantic-core==2.46.4 \ + --hash=sha256:00c603d540afdd6b80eb39f078f33ebd46211f02f33e34a32d9f053bba711de0 \ + --hash=sha256:0186750b482eefa11d7f435892b09c5c606193ef3375bcf94aa00ae6bfb66262 \ + --hash=sha256:041bde0a48fd37cf71cab1c9d56d3e8625a3793fef1f7dd232b3ff37e978ecda \ + --hash=sha256:0c563b08bca408dc7f65f700633d8442fffb2421fc47b8101377e9fd65051ff0 \ + --hash=sha256:0cbe8b01f948de4286c74cdd6c667aceb38f5c1e26f0693b3983d9d74887c65e \ + --hash=sha256:0ce40cd7b21210e99342afafbd4d0f76d784eb5b1d60f3bdc566be4983c6c73b \ + --hash=sha256:0e96592440881c74a213e5ad528e2b24d3d4f940de2766bed9010ab1d9e51594 \ + --hash=sha256:10e17cbb10a330363733efc4d7c4d0dd827ac0909b8f6a6542298fed1ea62f29 \ + --hash=sha256:133878133d271ade3d41d1bfb2a45ec38dbdbda40bc065921c6b04e4630127e2 \ + --hash=sha256:14d4edf427bdcf950a8a02d7cb44a08614388dd6e1bdcbf4f67504fa7887da9c \ + --hash=sha256:14f4c5d6db102bd796a627bbb3a17b4cf4574b9ae861d8b7c9a9661c6dd3362d \ + --hash=sha256:17299feefe090f2caa5b8e37222bb5f663e4935a8bfa6931d4102e5df1a9f398 \ + --hash=sha256:184c081504d17f1c1066e430e117142b2c77d9448a97f7b65c6ac9fd9aee238d \ + --hash=sha256:18e5ceec2ab67e6d5f1a9085e5a24c9c4e2ac4545730bfe668680bca05e555f3 \ + --hash=sha256:19e51f073cd3df251856a8a4189fbdf1de4012c3ebacfb1884f94f1eb406079f \ + --hash=sha256:1a7dd0b3ee80d90150e3495a3a13ac34dbcbfd4f012996a6a1d8900e91b5c0fb \ + --hash=sha256:1d8ba486450b14f3b1d63bc521d410ec7565e52f887b9fb671791886436a42f7 \ + --hash=sha256:2108ba5c1c1eca18030634489dc544844144ee36357f2f9f780b93e7ddbb44b5 \ + --hash=sha256:228ee9bae8bef5b1e97ec58302f80357c37199e0d0a99174e138d28e6957b9d9 \ + --hash=sha256:23ace664830ee0bfe014a0c7bc248b1f7f25ed7ad103852c317624a1083af462 \ + --hash=sha256:2412e734dcb48da14d4e4006b82b46b74f2518b8a26ee7e58c6844a6cd6d03c4 \ + --hash=sha256:29c61fc04a3d840155ff08e475a04809278972fe6aef51e2720554e96367e34b \ + --hash=sha256:2f84c03c8607173d16b5a854ec68a2f9079ae03237a54fb506d13af47e1d018d \ + --hash=sha256:3009f12e4e90b7f88b4f9adb1b0c4a3d58fe7820f3238c190047209d148026df \ + --hash=sha256:3245406455a5d98187ec35530fd772b1d799b26667980872c8d4614991e2c4a2 \ + --hash=sha256:3447661d99f75a3683a4cf5c87da72f2161964611864dbbeac7fbb118bb4bfc0 \ + --hash=sha256:372429a130e469c9cd698925ce5fc50940b7a1336b0d82038e63d5bbc4edc519 \ + --hash=sha256:395aebd9183f9d112f569aeb5b2214d1a10a33bec8456447f7fbdfa51d38d4cd \ + --hash=sha256:3a233125ac121aa3ffba9a2b59edfc4a985a76092dc8279586ab4b71390875e7 \ + --hash=sha256:3be77f45df024d789a672ae34f8b06fb346c4f9f46ea714956660ea4862e89ac \ + --hash=sha256:3bf92c5d0e00fefaab325a4d27828fe6b6e2a21848686b5b60d2d9eeb09d76c6 \ + --hash=sha256:3ecbc122d18468d06ca279dc26a8c2e2d5acb10943bb35e36ae92096dc3b5565 \ + --hash=sha256:3fb702cd90b0446a3a1c5e470bfa0dd23c0233b676a9099ddcc964fa6ca13898 \ + --hash=sha256:428e04521a40150c85216fc8b85e8d39fece235a9cf5e383761238c7fa9b96fb \ + --hash=sha256:432c179df7874eeb73307aad2df0755e1ae0efa61ff0ea89b93e194411ae3928 \ + --hash=sha256:4a05d69cba51d852c5c3e92758653245a50c0b646ced0cf05bd793ed592839d6 \ + --hash=sha256:4c63ebc82684aa89d9a3bcbd13d515b3be44250dc68dd3bd81526c1cb31286c3 \ + --hash=sha256:4fc73cb559bdb54b1134a706a2802a4cddd27a0633f5abb7e53056268751ac6a \ + --hash=sha256:4fcbe087dbc2068af7eda3aa87634eba216dbda64d1ae73c8684b621d33f6596 \ + --hash=sha256:56cb4851bcaf3d117eddcef4fe66afd750a50274b0da8e22be256d10e5611987 \ + --hash=sha256:5855698a4856556d86e8e6cd8434bc3ac0314ee8e12089ae0e143f64c6256e4e \ + --hash=sha256:5a4330cdbc57162e4b3aa303f588ba752257694c9c9be3e7ebb11b4aca659b5d \ + --hash=sha256:5b712b53160b79a5850310b912a5ef8e57e56947c8ad690c227f5c9d7e561712 \ + --hash=sha256:5d5902252db0d3cedf8d4a1bc68f70eeb430f7e4c7104c8c476753519b423008 \ + --hash=sha256:617d7e2ca7dcb8c5cf6bcb8c59b8832c94b36196bbf1cbd1bfb56ed341905edd \ + --hash=sha256:62f875393d7f270851f20523dd2e29f082bcc82292d66db2b64ea71f64b6e1c1 \ + --hash=sha256:633147d34cf4550417f12e2b1a0383973bdf5cdfde212cb09e9a581cf10820be \ + --hash=sha256:66ce7632c22d837c95301830e111ad0128a32b8207533b60896a96c4915192ea \ + --hash=sha256:6b3ace8194b0e5204818c92802dcdca7fc6d88aabbb799d7c795540d9cd6d292 \ + --hash=sha256:6f2eeda33a839975441c86a4119e1383c50b47faf0cbb5176985565c6bb02c33 \ + --hash=sha256:7027560ee92211647d0d34e3f7cd6f50da56399d26a9c8ad0da286d3869a53f3 \ + --hash=sha256:7283d57845ecf5a163403eb0702dfc220cc4fbdd18919cb5ccea4f95ee1cdab4 \ + --hash=sha256:7a5f930472650a82629163023e630d160863fce524c616f4e5186e5de9d9a49b \ + --hash=sha256:7bfb192b3f4b9e8a89b6277b6ce787564f62cfd272055f6e685726b111dc7826 \ + --hash=sha256:811ff8e9c313ab425368bcbb36e5c4ebd7108c2bbf4e4089cfbb0b01eff63fac \ + --hash=sha256:8233f2947cf85404441fd7e0085f53b10c93e0ee78611099b5c7237e36aacbf7 \ + --hash=sha256:82cf5301172168103724d49a1444d3378cb20cdee30b116a1bd6031236298a5d \ + --hash=sha256:8358a950c8909158e3df31538a7e4edc2d7265a7c54b47f0864d9e5bae9dcebf \ + --hash=sha256:85bb3611ff1802f3ee7fdd7dbff26b56f343fb432d57a4728fdd49b6ef35e2f4 \ + --hash=sha256:86e1a4418c6cd97d60c95c71164158eaf7324fae7b0923264016baa993eba6fc \ + --hash=sha256:8b9bab013d1c7a79d3501ff86d0bc9c31bf587db4551677b96bec07df78c6b15 \ + --hash=sha256:8c5dac79fa1614d1e06ca695109c6105923bd9c7d1d6c918d4e637b7e6b32fd3 \ + --hash=sha256:8d0820e8192167f80d88d64038e609c31452eeca865b4e1d9950a27a4609b00b \ + --hash=sha256:8daafc69c93ee8a0204506a3b6b30f586ef54028f52aeeeb5c4cfc5184fd5914 \ + --hash=sha256:9037063db01f09b09e237c282b6792bd4da634b5402c4e7f0c61effed7701a04 \ + --hash=sha256:905a0ed8ea6f2d61c1738835f99b699348d7857379083e5fc497fa0c967a407c \ + --hash=sha256:90884113d8b48f760e9587002789ddd741e76ab9f89518cd1e43b1f1a52ec44b \ + --hash=sha256:91a06d2e259ecfbd8c901d70c3c507900458498142b3026a296b7de4d1322cc9 \ + --hash=sha256:926c9541b14b12b1681dca8a0b75feb510b06c6341b70a8e500c2fdcff837cce \ + --hash=sha256:9401557acd873c3a7f3eb9383edef8ac4968f9510e340f4808d427e75667e7b4 \ + --hash=sha256:9551187363ffc0de2a00b2e47c25aeaeb1020b69b668762966df15fc5659dd5a \ + --hash=sha256:962ccbab7b642487b1d8b7df90ef677e03134cf1fd8880bf698649b22a69371f \ + --hash=sha256:97e7cf2be5c77b7d1a9713a05605d49460d02c6078d38d8bef3cbe323c548424 \ + --hash=sha256:9aa768456404a8bf48a4406685ac2bec8e72b62c69313734fa3b73cf33b3a894 \ + --hash=sha256:9bc519fbf2b7578398853d815009ae5e4d4603d12f4e3f91da8c06852d3da3e9 \ + --hash=sha256:9d56801be94b86a9da183e5f3766e6310752b99ff647e38b09a9500d88e46e76 \ + --hash=sha256:9f444c499b3eefd3a92e348059471ea0c3a6e303d9c1cec09fa748fd9f895201 \ + --hash=sha256:9fa8ae11da9e2b3126c6426f147e0fba88d96d65921799bb30c6abd1cb2c97fb \ + --hash=sha256:a0f62d0a58f4e7da165457e995725421e0064f2255d8eccebc49f41bbc23b109 \ + --hash=sha256:a396dcc17e5a0b164dbe026896245a4fa9ff402edca1dff0be3d53a517f74de4 \ + --hash=sha256:aaa2a54443eff1950ba5ddc6b6ccda0d9c84a364276a62f969bdf2a390650848 \ + --hash=sha256:ad785e92e6dc634c21555edc8bd6b64957ab844541bcb96a1366c202951ae526 \ + --hash=sha256:af8244b2bef6aaad6d92cda81372de7f8c8d36c9f0c3ea36e827c60e7d9467a0 \ + --hash=sha256:b078afbc25f3a1436c7a1d2cd3e322497ee99615ba97c563566fdf46aff1ee01 \ + --hash=sha256:b2f69dec1725e79a012d920df1707de5caf7ed5e08f3be4435e25803efc47458 \ + --hash=sha256:b8458003118a712e66286df6a707db01c52c0f52f7db8e4a38f0da1d3b94fc4e \ + --hash=sha256:bb63e0198ca18aad131c089b9204c23079c3afa95487e561f4c522d519e55aba \ + --hash=sha256:bfec22eab3c8cc2ceec0248aec886624116dc079afa027ecc8ad4a7e62010f8a \ + --hash=sha256:c1747f85cee84c26985853c6f3d9bd3e75da5212912443fa111c113b9c246f39 \ + --hash=sha256:c1b3f518abeca3aa13c712fd202306e145abf59a18b094a6bafb2d2bbf59192c \ + --hash=sha256:c50f2528cf200c5eed56faf3f4e22fcd5f38c157a8b78576e6ba3168ec35f000 \ + --hash=sha256:c68fcd102d71ea85c5b2dfac3f4f8476eff42a9e078fd5faefff6d145063536b \ + --hash=sha256:c7a7bd4e39e8e4c12c39cd480356842b6a8a06e41b23a55a5e3e191718838ddf \ + --hash=sha256:c94f0688e7b8d0a67abf40e57a7eaaecd17cc9586706a31b76c031f63df052b4 \ + --hash=sha256:cbaf13819775b7f769bf4a1f066cb6df7a28d4480081a589828ef190226881cd \ + --hash=sha256:cd2213145bcc2ba85884d0ac63d222fece9209678f77b9b4d76f054c561adb28 \ + --hash=sha256:ce5c1d2a8b27468f433ca974829c44060b8097eedc39933e3c206a90ee49c4a9 \ + --hash=sha256:d396ec2b979760aaf3218e76c24e65bd0aca24983298653b3a9d7a45f9e47b30 \ + --hash=sha256:d51026d73fcfd93610abc7b27789c26b313920fcfb20e27462d74a7f8b06e983 \ + --hash=sha256:d80ee3d731373b24cebbc10d689ca4ee1875caf0d5703a245db18efd4dd37fc1 \ + --hash=sha256:d995260fdf4e1db774581b4900e0f832abe3c7c84996726bbc161b19c8f29e76 \ + --hash=sha256:da4b951fe36dc7c3a1ccb4e3cd1747c3542b8c9ceede8fc86cae054e764485f5 \ + --hash=sha256:daa27d92c36f24388fe3ad306b174781c747627f134452e4f128ea00ce1fe8c4 \ + --hash=sha256:db06ffe51636ffe9ca531fe9023dd64bdd794be8754cb5df57c5498ae5b518a7 \ + --hash=sha256:e0d65b8c354be7fb5f720c3caa8bc940bc2d20ce749c8e06135f07f8ed95dd7c \ + --hash=sha256:e68b7a074f65a2fd746c52a7ce6142ab7006074ac269ace0c25cd8ba171f8066 \ + --hash=sha256:e739fee756ba1010f8bcccb534252e85a35fe45ae92c295a06059ce58b74ccd3 \ + --hash=sha256:e846ae7835bf0703ae43f534ab79a867146dadd59dc9ca5c8b53d5c8f7c9ef02 \ + --hash=sha256:e9c26f834c65f5752f3f06cb08cb86a913ceb7274d0db6e267808a708b46bc89 \ + --hash=sha256:ea793e075b70290d89d8142074262885d3f7da19634845135751bd6344f73b50 \ + --hash=sha256:f027324c56cd5406ca49c124b0db10e56c69064fec039acc571c29020cc87c76 \ + --hash=sha256:f13a646d65d09fbf1bc6b3a9635d30095c8e7e5cc419ff35ecc563c5fd04cd49 \ + --hash=sha256:f47286a97f0bc9b8859519809077b91b2cefe4ae47fcbf5e466a009c1c5d742b \ + --hash=sha256:f747929cf940cddb5b3668a390056ddd5ba2e5010615ea2dcf4f9c4f3ab8791d \ + --hash=sha256:f99626688942fb746e545232e7726926f3be91b5975f8b55327665fafda991c7 \ + --hash=sha256:f9fa868638bf362d3d138ea55829cefb3d5f4b0d7f142234382a15e2485dbec4 \ + --hash=sha256:fbdb89b3e1c94a30cc5edfce477c6e6a5dc4d8f84665b455c27582f211a1c72c \ + --hash=sha256:fc010ab034c8c7452522748bf937df58020d256ccae0874463d1f4d01758af8e \ + --hash=sha256:fc3e9034a63de20e15e8ade85358bc6efc614008cab72898b4b4952bea0509ff \ + --hash=sha256:fd8b3d9fd264be37976686c7f65cd52a83f5e84f4bfd2adf9c1d469676bbb6ae +pygments==2.20.0 \ + --hash=sha256:6757cd03768053ff99f3039c1a36d6c0aa0b263438fcab17520b30a303a82b5f \ + --hash=sha256:81a9e26dd42fd28a23a2d169d86d7ac03b46e2f8b59ed4698fb4785f946d0176 +pyjwt==2.13.0 \ + --hash=sha256:41571c89ca91598c79e8ef18a2d07367d4810fbbd6f637794879baf1b7703423 \ + --hash=sha256:66adcc2aff09b3f1bbd95fc1e1577df8ac8723c978552fd43304c8a290ac5728 +pymupdf==1.28.2 \ + --hash=sha256:2e1b574c0fd2cb238021033fd3c0f9c4388816638df064e4bfb56d9d81736dc8 \ + --hash=sha256:3050a233dde1211efe89ada74e2add6238436434159f46097a1423aad2842545 \ + --hash=sha256:397d6715c1f0df7548a92d0afd8ce370fc48fa47aeefac16be2bc04a16a8227f \ + --hash=sha256:530ef543a3885b3b81cb72a854e7c5a625a9233201221132bb6c31698c6a2bdb \ + --hash=sha256:5e0be7908a715aa20333caddd73f1d6f01e4cd0c26e869fa2dd0b7f344da2249 \ + --hash=sha256:5fc315b425ff1f7afdd1ea2f348205cb19b806767daae7ce4d64115799c2bae1 \ + --hash=sha256:7113846b35dbf0a033f088e4f4fb543dabeb4b0b12c112966a1ca1ee2d5eacae \ + --hash=sha256:ebd244918798502d7b4504c90410d1711a4d7675a32584ca30f1bab419ecbffe \ + --hash=sha256:f89fb2d86d07d643a269f17a093105057e20c79c1d06c103b53600067b6d2b01 \ + --hash=sha256:fd481ed48bef56305c41fb7e05a055c03345c899c7b101dad086258b438f8168 \ + --hash=sha256:ffe91a24edc75c80da2a4b62f50fc0f54632d34fc8fe4cbc48e5c7ff07cf8fb4 +pytest==9.1.1 \ + --hash=sha256:1088fbde8f2b49d95a549a195707afa7a76a3ce9bcadc26b6d71f0ffda5fe313 \ + --hash=sha256:37a86b45efb9a47a61a36449063e8e18d0cab3161329fc099eb21783169c4f0c +pytest-asyncio==1.4.0 \ + --hash=sha256:933ca923a23075a87fb7070c0ec272a6848489824d887c85c812670932835aa1 \ + --hash=sha256:c6c0d2259945122819f171a32ecea2c349ead889ee28176caaf492143424be42 +python-dotenv==1.2.2 \ + --hash=sha256:1d8214789a24de455a8b8bd8ae6fe3c6b69a5e3d64aa8a8e5d68e694bbcb285a \ + --hash=sha256:2c371a91fbd7ba082c2c1dc1f8bf89ca22564a087c2c287cd9b662adde799cf3 +python-multipart==0.0.32 \ + --hash=sha256:be54b7f3fa167bb83e4fcd936b887b708f4e57fe75911c02aebf53efaf8d938e \ + --hash=sha256:ff6d3f776f16878c894e52e107296ffc890e913c611b1a4ec6c44e2821fe2e23 +pywin32-ctypes==0.2.3 ; sys_platform == 'win32' \ + --hash=sha256:8a1513379d709975552d202d942d9837758905c8d01eb82b8bcc30918929e7b8 \ + --hash=sha256:d162dc04946d704503b2edc4d55f3dba5c1d539ead017afa00142c38b9885755 +pyyaml==6.0.3 \ + --hash=sha256:00c4bdeba853cc34e7dd471f16b4114f4162dc03e6b7afcc2128711f0eca823c \ + --hash=sha256:0150219816b6a1fa26fb4699fb7daa9caf09eb1999f3b70fb6e786805e80375a \ + --hash=sha256:02893d100e99e03eda1c8fd5c441d8c60103fd175728e23e431db1b589cf5ab3 \ + --hash=sha256:02ea2dfa234451bbb8772601d7b8e426c2bfa197136796224e50e35a78777956 \ + --hash=sha256:0f29edc409a6392443abf94b9cf89ce99889a1dd5376d94316ae5145dfedd5d6 \ + --hash=sha256:10892704fc220243f5305762e276552a0395f7beb4dbf9b14ec8fd43b57f126c \ + --hash=sha256:16249ee61e95f858e83976573de0f5b2893b3677ba71c9dd36b9cf8be9ac6d65 \ + --hash=sha256:1d37d57ad971609cf3c53ba6a7e365e40660e3be0e5175fa9f2365a379d6095a \ + --hash=sha256:1ebe39cb5fc479422b83de611d14e2c0d3bb2a18bbcb01f229ab3cfbd8fee7a0 \ + --hash=sha256:214ed4befebe12df36bcc8bc2b64b396ca31be9304b8f59e25c11cf94a4c033b \ + --hash=sha256:2283a07e2c21a2aa78d9c4442724ec1eb15f5e42a723b99cb3d822d48f5f7ad1 \ + --hash=sha256:22ba7cfcad58ef3ecddc7ed1db3409af68d023b7f940da23c6c2a1890976eda6 \ + --hash=sha256:27c0abcb4a5dac13684a37f76e701e054692a9b2d3064b70f5e4eb54810553d7 \ + --hash=sha256:28c8d926f98f432f88adc23edf2e6d4921ac26fb084b028c733d01868d19007e \ + --hash=sha256:2e71d11abed7344e42a8849600193d15b6def118602c4c176f748e4583246007 \ + --hash=sha256:34d5fcd24b8445fadc33f9cf348c1047101756fd760b4dacb5c3e99755703310 \ + --hash=sha256:37503bfbfc9d2c40b344d06b2199cf0e96e97957ab1c1b546fd4f87e53e5d3e4 \ + --hash=sha256:3c5677e12444c15717b902a5798264fa7909e41153cdf9ef7ad571b704a63dd9 \ + --hash=sha256:3ff07ec89bae51176c0549bc4c63aa6202991da2d9a6129d7aef7f1407d3f295 \ + --hash=sha256:41715c910c881bc081f1e8872880d3c650acf13dfa8214bad49ed4cede7c34ea \ + --hash=sha256:418cf3f2111bc80e0933b2cd8cd04f286338bb88bdc7bc8e6dd775ebde60b5e0 \ + --hash=sha256:44edc647873928551a01e7a563d7452ccdebee747728c1080d881d68af7b997e \ + --hash=sha256:4a2e8cebe2ff6ab7d1050ecd59c25d4c8bd7e6f400f5f82b96557ac0abafd0ac \ + --hash=sha256:4ad1906908f2f5ae4e5a8ddfce73c320c2a1429ec52eafd27138b7f1cbe341c9 \ + --hash=sha256:501a031947e3a9025ed4405a168e6ef5ae3126c59f90ce0cd6f2bfc477be31b7 \ + --hash=sha256:5190d403f121660ce8d1d2c1bb2ef1bd05b5f68533fc5c2ea899bd15f4399b35 \ + --hash=sha256:5498cd1645aa724a7c71c8f378eb29ebe23da2fc0d7a08071d89469bf1d2defb \ + --hash=sha256:5cf4e27da7e3fbed4d6c3d8e797387aaad68102272f8f9752883bc32d61cb87b \ + --hash=sha256:5e0b74767e5f8c593e8c9b5912019159ed0533c70051e9cce3e8b6aa699fcd69 \ + --hash=sha256:5ed875a24292240029e4483f9d4a4b8a1ae08843b9c54f43fcc11e404532a8a5 \ + --hash=sha256:5fcd34e47f6e0b794d17de1b4ff496c00986e1c83f7ab2fb8fcfe9616ff7477b \ + --hash=sha256:5fdec68f91a0c6739b380c83b951e2c72ac0197ace422360e6d5a959d8d97b2c \ + --hash=sha256:6344df0d5755a2c9a276d4473ae6b90647e216ab4757f8426893b5dd2ac3f369 \ + --hash=sha256:64386e5e707d03a7e172c0701abfb7e10f0fb753ee1d773128192742712a98fd \ + --hash=sha256:652cb6edd41e718550aad172851962662ff2681490a8a711af6a4d288dd96824 \ + --hash=sha256:66291b10affd76d76f54fad28e22e51719ef9ba22b29e1d7d03d6777a9174198 \ + --hash=sha256:66e1674c3ef6f541c35191caae2d429b967b99e02040f5ba928632d9a7f0f065 \ + --hash=sha256:6adc77889b628398debc7b65c073bcb99c4a0237b248cacaf3fe8a557563ef6c \ + --hash=sha256:79005a0d97d5ddabfeeea4cf676af11e647e41d81c9a7722a193022accdb6b7c \ + --hash=sha256:7c6610def4f163542a622a73fb39f534f8c101d690126992300bf3207eab9764 \ + --hash=sha256:7f047e29dcae44602496db43be01ad42fc6f1cc0d8cd6c83d342306c32270196 \ + --hash=sha256:8098f252adfa6c80ab48096053f512f2321f0b998f98150cea9bd23d83e1467b \ + --hash=sha256:850774a7879607d3a6f50d36d04f00ee69e7fc816450e5f7e58d7f17f1ae5c00 \ + --hash=sha256:8d1fab6bb153a416f9aeb4b8763bc0f22a5586065f86f7664fc23339fc1c1fac \ + --hash=sha256:8da9669d359f02c0b91ccc01cac4a67f16afec0dac22c2ad09f46bee0697eba8 \ + --hash=sha256:8dc52c23056b9ddd46818a57b78404882310fb473d63f17b07d5c40421e47f8e \ + --hash=sha256:9149cad251584d5fb4981be1ecde53a1ca46c891a79788c0df828d2f166bda28 \ + --hash=sha256:93dda82c9c22deb0a405ea4dc5f2d0cda384168e466364dec6255b293923b2f3 \ + --hash=sha256:96b533f0e99f6579b3d4d4995707cf36df9100d67e0c8303a0c55b27b5f99bc5 \ + --hash=sha256:9c57bb8c96f6d1808c030b1687b9b5fb476abaa47f0db9c0101f5e9f394e97f4 \ + --hash=sha256:9c7708761fccb9397fe64bbc0395abcae8c4bf7b0eac081e12b809bf47700d0b \ + --hash=sha256:9f3bfb4965eb874431221a3ff3fdcddc7e74e3b07799e0e84ca4a0f867d449bf \ + --hash=sha256:a33284e20b78bd4a18c8c2282d549d10bc8408a2a7ff57653c0cf0b9be0afce5 \ + --hash=sha256:a80cb027f6b349846a3bf6d73b5e95e782175e52f22108cfa17876aaeff93702 \ + --hash=sha256:b30236e45cf30d2b8e7b3e85881719e98507abed1011bf463a8fa23e9c3e98a8 \ + --hash=sha256:b3bc83488de33889877a0f2543ade9f70c67d66d9ebb4ac959502e12de895788 \ + --hash=sha256:b865addae83924361678b652338317d1bd7e79b1f4596f96b96c77a5a34b34da \ + --hash=sha256:b8bb0864c5a28024fac8a632c443c87c5aa6f215c0b126c449ae1a150412f31d \ + --hash=sha256:ba1cc08a7ccde2d2ec775841541641e4548226580ab850948cbfda66a1befcdc \ + --hash=sha256:bdb2c67c6c1390b63c6ff89f210c8fd09d9a1217a465701eac7316313c915e4c \ + --hash=sha256:c1ff362665ae507275af2853520967820d9124984e0f7466736aea23d8611fba \ + --hash=sha256:c2514fceb77bc5e7a2f7adfaa1feb2fb311607c9cb518dbc378688ec73d8292f \ + --hash=sha256:c3355370a2c156cffb25e876646f149d5d68f5e0a3ce86a5084dd0b64a994917 \ + --hash=sha256:c458b6d084f9b935061bc36216e8a69a7e293a2f1e68bf956dcd9e6cbcd143f5 \ + --hash=sha256:d0eae10f8159e8fdad514efdc92d74fd8d682c933a6dd088030f3834bc8e6b26 \ + --hash=sha256:d76623373421df22fb4cf8817020cbb7ef15c725b9d5e45f17e189bfc384190f \ + --hash=sha256:ebc55a14a21cb14062aa4162f906cd962b28e2e9ea38f9b4391244cd8de4ae0b \ + --hash=sha256:eda16858a3cab07b80edaf74336ece1f986ba330fdb8ee0d6c0d68fe82bc96be \ + --hash=sha256:ee2922902c45ae8ccada2c5b501ab86c36525b883eff4255313a253a3160861c \ + --hash=sha256:efd7b85f94a6f21e4932043973a7ba2613b059c4a000551892ac9f1d11f5baf3 \ + --hash=sha256:f7057c9a337546edc7973c0d3ba84ddcdf0daa14533c2065749c9075001090e6 \ + --hash=sha256:fa160448684b4e94d80416c0fa4aac48967a969efe22931448d853ada8baf926 \ + --hash=sha256:fc09d0aa354569bc501d4e787133afc08552722d3ab34836a80547331bb5d4a0 +regex==2026.7.19 \ + --hash=sha256:062f8cb7a9739c4835d22bd96f370c59aba89f257adcfa53be3cc209e08d3ae0 \ + --hash=sha256:064f1760a5a4ade65c5419be23e782f29147528e8a66e0c42dd4cedb8d4e9fc6 \ + --hash=sha256:09523a592938aa9f587fb74467c63ff0cf88fc3df14c82ab0f0517dcf76aaa62 \ + --hash=sha256:09d3007fc76249a83cdd33de160d50e6cb77f54e09d8fa9e7148e10607ce24af \ + --hash=sha256:09f3e5287f94f17b709dc9a9e70865855feee835c861613be144218ce4ca82cc \ + --hash=sha256:0c41c63992bf1874cebb6e7f56fd7d3c007924659a604ae3d90e427d40d4fd13 \ + --hash=sha256:0e9554c8785eac5cffe6300f69a91f58ba72bc88a5f8d661235ad7c6aa5b8ccd \ + --hash=sha256:1123ef4211d763ee771d47916a1596e2f4915794f7aabdc1adcb20e4249a6951 \ + --hash=sha256:15b364b9b98d6d2fe1a85034c23a3180ff913f46caddc3895f6fd65186255ccc \ + --hash=sha256:1649eb39fcc9ea80c4d2f110fde2b8ab2aef3877b98f02ab9b14e961f418c511 \ + --hash=sha256:17ed5692f6acc4183e98331101a5f9e4f64d72fe58b753da4d444a2c77d05b12 \ + --hash=sha256:199535629f25caf89698039af3d1ad5fcae7f933e2112c73f1cdf49165c99518 \ + --hash=sha256:1c398716054621aa300b3d411f467dda903806c5da0df6945ab73982b8d115db \ + --hash=sha256:1d3372064506b94dd2c67c845f2db8062e9e9ba84d04e33cb96d7d33c11fe1ae \ + --hash=sha256:1d58561843f0ff7dc78b4c28b5e2dc388f3eff94ebc8a232a3adba961fc00009 \ + --hash=sha256:1d793a7988e04fcb1e2e135567443d82173225d657419ec09414a9b5a145b986 \ + --hash=sha256:1ebac3474b8589fce2f9b225b650afd61448f7c73a5d0255a10cc6366471aed1 \ + --hash=sha256:20568e182eb82d39a6bf7cff3fd58566f14c75c6f74b2c8c96537eecf9010e3a \ + --hash=sha256:22a992de9a0d91bda927bf02b94351d737a0302905432c88a53de7c4b9ce62e2 \ + --hash=sha256:2955907b7157a6660f27079edf7e0229e9c9c5325c77a2ef6a890cba91efa6f0 \ + --hash=sha256:2c4e61e2e1be56f63ec3cc618aa9e0de81ef6f43d177205451840022e24f5b78 \ + --hash=sha256:2cc3460cedf7579948486eab03bc9ad7089df4d7281c0f47f4afe03e8d13f02d \ + --hash=sha256:2ce9e679f776649746729b6c86382da519ef649c8e34cc41df0d2e5e0f6c36d4 \ + --hash=sha256:2ef7eeb108c47ce7bcc9513e51bcb1bf57e8f483d52fce68a8642e3527141ae0 \ + --hash=sha256:3080a7fd38ef049bd489e01c970c97dd84ff446a885b0f1f6b26d9b1ad13ce11 \ + --hash=sha256:343a4504e3fb688c47cad451221ca5d4814f42b1e16c0065bde9cbf7f473bd52 \ + --hash=sha256:36aacfb15faaff3ced55afbf35ec72f50d4aee22082c4f7fe0573a33e2fca92e \ + --hash=sha256:3d3143f159261b1ce5b24c261c590e5913370c3200c5e9ebbb92b5aa5e111902 \ + --hash=sha256:40b34dd88658e4fedd2fddbf0275ac970d00614b731357f425722a3ed1983d11 \ + --hash=sha256:4458124d71339f505bf1fb94f69fd1bb8fa9d2481eebfef27c10ef4f2b9e12f6 \ + --hash=sha256:4896db1f4ce0576765b8272aa922df324e0f5b9bb2c3d03044ff32a7234a9aba \ + --hash=sha256:4a0530bb1b8c1c985e7e2122e2b4d3aedd8a3c21c6bfddae6767c4405668b56e \ + --hash=sha256:4aa5435cdb3eb6f55fe98a171b05e3fbcd95fadaa4aa32acf62afd9b0cfdbcac \ + --hash=sha256:4c3501bfa814ab07b5580741f9bf78dfdfe146a04057f82df9e2402d2a975939 \ + --hash=sha256:4e5413bd5f13d3a4e3539ca98f70f75e7fca92518dd7f117f030ebedd10b60cb \ + --hash=sha256:4e6883a021db30511d9fb8cfb0f222ce1f2c369f7d4d8b0448f449a93ba0bdfc \ + --hash=sha256:52579c60a6078be70a0e49c81d6e56d677f34cd439af281a0083b8c7bc75c095 \ + --hash=sha256:555497390743af1a65045fa4527782d10ff5b88970359412baa4a1e628fe393b \ + --hash=sha256:56ad4d9f77df871a99e25c37091052a02528ec0eb059de928ee33956b854b45b \ + --hash=sha256:571fde9741eb0ccde23dd4e0c1d50fbae910e901fa7e629faf39b2dda740d220 \ + --hash=sha256:572fc57b0009c735ee56c175ea021b637a15551a312f56734277f923d6fd0f6c \ + --hash=sha256:59787bd5f8c70aa339084e961d2996b53fbdeab4d5393bba5c1fe1fc32e02bae \ + --hash=sha256:5a2721c8720e2cb3c209925dfb9200199b4b07361c9e01d321719404b21458b3 \ + --hash=sha256:5cc26a66e212fa5d6c6170c3a40d99d888db3020c6fdab1523250d4341382e44 \ + --hash=sha256:5ebee1ee89c39c953baac6924fcde08c5bb427c4057510862f9d7c7bdb3d8665 \ + --hash=sha256:60be8693a1dadc210bbcbc0db3e26da5f7d01d1d5a3da594e99b4fa42df404f5 \ + --hash=sha256:618a0aed532be87294c4477b0481f3aa0f1520f4014a4374dd4cf789b4cd2c97 \ + --hash=sha256:61bb1bd45520aacd56dd80943bd34991fb5350afdd1f36f2282230fd5154a218 \ + --hash=sha256:6383cd2ed53a646c659ba1fe65727db76437fdaa069e697a0b44a51d5843d864 \ + --hash=sha256:64729333167c2dcaaa56a331d40ee097bd9c5617ffd51dabb09eaddafb1b532e \ + --hash=sha256:64b6ca7391a1395c2638dd5c7456d67bea44fc6c5e8e92c5dc8aa6a8f23292b4 \ + --hash=sha256:65dcd28d3eba2ab7c2fd906485cc301392b47cc2234790d27d4e4814e02cdfda \ + --hash=sha256:65fa6cb38ed5e9c3637e68e544f598b39c3b86b808ed0627a67b68320384b459 \ + --hash=sha256:66bd62c59a5427746e8c44becae1d9b99d22fb13f30f492083dfb9ad7c45cc18 \ + --hash=sha256:6e44c0e7c5664be20aee92085153150c0a7967310a73a43c0f832b7cd35d0dd3 \ + --hash=sha256:6f8c6e7a1cfa3dc9d0ee2de0e65e834537fa29992cc3976ffec914afc35c5dd5 \ + --hash=sha256:7322ec6cc9fba9d49ab888bb82d67ac5625627aa168f0165139b17018df3fb8a \ + --hash=sha256:73b133a9e6fb512858e7f065e96f1180aa46646bc74a83aea62f1d314f3dd035 \ + --hash=sha256:73f272fba87b8ccfe70a137d02a54af386f6d27aa509fbffdd978f5947aae1aa \ + --hash=sha256:7e77b324909c1617cbb4c668677e2c6ae13f44d7c1de0d4f15f2e3c10f3315b5 \ + --hash=sha256:80115dd39481fd3a4b4080220799dbcacb921a844de4b827264ececacbe17c78 \ + --hash=sha256:87ccab0db8d5f4fbb0272642113c1adb2ffc698c16d3a0944580222331fa7a20 \ + --hash=sha256:89dfee3319f5ae3f75ebd5c2445a809bb320252ba5529ffdafea4ef25d79cf1a \ + --hash=sha256:8ac59a0900474a52b7c04af8196affc22bd9842acb0950df12f7b813e983609a \ + --hash=sha256:8cae6fd77a5b72dae505084b1a2ee0360139faf72fedbab667cd7cc65aae7a6a \ + --hash=sha256:8d3469c91dd92ee41b7c95280edbd975ef1ba9195086686623a1c6e8935ce965 \ + --hash=sha256:90c633e7e8d6bf4e992b8b36ce69e018f834b641dd6de8cea6d78c06ffa119c5 \ + --hash=sha256:93db40c8de0815baab96a06e08a984bac71f989d13bab789e382158c5d426797 \ + --hash=sha256:9724e6cb5e478cd7d8cabf027826178739cb18cf0e117d0e32814d479fa02276 \ + --hash=sha256:98c6ac18480fcdb33f35439183f1d2e79760ab41930309c6d951cb1f8e46694c \ + --hash=sha256:9a15e785f244f3e07847b984ce8773fc3da10a9f3c131cc49a4c5b4d672b4547 \ + --hash=sha256:9b60d7814174f059e5de4ab98271cc5ba9259cfea55273a81544dceea32dc8d9 \ + --hash=sha256:9be2a6647740dd3cca6acb24e87f03d7632cd280dbce9bbe40c26353a215a45d \ + --hash=sha256:9c7472192ebfad53a6be7c4a8bfb2d64b81c0e93a1fc8c57e1dd0b638297b5d1 \ + --hash=sha256:9dce8ec9695f531a1b8a6f314fd4b393adcccf2ea861db480cdf97a301d01a68 \ + --hash=sha256:9e50d748a32da622f256e8d505867f5d3c43a837c6a9f0efb149655fadd1042a \ + --hash=sha256:a81758ed242b861b72e778ba34d41366441a2e10b16b472784c88da2dea7e2dd \ + --hash=sha256:ac777001cdfc28b72477d93c8564bb7583081ea8fb45cdca3d568e0a4f87183c \ + --hash=sha256:b2b506b1788df5fecd270a10d5e70a95fe77b87ea2b370a318043f6f5f817ee6 \ + --hash=sha256:b2ea4a3e8357be8849e833beeae757ac3c7a6b3fc055c03c808a53c91ad30d82 \ + --hash=sha256:bf1516fe58fc104f39b2d1dbe2d5e27d0cd45c4be2e42ba6ee0cc763701ec3c7 \ + --hash=sha256:c0d702548d89d572b2929879bc883bb7a4c4709efafe4512cadee56c55c9bd15 \ + --hash=sha256:c10b82c2634df08dfb13b1f04e38fe310d086ee092f4f69c0c8da234251e556e \ + --hash=sha256:c42572142ed0b9d5d261ba727157c426510da78e20828b66bbb855098b8a4e38 \ + --hash=sha256:c4585c3e64b4f9e583b4d2683f18f5d5d872b3d71dcf24594b74ecc23602fa96 \ + --hash=sha256:c639ea314df70a7b2811e8020448c75af8c9445f5a60f8a4ced81c306a9380c2 \ + --hash=sha256:c670fe7be5b6020b76bc6e8d2196074657e1327595bca93a389e1a76ab130ad8 \ + --hash=sha256:cc1b2440423a851fad781309dd87843868f4f66a6bcd1ddb9225cf4ec2c84732 \ + --hash=sha256:cd3584591ea4429026cdb931b054342c2bcf189b44ff367f8d5c15bc092a2966 \ + --hash=sha256:d15df07081d91b76ff20d43f94592ee110330152d617b730fdbe5ef9fb680053 \ + --hash=sha256:d19662dbedbe783d323196312d38f5ba53cf56296378252171985da6899887d3 \ + --hash=sha256:d24ecb4f5e009ea0bd275ee37ad9953b32005e2e5e60f8bbae16da0dbbf0d3a0 \ + --hash=sha256:d446c6ac40bb6e05025ccee55b84d80fe9bf8e93010ffc4bb9484f13d498835f \ + --hash=sha256:d51ffd3427640fa2da6ade574ceba932f210ad095f65fcc450a2b0a0d454868e \ + --hash=sha256:d6ce43a0269d68cee79a7d1ade7def53c20f8f2a047b92d7b5d5bcc73ae88327 \ + --hash=sha256:d721e53758b2cca74990185eb0671dd466d7a388a1a45d0c6f4c13cef41a68ac \ + --hash=sha256:d7da47a0f248977f08e2cb659ff3c17ddc13a4d39b3a7baa0a81bf5b415430f6 \ + --hash=sha256:db47b561c9afd884baa1f96f797c9ca369872c4b65912bc691cfa99e68340af2 \ + --hash=sha256:dbe6493fbd27321b1d1f2dd4f5c7e5bd4d8b1d7cab7f32fd67db3d0b2ed8248a \ + --hash=sha256:dbece16025afda5e3031af0c4059207e61dcf73ef13af844964f57f387d1c435 \ + --hash=sha256:ddd67571c10869f65a5d7dde536d1e066e306cc90de57d7de4d5f34802428bb5 \ + --hash=sha256:de9208bb427130c82a5dbfd104f92c8876fc9559278c880b3002755bbbe9c83d \ + --hash=sha256:e30d40268a28d54ce0437031750497004c22602b8e3ab891f759b795a003b312 \ + --hash=sha256:e8b0abe7d870f53ca5143895fef7d1041a0c831a140d3dc2c760dd7ba25d4a8b \ + --hash=sha256:f035d9dc1d25eff9d361456572231c7d27b5ccd473ca7dc0adfce732bd006d40 \ + --hash=sha256:f04b9f56b0e0614c0126be12c2c2d9f8850c1e57af302bd0a63bed379d4af974 \ + --hash=sha256:f0fa4fa9c3632d708742baf2282f2055c11d888a790362670a403cbf48a2c404 \ + --hash=sha256:f2e7f8e2ab6c2922be02c7ec45185aa5bd771e2e57b95455ee343a44d8130dff \ + --hash=sha256:f8f6fa298bb4f7f58a33334406218ba74716e68feddf5e4e54cd5d8082705abf \ + --hash=sha256:fbf300e2070bb35038660b3be1be4b91b0024edb41517e6996320b49b92b4175 \ + --hash=sha256:fce7760bf283405b2c7999cab3da4e72f7deca6396013115e3f7a955db9760da \ + --hash=sha256:fcee38cd8e5089d6d4f048ba1233b3ad76e5954f545382180889112ff5cb712d \ + --hash=sha256:fe31f28c94402043161876a258a9c6f757cb485905c7614ce8d6cd40e6b7bdc1 \ + --hash=sha256:ffd8893ccc1c2fce6e0d6ca402d716fe1b29db70c7132609a05955e31b2aa8f2 +requests==2.34.2 \ + --hash=sha256:2a0d60c172f83ac6ab31e4554906c0f3b3588d37b5cb939b1c061f4907e278e0 \ + --hash=sha256:f288924cae4e29463698d6d60bc6a4da69c89185ad1e0bcc4104f584e960b9ed +requests-toolbelt==1.0.0 \ + --hash=sha256:7681a0a3d047012b5bdc0ee37d7f8f07ebe76ab08caeccfc3921ce23c88d5bc6 \ + --hash=sha256:cccfdd665f0a24fcf4726e690f65639d272bb0637b9b92dfd91a5568ccf6bd06 +ruff==0.16.2 \ + --hash=sha256:20e66910f2c37cc753f9ef6580c914a621b80c4fa3549d3e3521e29d0f5bfc3f \ + --hash=sha256:335c621622c4650330be50842561c6586ac6971bb8ab5407fe34dcc9efb16bbe \ + --hash=sha256:3c8de4cf2181f01d57946d87d777aa52916976fc09942aed89938fab5e013318 \ + --hash=sha256:40ea8c0594feb894e89c8c61ab9c103d38b0ea72dfde6c594107147ca31b1140 \ + --hash=sha256:4ce4e02bad779bef557f541a1b31f20d6abeae1cc05ed1b1ac019d4ffd1044c8 \ + --hash=sha256:81806b08329130005dd4a8a8394a0c9da8c6f4cafb16ba438d2a2ee6a18bedf1 \ + --hash=sha256:9a48cc05c6fbc811ca81b5d7ba95375affea6582d1b8024e455e41afbbf55344 \ + --hash=sha256:a2c0d14fcbb26c91f0f867a6dc9bd71bbc30b1b6151829c884f23faeab2e5700 \ + --hash=sha256:ab3d62dde0b19facdd632008cc4827fc28ada7736c6bd35ab6f1050f0bfed53f \ + --hash=sha256:bcabe2f6d0fc7819f1431793005af4e4de7371927d037345bf941252b195b9fa \ + --hash=sha256:bf3a63d78fb39f4bf5ac8ae52051c5520505301abe19ba4e204c453b3f09bb0b \ + --hash=sha256:c24788a980581e1d7ea3a0cbe4344c4fbeb0a6a9b1f4713aa46bb104f8294690 \ + --hash=sha256:c3d7828d12e8927a6fc65fe38e2c2541b9e762d360a1786d752cb1b8883b3c9c \ + --hash=sha256:c7e36fbfba65510548156902bcf1350a979a958ce0347ce0f90d73894036b39f \ + --hash=sha256:d614e95cedf38a2053fd351c55b103ba30d017d61688fdbfd40ee0412852a99f \ + --hash=sha256:e0422abdf70070255fc4073ce9dfc814cc03db577013761ddd09bc1e4a9a4fbd \ + --hash=sha256:e43e1f5b8388da9eca1b9e88328d47a5cec794633ccf6f7484ac2dd15eee92c0 \ + --hash=sha256:f0eab35f80df8f134aae5d1630e751901321d317cc8e50dc39e36fa3ed34cd12 +secretstorage==3.5.0 ; sys_platform == 'linux' \ + --hash=sha256:0ce65888c0725fcb2c5bc0fdb8e5438eece02c523557ea40ce0703c266248137 \ + --hash=sha256:f04b8e4689cbce351744d5537bf6b1329c6fc68f91fa666f60a380edddcd11be +sniffio==1.3.1 \ + --hash=sha256:2f6da418d1f1e0fddd844478f41680e794e6051915791a034ff65e5f100525a2 \ + --hash=sha256:f4324edc670a0f49750a81b895f35c3adb843cca46f0530f79fc1babb23789dc +starlette==1.6.0 \ + --hash=sha256:a86dd39d14bb45f85a3d18525215a9ef0cfd1f192ac793220e72598c90335f0c \ + --hash=sha256:d4e3ac5e546444960c710297a3c9fc3f7ebae1b7e963f3d36173b49da535be9b +structlog==26.1.0 \ + --hash=sha256:e081a26d6c373e6d201eca24eede26d8ffab07f88f477822e679183428d3d91e \ + --hash=sha256:f63a716cbd1b1291cf7661de7794b455acfa4c43c5bcf1630e6ad5ddc1adb3b7 +tenacity==9.1.4 \ + --hash=sha256:6095a360c919085f28c6527de529e76a06ad89b23659fa881ae0649b867a9d55 \ + --hash=sha256:adb31d4c263f2bd041081ab33b498309a57c77f9acf2db65aadf0898179cf93a +tiktoken==0.13.0 \ + --hash=sha256:059c8ecf554eb5b41e6e054ba467b871b03277d267dee7244380aca4359747d4 \ + --hash=sha256:115c4f26ffa11caac8b54eea35c2ad38c612c20a48d35dd15d70a02ac6f51f58 \ + --hash=sha256:125bc05005e747f993a83dc67934249932d6e4209854452cd4c0b1d53fba3ba2 \ + --hash=sha256:165cf1820ea4a354985c2490a5205d4cc74661c934aca79dd0368232fff94e0f \ + --hash=sha256:2a3b536c55802fe42f4b4644d2be4f04bf788506b48de0a0a658cb58f8bce232 \ + --hash=sha256:2b920b35805cd64585a37c3dc7ce65fba4d2d36016be01e1d7942482ca29093a \ + --hash=sha256:2c397ddda233208345b01bd30f2fca79ff730e55731d0108a603f9bc57f6af3b \ + --hash=sha256:303f7d91b4fce3baddbcde05c139091d4caa5026ac7214c1dc7ff7a71ee429ff \ + --hash=sha256:32ac870a806cfb260a02d0cb70426aef02e038297f8ad50df5040bb5af360791 \ + --hash=sha256:32e0c12305105002c047b3bb1070b0dd9a73b0cb3b2856a8972b810e7a4f5881 \ + --hash=sha256:35e1ea1e0631c04f551297284a1ab7e1f65a3c55a9a48728d5e0f66b4527c04a \ + --hash=sha256:36217497eaffc158607a3b26f065300db2aefd43b115263f3b9688ce38146173 \ + --hash=sha256:3f277ebea5edd7b8bf03c6f9431e1d67d517530115572b2dc1d465326e8f88c7 \ + --hash=sha256:43cee3e5400573b2046fbf092cc7a5bc30164f9e4c95ce20714da929df48737a \ + --hash=sha256:44733b99bfd72b590cd0936b1c01b3b4dd73122db2d544bc1ceeb18a7678c910 \ + --hash=sha256:472527e9132952f2fbf77cd290658bacf003d4d5a3fabc18e5fbd407cbae4d9b \ + --hash=sha256:477c9a38e20d0ed248090509acf1e839ad3967a4f00b4b0f958210049f656dee \ + --hash=sha256:47b1df8d73390a24f94980c75158cdd5c56d256f16d55f30cb49c230caba9ba4 \ + --hash=sha256:493af3aa28a4aaf2e3d2600a2ee717252c9bf5ab38fff94eb5a02db5ab77e5ad \ + --hash=sha256:4d9980f11429ed2d737c463bb1fb78cf330caa026adf002f714aced7849a687b \ + --hash=sha256:4e2f67d27c9626cdd25fe33d9313c5cdb3d8d82da646b68d6eb8e7e9c20e6448 \ + --hash=sha256:51384448aa508e4df84c0f7c1dc3211c7f7b8096325660ee5fc82f3e11b381ce \ + --hash=sha256:5ba5fd62507a932d1241346179e3b39bc7bf7408f03c272652d93b3bedf5db24 \ + --hash=sha256:5cb65b60b9408563676d874a3a4ee573370066f0dc4e29d84e82e989c6517424 \ + --hash=sha256:5d48843bee149630eb735a99e1f4a85b47308d21868ea63163f6e87768d3cfed \ + --hash=sha256:5df5d1507bd245f1ccad4a074698240021239e455eb0bb4ced4e3d7181872154 \ + --hash=sha256:5e6358911cab4adee6712da27d65573496a4f68cf8a2b5fca6a4ad10fc5748cf \ + --hash=sha256:6644c9c2b5cf3916f5a3641d7d12fdb3f006a7b3d9ff6acdaec44e29ab1ff91e \ + --hash=sha256:6b1615f0ff71953d19729ceb18865429c185b0a23c5353f1bbca34a394bf60f7 \ + --hash=sha256:6c43a675ca14f6f2749ba7f12075d37456015a24b859f2517b9beb4ef30807ec \ + --hash=sha256:6eb4a5bfbc6426938026b1a334e898ac53541360d62d8c689870160cc80abd67 \ + --hash=sha256:75ab9bc99fa020a4c283424590ecd7f3afd70c1c281cb3fa3192a6c3af9f9615 \ + --hash=sha256:7ab10f4a21c2999846940113f6dbd72e0fa06a24119feddd74cc47e85818e06d \ + --hash=sha256:7bfe1849caa65d1e1d9871817170ec497bbb7984e182012e1bdce72f66608cdb \ + --hash=sha256:7d40c6c5aab171dcd6eb8455bc567bde404bb9def60cdb8c1299cc782b242bb9 \ + --hash=sha256:7de52e3f566d19b3b11bd37eea552c6c305ad74081f736882bd44d148ed4c48d \ + --hash=sha256:85b78cc3a2c3d48723ca751fa981f1fedccd54194ca0471b957364353a898b07 \ + --hash=sha256:8f2d16e7a7c783ad81f36e457d046d1f1c8af70b22aec8a13238efe531977c41 \ + --hash=sha256:8fe806a50664e83a6ffd56cbd1e4f5dcc6cd32a3e7538f70dc38b1a271384545 \ + --hash=sha256:91c180fe255bd5a86d8316210d2833a1d4d33d026cd86a67812f4773743c8d26 \ + --hash=sha256:95097e4f89b06403976e498abf61a0ee73a7497e73fb599cb211d8197a054d91 \ + --hash=sha256:975cbd78d085d75d26b59660e262736dcaed1e35f8f142cd6291025c01d25486 \ + --hash=sha256:9b842981fa91accdffd48ff6408a977b7a91c3fbda55d353c3c68114d5c9d69e \ + --hash=sha256:9b8858b29804b3a0add25ce9e62fb00f89f621dc754d75d03ca419d17e8ddf67 \ + --hash=sha256:a116178fa7e1b4065bff05214360373a65cac22f965be7b3f73d00a0dbfe7649 \ + --hash=sha256:a2937ad042d49d50eac6e1ba07c5661d4bd3942a5b1e0c0d08475c4df83676e1 \ + --hash=sha256:b8ac2d6420ff05841a89ba5205c6d45f56c4f6843454f3c884b7eb1a2a8dddb2 \ + --hash=sha256:b967dfb9d0adf9a631953b1b40717684f04478270fc51bbccdd2f838d67a2f00 \ + --hash=sha256:c9435714c3a84c2319499de9a300c0e604449dd0799ff246458b3bb6a7f433c1 \ + --hash=sha256:ca8b310bd93b3772cb1b7922d915446864860f562bdfe4825c63a0aed3fb28cd \ + --hash=sha256:cb99cb5127449f58d0a2d5f5ccfb390d8dbdfd919c221246caaee29d8725ed51 \ + --hash=sha256:d108bc2d470fc53c8ecd24f2c0fd2b5f98c33e87cdb6aa2e9b8c5dced703d273 \ + --hash=sha256:da86f8c96ac1c235d7a3b3eebff1eacfdbcfb8ad792706943268d4d2938fbafe \ + --hash=sha256:e28157350f7ebf35008dd8e9e0fdb621f976e4230c881099c85e8cf07eaa50e2 \ + --hash=sha256:eaaaef47c2406277181d2086484c317bf7fc433e2d5d03ff94f56b0dcec87471 \ + --hash=sha256:ed5a30027cb4d8c7ca8b273d4766f3db3cf58fad9e9f3b1a68a351ffb54873d5 \ + --hash=sha256:fc1c44cd37b43fc46bae593129164f4f281e82ea116b57a85aa81bda57eafc94 +tqdm==4.70.0 \ + --hash=sha256:55b0b0dbd97462d06ebee91e4dac24ed4d4702be82b24f07e6c1d27e08cea220 \ + --hash=sha256:7f585706bfddbdebf89daac705b2dfcc16890130727d3197ca62c732b4310953 +typing-extensions==4.16.0 \ + --hash=sha256:481caa481374e813c1b176ada14e97f1f67a4539ce9cfeb3f350d78d6370c2e8 \ + --hash=sha256:dc983d19a509c94dba722ee6abd33940f7c05a89e243c47e907eb4db6f1a43e5 +typing-inspection==0.4.2 \ + --hash=sha256:4ed1cacbdc298c220f1bd249ed5287caa16f34d44ef4e9c3d0cbad5b521545e7 \ + --hash=sha256:ba561c48a67c5958007083d386c3295464928b01faa735ab8547c5692e87f464 +urllib3==2.7.0 \ + --hash=sha256:231e0ec3b63ceb14667c67be60f2f2c40a518cb38b03af60abc813da26505f4c \ + --hash=sha256:9fb4c81ebbb1ce9531cce37674bbc6f1360472bc18ca9a553ede278ef7276897 +uuid-utils==0.17.0 \ + --hash=sha256:03815cea572c8a693cab5475b9d750cc161470961c7defa27e9286cad62f38f5 \ + --hash=sha256:04452640d8b6920c480c16e5afe91ff896d236e0c972830f9247e0898d38c803 \ + --hash=sha256:09a55b7a5ae764985cb46467496a1787678d0a1400356157a080ad95b1a36869 \ + --hash=sha256:0ab4a66e7a035ad6625cfc1fbdb34f5c2d25a80ae1ef4bfee458ea2036333c6d \ + --hash=sha256:0bc4c431ccd59c764080ceb43b126043325fe17861b87759d026a0cdd8423bb2 \ + --hash=sha256:0f3729e839209f3457d0d8b6a35a376fdf65577a5aecaf4cc3587d3305759ba6 \ + --hash=sha256:0fcca4e838af9ac9243b3358d7c14afa4dca286a87781124c272d6c4cad9c968 \ + --hash=sha256:1019476b6bdc047216ef7414be5babe0fa5ccfde977c0cac4fd6c75ddec66ff7 \ + --hash=sha256:14dc2f46abb1091260c0d203fcbdf4e045042cc07e49183fd3b255904b95eb70 \ + --hash=sha256:1776a80d16369999b21627028cc5dbce819be83e1e079fdd7a51b587d2916db9 \ + --hash=sha256:1edf2f8732e4ed95bd7b65f2658f4aa072efaaff321144f4e0d4bf6a22709263 \ + --hash=sha256:1fd6f0e8a162dc0e9255b6aebe3cd175e76c33202f1bf39da9e6294b93db0099 \ + --hash=sha256:21c79b61ff750abcf057163dd764ccb6196cde7a26cda1b31b45cd97769e03b3 \ + --hash=sha256:220b52746d99e11964badac3c0869016e0c24bafb70a7dd5c2c072a6be3da9cc \ + --hash=sha256:237722b6581bb5b4eb4cefbcbe5c6e2980a440aabe781fbe50ebf1cb71eee4cc \ + --hash=sha256:239d8a281fe10bae33205b5d43185834d556b18434e0a113b5dc1dfb2fd97e91 \ + --hash=sha256:29179ffb7b317239b6d6afb100d14c439c728770460718280b9c0a42d2561ec2 \ + --hash=sha256:2db386941cfdecdd0b5a8ceeed5cf7479c83d1730dcf64a48d43cfa018cc3310 \ + --hash=sha256:2dd4a21baaac9a88486f0dd166c5793feb101a0bb9f006f2c401657fff5a1343 \ + --hash=sha256:309a35f12d99dde19032bc2259cda6431c85eeac0879134dc777cc3087d7e1cb \ + --hash=sha256:3150d836290c88f1d26eb59c4db280d87417dd3bfaadd2889c77416c8f0ff6fa \ + --hash=sha256:32abaafc8e91928b3d9f4d82e42d2094041e38ad6bb964066faadff28e4162f1 \ + --hash=sha256:32df1944808877702ceea398c103881c09a679bb672a215e01c2a84231266bf9 \ + --hash=sha256:344f7c755e280ea0ba6aeb08022190d867a80000b1715cacded54fc4b5633607 \ + --hash=sha256:351462debd866f1f25e4d4f5c7fac89525b52151f0102a1bdfe94a999b046f5f \ + --hash=sha256:375cde148430d60a4a07c03abaa0774c4fddfdd90de99b4ba02f24088bc9d750 \ + --hash=sha256:387cf7437c94ddec08651a0f1081381299c7075bc48a6251d8922bf39973378a \ + --hash=sha256:3dac0ad0cd9a2818d1775215365a4e8c2f8ada215529dd26f3f8cceeb67a6988 \ + --hash=sha256:405233a5f625b3d995648f4647fa6befa4567cf3f74e1f6b9837e16f7310f0e0 \ + --hash=sha256:4134353bfe3026ddab8e886002dc52bc5a0ab04611aabb0eaae23c32e6e57f64 \ + --hash=sha256:42275ebd0e8e74e32cdbfb8bd88fc99576567d51d54a508020611fd8f4f463a0 \ + --hash=sha256:4441600447d340ae103a353f01dbcd22ff680e5ee1a22988efe8d7b791d8fdb3 \ + --hash=sha256:46a73cacdf512f473a81f65dbf84186e08cfe6e9118fa582b6c6b33a8288a30d \ + --hash=sha256:4bf4d9cd1e80e73922073b9b27c143bedeb109d65f94cd12712e2c87118f2b7d \ + --hash=sha256:4e2ac1c0b56f2c91b6f158e29ed96b1503223fe8aa6e79b1be1dc55bd8a5131c \ + --hash=sha256:52db0e471d3d2632d35445af352591f40a8f32959a412981d9f51e068bb9514b \ + --hash=sha256:53ce348ef4c6e98c02c19c522af01334fe94476ce9af0db8c4482f9f142ae9c1 \ + --hash=sha256:5641071337eb11d61a001ea08793bf72216f3241f0a433ed2764804b2a3e3cc7 \ + --hash=sha256:5670c52a438e21483ce715776144914a4e2a2a5c62d9dee15f8a3e90cf128ae6 \ + --hash=sha256:56aa6488b931246fae11924e4bd0e2b32677e63945eecb71c29e3c2ca0dc3131 \ + --hash=sha256:570db214f6d8507587a8faa968a3fe65e957daeb7bc48b27dc7f69bc3ecdd6f1 \ + --hash=sha256:58838921e377791ef22c64cc92141bfae030f43651ff9272f0f28a208a9e6a5a \ + --hash=sha256:589d9da7de8fa7f739bb970ac4632c9a268213117d634e1c4a58c1c1e821ca05 \ + --hash=sha256:5a4370089c8b2e42f1db51d76408c7fa8eaa2934bf854d17983d16179c07c098 \ + --hash=sha256:622cdde768300591ac79bfcd7bb3468e4b191b1105d5dbfe8d87c39d8f63dd46 \ + --hash=sha256:673d89cc434cc9b97a0b4cf61272f6fca70a81f64eb0afbface2a0d9f77f06cd \ + --hash=sha256:6a019a31bc4db89a0903a3e4f6b218571f3a6ff0ad4b3d3fe1c8f91a05ff6e3e \ + --hash=sha256:6c142bd0cb4dba31c10babe00d59f7ef6460f0ef55eaa9c1a9da270684af996a \ + --hash=sha256:6f29689a76fe7a49cbd629a794d0ec1eab48814e323a00a146a741b0195bde68 \ + --hash=sha256:75d7411e8eb9259764dd60310738540649057cda4509b4af14b36b7f663bfeb0 \ + --hash=sha256:793229621e1ad6cac55f015cfa9f4eff102accbc3da25d607b91c6b0bec167fb \ + --hash=sha256:7a49f47ac26df3e431c56b825c1bae8e6d3d591fdbb7438c227cc9845a7e3d73 \ + --hash=sha256:7b9044ce4acbf392d4b3a503fe377641f4deff82e6c341c36ef27af0dea76cdf \ + --hash=sha256:7c89359affecebe2e39e6a116d069b363c936511a9572b308402489a26957d89 \ + --hash=sha256:84ed3a2d5cd3ae6db87af20bfed3331116195ba4757ad7177fc8f12c1bbce2a9 \ + --hash=sha256:89a0980d49683c00539c59cd9f46b1908c538e6b5b0a48ad12187bb856d0f391 \ + --hash=sha256:8b72c2002202038666bf647f9a790906214c7c11cd0d6efef77b7d07bef3034a \ + --hash=sha256:8eb3e5caca8d3a6f72ea4cce024583f989f6f2e9186f98800213fff0176e8bcc \ + --hash=sha256:9082e709014946b1f6e96ae6ecd93652efca2d2a6a3ab67dbe151c8b4bf193a4 \ + --hash=sha256:9205068badf453d2f0821fd5d340389b4679992d7ff79d4f3e5608996dd1b287 \ + --hash=sha256:9472a8de37faf8bd216c628e0e68c8f6bef730d3ba0a5060f3b0fa460c992ac2 \ + --hash=sha256:967955620df45e6cffe2e9950cb9903cb455649396f896b26b04363a91a5054b \ + --hash=sha256:975c17da26c5b9d46c336b03c52a057ac28378d6f9d98b58d32a038589bb3912 \ + --hash=sha256:981cc10163988defea96e8d6c507df151eab8f483e7df9ae543d5a41a4be073b \ + --hash=sha256:98c88d3edd08e7245562e9815996dbc6f0bd4745e1c76462f24af5ae4e187dd1 \ + --hash=sha256:9a91c4814c7150a4d798da691b7804eacd78c4b84fb392a60fa0de21341861eb \ + --hash=sha256:9e311f908d2f842fca4c7dcebc4f10306b8089b204ef04cf6704b4332c9ff6ff \ + --hash=sha256:9e753e81457241e2200c56a898e268e8fa25796271af0489c608f24d8e631eed \ + --hash=sha256:a46bedc273b6f58f11dee816ff74999625ef8d007890f411b7a4975bf1c89330 \ + --hash=sha256:abb5667a36119019b3fa320c4d10c21ebccfcc87c8a739e6a0056cee7f48dde2 \ + --hash=sha256:b3131a82d0c7611f0aa480a6d36929e001a3f54ba0fc029a8118a5863cce513c \ + --hash=sha256:b5d11cccba076a32321ef1380dea956821f0b51794ef59df64e58fb1cd543aae \ + --hash=sha256:b6c5d2d71e1f17329150ad9427d27f4a3f29a01792e7ecdc64a98ac5368fc4d5 \ + --hash=sha256:b776c7fc8755c7de06dd5a22b47c40ae84f67d13277ebb233cc84933ba4dcbcd \ + --hash=sha256:c00d182e31034250690f417b9068b78eab423c10d76766664e82d9860c340479 \ + --hash=sha256:c351737e2e65497c7200ab4ffb8af97e9f48be6488309abdd265fe08d66ee92f \ + --hash=sha256:c4f845166b09acc65c5213a35551a7f81c17fa010ab467229b5813f79d17fe13 \ + --hash=sha256:c589f5023d471ce75dd2cce61acb25ed6347e562041588a1a366808f22d7176c \ + --hash=sha256:cee808b405e9095506f4e4e89924bec7ea77eac3129b6fe36eda04364b3b343b \ + --hash=sha256:d11a7bc1e02da8984d32e6de9e0826c6edac00eac17de270f372bf32f9a0af63 \ + --hash=sha256:d27c531edb8d1f38ca2eddaa1fa24913a460aeb721f2efd4ef42a124ce94e354 \ + --hash=sha256:d2d9a63a9e6f2416ace8c109043a9280d6b34f34bb2e5421903e149403db40a6 \ + --hash=sha256:d561a4c5747a1e6c7fa7c49a0292e78b4e8c456332caa084fc7abad8de828652 \ + --hash=sha256:d63010803d7c368963bbe6f7ec379593e76dd581d7db0f29118d88713c9e0354 \ + --hash=sha256:dd741c73440b328f937dc53b344ecadc46bc4f0cec0333a8f42b55f3468ce7ec \ + --hash=sha256:de1064663aa7c839286488a319d2b3b478ca5ab5b2091ade888ed0eeca11a98a \ + --hash=sha256:e252db239eb41c32248e096e0d170bce5896a4fd3405556362bc3dd83d912206 \ + --hash=sha256:e288a06cbbbcd01b44386e767985c9e21d2ad9bf59829aa7058d9a2a494804ab \ + --hash=sha256:e59b60a0a4cb7541480e02090d37dc2df3b72df4c2e776fff64ce3a4e3dd4637 \ + --hash=sha256:e671b2322ef09106ecb1ca0f4c398b134d5e2c1f80d7a4f3336847a3072c0e94 \ + --hash=sha256:e7b04935a79c03c41ad08d0a5f390aac968bfb561f1268897bc5b0f077971efd \ + --hash=sha256:f7e9b8728ba07a3cb2f29d5aa1a266c2664eb8ef0fd43afa34627c92f7fac8f0 \ + --hash=sha256:f9b093cb3b6c9d6233ef45a05cab064d2aa0a8cb3c5777084c9e20fcb77c2371 \ + --hash=sha256:fae8b282f0cb22a5de222999f7723f4e5ec04f6fcdf4aaef879b5b36625ae2b0 +uvicorn==0.52.1 \ + --hash=sha256:112ec661814189acbccd3f7b86460147cc065fc92c0821afa78918780e4354dd \ + --hash=sha256:e4403f9d93188cf9d1088e9f40e3acd12630e2df8675316704379a7fc20fff6a +uvloop==0.22.1 ; platform_python_implementation != 'PyPy' and sys_platform != 'cygwin' and sys_platform != 'win32' \ + --hash=sha256:017bd46f9e7b78e81606329d07141d3da446f8798c6baeec124260e22c262772 \ + --hash=sha256:0530a5fbad9c9e4ee3f2b33b148c6a64d47bbad8000ea63704fa8260f4cf728e \ + --hash=sha256:05e4b5f86e621cf3927631789999e697e58f0d2d32675b67d9ca9eb0bca55743 \ + --hash=sha256:0ae676de143db2b2f60a9696d7eca5bb9d0dd6cc3ac3dad59a8ae7e95f9e1b54 \ + --hash=sha256:1489cf791aa7b6e8c8be1c5a080bae3a672791fcb4e9e12249b05862a2ca9cec \ + --hash=sha256:17d4e97258b0172dfa107b89aa1eeba3016f4b1974ce85ca3ef6a66b35cbf659 \ + --hash=sha256:1cdf5192ab3e674ca26da2eada35b288d2fa49fdd0f357a19f0e7c4e7d5077c8 \ + --hash=sha256:1f38ec5e3f18c8a10ded09742f7fb8de0108796eb673f30ce7762ce1b8550cad \ + --hash=sha256:286322a90bea1f9422a470d5d2ad82d38080be0a29c4dd9b3e6384320a4d11e7 \ + --hash=sha256:297c27d8003520596236bdb2335e6b3f649480bd09e00d1e3a99144b691d2a35 \ + --hash=sha256:37554f70528f60cad66945b885eb01f1bb514f132d92b6eeed1c90fd54ed6289 \ + --hash=sha256:3879b88423ec7e97cd4eba2a443aa26ed4e59b45e6b76aabf13fe2f27023a142 \ + --hash=sha256:3b7f102bf3cb1995cfeaee9321105e8f5da76fdb104cdad8986f85461a1b7b77 \ + --hash=sha256:40631b049d5972c6755b06d0bfe8233b1bd9a8a6392d9d1c45c10b6f9e9b2733 \ + --hash=sha256:481c990a7abe2c6f4fc3d98781cc9426ebd7f03a9aaa7eb03d3bfc68ac2a46bd \ + --hash=sha256:4a968a72422a097b09042d5fa2c5c590251ad484acf910a651b4b620acd7f193 \ + --hash=sha256:4baa86acedf1d62115c1dc6ad1e17134476688f08c6efd8a2ab076e815665c74 \ + --hash=sha256:512fec6815e2dd45161054592441ef76c830eddaad55c8aa30952e6fe1ed07c0 \ + --hash=sha256:51eb9bd88391483410daad430813d982010f9c9c89512321f5b60e2cddbdddd6 \ + --hash=sha256:535cc37b3a04f6cd2c1ef65fa1d370c9a35b6695df735fcff5427323f2cd5473 \ + --hash=sha256:53c85520781d84a4b8b230e24a5af5b0778efdb39142b424990ff1ef7c48ba21 \ + --hash=sha256:55502bc2c653ed2e9692e8c55cb95b397d33f9f2911e929dc97c4d6b26d04242 \ + --hash=sha256:561577354eb94200d75aca23fbde86ee11be36b00e52a4eaf8f50fb0c86b7705 \ + --hash=sha256:56a2d1fae65fd82197cb8c53c367310b3eabe1bbb9fb5a04d28e3e3520e4f702 \ + --hash=sha256:57df59d8b48feb0e613d9b1f5e57b7532e97cbaf0d61f7aa9aa32221e84bc4b6 \ + --hash=sha256:6c84bae345b9147082b17371e3dd5d42775bddce91f885499017f4607fdaf39f \ + --hash=sha256:6cde23eeda1a25c75b2e07d39970f3374105d5eafbaab2a4482be82f272d5a5e \ + --hash=sha256:6e2ea3d6190a2968f4a14a23019d3b16870dd2190cd69c8180f7c632d21de68d \ + --hash=sha256:700e674a166ca5778255e0e1dc4e9d79ab2acc57b9171b79e65feba7184b3370 \ + --hash=sha256:7b5b1ac819a3f946d3b2ee07f09149578ae76066d70b44df3fa990add49a82e4 \ + --hash=sha256:7cd375a12b71d33d46af85a3343b35d98e8116134ba404bd657b3b1d15988792 \ + --hash=sha256:80eee091fe128e425177fbd82f8635769e2f32ec9daf6468286ec57ec0313efa \ + --hash=sha256:93f617675b2d03af4e72a5333ef89450dfaa5321303ede6e67ba9c9d26878079 \ + --hash=sha256:a592b043a47ad17911add5fbd087c76716d7c9ccc1d64ec9249ceafd735f03c2 \ + --hash=sha256:ac33ed96229b7790eb729702751c0e93ac5bc3bcf52ae9eccbff30da09194b86 \ + --hash=sha256:b31dc2fccbd42adc73bc4e7cdbae4fc5086cf378979e53ca5d0301838c5682c6 \ + --hash=sha256:b45649628d816c030dba3c80f8e2689bab1c89518ed10d426036cdc47874dfc4 \ + --hash=sha256:b76324e2dc033a0b2f435f33eb88ff9913c156ef78e153fb210e03c13da746b3 \ + --hash=sha256:b91328c72635f6f9e0282e4a57da7470c7350ab1c9f48546c0f2866205349d21 \ + --hash=sha256:badb4d8e58ee08dad957002027830d5c3b06aea446a6a3744483c2b3b745345c \ + --hash=sha256:bc5ef13bbc10b5335792360623cc378d52d7e62c2de64660616478c32cd0598e \ + --hash=sha256:c1955d5a1dd43198244d47664a5858082a3239766a839b2102a269aaff7a4e25 \ + --hash=sha256:c3e5c6727a57cb6558592a95019e504f605d1c54eb86463ee9f7a2dbd411c820 \ + --hash=sha256:c60ebcd36f7b240b30788554b6f0782454826a0ed765d8430652621b5de674b9 \ + --hash=sha256:daf620c2995d193449393d6c62131b3fbd40a63bf7b307a1527856ace637fe88 \ + --hash=sha256:e047cc068570bac9866237739607d1313b9253c3051ad84738cbb095be0537b2 \ + --hash=sha256:ea721dd3203b809039fcc2983f14608dae82b212288b346e0bfe46ec2fab0b7c \ + --hash=sha256:ef6f0d4cc8a9fa1f6a910230cd53545d9a14479311e87e3cb225495952eb672c \ + --hash=sha256:fe94b4564e865d968414598eea1a6de60adba0c040ba4ed05ac1300de402cd42 +watchfiles==1.2.0 \ + --hash=sha256:01859b11fd9fbca670f4d5da00fbac282cfea9bd67a2125d8b2833a3b5617ea9 \ + --hash=sha256:01ea8d66f0693b9b60a6541c8d10263091ca9a9060d242f3c1f3143f9aad2c98 \ + --hash=sha256:027ae72bfdfd254862065d8b3e2a815c6ab9b1853ce41e6648ece84afd34a551 \ + --hash=sha256:03b14855c6f35539e2d95c442ae9530a75762f1e26567152b9ed05f96534a74d \ + --hash=sha256:054dc20fd2e3132b4c3883b4a00d72fd6e1f56fdaf89fccd12e8057d74cd74d7 \ + --hash=sha256:094b9b70103d4e963499bdea001ee3c2697b144cd9ae6218a62c0f89ec9e31db \ + --hash=sha256:0a105bc2283f67e8fbec74253ec2d94925de92ed72c0393f1206bf326b7b7b69 \ + --hash=sha256:0a37faaed405c67e28e6be45a1fa4f206ef5a2860f27c237db9fa30704c38242 \ + --hash=sha256:0c4997d4e4a55f0d02b6cde327322daf3a0400e5df6c6b15948994bf72497925 \ + --hash=sha256:0cb4d80e212f116474a545c21c912b445f16bb0cef9e6a73a498164223e14e2f \ + --hash=sha256:0d191c054d0715c3c95c99df9b8dbf6fd096d8c1e021e8f212e1bd8bc444ccb5 \ + --hash=sha256:0e831a271c035d89789cffc386b6aa1375f39f1cd25eb7ca0997e4970d152fc5 \ + --hash=sha256:10d86db20695afe7997ac9e1717637d6714a8d0220458c33f3d2061f54cec427 \ + --hash=sha256:11743adfa510bfffebe97659fb280182b5c9b238708f667e866f308c3430dc19 \ + --hash=sha256:1bc6195825b7dcd217968bb1f801a60fd4c16e8eeab5bedc7fe917d7d5995ab4 \ + --hash=sha256:204f299afcbd65918ab78dbc52626b0ae45e9d8cef403fdbf33ecf9e40eac66e \ + --hash=sha256:20aa0e708b920bde876a4aa82dc7dd6ebea228a63a67cda6632c2fc87b787efa \ + --hash=sha256:23282a321c8baf9b3a3c4afff673f9fe65eb7fdc2338d765ccad9d3d1916a5ba \ + --hash=sha256:24b2405c0a46738dd9e1cf7135aa5dbdb9d42d024628651b3b13d5117e99f8df \ + --hash=sha256:2581a94056e55d7d0a31a823ea92bf73749c489ca2285bfdc0fbe6b2bb49d50c \ + --hash=sha256:2995c176de7692b86a2e4c58d9ec718f753150a979cb4a754e2b4ffa38e70906 \ + --hash=sha256:2b37d10b5a63bd4d87e18472d80fa525bd670586fae62e5dd580452764879b65 \ + --hash=sha256:2cb93af48550faf1cea04c303107c8b75833de7013e57ce27d3b8d21d8d0f58c \ + --hash=sha256:2d95ddc1eb6914154253d239089900813f6a767e174b8e6a50e7fdacb7e4236c \ + --hash=sha256:3416ff151bb6b5a8d8d11664974fbef4d9305b9b2957839ab5a270468fd8df30 \ + --hash=sha256:3651aa7058595e9cfb75d35dd5ada2bf9f48a5b8a0f3562821d3e210c507e077 \ + --hash=sha256:37a6721cdf3f65dbb13aa9503510ccb4451603ac837e44d265d7992a597e1374 \ + --hash=sha256:41bc1199f7523b3f82843c88cbb979180c949caef0342cf90968f178e5d49b01 \ + --hash=sha256:43d818978d06062d9b22c4fab2ebe44cf5213d42dc8e62bda8c2760cfa2eeb33 \ + --hash=sha256:4429f3b105524a10b72c3a819b091c495d2811d419c1e1e8df773a5a5974f831 \ + --hash=sha256:4543579a9bdb0c9560039b4ffddbdb39545707659fbc430ce4c10f3f68d557f9 \ + --hash=sha256:4674d49eb94706dfe666c069fc0a1b646ffcf920473492e209f6d5f60d3f0cc2 \ + --hash=sha256:4c887eba18b7945ac73067a8b4a66f21cd46c2539b2bc68588f7be6c7eb6d26b \ + --hash=sha256:4e4ff8e37f99cf1da89e255e07c9c4b37c214038c4283707bdec308cb1b0ea1f \ + --hash=sha256:4f34e26a19f91f710c08e0183429f0d1d15df734e6bc78c31e77b9ea9c433658 \ + --hash=sha256:5327989a465505f05cfe06f04fa9d0c2fd5432bb243e10e6f012b1bdca3c8579 \ + --hash=sha256:53b2290c92e0506d102cd448fbc610d87079553f86caa39d67440856a8b8bba5 \ + --hash=sha256:56d8641cf834c2836922899105bd3ce3d0dfc69291d52edf0b4d0436829b34c0 \ + --hash=sha256:57a2d9fa4fb4c2ecae57b13dfff2c7ab53e21a2ba674fe9f05506680fcdcc0d7 \ + --hash=sha256:63ac26eefbf4af1741247d6fb68b11c49a25b2f7413fbd318a83a12aaa9cf666 \ + --hash=sha256:6543cf55d170003296d185c0af981f3e1311564907e1f4e08671fc7693a890a5 \ + --hash=sha256:704fd259e332e01f9b9c178f4bce9e49027e5587cc2600eeeaf8e76e1c846201 \ + --hash=sha256:71283b39fd17e5408eb123bd37aeecfd9d54c81fc184421943208aadb879d103 \ + --hash=sha256:71cd71740ed2c15211ebb237ced4e39a1cdf6f80566e5fe95428da1626f4fde6 \ + --hash=sha256:7571e4464cb6e434958f867f7f730b8ab0b75e3f8e5eac0499168486ab3c33a8 \ + --hash=sha256:772b80df316480d894a0e3165fdd19cf77f5d17f9a787f94029465ad0e3529d1 \ + --hash=sha256:77a0feab9af4c021c581f695258c642b3d10c5fd4c676e33a0d8606425d82631 \ + --hash=sha256:7a2cffd17d27d2ecbb310c2b1d8174f222a5495b1a721894afa88ec11e25b898 \ + --hash=sha256:7a7ce236284f002a156f70add88efe5c70879cccbb658be0822c54b1306fc09d \ + --hash=sha256:7ba0480b9a74af058f43b337e937a451e109295c420916d68ad24e3dc02f5e44 \ + --hash=sha256:8520a4ab0e37f770afc34459c4f8f7019e153f9124dc101c15538365875d1ab2 \ + --hash=sha256:86bc13c25a8d1fcd70b51d0ce7c9b65e90de5666fcbfd3e34957cc73ee19aeb5 \ + --hash=sha256:89d8c2394a065ca86f5d2910ff263ae67c127e1376ccc4f9fc35c71db879f80a \ + --hash=sha256:8c520725602756229f045b032a1ff33d7ef0f7404189d62f6c2438cb6d8ef6a1 \ + --hash=sha256:8f200104103feb097de4cab8fe4f5dd18a2026934c7dea98c55a2f5fd6d5a33b \ + --hash=sha256:8f70d8b291ef6e88d19b1f297a6905ddb978888d9272b0d05e6f53309856bcfc \ + --hash=sha256:8fa585ede612ee9f9e91b18bebf9ba11b9ae29a4e3a0d0cf6fca3e382133f0d5 \ + --hash=sha256:922c0e019fe68b3ae392965a766b02a71ba1168c932cebc3733cd52c5fe5b377 \ + --hash=sha256:9342472aff9b093c5acd4f6d8f70ae0937964ab56542502bcf5579782da69ae8 \ + --hash=sha256:9649193aa27bd9ff2e80ff29bfaa93085496c7a3a377592823cc58b77ee88add \ + --hash=sha256:9f04b092229ad2c50126dd3c922c8822e51e605993764a33058d4a791ab42281 \ + --hash=sha256:a0f27f01bee51861392bb6b7c4fdb290b27d1eb194e9e28788d68102a0e898d9 \ + --hash=sha256:a16ffe19bf5cf9f5edaa1ad1dd830c5a816e8feec430c522302ab55483a4b994 \ + --hash=sha256:a204794696ffb8f9b10fba6f7cb5216d42f3b2b71860ccac6b6e42f5f10973b0 \ + --hash=sha256:a711b51aec4370d0dcda5b6c09463206f133a5759341d7744b953a7b62e1100e \ + --hash=sha256:a88fc94e647bc4eec523f1caa540258eb71d14278b9daf72fa1e2658a98df0f0 \ + --hash=sha256:ae99b14c5f21e026e0e9d96f40e07d8570ebee6cafd9d8fc318354606daa7a28 \ + --hash=sha256:b0ef001f8c25ad0fa9529f914c1600647ecd0f542d11c19b7894768c67b6acb7 \ + --hash=sha256:b141a4891c995a039cd89e9a49e62df1dc8a559a5d1a6e4c7106d16c12777a55 \ + --hash=sha256:b4e77f6a55f858504069abd35d336a637555c09bca453dde1ee1e5ada8a6a1fb \ + --hash=sha256:b62f042afde2dde21ec1d2c1a74361e804673df86f51e418a999c9acfe671b07 \ + --hash=sha256:b718bf356bbc15e559bd8ef41782b573b8ae0e3f177ab244b440568d7ea02cfb \ + --hash=sha256:b8c8358484d5fa12ef34f05b7f4168eaf1932f408725ff6d023c33ec17bd79d4 \ + --hash=sha256:b974946a10af379d425e2eef5b62f5c6ebeaccf91d45eaad6f5b27ecd4f91aa0 \ + --hash=sha256:b9909cc2b48468b575eefa944919e1fe8a36c5849d5c7c168f80a8c1db69398e \ + --hash=sha256:b9f732dc58b2dbe69e464ccf8fff7a03b0dd0be439da4c0720d3558527d3d6b4 \ + --hash=sha256:bb68bf4df85abebe5efddc53cf2075520f243a59868d9b3973278b23e76962a9 \ + --hash=sha256:bb7e52ecf68ba46d22df23467b87cffeb2146908aa523ebfe803019618cfda06 \ + --hash=sha256:bc13eb17538be00c874699dc0abe4ee2bc8d50bb1166a6b9e175ef3fd7eb8f26 \ + --hash=sha256:c0db965c5f79aa49fe672d297cf1febc5ad149b658594944f49a54a2b96270a7 \ + --hash=sha256:c16cb06dd17d43b9d185094268459eac92c9538356f050e55b54e82cf700e1d4 \ + --hash=sha256:c525543d91961c6955b2636b308569e84a1d1c5f5f2932041ab9ef46422f43e3 \ + --hash=sha256:c5c19526f4e54a00f2666a6c0e9e40d582c09e865055ea7378bf0009aab857b3 \ + --hash=sha256:c995fba777f1ea992f090f9236e9284cf7a5d1a0130dd5a3d82c598cacd76838 \ + --hash=sha256:ca148d73dea36c9763aaa351e4d7a51780ec1584217c45276f4fe8239c768b71 \ + --hash=sha256:cee9d5efd929efdac5f7e58f72b3376f676b64050a91c5b99a7094c5b2317488 \ + --hash=sha256:d158cd89df6053823533e06fb1d73c549133bff5f0396170c0e53d9559340717 \ + --hash=sha256:d20029a60a71a052a24c4db7673bc4de39ab89adbaccbfb5d67987c5d73f424d \ + --hash=sha256:d413349d565dab74297f2a63e84a097936be69bf8f3b3801f27f380e32040f44 \ + --hash=sha256:d4a4b147f5dca2a5d325a06a832fb43f345751adfbc63204aec30e0d9ca965a2 \ + --hash=sha256:d516b3283a758e087841aedb8031549fb41ced08f3db10aa6d2bf32dc042525b \ + --hash=sha256:d73a585accffa5ae39c17264c36ec3166d2fad7000c780f5ef83b2722afb9dd2 \ + --hash=sha256:dbd6c97045dad81227c8d040173da044c1de08de64a5ea8b555da4aee1d5fa22 \ + --hash=sha256:e0618518f282c4ebff60f5e5b1247b6d91bb8b9f4476947563a1e74acc66f3c6 \ + --hash=sha256:e140ed30ebde76796b686e67c182cff10ea2fbab186fafd1560f74bb5a473a6e \ + --hash=sha256:e1cfd51e97e13ff3bd047c140764d277fc9b95b7cb5da59e46a47d167adab310 \ + --hash=sha256:e2ca07fa7d89195ec0865d3d285666286740bfa83d83e5cee204043a31ecc165 \ + --hash=sha256:e53a384f76b631c3ae5334ce6a52f0baa3a911eb94a4eac7f160079868b716d5 \ + --hash=sha256:eb283ee99e21ad6443c8cdb06ac5b34b1308c329cbdf03fa02b445363714c799 \ + --hash=sha256:eb72919d93e3a16fc451d3aa3d4b1698423daca1b382d3d959c9ac51297c12a8 \ + --hash=sha256:ecb47f183a8025b2aa18b546725c3657e542112ae9c0613a2af79b4fa8d04ad7 \ + --hash=sha256:f155b3a1b2a5fc89cdc70d47ee5d54e3b75e88efa34982028a35daef9ba00379 \ + --hash=sha256:f22943b7770483f6ea0721c6b11d022947a98eb0acae14694de034f4d0d38925 \ + --hash=sha256:f28b2725eb8cce327b9b3ab02415c853011dc55c95832fe90de6bc56f5315f72 \ + --hash=sha256:f88af53d6ddaf72179ef613ddc905e6f4785f712b49b80b3bef9f3525e6194b4 \ + --hash=sha256:faea288b6f0ab1902ef08f4ca6de005dccf856c4e0c4f21b8c5fce02d90a1b08 \ + --hash=sha256:fff610d7bb2256a317bb1e96f0d7862c7aa8076733ee5df0fd41bbe76a24a4f4 +websockets==15.0.1 \ + --hash=sha256:0701bc3cfcb9164d04a14b149fd74be7347a530ad3bbf15ab2c678a2cd3dd9a2 \ + --hash=sha256:0a34631031a8f05657e8e90903e656959234f3a04552259458aac0b0f9ae6fd9 \ + --hash=sha256:0af68c55afbd5f07986df82831c7bff04846928ea8d1fd7f30052638788bc9b5 \ + --hash=sha256:0c9e74d766f2818bb95f84c25be4dea09841ac0f734d1966f415e4edfc4ef1c3 \ + --hash=sha256:0f3c1e2ab208db911594ae5b4f79addeb3501604a165019dd221c0bdcabe4db8 \ + --hash=sha256:0fdfe3e2a29e4db3659dbd5bbf04560cea53dd9610273917799f1cde46aa725e \ + --hash=sha256:1009ee0c7739c08a0cd59de430d6de452a55e42d6b522de7aa15e6f67db0b8e1 \ + --hash=sha256:1234d4ef35db82f5446dca8e35a7da7964d02c127b095e172e54397fb6a6c256 \ + --hash=sha256:16b6c1b3e57799b9d38427dda63edcbe4926352c47cf88588c0be4ace18dac85 \ + --hash=sha256:2034693ad3097d5355bfdacfffcbd3ef5694f9718ab7f29c29689a9eae841880 \ + --hash=sha256:21c1fa28a6a7e3cbdc171c694398b6df4744613ce9b36b1a498e816787e28123 \ + --hash=sha256:229cf1d3ca6c1804400b0a9790dc66528e08a6a1feec0d5040e8b9eb14422375 \ + --hash=sha256:27ccee0071a0e75d22cb35849b1db43f2ecd3e161041ac1ee9d2352ddf72f065 \ + --hash=sha256:363c6f671b761efcb30608d24925a382497c12c506b51661883c3e22337265ed \ + --hash=sha256:39c1fec2c11dc8d89bba6b2bf1556af381611a173ac2b511cf7231622058af41 \ + --hash=sha256:3b1ac0d3e594bf121308112697cf4b32be538fb1444468fb0a6ae4feebc83411 \ + --hash=sha256:3be571a8b5afed347da347bfcf27ba12b069d9d7f42cb8c7028b5e98bbb12597 \ + --hash=sha256:3c714d2fc58b5ca3e285461a4cc0c9a66bd0e24c5da9911e30158286c9b5be7f \ + --hash=sha256:3d00075aa65772e7ce9e990cab3ff1de702aa09be3940d1dc88d5abf1ab8a09c \ + --hash=sha256:3e90baa811a5d73f3ca0bcbf32064d663ed81318ab225ee4f427ad4e26e5aff3 \ + --hash=sha256:47819cea040f31d670cc8d324bb6435c6f133b8c7a19ec3d61634e62f8d8f9eb \ + --hash=sha256:47b099e1f4fbc95b701b6e85768e1fcdaf1630f3cbe4765fa216596f12310e2e \ + --hash=sha256:4a9fac8e469d04ce6c25bb2610dc535235bd4aa14996b4e6dbebf5e007eba5ee \ + --hash=sha256:4b826973a4a2ae47ba357e4e82fa44a463b8f168e1ca775ac64521442b19e87f \ + --hash=sha256:4c2529b320eb9e35af0fa3016c187dffb84a3ecc572bcee7c3ce302bfeba52bf \ + --hash=sha256:54479983bd5fb469c38f2f5c7e3a24f9a4e70594cd68cd1fa6b9340dadaff7cf \ + --hash=sha256:558d023b3df0bffe50a04e710bc87742de35060580a293c2a984299ed83bc4e4 \ + --hash=sha256:5756779642579d902eed757b21b0164cd6fe338506a8083eb58af5c372e39d9a \ + --hash=sha256:592f1a9fe869c778694f0aa806ba0374e97648ab57936f092fd9d87f8bc03665 \ + --hash=sha256:595b6c3969023ecf9041b2936ac3827e4623bfa3ccf007575f04c5a6aa318c22 \ + --hash=sha256:5a939de6b7b4e18ca683218320fc67ea886038265fd1ed30173f5ce3f8e85675 \ + --hash=sha256:5d54b09eba2bada6011aea5375542a157637b91029687eb4fdb2dab11059c1b4 \ + --hash=sha256:5df592cd503496351d6dc14f7cdad49f268d8e618f80dce0cd5a36b93c3fc08d \ + --hash=sha256:5f4c04ead5aed67c8a1a20491d54cdfba5884507a48dd798ecaf13c74c4489f5 \ + --hash=sha256:64dee438fed052b52e4f98f76c5790513235efaa1ef7f3f2192c392cd7c91b65 \ + --hash=sha256:66dd88c918e3287efc22409d426c8f729688d89a0c587c88971a0faa2c2f3792 \ + --hash=sha256:678999709e68425ae2593acf2e3ebcbcf2e69885a5ee78f9eb80e6e371f1bf57 \ + --hash=sha256:67f2b6de947f8c757db2db9c71527933ad0019737ec374a8a6be9a956786aaf9 \ + --hash=sha256:693f0192126df6c2327cce3baa7c06f2a117575e32ab2308f7f8216c29d9e2e3 \ + --hash=sha256:746ee8dba912cd6fc889a8147168991d50ed70447bf18bcda7039f7d2e3d9151 \ + --hash=sha256:756c56e867a90fb00177d530dca4b097dd753cde348448a1012ed6c5131f8b7d \ + --hash=sha256:76d1f20b1c7a2fa82367e04982e708723ba0e7b8d43aa643d3dcd404d74f1475 \ + --hash=sha256:7f493881579c90fc262d9cdbaa05a6b54b3811c2f300766748db79f098db9940 \ + --hash=sha256:823c248b690b2fd9303ba00c4f66cd5e2d8c3ba4aa968b2779be9532a4dad431 \ + --hash=sha256:82544de02076bafba038ce055ee6412d68da13ab47f0c60cab827346de828dee \ + --hash=sha256:8dd8327c795b3e3f219760fa603dcae1dcc148172290a8ab15158cf85a953413 \ + --hash=sha256:8fdc51055e6ff4adeb88d58a11042ec9a5eae317a0a53d12c062c8a8865909e8 \ + --hash=sha256:a625e06551975f4b7ea7102bc43895b90742746797e2e14b70ed61c43a90f09b \ + --hash=sha256:abdc0c6c8c648b4805c5eacd131910d2a7f6455dfd3becab248ef108e89ab16a \ + --hash=sha256:ac017dd64572e5c3bd01939121e4d16cf30e5d7e110a119399cf3133b63ad054 \ + --hash=sha256:ac1e5c9054fe23226fb11e05a6e630837f074174c4c2f0fe442996112a6de4fb \ + --hash=sha256:ac60e3b188ec7574cb761b08d50fcedf9d77f1530352db4eef1707fe9dee7205 \ + --hash=sha256:b359ed09954d7c18bbc1680f380c7301f92c60bf924171629c5db97febb12f04 \ + --hash=sha256:b7643a03db5c95c799b89b31c036d5f27eeb4d259c798e878d6937d71832b1e4 \ + --hash=sha256:ba9e56e8ceeeedb2e080147ba85ffcd5cd0711b89576b83784d8605a7df455fa \ + --hash=sha256:c338ffa0520bdb12fbc527265235639fb76e7bc7faafbb93f6ba80d9c06578a9 \ + --hash=sha256:cad21560da69f4ce7658ca2cb83138fb4cf695a2ba3e475e0559e05991aa8122 \ + --hash=sha256:d08eb4c2b7d6c41da6ca0600c077e93f5adcfd979cd777d747e9ee624556da4b \ + --hash=sha256:d50fd1ee42388dcfb2b3676132c78116490976f1300da28eb629272d5d93e905 \ + --hash=sha256:d591f8de75824cbb7acad4e05d2d710484f15f29d4a915092675ad3456f11770 \ + --hash=sha256:d5f6b181bb38171a8ad1d6aa58a67a6aa9d4b38d0f8c5f496b9e42561dfc62fe \ + --hash=sha256:d63efaa0cd96cf0c5fe4d581521d9fa87744540d4bc999ae6e08595a1014b45b \ + --hash=sha256:d99e5546bf73dbad5bf3547174cd6cb8ba7273062a23808ffea025ecb1cf8562 \ + --hash=sha256:e09473f095a819042ecb2ab9465aee615bd9c2028e4ef7d933600a8401c79561 \ + --hash=sha256:e8b56bdcdb4505c8078cb6c7157d9811a85790f2f2b3632c7d1462ab5783d215 \ + --hash=sha256:ee443ef070bb3b6ed74514f5efaa37a252af57c90eb33b956d35c8e9c10a1931 \ + --hash=sha256:f29d80eb9a9263b8d109135351caf568cc3f80b9928bccde535c235de55c22d9 \ + --hash=sha256:f7a866fbc1e97b5c617ee4116daaa09b722101d4a3c170c787450ba409f9736f \ + --hash=sha256:fcd5cf9e305d7b8338754470cf69cf81f420459dbae8a3b40cee57417f4614a7 +xxhash==3.8.1 \ + --hash=sha256:000435984a0469b0f822fe76f35bddea0f96a4d6521b3339a60a6428cdee1edc \ + --hash=sha256:00de40f3b42240db23a82a5c682b55d7263d84a26a953240c1aee463409660e3 \ + --hash=sha256:01cab782f8a0a05ecad2c63d7ef10f7ab475f660e0d6419d069418c14d88de7c \ + --hash=sha256:0204701e6d01f64254e0e5ff4255812b1febe027ddd7dda63372e27f98b5e91f \ + --hash=sha256:027dee4355f3fcc41481650d846cf6cfc895c85a1ab7acd063063821a0df5b4c \ + --hash=sha256:036a024d8b9c01f70782e09ed98d532e76fd23f950ae7154bd950fe94e90ebec \ + --hash=sha256:0418ec8b2331b9d4d575fc9284427e8e69449d7172e99e1a86fcdd1f51a0a937 \ + --hash=sha256:08ea2081f5e88615fec8622a9f87fbe21b8ea58d88cfc02163ca11026ee62a92 \ + --hash=sha256:095e1323fa108be1292c54c86da3ef3c7a7dc015b105a52133973bc07a6ad11a \ + --hash=sha256:09a204dd4bb0823daf938cdd0dc8057d5f1e14fe3cbde929424255f23f9de872 \ + --hash=sha256:0dfdf19b0d5433a75d61f19dc85737af0f0b95e445c1ad69c855115d05efed45 \ + --hash=sha256:0fe37f72a207223d22a4eddc3149d4298993385aa9daef25c039246ca5a309f3 \ + --hash=sha256:10e4393ec33633c2f05ad01869e546ad080b1a18f2650503731f153774608b31 \ + --hash=sha256:1153265daa10750a9bf8e9b01753d7618024a300925591efaf16b1b7fa536699 \ + --hash=sha256:115772daeb71b2f3b9381177017f53e6cf3f3439c840737fdabd21aba6e54920 \ + --hash=sha256:12a3cf79dadbab9631230ebc4c51c7c60f1e9cdfb890c15fb733eaafe2e7713c \ + --hash=sha256:12eaeaa9ab8b9e6033a1fa5f6b338aaf55ff4df4bee11b59fd6ee03b19186ee4 \ + --hash=sha256:131324f719957b988861714de7d6ddf57b47abec3b0cc691302ffeaba0e05e10 \ + --hash=sha256:15790b686f8723b845fec6f612a343beb815a25c83117a7fa408d7c8ee5aa8fd \ + --hash=sha256:1731407102b9332cd3c9dadee07db498bc3d437b95d752b5b1a5f7eb730a3738 \ + --hash=sha256:1b86ae798a976ccbc1d02af6ccb98f5b4d24756b1f65e995f11d10fe071f486f \ + --hash=sha256:1c332dd48b8cb050da2bb2a3c96d72b1664168650a250ef9718e423df7989e05 \ + --hash=sha256:1da930bbcac3e8fbe2191850e2abb57977a99348c12c4b385e1058ac1b0a9ecc \ + --hash=sha256:1f44275ddb0978b67a58a951501903f04d49335a91f7681c9ce122ecb8ccb329 \ + --hash=sha256:1ffcc98d8878e449e86dec008cea6f44cfd3a954d2ef24ae7d1cc9f725beec7d \ + --hash=sha256:220d68130f83f7cc86d6edfdeab176adc73d7200bf3a8ec10c629e8cf605c215 \ + --hash=sha256:2256e80e4960ee282f63428adb349cb7f8bd8efe4db770d88eb815f4b9860724 \ + --hash=sha256:23e710118a5778a45db740b431943a3f2a82a571a052c2768cce6544d9c8c62e \ + --hash=sha256:264710bd335016f303763ce1275c6486df30bb57c2245c91b224c983d7ac39b8 \ + --hash=sha256:2666f059a1588a99267e33605365ed89cea92f424b3522806a9f4bd8ad2e3d62 \ + --hash=sha256:27a9e475157f7315826118e3f3127909a0fe25f1b43d3d3be9c584f9d265f937 \ + --hash=sha256:27cfc2f1ed76f956f36dfe0c56e5f5a3e94cd91eb78b893f63e2ef2ae404fcdf \ + --hash=sha256:2bc7113e6f2b6b3922dd61796ca9f36af09da3773898e7003038dc992fc83b8d \ + --hash=sha256:2e32855b6f9e5b18f449e59d45e3d5778bdeb660632ef2693cca267a11246c75 \ + --hash=sha256:2f1c68394818e0595569c2ff3cbc1e6d5a36a434e796f5c526b987b80c8a8c62 \ + --hash=sha256:2f8c25a7061d952de589bd0ea0eaadee32378ff83dd6a677b267f9cd86f401f8 \ + --hash=sha256:314d05fbc55719ae2438eaaba77bf2508ca4f030b26fa4c9c8c380e81c48fa33 \ + --hash=sha256:32a94ad2763e0263d9102037d349002c3d3c401e42770542c3eeb4801f311661 \ + --hash=sha256:32ab1e5432690276e71192be7401b55f96db2d0eedea5d44eb1f164505669cc0 \ + --hash=sha256:345b07b78e2bf583d71682aa34ae5b5fab575f7a1cb31e10263ebbc6f89f8c42 \ + --hash=sha256:3557bec8fcb11738a8920eeb68974bc76b75262f6947998d3147954ce0a4b893 \ + --hash=sha256:358650d5bda9c635da699c53adf4e8134af492ecc79c960f917eebf088bb6799 \ + --hash=sha256:36fc69160465ae75c6ec4ac9f781bb2aa16ae7ff869e73c26fee85fbb11b9887 \ + --hash=sha256:37d5a56c36dcc0b9a87b814cd992598d33863ff683749de6c86081f278d5e629 \ + --hash=sha256:38c887aedb696ef8bca19983206d270848558cfae4a91afa6a2fb05dde58ffc5 \ + --hash=sha256:39c9d5b61508b0bb68f29e54546de0ed2a74943c6a18585535a7e37356f1dd12 \ + --hash=sha256:3a800912a2e5e975d4128969d645c4a2a80aa886ccd6c9b1c6f44529e327e8cf \ + --hash=sha256:3bc0fa90830df1e1277f33cc6e55de9990b83c0319fd8c7412866cfde38b025e \ + --hash=sha256:3c0d84c5f2e086b120bae4e7f551cbda804c1deb10d958478bed4f89ba286dfe \ + --hash=sha256:3c682fcd96eb4bf64be32a4d95f96107e1588005831bd8a741b324fdda01b913 \ + --hash=sha256:402db908ea70eaf9800d9182a66596fc86f36655df8f63fdecf7c11da741d86f \ + --hash=sha256:445e0f5a31f2f3546ae0895d4811e159518cdc9d824c11419898d40cfadb677e \ + --hash=sha256:4482380b462ca9e59994d072a877ecadd1cf51102daeeab2db696f96ab763723 \ + --hash=sha256:44c89d915a75c11d2547eaee9098fcd80398987c4bff2974a0497a925bf92c07 \ + --hash=sha256:454d78e786602278a2a4383d08048482052f4f0c61fa677ca590af08914d9bca \ + --hash=sha256:460261045936975193bfd20549a0de1cd52a33b405cbb972f0d80940c42266cd \ + --hash=sha256:46b39976d008e2a845758650f0ff7136bca004f40da0c8798bd37ac37860154f \ + --hash=sha256:498017fbf2d13a768b3110d084bde39f2bd8664c1de0b8084f8ccc84425b7c88 \ + --hash=sha256:49aa8692507835dcc1e8ad8021f20c74c2dc13d83b5112e87877faa2a0035b20 \ + --hash=sha256:4b512261801b1e5fde7b6ebf2fef7977339c620cbbca88a0040ad9ad134f4d02 \ + --hash=sha256:4bec8b2c909bcfae9a0dc702346007e02a8c9ba5bbde83ffb224aa194f4f9efc \ + --hash=sha256:4d365ee1892c1fa803536f8c6ce21d24b29c9718ec75eb856095c07830f8c478 \ + --hash=sha256:4d6e88ddb3c741fbf29e1e7faf429880f8cd1d7aff4303247435a549726b4fb1 \ + --hash=sha256:4df57c0b161ec1b3ed0526a67b0db0914b557e86ee8aae51887aec941b261542 \ + --hash=sha256:4e0e1b0fb0259c1b75d1251ac0bb4d7ab675d36f7a6bf4ba6aa630dae94f9ffa \ + --hash=sha256:4ef09bbc2519a93cd0f95f2ceb5f7b85919dffea643278e02362bf40e3c4bed1 \ + --hash=sha256:5013be3bea7612852c62a7437f3302c1cfb91ca7e703b194459db0b2b2e0d792 \ + --hash=sha256:5177aa44eddaa97c6ef0cc00c6d540edb64d51781d2f8fb941612ec61a92c9ed \ + --hash=sha256:51f71a6e2ad071e70c937e41fcb6c19f82c3f9f49831eba850ed4a106ffbb647 \ + --hash=sha256:538f5f865df6cd8c32dd63158a0e5b4f5dd08d732a7da8b7228a5a0776c8ce55 \ + --hash=sha256:53f3ed9118397074ff63a79b66b7fec1c84c782eecde35c5bc94e420a971c231 \ + --hash=sha256:559e3cabe522231909f9de98ef06929edbd53782046bd21aae0c72db6f2a0775 \ + --hash=sha256:57189a69c0891e4818853feaa521c972d22c880a001453addea015f48e3c3398 \ + --hash=sha256:57f80a898544db78ec6b0be6183bd1bc008933193d4199f5cde36b0e6bd5e062 \ + --hash=sha256:58346024d47e84f7d8b3e7f5d6faa1d58acbbe49a8771497872059f58c1d8ea5 \ + --hash=sha256:594131ce1aad18db3689781f806db1b065cdaa04f4df36b4c038d2013aefd0bf \ + --hash=sha256:5b96f0024e9840f449bd91b2d005c921a4b666055a0d1b6492463799f32aae22 \ + --hash=sha256:5c566b123dce7e4867ca518434cdfb9f84e5023771235b2e3107a26c9a41cbd8 \ + --hash=sha256:5d3dfb1f0ff146da7952867a9414f0c7a29762f8825a84879592612fd6139342 \ + --hash=sha256:5da703225374e3a4c8d4fd90e26fe7213a52004ec77f88b42b42e9e86d8c6d57 \ + --hash=sha256:5db43f249b4be9f99ef4b967863f37094fb40e67effafb78ba4f0356b6396104 \ + --hash=sha256:5eed32dad81d6ba8e62dc7b9ffa0500199385d7810a8dd9d4eafaceb8c6e20bb \ + --hash=sha256:602efcad4a42c184e81d43a2b7e6e4f524d619878f2b6ee2ba469011f47c8147 \ + --hash=sha256:614bca2c7cfa87ec95b703e691c3c5eb6c448b6dabbe9776ac53883152951729 \ + --hash=sha256:632a34590c090d1285ed5efa5a02be919f3f9a56a64bd25f693fe1e2d27a27fb \ + --hash=sha256:64af54dd1c3a45a27c04942f9a1a4683322bdd127f4745cca4e02549c1d2d2bb \ + --hash=sha256:6536d8677d2fff7e64cd0b98b976df9de7aee0e69590044c2af5f51b76b7a170 \ + --hash=sha256:656256c9f9303e47f07d5cb8ae4468285370adfafd7ba48aea33a458e7697626 \ + --hash=sha256:6696c8752aded28ff3b16f33ef28ce28fb5d209b80c206746f943199fcf5fd65 \ + --hash=sha256:6c7574528bc922f8757f34dd78ed60ab52b1c7973b630f5eae7ba33ec133ce71 \ + --hash=sha256:6cee733fe4ccb1737e0997135283c82341e5cfa9cf214b165f9087fb663aaf4f \ + --hash=sha256:6cf633fe83b1d4e6519d7259b33afe40fbba5d3f438730156971dd0cf7730610 \ + --hash=sha256:714503083a1f2065c9ad15340dd49ac8a8e948a505a705ffa1750cb951519113 \ + --hash=sha256:717b12fdc51819833704e85e6926d76981ffa3f780ef92e33ebb8b26d46bb230 \ + --hash=sha256:7258ee276e8772599bc19e14b36f6260306e21b637190cd7cb489a2449d48684 \ + --hash=sha256:72eb5ae575cc7ae2b23f6f8064a8b10f638c7149819ae9cc6d20ebd4d37a1629 \ + --hash=sha256:7345007c12780985de4fd740148776d1eee18c0d41407c6fa1e48c5450304fe5 \ + --hash=sha256:747476436f6891b9773374ce8d48edcc8b12cb5b61b67c6fb6289633747d088f \ + --hash=sha256:77f74e45a1e5574bbbf80181c8027b3a4c65c2248fffbd557bd596fff13102f9 \ + --hash=sha256:7801b7223db017b9c0c9ccf37e44524edb35a1544a1c032add22c061c6af0276 \ + --hash=sha256:78c794b643d214f1522e7a288bcf5a2de120d26cd170516749a4009dc92722c9 \ + --hash=sha256:7b5f97ecfede10d5b2870383620e2d25c8561e217c7bf9081073802b54248d2b \ + --hash=sha256:7dc4bdf008f77c88d544849c48c1a40faf25a5eff6cc466de2e8edc37c191fce \ + --hash=sha256:81f4ed9ca9644bc95cd976bfe10f7a4cafab8ffdc3aed52877d4600e445be7ef \ + --hash=sha256:82c0cedd280eab2e8291270e6c04894dbc096f8159a39dcf1807429f026ca3cc \ + --hash=sha256:8304be0982130954b7fd3aad18e2c6f8ee40254bc3d2e635991c16d77c91e2bd \ + --hash=sha256:83697b0ea1f10e7f5d8b26a4906fa851393c61546c63839643a2b7fe2d868061 \ + --hash=sha256:836f11d4474d3228e9909d97216faa4f7505df41cfaf3927eb29809de785a78d \ + --hash=sha256:83b9130b80b216d56fdf9e87131946b353c9627930c061955a101ea82b09fed9 \ + --hash=sha256:83d879362ddd0fedd3f2ab8ce7cce3da2049a6d51d16da8af73011c6edf4752f \ + --hash=sha256:848182a391fffdc25605443e832f5b443f25498edeccf9a64343fd84421ca04b \ + --hash=sha256:852bfe059720632e2f16a6a4745e41d20937b2bf2a42a401e2412046bb6971cc \ + --hash=sha256:868a8dcaff1a84ba78038e1cef14fc88ccf84d9b4d12ea604696e0693296aa56 \ + --hash=sha256:89b11a5cdd441aa463f6d34ca0241602bc09b001a76994b6059828494108c673 \ + --hash=sha256:89df64c10adfe340fb00330042537cdd6bf0d8d78bad73f29cfe5427eed7b084 \ + --hash=sha256:8ea8a141eeced4f6262ab6dd71c681ac546a558c30bb586abe087d814b5f85ea \ + --hash=sha256:8f454166c2ffed45636c8d501741e649851ba2f346c4eb73a64c07ac00428f20 \ + --hash=sha256:8f759eed402448c2bdbb492e4fba1f20668ffe29688605ea61f0f67f9e4e386d \ + --hash=sha256:9043877a917be88ccf230aa5667c1bd059bce80f4c2727e4defa1b29b7f48b08 \ + --hash=sha256:942bc86e9be6fdd6e1175048f5fe8f8fdaaf2309dd1323ef1e155a69cd346780 \ + --hash=sha256:947a585bcaa235702b7c59433b485489397f9a163b3f56058b9463a46fd9b74c \ + --hash=sha256:950ac754d16daea42038f38e7465eb84cda4d08d7343c1c915771b29470f065a \ + --hash=sha256:98d8ac1129b4dd39098cffed94d1284aceb61c3aa396757ccc736ac392e4cee5 \ + --hash=sha256:98ee81b4b7f3023c9cb04a78cc67610baffcb5812d92f2096cb5a5efc6f19437 \ + --hash=sha256:9b2ce44bf8f4a1d01f418b3110ff8dff32fd3f3e836c0e06333c3725f243fa6c \ + --hash=sha256:9d45eee3a95a8b61e5b568580caac91f1502ddb731aaf8f4aa448a98660b2fb4 \ + --hash=sha256:9db455cb649dcfe4504d6d68a6d83a7315a99a3ca59871dc3ff840671f99adba \ + --hash=sha256:9df56e6df96a60590935e22373041cccc91fd55858763dcffb55bf63b3a2b396 \ + --hash=sha256:9e80238259655bf69d7bcd08226a970d7f42605f3157786bfa76dd13472d7fa0 \ + --hash=sha256:9f23083e1bd9d901f844af7a126727c486e7eada9a1a6791c8f7e73f94fac656 \ + --hash=sha256:a2489d3a776fa380cb8e71f54c7fda268a9baf3de9b1395093fd280f95735907 \ + --hash=sha256:a5cd96f6dcdf4fa657b2d95668d71d58455248f98712ecffaa9c528edf40ccae \ + --hash=sha256:a5eed9d41995a83f3332b4e3396abb7f433cac584222bd7e305b606d8353861e \ + --hash=sha256:a6617f30641ba0d8baa1635fbefb1dffc5165ec36d26921bd5cee13497cd937a \ + --hash=sha256:a6e088bd7870775624256a0d84c2a6714afd223b2eeb56b0ca58398e52a32fda \ + --hash=sha256:a98b2f95cab589e0f5e92c48431afb4d56238b8bf6668edcc66166180e9b509b \ + --hash=sha256:ad52a0e4bcc0ba956a953a169d1feec2734a64981d689e4fc8f490f7bf91af60 \ + --hash=sha256:af0c9fedc4a2c24e8664953882fe8185f3790b8338c9c700f76f5ad660817711 \ + --hash=sha256:afe6380a0e9653a87aa1e6e88fb47718113e5563c7a1cb2bcc23c1d8e17e3961 \ + --hash=sha256:affb37f152e55b5e4494bb9d0107f7bb08515c6704fbed82d9f61214d74adc17 \ + --hash=sha256:b0093cf7eeb91b84776e8742113afa4bdf47533d36cf719179aaaf1f56f6f8bf \ + --hash=sha256:b0de4bf3aa66363552d52c6a89003c479911f12098cd48a53d44a0f7a25f7c46 \ + --hash=sha256:b30e01a0b97a4bc3f519a4d7a82da3dc53251fb0de5eeea8660dcd4ff094c0c2 \ + --hash=sha256:b3ba794c3d885803db6c3116686923f1ec13bc86e621e169a375282b63ea1cc6 \ + --hash=sha256:b3e1107fe5ca030f946dfa59fdbb66b5df121c8432f14b0bdd282d17b297f4eb \ + --hash=sha256:b5196cc2574cfec572a5f3fb7cfa5ade27305ae3d06516a082132441aff4c83a \ + --hash=sha256:b6fa3116e40e14e7782fb1a9f872f94b5997de21127c95545ce40196ac1351c5 \ + --hash=sha256:bb70573d2995d23932e2871120f78d798ebc3572e54c09e694a18ced95c5f8d9 \ + --hash=sha256:bbcdf9c92d21c65bc75426eecea724c8fa0d35a6e201fdf1630011d4cc3aa685 \ + --hash=sha256:bcab50a389cc04d87f90092af78a6adba2ab3deca63175a3344ca83514045315 \ + --hash=sha256:bf28f55e427e0483acb1f666bd0d869b6d5e5a716680c216ad7befe3d4cfba2e \ + --hash=sha256:bfcd82852c62a60e314670a9602de354c4460f8adad916e2e42a20860c7870bc \ + --hash=sha256:c240939e963653054fc7e4a17c382829cda4aa88a7daf0af841715dbded1b497 \ + --hash=sha256:c31a2649bcf1fe97cf11c79848d761df33ac46b3896942d31b640557b486ff6b \ + --hash=sha256:c4ed42965c2cd9081f011be22f69d0e65d3b6165fe7734072fd0c232840bbd4e \ + --hash=sha256:c73b6f652f0745425aa6378319c331293b5341756262e9408ed3d45f183375e6 \ + --hash=sha256:c85949d02c85adf6d786eb94858e124989a632a4e65739835b2fc5761827fac3 \ + --hash=sha256:c919f38cd3f0b5e8d30b81fd6cac688cf9221560340f0c35cbbb8b2bd77ad6ac \ + --hash=sha256:c959f88160b13b4e730b0d75b459b7929fc0d2225c284c9683ac95d6feeeac6a \ + --hash=sha256:cb3fe820c27593f170770d6c8d791936cf6275d9269405fbb7b30a55363c10c8 \ + --hash=sha256:cf399fac542a1c7a4734a435b93df2c55e858c7d31abf6c1bdf46f9ae67fbfd0 \ + --hash=sha256:d0b48cdf690a64cedf7258c3dc9506cc41fc86edd7739c40e3098952265dc068 \ + --hash=sha256:d247b34bf433c92b41689318fd25d246313cab2275a6a47e2efac178b80d6efe \ + --hash=sha256:d48acabb1e5cb0071009f80d71d7f01b6ba2c1d4b869b1352bb5df3f11bf7dfd \ + --hash=sha256:d5006c65ec507a333479e76e00e2c368781f16c24ededa764763956b32a0e93e \ + --hash=sha256:d58ce8b6cfa9c4d2f230557f69caf7c06369e318015d0b19485095bc2c5963ab \ + --hash=sha256:d59e71153fe9ff85648d00e18649b07e9b22c797291abb7e27274fa06df8b838 \ + --hash=sha256:d6a5c0bce213b23b0166fe0d35bcbbe23ce4b968f257cc7eb6fd57cb8e1e6297 \ + --hash=sha256:daa86e4b68221d38e669bb236ba112d0335353829fb627c82e5909e4bbe8694c \ + --hash=sha256:db77278a6eddadbf44ce5aae2fee5ebb4d061f026b1ce2130d058cd4d7a7b670 \ + --hash=sha256:dfe0580fbfd5e4af87d0cc52d2044f155d55ebd8c8a93568758a2ea7d8e15975 \ + --hash=sha256:e14800b9b10bb39d7a60ad4a310e403164d7b8988a27ae933d4e40618a44088e \ + --hash=sha256:e2a845687219ba3214126f14a8a5861f97c9e065a7d0b8252adb6df13eea86fb \ + --hash=sha256:e3b87cbd974512c0c5fc7b469c36b2cdc9ee6d76e4ec78bccb2c7184611c49b0 \ + --hash=sha256:e4a6443968c4e8dc69967e12776776a5952c119cc1bd94168ad1c5ad667c2be1 \ + --hash=sha256:e605e0b8abca9457abd5bee737e086ab145a20c25083ef1113013612268872ff \ + --hash=sha256:e6e49370822c1f4d8d90e678b06dbcb08b51a026a7c4b55479e7d467f2e813bc \ + --hash=sha256:e710ad822c493fb80a4fbc1e3d0a807b1422cb90adbe64378f98291b7fa48fef \ + --hash=sha256:ea6a3e734b0fd41b82784a400be946821900daebe610c050a5e0760838a34f99 \ + --hash=sha256:ec55d80e9b8a519d742669e0b49e8ce9e6747be42bf3c138158b6543a9c8e489 \ + --hash=sha256:ed8bcdab6692fd4ad0dd6241807a24a640a376764460023b8d462d745e6b7b27 \ + --hash=sha256:f377012b86c0a23a1df0cf5a1b05aa7187649e472f71c7892e5f2c2815bbe74f \ + --hash=sha256:f6114692261eff4266386cdec0f7d87eee24e317ab397c218b7ae6a76b4c6339 \ + --hash=sha256:f8044cf4c77f37968b8c4cbcbf7a0f355d8a437877ae18eba23e3aad953a6cc7 \ + --hash=sha256:f8ed8940435834141061da26d27c4dd0d18fb69777bf431f5c6cc46b43349113 \ + --hash=sha256:f93e408255ddce525189bf11feaa1be7ee35e55f486c299c97d9caa68d724a5b \ + --hash=sha256:fb9e256a357dfcede7818c6d34e70db2d6b664394803d1de4b6984d2de76c0f1 +zstandard==0.25.0 \ + --hash=sha256:011d388c76b11a0c165374ce660ce2c8efa8e5d87f34996aa80f9c0816698b64 \ + --hash=sha256:01582723b3ccd6939ab7b3a78622c573799d5d8737b534b86d0e06ac18dbde4a \ + --hash=sha256:05353cef599a7b0b98baca9b068dd36810c3ef0f42bf282583f438caf6ddcee3 \ + --hash=sha256:05df5136bc5a011f33cd25bc9f506e7426c0c9b3f9954f056831ce68f3b6689f \ + --hash=sha256:06acb75eebeedb77b69048031282737717a63e71e4ae3f77cc0c3b9508320df6 \ + --hash=sha256:07b527a69c1e1c8b5ab1ab14e2afe0675614a09182213f21a0717b62027b5936 \ + --hash=sha256:0bbc9a0c65ce0eea3c34a691e3c4b6889f5f3909ba4822ab385fab9057099431 \ + --hash=sha256:0be7622c37c183406f3dbf0cba104118eb16a4ea7359eeb5752f0794882fc250 \ + --hash=sha256:106281ae350e494f4ac8a80470e66d1fe27e497052c8d9c3b95dc4cf1ade81aa \ + --hash=sha256:10ef2a79ab8e2974e2075fb984e5b9806c64134810fac21576f0668e7ea19f8f \ + --hash=sha256:1673b7199bbe763365b81a4f3252b8e80f44c9e323fc42940dc8843bfeaf9851 \ + --hash=sha256:172de1f06947577d3a3005416977cce6168f2261284c02080e7ad0185faeced3 \ + --hash=sha256:181eb40e0b6a29b3cd2849f825e0fa34397f649170673d385f3598ae17cca2e9 \ + --hash=sha256:1869da9571d5e94a85a5e8d57e4e8807b175c9e4a6294e3b66fa4efb074d90f6 \ + --hash=sha256:19796b39075201d51d5f5f790bf849221e58b48a39a5fc74837675d8bafc7362 \ + --hash=sha256:1cd5da4d8e8ee0e88be976c294db744773459d51bb32f707a0f166e5ad5c8649 \ + --hash=sha256:1f3689581a72eaba9131b1d9bdbfe520ccd169999219b41000ede2fca5c1bfdb \ + --hash=sha256:1f830a0dac88719af0ae43b8b2d6aef487d437036468ef3c2ea59c51f9d55fd5 \ + --hash=sha256:223415140608d0f0da010499eaa8ccdb9af210a543fac54bce15babbcfc78439 \ + --hash=sha256:22a06c5df3751bb7dc67406f5374734ccee8ed37fc5981bf1ad7041831fa1137 \ + --hash=sha256:22a086cff1b6ceca18a8dd6096ec631e430e93a8e70a9ca5efa7561a00f826fa \ + --hash=sha256:23ebc8f17a03133b4426bcc04aabd68f8236eb78c3760f12783385171b0fd8bd \ + --hash=sha256:25f8f3cd45087d089aef5ba3848cd9efe3ad41163d3400862fb42f81a3a46701 \ + --hash=sha256:2b6bd67528ee8b5c5f10255735abc21aa106931f0dbaf297c7be0c886353c3d0 \ + --hash=sha256:2e54296a283f3ab5a26fc9b8b5d4978ea0532f37b231644f367aa588930aa043 \ + --hash=sha256:3756b3e9da9b83da1796f8809dd57cb024f838b9eeafde28f3cb472012797ac1 \ + --hash=sha256:37daddd452c0ffb65da00620afb8e17abd4adaae6ce6310702841760c2c26860 \ + --hash=sha256:3a39c94ad7866160a4a46d772e43311a743c316942037671beb264e395bdd611 \ + --hash=sha256:3b870ce5a02d4b22286cf4944c628e0f0881b11b3f14667c1d62185a99e04f53 \ + --hash=sha256:3c83b0188c852a47cd13ef3bf9209fb0a77fa5374958b8c53aaa699398c6bd7b \ + --hash=sha256:4203ce3b31aec23012d3a4cf4a2ed64d12fea5269c49aed5e4c3611b938e4088 \ + --hash=sha256:457ed498fc58cdc12fc48f7950e02740d4f7ae9493dd4ab2168a47c93c31298e \ + --hash=sha256:474d2596a2dbc241a556e965fb76002c1ce655445e4e3bf38e5477d413165ffa \ + --hash=sha256:4b14abacf83dfb5c25eb4e4a79520de9e7e205f72c9ee7702f91233ae57d33a2 \ + --hash=sha256:4b6d83057e713ff235a12e73916b6d356e3084fd3d14ced499d84240f3eecee0 \ + --hash=sha256:4d441506e9b372386a5271c64125f72d5df6d2a8e8a2a45a0ae09b03cb781ef7 \ + --hash=sha256:4f187a0bb61b35119d1926aee039524d1f93aaf38a9916b8c4b78ac8514a0aaf \ + --hash=sha256:51526324f1b23229001eb3735bc8c94f9c578b1bd9e867a0a646a3b17109f388 \ + --hash=sha256:53e08b2445a6bc241261fea89d065536f00a581f02535f8122eba42db9375530 \ + --hash=sha256:53f94448fe5b10ee75d246497168e5825135d54325458c4bfffbaafabcc0a577 \ + --hash=sha256:5a56ba0db2d244117ed744dfa8f6f5b366e14148e00de44723413b2f3938a902 \ + --hash=sha256:5f1ad7bf88535edcf30038f6919abe087f606f62c00a87d7e33e7fc57cb69fcc \ + --hash=sha256:5f5e4c2a23ca271c218ac025bd7d635597048b366d6f31f420aaeb715239fc98 \ + --hash=sha256:6a573a35693e03cf1d67799fd01b50ff578515a8aeadd4595d2a7fa9f3ec002a \ + --hash=sha256:6c0e5a65158a7946e7a7affa6418878ef97ab66636f13353b8502d7ea03c8097 \ + --hash=sha256:6dffecc361d079bb48d7caef5d673c88c8988d3d33fb74ab95b7ee6da42652ea \ + --hash=sha256:7030defa83eef3e51ff26f0b7bfb229f0204b66fe18e04359ce3474ac33cbc09 \ + --hash=sha256:7149623bba7fdf7e7f24312953bcf73cae103db8cae49f8154dd1eadc8a29ecb \ + --hash=sha256:72d35d7aa0bba323965da807a462b0966c91608ef3a48ba761678cb20ce5d8b7 \ + --hash=sha256:75ffc32a569fb049499e63ce68c743155477610532da1eb38e7f24bf7cd29e74 \ + --hash=sha256:7713e1179d162cf5c7906da876ec2ccb9c3a9dcbdffef0cc7f70c3667a205f0b \ + --hash=sha256:78228d8a6a1c177a96b94f7e2e8d012c55f9c760761980da16ae7546a15a8e9b \ + --hash=sha256:7b3c3a3ab9daa3eed242d6ecceead93aebbb8f5f84318d82cee643e019c4b73b \ + --hash=sha256:809c5bcb2c67cd0ed81e9229d227d4ca28f82d0f778fc5fea624a9def3963f91 \ + --hash=sha256:81dad8d145d8fd981b2962b686b2241d3a1ea07733e76a2f15435dfb7fb60150 \ + --hash=sha256:85304a43f4d513f5464ceb938aa02c1e78c2943b29f44a750b48b25ac999a049 \ + --hash=sha256:89c4b48479a43f820b749df49cd7ba2dbc2b1b78560ecb5ab52985574fd40b27 \ + --hash=sha256:8e735494da3db08694d26480f1493ad2cf86e99bdd53e8e9771b2752a5c0246a \ + --hash=sha256:913cbd31a400febff93b564a23e17c3ed2d56c064006f54efec210d586171c00 \ + --hash=sha256:9174f4ed06f790a6869b41cba05b43eeb9a35f8993c4422ab853b705e8112bbd \ + --hash=sha256:9300d02ea7c6506f00e627e287e0492a5eb0371ec1670ae852fefffa6164b072 \ + --hash=sha256:933b65d7680ea337180733cf9e87293cc5500cc0eb3fc8769f4d3c88d724ec5c \ + --hash=sha256:9654dbc012d8b06fc3d19cc825af3f7bf8ae242226df5f83936cb39f5fdc846c \ + --hash=sha256:98750a309eb2f020da61e727de7d7ba3c57c97cf6213f6f6277bb7fb42a8e065 \ + --hash=sha256:99c0c846e6e61718715a3c9437ccc625de26593fea60189567f0118dc9db7512 \ + --hash=sha256:a1a4ae2dec3993a32247995bdfe367fc3266da832d82f8438c8570f989753de1 \ + --hash=sha256:a3f79487c687b1fc69f19e487cd949bf3aae653d181dfb5fde3bf6d18894706f \ + --hash=sha256:a4089a10e598eae6393756b036e0f419e8c1d60f44a831520f9af41c14216cf2 \ + --hash=sha256:a51ff14f8017338e2f2e5dab738ce1ec3b5a851f23b18c1ae1359b1eecbee6df \ + --hash=sha256:a5a419712cf88862a45a23def0ae063686db3d324cec7edbe40509d1a79a0aab \ + --hash=sha256:a9ec8c642d1ec73287ae3e726792dd86c96f5681eb8df274a757bf62b750eae7 \ + --hash=sha256:aaf21ba8fb76d102b696781bddaa0954b782536446083ae3fdaa6f16b25a1c4b \ + --hash=sha256:ab85470ab54c2cb96e176f40342d9ed41e58ca5733be6a893b730e7af9c40550 \ + --hash=sha256:b9af1fe743828123e12b41dd8091eca1074d0c1569cc42e6e1eee98027f2bbd0 \ + --hash=sha256:bfc4e20784722098822e3eee42b8e576b379ed72cca4a7cb856ae733e62192ea \ + --hash=sha256:bfd06b1c5584b657a2892a6014c2f4c20e0db0208c159148fa78c65f7e0b0277 \ + --hash=sha256:c19bcdd826e95671065f8692b5a4aa95c52dc7a02a4c5a0cac46deb879a017a2 \ + --hash=sha256:c2ba942c94e0691467ab901fc51b6f2085ff48f2eea77b1a48240f011e8247c7 \ + --hash=sha256:c8e167d5adf59476fa3e37bee730890e389410c354771a62e3c076c86f9f7778 \ + --hash=sha256:ca54090275939dc8ec5dea2d2afb400e0f83444b2fc24e07df7fdef677110859 \ + --hash=sha256:d7541afd73985c630bafcd6338d2518ae96060075f9463d7dc14cfb33514383d \ + --hash=sha256:d8c56bb4e6c795fc77d74d8e8b80846e1fb8292fc0b5060cd8131d522974b751 \ + --hash=sha256:da469dc041701583e34de852d8634703550348d5822e66a0c827d39b05365b12 \ + --hash=sha256:daab68faadb847063d0c56f361a289c4f268706b598afbf9ad113cbe5c38b6b2 \ + --hash=sha256:e05ab82ea7753354bb054b92e2f288afb750e6b439ff6ca78af52939ebbc476d \ + --hash=sha256:e09bb6252b6476d8d56100e8147b803befa9a12cea144bbe629dd508800d1ad0 \ + --hash=sha256:e29f0cf06974c899b2c188ef7f783607dbef36da4c242eb6c82dcd8b512855e3 \ + --hash=sha256:e59fdc271772f6686e01e1b3b74537259800f57e24280be3f29c8a0deb1904dd \ + --hash=sha256:e7360eae90809efd19b886e59a09dad07da4ca9ba096752e61a2e03c8aca188e \ + --hash=sha256:e96594a5537722fdfb79951672a2a63aec5ebfb823e7560586f7484819f2a08f \ + --hash=sha256:ea9d54cc3d8064260114a0bbf3479fc4a98b21dffc89b3459edd506b69262f6e \ + --hash=sha256:ec996f12524f88e151c339688c3897194821d7f03081ab35d31d1e12ec975e94 \ + --hash=sha256:f27662e4f7dbf9f9c12391cb37b4c4c3cb90ffbd3b1fb9284dadbbb8935fa708 \ + --hash=sha256:f373da2c1757bb7f1acaf09369cdc1d51d84131e50d5fa9863982fd626466313 \ + --hash=sha256:f5aeea11ded7320a84dcdd62a3d95b5186834224a9e55b92ccae35d21a8b63d4 \ + --hash=sha256:f604efd28f239cc21b3adb53eb061e2a205dc164be408e553b41ba2ffe0ca15c \ + --hash=sha256:f67e8f1a324a900e75b5e28ffb152bcac9fbed1cc7b43f99cd90f395c4375344 \ + --hash=sha256:fd7a5004eb1980d3cefe26b2685bcb0b17989901a70a1040d1ac86f1d898c551 \ + --hash=sha256:ffef5a74088f1e09947aecf91011136665152e0b4b359c42be3373897fb39b01 diff --git a/backend/requirements.txt b/backend/requirements.txt index d8d5ae2..23c24b5 100644 --- a/backend/requirements.txt +++ b/backend/requirements.txt @@ -6,6 +6,7 @@ python-dotenv>=1.0.0 python-multipart>=0.0.18 # email-validator is the dependency that pydantic[email] pulls in for EmailStr. email-validator>=2.2.0 +PyJWT[crypto]>=2.13.0 # LLM langchain>=0.3.14 @@ -29,6 +30,8 @@ asyncpg>=0.30.0 # Utilities structlog>=24.4.0 prometheus-client>=0.21.0 +packaging>=24.0 +keyring==25.7.0 # Tests / local verification pytest>=8.3.0 diff --git a/backend/services/agent_run_control.py b/backend/services/agent_run_control.py new file mode 100644 index 0000000..dcb7fdf --- /dev/null +++ b/backend/services/agent_run_control.py @@ -0,0 +1,280 @@ +"""Fail-closed control-plane helpers for durable AgentRun events and AI egress.""" + +from __future__ import annotations + +import hashlib +import json +import math +import re +import time +import uuid +from collections.abc import Iterable, Mapping +from typing import Any + +from backend.api.security.ai_policy import CLOUD_PROVIDERS, LOCAL_PROVIDERS, EffectiveAiPolicy + + +_RANGE_PATTERN = re.compile( + r"(?:'[^']+'|[A-Za-z0-9_ .-]+)!\$?[A-Z]{1,3}\$?\d+(?::\$?[A-Z]{1,3}\$?\d+)?" +) +_SENSITIVE_KEY_PATTERN = re.compile( + r"(?:api[_-]?key|authorization|bearer|credential|password|secret|token)", + re.IGNORECASE, +) + + +def canonical_digest(value: Any) -> str: + encoded = json.dumps( + value, + sort_keys=True, + separators=(",", ":"), + ensure_ascii=False, + default=str, + ).encode("utf-8") + return hashlib.sha256(encoded).hexdigest() + + +def egress_request_digest(request_material: Mapping[str, Any]) -> str: + """Bind approval to every material input that may cross the provider boundary.""" + documents = request_material.get("documents") or [] + normalized_documents = [ + dict(document) + for document in documents + if isinstance(document, Mapping) + ] + material = { + "schema": "2026-08-agent-egress-request-v1", + "sessionId": str( + request_material.get("session_id") + or request_material.get("sessionId") + or "" + ), + "prompt": str(request_material.get("prompt") or ""), + "provider": str(request_material.get("provider") or "").strip().lower(), + "model": str(request_material.get("model") or ""), + "workbookContext": request_material.get("workbook_context") + or request_material.get("workbookContext") + or {}, + "documents": normalized_documents, + "templateId": request_material.get("template_id") + or request_material.get("templateId"), + "templateParams": request_material.get("template_params") + or request_material.get("templateParams") + or {}, + "validationProfile": request_material.get("validation_profile") + or request_material.get("validationProfile"), + "accuracyMode": request_material.get("accuracy_mode") + or request_material.get("accuracyMode"), + "approvalPolicy": request_material.get("approval_policy") + or request_material.get("approvalPolicy"), + } + return canonical_digest(material) + + +def build_egress_preview( + *, + session_id: str, + provider: str, + model: str, + request_digest: str, + data_classes: Iterable[str], + workbook_ranges: Iterable[str] = (), + redactions: Iterable[str] = (), + estimated_input_tokens: int = 0, + now: float | None = None, + ttl_seconds: int = 300, +) -> dict[str, Any]: + issued_at = float(time.time() if now is None else now) + ttl = min(max(int(ttl_seconds), 30), 900) + expires_at = issued_at + ttl + normalized_provider = str(provider or "").strip().lower() + normalized_classes = sorted({str(item) for item in data_classes if str(item)}) + normalized_ranges = sorted({str(item) for item in workbook_ranges if str(item)})[:100] + normalized_redactions = sorted({str(item) for item in redactions if str(item)})[:100] + storage_disabled = normalized_provider == "openai" + approval_id = f"egress_{uuid.uuid4().hex}" + digest_material = { + "schema": "2026-08-agent-egress-approval-v1", + "sessionId": session_id, + "provider": normalized_provider, + "model": str(model or ""), + "requestDigest": request_digest, + "dataClasses": normalized_classes, + "workbookRanges": normalized_ranges, + "redactions": normalized_redactions, + "estimatedInputTokens": max(int(estimated_input_tokens), 0), + "providerStorageDisabled": storage_disabled, + "providerStorageControl": "store:false" + if storage_disabled + else "provider_api_has_no_request_storage_flag", + "fallbackAllowed": False, + "cloudFallbackAllowed": False, + "retentionPosture": "provider_storage_disabled" + if storage_disabled + else "provider_contract_required", + "expiresAt": expires_at, + } + return { + **digest_material, + "approvalId": approval_id, + "approvalDigest": canonical_digest(digest_material), + "issuedAt": issued_at, + "consumedAt": None, + "consumptionId": "", + } + + +def build_egress_receipt( + *, + provider: str, + model: str, + request_digest: str, + approval_id: str = "", + approval_digest: str = "", + actor_digest: str = "", + provider_call_attempted: bool = False, + now: float | None = None, +) -> dict[str, Any]: + normalized_provider = str(provider or "").strip().lower() + created_at = float(time.time() if now is None else now) + is_cloud = normalized_provider in CLOUD_PROVIDERS + storage_disabled = normalized_provider == "openai" + body = { + "schemaVersion": "2026-08-agent-egress-receipt-v1", + "provider": normalized_provider, + "model": str(model or ""), + "scope": "cloud" if is_cloud else "local", + "requestDigest": request_digest, + "approvalId": approval_id, + "approvalDigest": approval_digest, + "approvalRequired": is_cloud, + "actorDigest": actor_digest, + "fallbackAllowed": False, + "cloudFallbackAllowed": False, + "providerStorageDisabled": storage_disabled, + "providerStorageControl": ( + "store:false" + if storage_disabled + else "not_applicable" + if normalized_provider in LOCAL_PROVIDERS + else "provider_api_has_no_request_storage_flag" + ), + "providerCallAttempted": bool(provider_call_attempted), + "egressPerformed": bool(is_cloud and provider_call_attempted), + "createdAt": created_at, + } + return { + **body, + "receiptId": f"egress_receipt_{canonical_digest(body)[:24]}", + } + + +def resolve_no_fallback_provider( + policy: EffectiveAiPolicy, + requested_provider: str | None, +) -> str: + """Choose exactly one policy-approved route; never construct a fallback chain.""" + if requested_provider: + return policy.require_provider(requested_provider) + for candidate in ("lmstudio", "ollama"): + try: + return policy.require_provider(candidate) + except ValueError: + continue + from backend.api.security.ai_policy import AiPolicyViolation + + raise AiPolicyViolation( + "No local AI provider is approved; cloud fallback is disabled for agent runs.", + "no_approved_local_ai_provider", + ) + + +def classify_egress_data( + *, + prompt: str, + workbook_context: Mapping[str, Any] | None, + documents: Iterable[Mapping[str, Any]], + additional_material: Mapping[str, Any] | None = None, +) -> tuple[list[str], list[str], list[str], int]: + context = dict(workbook_context or {}) + document_rows = [dict(item) for item in documents] + classes = ["user_prompt"] if prompt else [] + if context: + classes.extend(["workbook_values", "metadata"]) + if _contains_formula(context): + classes.append("workbook_formulas") + if document_rows: + classes.append("document_text") + material = { + "prompt": prompt, + "workbookContext": context, + "documents": document_rows, + "additionalMaterial": dict(additional_material or {}), + } + ranges = sorted(_extract_ranges(material))[:100] + redactions = sorted(_sensitive_paths(material))[:100] + estimated_tokens = math.ceil( + len(json.dumps(material, ensure_ascii=False, default=str)) / 4 + ) + return sorted(set(classes)), ranges, redactions, estimated_tokens + + +def redact_egress_payload(value: Any) -> Any: + """Apply the exact key-based redactions advertised by ``classify_egress_data``.""" + if isinstance(value, Mapping): + return { + str(key): "[REDACTED]" + if _SENSITIVE_KEY_PATTERN.search(str(key)) + else redact_egress_payload(item) + for key, item in value.items() + } + if isinstance(value, list): + return [redact_egress_payload(item) for item in value] + if isinstance(value, tuple): + return [redact_egress_payload(item) for item in value] + return value + + +def actor_subject_digest(subject: str) -> str: + normalized = str(subject or "").strip() + return hashlib.sha256(normalized.encode("utf-8")).hexdigest()[:16] if normalized else "" + + +def _contains_formula(value: Any) -> bool: + if isinstance(value, Mapping): + return any( + str(key).lower() in {"formula", "formulas"} or _contains_formula(item) + for key, item in value.items() + ) + if isinstance(value, list): + return any(_contains_formula(item) for item in value) + return isinstance(value, str) and value.startswith("=") + + +def _extract_ranges(value: Any) -> set[str]: + found: set[str] = set() + if isinstance(value, Mapping): + for key, item in value.items(): + found.update(_RANGE_PATTERN.findall(str(key))) + found.update(_extract_ranges(item)) + elif isinstance(value, list): + for item in value: + found.update(_extract_ranges(item)) + elif isinstance(value, str): + found.update(_RANGE_PATTERN.findall(value)) + return found + + +def _sensitive_paths(value: Any, path: str = "") -> set[str]: + found: set[str] = set() + if isinstance(value, Mapping): + for key, item in value.items(): + item_path = f"{path}.{key}" if path else str(key) + if _SENSITIVE_KEY_PATTERN.search(str(key)): + found.add(item_path) + else: + found.update(_sensitive_paths(item, item_path)) + elif isinstance(value, list): + for index, item in enumerate(value): + found.update(_sensitive_paths(item, f"{path}[{index}]")) + return found diff --git a/backend/services/agent_runner.py b/backend/services/agent_runner.py index f525251..a4c5c09 100644 --- a/backend/services/agent_runner.py +++ b/backend/services/agent_runner.py @@ -9,14 +9,9 @@ AgentCheckpoint, AgentRun, AgentRunStep, - OperationPlan, - OperationPlanV2, - OperationPlanV3, OperationSimulationResult, RepairPlan, - SourceEvidenceGraph, WorkbookInvariantReport, - WorkbookSemanticMapV2, WorkbookOperationV2, ) from backend.services.operation_planner import build_operation_plan @@ -198,13 +193,23 @@ def create_repair_plan(self, run: AgentRun) -> tuple[AgentRun, RepairPlan]: for item in (formula_graph.get("materialFormulaFailures") if isinstance(formula_graph, dict) else []) or [] if isinstance(item, dict) ) - operations = [_repair_read_operation(item) for item in dirty_ranges or [] if isinstance(item, dict)] + operations = [ + operation + for item in dirty_ranges or [] + if isinstance(item, dict) + and (operation := _repair_read_operation(item)) is not None + ] operations.extend( - _repair_read_operation({"sheet": cell.split("!", 1)[0], "range": cell.split("!", 1)[1]}) + operation for cell in (simulation.get("affectedOutputCells") if isinstance(simulation, dict) else []) or [] if isinstance(cell, str) and "!" in cell + and ( + operation := _repair_read_operation( + {"sheet": cell.split("!", 1)[0], "range": cell.split("!", 1)[1]} + ) + ) is not None ) - operations = [op for op in operations if op is not None][:25] + operations = operations[:25] checkpoint_required = any( isinstance(item, dict) and (item.get("severity") == "error" or item.get("requiresCheckpoint")) for item in [*(issues or []), *(findings or [])] diff --git a/backend/services/agent_runner_v3.py b/backend/services/agent_runner_v3.py index 8decc98..0ae9367 100644 --- a/backend/services/agent_runner_v3.py +++ b/backend/services/agent_runner_v3.py @@ -3,6 +3,7 @@ from __future__ import annotations import hashlib +import json import time from typing import Any @@ -20,7 +21,7 @@ async def create_run(self, **kwargs: Any) -> dict[str, Any]: payload.update( { "schemaVersion": "2026-05-agent-run-v3", - "idempotencyKey": _idempotency_key(payload), + "idempotencyKey": _idempotency_key(payload, kwargs), "stepArtifacts": [item.model_dump(mode="json", by_alias=True) for item in step_artifacts], "backgroundJob": { "mode": "local_inline_persisted_steps", @@ -89,6 +90,34 @@ def _accuracy_report(result: dict[str, Any]) -> dict[str, Any]: } -def _idempotency_key(payload: dict[str, Any]) -> str: - material = "|".join([str(payload.get("sessionId")), str(payload.get("prompt")), str(payload.get("createdAt"))]) - return hashlib.sha256(material.encode("utf-8")).hexdigest()[:24] +def _idempotency_key(payload: dict[str, Any], request_inputs: dict[str, Any] | None = None) -> str: + """Return a stable digest of semantic request inputs. + + Runtime-generated run IDs and timestamps are intentionally excluded, so + retrying the same request produces the same key. Workbook context and + model/policy selections remain part of the digest so materially different + work cannot collide merely because it shares a prompt. + """ + request_inputs = request_inputs or {} + documents = request_inputs.get("documents") or [] + document_ids = [ + str(item.get("document_id") or item.get("id") or "") + for item in documents + if isinstance(item, dict) + ] + material = { + "schema": "2026-08-agent-run-idempotency-v1", + "session_id": request_inputs.get("session_id") or payload.get("sessionId") or payload.get("session_id"), + "prompt": request_inputs.get("prompt") or payload.get("prompt"), + "provider": request_inputs.get("provider"), + "model": request_inputs.get("model"), + "template_id": request_inputs.get("template_id"), + "template_params": request_inputs.get("template_params") or {}, + "validation_profile": request_inputs.get("validation_profile"), + "accuracy_mode": request_inputs.get("accuracy_mode"), + "approval_policy": request_inputs.get("approval_policy"), + "source_document_ids": sorted(item for item in document_ids if item), + "workbook_context": request_inputs.get("workbook_context") or {}, + } + canonical = json.dumps(material, sort_keys=True, separators=(",", ":"), default=str) + return hashlib.sha256(canonical.encode("utf-8")).hexdigest()[:24] diff --git a/backend/services/brain_kernel.py b/backend/services/brain_kernel.py index eff20b4..9dc63ae 100644 --- a/backend/services/brain_kernel.py +++ b/backend/services/brain_kernel.py @@ -127,11 +127,14 @@ def build_brain_artifact_bundle( ) for artifact_id in source_artifact_ids or []: lower = str(artifact_id).lower() - if not bundle.source_evidence_id and any(token in lower for token in ("source", "evidence", "sev_")): + is_source_artifact = any(token in lower for token in ("source", "evidence", "sev_")) + is_truth_artifact = "truth" in lower + is_formula_artifact = "formula" in lower or ("graph" in lower and not is_source_artifact and not is_truth_artifact) + if not bundle.source_evidence_id and is_source_artifact: bundle.source_evidence_id = str(artifact_id) - if not bundle.truth_model_id and "truth" in lower: + if not bundle.truth_model_id and is_truth_artifact: bundle.truth_model_id = str(artifact_id) - if not bundle.formula_graph_id and any(token in lower for token in ("graph", "formula")): + if not bundle.formula_graph_id and is_formula_artifact: bundle.formula_graph_id = str(artifact_id) bundle.rollback_snapshot_ids = _merge_strings(bundle.rollback_snapshot_ids, rollback_snapshot_ids or []) bundle.audit_ids = _merge_strings(bundle.audit_ids, audit_ids or []) diff --git a/backend/services/cell_value_safety.py b/backend/services/cell_value_safety.py new file mode 100644 index 0000000..92b2d73 --- /dev/null +++ b/backend/services/cell_value_safety.py @@ -0,0 +1,73 @@ +"""Cell value safety helpers. + +Defense against formula/CSV injection when AI-generated or document-extracted +values flow into cell writes. We don't *block* the user from explicitly writing +a formula — but if a value is supposed to be literal text and starts with +``=``, ``+``, ``-``, ``@``, or a leading tab/CR, Excel will evaluate it as a +formula. Prefixing with a single quote tells Excel to treat the cell as text. + +Use ``sanitize_literal`` at the AI/operation boundary for values that should +be text. ``is_potential_formula`` is the predicate behind risk scoring; use +it to surface a warning when a ``write_cell`` op carries a value that *looks* +like a formula but isn't declared as one. +""" + +from __future__ import annotations + +from typing import Any + +# Microsoft's own guidance: Excel treats a cell starting with any of these +# characters as a formula trigger. Tab/CR can also be exploited to construct +# functional payloads after a paste, so we treat them defensively too. +FORMULA_TRIGGERS: tuple[str, ...] = ("=", "+", "-", "@", "\t", "\r") + + +def is_potential_formula(value: Any) -> bool: + """Return True if a value would be evaluated as a formula in Excel.""" + if value is None: + return False + text = value if isinstance(value, str) else str(value) + if not text: + return False + # Negative-number literals are not exploitable on their own ("-5"), but + # combined with a function call ("-SUM(...)") they are. Be conservative: + # if the value starts with - or + and contains an alphabetic character, + # treat as suspicious. + head = text[0] + if head not in FORMULA_TRIGGERS: + return False + if head in ("-", "+"): + rest = text[1:].lstrip() + return any(ch.isalpha() or ch == "(" for ch in rest[:8]) + return True + + +def sanitize_literal(value: Any) -> str: + """Return value as a literal-cell string, prepending ``'`` if necessary. + + Non-strings are coerced via ``str``. If the result starts with a + formula trigger, a single quote is prepended so Excel renders the value + as text. Idempotent — already-prefixed strings are returned unchanged. + """ + if value is None: + return "" + text = value if isinstance(value, str) else str(value) + if not text: + return "" + if text.startswith("'"): + # Already quoted as literal — leave alone. + return text + if is_potential_formula(text): + return "'" + text + return text + + +def explain_risk(value: Any) -> str | None: + """If value is suspicious, return a one-line risk explanation; else None.""" + if not is_potential_formula(value): + return None + head = str(value)[:1] + return ( + f"Cell value begins with {head!r} and would be evaluated as a formula by Excel. " + "Use sanitize_literal() or mark the operation as set_cell_formula." + ) diff --git a/backend/services/document_cleanup.py b/backend/services/document_cleanup.py new file mode 100644 index 0000000..f9e030a --- /dev/null +++ b/backend/services/document_cleanup.py @@ -0,0 +1,267 @@ +"""Managed staging, quarantine, and recovery for unregistered document uploads.""" + +from __future__ import annotations + +import hashlib +import json +import os +import tempfile +import time +import uuid +from pathlib import Path +from typing import Any, BinaryIO + +import structlog + +logger = structlog.get_logger() + + +class DocumentCleanupError(RuntimeError): + """Raised when raw upload bytes cannot be deleted or durably registered.""" + + +def stage_managed_upload( + content: bytes, + destination: Path, + *, + data_root: Path, + session_id: str, + document_id: str, +) -> None: + """Write upload bytes to a private staging file and atomically publish them.""" + documents_root, _, _ = _managed_roots(data_root) + documents_root.mkdir(parents=True, exist_ok=True) + resolved_destination = destination.resolve(strict=False) + if not resolved_destination.is_relative_to(documents_root.resolve()): + raise ValueError("Document upload destination is outside the managed document store") + + descriptor, raw_staging_path = tempfile.mkstemp( + prefix=f".{document_id}.", + suffix=".upload", + dir=documents_root, + ) + staging_path = Path(raw_staging_path) + try: + if hasattr(os, "fchmod"): + os.fchmod(descriptor, 0o600) + with os.fdopen(descriptor, "wb") as stream: + descriptor = -1 + _write_upload_stream(stream, content) + os.replace(staging_path, resolved_destination) + os.chmod(resolved_destination, 0o600) + except BaseException as exc: + if descriptor >= 0: + try: + os.close(descriptor) + except OSError: + pass + cleanup_errors: list[Exception] = [] + for candidate in (staging_path, resolved_destination): + try: + cleanup_unregistered_upload( + candidate, + data_root=data_root, + session_id=session_id, + document_id=document_id, + reason="upload_write_failed", + ) + except Exception as cleanup_exc: # pragma: no cover - catastrophic storage failure + cleanup_errors.append(cleanup_exc) + if cleanup_errors: + raise DocumentCleanupError("Upload write failed and raw-byte cleanup could not be durably registered") from exc + raise + + +def cleanup_unregistered_upload( + path: Path, + *, + data_root: Path, + session_id: str, + document_id: str, + reason: str, +) -> bool: + """Delete raw bytes, or quarantine and register them for verified recovery.""" + documents_root, quarantine_root, _ = _managed_roots(data_root) + candidate = path.resolve(strict=False) + if not _is_managed_target(candidate, documents_root, quarantine_root): + raise ValueError("Refusing to clean an upload outside managed storage") + + deletion_error = "" + try: + _unlink_file(candidate) + except OSError as exc: + deletion_error = exc.__class__.__name__ + if not candidate.exists(): + return True + + quarantined = False + quarantine_root.mkdir(parents=True, exist_ok=True) + quarantine_path = quarantine_root / f"{document_id}_{uuid.uuid4().hex}.quarantine" + if candidate.is_relative_to(documents_root.resolve()): + try: + os.replace(candidate, quarantine_path) + os.chmod(quarantine_path, 0o600) + candidate = quarantine_path.resolve(strict=False) + quarantined = True + except OSError as exc: + deletion_error = f"{deletion_error or 'unlink_failed'}:{exc.__class__.__name__}" + + if not candidate.exists(): + return True + + try: + receipt_path = _register_cleanup( + candidate, + data_root=data_root, + session_id=session_id, + document_id=document_id, + reason=reason, + quarantined=quarantined, + error=deletion_error or "delete_verification_failed", + ) + except Exception as exc: + logger.critical( + "document_upload_cleanup_registration_failed", + document_id=document_id, + path=str(candidate), + error=str(exc), + ) + raise DocumentCleanupError("Raw upload bytes remain and their cleanup could not be durably registered") from exc + logger.error( + "document_upload_cleanup_registered", + document_id=document_id, + quarantined=quarantined, + cleanup_receipt=str(receipt_path), + error=deletion_error, + ) + return False + + +def recover_pending_upload_cleanups(data_root: Path) -> int: + """Retry durable cleanup receipts and remove each receipt only after proof.""" + documents_root, quarantine_root, cleanup_root = _managed_roots(data_root) + recovered = 0 + receipt_paths = sorted(cleanup_root.glob("cleanup_*.json")) if cleanup_root.exists() else [] + for receipt_path in receipt_paths: + try: + payload = json.loads(receipt_path.read_text(encoding="utf-8")) + target = Path(str(payload.get("target_path") or "")).resolve(strict=False) + if not _is_managed_target(target, documents_root, quarantine_root): + raise ValueError("Cleanup receipt target is outside managed storage") + try: + _unlink_file(target) + except OSError as exc: + logger.error( + "document_upload_cleanup_retry_failed", + cleanup_id=payload.get("cleanup_id"), + error=exc.__class__.__name__, + ) + continue + if target.exists(): + logger.error( + "document_upload_cleanup_retry_unverified", + cleanup_id=payload.get("cleanup_id"), + ) + continue + _unlink_file(receipt_path) + if receipt_path.exists(): + continue + recovered += 1 + except (OSError, ValueError, TypeError, json.JSONDecodeError) as exc: + logger.error( + "document_upload_cleanup_receipt_failed", + receipt=str(receipt_path), + error=str(exc), + ) + for quarantine_path in sorted(quarantine_root.glob("*.quarantine")): + try: + _unlink_file(quarantine_path) + if not quarantine_path.exists(): + recovered += 1 + except OSError as exc: + logger.error( + "document_upload_quarantine_retry_failed", + path=str(quarantine_path), + error=exc.__class__.__name__, + ) + return recovered + + +def _write_upload_stream(stream: BinaryIO, content: bytes) -> None: + stream.write(content) + stream.flush() + os.fsync(stream.fileno()) + + +def _unlink_file(path: Path) -> None: + path.unlink(missing_ok=True) + + +def _register_cleanup( + target: Path, + *, + data_root: Path, + session_id: str, + document_id: str, + reason: str, + quarantined: bool, + error: str, +) -> Path: + _, _, cleanup_root = _managed_roots(data_root) + cleanup_root.mkdir(parents=True, exist_ok=True) + cleanup_id = f"cleanup_{uuid.uuid4().hex}" + receipt_path = cleanup_root / f"{cleanup_id}.json" + _atomic_json_write( + receipt_path, + { + "schema_version": "2026-08-document-cleanup-v1", + "cleanup_id": cleanup_id, + "document_id": document_id, + "session_id_hash": hashlib.sha256(session_id.encode("utf-8")).hexdigest(), + "target_path": str(target), + "reason": str(reason)[:128], + "quarantined": quarantined, + "last_error": str(error)[:128], + "created_at": time.time(), + }, + ) + return receipt_path + + +def _atomic_json_write(path: Path, payload: dict[str, Any]) -> None: + descriptor, raw_temporary_path = tempfile.mkstemp( + prefix=f".{path.name}.", + suffix=".tmp", + dir=path.parent, + ) + temporary_path = Path(raw_temporary_path) + try: + if hasattr(os, "fchmod"): + os.fchmod(descriptor, 0o600) + with os.fdopen(descriptor, "w", encoding="utf-8") as stream: + descriptor = -1 + json.dump(payload, stream, sort_keys=True) + stream.flush() + os.fsync(stream.fileno()) + os.replace(temporary_path, path) + os.chmod(path, 0o600) + except BaseException: + if descriptor >= 0: + try: + os.close(descriptor) + except OSError: + pass + try: + temporary_path.unlink(missing_ok=True) + except OSError: + pass + raise + + +def _managed_roots(data_root: Path) -> tuple[Path, Path, Path]: + root = Path(data_root).resolve(strict=False) + return root / "documents", root / "document-quarantine", root / "document-cleanup" + + +def _is_managed_target(path: Path, documents_root: Path, quarantine_root: Path) -> bool: + return path.is_relative_to(documents_root.resolve(strict=False)) or path.is_relative_to(quarantine_root.resolve(strict=False)) diff --git a/backend/services/formula_graph.py b/backend/services/formula_graph.py index a4d14d5..25b5c45 100644 --- a/backend/services/formula_graph.py +++ b/backend/services/formula_graph.py @@ -248,7 +248,7 @@ def _formula_impact_report( def _output_rankings(output_cells: list[str], centrality: dict[str, float], dependents: dict[str, list[str]]) -> list[dict[str, Any]]: - rows = [] + rows: list[dict[str, Any]] = [] for cell in output_cells: rows.append( { @@ -258,7 +258,7 @@ def _output_rankings(output_cells: list[str], centrality: dict[str, float], depe "reason": "Output-like formula with dependency graph evidence", } ) - rows.sort(key=lambda item: item["score"], reverse=True) + rows.sort(key=lambda item: float(item["score"]), reverse=True) return rows[:100] diff --git a/backend/services/lifecycle.py b/backend/services/lifecycle.py new file mode 100644 index 0000000..758d148 --- /dev/null +++ b/backend/services/lifecycle.py @@ -0,0 +1,1163 @@ +"""Coordinated, verifiable deletion for session-bound data.""" + +from __future__ import annotations + +import asyncio +import base64 +import hashlib +import hmac +import inspect +import json +import os +import threading +import time +import uuid +from contextlib import asynccontextmanager +from dataclasses import asdict, dataclass, field +from enum import Enum +from pathlib import Path +from typing import Any, AsyncIterator, Awaitable, Callable, Mapping + +import structlog + +from backend.config import DATA_DIR + +logger = structlog.get_logger() + +LifecyclePurger = Callable[[str], Any | Awaitable[Any]] +RECEIPT_SCHEMA_VERSION = "2026-08-deletion-receipt-v1" +RECEIPT_ALGORITHM = "HMAC-SHA256" +RECEIPT_CURRENT_KEY_ID_ENV = "EXCEL_AGENT_LIFECYCLE_RECEIPT_KEY_ID" +RECEIPT_CURRENT_KEY_ENV = "EXCEL_AGENT_LIFECYCLE_RECEIPT_HMAC_KEY" +RECEIPT_PREVIOUS_KEYS_ENV = "EXCEL_AGENT_LIFECYCLE_RECEIPT_PREVIOUS_KEYS" +LEGACY_RECEIPT_QUARANTINE_DIR = "quarantine/legacy-unsigned" +LEGACY_RECEIPT_QUARANTINE_GUIDANCE = "RECOVERY.txt" + + +class LifecycleState(str, Enum): + ACTIVE = "active" + PURGING = "purging" + VERIFIED = "verified" + FAILED = "failed" + + +@dataclass +class DeletionReceipt: + receipt_id: str + session_id_hash: str + state: LifecycleState + started_at: float + completed_at: float = 0.0 + database_deleted: bool = False + document_files_deleted: int = 0 + workbook_jobs_revoked: int = 0 + workbook_job_threads_pending: int = 0 + workflows_deleted: int = 0 + registered_purgers: dict[str, Any] = field(default_factory=dict) + errors: list[str] = field(default_factory=list) + + def to_dict(self) -> dict[str, Any]: + payload = asdict(self) + payload["state"] = self.state.value + return payload + + +class LifecycleCoordinator: + """Own the cross-store lifecycle boundary for one session.""" + + def __init__( + self, + *, + store: Any | None = None, + job_manager: Any | None = None, + receipts_dir: Path | str | None = None, + receipt_signing_keys: Mapping[str, bytes | str] | None = None, + current_receipt_key_id: str | None = None, + ) -> None: + self._store = store + self._job_manager = job_manager + self._receipts_dir = Path(receipts_dir or (DATA_DIR / "lifecycle-receipts")) + self._states: dict[str, LifecycleState] = {} + self._hashed_states: dict[str, LifecycleState] = {} + self._latest_receipts: dict[str, DeletionReceipt] = {} + self._receipts_by_identity: dict[tuple[str, str], DeletionReceipt] = {} + self._purgers: dict[str, LifecyclePurger] = {} + self._active_tasks: dict[str, set[asyncio.Task[Any]]] = {} + self._lock = threading.RLock() + self._receipt_signing_keys = _normalize_receipt_keys(receipt_signing_keys or {}) + self._current_receipt_key_id = str(current_receipt_key_id or "").strip() + if self._receipt_signing_keys and not self._current_receipt_key_id: + self._current_receipt_key_id = next(iter(self._receipt_signing_keys)) + if self._current_receipt_key_id and self._current_receipt_key_id not in self._receipt_signing_keys: + raise ValueError("Current deletion-receipt key ID is not present in the verification key set") + self._load_receipt_states() + + def state_for(self, session_id: str) -> LifecycleState: + normalized_session_id = str(session_id or "").strip() + with self._lock: + direct = self._states.get(normalized_session_id) + if direct is not None: + return direct + hashed = self._hashed_states.get(_session_hash(normalized_session_id)) + if hashed is not None: + return hashed + store = self._store + getter = getattr(store, "get_session_lifecycle_state", None) if store is not None else None + if callable(getter): + try: + durable_state = getter(normalized_session_id) + if durable_state: + return LifecycleState(str(durable_state)) + except Exception: + logger.exception("lifecycle_durable_state_read_failed") + return LifecycleState.FAILED + return LifecycleState.ACTIVE + + def reads_revoked(self, session_id: str) -> bool: + """Return whether a session has crossed the irreversible purge boundary.""" + return self.state_for(str(session_id or "").strip()) is not LifecycleState.ACTIVE + + def writes_revoked(self, session_id: str) -> bool: + """Return whether new durable state is forbidden for a session.""" + return self.state_for(str(session_id or "").strip()) is not LifecycleState.ACTIVE + + def register_purger(self, name: str, purger: LifecyclePurger) -> None: + normalized = str(name or "").strip() + if not normalized or not callable(purger): + raise ValueError("Lifecycle purger requires a name and callable") + with self._lock: + self._purgers[normalized] = purger + + @asynccontextmanager + async def active_task(self, session_id: str) -> AsyncIterator[None]: + """Register one request task so purge can revoke and join active AI work.""" + normalized_session_id = str(session_id or "").strip() + if not normalized_session_id: + raise ValueError("session_id is required") + task = asyncio.current_task() + if task is None: + raise RuntimeError("Active lifecycle work requires an asyncio task") + if self.writes_revoked(normalized_session_id): + raise RuntimeError("Session lifecycle is not active") + session_id_hash = _session_hash(normalized_session_id) + with self._lock: + direct = self._states.get(normalized_session_id) + hashed = self._hashed_states.get(session_id_hash) + if (direct is not None and direct is not LifecycleState.ACTIVE) or ( + hashed is not None and hashed is not LifecycleState.ACTIVE + ): + raise RuntimeError("Session lifecycle is not active") + self._active_tasks.setdefault(normalized_session_id, set()).add(task) + try: + # Close the race between the durable check above and local registry + # insertion. A concurrent purge either sees this task or this check + # observes its irreversible tombstone. + if self.writes_revoked(normalized_session_id): + raise RuntimeError("Session lifecycle is not active") + yield + finally: + with self._lock: + tasks = self._active_tasks.get(normalized_session_id) + if tasks is not None: + tasks.discard(task) + if not tasks: + self._active_tasks.pop(normalized_session_id, None) + + async def _cancel_and_join_active_tasks( + self, + session_id: str, + *, + join_timeout: float = 2.0, + ) -> dict[str, int]: + """Cancel session work without canceling the task performing its own purge.""" + current = asyncio.current_task() + current_loop = asyncio.get_running_loop() + with self._lock: + registered = { + task + for task in self._active_tasks.get(session_id, set()) + if not task.done() + } + self_excluded = int(current is not None and current in registered) + targets = {task for task in registered if task is not current} + local_targets = {task for task in targets if task.get_loop() is current_loop} + foreign_targets = targets - local_targets + for task in local_targets: + task.cancel("session lifecycle purge") + for task in foreign_targets: + task.get_loop().call_soon_threadsafe(task.cancel, "session lifecycle purge") + + done: set[asyncio.Task[Any]] = set() + pending: set[asyncio.Task[Any]] = set(local_targets) + timeout = max(float(join_timeout), 0.0) + if local_targets: + done, pending = await asyncio.wait(local_targets, timeout=timeout) + deadline = time.monotonic() + timeout + while foreign_targets and time.monotonic() < deadline: + finished = {task for task in foreign_targets if task.done()} + done.update(finished) + foreign_targets.difference_update(finished) + if foreign_targets: + await asyncio.sleep(0.01) + pending.update(foreign_targets) + with self._lock: + tracked = self._active_tasks.get(session_id) + if tracked is not None: + tracked.difference_update(done) + if not tracked: + self._active_tasks.pop(session_id, None) + return { + "tasks_revoked": len(targets), + "tasks_joined": len(done), + "tasks_pending": len(pending), + "self_excluded": self_excluded, + } + + async def purge_session(self, session_id: str) -> DeletionReceipt: + normalized_session_id = str(session_id or "").strip() + if not normalized_session_id: + raise ValueError("session_id is required") + store = self._store_instance() + session_id_hash = _session_hash(normalized_session_id) + durable_record = await asyncio.to_thread( + self._store_lifecycle_record, + store, + normalized_session_id, + ) + durable_state = str((durable_record or {}).get("state") or "") + durable_receipt_id = str((durable_record or {}).get("receipt_id") or "") + if durable_receipt_id: + # The same signed receipt file advances from PURGING to VERIFIED. + # Always discard the cached identity and re-authenticate disk state; + # otherwise an idle replica can reject a peer's terminal update. + with self._lock: + self._receipts_by_identity.pop((session_id_hash, durable_receipt_id), None) + self._load_receipt_states() + terminal_receipt: DeletionReceipt | None = None + with self._lock: + trusted_receipt = ( + self._receipts_by_identity.get((session_id_hash, durable_receipt_id)) + if durable_receipt_id + else self._latest_receipts.get(session_id_hash) + ) + if ( + durable_state == LifecycleState.VERIFIED.value + and trusted_receipt is not None + and trusted_receipt.state is LifecycleState.VERIFIED + and trusted_receipt.receipt_id == durable_receipt_id + ): + self._states[normalized_session_id] = LifecycleState.VERIFIED + self._hashed_states[session_id_hash] = LifecycleState.VERIFIED + terminal_receipt = trusted_receipt + elif durable_state == LifecycleState.VERIFIED.value: + raise RuntimeError("Verified deletion tombstone has no matching trusted receipt") + elif trusted_receipt is not None and trusted_receipt.state is LifecycleState.VERIFIED: + if not durable_record: + raise RuntimeError("Trusted deletion receipt has no durable lifecycle tombstone") + if durable_state != LifecycleState.PURGING.value: + raise RuntimeError("Trusted deletion receipt conflicts with durable lifecycle state") + if terminal_receipt is None: + resume_existing = ( + durable_state in {LifecycleState.PURGING.value, LifecycleState.FAILED.value} + and trusted_receipt is not None + and trusted_receipt.receipt_id == durable_receipt_id + ) + if durable_state == LifecycleState.PURGING.value and not resume_existing: + raise RuntimeError("Durable purge marker has no matching trusted receipt") + if ( + self._states.get(normalized_session_id) is LifecycleState.PURGING + and not resume_existing + ): + raise RuntimeError("Session purge is already in progress") + self._states[normalized_session_id] = LifecycleState.PURGING + self._hashed_states[session_id_hash] = LifecycleState.PURGING + else: + resume_existing = False + purgers = dict(self._purgers) + + if terminal_receipt is not None: + return await self._reconcile_verified_receipt( + store, + normalized_session_id, + terminal_receipt, + ) + + if ( + durable_state == LifecycleState.PURGING.value + and trusted_receipt is not None + and trusted_receipt.state is LifecycleState.VERIFIED + ): + return await self._reconcile_verified_receipt( + store, + normalized_session_id, + trusted_receipt, + ) + + receipt = ( + _reset_receipt_for_retry(trusted_receipt) + if resume_existing and trusted_receipt is not None + else DeletionReceipt( + receipt_id=f"del_{uuid.uuid4().hex}", + session_id_hash=session_id_hash, + state=LifecycleState.PURGING, + started_at=time.time(), + ) + ) + try: + receipt = await asyncio.to_thread(self._persist_receipt, receipt) + except Exception as exc: + # replace(2) may have installed the signed VERIFIED receipt before + # its parent-directory fsync failed. Never write a conflicting + # FAILED tombstone in that ambiguous state: keep durable PURGING, + # fail the request, and let a retry re-fsync and reconcile it. + target = self._receipts_dir / f"{receipt.receipt_id}.json" + try: + on_disk = self._verify_receipt_envelope( + json.loads(target.read_text(encoding="utf-8")) + ) + except (OSError, ValueError, TypeError, json.JSONDecodeError, RuntimeError): + on_disk = None + if ( + on_disk is not None + and on_disk.state is LifecycleState.VERIFIED + and on_disk.receipt_id == receipt.receipt_id + and on_disk.session_id_hash == receipt.session_id_hash + ): + with self._lock: + self._states[normalized_session_id] = LifecycleState.PURGING + self._hashed_states[receipt.session_id_hash] = LifecycleState.PURGING + raise RuntimeError("Could not durably persist the final deletion receipt") from exc + with self._lock: + self._states[normalized_session_id] = LifecycleState.FAILED + self._hashed_states[receipt.session_id_hash] = LifecycleState.FAILED + raise RuntimeError("Could not seal the session purge marker") from exc + if receipt.state is LifecycleState.VERIFIED: + return await self._reconcile_verified_receipt( + store, + normalized_session_id, + receipt, + ) + try: + await asyncio.to_thread( + self._persist_store_state, + store, + normalized_session_id, + receipt, + ) + except Exception as exc: + failed_receipt = _receipt_with_failure( + receipt, + f"lifecycle_state_persist_failed:{exc.__class__.__name__}", + ) + with self._lock: + self._states[normalized_session_id] = LifecycleState.FAILED + self._hashed_states[receipt.session_id_hash] = LifecycleState.FAILED + try: + persisted = await asyncio.to_thread(self._persist_receipt, failed_receipt) + except Exception: + logger.exception("lifecycle_failed_marker_persist_failed") + else: + if persisted.state is LifecycleState.VERIFIED: + return await self._reconcile_verified_receipt( + store, + normalized_session_id, + persisted, + ) + raise RuntimeError("Could not persist the durable session purge state") from exc + try: + active_tasks = await self._cancel_and_join_active_tasks(normalized_session_id) + receipt.registered_purgers["active_ai_tasks"] = active_tasks + if active_tasks["tasks_pending"]: + receipt.errors.append("active_ai_tasks_pending") + except Exception as exc: + receipt.errors.append(f"active_ai_task_purge_failed:{exc.__class__.__name__}") + try: + session_present = bool(await asyncio.to_thread(store.session_exists, normalized_session_id)) + except Exception as exc: + session_present = False + receipt.errors.append(f"session_inventory_failed:{exc.__class__.__name__}") + if not session_present and not receipt.errors: + verified = await asyncio.to_thread( + self._store_deletion_verified, + store, + normalized_session_id, + ) + if verified: + receipt.database_deleted = True + else: + receipt.errors.append("session_inventory_missing") + + try: + file_paths = await asyncio.to_thread(store.list_document_file_paths, normalized_session_id) + except Exception as exc: + file_paths = [] + receipt.errors.append(f"document_inventory_failed:{exc.__class__.__name__}") + + try: + jobs = await asyncio.to_thread( + self._job_manager_instance().purge_session, + normalized_session_id, + ) + receipt.workbook_jobs_revoked = int(jobs.get("jobs_revoked") or 0) + receipt.workbook_job_threads_pending = int(jobs.get("threads_pending") or 0) + if receipt.workbook_job_threads_pending: + receipt.errors.append("workbook_job_threads_pending") + except Exception as exc: + receipt.errors.append(f"workbook_job_purge_failed:{exc.__class__.__name__}") + + # The database is the only durable inventory for managed files and + # session-bound stores. Keep it intact until every external cleanup + # succeeds so a transient failure remains safely retryable. + if not receipt.errors: + for raw_path in file_paths: + try: + deleted = await asyncio.to_thread(_delete_managed_document, raw_path) + receipt.document_files_deleted += int(deleted) + except Exception as exc: + receipt.errors.append(f"document_file_purge_failed:{exc.__class__.__name__}") + + if not receipt.errors: + try: + from backend.agents.workflows import get_workflow_engine + + receipt.workflows_deleted = await asyncio.to_thread( + get_workflow_engine().purge_session, + normalized_session_id, + ) + except Exception as exc: + receipt.errors.append(f"workflow_purge_failed:{exc.__class__.__name__}") + + if not receipt.errors: + for name, purger in purgers.items(): + try: + result = purger(normalized_session_id) + if inspect.isawaitable(result): + result = await result + receipt.registered_purgers[name] = result + except Exception as exc: + receipt.errors.append(f"registered_purger_failed:{name}:{exc.__class__.__name__}") + + if not receipt.errors: + try: + from backend.agents.workflows import get_workflow_engine + + remaining_workflows = await asyncio.to_thread( + get_workflow_engine().count_session_workflows, + normalized_session_id, + ) + if remaining_workflows: + receipt.errors.append( + f"workflow_inventory_remaining:{remaining_workflows}" + ) + except Exception as exc: + receipt.errors.append( + f"workflow_inventory_failed:{exc.__class__.__name__}" + ) + + if not receipt.errors: + try: + receipt.database_deleted = bool( + await asyncio.to_thread(store.delete_session, normalized_session_id) + ) + if not receipt.database_deleted: + receipt.errors.append("database_purge_noop") + except Exception as exc: + receipt.errors.append(f"database_purge_failed:{exc.__class__.__name__}") + + if receipt.database_deleted: + try: + if await asyncio.to_thread(store.session_exists, normalized_session_id): + receipt.errors.append("session_still_exists") + except Exception as exc: + receipt.errors.append(f"deletion_verification_failed:{exc.__class__.__name__}") + + receipt.completed_at = time.time() + receipt.state = LifecycleState.FAILED if receipt.errors else LifecycleState.VERIFIED + with self._lock: + self._states[normalized_session_id] = receipt.state + self._hashed_states[receipt.session_id_hash] = receipt.state + try: + receipt = await asyncio.to_thread(self._persist_receipt, receipt) + except Exception as exc: + target = self._receipts_dir / f"{receipt.receipt_id}.json" + try: + on_disk = self._verify_receipt_envelope( + json.loads(target.read_text(encoding="utf-8")) + ) + except (OSError, ValueError, TypeError, json.JSONDecodeError, RuntimeError): + on_disk = None + if ( + on_disk is not None + and on_disk.state is LifecycleState.VERIFIED + and on_disk.receipt_id == receipt.receipt_id + and on_disk.session_id_hash == receipt.session_id_hash + ): + with self._lock: + self._states[normalized_session_id] = LifecycleState.PURGING + self._hashed_states[receipt.session_id_hash] = LifecycleState.PURGING + raise RuntimeError("Could not durably persist the final deletion receipt") from exc + with self._lock: + self._states[normalized_session_id] = LifecycleState.FAILED + self._hashed_states[receipt.session_id_hash] = LifecycleState.FAILED + failed_receipt = _receipt_with_failure( + receipt, + f"receipt_persist_failed:{exc.__class__.__name__}", + ) + try: + await asyncio.to_thread( + self._persist_store_state, + store, + normalized_session_id, + failed_receipt, + ) + except Exception: + logger.exception("lifecycle_failed_state_persist_failed") + raise RuntimeError("Could not persist the final deletion receipt") from exc + if receipt.state is LifecycleState.VERIFIED and receipt.errors: + raise RuntimeError("Verified deletion receipt cannot contain cleanup errors") + try: + await asyncio.to_thread( + self._persist_store_state, + store, + normalized_session_id, + receipt, + ) + except Exception as exc: + if receipt.state is LifecycleState.VERIFIED: + return await self._reconcile_verified_receipt( + store, + normalized_session_id, + receipt, + cause=exc, + ) + failed_receipt = _receipt_with_failure( + receipt, + f"lifecycle_state_persist_failed:{exc.__class__.__name__}", + ) + with self._lock: + self._states[normalized_session_id] = LifecycleState.FAILED + self._hashed_states[receipt.session_id_hash] = LifecycleState.FAILED + try: + persisted = await asyncio.to_thread(self._persist_receipt, failed_receipt) + except Exception: + logger.exception("lifecycle_failed_marker_persist_failed") + else: + if persisted.state is LifecycleState.VERIFIED: + return await self._reconcile_verified_receipt( + store, + normalized_session_id, + persisted, + cause=exc, + ) + raise RuntimeError("Could not persist the final durable lifecycle state") from exc + log = logger.info if receipt.state is LifecycleState.VERIFIED else logger.error + log("session_lifecycle_purge_completed", **receipt.to_dict()) + return receipt + + async def purge_before(self, cutoff_timestamp: float) -> list[DeletionReceipt]: + store = self._store_instance() + session_ids = await asyncio.to_thread(store.list_session_ids_before, cutoff_timestamp) + receipts = [] + for session_id in session_ids: + receipts.append(await self.purge_session(session_id)) + return receipts + + def _store_instance(self): + if self._store is None: + from backend.storage import get_session_store + + self._store = get_session_store() + return self._store + + def _job_manager_instance(self): + if self._job_manager is None: + from backend.services.workbook_jobs import get_workbook_job_manager + + self._job_manager = get_workbook_job_manager() + return self._job_manager + + async def _reconcile_verified_receipt( + self, + store: Any, + session_id: str, + receipt: DeletionReceipt, + *, + cause: Exception | None = None, + ) -> DeletionReceipt: + if receipt.state is not LifecycleState.VERIFIED: + raise RuntimeError("Only VERIFIED receipts can reconcile terminal lifecycle state") + await self._revoke_local_terminal_state(session_id) + try: + receipt = await asyncio.to_thread(self._persist_receipt, receipt) + except Exception as persist_exc: + raise RuntimeError("Could not durably seal the verified deletion receipt") from persist_exc + deletion_verified = await asyncio.to_thread( + self._store_deletion_verified, + store, + session_id, + ) + if not deletion_verified: + raise RuntimeError("Verified deletion receipt conflicts with surviving durable rows") from cause + durable = await asyncio.to_thread(self._store_lifecycle_record, store, session_id) + durable_state = str((durable or {}).get("state") or "") + durable_receipt_id = str((durable or {}).get("receipt_id") or "") + if durable_receipt_id != receipt.receipt_id: + raise RuntimeError("Verified deletion receipt conflicts with durable receipt identity") from cause + if durable_state == LifecycleState.PURGING.value: + try: + await asyncio.to_thread( + self._persist_store_state, + store, + session_id, + receipt, + ) + except Exception as persist_exc: + durable = await asyncio.to_thread(self._store_lifecycle_record, store, session_id) + durable_state = str((durable or {}).get("state") or "") + durable_receipt_id = str((durable or {}).get("receipt_id") or "") + if not ( + durable_state == LifecycleState.VERIFIED.value + and durable_receipt_id == receipt.receipt_id + ): + raise RuntimeError("Could not reconcile the verified deletion tombstone") from persist_exc + elif durable_state != LifecycleState.VERIFIED.value: + raise RuntimeError("Verified deletion receipt conflicts with durable lifecycle state") from cause + with self._lock: + self._states[session_id] = LifecycleState.VERIFIED + self._hashed_states[receipt.session_id_hash] = LifecycleState.VERIFIED + return _clone_receipt(receipt) + + async def _revoke_local_terminal_state(self, session_id: str) -> None: + """Make a peer-verified tombstone terminal for this process too.""" + active_tasks = await self._cancel_and_join_active_tasks(session_id) + failures: list[str] = [] + if active_tasks["tasks_pending"]: + failures.append("active_ai_tasks_pending") + + try: + jobs = await asyncio.to_thread( + self._job_manager_instance().purge_session, + session_id, + ) + if int(jobs.get("threads_pending") or 0): + failures.append("workbook_job_threads_pending") + except Exception as exc: + failures.append(f"workbook_job_purge_failed:{exc.__class__.__name__}") + + try: + from backend.agents.workflows import get_workflow_engine + + await asyncio.to_thread(get_workflow_engine().purge_session, session_id) + except Exception as exc: + failures.append(f"workflow_purge_failed:{exc.__class__.__name__}") + + with self._lock: + purgers = dict(self._purgers) + for name, purger in purgers.items(): + try: + result = purger(session_id) + if inspect.isawaitable(result): + await result + except Exception as exc: + failures.append(f"registered_purger_failed:{name}:{exc.__class__.__name__}") + if failures: + raise RuntimeError( + "Verified deletion tombstone could not revoke local state: " + ",".join(failures) + ) + + def _persist_receipt(self, receipt: DeletionReceipt) -> DeletionReceipt: + receipt = _clone_receipt(receipt) + self._receipts_dir.mkdir(parents=True, exist_ok=True) + lock_file = _acquire_receipt_file_lock(self._receipts_dir, receipt.receipt_id) + try: + return self._persist_receipt_locked(receipt) + finally: + _release_receipt_file_lock(lock_file) + + def _persist_receipt_locked(self, receipt: DeletionReceipt) -> DeletionReceipt: + target = self._receipts_dir / f"{receipt.receipt_id}.json" + if target.exists(): + existing_envelope = json.loads(target.read_text(encoding="utf-8")) + existing = self._verify_receipt_envelope(existing_envelope) + if ( + existing.receipt_id != receipt.receipt_id + or existing.session_id_hash != receipt.session_id_hash + ): + raise RuntimeError("Deletion receipt identity is immutable") + if existing.state is LifecycleState.VERIFIED: + canonical = _clone_receipt(existing) + _fsync_directory(self._receipts_dir) + self._remember_receipt(canonical) + return canonical + key_id, signing_key, _verification_keys = self._receipt_signing_material() + signed_payload = { + "schema_version": RECEIPT_SCHEMA_VERSION, + "algorithm": RECEIPT_ALGORITHM, + "key_id": key_id, + "receipt": receipt.to_dict(), + } + envelope = { + **signed_payload, + "signature": hmac.new( + signing_key, + _canonical_json_bytes(signed_payload), + hashlib.sha256, + ).hexdigest(), + } + temporary = target.with_name(f".{target.name}.{uuid.uuid4().hex}.tmp") + with temporary.open("w", encoding="utf-8") as receipt_file: + receipt_file.write(json.dumps(envelope, default=str, indent=2, sort_keys=True)) + receipt_file.flush() + os.fsync(receipt_file.fileno()) + temporary.chmod(0o600) + temporary.replace(target) + _fsync_directory(self._receipts_dir) + persisted = _clone_receipt(receipt) + self._remember_receipt(persisted) + return persisted + + def _remember_receipt(self, receipt: DeletionReceipt) -> None: + remembered = _clone_receipt(receipt) + with self._lock: + self._latest_receipts[remembered.session_id_hash] = remembered + self._receipts_by_identity[(remembered.session_id_hash, remembered.receipt_id)] = remembered + + def _receipt_signing_material(self) -> tuple[str, bytes, dict[str, bytes]]: + with self._lock: + if self._receipt_signing_keys: + key_id = self._current_receipt_key_id + if not key_id: + raise RuntimeError("Deletion receipt signing key ID is unavailable") + return key_id, self._receipt_signing_keys[key_id], dict(self._receipt_signing_keys) + + configured_keys: dict[str, bytes] = {} + current_key_id = str(os.getenv(RECEIPT_CURRENT_KEY_ID_ENV) or "").strip() + current_encoded = str(os.getenv(RECEIPT_CURRENT_KEY_ENV) or "").strip() + if current_encoded: + current_key_id = current_key_id or "environment-v1" + configured_keys[current_key_id] = _decode_receipt_secret(current_encoded) + + previous_encoded = str(os.getenv(RECEIPT_PREVIOUS_KEYS_ENV) or "").strip() + if previous_encoded: + try: + previous_payload = json.loads(previous_encoded) + except json.JSONDecodeError as exc: + raise RuntimeError("Previous deletion-receipt keys must be a JSON object") from exc + if not isinstance(previous_payload, dict): + raise RuntimeError("Previous deletion-receipt keys must be a JSON object") + previous_keys: dict[str, bytes] = {} + for raw_key_id, raw_secret in previous_payload.items(): + if not isinstance(raw_secret, str): + raise RuntimeError("Previous deletion-receipt keys must be URL-safe base64 strings") + previous_keys[str(raw_key_id)] = _decode_receipt_secret(raw_secret) + normalized_previous_keys = _normalize_receipt_keys(previous_keys) + if current_key_id and current_key_id in normalized_previous_keys: + raise RuntimeError("Previous deletion-receipt keys must not redefine the current key ID") + configured_keys.update(normalized_previous_keys) + + if not current_encoded: + try: + from backend.api.security.local_pairing import KeyringSecretLoader + + current_key_id = current_key_id or "os-keychain-v1" + configured_keys[current_key_id] = KeyringSecretLoader( + service="Excel Agent Lifecycle Receipts", + account="deletion-receipt-hmac-v1", + )() + except Exception as exc: + raise RuntimeError( + "Deletion receipt signing requires an OS-keychain key or an injected environment key" + ) from exc + + if not current_key_id or current_key_id not in configured_keys: + raise RuntimeError("Current deletion-receipt signing key is unavailable") + with self._lock: + self._receipt_signing_keys = configured_keys + self._current_receipt_key_id = current_key_id + return current_key_id, configured_keys[current_key_id], dict(configured_keys) + + @staticmethod + def _persist_store_state( + store: Any, + session_id: str, + receipt: DeletionReceipt, + ) -> None: + setter = getattr(store, "set_session_lifecycle_state", None) + if not callable(setter): + return + setter( + session_id, + receipt.state.value, + receipt_id=receipt.receipt_id, + started_at=receipt.started_at, + completed_at=receipt.completed_at, + errors=list(receipt.errors), + ) + + @staticmethod + def _store_lifecycle_record(store: Any, session_id: str) -> dict[str, Any] | None: + getter = getattr(store, "get_session_lifecycle_record", None) + if callable(getter): + record = getter(session_id) + return dict(record) if record else None + state_getter = getattr(store, "get_session_lifecycle_state", None) + if callable(state_getter): + state = state_getter(session_id) + return {"state": state} if state else None + return None + + @staticmethod + def _store_deletion_verified(store: Any, session_id: str) -> bool: + verifier = getattr(store, "session_deletion_verified", None) + if callable(verifier): + return bool(verifier(session_id)) + return False + + def _load_receipt_states(self) -> None: + latest: dict[str, tuple[float, DeletionReceipt]] = {} + receipts_by_identity: dict[tuple[str, str], DeletionReceipt] = {} + untrusted_hashes: set[str] = set() + verification_failures: dict[str, int] = {} + unsigned_paths: list[Path] = [] + quarantine_root = self._receipts_dir / LEGACY_RECEIPT_QUARANTINE_DIR + try: + receipt_paths = list(self._receipts_dir.glob("del_*.json")) + except OSError as exc: + logger.error("lifecycle_receipt_inventory_failed", error=exc.__class__.__name__) + return + for path in receipt_paths: + envelope: Any = None + try: + envelope = json.loads(path.read_text(encoding="utf-8")) + if _is_unsigned_legacy_receipt(envelope): + session_id_hash = _untrusted_receipt_hash(envelope) + if session_id_hash: + untrusted_hashes.add(session_id_hash) + unsigned_paths.append(path) + continue + receipt = self._verify_receipt_envelope(envelope) + session_id_hash = receipt.session_id_hash + timestamp = float(receipt.completed_at or receipt.started_at or 0.0) + except (OSError, ValueError, TypeError, json.JSONDecodeError, RuntimeError) as exc: + session_id_hash = _untrusted_receipt_hash(envelope) + if session_id_hash: + untrusted_hashes.add(session_id_hash) + error_name = exc.__class__.__name__ + verification_failures[error_name] = verification_failures.get(error_name, 0) + 1 + continue + receipts_by_identity[(session_id_hash, receipt.receipt_id)] = receipt + previous = latest.get(session_id_hash) + if previous is None or timestamp >= previous[0]: + latest[session_id_hash] = (timestamp, receipt) + + quarantined_hashes, quarantine_failures = self._quarantine_unsigned_receipts(unsigned_paths) + untrusted_hashes.update(quarantined_hashes) + try: + quarantined_paths = list(quarantine_root.glob("del_*.json")) + except OSError: + quarantined_paths = [] + quarantine_failures += 1 + for path in quarantined_paths: + try: + quarantined = json.loads(path.read_text(encoding="utf-8")) + except (OSError, ValueError, TypeError, json.JSONDecodeError): + continue + session_id_hash = _untrusted_receipt_hash(quarantined) + if session_id_hash: + untrusted_hashes.add(session_id_hash) + + if unsigned_paths: + logger.warning( + "lifecycle_unsigned_receipts_quarantined", + discovered=len(unsigned_paths), + quarantined=len(unsigned_paths) - quarantine_failures, + quarantine_failures=quarantine_failures, + recovery_guidance=str(quarantine_root / LEGACY_RECEIPT_QUARANTINE_GUIDANCE), + ) + if verification_failures: + logger.error( + "lifecycle_receipt_verification_failures", + total=sum(verification_failures.values()), + errors=verification_failures, + ) + with self._lock: + self._receipts_by_identity.update(receipts_by_identity) + self._latest_receipts.update( + {session_hash: receipt for session_hash, (_, receipt) in latest.items()} + ) + self._hashed_states.update( + {session_hash: receipt.state for session_hash, (_, receipt) in latest.items()} + ) + self._hashed_states.update( + {session_hash: LifecycleState.FAILED for session_hash in untrusted_hashes} + ) + + def _quarantine_unsigned_receipts(self, paths: list[Path]) -> tuple[set[str], int]: + if not paths: + return set(), 0 + quarantine_root = self._receipts_dir / LEGACY_RECEIPT_QUARANTINE_DIR + quarantined_hashes: set[str] = set() + failures = 0 + try: + quarantine_root.mkdir(parents=True, exist_ok=True) + quarantine_root.chmod(0o700) + guidance = quarantine_root / LEGACY_RECEIPT_QUARANTINE_GUIDANCE + if not guidance.exists(): + guidance.write_text( + "Unsigned legacy deletion receipts are preserved here for forensic review but are never trusted.\n" + "Their session hashes remain fail-closed. Restore a signed receipt from a trusted backup, or have an " + "operator re-run the lifecycle purge after verifying the durable database tombstone and inventory.\n", + encoding="utf-8", + ) + guidance.chmod(0o600) + except OSError: + return set(), len(paths) + for source in paths: + try: + payload = json.loads(source.read_text(encoding="utf-8")) + session_id_hash = _untrusted_receipt_hash(payload) + target = quarantine_root / source.name + if target.exists(): + target = quarantine_root / f"{source.stem}-{uuid.uuid4().hex[:8]}{source.suffix}" + source.replace(target) + target.chmod(0o600) + if session_id_hash: + quarantined_hashes.add(session_id_hash) + except (OSError, ValueError, TypeError, json.JSONDecodeError): + failures += 1 + return quarantined_hashes, failures + + def _verify_receipt_envelope(self, envelope: Any) -> DeletionReceipt: + if not isinstance(envelope, dict): + raise ValueError("Deletion receipt envelope must be an object") + if str(envelope.get("schema_version") or "") != RECEIPT_SCHEMA_VERSION: + raise ValueError("Deletion receipt schema is unsupported") + if str(envelope.get("algorithm") or "") != RECEIPT_ALGORITHM: + raise ValueError("Deletion receipt algorithm is unsupported") + key_id = str(envelope.get("key_id") or "").strip() + signature = str(envelope.get("signature") or "").strip().lower() + if len(signature) != 64 or any(char not in "0123456789abcdef" for char in signature): + raise ValueError("Deletion receipt signature is malformed") + _current_key_id, _current_key, verification_keys = self._receipt_signing_material() + verification_key = verification_keys.get(key_id) + if verification_key is None: + raise RuntimeError("Deletion receipt key ID is unknown") + signed_payload = { + "schema_version": envelope.get("schema_version"), + "algorithm": envelope.get("algorithm"), + "key_id": key_id, + "receipt": envelope.get("receipt"), + } + expected = hmac.new( + verification_key, + _canonical_json_bytes(signed_payload), + hashlib.sha256, + ).hexdigest() + if not hmac.compare_digest(signature, expected): + raise ValueError("Deletion receipt signature verification failed") + return _receipt_from_payload(envelope.get("receipt")) + + +def _canonical_json_bytes(payload: Any) -> bytes: + return json.dumps( + payload, + default=str, + ensure_ascii=False, + separators=(",", ":"), + sort_keys=True, + ).encode("utf-8") + + +def _normalize_receipt_keys(keys: Mapping[str, bytes | str]) -> dict[str, bytes]: + normalized: dict[str, bytes] = {} + for raw_key_id, raw_secret in keys.items(): + key_id = str(raw_key_id or "").strip() + if not key_id: + raise ValueError("Deletion-receipt key IDs must be non-empty") + secret = raw_secret if isinstance(raw_secret, bytes) else str(raw_secret).encode("utf-8") + if len(secret) < 32: + raise ValueError("Deletion-receipt signing keys must contain at least 32 bytes") + normalized[key_id] = bytes(secret) + return normalized + + +def _decode_receipt_secret(encoded: str) -> bytes: + value = str(encoded or "").strip() + if value.startswith("base64:"): + value = value.split(":", 1)[1] + try: + padding = "=" * ((4 - len(value) % 4) % 4) + secret = base64.urlsafe_b64decode(value + padding) + except (ValueError, TypeError) as exc: + raise ValueError("Deletion-receipt signing key must be URL-safe base64") from exc + if len(secret) < 32: + raise ValueError("Deletion-receipt signing key must decode to at least 32 bytes") + return secret + + +def _receipt_from_payload(payload: Any) -> DeletionReceipt: + if not isinstance(payload, dict): + raise ValueError("Deletion receipt payload must be an object") + receipt_id = str(payload.get("receipt_id") or "") + session_id_hash = str(payload.get("session_id_hash") or "") + if not receipt_id.startswith("del_"): + raise ValueError("Deletion receipt ID is malformed") + if not _valid_session_hash(session_id_hash): + raise ValueError("Deletion receipt session hash is malformed") + state = LifecycleState(str(payload.get("state") or "")) + if state is LifecycleState.ACTIVE: + raise ValueError("Deletion receipt cannot assert an active lifecycle") + registered_purgers = payload.get("registered_purgers") + errors = payload.get("errors") + if not isinstance(registered_purgers, dict) or not isinstance(errors, list): + raise ValueError("Deletion receipt collections are malformed") + return DeletionReceipt( + receipt_id=receipt_id, + session_id_hash=session_id_hash, + state=state, + started_at=float(payload.get("started_at") or 0.0), + completed_at=float(payload.get("completed_at") or 0.0), + database_deleted=bool(payload.get("database_deleted")), + document_files_deleted=int(payload.get("document_files_deleted") or 0), + workbook_jobs_revoked=int(payload.get("workbook_jobs_revoked") or 0), + workbook_job_threads_pending=int(payload.get("workbook_job_threads_pending") or 0), + workflows_deleted=int(payload.get("workflows_deleted") or 0), + registered_purgers=dict(registered_purgers), + errors=[str(error) for error in errors], + ) + + +def _reset_receipt_for_retry(receipt: DeletionReceipt) -> DeletionReceipt: + return DeletionReceipt( + receipt_id=receipt.receipt_id, + session_id_hash=receipt.session_id_hash, + state=LifecycleState.PURGING, + started_at=receipt.started_at or time.time(), + ) + + +def _clone_receipt(receipt: DeletionReceipt) -> DeletionReceipt: + return _receipt_from_payload(receipt.to_dict()) + + +def _receipt_with_failure(receipt: DeletionReceipt, error: str) -> DeletionReceipt: + failed = _clone_receipt(receipt) + failed.errors.append(str(error)) + failed.completed_at = time.time() + failed.state = LifecycleState.FAILED + return failed + + +def _fsync_directory(path: Path) -> None: + """Durably commit an atomic rename to its parent directory where supported.""" + if os.name == "nt": + # Windows does not expose a portable directory handle compatible with + # os.fsync; ReplaceFile/MoveFileEx durability is delegated to the host. + return + flags = os.O_RDONLY | getattr(os, "O_DIRECTORY", 0) + directory_fd = os.open(path, flags) + try: + os.fsync(directory_fd) + finally: + os.close(directory_fd) + + +def _acquire_receipt_file_lock(receipts_dir: Path, receipt_id: str): + lock_dir = receipts_dir / ".locks" + lock_dir.mkdir(parents=True, exist_ok=True) + lock_dir.chmod(0o700) + lock_name = hashlib.sha256(str(receipt_id or "").encode("utf-8")).hexdigest() + lock_file = (lock_dir / f"{lock_name}.lock").open("a+b") + lock_file_path = Path(lock_file.name) + lock_file_path.chmod(0o600) + if os.name == "nt": + import msvcrt + + lock_file.seek(0, os.SEEK_END) + if lock_file.tell() == 0: + lock_file.write(b"\0") + lock_file.flush() + lock_file.seek(0) + locking = vars(msvcrt)["locking"] + locking(lock_file.fileno(), vars(msvcrt)["LK_LOCK"], 1) + else: + import fcntl + + fcntl.flock(lock_file.fileno(), fcntl.LOCK_EX) + return lock_file + + +def _release_receipt_file_lock(lock_file) -> None: + try: + if os.name == "nt": + import msvcrt + + lock_file.seek(0) + locking = vars(msvcrt)["locking"] + locking(lock_file.fileno(), vars(msvcrt)["LK_UNLCK"], 1) + else: + import fcntl + + fcntl.flock(lock_file.fileno(), fcntl.LOCK_UN) + finally: + lock_file.close() + + +def _valid_session_hash(value: str) -> bool: + return len(value) == 64 and not any(char not in "0123456789abcdef" for char in value) + + +def _untrusted_receipt_hash(envelope: Any) -> str: + if not isinstance(envelope, dict): + return "" + payload = envelope.get("receipt") + if not isinstance(payload, dict): + payload = envelope + candidate = str(payload.get("session_id_hash") or "") + return candidate if _valid_session_hash(candidate) else "" + + +def _is_unsigned_legacy_receipt(envelope: Any) -> bool: + if not isinstance(envelope, dict): + return False + if any(key in envelope for key in ("schema_version", "algorithm", "key_id", "signature")): + return False + return str(envelope.get("receipt_id") or "").startswith("del_") and bool( + _untrusted_receipt_hash(envelope) + ) + + +def _session_hash(session_id: str) -> str: + return hashlib.sha256(str(session_id or "").encode("utf-8")).hexdigest() + + +def _delete_managed_document(raw_path: str) -> bool: + managed_root = (DATA_DIR / "documents").resolve() + candidate = Path(str(raw_path or "")).resolve(strict=False) + if not candidate.is_relative_to(managed_root): + raise ValueError("Recorded document path is outside the managed store") + if not candidate.exists(): + return False + if not candidate.is_file(): + raise ValueError("Recorded document path is not a file") + candidate.unlink() + return True + + +_coordinator: LifecycleCoordinator | None = None + + +def _register_builtin_purgers(coordinator: LifecycleCoordinator) -> None: + from backend.api.teams import get_team_manager + from backend.excel.diff_engine import purge_diff_engine + + coordinator.register_purger("diff_engine", purge_diff_engine) + coordinator.register_purger("team_session", get_team_manager().purge_session) + + +def get_lifecycle_coordinator() -> LifecycleCoordinator: + global _coordinator + if _coordinator is None: + _coordinator = LifecycleCoordinator() + _register_builtin_purgers(_coordinator) + return _coordinator diff --git a/backend/services/operation_planner.py b/backend/services/operation_planner.py index 2e54d4b..48ad0ab 100644 --- a/backend/services/operation_planner.py +++ b/backend/services/operation_planner.py @@ -8,13 +8,14 @@ from __future__ import annotations import asyncio -from typing import Any, Callable, Optional +from typing import Any, Optional import structlog from backend.agents.react_engine import ReActEngine from backend.agents.rules import get_rules_engine from backend.agents.template_router import try_template_route +from backend.agents.tools import execute_tool_call from backend.excel.context_extractor import extract_context from backend.excel.formatting_engine import post_process_operations from backend.llm.base import LLMMessage @@ -27,6 +28,14 @@ logger = structlog.get_logger() +class OperationPlanningError(RuntimeError): + """Raised when an AI-authored plan cannot be proven safe for review.""" + + +class OperationNormalizationError(OperationPlanningError): + """Raised when any proposed operation fails canonical schema conversion.""" + + async def build_operation_plan( *, prompt: str, @@ -94,10 +103,22 @@ async def build_operation_plan( trace = await _run_react_plan( prompt=prompt, workbook_context=processed_context, + tool_context=workbook_context or {}, provider_override=provider_override, model_override=model_override, ) + if not trace.success: + logger.error( + "operation_plan_rejected", + failure_code=trace.failure_code, + iterations=trace.iterations, + partial_operations=len(trace.total_operations), + ) + raise OperationPlanningError( + f"Cannot safely prepare plan: {trace.failure_code or 'planning_failed'}" + ) + legacy_operations = trace.total_operations operations = _normalize_operations(post_process_operations(legacy_operations)) summary = trace.final_summary or f"Prepared {len(operations)} workbook operation(s) for review." @@ -115,6 +136,9 @@ async def build_operation_plan( "planner": "react_tool_loop", "iterations": trace.iterations, "duration_ms": trace.total_duration_ms, + "completed": trace.completed, + "tool_calls": sum(len(step.tool_calls) for step in trace.steps), + "tool_results": sum(len(step.tool_results) for step in trace.steps), "trace_steps": [ { "step": step.step_num, @@ -122,6 +146,8 @@ async def build_operation_plan( "summary": step.content[:300], "operations": len(step.operations), "duration_ms": step.duration_ms, + "tool_calls": len(step.tool_calls), + "tool_results": step.tool_results, } for step in trace.steps ], @@ -135,6 +161,7 @@ async def _run_react_plan( *, prompt: str, workbook_context: Optional[dict], + tool_context: dict[str, Any], provider_override: Optional[str], model_override: Optional[str], ): @@ -167,14 +194,16 @@ def validator(operations: list[dict], context: dict | None): engine = ReActEngine( llm_generate=llm_generate, validator=validator, + tool_executor=execute_tool_call, max_iterations=8, - max_operations_per_step=40, + max_operations_per_step=30, ) try: return await asyncio.wait_for( engine.execute( task=prompt, workbook_context=workbook_context, + tool_context=tool_context, rules_prompt=get_rules_engine().get_rules_prompt(), ), timeout=75, @@ -185,11 +214,17 @@ def validator(operations: list[dict], context: dict | None): def _normalize_operations(raw_operations: list[dict[str, Any]]) -> list[WorkbookOperation]: normalized: list[WorkbookOperation] = [] + failures: list[str] = [] for idx, op in enumerate(raw_operations or []): try: normalized.append(legacy_to_operation(op, idx)) except Exception as exc: - logger.warning("operation_normalize_skipped", index=idx, error=str(exc), operation=op) + logger.error("operation_normalize_rejected", index=idx, error=str(exc), operation=op) + failures.append(f"operation[{idx}]: {exc}") + if failures: + raise OperationNormalizationError( + "Proposed operations failed canonical validation: " + "; ".join(failures[:5]) + ) return normalized diff --git a/backend/services/operation_registry.py b/backend/services/operation_registry.py index ebef5b3..65d1841 100644 --- a/backend/services/operation_registry.py +++ b/backend/services/operation_registry.py @@ -182,16 +182,18 @@ def get_operation_registry_v3() -> dict[str, dict]: v7_ready = op_type not in GATED_OPERATIONS mock_coverage_status = _mock_coverage_status(op_type) rollback_proof_class = _rollback_proof_class(metadata) + apply_ready = v7_ready and rollback_proof_class != "manual" and mock_coverage_status != "missing" + rollback_strategy_v7 = _rollback_strategy_v7(metadata, rollback_proof_class) metadata.update( { "schema_version": "2026-05-operation-registry-v3-entry", "checkpoint_policy": "required" if risk in {"blocked", "high"} or metadata.get("approval_required") else metadata.get("checkpoint_policy", "none"), "execution_group_policy": "read" if family == "read" else "checkpoint" if risk == "blocked" else "safe_group", "simulation_support": "mandatory" if family != "read" else "not_required", - "rollback_strategy_v7": metadata.get("snapshot_strategy") if family.startswith("range") or family == "range_write" else "manual", + "rollback_strategy_v7": rollback_strategy_v7, "generated_frontend_handler_key": f"{metadata.get('frontend_handler', 'range')}:{op_type}", "v7_ready": v7_ready, - "applyReady": v7_ready, + "applyReady": apply_ready, "safeGated": op_type in SAFE_GATED_OPERATIONS, "hostRequirement": metadata.get("required_requirement_set", "ExcelApi 1.1"), "readinessReason": readiness_reason, @@ -210,11 +212,11 @@ def get_operation_registry_v3() -> dict[str, dict]: }, "rollbackProof": { "proofClass": rollback_proof_class, - "strategy": metadata.get("rollback_strategy_v7") or metadata.get("rollback_strategy") or "manual", + "strategy": rollback_strategy_v7, "warningsRequired": rollback_proof_class == "manual", }, "dependencyRisk": {"status": "passed", "advisories": []}, - "defaultSwitchBlockers": _default_switch_blockers(op_type, v7_ready, safe_gated_reason, mock_coverage_status), + "defaultSwitchBlockers": _default_switch_blockers(op_type, apply_ready, safe_gated_reason, mock_coverage_status), } ) return registry @@ -269,8 +271,17 @@ def operation_metadata(op_type: str) -> dict: def _rollback_proof_class(metadata: dict) -> str: strategy = str(metadata.get("snapshot_strategy") or metadata.get("rollback_strategy") or "manual") family = str(metadata.get("family") or "") + op_type = str(metadata.get("type") or "") if family == "read": return "not_required" + if op_type in {"merge_range", "unmerge_range"}: + return "manual" + if family == "chart": + return "chart_restore_operations" + if family == "table": + return "table_restore_operations" + if op_type in {"add_comment", "update_comment", "delete_comment"}: + return "comment_restore_operations" if strategy in {"values_formulas_formats", "formats", "restore_range"} or family.startswith("range"): return "range_restore_operations" if strategy == "created_object_pointer": @@ -278,8 +289,29 @@ def _rollback_proof_class(metadata: dict) -> str: return "manual" +def _rollback_strategy_v7(metadata: dict, proof_class: str) -> str: + if proof_class in {"chart_restore_operations", "table_restore_operations", "comment_restore_operations", "range_restore_operations", "not_required"}: + return proof_class + return str(metadata.get("rollback_strategy") or "manual") + + def _mock_coverage_status(op_type: str) -> str: - if op_type in {"write_cell", "write_range", "clear_range", "screenshot_range", "create_sparkline"}: + if op_type in { + "write_cell", + "write_range", + "clear_range", + "screenshot_range", + "create_sparkline", + "create_chart", + "update_chart", + "delete_chart", + "create_table", + "update_table", + "delete_table", + "add_comment", + "update_comment", + "delete_comment", + }: return "behavioral" if op_type in SAFE_GATED_OPERATIONS: return "safe_gated" diff --git a/backend/services/operation_validation.py b/backend/services/operation_validation.py index 221b839..a697a55 100644 --- a/backend/services/operation_validation.py +++ b/backend/services/operation_validation.py @@ -3,12 +3,14 @@ from __future__ import annotations import re +from collections.abc import Sequence from backend.api.validators import validate_range_reference from backend.config import OFFICECLI_SCHEMA_MODE from backend.excel.formula_parser import extract_cell_references from backend.excel.formula_validator import validate_formula from backend.excel.xlsx_schema_registry import schema_issues, schema_rule_for +from backend.services.cell_value_safety import is_potential_formula from backend.models.operations import ( OperationPlan, OperationPlanV2, @@ -54,6 +56,8 @@ def validate_workbook_operations( validation_profile: str | None = None, ) -> ValidationReport: issues: list[ValidationIssue] = [] + issues.extend(_registry_apply_issues(operations)) + issues.extend(_unsupported_contract_field_issues(operations)) seen_targets: set[tuple[str, str]] = set() existing_cells = _existing_cells(workbook_context) formula_cells = _formula_cells(workbook_context) @@ -89,6 +93,23 @@ def validate_workbook_operations( seen_targets.add(target) if op.operation_type in {"write_cell", "write_range"}: + # Formula/CSV injection: a write_cell with a string value that + # would Excel-evaluate as a formula is almost certainly an AI or + # source-extraction mistake; the caller should be using set_formula. + if op.value is not None and is_potential_formula(op.value): + issues.append( + ValidationIssue( + severity="warning", + code="formula_like_literal", + risk="review", + message=( + "write_cell value starts with a formula trigger and would be evaluated by Excel; " + "use a formula operation or prefix the value to keep it literal." + ), + operation_id=op.id, + range=op.range_ref, + ) + ) touched = _expand_range_safe(op.range_ref) overlaps_existing = sorted(touched & existing_cells) overlaps_formulas = sorted(touched & formula_cells) @@ -335,6 +356,8 @@ def validate_workbook_operations_v2( ) -> ValidationReport: """Validate the broader V2 contract before it reaches Office.js.""" issues: list[ValidationIssue] = [] + issues.extend(_registry_apply_issues(operations)) + issues.extend(_unsupported_contract_field_issues(operations)) existing_cells = _existing_cells(workbook_context) formula_cells = _formula_cells(workbook_context) protected = {str(name).lower() for name, value in (workbook_context or {}).get("sheet_protection", {}).items() if value} @@ -571,7 +594,12 @@ def _existing_cells(workbook_context: dict | None) -> set[str]: if not workbook_context: return set() snapshot = workbook_context.get("cell_snapshot") or {} - return {str(cell).upper() for cell in snapshot.keys()} + cells: set[str] = set() + for cell in snapshot.keys(): + qualified = str(cell).upper() + cells.add(qualified) + cells.add(qualified.rsplit("!", 1)[-1]) + return cells def _formula_cells(workbook_context: dict | None) -> set[str]: @@ -581,10 +609,84 @@ def _formula_cells(workbook_context: dict | None) -> set[str]: cells = set() for cell, info in snapshot.items(): if isinstance(info, dict) and info.get("formula"): - cells.add(str(cell).upper()) + qualified = str(cell).upper() + cells.add(qualified) + cells.add(qualified.rsplit("!", 1)[-1]) return cells +def _registry_apply_issues(operations: Sequence[object]) -> list[ValidationIssue]: + """Fail closed when registry metadata says an operation cannot be applied safely.""" + from backend.services.operation_registry import get_operation_registry_v3 + + registry = get_operation_registry_v3() + issues: list[ValidationIssue] = [] + for op in operations: + op_type = str(getattr(op, "operation_type", None) or getattr(op, "type", "")) + metadata = registry.get(op_type) or {} + operation_id = getattr(op, "id", None) + range_ref = getattr(op, "range_ref", None) or getattr(op, "range", None) + if metadata.get("safeGated"): + issues.append( + ValidationIssue( + severity="error", + risk="blocked", + code="operation_safe_gated", + message=str(metadata.get("readinessReason") or f"{op_type} is safely gated"), + operation_id=operation_id, + range=range_ref, + schema_family=str(metadata.get("schema_family") or metadata.get("family") or ""), + approval_required=True, + ) + ) + elif not metadata.get("applyReady", False): + issues.append( + ValidationIssue( + severity="error", + risk="blocked", + code="operation_not_apply_ready", + message=f"{op_type} is not apply-ready because automatic rollback proof is unavailable", + operation_id=operation_id, + range=range_ref, + schema_family=str(metadata.get("schema_family") or metadata.get("family") or ""), + approval_required=True, + ) + ) + return issues + + +def _unsupported_contract_field_issues(operations: Sequence[object]) -> list[ValidationIssue]: + """Reject modeled fields the Office adapter cannot yet apply and roll back exactly.""" + issues: list[ValidationIssue] = [] + for op in operations: + op_type = str(getattr(op, "operation_type", None) or getattr(op, "type", "")) + if op_type not in {"create_chart", "update_chart"}: + continue + unsupported = [ + field + for field in ("position", "width", "height") + if getattr(op, field, None) is not None + ] + if not unsupported: + continue + issues.append( + ValidationIssue( + severity="error", + risk="blocked", + code="unsupported_contract_fields", + message=( + "Chart geometry is not apply-ready with exact rollback proof; " + f"remove unsupported field(s): {', '.join(unsupported)}" + ), + operation_id=getattr(op, "id", None), + range=getattr(op, "range_ref", None) or getattr(op, "range", None), + schema_family="chart", + approval_required=True, + ) + ) + return issues + + def _formula_reference_issues(operation_id: str | None, range_ref: str, formula: str) -> list[ValidationIssue]: issues: list[ValidationIssue] = [] upper = formula.upper() diff --git a/backend/services/plan_state_machine.py b/backend/services/plan_state_machine.py new file mode 100644 index 0000000..bc03f5f --- /dev/null +++ b/backend/services/plan_state_machine.py @@ -0,0 +1,93 @@ +"""Plan state machine. + +Centralizes valid status / lifecycle transitions for OperationPlan (v1/v2) and +OperationPlanV3 so that ad-hoc status assignments cannot back-date a plan or +move a terminal plan back to active. + +Used by routes that mutate plan status; raises ``InvalidPlanTransition`` when +a transition is illegal. Callers should map that to HTTP 409. +""" + +from __future__ import annotations + +from typing import Iterable + +# OperationPlan / OperationPlanV2 use the same Literal set: +# draft, approved, applied, failed, rollback_pending, rolled_back, abandoned +PLAN_STATUS_TRANSITIONS: dict[str, frozenset[str]] = { + "draft": frozenset({"draft", "approved", "abandoned"}), + "approved": frozenset({"applied", "failed", "abandoned", "approved"}), + "applied": frozenset({"rollback_pending", "applied"}), + "rollback_pending": frozenset({"rolled_back", "applied", "rollback_pending"}), + "failed": frozenset(), + "rolled_back": frozenset(), + "abandoned": frozenset(), +} + +# OperationPlanV3 lifecycle states +PLAN_V3_LIFECYCLE_TRANSITIONS: dict[str, frozenset[str]] = { + "draft": frozenset({"inspection_required", "ready", "replanned", "failed", "draft"}), + "inspection_required": frozenset({"ready", "replanned", "draft", "failed", "inspection_required"}), + "replanned": frozenset({"ready", "draft", "failed", "replanned"}), + "ready": frozenset({"approved", "replanned", "failed", "ready"}), + "approved": frozenset({"applied", "failed", "approved"}), + "applied": frozenset({"verified", "failed", "applied"}), + "verified": frozenset(), + "failed": frozenset(), +} + + +class InvalidPlanTransition(Exception): + """Raised when a plan status / lifecycle transition is not permitted.""" + + def __init__(self, kind: str, current: str, requested: str, plan_id: str = "") -> None: + self.kind = kind + self.current = current + self.requested = requested + self.plan_id = plan_id + super().__init__( + f"Illegal {kind} transition from {current!r} to {requested!r}" + + (f" for plan {plan_id}" if plan_id else "") + ) + + +def can_transition(kind: str, current: str, requested: str) -> bool: + """Return True if a transition is permitted under ``kind`` (``"plan"`` or ``"v3"``).""" + + table = PLAN_STATUS_TRANSITIONS if kind == "plan" else PLAN_V3_LIFECYCLE_TRANSITIONS + if current not in table: + # Unknown current state: treat as terminal — refuse any new transition. + return False + return requested in table[current] + + +def ensure_transition( + kind: str, + current: str, + requested: str, + *, + plan_id: str = "", + allow_same: bool = True, +) -> None: + """Validate a plan status / V3 lifecycle transition or raise. + + ``allow_same=True`` lets a status be re-applied (the table already lists each + state as a permitted target of itself; the flag exists for callers that + explicitly want to reject no-op writes). + """ + + if not allow_same and current == requested: + raise InvalidPlanTransition(kind, current, requested, plan_id) + if not can_transition(kind, current, requested): + raise InvalidPlanTransition(kind, current, requested, plan_id) + + +def filter_terminal(statuses: Iterable[str]) -> list[str]: + """Return the subset of statuses that are terminal under either machine.""" + + terminals: set[str] = set() + for kind_table in (PLAN_STATUS_TRANSITIONS, PLAN_V3_LIFECYCLE_TRANSITIONS): + for state, allowed in kind_table.items(): + if not allowed: + terminals.add(state) + return [s for s in statuses if s in terminals] diff --git a/backend/services/repair_planning.py b/backend/services/repair_planning.py index 9fc181d..9d8dd86 100644 --- a/backend/services/repair_planning.py +++ b/backend/services/repair_planning.py @@ -8,7 +8,6 @@ from backend.models.operations import ( OperationSimulationV4, - RepairPlanV2, RepairPlanV3, WorkbookOperationV2, WorkbookTruthModel, @@ -103,7 +102,7 @@ def _repair_evidence(suggestions: list[dict[str, Any]], truth_model: WorkbookTru return evidence[:50] -def _confidence(simulation: OperationSimulationV3, truth_model: WorkbookTruthModel | None) -> float: +def _confidence(simulation: OperationSimulationV4, truth_model: WorkbookTruthModel | None) -> float: base = float(getattr(truth_model, "accuracy_score", 0.55) if truth_model is not None else 0.55) if simulation.simulation_gate == "passed": base += 0.08 diff --git a/backend/services/retention.py b/backend/services/retention.py new file mode 100644 index 0000000..4c23d43 --- /dev/null +++ b/backend/services/retention.py @@ -0,0 +1,59 @@ +"""Terminal workflow hooks for executable zero-hour retention.""" + +from __future__ import annotations + +from typing import Any + +import structlog + +logger = structlog.get_logger() + +TERMINAL_STATUSES = frozenset( + { + "abandoned", + "applied", + "cancelled", + "canceled", + "completed", + "failed", + "rolled_back", + } +) + + +class ImmediateRetentionError(RuntimeError): + """Immediate retention was promised but verified deletion did not finish.""" + + +async def purge_terminal_session_if_immediate( + session_id: str, + terminal_status: str, +) -> dict[str, Any] | None: + """Await verified deletion after a terminal result has been formed.""" + normalized_status = str(terminal_status or "").strip().lower() + if normalized_status not in TERMINAL_STATUSES: + return None + + from backend.api.security.rbac import get_zdr + + zdr = get_zdr() + if not zdr.enabled or zdr.retention_hours != 0: + return None + receipt = await zdr.purge_session(session_id) + if not receipt or receipt.get("state") != "verified": + logger.error( + "terminal_zero_hour_purge_unverified", + session_id=session_id, + terminal_status=normalized_status, + receipt=receipt, + ) + raise ImmediateRetentionError( + "Terminal result was recorded, but immediate session deletion was not verified" + ) + logger.info( + "terminal_zero_hour_purge_verified", + session_id=session_id, + terminal_status=normalized_status, + receipt_id=receipt.get("receiptId") or receipt.get("receipt_id"), + ) + return receipt diff --git a/backend/services/source_evidence.py b/backend/services/source_evidence.py index 2ddd2b6..3a95e70 100644 --- a/backend/services/source_evidence.py +++ b/backend/services/source_evidence.py @@ -7,6 +7,7 @@ from typing import Any from backend.models.operations import SourceEvidence, SourceEvidenceGraph +from backend.services.cell_value_safety import sanitize_literal FINANCIAL_LABELS = { @@ -76,12 +77,16 @@ def _evidence_from_document(document: dict[str, Any], workbook_context: dict[str continue rows.extend(_evidence_from_row(row, doc_id, filename, page, table_name, row_index, workbook_context)) for key, value in (summary.get("extracted_params") or {}).items(): + # Document-extracted values may begin with formula triggers (=, +, -, @). + # If they are later written into a cell as text, Excel would evaluate + # them; sanitize_literal prepends a single quote when needed so the + # source evidence carries a safe-to-paste representation. rows.append( SourceEvidence( sourceDocumentId=doc_id, source=filename, - label=str(key).replace("_", " ").title(), - value=value, + label=sanitize_literal(str(key).replace("_", " ").title()), + value=sanitize_literal(value) if isinstance(value, str) else value, extractedText=f"{key}: {value}", confidence=0.78, transformation="summary extracted parameter", diff --git a/backend/services/workbook_invariants.py b/backend/services/workbook_invariants.py index ae12e68..06ad509 100644 --- a/backend/services/workbook_invariants.py +++ b/backend/services/workbook_invariants.py @@ -30,18 +30,20 @@ def check( workbook_context: dict[str, Any] | None = None, ) -> WorkbookInvariantReport: if isinstance(semantic_map, dict): - semantic_map = WorkbookSemanticMapV2.model_validate(semantic_map) - model_type = semantic_map.model_type or "unknown" + semantic_map_obj = WorkbookSemanticMapV2.model_validate(semantic_map) + else: + semantic_map_obj = semantic_map + model_type = semantic_map_obj.model_type or "unknown" rules = RULES_BY_MODEL.get(model_type, RULES_BY_MODEL["unknown"]) - labels = " | ".join(entity.label.lower() for entity in semantic_map.entities) - formulas = " | ".join(item.formula.lower() for item in semantic_map.formulas) + labels = " | ".join(entity.label.lower() for entity in semantic_map_obj.entities) + formulas = " | ".join(item.formula.lower() for item in semantic_map_obj.formulas) findings: list[WorkbookInvariantFinding] = [] for rule in rules: - finding = _evaluate_rule(rule, labels, formulas, semantic_map, workbook_context or {}) + finding = _evaluate_rule(rule, labels, formulas, semantic_map_obj, workbook_context or {}) if finding: findings.append(finding) status = "failed" if any(item.severity == "error" for item in findings) else "warning" if findings else "passed" - accuracy_score = round(max(0.0, semantic_map.accuracy_score - 0.08 * len([f for f in findings if f.severity != "info"])), 3) + accuracy_score = round(max(0.0, semantic_map_obj.accuracy_score - 0.08 * len([f for f in findings if f.severity != "info"])), 3) return WorkbookInvariantReport( status=status, modelType=model_type, @@ -49,9 +51,9 @@ def check( checkedRules=rules, accuracyScore=accuracy_score, metadata={ - "formula_count": len(semantic_map.formulas), - "entity_count": len(semantic_map.entities), - "source_coverage": semantic_map.source_coverage, + "formula_count": len(semantic_map_obj.formulas), + "entity_count": len(semantic_map_obj.entities), + "source_coverage": semantic_map_obj.source_coverage, }, ) diff --git a/backend/services/workbook_jobs.py b/backend/services/workbook_jobs.py index edc2a4f..6254f8d 100644 --- a/backend/services/workbook_jobs.py +++ b/backend/services/workbook_jobs.py @@ -1,23 +1,38 @@ -"""Lightweight async workbook job manager. +"""Durable async workbook job manager. Large workbook ingest/export/formula-analysis tasks should not be forced into a -single synchronous HTTP request. This in-memory manager gives the API a -Shortcut-style runId/status contract while keeping runtime dependencies small. +single synchronous HTTP request. This manager gives the API a Shortcut-style +runId/status/result contract while persisting job envelopes to disk so polling +survives process restarts. """ from __future__ import annotations import base64 +import json +import os import threading import time import uuid +from pathlib import Path from dataclasses import dataclass, field from typing import Any, Literal +from backend.api.security.ownership import OwnershipNotFoundError, current_principal +from backend.operations.work_admission import ( + CancellationSignal, + WorkBudgetError, + WorkExecutionTimeout, + work_context, +) +from backend.storage.protocol import SessionStoreProtocol + WorkbookJobCommand = Literal[ "ingest", + "open-workbook-v1", "export", + "export-workbook-v1", "diff", "formula-analyze", "semantic-map", @@ -29,6 +44,7 @@ "openxml-qa", "export-template", "roundtrip-qa", + "roundtrip-qa-v2", "truth-model-v4", "formula-graph-v4", "simulate-operations-v5", @@ -44,7 +60,12 @@ "workbook-diff-v3", "fixture-validate", ] -WorkbookJobStatus = Literal["queued", "running", "completed", "failed"] +WorkbookJobStatus = Literal["queued", "running", "completed", "failed", "cancelled"] +ROOT = Path(__file__).resolve().parents[2] +DEFAULT_JOB_DIR = ROOT / "data" / "workbook-jobs" +MAX_ACTIVE_JOBS_PER_ACCOUNT = int(os.getenv("WORKBOOK_JOB_MAX_ACTIVE_PER_ACCOUNT", "8")) +WORKER_ONLY_JOB_COMMANDS = {"open-workbook-v1", "export-workbook-v1", "roundtrip-qa-v2"} +PYTHON_FIRST_JOB_COMMANDS = {"source-extract"} @dataclass @@ -54,8 +75,14 @@ class WorkbookJob: status: WorkbookJobStatus = "queued" created_at: float = field(default_factory=time.time) updated_at: float = field(default_factory=time.time) + account_id: str = "local" + session_id: str = "" + owner_user_id: str = "" + progress: int = 0 result: Any = None error: str = "" + payload_size_bytes: int = 0 + result_size_bytes: int = 0 def to_dict(self, include_result: bool = True) -> dict[str, Any]: payload = { @@ -64,6 +91,15 @@ def to_dict(self, include_result: bool = True) -> dict[str, Any]: "status": self.status, "created_at": self.created_at, "updated_at": self.updated_at, + "account_id": self.account_id, + "session_id": self.session_id, + "owner_user_id": self.owner_user_id, + "progress": self.progress, + "payload_size_bytes": self.payload_size_bytes, + "result_size_bytes": self.result_size_bytes, + "status_url": f"/api/v1/workbook/jobs/{self.run_id}", + "result_url": f"/api/v1/workbook/jobs/{self.run_id}/result", + "download_url": f"/api/v1/workbook/jobs/{self.run_id}/download", } if self.error: payload["error"] = self.error @@ -71,45 +107,413 @@ def to_dict(self, include_result: bool = True) -> dict[str, Any]: payload["result"] = self.result return payload + @classmethod + def from_dict(cls, data: dict[str, Any]) -> "WorkbookJob": + return cls( + run_id=str(data["run_id"]), + command=data["command"], + status=data.get("status", "queued"), + created_at=float(data.get("created_at") or time.time()), + updated_at=float(data.get("updated_at") or time.time()), + account_id=str(data.get("account_id") or "local"), + session_id=str(data.get("session_id") or ""), + owner_user_id=str(data.get("owner_user_id") or ""), + progress=int(data.get("progress") or 0), + result=data.get("result"), + error=str(data.get("error") or ""), + payload_size_bytes=int(data.get("payload_size_bytes") or 0), + result_size_bytes=int(data.get("result_size_bytes") or 0), + ) + class WorkbookJobManager: - def __init__(self) -> None: + def __init__( + self, + storage_dir: Path | str = DEFAULT_JOB_DIR, + *, + store: SessionStoreProtocol | None = None, + use_session_store: bool = False, + ) -> None: + self.storage_dir = Path(storage_dir) + self.storage_dir.mkdir(parents=True, exist_ok=True) + self._store = store + self._use_session_store = use_session_store or store is not None + self._last_store_error = "" self._jobs: dict[str, WorkbookJob] = {} - - def submit(self, command: WorkbookJobCommand, payload: dict[str, Any]) -> WorkbookJob: + self._threads: dict[str, threading.Thread] = {} + self._cancellations: dict[str, CancellationSignal] = {} + self._purging_sessions: set[str] = set() + self._purge_pending_threads: dict[str, set[threading.Thread]] = {} + self._lock = threading.RLock() + self._load_jobs() + + def submit( + self, + command: WorkbookJobCommand, + payload: dict[str, Any], + *, + account_id: str = "local", + session_id: str = "", + ) -> WorkbookJob: + principal = current_principal() + account_id = account_id or "local" + session_id = session_id or str(payload.get("session_id") or payload.get("sessionId") or "") + with self._lock: + if session_id and session_id in self._purging_sessions: + raise RuntimeError("Workbook session is being purged") + owner_user_id = "" + if principal.authenticated and not principal.is_admin: + if not principal.user_id or not session_id: + raise OwnershipNotFoundError("Workbook job not found") + owner_user_id = principal.user_id + account_id = principal.user_id + elif principal.authenticated: + owner_user_id = principal.user_id + account_id = principal.user_id or "authenticated" + store = self._store_or_none() + create_session = getattr(store, "get_or_create_session", None) if store else None + with self._lock: + if session_id and session_id in self._purging_sessions: + raise RuntimeError("Workbook session is being purged") + if session_id and callable(create_session): + create_session(session_id) + active = [ + job + for job in self._jobs.values() + if job.account_id == account_id and job.status in {"queued", "running"} + ] + if len(active) >= MAX_ACTIVE_JOBS_PER_ACCOUNT: + raise RuntimeError(f"Active workbook job limit reached for account '{account_id}'") run_id = f"run_{uuid.uuid4().hex[:12]}" - job = WorkbookJob(run_id=run_id, command=command) - self._jobs[run_id] = job + payload_size = len(json.dumps(payload, default=str).encode("utf-8")) + job = WorkbookJob( + run_id=run_id, + command=command, + account_id=account_id, + session_id=session_id, + owner_user_id=owner_user_id, + payload_size_bytes=payload_size, + ) + with self._lock: + if session_id and session_id in self._purging_sessions: + raise RuntimeError("Workbook session is being purged") + self._jobs[run_id] = job + self._cancellations[run_id] = CancellationSignal() + try: + self._persist(job) + except Exception: + self._jobs.pop(run_id, None) + self._cancellations.pop(run_id, None) + raise thread = threading.Thread(target=self._run, args=(job, payload), daemon=True) + with self._lock: + self._threads[run_id] = thread thread.start() return job def get(self, run_id: str) -> WorkbookJob | None: - return self._jobs.get(run_id) + store = self._store_or_none() + if self._use_session_store and store is None: + return None + with self._lock: + job = self._jobs.get(run_id) + if job and self._job_is_active(job) and self._job_is_visible(job): + return job + if job: + return None + if not store: + return None + record = store.get_workbook_job(run_id) + if not record: + return None + with self._lock: + cached = self._cache_job_record(record, mark_interrupted=False) + return cached if cached and self._job_is_active(cached) and self._job_is_visible(cached) else None + + def list( + self, + *, + limit: int = 50, + status: str | None = None, + session_id: str | None = None, + account_id: str | None = None, + ) -> list[WorkbookJob]: + store = self._store_or_none() + if self._use_session_store and store is None: + return [] + if store: + for record in store.list_workbook_jobs( + limit=max(limit, 0), + status=status, + session_id=session_id, + account_id=account_id, + ): + with self._lock: + cached = self._cache_job_record(record, mark_interrupted=False) + if cached and not self._job_is_active(cached): + self._jobs.pop(cached.run_id, None) + with self._lock: + jobs = list(self._jobs.values()) + jobs = [job for job in jobs if self._job_is_active(job) and self._job_is_visible(job)] + if status: + jobs = [job for job in jobs if job.status == status] + if session_id: + jobs = [job for job in jobs if job.session_id == session_id] + if account_id: + jobs = [job for job in jobs if job.account_id == account_id] + return sorted(jobs, key=lambda job: job.created_at, reverse=True)[: max(limit, 0)] + + def cancel(self, run_id: str) -> WorkbookJob | None: + with self._lock: + job = self._jobs.get(run_id) + if not job or not self._job_is_visible(job): + return None + if job.status in {"completed", "failed", "cancelled"}: + return job + job.status = "cancelled" + job.progress = min(job.progress, 99) + job.updated_at = time.time() + job.error = "cancelled by request" + cancellation = self._cancellations.get(run_id) + if cancellation is not None: + cancellation.cancel("request") + self._persist(job) + return job + + @staticmethod + def _job_is_visible(job: WorkbookJob) -> bool: + principal = current_principal() + if not principal.authenticated or principal.is_admin: + return True + return bool(principal.user_id) and job.owner_user_id == principal.user_id def _run(self, job: WorkbookJob, payload: dict[str, Any]) -> None: - job.status = "running" - job.updated_at = time.time() try: - job.result = _execute_command(job.command, payload) - job.status = "completed" + with self._lock: + if job.status == "cancelled" or job.session_id in self._purging_sessions: + return + job.status = "running" + job.progress = 25 + job.updated_at = time.time() + self._persist(job) + cancellation = self._cancellations.get(job.run_id) + with work_context(session_id=job.session_id, cancellation_event=cancellation): + result = _execute_command(job.command, payload) + with self._lock: + if job.status == "cancelled" or job.session_id in self._purging_sessions: + return + job.result = result + job.status = "completed" + job.progress = 100 + job.result_size_bytes = len(json.dumps(result, default=str).encode("utf-8")) + job.updated_at = time.time() + self._persist(job) except Exception as exc: - job.status = "failed" - job.error = str(exc) + with self._lock: + if job.status != "cancelled" and job.session_id not in self._purging_sessions: + job.status = "failed" + job.error = str(exc) + job.progress = 100 + job.updated_at = time.time() + try: + self._persist(job) + except Exception as persist_exc: + self._last_store_error = str(persist_exc) finally: + with self._lock: + self._threads.pop(job.run_id, None) + self._cancellations.pop(job.run_id, None) + current = threading.current_thread() + for pending in self._purge_pending_threads.values(): + pending.discard(current) + + def _path(self, run_id: str) -> Path: + return self.storage_dir / f"{run_id}.json" + + def _persist(self, job: WorkbookJob) -> None: + record = job.to_dict(include_result=True) + store = self._store_or_none() + if store: + try: + store.save_workbook_job(record) + except Exception as exc: + self._last_store_error = str(exc) + raise RuntimeError("Workbook job persistence failed") from exc + self._path(job.run_id).unlink(missing_ok=True) + return + if self._use_session_store: + raise RuntimeError("Workbook job store is unavailable") + tmp = self._path(job.run_id).with_suffix(".tmp") + tmp.write_text(json.dumps(record, indent=2, sort_keys=True), encoding="utf-8") + tmp.replace(self._path(job.run_id)) + + def _load_jobs(self) -> None: + loaded: set[str] = set() + store = self._store_or_none() + if store: + try: + records = store.list_workbook_jobs(limit=1000) + except Exception as exc: + self._last_store_error = str(exc) + records = [] + for record in records: + job = self._cache_job_record(record, mark_interrupted=True) + if job: + loaded.add(job.run_id) + for path in self.storage_dir.glob("run_*.json"): + if path.stem in loaded: + path.unlink(missing_ok=True) + else: + self._quarantine(path) + return + if self._use_session_store: + for path in self.storage_dir.glob("run_*.json"): + self._quarantine(path) + return + for path in self.storage_dir.glob("run_*.json"): + try: + record = json.loads(path.read_text(encoding="utf-8")) + except (OSError, json.JSONDecodeError, KeyError, TypeError, ValueError): + continue + run_id = str(record.get("run_id") or "") + if run_id and run_id in loaded: + continue + self._cache_job_record(record, mark_interrupted=True) + + def purge_session(self, session_id: str, *, join_timeout: float = 2.0) -> dict[str, Any]: + """Revoke and remove all in-process and legacy-file state for a session.""" + normalized_session_id = str(session_id or "").strip() + if not normalized_session_id: + return {"jobs_revoked": 0, "threads_joined": 0, "threads_pending": 0, "files_removed": 0} + with self._lock: + self._purging_sessions.add(normalized_session_id) + run_ids = [ + job.run_id + for job in self._jobs.values() + if job.session_id == normalized_session_id + ] + threads = { + thread + for thread in self._purge_pending_threads.pop(normalized_session_id, set()) + if thread.is_alive() + } + threads.update( + self._threads[run_id] + for run_id in run_ids + if run_id in self._threads and self._threads[run_id].is_alive() + ) + for run_id in run_ids: + job = self._jobs.get(run_id) + if job: + job.status = "cancelled" + job.error = "cancelled by session purge" + cancellation = self._cancellations.get(run_id) + if cancellation is not None: + cancellation.cancel("session_purge") + self._jobs.pop(run_id, None) + deadline = time.monotonic() + max(float(join_timeout), 0.0) + joined = 0 + for thread in threads: + thread.join(timeout=max(deadline - time.monotonic(), 0.0)) + if not thread.is_alive(): + joined += 1 + remaining = {thread for thread in threads if thread.is_alive()} + with self._lock: + if remaining: + self._purge_pending_threads[normalized_session_id] = remaining + else: + self._purge_pending_threads.pop(normalized_session_id, None) + files_removed = self._remove_session_files(normalized_session_id) + return { + "jobs_revoked": len(run_ids), + "threads_joined": joined, + "threads_pending": len(remaining), + "files_removed": files_removed, + } + + def _job_is_active(self, job: WorkbookJob) -> bool: + if job.session_id in self._purging_sessions: + return False + if not job.session_id: + return True + store = self._store_or_none() + checker = getattr(store, "session_exists", None) if store else None + return bool(checker(job.session_id)) if callable(checker) else True + + def _quarantine(self, path: Path) -> None: + quarantine_dir = self.storage_dir / "quarantine" + quarantine_dir.mkdir(parents=True, exist_ok=True) + target = quarantine_dir / path.name + if target.exists(): + path.unlink(missing_ok=True) + else: + path.replace(target) + + def _remove_session_files(self, session_id: str) -> int: + removed = 0 + candidates = [*self.storage_dir.glob("run_*.json"), *(self.storage_dir / "quarantine").glob("run_*.json")] + for path in candidates: + try: + record = json.loads(path.read_text(encoding="utf-8")) + except (OSError, json.JSONDecodeError, TypeError, ValueError): + continue + if str(record.get("session_id") or "") == session_id: + path.unlink(missing_ok=True) + removed += 1 + return removed + + def _store_or_none(self) -> SessionStoreProtocol | None: + if not self._use_session_store: + return None + if self._store is not None: + return self._store + try: + from backend.storage import get_session_store + + self._store = get_session_store() + except Exception as exc: + self._last_store_error = str(exc) + return None + return self._store + + def _cache_job_record(self, record: dict[str, Any], *, mark_interrupted: bool) -> WorkbookJob | None: + try: + job = WorkbookJob.from_dict(record) + except (KeyError, TypeError, ValueError): + return None + if mark_interrupted and job.status in {"queued", "running"}: + job.status = "failed" + job.error = "interrupted by process restart" + job.progress = 100 job.updated_at = time.time() + self._jobs[job.run_id] = job + if mark_interrupted: + self._persist(job) + return job def _execute_command(command: WorkbookJobCommand, payload: dict[str, Any]) -> Any: + if command in PYTHON_FIRST_JOB_COMMANDS: + return _execute_python_command(command, payload) + + worker_error = "" try: from backend.excel.workbook_worker import run_worker worker = run_worker(command, payload) if worker.ok: return worker.result - except Exception: - pass + worker_error = worker.error or "Workbook worker failed" + except (WorkBudgetError, WorkExecutionTimeout): + raise + except Exception as exc: + worker_error = str(exc) or exc.__class__.__name__ + if worker_error: + raise RuntimeError(worker_error) + raise RuntimeError("Workbook worker failed without an error") + +def _execute_python_command(command: WorkbookJobCommand, payload: dict[str, Any]) -> Any: if command == "ingest": from backend.excel.workbook_files import WorkbookIngestionService @@ -321,8 +725,11 @@ def _template_sheets(template: str, payload: dict[str, Any]) -> dict[str, list[l return {"Summary": [["Template", template], ["Generated By", "Finance AI"]]} -_manager = WorkbookJobManager() +_manager: WorkbookJobManager | None = None def get_workbook_job_manager() -> WorkbookJobManager: + global _manager + if _manager is None: + _manager = WorkbookJobManager(use_session_store=True) return _manager diff --git a/backend/services/workbook_truth.py b/backend/services/workbook_truth.py index 1b5845f..3ba17ae 100644 --- a/backend/services/workbook_truth.py +++ b/backend/services/workbook_truth.py @@ -10,7 +10,6 @@ OperationTarget, SemanticEntity, SemanticTargetResolution, - WorkbookTruthModel, WorkbookTruthModelV3, ) from backend.services.finance_invariants import get_finance_invariant_pack_service @@ -149,7 +148,7 @@ def _evidence_rankings( ) -> list[dict[str, Any]]: selected = str(context.get("selected_range") or "") named = {str(item.get("name") or "").lower(): item for item in context.get("named_ranges") or [] if isinstance(item, dict)} - rows = [] + rows: list[dict[str, Any]] = [] for entity in entities: score = float(entity.confidence) evidence = list(entity.evidence) @@ -178,7 +177,7 @@ def _evidence_rankings( "evidence": evidence[:8], } ) - rows.sort(key=lambda item: item["score"], reverse=True) + rows.sort(key=lambda item: float(item["score"]), reverse=True) return rows[:200] diff --git a/backend/session_store.py b/backend/session_store.py index e5359e9..cc28fb7 100644 --- a/backend/session_store.py +++ b/backend/session_store.py @@ -16,19 +16,54 @@ import os import secrets import sqlite3 +import threading import time import uuid from contextlib import contextmanager -from dataclasses import dataclass, field from pathlib import Path from typing import Optional import structlog -from backend.storage.migrations import migration_status, run_migrations +from backend.api.security.ownership import OwnershipNotFoundError, current_principal +from backend.storage.enterprise_identity import ( + api_key_status_allows, + prepare_enterprise_identity_binding, +) +from backend.storage.migrations import ( + SQLITE_SESSION_LIFECYCLE_TABLES, + assert_sqlite_session_lifecycle_integrity, + migration_status, + purge_sqlite_session_rows, + run_migrations, +) +from backend.storage.protocol import ConversationMessage +from backend.storage.v05_import_store import ( + MATERIALIZATION_MAX_ENTRY_BYTES, + MATERIALIZATION_MAX_PAYLOAD_BYTES, + MATERIALIZATION_MAX_PAYLOADS, + NEUTRAL_IMPORT_STATES, + NeutralV05ImportError, + binding_sha256 as v05_binding_sha256, + build_import_receipt, + canonical_json as v05_canonical_json, + ensure_row_binding, + neutral_import_id, + neutral_session_id, + normalize_import_bundle, + normalize_owner_id, + normalize_sha256, + public_import_status, + sha256_bytes, + sha256_text, + validate_import_receipt, +) logger = structlog.get_logger() +_SESSION_LOCKS_GUARD = threading.Lock() +_SESSION_LOCKS: dict[tuple[str, str], threading.RLock] = {} + # Database file location (relative to project root, overridable for deployments) try: from backend.config import DB_PATH as CONFIG_DB_PATH @@ -38,14 +73,6 @@ DB_PATH = Path(CONFIG_DB_PATH) -@dataclass -class ConversationMessage: - role: str # "user" | "assistant" | "system" - content: str - timestamp: float - metadata: dict = field(default_factory=dict) - - def _api_key_pepper() -> bytes: """Server-side pepper used to derive enterprise API-key hashes. @@ -75,34 +102,39 @@ def _enterprise_api_key_candidates(api_key: str) -> list[str]: return [_hash_enterprise_api_key(api_key), legacy] +def _enterprise_user_allows_api_key(user: Optional[dict], *, require_mfa: bool = False) -> bool: + if not user: + return False + if not api_key_status_allows(user.get("status")): + return False + return not require_mfa or bool(user.get("mfa_enabled")) + + class SessionStore: """SQLite-backed session store for conversation memory and task history.""" def __init__(self, db_path: Path | str = DB_PATH): self.db_path = Path(db_path) if not isinstance(db_path, Path) else db_path self.db_path.parent.mkdir(parents=True, exist_ok=True) + # The provider is part of the primary connection path. Encryption can + # therefore never be reported as enabled while SessionStore continues + # opening the same file through plaintext sqlite3 connections. + from backend.storage.encrypted_store import EncryptedSessionStore + + self._connection_provider = EncryptedSessionStore(db_path=str(self.db_path)) self._init_schema() @contextmanager def _conn(self): - conn = sqlite3.connect(str(self.db_path), timeout=10) - conn.row_factory = sqlite3.Row - conn.execute("PRAGMA journal_mode=WAL") - conn.execute("PRAGMA foreign_keys=ON") - try: + with self._connection_provider.connection() as conn: yield conn - conn.commit() - except Exception: - conn.rollback() - raise - finally: - conn.close() def _init_schema(self): with self._conn() as conn: conn.executescript(""" CREATE TABLE IF NOT EXISTS sessions ( session_id TEXT PRIMARY KEY, + owner_user_id TEXT, created_at REAL NOT NULL, updated_at REAL NOT NULL, metadata TEXT DEFAULT '{}' @@ -216,6 +248,8 @@ def _init_schema(self): status TEXT NOT NULL DEFAULT 'active', mfa_enabled INTEGER NOT NULL DEFAULT 0, metadata_json TEXT NOT NULL DEFAULT '{}', + oidc_issuer TEXT NOT NULL DEFAULT '', + oidc_subject TEXT NOT NULL DEFAULT '', created_at REAL NOT NULL, updated_at REAL NOT NULL ); @@ -274,31 +308,1070 @@ def get_migration_status(self) -> dict: with self._conn() as conn: return migration_status(conn) - # ── Session Management ─────────────────────────────────────── + # ── One-way v0.5 neutral migration intake ───────────────────── + + def consume_v05_import(self, request: dict, payloads: list[dict]) -> dict: + """Persist exact portable bytes without creating current domain state.""" + normalized, normalized_payloads = normalize_import_bundle(request, payloads) + principal = current_principal() + if ( + principal.authenticated + and not principal.is_admin + and principal.user_id != normalized["ownerId"] + ): + raise NeutralV05ImportError("owner_scope_forbidden") + from backend.services.lifecycle import get_lifecycle_coordinator + + target_session_id = str(normalized["targetSessionId"]) + if get_lifecycle_coordinator().writes_revoked(target_session_id): + raise NeutralV05ImportError("session_fenced") + + claim_version: int | None = None + try: + with self._session_lock(target_session_id): + prepared = self._prepare_v05_import(normalized, normalized_payloads) + if prepared["state"] == "imported": + return prepared + claimed = self._claim_v05_import(normalized, normalized_payloads) + if claimed["state"] == "imported": + return claimed + claim_version = int(claimed["version"]) + return self._complete_v05_import( + normalized, + normalized_payloads, + expected_version=claim_version, + ) + except NeutralV05ImportError: + raise + except Exception as exc: + if claim_version is not None: + self._mark_v05_import_retryable_failure( + normalized, + expected_version=claim_version, + failure_code="storage_write_failed", + ) + raise NeutralV05ImportError("storage_write_failed") from exc - def get_or_create_session(self, session_id: str) -> dict: - """Get or create a session.""" + def get_v05_import(self, import_id: str) -> Optional[dict]: + normalized_import_id = str(import_id or "").strip() + prefix = "v05-import-" + if not normalized_import_id.startswith(prefix): + return None + try: + if neutral_import_id(normalized_import_id.removeprefix(prefix)) != normalized_import_id: + return None + except NeutralV05ImportError: + return None + with self._read_conn() as conn: + row = conn.execute( + "SELECT * FROM migration_v05_imports WHERE import_id = ?", + (normalized_import_id,), + ).fetchone() + if not row: + return None + row_dict = dict(row) + if not self._session_visible_in_conn(conn, str(row_dict["session_id"])): + return None + request = self._v05_request_from_row(row_dict) + return self._v05_status_in_conn(conn, row_dict, request) + + def read_v05_import_bundle( + self, + import_id: str, + *, + expected_binding_sha256: str, + expected_content_sha256: str, + expected_owner_id: str, + expected_owner_scope_sha256: str, + expected_target_session_id: str, + ) -> Optional[dict]: + """Return an authenticated neutral bundle for a reviewed materializer. + + The raw payload bytes never leave the lifecycle/ownership-fenced + storage boundary unless the caller already knows both immutable + binding digests. The complete ledger and digest-bound neutral receipt are + revalidated in the same SQLite authorization snapshot. + """ + + normalized_import_id = str(import_id or "").strip() + prefix = "v05-import-" + if not normalized_import_id.startswith(prefix): + return None + try: + if neutral_import_id(normalized_import_id.removeprefix(prefix)) != normalized_import_id: + return None + expected_binding = normalize_sha256(expected_binding_sha256) + expected_content = normalize_sha256(expected_content_sha256) + expected_owner = normalize_owner_id(expected_owner_id) + expected_owner_scope = normalize_sha256(expected_owner_scope_sha256) + expected_session = str(expected_target_session_id or "").strip() + if ( + normalized_import_id != neutral_import_id(expected_content) + or expected_session != neutral_session_id(expected_content) + or not hmac.compare_digest(expected_owner_scope, sha256_text(expected_owner)) + ): + return None + except NeutralV05ImportError: + return None + with self._read_conn() as conn: + row = conn.execute( + "SELECT * FROM migration_v05_imports WHERE import_id = ?", + (normalized_import_id,), + ).fetchone() + if not row: + return None + row_dict = dict(row) + session_id = str(row_dict["session_id"]) + if not self._session_visible_in_conn(conn, session_id): + return None + owner_row = conn.execute( + "SELECT owner_user_id FROM sessions WHERE session_id = ?", + (session_id,), + ).fetchone() + if not owner_row: + return None + try: + stored_owner = normalize_owner_id(owner_row["owner_user_id"]) + except NeutralV05ImportError as exc: + raise NeutralV05ImportError("corrupt_import_state") from exc + if ( + stored_owner != expected_owner + or not hmac.compare_digest(sha256_text(stored_owner), expected_owner_scope) + ): + return None + request = self._v05_request_from_row(row_dict) + if ( + not hmac.compare_digest(request["bindingSha256"], expected_binding) + or not hmac.compare_digest(request["contentSha256"], expected_content) + or not hmac.compare_digest(request["ownerScopeSha256"], expected_owner_scope) + or request["targetSessionId"] != expected_session + ): + return None + request["ownerId"] = stored_owner + if str(row_dict.get("state") or "") != "imported": + raise NeutralV05ImportError("import_not_ready") + self._enforce_v05_materialization_inventory_budget( + conn, + normalized_import_id, + ) + rows = conn.execute( + "SELECT relative_path, payload_kind, payload_sha256, size_bytes, payload_bytes " + "FROM migration_v05_payloads WHERE import_id = ? ORDER BY relative_path", + (normalized_import_id,), + ).fetchall() + payloads = tuple( + { + "content": payload["payload_bytes"], + "kind": payload["payload_kind"], + "path": payload["relative_path"], + "sha256": payload["payload_sha256"], + "sizeBytes": payload["size_bytes"], + } + for payload in rows + ) + self._verify_v05_payload_inventory( + conn, + request, + payloads=payloads, + require_complete=True, + ) + status = self._v05_status_in_conn( + conn, + row_dict, + request, + payloads=payloads, + ) + normalized_request, normalized_payloads = normalize_import_bundle( + request, + list(payloads), + ) + normalized_receipt = validate_import_receipt( + status["receipt"], + normalized_request, + stored_payloads=len(normalized_payloads), + ) + return { + "payloads": normalized_payloads, + "receipt": normalized_receipt, + "request": normalized_request, + } + + @staticmethod + def _enforce_v05_materialization_inventory_budget( + conn: sqlite3.Connection, + import_id: str, + ) -> None: + inventory = conn.execute( + "SELECT COUNT(*) AS payload_count, " + "COALESCE(SUM(size_bytes), 0) AS declared_bytes, " + "COALESCE(MAX(size_bytes), 0) AS max_declared_bytes, " + "COALESCE(SUM(length(payload_bytes)), 0) AS actual_bytes, " + "COALESCE(MAX(length(payload_bytes)), 0) AS max_actual_bytes " + "FROM migration_v05_payloads WHERE import_id = ?", + (import_id,), + ).fetchone() + if not inventory: + raise NeutralV05ImportError("corrupt_import_payloads") + values = { + key: inventory[key] + for key in ( + "payload_count", + "declared_bytes", + "max_declared_bytes", + "actual_bytes", + "max_actual_bytes", + ) + } + if any( + not isinstance(value, int) or isinstance(value, bool) or value < 0 + for value in values.values() + ): + raise NeutralV05ImportError("corrupt_import_payloads") + if values["declared_bytes"] != values["actual_bytes"]: + raise NeutralV05ImportError("corrupt_import_payloads") + if values["max_declared_bytes"] != values["max_actual_bytes"]: + raise NeutralV05ImportError("corrupt_import_payloads") + if values["payload_count"] == 0: + raise NeutralV05ImportError("corrupt_import_payloads") + if ( + values["payload_count"] > MATERIALIZATION_MAX_PAYLOADS + or values["actual_bytes"] > MATERIALIZATION_MAX_PAYLOAD_BYTES + or values["max_actual_bytes"] > MATERIALIZATION_MAX_ENTRY_BYTES + ): + raise NeutralV05ImportError("materialization_budget_exceeded") + + def _prepare_v05_import( + self, + request: dict, + payloads: tuple[dict, ...], + ) -> dict: now = time.time() + session_id = str(request["targetSessionId"]) with self._conn() as conn: - row = conn.execute("SELECT * FROM sessions WHERE session_id = ?", (session_id,)).fetchone() - if row: - conn.execute("UPDATE sessions SET updated_at = ? WHERE session_id = ?", (now, session_id)) - return dict(row) + conn.execute("BEGIN IMMEDIATE") + lifecycle = conn.execute( + "SELECT state FROM session_lifecycle WHERE session_id = ?", + (session_id,), + ).fetchone() + session = conn.execute( + "SELECT owner_user_id FROM sessions WHERE session_id = ?", + (session_id,), + ).fetchone() + if lifecycle and str(lifecycle["state"] or "") != "active": + raise NeutralV05ImportError("session_fenced") + if session: + if lifecycle is None: + raise NeutralV05ImportError("corrupt_import_state") + if str(session["owner_user_id"] or "") != request["ownerId"]: + raise NeutralV05ImportError("owner_session_conflict") else: + if lifecycle is not None: + raise NeutralV05ImportError("session_fenced") + conn.execute( + "INSERT INTO session_lifecycle " + "(session_id, state, receipt_id, started_at, completed_at, errors_json, updated_at) " + "VALUES (?, 'active', '', 0, 0, '[]', ?)", + (session_id, now), + ) + conn.execute( + "INSERT INTO sessions " + "(session_id, owner_user_id, created_at, updated_at, metadata) " + "VALUES (?, ?, ?, ?, ?)", + ( + session_id, + request["ownerId"], + now, + now, + v05_canonical_json( + { + "contentSha256": request["contentSha256"], + "neutralV05Import": True, + } + ).decode("utf-8"), + ), + ) + + row = conn.execute( + "SELECT * FROM migration_v05_imports WHERE import_id = ?", + (request["importId"],), + ).fetchone() + if row: + return self._v05_status_in_conn(conn, dict(row), request, payloads=payloads) + conn.execute( + """ + INSERT INTO migration_v05_imports + (import_id, session_id, schema_version, state, version, attempt_count, + binding_sha256, archive_sha256, content_sha256, owner_scope_sha256, + manifest_sha256, stage_receipt_sha256, counts_json, payload_count, + receipt_json, failure_code, created_at, updated_at) + VALUES (?, ?, ?, 'validated_pending_import', 0, 0, ?, ?, ?, ?, ?, ?, ?, 0, '', '', ?, ?) + """, + ( + request["importId"], + session_id, + request["schemaVersion"], + request["bindingSha256"], + request["archiveSha256"], + request["contentSha256"], + request["ownerScopeSha256"], + request["manifestSha256"], + request["stageReceiptSha256"], + v05_canonical_json(request["counts"]).decode("utf-8"), + now, + now, + ), + ) + inserted = conn.execute( + "SELECT * FROM migration_v05_imports WHERE import_id = ?", + (request["importId"],), + ).fetchone() + if not inserted: + raise NeutralV05ImportError("storage_write_failed") + return self._v05_status_in_conn(conn, dict(inserted), request, payloads=payloads) + + def _claim_v05_import( + self, + request: dict, + payloads: tuple[dict, ...], + ) -> dict: + with self._conn() as conn: + conn.execute("BEGIN IMMEDIATE") + self._require_v05_import_session(conn, request) + row = conn.execute( + "SELECT * FROM migration_v05_imports WHERE import_id = ?", + (request["importId"],), + ).fetchone() + if not row: + raise NeutralV05ImportError("corrupt_import_state") + row_dict = dict(row) + ensure_row_binding(row_dict, request) + state = str(row_dict["state"] or "") + if state == "imported": + return self._v05_status_in_conn(conn, row_dict, request, payloads=payloads) + if state == "importing": + return self._v05_status_in_conn(conn, row_dict, request, payloads=payloads) + if state not in {"validated_pending_import", "failed_retryable"}: + raise NeutralV05ImportError("corrupt_import_state") + version = int(row_dict["version"]) + result = conn.execute( + "UPDATE migration_v05_imports " + "SET state = 'importing', version = version + 1, " + "attempt_count = attempt_count + 1, failure_code = '', updated_at = ? " + "WHERE import_id = ? AND binding_sha256 = ? AND version = ? AND state = ?", + ( + time.time(), + request["importId"], + request["bindingSha256"], + version, + state, + ), + ) + if result.rowcount != 1: + raise NeutralV05ImportError("import_cas_conflict") + claimed = conn.execute( + "SELECT * FROM migration_v05_imports WHERE import_id = ?", + (request["importId"],), + ).fetchone() + if not claimed: + raise NeutralV05ImportError("corrupt_import_state") + return self._v05_status_in_conn(conn, dict(claimed), request, payloads=payloads) + + def _complete_v05_import( + self, + request: dict, + payloads: tuple[dict, ...], + *, + expected_version: int, + ) -> dict: + with self._conn() as conn: + conn.execute("BEGIN IMMEDIATE") + self._require_v05_import_session(conn, request) + row = conn.execute( + "SELECT * FROM migration_v05_imports WHERE import_id = ?", + (request["importId"],), + ).fetchone() + if not row: + raise NeutralV05ImportError("corrupt_import_state") + row_dict = dict(row) + ensure_row_binding(row_dict, request) + if str(row_dict["state"] or "") == "imported": + return self._v05_status_in_conn(conn, row_dict, request, payloads=payloads) + if str(row_dict["state"] or "") != "importing" or int(row_dict["version"]) != expected_version: + raise NeutralV05ImportError("import_cas_conflict") + existing = conn.execute( + "SELECT COUNT(*) FROM migration_v05_payloads WHERE import_id = ?", + (request["importId"],), + ).fetchone()[0] + if int(existing) != 0: + raise NeutralV05ImportError("corrupt_import_state") + for payload in payloads: + conn.execute( + """ + INSERT INTO migration_v05_payloads + (import_id, session_id, relative_path, payload_kind, + payload_sha256, size_bytes, payload_bytes) + VALUES (?, ?, ?, ?, ?, ?, ?) + """, + ( + request["importId"], + request["targetSessionId"], + payload["path"], + payload["kind"], + payload["sha256"], + payload["sizeBytes"], + payload["content"], + ), + ) + self._verify_v05_payload_inventory( + conn, + request, + payloads=payloads, + require_complete=True, + ) + receipt = build_import_receipt(request, stored_payloads=len(payloads)) + result = conn.execute( + "UPDATE migration_v05_imports " + "SET state = 'imported', version = version + 1, payload_count = ?, " + "receipt_json = ?, failure_code = '', updated_at = ? " + "WHERE import_id = ? AND binding_sha256 = ? AND version = ? AND state = 'importing'", + ( + len(payloads), + v05_canonical_json(receipt).decode("utf-8"), + time.time(), + request["importId"], + request["bindingSha256"], + expected_version, + ), + ) + if result.rowcount != 1: + raise NeutralV05ImportError("import_cas_conflict") + completed = conn.execute( + "SELECT * FROM migration_v05_imports WHERE import_id = ?", + (request["importId"],), + ).fetchone() + if not completed: + raise NeutralV05ImportError("corrupt_import_state") + return self._v05_status_in_conn(conn, dict(completed), request, payloads=payloads) + + def _mark_v05_import_retryable_failure( + self, + request: dict, + *, + expected_version: int, + failure_code: str, + ) -> None: + try: + with self._conn() as conn: + conn.execute("BEGIN IMMEDIATE") conn.execute( - "INSERT INTO sessions (session_id, created_at, updated_at) VALUES (?, ?, ?)", - (session_id, now, now) + "UPDATE migration_v05_imports " + "SET state = 'failed_retryable', version = version + 1, " + "failure_code = ?, updated_at = ? " + "WHERE import_id = ? AND binding_sha256 = ? " + "AND state = 'importing' AND version = ?", + ( + failure_code, + time.time(), + request["importId"], + request["bindingSha256"], + expected_version, + ), ) - return {"session_id": session_id, "created_at": now, "updated_at": now} + except Exception: + return + + def _require_v05_import_session(self, conn: sqlite3.Connection, request: dict) -> None: + row = conn.execute( + "SELECT session.owner_user_id, lifecycle.state " + "FROM sessions AS session " + "JOIN session_lifecycle AS lifecycle ON lifecycle.session_id = session.session_id " + "WHERE session.session_id = ?", + (request["targetSessionId"],), + ).fetchone() + if not row or str(row["state"] or "") != "active": + raise NeutralV05ImportError("session_fenced") + if str(row["owner_user_id"] or "") != request["ownerId"]: + raise NeutralV05ImportError("owner_session_conflict") + principal = current_principal() + if ( + principal.authenticated + and not principal.is_admin + and principal.user_id != request["ownerId"] + ): + raise NeutralV05ImportError("owner_scope_forbidden") + + def _v05_status_in_conn( + self, + conn: sqlite3.Connection, + row: dict, + request: dict, + *, + payloads: tuple[dict, ...] | None = None, + ) -> dict: + state = str(row.get("state") or "") + if state not in NEUTRAL_IMPORT_STATES: + raise NeutralV05ImportError("corrupt_import_state") + stored_payloads = self._verify_v05_payload_inventory( + conn, + request, + payloads=payloads if state == "imported" else None, + require_complete=state == "imported", + ) + if state != "imported" and stored_payloads != 0: + raise NeutralV05ImportError("corrupt_import_state") + payload_count = row.get("payload_count") + if ( + not isinstance(payload_count, int) + or isinstance(payload_count, bool) + or payload_count < 0 + ): + raise NeutralV05ImportError("corrupt_import_state") + if state == "imported" and payload_count != stored_payloads: + raise NeutralV05ImportError("corrupt_import_state") + return public_import_status( + row, + request, + stored_payloads=stored_payloads, + ) - def list_sessions(self, limit: int = 20) -> list[dict]: + @staticmethod + def _verify_v05_payload_inventory( + conn: sqlite3.Connection, + request: dict, + *, + payloads: tuple[dict, ...] | None, + require_complete: bool = False, + ) -> int: + rows = conn.execute( + "SELECT session_id, relative_path, payload_kind, payload_sha256, " + "size_bytes, payload_bytes FROM migration_v05_payloads " + "WHERE import_id = ? ORDER BY relative_path", + (request["importId"],), + ).fetchall() + if payloads is not None and len(rows) != len(payloads): + raise NeutralV05ImportError("corrupt_import_payloads") + if not rows: + if require_complete: + raise NeutralV05ImportError("corrupt_import_payloads") + return 0 + descriptors: list[dict] = [] + actual_counts = {"documents": 0, "sessions": 0, "workflows": 0} + for index, raw_row in enumerate(rows): + row = dict(raw_row) + content = row.get("payload_bytes") + size_bytes = row.get("size_bytes") + session_id = row.get("session_id") + payload_kind = row.get("payload_kind") + relative_path = row.get("relative_path") + payload_sha256 = row.get("payload_sha256") + if ( + not isinstance(content, bytes) + or not isinstance(size_bytes, int) + or isinstance(size_bytes, bool) + or not isinstance(session_id, str) + or not isinstance(payload_kind, str) + or payload_kind not in {"blob", "document", "session", "workflow_recipe"} + or not isinstance(relative_path, str) + or not relative_path + or not isinstance(payload_sha256, str) + or session_id != request["targetSessionId"] + or size_bytes != len(content) + or payload_sha256 != sha256_bytes(content) + ): + raise NeutralV05ImportError("corrupt_import_payloads") + descriptor = { + "kind": payload_kind, + "path": relative_path, + "sha256": payload_sha256, + "size_bytes": size_bytes, + } + descriptors.append(descriptor) + if descriptor["kind"] == "document": + actual_counts["documents"] += 1 + elif descriptor["kind"] == "session": + actual_counts["sessions"] += 1 + elif descriptor["kind"] == "workflow_recipe": + actual_counts["workflows"] += 1 + if payloads is not None: + expected = payloads[index] + if ( + descriptor["path"] != expected["path"] + or descriptor["kind"] != expected["kind"] + or descriptor["sha256"] != expected["sha256"] + or descriptor["size_bytes"] != expected["sizeBytes"] + or content != expected["content"] + ): + raise NeutralV05ImportError("corrupt_import_payloads") + if sha256_bytes(v05_canonical_json(descriptors, newline=True)) != request[ + "contentSha256" + ] or actual_counts != request["counts"]: + raise NeutralV05ImportError("corrupt_import_payloads") + return len(rows) + + @staticmethod + def _v05_request_from_row(row: dict) -> dict: + try: + counts = json.loads(str(row["counts_json"] or "")) + except (TypeError, ValueError) as exc: + raise NeutralV05ImportError("corrupt_import_state") from exc + request = { + "archiveSha256": str(row["archive_sha256"] or ""), + "bindingSha256": str(row["binding_sha256"] or ""), + "contentSha256": str(row["content_sha256"] or ""), + "counts": counts, + "importId": str(row["import_id"] or ""), + "manifestSha256": str(row["manifest_sha256"] or ""), + "ownerScopeSha256": str(row["owner_scope_sha256"] or ""), + "schemaVersion": str(row["schema_version"] or ""), + "stageReceiptSha256": str(row["stage_receipt_sha256"] or ""), + "targetSessionId": str(row["session_id"] or ""), + } + if ( + request["importId"] != neutral_import_id(request["contentSha256"]) + or request["targetSessionId"] != neutral_session_id(request["contentSha256"]) + ): + raise NeutralV05ImportError("corrupt_import_state") + if v05_binding_sha256(request) != request["bindingSha256"]: + raise NeutralV05ImportError("corrupt_import_state") + ensure_row_binding(row, request) + return request + + # ── Async Workbook Jobs ──────────────────────────────────────── + + def save_workbook_job(self, job: dict) -> str: + """Upsert an async workbook job envelope.""" + run_id = str(job["run_id"]) + record = dict(job) + principal = current_principal() + record.setdefault("account_id", "local") + record.setdefault("session_id", "") + if principal.authenticated and not principal.is_admin: + if not principal.user_id or not record["session_id"]: + raise OwnershipNotFoundError("Workbook job not found") + record["owner_user_id"] = principal.user_id + record["account_id"] = principal.user_id + elif principal.authenticated: + record.setdefault("owner_user_id", principal.user_id) + else: + record.setdefault("owner_user_id", "") + record.setdefault("command", "workbook") + record.setdefault("status", "queued") + record.setdefault("created_at", time.time()) + record["updated_at"] = float(record.get("updated_at") or time.time()) + result = record.pop("result", None) + session_id = str(record.get("session_id") or "").strip() + connection_context = self._session_write_conn(session_id) if session_id else self._conn() + with connection_context as conn: + conn.execute( + """ + INSERT INTO background_jobs + (job_id, job_type, status, request_id, session_id, owner_user_id, + payload_json, result_json, error, created_at, updated_at) + VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?) + ON CONFLICT(job_id) DO UPDATE SET + status = excluded.status, + request_id = excluded.request_id, + session_id = excluded.session_id, + owner_user_id = excluded.owner_user_id, + payload_json = excluded.payload_json, + result_json = excluded.result_json, + error = excluded.error, + updated_at = excluded.updated_at + """, + ( + run_id, + "workbook_job", + str(record.get("status") or "queued"), + str(record.get("account_id") or "local"), + str(record.get("session_id") or ""), + str(record.get("owner_user_id") or ""), + json.dumps(record), + json.dumps(result if result is not None else {}), + str(record.get("error") or ""), + float(record.get("created_at") or time.time()), + float(record.get("updated_at") or time.time()), + ), + ) + return run_id + + def get_workbook_job(self, run_id: str) -> Optional[dict]: + with self._read_conn() as conn: + row = conn.execute( + "SELECT * FROM background_jobs WHERE job_type = ? AND job_id = ?", + ("workbook_job", run_id), + ).fetchone() + if not row: + return None + row_session_id = str(row["session_id"] or "") + if row_session_id and not self._session_visible_in_conn(conn, row_session_id): + return None + principal = current_principal() + if principal.authenticated and not principal.is_admin: + if not principal.user_id or str(row["owner_user_id"] or "") != principal.user_id: + return None + return self._workbook_job_from_row(row) + + def list_workbook_jobs( + self, + limit: int = 50, + status: Optional[str] = None, + session_id: Optional[str] = None, + account_id: Optional[str] = None, + ) -> list[dict]: + clauses = ["job_type = ?"] + params: list[object] = ["workbook_job"] + principal = current_principal() + if principal.authenticated and not principal.is_admin: + if not principal.user_id: + return [] + clauses.append("owner_user_id = ?") + params.append(principal.user_id) + account_id = None + if status: + clauses.append("status = ?") + params.append(status) + if session_id: + clauses.append("session_id = ?") + params.append(session_id) + if account_id: + clauses.append("request_id = ?") + params.append(account_id) + params.append(max(limit, 0)) + with self._read_conn() as conn: + rows = conn.execute( + f""" + SELECT * + FROM background_jobs + WHERE {' AND '.join(clauses)} + ORDER BY created_at DESC + LIMIT ? + """, + params, + ).fetchall() + return [ + self._workbook_job_from_row(row) + for row in rows + if not str(row["session_id"] or "") + or self._session_visible_in_conn(conn, str(row["session_id"] or "")) + ] + + @staticmethod + def _workbook_job_from_row(row: sqlite3.Row) -> dict: + record = json.loads(row["payload_json"] or "{}") + record["run_id"] = row["job_id"] + record["status"] = row["status"] + record["account_id"] = row["request_id"] or record.get("account_id") or "local" + record["session_id"] = row["session_id"] or record.get("session_id") or "" + record["owner_user_id"] = row["owner_user_id"] or record.get("owner_user_id") or "" + record["created_at"] = row["created_at"] + record["updated_at"] = row["updated_at"] + record["error"] = row["error"] or record.get("error", "") + result = json.loads(row["result_json"] or "{}") + if row["status"] == "completed" or result: + record["result"] = result + return record + + # ── Session Management ─────────────────────────────────────── + + def get_or_create_session(self, session_id: str, owner_user_id: Optional[str] = None) -> dict: + """Get or create a session.""" + normalized_session_id = str(session_id or "").strip() + if not normalized_session_id: + raise OwnershipNotFoundError("Session not found") + from backend.services.lifecycle import get_lifecycle_coordinator + + if get_lifecycle_coordinator().writes_revoked(normalized_session_id): + raise OwnershipNotFoundError("Session not found") + principal = current_principal() + if principal.authenticated and not principal.is_admin: + if not principal.user_id: + raise OwnershipNotFoundError("Session not found") + owner_user_id = principal.user_id + elif owner_user_id is None and principal.authenticated: + owner_user_id = principal.user_id or None + now = time.time() + with self._session_lock(normalized_session_id): + with self._conn() as conn: + conn.execute("BEGIN IMMEDIATE") + lifecycle = conn.execute( + "SELECT state FROM session_lifecycle WHERE session_id = ?", + (normalized_session_id,), + ).fetchone() + if lifecycle and str(lifecycle["state"] or "") != "active": + raise OwnershipNotFoundError("Session not found") + row = conn.execute( + "SELECT * FROM sessions WHERE session_id = ?", + (normalized_session_id,), + ).fetchone() + if row: + existing_owner = str(row["owner_user_id"] or "") + if principal.authenticated and not principal.is_admin and existing_owner != principal.user_id: + raise OwnershipNotFoundError("Session not found") + if owner_user_id and existing_owner and existing_owner != owner_user_id and not principal.is_admin: + raise OwnershipNotFoundError("Session not found") + if lifecycle is None: + conn.execute( + "INSERT INTO session_lifecycle " + "(session_id, state, receipt_id, started_at, completed_at, errors_json, updated_at) " + "VALUES (?, 'active', '', 0, 0, '[]', ?)", + (normalized_session_id, now), + ) + conn.execute( + "UPDATE sessions SET updated_at = ? WHERE session_id = ?", + (now, normalized_session_id), + ) + return dict(row) + + conn.execute( + "INSERT INTO session_lifecycle " + "(session_id, state, receipt_id, started_at, completed_at, errors_json, updated_at) " + "VALUES (?, 'active', '', 0, 0, '[]', ?)", + (normalized_session_id, now), + ) + conn.execute( + "INSERT INTO sessions (session_id, owner_user_id, created_at, updated_at) VALUES (?, ?, ?, ?)", + (normalized_session_id, owner_user_id, now, now), + ) + return { + "session_id": normalized_session_id, + "owner_user_id": owner_user_id, + "created_at": now, + "updated_at": now, + } + + @contextmanager + def _session_lock(self, session_id: str): + key = (str(self.db_path.resolve()), str(session_id or "").strip()) + with _SESSION_LOCKS_GUARD: + lock = _SESSION_LOCKS.setdefault(key, threading.RLock()) + with lock: + yield + + def _require_session_write(self, conn: sqlite3.Connection, session_id: str) -> None: + normalized_session_id = str(session_id or "").strip() + if not normalized_session_id: + raise OwnershipNotFoundError("Session not found") + row = conn.execute( + "SELECT s.owner_user_id, lifecycle.state AS lifecycle_state " + "FROM sessions AS s " + "JOIN session_lifecycle AS lifecycle ON lifecycle.session_id = s.session_id " + "WHERE s.session_id = ?", + (normalized_session_id,), + ).fetchone() + if not row or str(row["lifecycle_state"] or "") != "active": + raise OwnershipNotFoundError("Session not found") + principal = current_principal() + existing_owner = str(row["owner_user_id"] or "") + if ( + principal.authenticated + and not principal.is_admin + and (not principal.user_id or existing_owner != principal.user_id) + ): + raise OwnershipNotFoundError("Session not found") + + @contextmanager + def _session_write_conn(self, session_id: str): + normalized_session_id = str(session_id or "").strip() + with self._session_lock(normalized_session_id): + with self._conn() as conn: + conn.execute("BEGIN IMMEDIATE") + self._require_session_write(conn, normalized_session_id) + yield conn + + @contextmanager + def _read_conn(self): + """Open a read whose authorization snapshot cannot race a purge commit.""" + with self._conn() as conn: + conn.execute("BEGIN IMMEDIATE") + yield conn + + def _session_visible_in_conn(self, conn: sqlite3.Connection, session_id: str) -> bool: + normalized_session_id = str(session_id or "").strip() + from backend.services.lifecycle import get_lifecycle_coordinator + + if get_lifecycle_coordinator().reads_revoked(normalized_session_id): + return False + row = conn.execute( + "SELECT s.owner_user_id FROM sessions AS s " + "JOIN session_lifecycle AS lifecycle ON lifecycle.session_id = s.session_id " + "WHERE s.session_id = ? AND lifecycle.state = 'active'", + (normalized_session_id,), + ).fetchone() + if not row: + return False + principal = current_principal() + if not principal.authenticated or principal.is_admin: + return True + return bool(principal.user_id) and str(row["owner_user_id"] or "") == principal.user_id + + @contextmanager + def _session_read_conn(self, session_id: str): + normalized_session_id = str(session_id or "").strip() + with self._session_lock(normalized_session_id): + with self._read_conn() as conn: + yield conn if self._session_visible_in_conn(conn, normalized_session_id) else None + + def get_session_lifecycle_state(self, session_id: str) -> Optional[str]: + with self._conn() as conn: + row = conn.execute( + "SELECT state FROM session_lifecycle WHERE session_id = ?", + (str(session_id or "").strip(),), + ).fetchone() + return str(row["state"]) if row else None + + def get_session_lifecycle_record(self, session_id: str) -> Optional[dict]: + with self._conn() as conn: + row = conn.execute( + "SELECT * FROM session_lifecycle WHERE session_id = ?", + (str(session_id or "").strip(),), + ).fetchone() + if not row: + return None + record = dict(row) + record["errors"] = json.loads(record.pop("errors_json") or "[]") + return record + + def set_session_lifecycle_state( + self, + session_id: str, + state: str, + *, + receipt_id: str, + started_at: float, + completed_at: float = 0.0, + errors: Optional[list[str]] = None, + ) -> None: + normalized_session_id = str(session_id or "").strip() + normalized_state = str(state or "").strip().lower() + normalized_receipt_id = str(receipt_id or "").strip() + if normalized_state not in {"purging", "verified", "failed"}: + raise ValueError("Lifecycle state must be purging, verified, or failed") + if not normalized_session_id or not normalized_receipt_id: + raise ValueError("session_id and receipt_id are required") + now = time.time() + with self._session_lock(normalized_session_id): + with self._conn() as conn: + conn.execute("BEGIN IMMEDIATE") + current = conn.execute( + "SELECT state, receipt_id FROM session_lifecycle WHERE session_id = ?", + (normalized_session_id,), + ).fetchone() + current_state = str(current["state"] or "") if current else "" + current_receipt_id = str(current["receipt_id"] or "") if current else "" + if normalized_state == "purging": + if current_state == "verified": + raise OwnershipNotFoundError("Session not found") + if current_state == "purging" and current_receipt_id not in {"", normalized_receipt_id}: + raise RuntimeError("Session purge is already in progress") + session_present = conn.execute( + "SELECT 1 FROM sessions WHERE session_id = ?", + (normalized_session_id,), + ).fetchone() + if not session_present and not current: + raise OwnershipNotFoundError("Session not found") + conn.execute( + "INSERT INTO session_lifecycle " + "(session_id, state, receipt_id, started_at, completed_at, errors_json, updated_at) " + "VALUES (?, 'purging', ?, ?, 0, '[]', ?) " + "ON CONFLICT(session_id) DO UPDATE SET " + "state = 'purging', receipt_id = excluded.receipt_id, " + "started_at = excluded.started_at, completed_at = 0, errors_json = '[]', " + "updated_at = excluded.updated_at", + (normalized_session_id, normalized_receipt_id, float(started_at), now), + ) + return + + if current_state != "purging" or current_receipt_id != normalized_receipt_id: + raise RuntimeError("Lifecycle completion does not match the active purge receipt") + if normalized_state == "verified" and self._session_has_durable_rows(conn, normalized_session_id): + raise RuntimeError("Cannot verify deletion while session-bound rows remain") + cursor = conn.execute( + "UPDATE session_lifecycle SET state = ?, completed_at = ?, errors_json = ?, updated_at = ? " + "WHERE session_id = ? AND state = 'purging' AND receipt_id = ?", + ( + normalized_state, + float(completed_at), + json.dumps(errors or []), + now, + normalized_session_id, + normalized_receipt_id, + ), + ) + if cursor.rowcount != 1: + raise RuntimeError("Lifecycle completion compare-and-set failed") + + def _session_has_durable_rows(self, conn: sqlite3.Connection, session_id: str) -> bool: + for table_name in SQLITE_SESSION_LIFECYCLE_TABLES: + row = conn.execute( + f"SELECT 1 FROM {table_name} WHERE session_id = ? LIMIT 1", # noqa: S608 -- static inventory + (session_id,), + ).fetchone() + if row: + return True + return False + + def session_deletion_verified(self, session_id: str) -> bool: + with self._conn() as conn: + return not self._session_has_durable_rows(conn, str(session_id or "").strip()) + + def get_session_owner(self, session_id: str) -> Optional[str]: + with self._conn() as conn: + row = conn.execute( + "SELECT owner_user_id FROM sessions WHERE session_id = ?", + (session_id,), + ).fetchone() + return str(row["owner_user_id"] or "") or None if row else None + + def session_exists(self, session_id: str) -> bool: + with self._conn() as conn: + row = conn.execute("SELECT 1 FROM sessions WHERE session_id = ?", (session_id,)).fetchone() + return row is not None + + def list_session_ids_before(self, cutoff_timestamp: float) -> list[str]: + with self._conn() as conn: + rows = conn.execute( + "SELECT s.session_id FROM sessions AS s " + "JOIN session_lifecycle AS lifecycle ON lifecycle.session_id = s.session_id " + "WHERE s.updated_at < ? AND lifecycle.state <> 'verified' ORDER BY s.updated_at", + (cutoff_timestamp,), + ).fetchall() + return [str(row["session_id"]) for row in rows] + + def list_document_file_paths(self, session_id: str) -> list[str]: + with self._conn() as conn: + rows = conn.execute( + "SELECT file_path FROM documents WHERE session_id = ? AND file_path <> ''", + (session_id,), + ).fetchall() + return [str(row["file_path"]) for row in rows if row["file_path"]] + + def _session_is_visible(self, session_id: str) -> bool: + with self._session_read_conn(session_id) as conn: + return conn is not None + + def list_sessions( + self, + limit: int = 20, + owner_user_id: Optional[str] = None, + ) -> list[dict]: """List recent sessions.""" + principal = current_principal() + if principal.authenticated and not principal.is_admin: + if not principal.user_id: + return [] + owner_user_id = principal.user_id + where = "WHERE lifecycle.state = 'active' " + if owner_user_id: + where += "AND s.owner_user_id = ? " + params: tuple[object, ...] = (owner_user_id, limit) if owner_user_id else (limit,) with self._conn() as conn: rows = conn.execute( "SELECT s.*, COUNT(c.id) as message_count FROM sessions s " + "JOIN session_lifecycle AS lifecycle ON lifecycle.session_id = s.session_id " "LEFT JOIN conversations c ON s.session_id = c.session_id " + f"{where}" "GROUP BY s.session_id ORDER BY s.updated_at DESC LIMIT ?", - (limit,) + params, ).fetchall() return [dict(r) for r in rows] @@ -307,9 +1380,8 @@ def list_sessions(self, limit: int = 20) -> list[dict]: def add_message(self, session_id: str, role: str, content: str, metadata: Optional[dict] = None): """Add a message to the conversation history.""" - self.get_or_create_session(session_id) now = time.time() - with self._conn() as conn: + with self._session_write_conn(session_id) as conn: conn.execute( "INSERT INTO conversations (session_id, role, content, timestamp, metadata) VALUES (?, ?, ?, ?, ?)", (session_id, role, content, now, json.dumps(metadata or {})) @@ -317,7 +1389,9 @@ def add_message(self, session_id: str, role: str, content: str, def get_conversation(self, session_id: str, limit: int = 50) -> list[ConversationMessage]: """Get the conversation history for a session (most recent N messages).""" - with self._conn() as conn: + with self._session_read_conn(session_id) as conn: + if conn is None: + return [] # Get the last N messages, then reverse to chronological order rows = conn.execute( "SELECT * FROM conversations WHERE session_id = ? ORDER BY timestamp DESC LIMIT ?", @@ -357,11 +1431,10 @@ def get_conversation_for_llm(self, session_id: str, max_messages: int = 10) -> l def record_task(self, session_id: str, task_id: str, prompt: str, mode: str, operations: list, summary: str, plan: Optional[list] = None, provider: str = "", - model: str = "", tokens: Optional[dict] = None): + model: str = "", tokens: Optional[dict] = None): """Record a completed task in the history.""" - self.get_or_create_session(session_id) now = time.time() - with self._conn() as conn: + with self._session_write_conn(session_id) as conn: conn.execute( "INSERT INTO task_history " "(session_id, task_id, prompt, mode, operations, summary, plan, provider, model, tokens_used, timestamp) " @@ -373,7 +1446,9 @@ def record_task(self, session_id: str, task_id: str, prompt: str, def get_task_history(self, session_id: str, limit: int = 20) -> list[dict]: """Get task history for a session.""" - with self._conn() as conn: + with self._session_read_conn(session_id) as conn: + if conn is None: + return [] rows = conn.execute( "SELECT * FROM task_history WHERE session_id = ? ORDER BY timestamp DESC LIMIT ?", (session_id, limit) @@ -389,14 +1464,25 @@ def get_task_history(self, session_id: str, limit: int = 20) -> list[dict]: def get_total_tokens(self, session_id: Optional[str] = None) -> dict: """Get total token usage (by session or global).""" - with self._conn() as conn: + if session_id: + read_context = self._session_read_conn(session_id) + else: + read_context = self._read_conn() + with read_context as conn: + if conn is None: + return {"prompt_tokens": 0, "completion_tokens": 0, "total_tokens": 0} if session_id: rows = conn.execute( "SELECT tokens_used FROM task_history WHERE session_id = ?", (session_id,) ).fetchall() else: - rows = conn.execute("SELECT tokens_used FROM task_history").fetchall() + rows = conn.execute( + "SELECT history.tokens_used FROM task_history AS history " + "JOIN sessions AS session ON session.session_id = history.session_id " + "JOIN session_lifecycle AS lifecycle ON lifecycle.session_id = history.session_id " + "WHERE lifecycle.state = 'active'" + ).fetchall() total = {"prompt_tokens": 0, "completion_tokens": 0} for r in rows: @@ -413,8 +1499,7 @@ def save_operation_plan(self, plan: dict): session_id = plan["session_id"] plan_id = plan["id"] now = time.time() - self.get_or_create_session(session_id) - with self._conn() as conn: + with self._session_write_conn(session_id) as conn: conn.execute( """ INSERT INTO operation_plans @@ -436,14 +1521,58 @@ def save_operation_plan(self, plan: dict): ), ) + def compare_and_swap_operation_plan( + self, + plan_id: str, + expected_plan: dict, + updated_plan: dict, + ) -> bool: + """Replace one exact durable plan revision under the session write fence.""" + expected_id = str(expected_plan.get("id") or expected_plan.get("plan_id") or "") + updated_id = str(updated_plan.get("id") or updated_plan.get("plan_id") or "") + expected_session_id = str(expected_plan.get("session_id") or "") + updated_session_id = str(updated_plan.get("session_id") or "") + if expected_id != plan_id or updated_id != plan_id: + raise ValueError("Operation plan identity is immutable") + if not expected_session_id or updated_session_id != expected_session_id: + raise ValueError("Operation plan session ownership is immutable") + + with self._session_write_conn(expected_session_id) as conn: + row = conn.execute( + "SELECT session_id, plan_json FROM operation_plans WHERE plan_id = ?", + (plan_id,), + ).fetchone() + if not row: + return False + if str(row["session_id"] or "") != expected_session_id: + raise ValueError("Operation plan session ownership is immutable") + current_json = str(row["plan_json"] or "") + if json.loads(current_json) != expected_plan: + return False + result = conn.execute( + "UPDATE operation_plans SET status = ?, plan_json = ?, updated_at = ? " + "WHERE plan_id = ? AND session_id = ? AND plan_json = ?", + ( + updated_plan.get("status", "draft"), + json.dumps(updated_plan), + time.time(), + plan_id, + expected_session_id, + current_json, + ), + ) + return result.rowcount == 1 + def get_operation_plan(self, plan_id: str) -> Optional[dict]: - with self._conn() as conn: + with self._read_conn() as conn: row = conn.execute( - "SELECT plan_json FROM operation_plans WHERE plan_id = ?", + "SELECT session_id, plan_json FROM operation_plans WHERE plan_id = ?", (plan_id,), ).fetchone() if not row: return None + if not self._session_visible_in_conn(conn, str(row["session_id"] or "")): + return None return json.loads(row["plan_json"]) def update_operation_plan_status(self, plan_id: str, status: str) -> Optional[dict]: @@ -452,7 +1581,7 @@ def update_operation_plan_status(self, plan_id: str, status: str) -> Optional[di return None plan["status"] = status plan["updated_at"] = time.time() - with self._conn() as conn: + with self._session_write_conn(str(plan.get("session_id") or "")) as conn: conn.execute( "UPDATE operation_plans SET status = ?, plan_json = ?, updated_at = ? WHERE plan_id = ?", (status, json.dumps(plan), plan["updated_at"], plan_id), @@ -467,27 +1596,29 @@ def record_snapshot( label: str = "", ) -> str: snapshot_id = f"snap_{int(time.time() * 1000)}_{uuid.uuid4().hex[:6]}" - self.get_or_create_session(session_id) - with self._conn() as conn: + state_json = json.dumps(workbook_state or {}, sort_keys=True, separators=(",", ":")) + state_hash = hashlib.sha256(state_json.encode("utf-8")).hexdigest() + with self._session_write_conn(session_id) as conn: conn.execute( """ INSERT INTO workbook_snapshots - (snapshot_id, session_id, plan_id, label, workbook_state, created_at) - VALUES (?, ?, ?, ?, ?, ?) + (snapshot_id, session_id, plan_id, label, workbook_state, state_hash, created_at) + VALUES (?, ?, ?, ?, ?, ?, ?) """, ( snapshot_id, session_id, plan_id, label, - json.dumps(workbook_state or {}), + state_json, + state_hash, time.time(), ), ) return snapshot_id def get_snapshot(self, snapshot_id: str) -> Optional[dict]: - with self._conn() as conn: + with self._read_conn() as conn: row = conn.execute( "SELECT * FROM workbook_snapshots WHERE snapshot_id = ?", (snapshot_id,), @@ -495,11 +1626,35 @@ def get_snapshot(self, snapshot_id: str) -> Optional[dict]: if not row: return None data = dict(row) - data["workbook_state"] = json.loads(data["workbook_state"] or "{}") + if not self._session_visible_in_conn(conn, str(data.get("session_id") or "")): + return None + raw_state = data.get("workbook_state") or "{}" + stored_hash = str(data.get("state_hash") or "") + data["workbook_state"] = json.loads(raw_state) + # Verify integrity: recompute from the canonical-serialized payload. + recomputed = hashlib.sha256( + json.dumps(data["workbook_state"], sort_keys=True, separators=(",", ":")).encode("utf-8") + ).hexdigest() + if not stored_hash: + # Pre-migration row — mark as legacy (no hash to verify against). + data["state_verified"] = None + data["state_hash"] = "" + else: + data["state_verified"] = stored_hash == recomputed + data["state_hash"] = stored_hash + if not data["state_verified"]: + logger.warning( + "snapshot_integrity_mismatch", + snapshot_id=snapshot_id, + expected=stored_hash, + recomputed=recomputed, + ) return data def list_snapshots(self, session_id: str, limit: int = 20) -> list[dict]: - with self._conn() as conn: + with self._session_read_conn(session_id) as conn: + if conn is None: + return [] rows = conn.execute( "SELECT * FROM workbook_snapshots WHERE session_id = ? ORDER BY created_at DESC LIMIT ?", (session_id, limit), @@ -520,8 +1675,7 @@ def record_audit_event( snapshot_id: Optional[str] = None, ) -> str: audit_id = f"audit_{int(time.time() * 1000)}_{uuid.uuid4().hex[:6]}" - self.get_or_create_session(session_id) - with self._conn() as conn: + with self._session_write_conn(session_id) as conn: conn.execute( """ INSERT INTO audit_events @@ -540,9 +1694,73 @@ def record_audit_event( ) return audit_id + def record_enterprise_admin_audit( + self, + event_type: str, + payload: Optional[dict] = None, + ) -> str: + principal = current_principal() + if principal.authenticated and not principal.is_admin: + raise OwnershipNotFoundError("Audit stream not found") + normalized_type = str(event_type or "").strip() + if not normalized_type.startswith("enterprise_admin."): + raise ValueError("Enterprise administrator audit type is required") + audit_id = f"admin_audit_{int(time.time() * 1000)}_{uuid.uuid4().hex[:8]}" + with self._conn() as conn: + conn.execute( + """ + INSERT INTO enterprise_admin_audits + (audit_id, event_type, payload_json, created_at) + VALUES (?, ?, ?, ?) + """, + (audit_id, normalized_type, json.dumps(payload or {}), time.time()), + ) + return audit_id + + def list_enterprise_admin_audits(self, limit: int = 1000) -> list[dict]: + principal = current_principal() + if principal.authenticated and not principal.is_admin: + raise OwnershipNotFoundError("Audit stream not found") + with self._conn() as conn: + rows = conn.execute( + "SELECT * FROM enterprise_admin_audits " + "ORDER BY created_at DESC LIMIT ?", + (max(int(limit), 0),), + ).fetchall() + return [ + { + "audit_id": row["audit_id"], + "event_type": row["event_type"], + "payload": json.loads(row["payload_json"] or "{}"), + "created_at": row["created_at"], + } + for row in rows + ] + def get_timeline(self, session_id: str, limit: int = 50) -> dict: """Return durable conversation, plan, snapshot, and audit history.""" - with self._conn() as conn: + empty_timeline = { + "session_id": session_id, + "messages": [], + "plans": [], + "snapshots": [], + "audit_events": [], + "workbook_maps": [], + "verification_runs": [], + "agent_runs": [], + "source_evidence_graphs": [], + "operation_simulations": [], + "documents": [], + "citations": [], + } + with self._session_read_conn(session_id) as conn: + if conn is None: + return empty_timeline + messages = conn.execute( + "SELECT * FROM conversations WHERE session_id = ? " + "ORDER BY timestamp DESC LIMIT ?", + (session_id, limit), + ).fetchall() plans = conn.execute( "SELECT * FROM operation_plans WHERE session_id = ? ORDER BY updated_at DESC LIMIT ?", (session_id, limit), @@ -581,12 +1799,25 @@ def get_timeline(self, session_id: str, limit: int = 50) -> dict: "FROM operation_simulations WHERE session_id = ? ORDER BY created_at DESC LIMIT ?", (session_id, limit), ).fetchall() + documents = conn.execute( + "SELECT * FROM documents WHERE session_id = ? ORDER BY created_at DESC LIMIT ?", + (session_id, limit), + ).fetchall() + citations = conn.execute( + "SELECT * FROM citations WHERE session_id = ? ORDER BY created_at DESC LIMIT ?", + (session_id, limit), + ).fetchall() return { "session_id": session_id, "messages": [ - {"role": m.role, "content": m.content, "timestamp": m.timestamp, "metadata": m.metadata} - for m in self.get_conversation(session_id, limit) + { + "role": row["role"], + "content": row["content"], + "timestamp": row["timestamp"], + "metadata": json.loads(row["metadata"] or "{}"), + } + for row in reversed(messages) ], "plans": [ { @@ -624,8 +1855,21 @@ def get_timeline(self, session_id: str, limit: int = 50) -> dict: for row in evidence_graphs ], "operation_simulations": [dict(row) for row in simulations], - "documents": self.list_documents(session_id, limit), - "citations": self.get_citations(session_id, limit=limit), + "documents": [ + { + **{k: row[k] for k in row.keys() if k not in {"summary_json", "parsed_json"}}, + "summary": json.loads(row["summary_json"] or "{}"), + "parsed": json.loads(row["parsed_json"] or "{}"), + } + for row in documents + ], + "citations": [ + { + **{k: row[k] for k in row.keys() if k != "citation_json"}, + "citation": json.loads(row["citation_json"] or "{}"), + } + for row in citations + ], } # ── Plan Engine V3 Artifacts ─────────────────────────────────── @@ -634,8 +1878,7 @@ def save_workbook_map(self, semantic_map: dict) -> str: map_id = semantic_map["mapId"] session_id = semantic_map["sessionId"] now = time.time() - self.get_or_create_session(session_id) - with self._conn() as conn: + with self._session_write_conn(session_id) as conn: conn.execute( """ INSERT INTO workbook_maps @@ -660,12 +1903,19 @@ def save_workbook_map(self, semantic_map: dict) -> str: return map_id def get_workbook_map(self, map_id: str) -> Optional[dict]: - with self._conn() as conn: - row = conn.execute("SELECT map_json FROM workbook_maps WHERE map_id = ?", (map_id,)).fetchone() - return json.loads(row["map_json"]) if row else None + with self._read_conn() as conn: + row = conn.execute( + "SELECT session_id, map_json FROM workbook_maps WHERE map_id = ?", + (map_id,), + ).fetchone() + if row and not self._session_visible_in_conn(conn, str(row["session_id"] or "")): + return None + return json.loads(row["map_json"]) if row else None def get_latest_workbook_map(self, session_id: str, workbook_fingerprint: str = "") -> Optional[dict]: - with self._conn() as conn: + with self._session_read_conn(session_id) as conn: + if conn is None: + return None if workbook_fingerprint: row = conn.execute( "SELECT map_json FROM workbook_maps WHERE session_id = ? AND workbook_fingerprint = ? ORDER BY updated_at DESC LIMIT 1", @@ -680,7 +1930,11 @@ def get_latest_workbook_map(self, session_id: str, workbook_fingerprint: str = " def save_plan_revision(self, plan_id: str, revision_type: str, reason: str, plan: dict) -> str: revision_id = f"rev_{int(time.time() * 1000)}_{uuid.uuid4().hex[:6]}" - with self._conn() as conn: + parent = self.get_operation_plan(plan_id) + if not parent: + raise OwnershipNotFoundError("Operation plan not found") + session_id = str(parent.get("session_id") or "") + with self._session_write_conn(session_id) as conn: conn.execute( """ INSERT INTO plan_revisions @@ -692,25 +1946,30 @@ def save_plan_revision(self, plan_id: str, revision_type: str, reason: str, plan return revision_id def list_plan_revisions(self, plan_id: str, limit: int = 50) -> list[dict]: - with self._conn() as conn: + with self._read_conn() as conn: + plan = conn.execute( + "SELECT session_id FROM operation_plans WHERE plan_id = ?", + (plan_id,), + ).fetchone() + if not plan or not self._session_visible_in_conn(conn, str(plan["session_id"] or "")): + return [] rows = conn.execute( "SELECT * FROM plan_revisions WHERE plan_id = ? ORDER BY created_at ASC LIMIT ?", (plan_id, limit), ).fetchall() - return [ - { - **{k: row[k] for k in row.keys() if k != "plan_json"}, - "plan": json.loads(row["plan_json"] or "{}"), - } - for row in rows - ] + return [ + { + **{k: row[k] for k in row.keys() if k != "plan_json"}, + "plan": json.loads(row["plan_json"] or "{}"), + } + for row in rows + ] def save_operation_audits(self, session_id: str, plan_id: str, audits: list[dict]) -> int: if not audits: return 0 - self.get_or_create_session(session_id) now = time.time() - with self._conn() as conn: + with self._session_write_conn(session_id) as conn: for audit in audits: conn.execute( """ @@ -740,8 +1999,7 @@ def save_operation_audits(self, session_id: str, plan_id: str, audits: list[dict def save_verification_run(self, session_id: str, plan_id: str, status: str, report: dict) -> str: verification_id = f"verify_{int(time.time() * 1000)}_{uuid.uuid4().hex[:6]}" - self.get_or_create_session(session_id) - with self._conn() as conn: + with self._session_write_conn(session_id) as conn: conn.execute( """ INSERT INTO verification_runs @@ -758,8 +2016,7 @@ def save_source_evidence_graph(self, graph: dict) -> str: graph_id = graph["graphId"] session_id = graph.get("sessionId", "") now = time.time() - self.get_or_create_session(session_id) - with self._conn() as conn: + with self._session_write_conn(session_id) as conn: conn.execute( """ INSERT INTO source_evidence_graphs @@ -782,9 +2039,14 @@ def save_source_evidence_graph(self, graph: dict) -> str: return graph_id def get_source_evidence_graph(self, graph_id: str) -> Optional[dict]: - with self._conn() as conn: - row = conn.execute("SELECT graph_json FROM source_evidence_graphs WHERE graph_id = ?", (graph_id,)).fetchone() - return json.loads(row["graph_json"]) if row else None + with self._read_conn() as conn: + row = conn.execute( + "SELECT session_id, graph_json FROM source_evidence_graphs WHERE graph_id = ?", + (graph_id,), + ).fetchone() + if row and not self._session_visible_in_conn(conn, str(row["session_id"] or "")): + return None + return json.loads(row["graph_json"]) if row else None def save_operation_simulation( self, @@ -795,8 +2057,7 @@ def save_operation_simulation( run_id: str = "", ) -> str: simulation_id = simulation["simulationId"] - self.get_or_create_session(session_id) - with self._conn() as conn: + with self._session_write_conn(session_id) as conn: conn.execute( """ INSERT INTO operation_simulations @@ -816,9 +2077,14 @@ def save_operation_simulation( return simulation_id def get_operation_simulation(self, simulation_id: str) -> Optional[dict]: - with self._conn() as conn: - row = conn.execute("SELECT simulation_json FROM operation_simulations WHERE simulation_id = ?", (simulation_id,)).fetchone() - return json.loads(row["simulation_json"]) if row else None + with self._read_conn() as conn: + row = conn.execute( + "SELECT session_id, simulation_json FROM operation_simulations WHERE simulation_id = ?", + (simulation_id,), + ).fetchone() + if row and not self._session_visible_in_conn(conn, str(row["session_id"] or "")): + return None + return json.loads(row["simulation_json"]) if row else None def save_workbook_invariant_run( self, @@ -829,8 +2095,7 @@ def save_workbook_invariant_run( plan_id: str = "", ) -> str: invariant_run_id = f"invariant_{int(time.time() * 1000)}_{uuid.uuid4().hex[:6]}" - self.get_or_create_session(session_id) - with self._conn() as conn: + with self._session_write_conn(session_id) as conn: conn.execute( """ INSERT INTO workbook_invariant_runs @@ -853,8 +2118,7 @@ def save_agent_run(self, run: dict) -> str: run_id = run["runId"] session_id = run["sessionId"] now = time.time() - self.get_or_create_session(session_id) - with self._conn() as conn: + with self._session_write_conn(session_id) as conn: conn.execute( """ INSERT INTO agent_runs @@ -879,12 +2143,295 @@ def save_agent_run(self, run: dict) -> str: return run_id def get_agent_run(self, run_id: str) -> Optional[dict]: - with self._conn() as conn: - row = conn.execute("SELECT run_json FROM agent_runs WHERE run_id = ?", (run_id,)).fetchone() - return json.loads(row["run_json"]) if row else None + with self._read_conn() as conn: + row = conn.execute( + "SELECT session_id, run_json FROM agent_runs WHERE run_id = ?", + (run_id,), + ).fetchone() + if row and not self._session_visible_in_conn(conn, str(row["session_id"] or "")): + return None + return json.loads(row["run_json"]) if row else None + + def append_agent_run_event( + self, + run_id: str, + *, + expected_sequence: int, + event_type: str, + event_payload: dict, + updated_run: dict, + idempotency_key: str = "", + ) -> dict: + """Atomically compare the run sequence, replace its snapshot, and append one event.""" + normalized_run_id = str(run_id or "").strip() + normalized_type = str(event_type or "").strip() + normalized_key = str(idempotency_key or "").strip() + if not normalized_run_id or not normalized_type: + raise ValueError("run_id and event_type are required") + if int(expected_sequence) < 0: + raise ValueError("expected_sequence must be non-negative") + + with self._read_conn() as conn: + owner_row = conn.execute( + "SELECT session_id FROM agent_runs WHERE run_id = ?", + (normalized_run_id,), + ).fetchone() + if not owner_row: + return {"outcome": "not_found", "run": None, "event": None} + session_id = str(owner_row["session_id"] or "") + + with self._session_write_conn(session_id) as conn: + row = conn.execute( + "SELECT session_id, run_json FROM agent_runs WHERE run_id = ?", + (normalized_run_id,), + ).fetchone() + if not row: + return {"outcome": "not_found", "run": None, "event": None} + current_run = json.loads(row["run_json"] or "{}") + event_rows = conn.execute( + "SELECT artifact_json FROM agent_run_artifacts " + "WHERE run_id = ? AND artifact_type = 'control_event'", + (normalized_run_id,), + ).fetchall() + events = [json.loads(item["artifact_json"] or "{}") for item in event_rows] + if normalized_key: + replay = next( + ( + item + for item in events + if str(item.get("idempotencyKey") or "") == normalized_key + ), + None, + ) + if replay: + if str(replay.get("eventType") or "") != normalized_type: + return { + "outcome": "idempotency_conflict", + "run": current_run, + "event": replay, + } + return {"outcome": "replayed", "run": current_run, "event": replay} + + metadata = current_run.get("metadata") + metadata = dict(metadata) if isinstance(metadata, dict) else {} + control = metadata.get("controlPlane") + control = dict(control) if isinstance(control, dict) else {} + stored_sequence = max( + int(control.get("eventSequence") or 0), + max((int(item.get("sequence") or 0) for item in events), default=0), + ) + if stored_sequence != int(expected_sequence): + return { + "outcome": "sequence_conflict", + "run": current_run, + "event": None, + "currentSequence": stored_sequence, + } + if str(current_run.get("status") or "") in {"completed", "failed", "cancelled"}: + return { + "outcome": "terminal_fence", + "run": current_run, + "event": None, + "currentSequence": stored_sequence, + } + + next_run = json.loads(json.dumps(updated_run, default=str)) + next_run_id = str(next_run.get("runId") or next_run.get("run_id") or "") + next_session_id = str(next_run.get("sessionId") or next_run.get("session_id") or "") + if next_run_id != normalized_run_id or next_session_id != session_id: + raise ValueError("Agent run identity and session ownership are immutable") + sequence = stored_sequence + 1 + occurred_at = time.time() + control.update( + { + "schemaVersion": "2026-08-agent-run-control-v1", + "eventSequence": sequence, + "cursor": str(sequence), + "updatedAt": occurred_at, + } + ) + next_metadata = next_run.get("metadata") + next_metadata = dict(next_metadata) if isinstance(next_metadata, dict) else {} + next_metadata["controlPlane"] = control + next_run["metadata"] = next_metadata + next_run["updatedAt"] = occurred_at + event_id = "run_event_" + hashlib.sha256( + f"{normalized_run_id}:{sequence}:{normalized_type}:{normalized_key}".encode("utf-8") + ).hexdigest()[:24] + event = { + "schemaVersion": "2026-08-agent-run-event-v1", + "eventId": event_id, + "runId": normalized_run_id, + "sequence": sequence, + "cursor": str(sequence), + "eventType": normalized_type, + "fromStatus": str(current_run.get("status") or ""), + "toStatus": str(next_run.get("status") or ""), + "payload": dict(event_payload or {}), + "idempotencyKey": normalized_key, + "occurredAt": occurred_at, + } + current_json = str(row["run_json"] or "{}") + serialized_run = json.dumps(next_run) + update = conn.execute( + "UPDATE agent_runs SET status = ?, prompt = ?, run_json = ?, updated_at = ? " + "WHERE run_id = ? AND session_id = ? AND run_json = ?", + ( + str(next_run.get("status") or "created"), + str(next_run.get("prompt") or ""), + serialized_run, + occurred_at, + normalized_run_id, + session_id, + current_json, + ), + ) + if update.rowcount != 1: + return { + "outcome": "sequence_conflict", + "run": current_run, + "event": None, + "currentSequence": stored_sequence, + } + conn.execute( + "INSERT INTO agent_run_artifacts " + "(artifact_id, run_id, session_id, artifact_type, artifact_json, created_at) " + "VALUES (?, ?, ?, 'control_event', ?, ?)", + ( + event_id, + normalized_run_id, + session_id, + json.dumps(event), + occurred_at, + ), + ) + return {"outcome": "appended", "run": next_run, "event": event} + + def list_agent_run_events( + self, + run_id: str, + after_sequence: int = 0, + limit: int = 100, + ) -> dict: + normalized_run_id = str(run_id or "").strip() + after = max(int(after_sequence), 0) + bounded_limit = min(max(int(limit), 1), 100) + with self._read_conn() as conn: + run = conn.execute( + "SELECT session_id FROM agent_runs WHERE run_id = ?", + (normalized_run_id,), + ).fetchone() + if not run or not self._session_visible_in_conn(conn, str(run["session_id"] or "")): + return { + "events": [], + "nextCursor": str(after), + "latestSequence": after, + "hasMore": False, + } + rows = conn.execute( + "SELECT artifact_json FROM agent_run_artifacts " + "WHERE run_id = ? AND artifact_type = 'control_event'", + (normalized_run_id,), + ).fetchall() + all_events = sorted( + (json.loads(row["artifact_json"] or "{}") for row in rows), + key=lambda item: int(item.get("sequence") or 0), + ) + latest = max((int(item.get("sequence") or 0) for item in all_events), default=after) + remaining = [item for item in all_events if int(item.get("sequence") or 0) > after] + page = remaining[:bounded_limit] + next_cursor = str(int(page[-1].get("sequence") or after)) if page else str(after) + return { + "events": page, + "nextCursor": next_cursor, + "latestSequence": latest, + "hasMore": len(remaining) > len(page), + } + + def save_agent_egress_preview(self, session_id: str, preview: dict) -> str: + approval_id = str(preview.get("approvalId") or "").strip() + preview_session_id = str(preview.get("sessionId") or "").strip() + if not approval_id or preview_session_id != str(session_id or "").strip(): + raise ValueError("Egress preview identity and session are required") + with self._session_write_conn(session_id) as conn: + existing = conn.execute( + "SELECT artifact_json FROM agent_run_artifacts " + "WHERE artifact_id = ? AND artifact_type = 'egress_preview'", + (approval_id,), + ).fetchone() + if existing: + current = json.loads(existing["artifact_json"] or "{}") + if current != preview: + raise ValueError("Egress approval identity is immutable") + return approval_id + conn.execute( + "INSERT INTO agent_run_artifacts " + "(artifact_id, run_id, session_id, artifact_type, artifact_json, created_at) " + "VALUES (?, ?, ?, 'egress_preview', ?, ?)", + ( + approval_id, + f"egress:{approval_id}", + session_id, + json.dumps(preview), + float(preview.get("issuedAt") or time.time()), + ), + ) + return approval_id + + def consume_agent_egress_approval( + self, + session_id: str, + *, + approval_id: str, + approved_digest: str, + request_digest: str, + consumption_id: str, + now: Optional[float] = None, + ) -> dict: + consumed_at = float(time.time() if now is None else now) + with self._session_write_conn(session_id) as conn: + row = conn.execute( + "SELECT artifact_json FROM agent_run_artifacts " + "WHERE artifact_id = ? AND session_id = ? AND artifact_type = 'egress_preview'", + (approval_id, session_id), + ).fetchone() + if not row: + return {"outcome": "not_found", "preview": None} + raw_preview = str(row["artifact_json"] or "{}") + preview = json.loads(raw_preview) + if not hmac.compare_digest( + str(preview.get("requestDigest") or ""), + str(request_digest or ""), + ) or not hmac.compare_digest( + str(preview.get("approvalDigest") or ""), + str(approved_digest or ""), + ): + return {"outcome": "digest_mismatch", "preview": preview} + if consumed_at > float(preview.get("expiresAt") or 0): + return {"outcome": "expired", "preview": preview} + if preview.get("consumedAt") is not None: + return {"outcome": "already_consumed", "preview": preview} + preview["consumedAt"] = consumed_at + preview["consumptionId"] = str(consumption_id or "") + result = conn.execute( + "UPDATE agent_run_artifacts SET artifact_json = ? " + "WHERE artifact_id = ? AND session_id = ? " + "AND artifact_type = 'egress_preview' AND artifact_json = ?", + ( + json.dumps(preview), + approval_id, + session_id, + raw_preview, + ), + ) + if result.rowcount != 1: + return {"outcome": "already_consumed", "preview": preview} + return {"outcome": "consumed", "preview": preview} def list_agent_runs(self, session_id: str, limit: int = 50) -> list[dict]: - with self._conn() as conn: + with self._session_read_conn(session_id) as conn: + if conn is None: + return [] rows = conn.execute( "SELECT run_json FROM agent_runs WHERE session_id = ? ORDER BY updated_at DESC LIMIT ?", (session_id, limit), @@ -897,8 +2444,7 @@ def save_workbook_truth_model(self, truth_model: dict) -> str: truth_id = truth_model["truthModelId"] session_id = truth_model.get("sessionId", "") now = time.time() - self.get_or_create_session(session_id) - with self._conn() as conn: + with self._session_write_conn(session_id) as conn: conn.execute( """ INSERT INTO workbook_truth_models @@ -923,18 +2469,19 @@ def save_workbook_truth_model(self, truth_model: dict) -> str: return truth_id def get_workbook_truth_model(self, truth_model_id: str) -> Optional[dict]: - with self._conn() as conn: + with self._read_conn() as conn: row = conn.execute( - "SELECT truth_json FROM workbook_truth_models WHERE truth_model_id = ?", + "SELECT session_id, truth_json FROM workbook_truth_models WHERE truth_model_id = ?", (truth_model_id,), ).fetchone() - return json.loads(row["truth_json"]) if row else None + if row and not self._session_visible_in_conn(conn, str(row["session_id"] or "")): + return None + return json.loads(row["truth_json"]) if row else None def save_formula_graph_artifact(self, graph: dict) -> str: graph_id = graph["graphId"] session_id = graph.get("sessionId", "") - self.get_or_create_session(session_id) - with self._conn() as conn: + with self._session_write_conn(session_id) as conn: conn.execute( """ INSERT INTO formula_graph_artifacts @@ -957,9 +2504,14 @@ def save_formula_graph_artifact(self, graph: dict) -> str: return graph_id def get_formula_graph_artifact(self, graph_id: str) -> Optional[dict]: - with self._conn() as conn: - row = conn.execute("SELECT graph_json FROM formula_graph_artifacts WHERE graph_id = ?", (graph_id,)).fetchone() - return json.loads(row["graph_json"]) if row else None + with self._read_conn() as conn: + row = conn.execute( + "SELECT session_id, graph_json FROM formula_graph_artifacts WHERE graph_id = ?", + (graph_id,), + ).fetchone() + if row and not self._session_visible_in_conn(conn, str(row["session_id"] or "")): + return None + return json.loads(row["graph_json"]) if row else None def save_operation_simulation_v2( self, @@ -970,8 +2522,7 @@ def save_operation_simulation_v2( run_id: str = "", ) -> str: simulation_id = simulation["simulationId"] - self.get_or_create_session(session_id) - with self._conn() as conn: + with self._session_write_conn(session_id) as conn: conn.execute( """ INSERT INTO operation_simulations_v2 @@ -1001,8 +2552,7 @@ def save_v7_workbook_review(self, review: dict) -> str: review_id = review["reviewId"] session_id = review.get("sessionId", "") now = time.time() - self.get_or_create_session(session_id) - with self._conn() as conn: + with self._session_write_conn(session_id) as conn: conn.execute( """ INSERT INTO v7_workbook_reviews @@ -1031,14 +2581,18 @@ def save_v7_workbook_review(self, review: dict) -> str: return review_id def get_v7_workbook_review(self, review_id: str) -> Optional[dict]: - with self._conn() as conn: - row = conn.execute("SELECT review_json FROM v7_workbook_reviews WHERE review_id = ?", (review_id,)).fetchone() - return json.loads(row["review_json"]) if row else None + with self._read_conn() as conn: + row = conn.execute( + "SELECT session_id, review_json FROM v7_workbook_reviews WHERE review_id = ?", + (review_id,), + ).fetchone() + if row and not self._session_visible_in_conn(conn, str(row["session_id"] or "")): + return None + return json.loads(row["review_json"]) if row else None def save_local_replay_artifact(self, session_id: str, artifact_type: str, payload: dict) -> str: replay_id = f"replay_{int(time.time() * 1000)}_{uuid.uuid4().hex[:6]}" - self.get_or_create_session(session_id) - with self._conn() as conn: + with self._session_write_conn(session_id) as conn: conn.execute( """ INSERT INTO local_replay_artifacts @@ -1050,7 +2604,7 @@ def save_local_replay_artifact(self, session_id: str, artifact_type: str, payloa return replay_id def list_local_replay_artifacts(self, limit: int = 100) -> list[dict]: - with self._conn() as conn: + with self._read_conn() as conn: rows = conn.execute( """ SELECT replay_id, session_id, artifact_type, anonymized_payload_json, created_at @@ -1060,19 +2614,20 @@ def list_local_replay_artifacts(self, limit: int = 100) -> list[dict]: """, (limit,), ).fetchall() - return [ - { - "replay_id": row["replay_id"], - "session_id": row["session_id"], - "artifact_type": row["artifact_type"], - "payload": json.loads(row["anonymized_payload_json"] or "{}"), - "created_at": row["created_at"], - } - for row in rows - ] + return [ + { + "replay_id": row["replay_id"], + "session_id": row["session_id"], + "artifact_type": row["artifact_type"], + "payload": json.loads(row["anonymized_payload_json"] or "{}"), + "created_at": row["created_at"], + } + for row in rows + if self._session_visible_in_conn(conn, str(row["session_id"] or "")) + ] def clear_local_replay_artifacts(self, session_id: str) -> int: - with self._conn() as conn: + with self._session_write_conn(session_id) as conn: cursor = conn.execute("DELETE FROM local_replay_artifacts WHERE session_id = ?", (session_id,)) return int(cursor.rowcount or 0) @@ -1080,8 +2635,7 @@ def save_local_replay_baseline(self, baseline: dict) -> str: baseline_id = baseline["baselineId"] now = float(baseline.get("createdAt") or time.time()) session_id = str(baseline.get("sessionId") or "local-replay-baseline") - self.get_or_create_session(session_id) - with self._conn() as conn: + with self._session_write_conn(session_id) as conn: conn.execute( """ INSERT INTO local_replay_baselines @@ -1093,7 +2647,8 @@ def save_local_replay_baseline(self, baseline: dict) -> str: evaluation_id = excluded.evaluation_id, replay_count = excluded.replay_count, drift_severity = excluded.drift_severity, - baseline_json = excluded.baseline_json + baseline_json = excluded.baseline_json, + created_at = excluded.created_at """, ( baseline_id, @@ -1109,17 +2664,21 @@ def save_local_replay_baseline(self, baseline: dict) -> str: return baseline_id def list_local_replay_baselines(self, limit: int = 20) -> list[dict]: - with self._conn() as conn: + with self._read_conn() as conn: rows = conn.execute( """ - SELECT baseline_json + SELECT session_id, baseline_json FROM local_replay_baselines ORDER BY created_at DESC LIMIT ? """, (limit,), ).fetchall() - return [json.loads(row["baseline_json"] or "{}") for row in rows] + return [ + json.loads(row["baseline_json"] or "{}") + for row in rows + if self._session_visible_in_conn(conn, str(row["session_id"] or "")) + ] def save_v7_default_readiness_report(self, report: dict) -> str: readiness_id = report["readinessId"] @@ -1177,8 +2736,7 @@ def list_v7_default_readiness_reports(self, limit: int = 20) -> list[dict]: def save_rollback_snapshot_v2(self, snapshot: dict) -> str: snapshot_id = snapshot["snapshotId"] session_id = snapshot.get("sessionId", "") - self.get_or_create_session(session_id) - with self._conn() as conn: + with self._session_write_conn(session_id) as conn: conn.execute( """ INSERT INTO rollback_snapshots_v2 @@ -1203,8 +2761,7 @@ def save_brain_run(self, run: dict) -> str: run_id = run["runId"] session_id = run.get("sessionId", "") now = time.time() - self.get_or_create_session(session_id) - with self._conn() as conn: + with self._session_write_conn(session_id) as conn: conn.execute( """ INSERT INTO agent_run_artifacts @@ -1222,19 +2779,23 @@ def save_brain_run(self, run: dict) -> str: return run_id def get_brain_run(self, run_id: str) -> Optional[dict]: - with self._conn() as conn: + with self._read_conn() as conn: row = conn.execute( """ - SELECT artifact_json + SELECT session_id, artifact_json FROM agent_run_artifacts WHERE artifact_id = ? AND artifact_type = 'brain_run' """, (run_id,), ).fetchone() - return json.loads(row["artifact_json"] or "{}") if row else None + if row and not self._session_visible_in_conn(conn, str(row["session_id"] or "")): + return None + return json.loads(row["artifact_json"] or "{}") if row else None def list_brain_runs(self, session_id: str, limit: int = 20) -> list[dict]: - with self._conn() as conn: + with self._session_read_conn(session_id) as conn: + if conn is None: + return [] rows = conn.execute( """ SELECT artifact_json @@ -1252,8 +2813,7 @@ def save_brain_artifact_bundle(self, bundle: dict) -> str: session_id = bundle.get("sessionId", "") run_id = (bundle.get("metadata") or {}).get("runId") or f"brain_{bundle.get('planId', '')}" now = time.time() - self.get_or_create_session(session_id) - with self._conn() as conn: + with self._session_write_conn(session_id) as conn: conn.execute( """ INSERT INTO agent_run_artifacts @@ -1271,21 +2831,22 @@ def save_brain_artifact_bundle(self, bundle: dict) -> str: return bundle_id def get_brain_artifact_bundle(self, bundle_id: str) -> Optional[dict]: - with self._conn() as conn: + with self._read_conn() as conn: row = conn.execute( """ - SELECT artifact_json + SELECT session_id, artifact_json FROM agent_run_artifacts WHERE artifact_id = ? AND artifact_type = 'brain_artifact_bundle' """, (bundle_id,), ).fetchone() - return json.loads(row["artifact_json"] or "{}") if row else None + if row and not self._session_visible_in_conn(conn, str(row["session_id"] or "")): + return None + return json.loads(row["artifact_json"] or "{}") if row else None def save_agent_run_artifact(self, run_id: str, session_id: str, artifact_type: str, artifact: dict) -> str: artifact_id = f"artifact_{artifact_type}_{int(time.time() * 1000)}_{uuid.uuid4().hex[:6]}" - self.get_or_create_session(session_id) - with self._conn() as conn: + with self._session_write_conn(session_id) as conn: conn.execute( """ INSERT INTO agent_run_artifacts @@ -1297,7 +2858,13 @@ def save_agent_run_artifact(self, run_id: str, session_id: str, artifact_type: s return artifact_id def list_agent_run_artifacts(self, run_id: str, limit: int = 100) -> list[dict]: - with self._conn() as conn: + with self._read_conn() as conn: + run = conn.execute( + "SELECT session_id FROM agent_runs WHERE run_id = ?", + (run_id,), + ).fetchone() + if not run or not self._session_visible_in_conn(conn, str(run["session_id"] or "")): + return [] rows = conn.execute( """ SELECT artifact_id, artifact_type, artifact_json, created_at @@ -1308,21 +2875,20 @@ def list_agent_run_artifacts(self, run_id: str, limit: int = 100) -> list[dict]: """, (run_id, limit), ).fetchall() - return [ - { - "artifact_id": row["artifact_id"], - "artifact_type": row["artifact_type"], - "created_at": row["created_at"], - "artifact": json.loads(row["artifact_json"] or "{}"), - } - for row in rows - ] + return [ + { + "artifact_id": row["artifact_id"], + "artifact_type": row["artifact_type"], + "created_at": row["created_at"], + "artifact": json.loads(row["artifact_json"] or "{}"), + } + for row in rows + ] def save_repair_plan_v2(self, session_id: str, repair_plan: dict) -> str: repair_id = repair_plan["repairId"] now = time.time() - self.get_or_create_session(session_id) - with self._conn() as conn: + with self._session_write_conn(session_id) as conn: conn.execute( """ INSERT INTO repair_plans_v2 @@ -1349,16 +2915,20 @@ def save_repair_plan_v2(self, session_id: str, repair_plan: dict) -> str: return repair_id def get_repair_plan_v2(self, repair_id: str) -> Optional[dict]: - with self._conn() as conn: - row = conn.execute("SELECT repair_json FROM repair_plans_v2 WHERE repair_id = ?", (repair_id,)).fetchone() - return json.loads(row["repair_json"]) if row else None + with self._read_conn() as conn: + row = conn.execute( + "SELECT session_id, repair_json FROM repair_plans_v2 WHERE repair_id = ?", + (repair_id,), + ).fetchone() + if row and not self._session_visible_in_conn(conn, str(row["session_id"] or "")): + return None + return json.loads(row["repair_json"]) if row else None def save_verification_report_v2(self, session_id: str, verification_report: dict) -> str: verification_id = verification_report.get("verificationId") or verification_report.get("reportId") or f"verify_{int(time.time() * 1000)}_{uuid.uuid4().hex[:6]}" verification_report.setdefault("verificationId", verification_id) now = time.time() - self.get_or_create_session(session_id) - with self._conn() as conn: + with self._session_write_conn(session_id) as conn: conn.execute( """ INSERT INTO verification_reports_v2 @@ -1410,8 +2980,7 @@ def save_agent_run_step_artifact_v3(self, run_id: str, session_id: str, step_id: artifact.setdefault("artifactId", artifact_id) artifact.setdefault("runId", run_id) artifact.setdefault("stepId", step_id) - self.get_or_create_session(session_id) - with self._conn() as conn: + with self._session_write_conn(session_id) as conn: conn.execute( """ INSERT INTO agent_run_artifacts @@ -1424,8 +2993,7 @@ def save_agent_run_step_artifact_v3(self, run_id: str, session_id: str, step_id: def save_operation_diff_preview_v2(self, session_id: str, diff_preview: dict) -> str: preview_id = diff_preview.get("diffPreviewId") or diff_preview.get("operationId") or f"diff_{int(time.time() * 1000)}_{uuid.uuid4().hex[:6]}" - self.get_or_create_session(session_id) - with self._conn() as conn: + with self._session_write_conn(session_id) as conn: conn.execute( """ INSERT INTO operation_diff_previews_v2 @@ -1448,9 +3016,8 @@ def save_operation_diff_preview_v2(self, session_id: str, diff_preview: dict) -> def save_source_cell_provenance_v2(self, session_id: str, provenance: dict) -> str: provenance_id = provenance.get("provenanceId") or f"prov_{int(time.time() * 1000)}_{uuid.uuid4().hex[:6]}" - self.get_or_create_session(session_id) now = time.time() - with self._conn() as conn: + with self._session_write_conn(session_id) as conn: conn.execute( """ INSERT INTO source_cell_provenance_v2 @@ -1485,9 +3052,8 @@ def save_document( parsed: Optional[dict] = None, document_id: Optional[str] = None, ) -> str: - self.get_or_create_session(session_id) doc_id = document_id or f"doc_{uuid.uuid4().hex[:12]}" - with self._conn() as conn: + with self._session_write_conn(session_id) as conn: conn.execute( """ INSERT INTO documents @@ -1549,7 +3115,7 @@ def _replace_document_chunks( ) def get_document(self, document_id: str) -> Optional[dict]: - with self._conn() as conn: + with self._read_conn() as conn: row = conn.execute( "SELECT * FROM documents WHERE document_id = ?", (document_id,), @@ -1557,12 +3123,16 @@ def get_document(self, document_id: str) -> Optional[dict]: if not row: return None data = dict(row) + if not self._session_visible_in_conn(conn, str(data.get("session_id") or "")): + return None data["summary"] = json.loads(data.pop("summary_json") or "{}") data["parsed"] = json.loads(data.pop("parsed_json") or "{}") return data def list_documents(self, session_id: str, limit: int = 20) -> list[dict]: - with self._conn() as conn: + with self._session_read_conn(session_id) as conn: + if conn is None: + return [] rows = conn.execute( "SELECT * FROM documents WHERE session_id = ? ORDER BY created_at DESC LIMIT ?", (session_id, limit), @@ -1583,9 +3153,8 @@ def save_citations( ) -> int: if not citations: return 0 - self.get_or_create_session(session_id) now = time.time() - with self._conn() as conn: + with self._session_write_conn(session_id) as conn: for citation in citations: conn.execute( """ @@ -1610,7 +3179,9 @@ def get_citations( plan_id: Optional[str] = None, limit: int = 200, ) -> list[dict]: - with self._conn() as conn: + with self._session_read_conn(session_id) as conn: + if conn is None: + return [] if plan_id: rows = conn.execute( "SELECT * FROM citations WHERE session_id = ? AND plan_id = ? ORDER BY created_at DESC LIMIT ?", @@ -1641,9 +3212,21 @@ def record_apply_attempt( workbook_fingerprint: str = "", payload: Optional[dict] = None, ) -> str: - attempt_id = f"apply_{int(time.time() * 1000)}_{uuid.uuid4().hex[:6]}" - self.get_or_create_session(session_id) - with self._conn() as conn: + # Dedupe: if there is an open (un-completed) attempt for this plan/fingerprint, + # return its attempt_id instead of inserting a new row. This prevents + # network-retried /tasks/apply calls from producing parallel attempts. + with self._session_write_conn(session_id) as conn: + existing = conn.execute( + """ + SELECT attempt_id, token_hash FROM apply_attempts + WHERE plan_id = ? AND completed_at IS NULL AND status = 'approved' + ORDER BY created_at DESC LIMIT 1 + """, + (plan_id,), + ).fetchone() + if existing and existing["token_hash"] == token_hash: + return existing["attempt_id"] + attempt_id = f"apply_{int(time.time() * 1000)}_{uuid.uuid4().hex[:6]}" conn.execute( """ INSERT INTO apply_attempts @@ -1665,16 +3248,18 @@ def record_apply_attempt( return attempt_id def get_latest_apply_attempt(self, plan_id: str) -> Optional[dict]: - with self._conn() as conn: + with self._read_conn() as conn: row = conn.execute( "SELECT * FROM apply_attempts WHERE plan_id = ? ORDER BY created_at DESC LIMIT 1", (plan_id,), ).fetchone() - if not row: - return None - data = dict(row) - data["payload"] = json.loads(data.pop("payload_json") or "{}") - return data + if not row: + return None + data = dict(row) + if not self._session_visible_in_conn(conn, str(data.get("session_id") or "")): + return None + data["payload"] = json.loads(data.pop("payload_json") or "{}") + return data def complete_apply_attempt( self, @@ -1687,9 +3272,16 @@ def complete_apply_attempt( attempt = self.get_latest_apply_attempt(plan_id) if not attempt or attempt["token_hash"] != token_hash: return None + # Token consumption: once an attempt has completed (success or failure), + # the token cannot be reused to flip status again. Returning the + # existing attempt without mutation makes /apply/result idempotent for + # the network-retry case. + if attempt.get("completed_at"): + attempt["token_consumed"] = True + return attempt existing_payload = attempt.get("payload", {}) existing_payload.update(payload or {}) - with self._conn() as conn: + with self._session_write_conn(str(attempt.get("session_id") or "")) as conn: conn.execute( """ UPDATE apply_attempts @@ -1701,6 +3293,7 @@ def complete_apply_attempt( attempt["status"] = status attempt["payload"] = existing_payload attempt["completed_at"] = time.time() + attempt["token_consumed"] = True return attempt def record_apply_result( @@ -1728,14 +3321,30 @@ def export_audit_csv(self, session_id: str) -> str: timeline = self.get_timeline(session_id, limit=1000) out = io.StringIO() writer = csv.writer(out) - writer.writerow(["created_at", "event_type", "plan_id", "snapshot_id", "payload"]) + writer.writerow([ + "created_at", + "event_type", + "plan_id", + "snapshot_id", + "payload", + "actor_user_id", + "actor_email", + "actor_role", + "actor_auth_source", + ]) for event in timeline["audit_events"]: + payload = event.get("payload", {}) + actor = payload.get("actor", {}) if isinstance(payload, dict) else {} writer.writerow([ event.get("created_at", ""), event.get("event_type", ""), event.get("plan_id", ""), event.get("snapshot_id", ""), - json.dumps(event.get("payload", {}), sort_keys=True), + json.dumps(payload, sort_keys=True), + actor.get("user_id", ""), + actor.get("email", ""), + actor.get("role", ""), + actor.get("auth_source", ""), ]) return out.getvalue() @@ -1799,38 +3408,79 @@ def upsert_enterprise_user( email: str, name: str, role: str, - status: str = "active", - mfa_enabled: bool = False, + status: Optional[str] = None, + mfa_enabled: Optional[bool] = None, metadata: Optional[dict] = None, + oidc_issuer: Optional[str] = None, + oidc_subject: Optional[str] = None, ) -> dict: now = time.time() with self._conn() as conn: - conn.execute( - """ - INSERT INTO enterprise_users - (user_id, email, name, role, status, mfa_enabled, metadata_json, created_at, updated_at) - VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?) - ON CONFLICT(user_id) DO UPDATE SET - email = excluded.email, - name = excluded.name, - role = excluded.role, - status = excluded.status, - mfa_enabled = excluded.mfa_enabled, - metadata_json = excluded.metadata_json, - updated_at = excluded.updated_at - """, - ( - user_id, - email, - name, - role, - status, - 1 if mfa_enabled else 0, - json.dumps(metadata or {}), - now, - now, - ), + # Serialize the read/merge/write sequence across threads and + # processes. Reading before BEGIN IMMEDIATE lets a stale profile + # PATCH overwrite a concurrent security-state update. + conn.execute("BEGIN IMMEDIATE") + row = conn.execute( + "SELECT * FROM enterprise_users WHERE user_id = ?", + (user_id,), + ).fetchone() + existing: Optional[dict] = None + if row: + existing = dict(row) + existing["mfa_enabled"] = bool(existing["mfa_enabled"]) + existing["metadata"] = json.loads(existing.pop("metadata_json") or "{}") + binding = prepare_enterprise_identity_binding( + status=status if status is not None else str((existing or {}).get("status") or "active"), + metadata=metadata, + oidc_issuer=oidc_issuer, + oidc_subject=oidc_subject, + existing=existing, ) + effective_mfa = bool((existing or {}).get("mfa_enabled")) if mfa_enabled is None else bool(mfa_enabled) + if binding.issuer: + conflict = conn.execute( + "SELECT user_id FROM enterprise_users " + "WHERE oidc_issuer = ? AND oidc_subject = ? AND user_id <> ?", + (binding.issuer, binding.subject, user_id), + ).fetchone() + if conflict: + raise ValueError("OIDC identity is already bound to another enterprise user") + try: + conn.execute( + """ + INSERT INTO enterprise_users + (user_id, email, name, role, status, mfa_enabled, metadata_json, + oidc_issuer, oidc_subject, created_at, updated_at) + VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?) + ON CONFLICT(user_id) DO UPDATE SET + email = excluded.email, + name = excluded.name, + role = excluded.role, + status = excluded.status, + mfa_enabled = excluded.mfa_enabled, + metadata_json = excluded.metadata_json, + oidc_issuer = excluded.oidc_issuer, + oidc_subject = excluded.oidc_subject, + updated_at = excluded.updated_at + """, + ( + user_id, + str(email).strip().lower(), + name, + role, + binding.status, + 1 if effective_mfa else 0, + json.dumps(binding.metadata), + binding.issuer, + binding.subject, + now, + now, + ), + ) + except sqlite3.IntegrityError as exc: + if "idx_enterprise_users_oidc_identity" in str(exc) or "oidc_issuer" in str(exc): + raise ValueError("OIDC identity is already bound to another enterprise user") from exc + raise return self.get_enterprise_user(user_id) or {} def get_enterprise_user(self, user_id: str) -> Optional[dict]: @@ -1841,11 +3491,25 @@ def get_enterprise_user(self, user_id: str) -> Optional[dict]: data = dict(row) data["mfa_enabled"] = bool(data["mfa_enabled"]) data["metadata"] = json.loads(data.pop("metadata_json") or "{}") + data["oidc_issuer"] = str(data.get("oidc_issuer") or "") + data["oidc_subject"] = str(data.get("oidc_subject") or "") return data def get_enterprise_user_by_email(self, email: str) -> Optional[dict]: with self._conn() as conn: - row = conn.execute("SELECT user_id FROM enterprise_users WHERE email = ?", (email,)).fetchone() + row = conn.execute( + "SELECT user_id FROM enterprise_users WHERE LOWER(email) = ?", + (str(email).strip().lower(),), + ).fetchone() + return self.get_enterprise_user(row["user_id"]) if row else None + + def get_enterprise_user_by_oidc_identity(self, issuer: str, subject: str) -> Optional[dict]: + with self._conn() as conn: + row = conn.execute( + "SELECT user_id FROM enterprise_users " + "WHERE oidc_issuer = ? AND oidc_subject = ? AND LOWER(status) = 'active'", + (str(issuer).strip(), str(subject).strip()), + ).fetchone() return self.get_enterprise_user(row["user_id"]) if row else None def list_enterprise_users(self, limit: int = 100) -> list[dict]: @@ -1929,6 +3593,14 @@ def create_enterprise_api_key(self, user_id: str, label: str = "") -> dict: ) return {"key_id": key_id, "user_id": user_id, "api_key": raw_key, "label": label} + def revoke_enterprise_api_key(self, key_id: str) -> bool: + with self._conn() as conn: + cur = conn.execute( + "UPDATE enterprise_api_keys SET active = 0 WHERE key_id = ?", + (key_id,), + ) + return cur.rowcount > 0 + def get_enterprise_user_by_api_key(self, api_key: str) -> Optional[dict]: # Verify the new HMAC-SHA-256 form first; fall back to the legacy # unsalted SHA-256 form so keys minted before the upgrade keep working. @@ -1936,11 +3608,21 @@ def get_enterprise_user_by_api_key(self, api_key: str) -> Optional[dict]: candidates = _enterprise_api_key_candidates(api_key) with self._conn() as conn: row = conn.execute( - f"SELECT user_id, key_hash FROM enterprise_api_keys WHERE key_hash IN ({','.join('?' for _ in candidates)}) AND active = 1", + f""" + SELECT k.user_id, k.key_hash, u.status + FROM enterprise_api_keys k + JOIN enterprise_users u ON u.user_id = k.user_id + WHERE k.key_hash IN ({','.join('?' for _ in candidates)}) AND k.active = 1 + """, candidates, ).fetchone() if not row: return None + if not api_key_status_allows(row["status"]): + return None + user = self.get_enterprise_user(row["user_id"]) + if not _enterprise_user_allows_api_key(user, require_mfa=self._enterprise_mfa_required()): + return None preferred_hash = candidates[0] if row["key_hash"] != preferred_hash: conn.execute( @@ -1952,7 +3634,7 @@ def get_enterprise_user_by_api_key(self, api_key: str) -> Optional[dict]: "UPDATE enterprise_api_keys SET last_used_at = ? WHERE key_hash = ?", (time.time(), preferred_hash), ) - return self.get_enterprise_user(row["user_id"]) + return user def list_enterprise_api_keys(self, user_id: Optional[str] = None) -> list[dict]: with self._conn() as conn: @@ -1993,6 +3675,14 @@ def get_enterprise_config(self, key: str) -> dict: "updated_at": row["updated_at"], } + def _enterprise_mfa_required(self) -> bool: + for key in ("security_policy", "sso", "provider_policy"): + stored = self.get_enterprise_config(key) + config = stored.get("config") if isinstance(stored, dict) else {} + if isinstance(config, dict) and (config.get("require_mfa") or config.get("requireMfa")): + return True + return False + def enterprise_dashboard(self) -> dict: with self._conn() as conn: users = conn.execute("SELECT COUNT(*) AS c FROM enterprise_users").fetchone()["c"] @@ -2013,51 +3703,31 @@ def enterprise_dashboard(self) -> dict: # ── Retention ─────────────────────────────────────────────────── def delete_session(self, session_id: str) -> bool: - with self._conn() as conn: - exists = conn.execute("SELECT 1 FROM sessions WHERE session_id = ?", (session_id,)).fetchone() - if not exists: - return False - conn.execute("DELETE FROM local_replay_artifacts WHERE session_id = ?", (session_id,)) - conn.execute("DELETE FROM v7_workbook_reviews WHERE session_id = ?", (session_id,)) - conn.execute("DELETE FROM verification_reports_v2 WHERE session_id = ?", (session_id,)) - conn.execute("DELETE FROM repair_plans_v2 WHERE session_id = ?", (session_id,)) - conn.execute("DELETE FROM agent_run_artifacts WHERE session_id = ?", (session_id,)) - conn.execute("DELETE FROM rollback_snapshots_v2 WHERE session_id = ?", (session_id,)) - conn.execute("DELETE FROM operation_simulations_v2 WHERE session_id = ?", (session_id,)) - conn.execute("DELETE FROM formula_graph_artifacts WHERE session_id = ?", (session_id,)) - conn.execute("DELETE FROM workbook_truth_models WHERE session_id = ?", (session_id,)) - conn.execute("DELETE FROM workbook_invariant_runs WHERE session_id = ?", (session_id,)) - conn.execute("DELETE FROM operation_simulations WHERE session_id = ?", (session_id,)) - conn.execute("DELETE FROM source_evidence_graphs WHERE session_id = ?", (session_id,)) - conn.execute("DELETE FROM agent_runs WHERE session_id = ?", (session_id,)) - conn.execute("DELETE FROM metric_events WHERE session_id = ?", (session_id,)) - conn.execute("DELETE FROM verification_runs WHERE session_id = ?", (session_id,)) - conn.execute("DELETE FROM operation_audits WHERE session_id = ?", (session_id,)) - conn.execute("DELETE FROM workbook_maps WHERE session_id = ?", (session_id,)) - plan_rows = conn.execute("SELECT plan_id FROM operation_plans WHERE session_id = ?", (session_id,)).fetchall() - for row in plan_rows: - conn.execute("DELETE FROM plan_revisions WHERE plan_id = ?", (row["plan_id"],)) - conn.execute("DELETE FROM background_jobs WHERE session_id = ?", (session_id,)) - conn.execute("DELETE FROM document_chunks WHERE session_id = ?", (session_id,)) - conn.execute("DELETE FROM apply_attempts WHERE session_id = ?", (session_id,)) - conn.execute("DELETE FROM citations WHERE session_id = ?", (session_id,)) - conn.execute("DELETE FROM audit_events WHERE session_id = ?", (session_id,)) - conn.execute("DELETE FROM workbook_snapshots WHERE session_id = ?", (session_id,)) - conn.execute("DELETE FROM operation_plans WHERE session_id = ?", (session_id,)) - conn.execute("DELETE FROM task_history WHERE session_id = ?", (session_id,)) - conn.execute("DELETE FROM conversations WHERE session_id = ?", (session_id,)) - conn.execute("DELETE FROM documents WHERE session_id = ?", (session_id,)) - conn.execute("DELETE FROM sessions WHERE session_id = ?", (session_id,)) - return True + normalized_session_id = str(session_id or "").strip() + with self._session_lock(normalized_session_id): + with self._conn() as conn: + conn.execute("BEGIN IMMEDIATE") + lifecycle = conn.execute( + "SELECT state FROM session_lifecycle WHERE session_id = ?", + (normalized_session_id,), + ).fetchone() + if not lifecycle or str(lifecycle["state"] or "") != "purging": + raise RuntimeError("Session deletion requires a durable PURGING tombstone") + exists = conn.execute( + "SELECT 1 FROM sessions WHERE session_id = ?", + (normalized_session_id,), + ).fetchone() + purge_sqlite_session_rows(conn, normalized_session_id) + if self._session_has_durable_rows(conn, normalized_session_id): + raise RuntimeError("SQLite session deletion verification failed") + assert_sqlite_session_lifecycle_integrity(conn) + return bool(exists) or lifecycle is not None def purge_sessions_before(self, cutoff_timestamp: float) -> int: - with self._conn() as conn: - rows = conn.execute("SELECT session_id FROM sessions WHERE updated_at < ?", (cutoff_timestamp,)).fetchall() - count = 0 - for row in rows: - if self.delete_session(row["session_id"]): - count += 1 - return count + _ = cutoff_timestamp + raise RuntimeError( + "Direct SQLite retention deletion is disabled; use LifecycleCoordinator.purge_before" + ) def _document_chunks_from_parsed(parsed: dict) -> list[dict]: diff --git a/backend/storage/__init__.py b/backend/storage/__init__.py index 83bdcbb..15fcad3 100644 --- a/backend/storage/__init__.py +++ b/backend/storage/__init__.py @@ -5,10 +5,199 @@ """ import inspect +from typing import Any, cast from backend.storage.protocol import SessionStoreProtocol, ConversationMessage -__all__ = ["get_session_store", "SessionStoreProtocol", "ConversationMessage"] +__all__ = [ + "get_session_store", + "SessionStoreProtocol", + "ConversationMessage", + "postgres_protocol_gaps", + "postgres_protocol_gap_report", +] + +POSTGRES_PROTOCOL_GAP_CATEGORIES: tuple[dict[str, Any], ...] = ( + { + "id": "session_history", + "label": "Session, conversation, task, and token history", + "risk": "critical", + "action": "Keep core session history and token accounting callable before enabling DB_ENGINE=postgres.", + "methods": ( + "get_or_create_session", + "get_session_owner", + "session_exists", + "get_session_lifecycle_state", + "get_session_lifecycle_record", + "set_session_lifecycle_state", + "session_deletion_verified", + "list_session_ids_before", + "list_sessions", + "add_message", + "get_conversation", + "get_conversation_for_llm", + "record_task", + "get_task_history", + "get_total_tokens", + "get_migration_status", + ), + }, + { + "id": "operation_apply_audit", + "label": "Operation plan, apply, snapshot, audit, and timeline", + "risk": "critical", + "action": "Implement durable apply lifecycle and audit/timeline methods before Postgres can run production workflows.", + "methods": ( + "save_operation_plan", + "compare_and_swap_operation_plan", + "get_operation_plan", + "update_operation_plan_status", + "record_snapshot", + "get_snapshot", + "record_audit_event", + "export_audit_csv", + "record_apply_attempt", + "get_latest_apply_attempt", + "complete_apply_attempt", + "record_apply_result", + "get_timeline", + "delete_session", + ), + }, + { + "id": "documents_citations_benchmarks", + "label": "Documents, citations, and benchmark history", + "risk": "high", + "action": "Add document metadata, citation, and benchmark run persistence or keep those routes on SQLite.", + "methods": ( + "save_document", + "get_document", + "list_documents", + "list_document_file_paths", + "save_citations", + "get_citations", + "save_benchmark_run", + "list_benchmark_runs", + ), + }, + { + "id": "workbook_job_runs", + "label": "Async workbook job runs", + "risk": "high", + "action": "Persist workbook job status, result envelopes, and list filters before routing heavy workbook work to Postgres.", + "methods": ( + "save_workbook_job", + "get_workbook_job", + "list_workbook_jobs", + ), + }, + { + "id": "one_way_migration_intake", + "label": "One-way neutral migration intake", + "risk": "critical", + "action": "Keep validated migration payloads owner-fenced, lifecycle-owned, and transactionally replayable on both stores.", + "methods": ( + "consume_v05_import", + "get_v05_import", + "read_v05_import_bundle", + ), + }, + { + "id": "enterprise_admin", + "label": "Enterprise users, teams, API keys, config, and retention", + "risk": "critical", + "action": "Implement enterprise admin and API-key auth methods before Postgres can back enterprise routes.", + "methods": ( + "upsert_enterprise_user", + "record_enterprise_admin_audit", + "list_enterprise_admin_audits", + "get_enterprise_user", + "get_enterprise_user_by_email", + "get_enterprise_user_by_oidc_identity", + "list_enterprise_users", + "create_enterprise_team", + "get_enterprise_team", + "list_enterprise_teams", + "add_enterprise_team_member", + "create_enterprise_api_key", + "get_enterprise_user_by_api_key", + "list_enterprise_api_keys", + "revoke_enterprise_api_key", + "set_enterprise_config", + "get_enterprise_config", + "enterprise_dashboard", + "purge_sessions_before", + ), + }, + { + "id": "workbook_accuracy_artifacts", + "label": "Workbook accuracy and verification artifacts", + "risk": "high", + "action": "Persist workbook maps, verification runs, plan revisions, and operation audits with retrievable IDs.", + "methods": ( + "save_workbook_map", + "get_workbook_map", + "save_source_evidence_graph", + "save_operation_simulation", + "save_workbook_invariant_run", + "save_agent_run", + "get_agent_run", + "save_workbook_truth_model", + "get_workbook_truth_model", + "save_formula_graph_artifact", + "get_formula_graph_artifact", + "save_operation_simulation_v2", + "save_v7_workbook_review", + "get_v7_workbook_review", + "save_rollback_snapshot_v2", + "save_workbook_benchmark_fixture_result", + "save_operation_diff_preview_v2", + "save_source_cell_provenance_v2", + "save_plan_revision", + "save_verification_run", + "save_operation_audits", + ), + }, + { + "id": "local_replay_readiness", + "label": "Local replay and V7 default readiness", + "risk": "medium", + "action": "Keep local replay capture, baselines, and default-readiness history queryable under Postgres.", + "methods": ( + "save_local_replay_artifact", + "list_local_replay_artifacts", + "clear_local_replay_artifacts", + "save_local_replay_baseline", + "list_local_replay_baselines", + "save_v7_default_readiness_report", + "get_latest_v7_default_readiness_report", + "list_v7_default_readiness_reports", + ), + }, + { + "id": "brain_agent_artifacts", + "label": "Brain, agent-run, repair, and step artifacts", + "risk": "high", + "action": "Preserve brain and agent artifact lineage methods before moving multi-step agent workflows to Postgres.", + "methods": ( + "save_brain_run", + "get_brain_run", + "list_brain_runs", + "save_brain_artifact_bundle", + "get_brain_artifact_bundle", + "save_agent_run_artifact", + "list_agent_run_artifacts", + "append_agent_run_event", + "list_agent_run_events", + "save_agent_egress_preview", + "consume_agent_egress_approval", + "save_repair_plan_v2", + "get_repair_plan_v2", + "save_verification_report_v2", + "save_agent_run_step_artifact_v3", + ), + }, +) _store: SessionStoreProtocol | None = None _store_engine: str | None = None @@ -32,33 +221,209 @@ def get_session_store() -> SessionStoreProtocol: raise RuntimeError("DATABASE_URL must be set when DB_ENGINE=postgres") from backend.storage.postgres import PostgresSessionStore - missing = postgres_protocol_gaps(PostgresSessionStore) - if missing: - joined = ", ".join(missing[:12]) - suffix = f" and {len(missing) - 12} more" if len(missing) > 12 else "" + parity = postgres_protocol_gap_report(PostgresSessionStore) + if not parity["ready"]: + details = _format_postgres_parity_failures(parity) raise RuntimeError( - "DB_ENGINE=postgres is not enabled because PostgresSessionStore lacks " - f"SessionStoreProtocol parity: {joined}{suffix}. " - "Implement these methods or set DB_ENGINE=sqlite for local/dev." + "DB_ENGINE=postgres is not enabled because PostgresSessionStore does not satisfy " + f"SessionStoreProtocol parity ({parity['blocking_count']} blocking issues: " + f"{parity['missing_count']} missing methods, " + f"{parity['signature_mismatch_count']} signature mismatches). " + f"{details}. Inspect backend.storage.postgres_protocol_gap_report(PostgresSessionStore) " + "for categorized next actions, or set DB_ENGINE=sqlite for local/dev." ) - _store = PostgresSessionStore(DATABASE_URL) + _store = cast(SessionStoreProtocol, PostgresSessionStore(DATABASE_URL)) else: from backend.session_store import get_session_store as _get_sqlite - _store = _get_sqlite() + _store = cast(SessionStoreProtocol, _get_sqlite()) _store_engine = engine + assert _store is not None return _store def postgres_protocol_gaps(store_cls: type) -> list[str]: """Return SessionStoreProtocol methods not implemented by the Postgres store.""" - required = [ + return postgres_protocol_gap_report(store_cls)["missing_methods"] + + +def postgres_protocol_gap_report(store_cls: type) -> dict: + """Return a categorized Postgres readiness report for SessionStoreProtocol parity.""" + required = _session_store_protocol_methods() + implemented = [name for name in required if callable(getattr(store_cls, name, None))] + missing = [name for name in required if name not in implemented] + signature_mismatches = [ + issue + for name in implemented + if (issue := _signature_mismatch(name, store_cls)) is not None + ] + categories = _categorized_gap_rows(required, implemented, missing, signature_mismatches) + blocking_categories = [row for row in categories if row["blocking_count"]] + blocking_methods = sorted(set(missing) | {issue["method"] for issue in signature_mismatches}) + + return { + "store": store_cls.__name__, + "protocol": SessionStoreProtocol.__name__, + "status": "ready" if not blocking_methods else "blocked", + "ready": not blocking_methods, + "startup_policy": "DB_ENGINE=postgres is allowed only when no missing methods or signature mismatches remain.", + "required_count": len(required), + "required_methods": required, + "implemented_count": len(implemented), + "missing_count": len(missing), + "signature_mismatch_count": len(signature_mismatches), + "blocking_count": len(blocking_methods), + "missing_methods": missing, + "signature_mismatches": signature_mismatches, + "blocking_methods": blocking_methods, + "categories": categories, + "blocking_categories": blocking_categories, + "next_actions": [row["action"] for row in blocking_categories], + } + + +def _session_store_protocol_methods() -> list[str]: + return sorted( name for name, value in inspect.getmembers(SessionStoreProtocol) if not name.startswith("_") and callable(value) + ) + + +def _categorized_gap_rows( + required: list[str], + implemented: list[str], + missing: list[str], + signature_mismatches: list[dict], +) -> list[dict]: + implemented_set = set(implemented) + missing_set = set(missing) + signature_by_method = {issue["method"]: issue for issue in signature_mismatches} + covered: set[str] = set() + rows: list[dict] = [] + + for category in POSTGRES_PROTOCOL_GAP_CATEGORIES: + methods = [method for method in category["methods"] if method in required] + covered.update(methods) + missing_methods = [method for method in methods if method in missing_set] + signature_issues = [signature_by_method[method] for method in methods if method in signature_by_method] + rows.append( + { + "id": category["id"], + "label": category["label"], + "risk": category["risk"], + "action": category["action"], + "required_count": len(methods), + "implemented_count": sum(1 for method in methods if method in implemented_set), + "missing_count": len(missing_methods), + "signature_mismatch_count": len(signature_issues), + "blocking_count": len(missing_methods) + len(signature_issues), + "missing_methods": missing_methods, + "signature_mismatches": signature_issues, + } + ) + + uncategorized = [method for method in required if method not in covered] + if uncategorized: + missing_methods = [method for method in uncategorized if method in missing_set] + signature_issues = [signature_by_method[method] for method in uncategorized if method in signature_by_method] + rows.append( + { + "id": "uncategorized", + "label": "Uncategorized protocol surface", + "risk": "high", + "action": "Classify these protocol methods before treating the Postgres readiness report as complete.", + "required_count": len(uncategorized), + "implemented_count": sum(1 for method in uncategorized if method in implemented_set), + "missing_count": len(missing_methods), + "signature_mismatch_count": len(signature_issues), + "blocking_count": len(missing_methods) + len(signature_issues), + "missing_methods": missing_methods, + "signature_mismatches": signature_issues, + } + ) + + return rows + + +def _signature_mismatch(method_name: str, store_cls: type) -> dict | None: + protocol_method = getattr(SessionStoreProtocol, method_name, None) + store_method = getattr(store_cls, method_name, None) + if not callable(protocol_method) or not callable(store_method): + return None + + try: + protocol_signature = inspect.signature(protocol_method) + store_signature = inspect.signature(store_method) + except (TypeError, ValueError): + return None + + protocol_parameters = _named_parameters(protocol_signature) + store_parameters = _named_parameters(store_signature) + store_accepts_any_args = any(param.kind is inspect.Parameter.VAR_POSITIONAL for param in store_signature.parameters.values()) + store_accepts_any_kwargs = any(param.kind is inspect.Parameter.VAR_KEYWORD for param in store_signature.parameters.values()) + if store_accepts_any_args and store_accepts_any_kwargs: + return None + + missing_required = [ + name + for name, param in protocol_parameters.items() + if _is_required_parameter(param) and name not in store_parameters and not store_accepts_any_kwargs + ] + extra_required = [ + name + for name, param in store_parameters.items() + if _is_required_parameter(param) and name not in protocol_parameters ] - missing: list[str] = [] - for name in sorted(required): - if not callable(getattr(store_cls, name, None)): - missing.append(name) - return missing + tightened_optional = [ + name + for name, param in store_parameters.items() + if name in protocol_parameters + and _is_required_parameter(param) + and not _is_required_parameter(protocol_parameters[name]) + ] + + if not missing_required and not extra_required and not tightened_optional: + return None + + return { + "method": method_name, + "protocol_signature": str(protocol_signature), + "store_signature": str(store_signature), + "missing_required_protocol_parameters": missing_required, + "store_requires_extra_parameters": extra_required, + "store_tightens_optional_parameters": tightened_optional, + } + + +def _named_parameters(signature: inspect.Signature) -> dict[str, inspect.Parameter]: + return { + name: param + for name, param in signature.parameters.items() + if name != "self" and param.kind not in (inspect.Parameter.VAR_POSITIONAL, inspect.Parameter.VAR_KEYWORD) + } + + +def _is_required_parameter(param: inspect.Parameter) -> bool: + return param.default is inspect.Parameter.empty + + +def _format_postgres_parity_failures(report: dict) -> str: + parts: list[str] = [] + for category in report["blocking_categories"][:4]: + detail_parts: list[str] = [] + if category["missing_methods"]: + missing = ", ".join(category["missing_methods"][:3]) + suffix = f" and {len(category['missing_methods']) - 3} more" if len(category["missing_methods"]) > 3 else "" + detail_parts.append(f"missing {missing}{suffix}") + if category["signature_mismatches"]: + methods = [issue["method"] for issue in category["signature_mismatches"]] + mismatches = ", ".join(methods[:2]) + suffix = f" and {len(methods) - 2} more" if len(methods) > 2 else "" + detail_parts.append(f"signature mismatch {mismatches}{suffix}") + parts.append(f"{category['label']} ({category['blocking_count']}): {', '.join(detail_parts)}") + + remaining = len(report["blocking_categories"]) - 4 + if remaining > 0: + parts.append(f"{remaining} more blocking categories") + return "; ".join(parts) diff --git a/backend/storage/encrypted_store.py b/backend/storage/encrypted_store.py index 6685618..d94b8a7 100644 --- a/backend/storage/encrypted_store.py +++ b/backend/storage/encrypted_store.py @@ -1,171 +1,1131 @@ -"""Encrypted Session Store — AES-256 encryption at rest for SQLite. - -Shortcut.ai gap closure: AES-256 encryption at rest. - -Wraps the existing SessionStore with optional sqlcipher encryption. -Falls back gracefully to standard SQLite if sqlcipher is not available. -""" +"""Fail-closed SQLCipher connection provider for the primary SQLite store.""" from __future__ import annotations import os +import re import sqlite3 +import hashlib +import hmac +import json +import stat +import threading +import time +from collections.abc import Iterator from contextlib import contextmanager +from dataclasses import dataclass, field from pathlib import Path -from typing import Optional +from typing import Any, Optional import structlog logger = structlog.get_logger() +ROTATION_RECEIPT_SCHEMA_VERSION = "2026-08-sqlcipher-key-rotation-v1" +_ROTATION_JOURNAL_SCHEMA_VERSION = "2026-08-sqlcipher-key-rotation-journal-v1" +_ROTATION_ID_RE = re.compile(r"[A-Za-z0-9][A-Za-z0-9._-]{2,127}") +_KEY_ID_RE = re.compile(r"[A-Za-z0-9][A-Za-z0-9._:/-]{2,255}") +_RELEASE_SHA_RE = re.compile(r"[0-9a-f]{40}") +_MAX_ROTATION_JOURNAL_BYTES = 256 * 1024 + + +class RotationError(RuntimeError): + """Privacy-safe, machine-readable SQLCipher rotation failure.""" + + def __init__(self, code: str, *, receipt: dict[str, Any] | None = None): + self.code = code + self.receipt = receipt + super().__init__(code) + + +@dataclass +class _ProcessDatabaseState: + condition: threading.Condition = field( + default_factory=lambda: threading.Condition(threading.RLock()) + ) + active_connections: int = 0 + rotating: bool = False + + +_PROCESS_DATABASE_STATES_LOCK = threading.Lock() +_PROCESS_DATABASE_STATES: dict[str, _ProcessDatabaseState] = {} + + +def _canonical_json_bytes(value: Any) -> bytes: + return ( + json.dumps(value, sort_keys=True, separators=(",", ":"), ensure_ascii=True) + + "\n" + ).encode("utf-8") + + +def canonical_rotation_receipt_bytes(receipt: dict[str, Any]) -> bytes: + """Canonical bytes suitable for a detached external Ed25519 signature.""" + return _canonical_json_bytes(receipt) + + +def _sha256_bytes(value: bytes) -> str: + return hashlib.sha256(value).hexdigest() + + +def _sha256_text(value: str) -> str: + return _sha256_bytes(value.encode("utf-8")) + + +def _secure_regular_file(file_stat: os.stat_result) -> bool: + if not stat.S_ISREG(file_stat.st_mode) or file_stat.st_nlink != 1: + return False + if os.name != "nt": + if file_stat.st_mode & 0o022: + return False + if file_stat.st_uid not in {os.geteuid(), 0}: + return False + return True + + +def _descriptor_matches_path(file_descriptor: int, path: Path) -> bool: + try: + descriptor_stat = os.fstat(file_descriptor) + path_stat = os.lstat(path) + except OSError: + return False + return ( + descriptor_stat.st_dev == path_stat.st_dev + and descriptor_stat.st_ino == path_stat.st_ino + ) + + +def _reject_json_duplicates(pairs: list[tuple[str, Any]]) -> dict[str, Any]: + result: dict[str, Any] = {} + for key, value in pairs: + if key in result: + raise RotationError("invalid_rotation_journal") + result[key] = value + return result + + +def _exact_sha256(value: object) -> bool: + return isinstance(value, str) and re.fullmatch(r"[0-9a-f]{64}", value) is not None + + +def _valid_verification(value: object, *, successful: bool) -> bool: + if not isinstance(value, dict): + return False + if successful: + return ( + set(value) + == { + "cipherIntegrity", + "cipherVersionSha256", + "databaseReadable", + "sqliteIntegrity", + } + and value["databaseReadable"] is True + and value["cipherIntegrity"] == "verified" + and value["sqliteIntegrity"] == "verified" + and _exact_sha256(value["cipherVersionSha256"]) + ) + return ( + set(value) == {"cipherIntegrity", "databaseReadable", "sqliteIntegrity"} + and value["databaseReadable"] is False + and value["cipherIntegrity"] == "failed" + and value["sqliteIntegrity"] == "not_verified" + ) + + +def _valid_rollback(value: object, *, passed: bool) -> bool: + if not isinstance(value, dict) or set(value) != { + "attempted", + "oldKeyRevalidated", + "status", + }: + return False + if not isinstance(value["attempted"], bool) or not isinstance( + value["oldKeyRevalidated"], bool + ): + return False + status = value["status"] + if passed: + return ( + value["attempted"] is False + and value["oldKeyRevalidated"] is False + and status == "not_needed" + ) + return status in {"not_needed", "state_unknown", "recovered", "failed"} + + +def _valid_receipt_envelope( + envelope: object, + *, + request: dict[str, Any], + expected_status: str, +) -> bool: + if not isinstance(envelope, dict) or set(envelope) != {"receipt", "receiptSha256"}: + return False + receipt = envelope["receipt"] + receipt_sha = envelope["receiptSha256"] + if not isinstance(receipt, dict) or not _exact_sha256(receipt_sha): + return False + if not hmac.compare_digest( + receipt_sha, + _sha256_bytes(canonical_rotation_receipt_bytes(receipt)), + ): + return False + required = { + "algorithm", + "databaseIdentitySha256", + "externallySignable", + "generatedAtUnixSeconds", + "keyMaterial", + "newKeyIdSha256", + "oldKeyIdSha256", + "postDatabaseSha256", + "postVerification", + "preDatabaseSha256", + "preVerification", + "recoveredInterruptedRotation", + "releaseSha", + "requestSha256", + "rollback", + "rotationIdSha256", + "schemaVersion", + "signatureAlgorithm", + "status", + } + if expected_status == "failed": + required.add("failureCode") + if set(receipt) != required: + return False + if ( + receipt["schemaVersion"] != ROTATION_RECEIPT_SCHEMA_VERSION + or receipt["status"] != expected_status + or receipt["algorithm"] != "SQLCipher-AES-256" + or receipt["keyMaterial"] != "raw-256-bit" + or receipt["externallySignable"] is not True + or receipt["signatureAlgorithm"] != "Ed25519" + or receipt["releaseSha"] != request["releaseSha"] + or receipt["requestSha256"] != request["requestSha256"] + or receipt["databaseIdentitySha256"] != request["databaseIdentitySha256"] + or receipt["oldKeyIdSha256"] != request["oldKeyIdSha256"] + or receipt["newKeyIdSha256"] != request["newKeyIdSha256"] + or receipt["rotationIdSha256"] != _sha256_text(str(request["rotationId"])) + or not _exact_sha256(receipt["preDatabaseSha256"]) + or not _valid_verification(receipt["preVerification"], successful=True) + or not isinstance(receipt["recoveredInterruptedRotation"], bool) + or not isinstance(receipt["generatedAtUnixSeconds"], int) + or isinstance(receipt["generatedAtUnixSeconds"], bool) + or receipt["generatedAtUnixSeconds"] < 0 + ): + return False + passed = expected_status == "passed" + if not _valid_rollback(receipt["rollback"], passed=passed): + return False + if passed: + return _exact_sha256(receipt["postDatabaseSha256"]) and _valid_verification( + receipt["postVerification"], successful=True + ) + return ( + receipt["failureCode"] + in { + "post_rotate_verification_failed_recovered", + "post_rotate_verification_failed_recovery_failed", + "rotation_apply_failed", + "rotation_apply_failed_key_state_unknown", + } + and ( + receipt["postDatabaseSha256"] is None + or _exact_sha256(receipt["postDatabaseSha256"]) + ) + and _valid_verification(receipt["postVerification"], successful=False) + ) + + +def _lock_file_descriptor(file_descriptor: int) -> None: + if os.name == "nt": # pragma: no cover - exercised on Windows runners + import msvcrt + + vars(msvcrt)["locking"](file_descriptor, vars(msvcrt)["LK_NBLCK"], 1) + else: + import fcntl + + fcntl.flock(file_descriptor, fcntl.LOCK_EX | fcntl.LOCK_NB) + + +def _unlock_file_descriptor(file_descriptor: int) -> None: + if os.name == "nt": # pragma: no cover - exercised on Windows runners + import msvcrt + + vars(msvcrt)["locking"](file_descriptor, vars(msvcrt)["LK_UNLCK"], 1) + else: + import fcntl + + fcntl.flock(file_descriptor, fcntl.LOCK_UN) + + +@contextmanager +def _exclusive_file_lock(lock_path: Path, timeout_seconds: float) -> Iterator[None]: + if timeout_seconds <= 0 or timeout_seconds > 300: + raise RotationError("invalid_lock_timeout") + if lock_path.is_symlink(): + raise RotationError("unsafe_rotation_lock") + flags = os.O_RDWR | os.O_CREAT + if hasattr(os, "O_NOFOLLOW"): + flags |= os.O_NOFOLLOW + try: + file_descriptor = os.open(lock_path, flags, 0o600) + except OSError as exc: + raise RotationError("rotation_lock_unavailable") from exc + deadline = time.monotonic() + timeout_seconds + locked = False + try: + if not _secure_regular_file(os.fstat(file_descriptor)) or not _descriptor_matches_path( + file_descriptor, lock_path + ): + raise RotationError("unsafe_rotation_lock") + if os.name == "nt" and os.fstat(file_descriptor).st_size == 0: # pragma: no cover + os.write(file_descriptor, b"\0") + os.fsync(file_descriptor) + os.lseek(file_descriptor, 0, os.SEEK_SET) + while not locked: + try: + _lock_file_descriptor(file_descriptor) + locked = True + except (BlockingIOError, OSError) as exc: + if time.monotonic() >= deadline: + raise RotationError("rotation_lock_timeout") from exc + time.sleep(min(0.05, max(0.001, deadline - time.monotonic()))) + if not _descriptor_matches_path(file_descriptor, lock_path): + raise RotationError("unsafe_rotation_lock") + yield + finally: + if locked: + _unlock_file_descriptor(file_descriptor) + os.close(file_descriptor) + + +def _read_bounded_json(path: Path) -> dict[str, Any]: + if path.is_symlink(): + raise RotationError("unsafe_rotation_journal") + flags = os.O_RDONLY + if hasattr(os, "O_NOFOLLOW"): + flags |= os.O_NOFOLLOW + try: + file_descriptor = os.open(path, flags) + except FileNotFoundError: + return {"schemaVersion": _ROTATION_JOURNAL_SCHEMA_VERSION, "rotations": {}} + except OSError as exc: + raise RotationError("unsafe_rotation_journal") from exc + try: + file_stat = os.fstat(file_descriptor) + if not _secure_regular_file(file_stat) or not _descriptor_matches_path( + file_descriptor, path + ): + raise RotationError("unsafe_rotation_journal") + if file_stat.st_size < 1 or file_stat.st_size > _MAX_ROTATION_JOURNAL_BYTES: + raise RotationError("invalid_rotation_journal") + with os.fdopen(file_descriptor, "rb", closefd=False) as handle: + raw = handle.read(_MAX_ROTATION_JOURNAL_BYTES + 1) + if len(raw) != file_stat.st_size or len(raw) > _MAX_ROTATION_JOURNAL_BYTES: + raise RotationError("invalid_rotation_journal") + try: + parsed = json.loads( + raw.decode("utf-8"), + object_pairs_hook=_reject_json_duplicates, + ) + except (UnicodeError, json.JSONDecodeError) as exc: + raise RotationError("invalid_rotation_journal") from exc + finally: + os.close(file_descriptor) + if ( + not isinstance(parsed, dict) + or set(parsed) != {"rotations", "schemaVersion"} + or parsed.get("schemaVersion") != _ROTATION_JOURNAL_SCHEMA_VERSION + or not isinstance(parsed.get("rotations"), dict) + ): + raise RotationError("invalid_rotation_journal") + return parsed + + +def _write_bounded_json(path: Path, value: dict[str, Any]) -> None: + raw = _canonical_json_bytes(value) + if len(raw) > _MAX_ROTATION_JOURNAL_BYTES: + raise RotationError("rotation_journal_capacity_exceeded") + path.parent.mkdir(parents=True, exist_ok=True) + temporary_path = path.with_name( + f".{path.name}.{os.getpid()}.{threading.get_ident()}.tmp" + ) + flags = os.O_WRONLY | os.O_CREAT | os.O_EXCL + if hasattr(os, "O_NOFOLLOW"): + flags |= os.O_NOFOLLOW + try: + file_descriptor = os.open(temporary_path, flags, 0o600) + try: + with os.fdopen(file_descriptor, "wb", closefd=False) as handle: + handle.write(raw) + handle.flush() + os.fsync(handle.fileno()) + finally: + os.close(file_descriptor) + os.replace(temporary_path, path) + try: + directory_descriptor = os.open(path.parent, os.O_RDONLY) + except OSError: + directory_descriptor = -1 + if directory_descriptor >= 0: + try: + os.fsync(directory_descriptor) + finally: + os.close(directory_descriptor) + except OSError as exc: + try: + temporary_path.unlink(missing_ok=True) + except OSError: + pass + raise RotationError("rotation_journal_write_failed") from exc + class EncryptedSessionStore: """SQLite session store with optional AES-256 encryption via sqlcipher. Configuration via environment variables: - DB_ENCRYPT=true — enable encryption - - DB_ENCRYPT_KEY — encryption passphrase (REQUIRED if DB_ENCRYPT=true) + - DB_ENCRYPT_KEY — 64 lowercase hex characters (32 raw key bytes) - DB_ENCRYPT_CIPHER — cipher name (default: aes-256-cbc) - DB_PATH — database file path """ - def __init__(self, db_path: Optional[str] = None, encrypt_key: Optional[str] = None): - from backend.session_store import DB_PATH as DEFAULT_DB_PATH + def __init__( + self, + db_path: Optional[str] = None, + encrypt_key: Optional[str] = None, + *, + key_id: Optional[str] = None, + encrypt_enabled: Optional[bool] = None, + ): + if db_path is None: + from backend.session_store import DB_PATH as default_db_path - self.db_path = Path(db_path) if db_path else DEFAULT_DB_PATH + self.db_path = default_db_path + else: + self.db_path = Path(db_path) self.db_path.parent.mkdir(parents=True, exist_ok=True) - self.encrypt_enabled = os.getenv("DB_ENCRYPT", "false").lower() == "true" - self.encrypt_key = encrypt_key or os.getenv("DB_ENCRYPT_KEY", "") + self.encrypt_enabled = ( + os.getenv("DB_ENCRYPT", "false").lower() == "true" + if encrypt_enabled is None + else encrypt_enabled + ) + self.encrypt_key = ( + encrypt_key if encrypt_key is not None else (os.getenv("DB_ENCRYPT_KEY") or "") + ) + self.key_id = key_id if key_id is not None else (os.getenv("DB_ENCRYPT_KEY_ID") or "") + database_state_key = _sha256_text(str(self.db_path.resolve(strict=False))) + with _PROCESS_DATABASE_STATES_LOCK: + self._process_state = _PROCESS_DATABASE_STATES.setdefault( + database_state_key, _ProcessDatabaseState() + ) + self._sqlcipher_driver: str | None = None self._sqlcipher_available = False if self.encrypt_enabled: self._sqlcipher_available = self._check_sqlcipher() if self._sqlcipher_available: - logger.info("db_encryption_enabled", cipher="aes-256-cbc") - else: - logger.warning( - "db_encryption_unavailable", - note="sqlcipher not installed. Install pysqlcipher3 or use sqlcipher-built sqlite3.", + logger.info( + "db_encryption_driver_available", + driver=self._sqlcipher_driver, ) + else: + logger.error("db_encryption_driver_unavailable") def _check_sqlcipher(self) -> bool: - """Check if sqlcipher is available.""" + """Require a non-empty cipher_version response from a real driver.""" try: import pysqlcipher3.dbapi2 as sqlcipher - return True + + if self._driver_has_cipher(sqlcipher): + self._sqlcipher_driver = "pysqlcipher3" + return True except ImportError: pass - # Check if the standard sqlite3 module has cipher support (some distributions do) + if self._driver_has_cipher(sqlite3): + self._sqlcipher_driver = "sqlite3-sqlcipher" + return True + + return False + + @staticmethod + def _driver_has_cipher(driver: Any) -> bool: + conn = None + try: + conn = driver.connect(":memory:") + row = conn.execute("PRAGMA cipher_version").fetchone() + return bool(row and isinstance(row[0], str) and row[0].strip()) + except Exception: + return False + finally: + if conn is not None: + conn.close() + + def _open_encrypted_connection(self, raw_key: str) -> Any: + """Open SQLCipher and force evaluation of the supplied raw key.""" + _validate_encrypt_key(raw_key) + if not self._sqlcipher_available: + raise RuntimeError("verified SQLCipher driver unavailable") + + if self._sqlcipher_driver == "pysqlcipher3": + import pysqlcipher3.dbapi2 as sqlcipher + + conn = sqlcipher.connect(str(self.db_path)) + elif self._sqlcipher_driver == "sqlite3-sqlcipher": + conn = sqlite3.connect(str(self.db_path)) + else: + raise RuntimeError("SQLCipher driver selection is inconsistent") + + try: + conn.execute(_sqlcipher_key_pragma(raw_key)) + conn.execute("PRAGMA cipher_page_size = 4096") + conn.execute("PRAGMA cipher_hmac_algorithm = HMAC_SHA512") + conn.execute("PRAGMA cipher_memory_security = ON") + cipher_row = conn.execute("PRAGMA cipher_version").fetchone() + if not cipher_row or not str(cipher_row[0]).strip(): + raise RuntimeError("SQLCipher cipher activation failed") + conn.execute("SELECT count(*) FROM sqlite_master").fetchone() + return conn + except Exception: + conn.close() + raise + + def _verify_key_and_integrity(self, raw_key: str) -> dict[str, Any]: + """Force key evaluation plus SQLCipher and SQLite integrity checks.""" + conn = self._open_encrypted_connection(raw_key) try: - conn = sqlite3.connect(":memory:") - conn.execute("PRAGMA cipher_version") + checkpoint_row = conn.execute("PRAGMA wal_checkpoint(FULL)").fetchone() + if checkpoint_row and int(checkpoint_row[0]) != 0: + raise RuntimeError("SQLCipher WAL checkpoint remained busy") + cipher_row = conn.execute("PRAGMA cipher_version").fetchone() + cipher_version = str(cipher_row[0]).strip() if cipher_row else "" + cipher_rows = conn.execute("PRAGMA cipher_integrity_check").fetchall() + cipher_values = [str(row[0]).strip().lower() for row in cipher_rows if row] + # SQLCipher versions differ: an empty result or one/more `ok` + # rows means success; any diagnostic row means failure. + if cipher_values and any(value != "ok" for value in cipher_values): + raise RuntimeError("SQLCipher integrity verification failed") + sqlite_row = conn.execute("PRAGMA integrity_check").fetchone() + if not sqlite_row or str(sqlite_row[0]).strip().lower() != "ok": + raise RuntimeError("SQLite integrity verification failed") + return { + "databaseReadable": True, + "cipherIntegrity": "verified", + "sqliteIntegrity": "verified", + "cipherVersionSha256": _sha256_text(cipher_version), + } + finally: conn.close() - return True + + def _apply_rekey(self, old_key: str, new_key: str) -> None: + """Apply a raw-key SQLCipher rekey without exposing either key.""" + _validate_encrypt_key(old_key) + _validate_encrypt_key(new_key) + conn = self._open_encrypted_connection(old_key) + try: + rekey_pragma = _sqlcipher_key_pragma(new_key).replace( + "PRAGMA key", "PRAGMA rekey", 1 + ) + conn.execute(rekey_pragma) + conn.commit() except Exception: - pass + conn.rollback() + raise + finally: + conn.close() - return False + @contextmanager + def _connection_admission(self) -> Iterator[None]: + """Fence application connections while a rotation owns the database.""" + if not self.encrypt_enabled: + yield + return + state = self._process_state + with state.condition: + while state.rotating: + state.condition.wait() + state.active_connections += 1 + try: + yield + finally: + with state.condition: + state.active_connections -= 1 + state.condition.notify_all() @contextmanager - def connection(self): + def connection(self) -> Iterator[Any]: """Get an encrypted database connection. - Encryption is enforced when DB_ENCRYPT=true AND the deployment is in - an enterprise/production profile: a missing sqlcipher binding becomes - a hard failure rather than a silent fallback to plaintext storage. + When ``DB_ENCRYPT=true`` every connection must use a verified SQLCipher + driver and an exact 256-bit raw key. There is no development fallback + because a fallback would make the same database alternate between + encrypted and plaintext access depending on the host. """ - from backend.config import PRODUCTION_MODE # local import avoids cycle - if self.encrypt_enabled: - if not self.encrypt_key: - raise RuntimeError("DB_ENCRYPT is true but DB_ENCRYPT_KEY is empty; refusing to open an unencrypted database") + _validate_encrypt_key(self.encrypt_key) if not self._sqlcipher_available: - if PRODUCTION_MODE: - raise RuntimeError( - "DB_ENCRYPT is true but sqlcipher is not available; install pysqlcipher3 or a sqlcipher-built sqlite3" - ) - logger.warning( - "db_encryption_unavailable_dev_fallback", - note="Continuing without encryption because production_mode is false", + raise RuntimeError( + "DB_ENCRYPT is true but a verified SQLCipher driver is unavailable; " + "refusing plaintext access" ) - if self._sqlcipher_available and self.encrypt_enabled and self.encrypt_key: - key_pragma = _sqlcipher_key_pragma(self.encrypt_key) + with self._connection_admission(): + conn: Any | None = None try: - import pysqlcipher3.dbapi2 as sqlcipher - conn = sqlcipher.connect(str(self.db_path)) - conn.execute(key_pragma) - conn.execute("PRAGMA cipher_page_size = 4096") - conn.execute("PRAGMA kdf_iter = 256000") - conn.execute("PRAGMA cipher_hmac_algorithm = HMAC_SHA512") - conn.execute("PRAGMA cipher_kdf_algorithm = PBKDF2_HMAC_SHA512") - except ImportError: - # Fallback to standard sqlite3 with PRAGMA key (only reachable - # when sqlcipher reported available but the import failed mid - # request — should be vanishingly rare). - conn = sqlite3.connect(str(self.db_path)) - try: - conn.execute(key_pragma) - except Exception: - pass - else: - conn = sqlite3.connect(str(self.db_path), timeout=10) + if self.encrypt_enabled: + conn = self._open_encrypted_connection(self.encrypt_key) + else: + conn = sqlite3.connect(str(self.db_path), timeout=10) + + conn.row_factory = sqlite3.Row + conn.execute("PRAGMA journal_mode=WAL") + conn.execute("PRAGMA foreign_keys=ON") + except Exception: + if conn is not None: + conn.close() + raise - conn.row_factory = sqlite3.Row - conn.execute("PRAGMA journal_mode=WAL") - conn.execute("PRAGMA foreign_keys=ON") + assert conn is not None + try: + yield conn + conn.commit() + except Exception: + conn.rollback() + raise + finally: + conn.close() + + @property + def _rotation_lock_path(self) -> Path: + return self.db_path.with_name(f".{self.db_path.name}.rotation.lock") + + @property + def _rotation_journal_path(self) -> Path: + return self.db_path.with_name(f".{self.db_path.name}.rotation.json") + + def _database_identity_sha256(self) -> str: + file_descriptor = self._open_database_read_descriptor() try: - yield conn - conn.commit() - except Exception: - conn.rollback() + file_stat = os.fstat(file_descriptor) + return _sha256_text(f"{file_stat.st_dev}:{file_stat.st_ino}") + finally: + os.close(file_descriptor) + + def _open_database_read_descriptor(self) -> int: + if self.db_path.is_symlink(): + raise RotationError("unsafe_database_file") + flags = os.O_RDONLY + if hasattr(os, "O_NOFOLLOW"): + flags |= os.O_NOFOLLOW + try: + file_descriptor = os.open(self.db_path, flags) + except OSError as exc: + raise RotationError("database_file_unavailable") from exc + if not _secure_regular_file(os.fstat(file_descriptor)) or not _descriptor_matches_path( + file_descriptor, self.db_path + ): + os.close(file_descriptor) + raise RotationError("unsafe_database_file") + return file_descriptor + + def _validate_storage_directory(self) -> None: + try: + directory = self.db_path.parent.resolve(strict=True) + directory_stat = directory.stat() + except OSError as exc: + raise RotationError("unsafe_storage_directory") from exc + if not stat.S_ISDIR(directory_stat.st_mode): + raise RotationError("unsafe_storage_directory") + if os.name != "nt" and ( + directory_stat.st_mode & 0o022 + or directory_stat.st_uid not in {os.geteuid(), 0} + ): + raise RotationError("unsafe_storage_directory") + + def _assert_database_identity(self, expected_sha256: str) -> None: + if not hmac.compare_digest(self._database_identity_sha256(), expected_sha256): + raise RotationError("database_identity_changed") + + def _database_sha256(self) -> str: + digest = hashlib.sha256() + file_descriptor = self._open_database_read_descriptor() + try: + with os.fdopen(file_descriptor, "rb", closefd=False) as handle: + for chunk in iter(lambda: handle.read(1024 * 1024), b""): + digest.update(chunk) + except RotationError: raise + except OSError as exc: + raise RotationError("database_digest_failed") from exc finally: - conn.close() + os.close(file_descriptor) + return digest.hexdigest() - def rekey(self, new_key: str): - """Re-encrypt the database with a new key.""" - if not self._sqlcipher_available: - logger.error("rekey_requires_sqlcipher") - return False + @contextmanager + def _rotation_admission(self, timeout_seconds: float) -> Iterator[None]: + deadline = time.monotonic() + timeout_seconds + self._validate_storage_directory() + with _exclusive_file_lock(self._rotation_lock_path, timeout_seconds): + state = self._process_state + with state.condition: + while state.rotating or state.active_connections: + remaining = deadline - time.monotonic() + if remaining <= 0: + raise RotationError("rotation_connection_drain_timeout") + state.condition.wait(timeout=remaining) + state.rotating = True + try: + yield + finally: + with state.condition: + state.rotating = False + state.condition.notify_all() + @staticmethod + def _validate_rotation_request( + *, + rotation_id: str, + old_key_id: str, + new_key_id: str, + new_key: str, + release_sha: str, + ) -> None: + if _ROTATION_ID_RE.fullmatch(rotation_id) is None: + raise RotationError("rotation_id_invalid") + if _KEY_ID_RE.fullmatch(old_key_id) is None or _KEY_ID_RE.fullmatch(new_key_id) is None: + raise RotationError("invalid_key_id") + if hmac.compare_digest(old_key_id, new_key_id): + raise RotationError("rotation_key_id_conflict") try: - with self.connection() as conn: - rekey_pragma = _sqlcipher_key_pragma(new_key).replace("PRAGMA key", "PRAGMA rekey") - conn.execute(rekey_pragma) + _validate_encrypt_key(new_key) + except RuntimeError as exc: + raise RotationError("invalid_new_key") from exc + if _RELEASE_SHA_RE.fullmatch(release_sha) is None: + raise RotationError("rotation_release_sha_invalid") + + def _rotation_request( + self, + *, + rotation_id: str, + old_key_id: str, + new_key_id: str, + new_key: str, + release_sha: str, + database_identity_sha256: str, + ) -> dict[str, Any]: + request = { + "schemaVersion": ROTATION_RECEIPT_SCHEMA_VERSION, + "rotationId": rotation_id, + "releaseSha": release_sha, + "databaseIdentitySha256": database_identity_sha256, + "oldKeyIdSha256": _sha256_text(old_key_id), + "newKeyIdSha256": _sha256_text(new_key_id), + "newKeyMaterialSha256": _sha256_bytes(bytes.fromhex(new_key)), + } + request["requestSha256"] = _sha256_bytes(_canonical_json_bytes(request)) + return request + + @staticmethod + def _receipt_envelope(receipt: dict[str, Any]) -> dict[str, Any]: + return { + "receipt": receipt, + "receiptSha256": _sha256_bytes(canonical_rotation_receipt_bytes(receipt)), + } + + def _build_rotation_receipt( + self, + *, + status: str, + request: dict[str, Any], + pre_verification: dict[str, Any], + post_verification: dict[str, Any], + pre_database_sha256: str, + post_database_sha256: str | None, + recovered_interrupted_rotation: bool, + rollback: dict[str, Any], + failure_code: str | None = None, + ) -> dict[str, Any]: + receipt: dict[str, Any] = { + "schemaVersion": ROTATION_RECEIPT_SCHEMA_VERSION, + "status": status, + "rotationIdSha256": _sha256_text(str(request["rotationId"])), + "releaseSha": request["releaseSha"], + "requestSha256": request["requestSha256"], + "databaseIdentitySha256": request["databaseIdentitySha256"], + "oldKeyIdSha256": request["oldKeyIdSha256"], + "newKeyIdSha256": request["newKeyIdSha256"], + "algorithm": "SQLCipher-AES-256", + "keyMaterial": "raw-256-bit", + "preVerification": pre_verification, + "postVerification": post_verification, + "preDatabaseSha256": pre_database_sha256, + "postDatabaseSha256": post_database_sha256, + "recoveredInterruptedRotation": recovered_interrupted_rotation, + "rollback": rollback, + "externallySignable": True, + "signatureAlgorithm": "Ed25519", + "generatedAtUnixSeconds": int(time.time()), + } + if failure_code is not None: + receipt["failureCode"] = failure_code + return self._receipt_envelope(receipt) + + def _save_rotation_record( + self, + journal: dict[str, Any], + rotation_id: str, + record: dict[str, Any], + ) -> None: + journal["rotations"][rotation_id] = record + _write_bounded_json(self._rotation_journal_path, journal) + + def _verify_or_none(self, raw_key: str) -> dict[str, Any] | None: + try: + return self._verify_key_and_integrity(raw_key) + except Exception: + return None + + @staticmethod + def _validate_existing_rotation_record( + record: object, + request: dict[str, Any], + ) -> dict[str, Any]: + if not isinstance(record, dict): + raise RotationError("invalid_rotation_journal") + status = record.get("status") + required = { + "preDatabaseSha256", + "preVerification", + "request", + "requestSha256", + "status", + } + if status in {"passed", "failed"}: + required.add("receiptEnvelope") + if set(record) != required: + raise RotationError("invalid_rotation_journal") + if ( + status not in {"in_progress", "passed", "failed"} + or record["request"] != request + or not isinstance(record["requestSha256"], str) + or not hmac.compare_digest(record["requestSha256"], request["requestSha256"]) + or not _exact_sha256(record["preDatabaseSha256"]) + or not _valid_verification(record["preVerification"], successful=True) + ): + raise RotationError("invalid_rotation_journal") + if status in {"passed", "failed"} and not _valid_receipt_envelope( + record["receiptEnvelope"], + request=request, + expected_status=status, + ): + raise RotationError("invalid_rotation_journal") + return record + + def rotate_key( + self, + *, + rotation_id: str, + old_key_id: str, + new_key_id: str, + new_key: str, + release_sha: str, + lock_timeout_seconds: float = 30.0, + ) -> dict[str, Any]: + """Rotate a SQLCipher key with durable replay and recovery fencing. + + Receipts contain only hashes and categorical verification results. They + are canonicalized for a detached external signature; this process does + not hold or assert an attestation signing key. + """ + self._validate_rotation_request( + rotation_id=rotation_id, + old_key_id=old_key_id, + new_key_id=new_key_id, + new_key=new_key, + release_sha=release_sha, + ) + if not self.encrypt_enabled: + raise RotationError("encryption_not_enabled") + if not self._sqlcipher_available: + raise RotationError("sqlcipher_not_available") + + with self._rotation_admission(lock_timeout_seconds): + database_identity_sha256 = self._database_identity_sha256() + request = self._rotation_request( + rotation_id=rotation_id, + old_key_id=old_key_id, + new_key_id=new_key_id, + new_key=new_key, + release_sha=release_sha, + database_identity_sha256=database_identity_sha256, + ) + self._assert_database_identity(database_identity_sha256) + journal = _read_bounded_json(self._rotation_journal_path) + existing = journal["rotations"].get(rotation_id) + if existing is not None: + if not isinstance(existing, dict): + raise RotationError("invalid_rotation_journal") + if not hmac.compare_digest( + str(existing.get("requestSha256", "")), request["requestSha256"] + ): + raise RotationError("rotation_id_conflict") + existing = self._validate_existing_rotation_record(existing, request) + existing_status = existing.get("status") + existing_receipt = existing.get("receiptEnvelope") + if existing_status == "passed" and isinstance(existing_receipt, dict): + self._assert_database_identity(database_identity_sha256) + if self._verify_or_none(new_key) is None: + raise RotationError("rotation_replay_verification_failed") + self.encrypt_key = new_key + self.key_id = new_key_id + return {"outcome": "replayed", "receipt": existing_receipt} + if existing_status == "failed" and isinstance(existing_receipt, dict): + raise RotationError("rotation_previously_failed", receipt=existing_receipt) + if existing_status != "in_progress": + raise RotationError("invalid_rotation_journal") + else: + existing = None + + post_verification: dict[str, Any] | None = None + if existing is None: + if not self.key_id: + raise RotationError("old_key_id_not_configured") + if not hmac.compare_digest(self.key_id, old_key_id): + raise RotationError("old_key_id_mismatch") + try: + _validate_encrypt_key(self.encrypt_key) + except RuntimeError as exc: + raise RotationError("invalid_old_key") from exc + if hmac.compare_digest(self.encrypt_key, new_key): + raise RotationError("rotation_key_material_not_distinct") + + pre_verification = self._verify_or_none(self.encrypt_key) + if pre_verification is None: + raise RotationError("pre_rotate_verification_failed") + pre_database_sha256 = self._database_sha256() + existing = { + "status": "in_progress", + "requestSha256": request["requestSha256"], + "request": request, + "preVerification": pre_verification, + "preDatabaseSha256": pre_database_sha256, + } + self._save_rotation_record(journal, rotation_id, existing) + old_key = self.encrypt_key + self._assert_database_identity(database_identity_sha256) + try: + self._apply_rekey(old_key, new_key) + except Exception as exc: + # A driver can report an error after SQLCipher has durably + # changed the key. Resolve the actual key state before + # deciding whether rollback or an operator intervention is + # required. + post_verification = self._verify_or_none(new_key) + if post_verification is not None: + recovered_interrupted_rotation = True + else: + old_key_revalidated = self._verify_or_none(old_key) is not None + failure_code = ( + "rotation_apply_failed" + if old_key_revalidated + else "rotation_apply_failed_key_state_unknown" + ) + failed = self._build_rotation_receipt( + status="failed", + request=request, + pre_verification=pre_verification, + post_verification={ + "databaseReadable": False, + "cipherIntegrity": "failed", + "sqliteIntegrity": "not_verified", + }, + pre_database_sha256=pre_database_sha256, + post_database_sha256=( + self._database_sha256() if old_key_revalidated else None + ), + recovered_interrupted_rotation=False, + rollback={ + "attempted": False, + "oldKeyRevalidated": old_key_revalidated, + "status": ( + "not_needed" if old_key_revalidated else "state_unknown" + ), + }, + failure_code=failure_code, + ) + existing.update(status="failed", receiptEnvelope=failed) + self._save_rotation_record(journal, rotation_id, existing) + raise RotationError(failure_code, receipt=failed) from exc + else: + recovered_interrupted_rotation = False + else: + pre_verification_value = existing.get("preVerification") + pre_database_value = existing.get("preDatabaseSha256") + if not isinstance(pre_verification_value, dict) or not isinstance( + pre_database_value, str + ): + raise RotationError("invalid_rotation_journal") + pre_verification = pre_verification_value + pre_database_sha256 = pre_database_value + recovered_interrupted_rotation = True + old_key = self.encrypt_key + self._assert_database_identity(database_identity_sha256) + post_verification = self._verify_or_none(new_key) + if post_verification is None: + if self._verify_or_none(old_key) is None: + raise RotationError("interrupted_rotation_key_state_unknown") + try: + self._assert_database_identity(database_identity_sha256) + self._apply_rekey(old_key, new_key) + except Exception as exc: + raise RotationError("interrupted_rotation_resume_failed") from exc + + if post_verification is None: + self._assert_database_identity(database_identity_sha256) + post_verification = self._verify_or_none(new_key) + if post_verification is None: + self._assert_database_identity(database_identity_sha256) + rollback_recovered = False + try: + self._apply_rekey(new_key, old_key) + rollback_recovered = self._verify_or_none(old_key) is not None + except Exception: + rollback_recovered = False + failure_code = ( + "post_rotate_verification_failed_recovered" + if rollback_recovered + else "post_rotate_verification_failed_recovery_failed" + ) + failed = self._build_rotation_receipt( + status="failed", + request=request, + pre_verification=pre_verification, + post_verification={ + "databaseReadable": False, + "cipherIntegrity": "failed", + "sqliteIntegrity": "not_verified", + }, + pre_database_sha256=pre_database_sha256, + post_database_sha256=(self._database_sha256() if rollback_recovered else None), + recovered_interrupted_rotation=recovered_interrupted_rotation, + rollback={ + "attempted": True, + "oldKeyRevalidated": rollback_recovered, + "status": "recovered" if rollback_recovered else "failed", + }, + failure_code=failure_code, + ) + existing.update(status="failed", receiptEnvelope=failed) + self._save_rotation_record(journal, rotation_id, existing) + raise RotationError(failure_code, receipt=failed) + + passed = self._build_rotation_receipt( + status="passed", + request=request, + pre_verification=pre_verification, + post_verification=post_verification, + pre_database_sha256=pre_database_sha256, + post_database_sha256=self._database_sha256(), + recovered_interrupted_rotation=recovered_interrupted_rotation, + rollback={ + "attempted": False, + "oldKeyRevalidated": False, + "status": "not_needed", + }, + ) + self._assert_database_identity(database_identity_sha256) + existing.update(status="passed", receiptEnvelope=passed) + self._save_rotation_record(journal, rotation_id, existing) self.encrypt_key = new_key - logger.info("db_rekeyed_successfully") - return True - except Exception as e: - logger.error("db_rekey_failed", error=str(e)) - return False + self.key_id = new_key_id + logger.info("db_key_rotation_verified", rotation_id_sha256=_sha256_text(rotation_id)) + return {"outcome": "rotated", "receipt": passed} - def get_encryption_status(self) -> dict: - """Get encryption configuration status.""" + def rekey(self, new_key: str) -> bool: + """Refuse an unreceipted legacy rekey; use :meth:`rotate_key`.""" + del new_key + logger.error("rekey_requires_evidenced_rotation") + return False + + def get_encryption_status(self) -> dict[str, Any]: + """Return bounded posture without paths, keys, or unverified claims.""" + key_valid = False + if self.encrypt_key: + try: + _validate_encrypt_key(self.encrypt_key) + key_valid = True + except RuntimeError: + pass + prerequisites_ready = ( + self.encrypt_enabled + and self._sqlcipher_available + and self._sqlcipher_driver is not None + and key_valid + ) + active = False + verification_failed = False + database_initialized = self.db_path.is_file() and not self.db_path.is_symlink() + if prerequisites_ready and database_initialized: + try: + self._verify_key_and_integrity(self.encrypt_key) + active = True + except Exception: + verification_failed = True + if active: + status = "active" + reason = None + elif self.encrypt_enabled: + status = "blocked" + if not self._sqlcipher_available or self._sqlcipher_driver is None: + reason = "driver_unavailable" + elif not key_valid: + reason = "key_invalid" + elif not database_initialized: + reason = "database_not_initialized" + elif verification_failed: + reason = "runtime_verification_failed" + else: + reason = "runtime_verification_not_completed" + else: + status = "disabled" + reason = "not_configured" return { - "enabled": self.encrypt_enabled, - "sqlcipher_available": self._sqlcipher_available, - "active": self.encrypt_enabled and self._sqlcipher_available and bool(self.encrypt_key), - "cipher": "aes-256-cbc" if self._sqlcipher_available else None, - "kdf": "PBKDF2_HMAC_SHA512" if self._sqlcipher_available else None, - "db_path": str(self.db_path), + "status": status, + "configured": self.encrypt_enabled, + "driverAvailable": self._sqlcipher_available, + "driver": self._sqlcipher_driver, + "keyConfigured": bool(self.encrypt_key), + "keyValid": key_valid, + "active": active, + "primaryStoreProtected": active, + "cipher": "AES-256" if active else None, + "kdf": None, + "reasonCode": reason, } def _sqlcipher_key_pragma(raw_key: str) -> str: """Render a safe `PRAGMA key = ...` statement. - SQLite's PRAGMA does not bind parameters, so the key has to be - interpolated. Use sqlcipher's hex-literal form (`PRAGMA key = "x'...'"`) - so the value is a deterministic, quote-free hex blob. This eliminates - the previous SQL-injection-shaped risk where a key containing a single - quote could escape the literal. + SQLite's PRAGMA does not bind parameters, so only an exactly validated + 256-bit lowercase hexadecimal raw key is accepted. SQLCipher's raw-key + literal form is quote-free and cannot escape the statement. """ - encoded = raw_key.encode("utf-8").hex() - return f"PRAGMA key = \"x'{encoded}'\"" + _validate_encrypt_key(raw_key) + return f"PRAGMA key = \"x'{raw_key}'\"" + + +def _validate_encrypt_key(raw_key: str) -> None: + if re.fullmatch(r"[0-9a-f]{64}", raw_key) is None: + raise RuntimeError( + "DB_ENCRYPT_KEY must contain exactly 64 lowercase hexadecimal characters" + ) # ── Singleton ──────────────────────────────────────────────────────────── @@ -177,3 +1137,63 @@ def get_encrypted_store() -> EncryptedSessionStore: if _store is None: _store = EncryptedSessionStore() return _store + + +def get_primary_storage_encryption_status() -> dict[str, Any]: + """Report only encryption that this process can prove for its primary DB.""" + from backend.config import DB_ENGINE + + if DB_ENGINE == "postgres": + from backend.operations.storage_encryption_attestation import ( + runtime_external_storage_encryption_status, + ) + + external = runtime_external_storage_encryption_status() + if external.get("authenticatedExternalAttestation") is True: + return { + "status": "active", + "configured": True, + "driverAvailable": None, + "driver": "postgres", + "keyConfigured": None, + "keyValid": None, + "active": True, + "primaryStoreProtected": True, + "cipher": external.get("algorithm"), + "kdf": None, + "reasonCode": None, + "authenticatedExternalAttestation": True, + "attestationReceiptSha256": external.get("receiptSha256"), + "attestationReleaseSha": external.get("releaseSha"), + "keyIdentitySha256": external.get("keyIdentitySha256"), + "provider": external.get("provider"), + } + if external.get("status") == "blocked": + return { + "status": "blocked", + "configured": True, + "driverAvailable": None, + "driver": "postgres", + "keyConfigured": None, + "keyValid": None, + "active": False, + "primaryStoreProtected": False, + "cipher": None, + "kdf": None, + "reasonCode": external.get("reasonCode"), + "authenticatedExternalAttestation": False, + } + return { + "status": "external_attestation_required", + "configured": False, + "driverAvailable": None, + "driver": "postgres", + "keyConfigured": None, + "keyValid": None, + "active": False, + "primaryStoreProtected": False, + "cipher": None, + "kdf": None, + "reasonCode": "postgres_encryption_at_rest_not_runtime_verifiable", + } + return get_encrypted_store().get_encryption_status() diff --git a/backend/storage/enterprise_identity.py b/backend/storage/enterprise_identity.py new file mode 100644 index 0000000..e9d54b4 --- /dev/null +++ b/backend/storage/enterprise_identity.py @@ -0,0 +1,119 @@ +"""Canonical enterprise OIDC identity binding rules. + +OIDC issuer and subject are the immutable authorization identity. Email and +other profile claims are deliberately excluded from this contract. +""" + +from __future__ import annotations + +from dataclasses import dataclass +from typing import Any, Mapping + + +ACTIVE_STATUS = "active" +API_KEY_ONLY_STATUS = "api_key_only" +BINDING_REQUIRED_STATUS = "binding_required" + + +@dataclass(frozen=True, slots=True) +class EnterpriseIdentityBinding: + status: str + metadata: dict[str, Any] + issuer: str + subject: str + + +def prepare_enterprise_identity_binding( + *, + status: str | None, + metadata: Mapping[str, Any] | None, + oidc_issuer: str | None, + oidc_subject: str | None, + existing: Mapping[str, Any] | None = None, +) -> EnterpriseIdentityBinding: + """Normalize and enforce a complete, unique-at-storage immutable binding.""" + normalized_status = str(status or ACTIVE_STATUS).strip().lower() + existing_metadata = existing.get("metadata") if isinstance(existing, Mapping) else None + existing_metadata = existing_metadata if isinstance(existing_metadata, Mapping) else {} + requested_metadata = dict(metadata or {}) + normalized_metadata = dict(existing_metadata) + normalized_metadata.update(requested_metadata) + + existing_issuer = _binding_value(existing, existing_metadata, "oidc_issuer") + existing_subject = _binding_value(existing, existing_metadata, "oidc_subject") + metadata_issuer = _clean_binding_value(requested_metadata.get("oidc_issuer")) + metadata_subject = _clean_binding_value(requested_metadata.get("oidc_subject")) + + if oidc_issuer is not None and metadata_issuer and _clean_binding_value(oidc_issuer) != metadata_issuer: + raise ValueError("OIDC issuer conflicts with metadata.oidc_issuer") + if oidc_subject is not None and metadata_subject and _clean_binding_value(oidc_subject) != metadata_subject: + raise ValueError("OIDC subject conflicts with metadata.oidc_subject") + + issuer = _requested_binding_value( + explicit=oidc_issuer, + metadata=metadata_issuer, + metadata_present="oidc_issuer" in requested_metadata, + existing=existing_issuer, + ) + subject = _requested_binding_value( + explicit=oidc_subject, + metadata=metadata_subject, + metadata_present="oidc_subject" in requested_metadata, + existing=existing_subject, + ) + + if bool(issuer) != bool(subject): + raise ValueError("OIDC identity binding requires both issuer and subject") + if bool(existing_issuer) != bool(existing_subject): + raise ValueError("Existing enterprise user has an incomplete OIDC identity binding") + if existing_issuer and (issuer != existing_issuer or subject != existing_subject): + raise ValueError("OIDC issuer and subject are immutable once bound") + if normalized_status == ACTIVE_STATUS and not (issuer and subject): + raise ValueError("Active enterprise user requires an immutable OIDC issuer and subject") + + if issuer: + normalized_metadata["oidc_issuer"] = issuer + normalized_metadata["oidc_subject"] = subject + normalized_metadata.pop("oidc_binding_state", None) + else: + normalized_metadata.pop("oidc_issuer", None) + normalized_metadata.pop("oidc_subject", None) + + return EnterpriseIdentityBinding( + status=normalized_status, + metadata=normalized_metadata, + issuer=issuer, + subject=subject, + ) + + +def api_key_status_allows(status: Any) -> bool: + """API-only legacy/bootstrap users remain usable without becoming OIDC users.""" + return str(status or ACTIVE_STATUS).strip().lower() in {ACTIVE_STATUS, API_KEY_ONLY_STATUS} + + +def _binding_value( + existing: Mapping[str, Any] | None, + metadata: Mapping[str, Any], + key: str, +) -> str: + direct = existing.get(key) if isinstance(existing, Mapping) else None + return _clean_binding_value(direct or metadata.get(key)) + + +def _requested_binding_value( + *, + explicit: str | None, + metadata: str, + metadata_present: bool, + existing: str, +) -> str: + if explicit is not None: + return _clean_binding_value(explicit) + if metadata_present: + return metadata + return existing + + +def _clean_binding_value(value: Any) -> str: + return str(value or "").strip() diff --git a/backend/storage/managed_object_store.py b/backend/storage/managed_object_store.py new file mode 100644 index 0000000..8b5b978 --- /dev/null +++ b/backend/storage/managed_object_store.py @@ -0,0 +1,2683 @@ +"""Opt-in encrypted managed-object storage for enterprise deployments. + +The substrate is deliberately disconnected from document/workbook routes. A +deployment must explicitly construct it with a metadata repository and key +material; the local-first application never enables it implicitly. +""" + +from __future__ import annotations + +import asyncio +import base64 +import binascii +import contextlib +import fcntl +import hashlib +import hmac +import io +import json +import math +import os +import re +import sqlite3 +import stat +import struct +import threading +import time +import uuid +from collections.abc import Callable, Iterator, Mapping +from dataclasses import dataclass +from enum import Enum +from pathlib import Path +from typing import Any, BinaryIO, Protocol, TypeVar + +from cryptography.exceptions import InvalidTag +from cryptography.hazmat.primitives.ciphers import Cipher, algorithms, modes + + +_MAGIC = b"EAMOBJ01" +_FORMAT_VERSION = "2026-08-managed-object-v1" +_ALGORITHM = "AES-256-GCM" +_TAG_BYTES = 16 +_NONCE_BYTES = 12 +_MAX_HEADER_BYTES = 8192 +_MAX_JOURNAL_BYTES = 16 * 1024 * 1024 +_ID_PATTERN = re.compile(r"^[A-Za-z0-9._-]{1,128}$") +_OBJECT_ID_PATTERN = re.compile(r"^obj_[0-9a-f]{64}$") +_LOCATOR_PATTERN = re.compile(r"^[0-9a-f]{32}/[0-9a-f]{32}\.mobj$") +_RECOVERABLE_CIPHERTEXT_PATTERN = re.compile( + r"^(?P[0-9a-f]{32})/(?P(?:[0-9a-f]{32}\.mobj|\.[0-9a-f]{32}\.tmp|\.delete-[0-9a-f]{32}\.tmp))$" +) +_ROTATION_ID_PATTERN = re.compile(r"^[A-Za-z0-9._-]{1,128}$") +_OBJECT_STATES = frozenset({"pending", "active", "deleting", "deleted", "quarantined"}) +_PURGE_LIFECYCLE_STATES = frozenset({"purging", "verified", "failed"}) + + +class ManagedObjectStoreError(RuntimeError): + """Base class containing only stable, privacy-safe public codes.""" + + +class ManagedObjectStoreDisabledError(ManagedObjectStoreError): + pass + + +class ManagedObjectNotFoundError(ManagedObjectStoreError): + pass + + +class ManagedObjectConflictError(ManagedObjectStoreError): + pass + + +class ManagedObjectIntegrityError(ManagedObjectStoreError): + pass + + +class ManagedObjectStoreLimitError(ManagedObjectStoreError): + pass + + +class ManagedObjectKind(str, Enum): + DOCUMENT = "document" + WORKBOOK = "workbook" + EVIDENCE = "evidence" + + +@dataclass(frozen=True) +class ManagedObjectScope: + tenant_id: str + owner_id: str + session_id: str + + def __post_init__(self) -> None: + for label, value in ( + ("tenant_id", self.tenant_id), + ("owner_id", self.owner_id), + ("session_id", self.session_id), + ): + normalized = str(value or "").strip() + if not normalized or len(normalized.encode("utf-8")) > 512: + raise ValueError(f"{label} is required and must be bounded") + object.__setattr__(self, label, normalized) + + +@dataclass(frozen=True) +class ManagedObjectStoreConfig: + enabled: bool + root: Path + current_key_id: str + current_key: bytes + previous_keys: Mapping[str, bytes] + address_key: bytes + max_object_bytes: int = 64 * 1024 * 1024 + total_bytes: int = 1024 * 1024 * 1024 + chunk_bytes: int = 256 * 1024 + + def __post_init__(self) -> None: + if not isinstance(self.enabled, bool): + raise ValueError("enabled must be a boolean") + object.__setattr__(self, "root", Path(self.root)) + object.__setattr__(self, "previous_keys", dict(self.previous_keys)) + if not self.enabled: + return + if not isinstance(self.current_key_id, str): + raise ValueError("Encryption key ID is invalid") + _validate_key_id(self.current_key_id) + _validate_32_byte_key(self.current_key, "current encryption key") + _validate_32_byte_key(self.address_key, "address key") + if hmac.compare_digest(self.current_key, self.address_key): + raise ValueError("Address and encryption keys must be distinct") + if len(self.previous_keys) > 4: + raise ValueError("At most four previous verification keys are supported") + for key_id, key in self.previous_keys.items(): + if not isinstance(key_id, str): + raise ValueError("Encryption key ID is invalid") + _validate_key_id(key_id) + _validate_32_byte_key(key, "previous encryption key") + if key_id == self.current_key_id: + raise ValueError("Current and previous key IDs must be distinct") + if any( + hmac.compare_digest(key, configured) + for configured in (self.current_key, self.address_key) + ): + raise ValueError("Managed-object key roles must be distinct") + previous_material = list(self.previous_keys.values()) + if len({bytes(key) for key in previous_material}) != len(previous_material): + raise ValueError("Previous encryption keys must be distinct") + if isinstance(self.chunk_bytes, bool) or not isinstance(self.chunk_bytes, int): + raise ValueError("chunk_bytes is outside the supported range") + if not 1 <= self.chunk_bytes <= 4 * 1024 * 1024: + raise ValueError("chunk_bytes is outside the supported range") + if isinstance(self.max_object_bytes, bool) or not isinstance(self.max_object_bytes, int): + raise ValueError("max_object_bytes is outside the supported range") + if not 1 <= self.max_object_bytes <= 1024 * 1024 * 1024: + raise ValueError("max_object_bytes is outside the supported range") + if isinstance(self.total_bytes, bool) or not isinstance(self.total_bytes, int): + raise ValueError("total_bytes is outside the supported range") + if not self.max_object_bytes <= self.total_bytes <= 16 * 1024 * 1024 * 1024: + raise ValueError("total_bytes must be at least max_object_bytes") + + @classmethod + def from_environment( + cls, + environ: Mapping[str, str] | None = None, + ) -> ManagedObjectStoreConfig: + env = os.environ if environ is None else environ + enabled = _parse_bool(env.get("MANAGED_OBJECT_STORE_ENABLED", "false")) + root = Path(env.get("MANAGED_OBJECT_STORE_ROOT", "data/managed-objects")) + if not enabled: + return cls(False, root, "", b"", {}, b"") + current_key_id = str(env.get("MANAGED_OBJECT_STORE_CURRENT_KEY_ID", "")).strip() + current_key = _decode_external_key(env.get("MANAGED_OBJECT_STORE_CURRENT_KEY", "")) + address_key = _decode_external_key(env.get("MANAGED_OBJECT_STORE_ADDRESS_KEY", "")) + previous_raw = str(env.get("MANAGED_OBJECT_STORE_PREVIOUS_KEYS", "") or "{}").strip() + try: + previous_payload = json.loads(previous_raw, object_pairs_hook=_unique_json_object) + except (json.JSONDecodeError, ValueError) as exc: + raise ValueError("Previous managed-object keys must be a JSON object") from exc + if not isinstance(previous_payload, dict): + raise ValueError("Previous managed-object keys must be a JSON object") + previous_keys = { + str(key_id): _decode_external_key(encoded) for key_id, encoded in previous_payload.items() if isinstance(encoded, str) + } + if len(previous_keys) != len(previous_payload): + raise ValueError("Previous managed-object keys must be encoded strings") + return cls( + enabled=True, + root=root, + current_key_id=current_key_id, + current_key=current_key, + previous_keys=previous_keys, + address_key=address_key, + max_object_bytes=_bounded_env_int( + env.get("MANAGED_OBJECT_STORE_MAX_OBJECT_BYTES"), + 64 * 1024 * 1024, + 1, + 1024 * 1024 * 1024, + ), + total_bytes=_bounded_env_int( + env.get("MANAGED_OBJECT_STORE_TOTAL_BYTES"), + 1024 * 1024 * 1024, + 1, + 16 * 1024 * 1024 * 1024, + ), + chunk_bytes=_bounded_env_int( + env.get("MANAGED_OBJECT_STORE_CHUNK_BYTES"), + 256 * 1024, + 1, + 4 * 1024 * 1024, + ), + ) + + +@dataclass(frozen=True) +class ManagedObjectReceipt: + object_id: str + object_kind: str + plaintext_size: int + generation: int + + def to_dict(self) -> dict[str, Any]: + return { + "object_id": self.object_id, + "object_kind": self.object_kind, + "plaintext_size": self.plaintext_size, + "generation": self.generation, + } + + +@dataclass(frozen=True) +class ManagedObjectDeletionReceipt: + deleted: bool = True + + def to_dict(self) -> dict[str, bool]: + return {"deleted": self.deleted} + + +@dataclass(frozen=True) +class _ObjectRecord: + object_id: str + tenant_scope: str + owner_scope: str + session_scope: str + session_id: str + object_kind: str + content_identity: str + locator: str + plaintext_size: int + ciphertext_size: int + key_id: str + generation: int + state: str + header_sha256: str + ciphertext_sha256: str + mutation_id: str + created_at: float + updated_at: float + + +@dataclass(frozen=True) +class _PutRequest: + scope: ManagedObjectScope + tenant_scope: str + owner_scope: str + session_scope: str + object_kind: str + content_identity: str + object_id: str + plaintext_size: int + key_id: str + locator: str + mutation_id: str + + +_T = TypeVar("_T") +_Writer = Callable[[_ObjectRecord], Mapping[str, Any]] +_Reader = Callable[[_ObjectRecord], _T] +_Deleter = Callable[[_ObjectRecord], None] +_SessionPurger = Callable[[list[_ObjectRecord]], dict[str, int]] +_Rotator = Callable[[_ObjectRecord], Mapping[str, Any]] + + +class ManagedObjectMetadataRepository(Protocol): + provider_name: str + + def initialize(self) -> None: ... + def put(self, request: _PutRequest, total_budget: int, writer: _Writer) -> _ObjectRecord: ... + def read(self, scope: ManagedObjectScope, bindings: Mapping[str, str], object_id: str, reader: _Reader[_T]) -> _T: ... + def delete(self, scope: ManagedObjectScope, bindings: Mapping[str, str], object_id: str, deleter: _Deleter) -> bool: ... + def purge_session(self, session_id: str, purger: _SessionPurger) -> dict[str, int]: ... + def records(self) -> list[_ObjectRecord]: ... + def rotate(self, old_key_id: str, new_key_id: str, rotator: _Rotator) -> int: ... + def stats(self) -> dict[str, int]: ... + + +class SQLiteManagedObjectMetadataRepository: + """SQLite metadata adapter sharing the lifecycle database transaction.""" + + provider_name = "sqlite" + + def __init__(self, db_path: Path | str) -> None: + self.db_path = Path(db_path) + self._schema_lock = threading.Lock() + + @contextlib.contextmanager + def _connection(self) -> Iterator[sqlite3.Connection]: + conn = sqlite3.connect(self.db_path, timeout=30, check_same_thread=False) + conn.row_factory = sqlite3.Row + conn.execute("PRAGMA foreign_keys = ON") + conn.execute("PRAGMA busy_timeout = 30000") + try: + yield conn + finally: + conn.close() + + @contextlib.contextmanager + def _transaction(self) -> Iterator[sqlite3.Connection]: + with self._connection() as conn: + conn.execute("BEGIN IMMEDIATE") + try: + yield conn + conn.commit() + except BaseException: + conn.rollback() + raise + + def initialize(self) -> None: + with self._schema_lock, self._connection() as conn: + required = { + str(row[0]) + for row in conn.execute( + "SELECT name FROM sqlite_master WHERE type = 'table' AND name IN ('sessions', 'session_lifecycle')" + ).fetchall() + } + if required != {"sessions", "session_lifecycle"}: + raise RuntimeError("Managed-object metadata requires an initialized lifecycle store") + conn.execute( + "CREATE TABLE IF NOT EXISTS managed_object_store_schema (schema_version TEXT PRIMARY KEY, installed_at REAL NOT NULL)" + ) + installed_versions = {str(row[0]) for row in conn.execute("SELECT schema_version FROM managed_object_store_schema").fetchall()} + if not installed_versions: + conn.execute( + "INSERT INTO managed_object_store_schema (schema_version, installed_at) VALUES (?, ?)", + (_FORMAT_VERSION, time.time()), + ) + elif installed_versions != {_FORMAT_VERSION}: + raise RuntimeError("Managed-object metadata schema version is incompatible") + conn.executescript( + """ + CREATE TABLE IF NOT EXISTS managed_encrypted_objects ( + object_id TEXT PRIMARY KEY, + tenant_scope TEXT NOT NULL, + owner_scope TEXT NOT NULL, + session_scope TEXT NOT NULL, + session_id TEXT NOT NULL REFERENCES sessions(session_id) ON DELETE RESTRICT, + object_kind TEXT NOT NULL CHECK (object_kind IN ('document', 'workbook', 'evidence')), + content_identity TEXT NOT NULL, + locator TEXT NOT NULL UNIQUE, + plaintext_size INTEGER NOT NULL CHECK (plaintext_size >= 0), + ciphertext_size INTEGER NOT NULL DEFAULT 0 CHECK (ciphertext_size >= 0), + key_id TEXT NOT NULL, + generation INTEGER NOT NULL DEFAULT 1 CHECK (generation >= 1), + state TEXT NOT NULL CHECK ( + state IN ('pending', 'active', 'deleting', 'deleted', 'quarantined') + ), + header_sha256 TEXT NOT NULL DEFAULT '', + ciphertext_sha256 TEXT NOT NULL DEFAULT '', + mutation_id TEXT NOT NULL, + created_at REAL NOT NULL, + updated_at REAL NOT NULL, + UNIQUE ( + tenant_scope, owner_scope, session_scope, object_kind, content_identity + ) + ); + CREATE INDEX IF NOT EXISTS idx_managed_objects_session + ON managed_encrypted_objects(session_id, state); + CREATE INDEX IF NOT EXISTS idx_managed_objects_key + ON managed_encrypted_objects(key_id, state); + + CREATE TRIGGER IF NOT EXISTS trg_managed_objects_active_insert + BEFORE INSERT ON managed_encrypted_objects + BEGIN + SELECT CASE WHEN NOT EXISTS ( + SELECT 1 FROM session_lifecycle AS lifecycle + WHERE lifecycle.session_id = NEW.session_id AND lifecycle.state = 'active' + ) THEN RAISE(ABORT, 'session lifecycle is not active') END; + END; + + CREATE TRIGGER IF NOT EXISTS trg_managed_objects_session_immutable + BEFORE UPDATE OF session_id ON managed_encrypted_objects + WHEN OLD.session_id <> NEW.session_id + BEGIN + SELECT RAISE(ABORT, 'session_id is immutable'); + END; + """ + ) + conn.commit() + + def _authorize( + self, + conn: sqlite3.Connection, + scope: ManagedObjectScope, + *, + require_active: bool = True, + ) -> None: + row = conn.execute( + "SELECT s.owner_user_id, lifecycle.state FROM sessions AS s " + "JOIN session_lifecycle AS lifecycle ON lifecycle.session_id = s.session_id " + "WHERE s.session_id = ?", + (scope.session_id,), + ).fetchone() + if ( + row is None + or (require_active and str(row["state"] or "") != "active") + or not hmac.compare_digest(str(row["owner_user_id"] or ""), scope.owner_id) + ): + raise ManagedObjectNotFoundError("managed_object_not_found") + + def put(self, request: _PutRequest, total_budget: int, writer: _Writer) -> _ObjectRecord: + with self._transaction() as conn: + self._authorize(conn, request.scope) + row = conn.execute( + "SELECT * FROM managed_encrypted_objects WHERE object_id = ?", + (request.object_id,), + ).fetchone() + if row is not None: + record = _sqlite_record(row) + if not _record_matches_request(record, request): + raise ManagedObjectConflictError("managed_object_conflict") + if record.state == "active": + return record + if record.state == "deleted": + raise ManagedObjectConflictError("managed_object_tombstoned") + raise ManagedObjectConflictError("managed_object_conflict") + + used = int( + conn.execute( + "SELECT COALESCE(SUM(plaintext_size), 0) FROM managed_encrypted_objects " + "WHERE state IN ('pending', 'active', 'deleting')" + ).fetchone()[0] + ) + if used + request.plaintext_size > total_budget: + raise ManagedObjectStoreLimitError("managed_object_store_limit") + now = time.time() + conn.execute( + """ + INSERT INTO managed_encrypted_objects ( + object_id, tenant_scope, owner_scope, session_scope, session_id, + object_kind, content_identity, locator, plaintext_size, + ciphertext_size, key_id, generation, state, header_sha256, + ciphertext_sha256, mutation_id, created_at, updated_at + ) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, 0, ?, 1, 'pending', '', '', ?, ?, ?) + """, + ( + request.object_id, + request.tenant_scope, + request.owner_scope, + request.session_scope, + request.scope.session_id, + request.object_kind, + request.content_identity, + request.locator, + request.plaintext_size, + request.key_id, + request.mutation_id, + now, + now, + ), + ) + pending = _sqlite_record( + conn.execute( + "SELECT * FROM managed_encrypted_objects WHERE object_id = ?", + (request.object_id,), + ).fetchone() + ) + stored = dict(writer(pending)) + result = conn.execute( + "UPDATE managed_encrypted_objects SET state = 'active', ciphertext_size = ?, " + "header_sha256 = ?, ciphertext_sha256 = ?, updated_at = ? " + "WHERE object_id = ? AND state = 'pending' AND mutation_id = ?", + ( + _strict_nonnegative_int(stored.get("ciphertext_size")), + _strict_sha256(stored.get("header_sha256")), + _strict_sha256(stored.get("ciphertext_sha256")), + time.time(), + request.object_id, + request.mutation_id, + ), + ) + if result.rowcount != 1: + raise ManagedObjectConflictError("managed_object_conflict") + return _sqlite_record( + conn.execute( + "SELECT * FROM managed_encrypted_objects WHERE object_id = ?", + (request.object_id,), + ).fetchone() + ) + + def read( + self, + scope: ManagedObjectScope, + bindings: Mapping[str, str], + object_id: str, + reader: _Reader[_T], + ) -> _T: + with self._transaction() as conn: + self._authorize(conn, scope) + row = conn.execute( + "SELECT * FROM managed_encrypted_objects WHERE object_id = ? AND state = 'active'", + (object_id,), + ).fetchone() + if row is None: + raise ManagedObjectNotFoundError("managed_object_not_found") + record = _sqlite_record(row) + if not _record_matches_scope(record, scope, bindings): + raise ManagedObjectNotFoundError("managed_object_not_found") + return reader(record) + + def delete( + self, + scope: ManagedObjectScope, + bindings: Mapping[str, str], + object_id: str, + deleter: _Deleter, + ) -> bool: + with self._transaction() as conn: + self._authorize(conn, scope) + row = conn.execute( + "SELECT * FROM managed_encrypted_objects WHERE object_id = ?", + (object_id,), + ).fetchone() + if row is None: + raise ManagedObjectNotFoundError("managed_object_not_found") + record = _sqlite_record(row) + if not _record_matches_scope(record, scope, bindings): + raise ManagedObjectNotFoundError("managed_object_not_found") + if record.state == "deleted": + return True + if record.state != "active": + raise ManagedObjectConflictError("managed_object_conflict") + conn.execute( + "UPDATE managed_encrypted_objects SET state = 'deleting', updated_at = ? WHERE object_id = ? AND state = 'active'", + (time.time(), object_id), + ) + deleter(record) + result = conn.execute( + "UPDATE managed_encrypted_objects SET state = 'deleted', ciphertext_size = 0, " + "header_sha256 = '', ciphertext_sha256 = '', updated_at = ? " + "WHERE object_id = ? AND state = 'deleting'", + (time.time(), object_id), + ) + if result.rowcount != 1: + raise ManagedObjectConflictError("managed_object_conflict") + return True + + def purge_session(self, session_id: str, purger: _SessionPurger) -> dict[str, int]: + normalized = str(session_id or "").strip() + if not normalized: + raise ValueError("session_id is required") + with self._transaction() as conn: + lifecycle = conn.execute( + "SELECT state FROM session_lifecycle WHERE session_id = ?", + (normalized,), + ).fetchone() + if lifecycle is None or str(lifecycle["state"] or "") not in _PURGE_LIFECYCLE_STATES: + raise RuntimeError("Managed-object purge requires a lifecycle tombstone") + rows = conn.execute( + "SELECT * FROM managed_encrypted_objects WHERE session_id = ? ORDER BY object_id", + (normalized,), + ).fetchall() + report = purger([_sqlite_record(row) for row in rows]) + if int(report.get("residual_files", -1)) != 0: + raise RuntimeError("Managed-object purge left residual ciphertext") + conn.execute( + "DELETE FROM managed_encrypted_objects WHERE session_id = ?", + (normalized,), + ) + remaining = int( + conn.execute( + "SELECT COUNT(*) FROM managed_encrypted_objects WHERE session_id = ?", + (normalized,), + ).fetchone()[0] + ) + if remaining: + raise RuntimeError("Managed-object purge inventory remains") + return report + + def records(self) -> list[_ObjectRecord]: + with self._connection() as conn: + rows = conn.execute("SELECT * FROM managed_encrypted_objects ORDER BY object_id").fetchall() + return [_sqlite_record(row) for row in rows] + + def rotate(self, old_key_id: str, new_key_id: str, rotator: _Rotator) -> int: + with self._transaction() as conn: + rows = conn.execute( + "SELECT objects.* FROM managed_encrypted_objects AS objects " + "JOIN session_lifecycle AS lifecycle ON lifecycle.session_id = objects.session_id " + "WHERE objects.state = 'active' AND objects.key_id = ? " + "AND lifecycle.state = 'active' ORDER BY objects.object_id", + (old_key_id,), + ).fetchall() + rotated = 0 + for row in rows: + record = _sqlite_record(row) + stored = dict(rotator(record)) + result = conn.execute( + "UPDATE managed_encrypted_objects SET key_id = ?, generation = generation + 1, " + "ciphertext_size = ?, header_sha256 = ?, ciphertext_sha256 = ?, updated_at = ? " + "WHERE object_id = ? AND state = 'active' AND key_id = ? AND generation = ?", + ( + new_key_id, + _strict_nonnegative_int(stored.get("ciphertext_size")), + _strict_sha256(stored.get("header_sha256")), + _strict_sha256(stored.get("ciphertext_sha256")), + time.time(), + record.object_id, + old_key_id, + record.generation, + ), + ) + if result.rowcount != 1: + raise ManagedObjectConflictError("managed_object_rotation_conflict") + rotated += 1 + return rotated + + def stats(self) -> dict[str, int]: + with self._connection() as conn: + row = conn.execute( + "SELECT COUNT(*) AS objects, COALESCE(SUM(plaintext_size), 0) AS plaintext_bytes, " + "COALESCE(SUM(ciphertext_size), 0) AS ciphertext_bytes " + "FROM managed_encrypted_objects WHERE state = 'active'" + ).fetchone() + return { + "objects": int(row["objects"]), + "plaintext_bytes": int(row["plaintext_bytes"]), + "ciphertext_bytes": int(row["ciphertext_bytes"]), + } + + +class PostgresManagedObjectMetadataRepository: + """PostgreSQL adapter with the same CAS and lifecycle transaction boundary.""" + + provider_name = "postgres" + + def __init__(self, dsn: str) -> None: + self._dsn = str(dsn or "").strip() + if not self._dsn: + raise ValueError("PostgreSQL DSN is required") + self._sync_loop: asyncio.AbstractEventLoop | None = None + self._sync_thread: threading.Thread | None = None + self._sync_lock = threading.Lock() + + def _run(self, coroutine: Any) -> Any: + try: + asyncio.get_running_loop() + except RuntimeError: + return asyncio.run(coroutine) + with self._sync_lock: + if self._sync_loop is None or self._sync_thread is None or not self._sync_thread.is_alive(): + ready = threading.Event() + + def run_loop() -> None: + loop = asyncio.new_event_loop() + asyncio.set_event_loop(loop) + self._sync_loop = loop + ready.set() + loop.run_forever() + + self._sync_thread = threading.Thread( + target=run_loop, + name="managed-object-postgres-loop", + daemon=True, + ) + self._sync_thread.start() + if not ready.wait(timeout=5): + raise RuntimeError("Managed-object PostgreSQL event loop did not start") + loop = self._sync_loop + if loop is None: + raise RuntimeError("Managed-object PostgreSQL event loop is unavailable") + future = asyncio.run_coroutine_threadsafe(coroutine, loop) + try: + return future.result(timeout=60) + except TimeoutError as exc: + future.cancel() + raise RuntimeError("Managed-object PostgreSQL operation timed out") from exc + + async def _connect(self) -> Any: + try: + import asyncpg + except ImportError as exc: + raise RuntimeError("asyncpg is required for PostgreSQL metadata") from exc + return await asyncpg.connect(dsn=self._dsn, command_timeout=30) + + def initialize(self) -> None: + self._run(self._initialize()) + + async def _initialize(self) -> None: + conn = await self._connect() + try: + present = await conn.fetchval( + "SELECT COUNT(*) FROM information_schema.tables WHERE table_schema = current_schema() AND table_name = ANY($1::text[])", + ["sessions", "session_lifecycle"], + ) + if int(present or 0) != 2: + raise RuntimeError("Managed-object metadata requires an initialized lifecycle store") + await conn.execute( + "CREATE TABLE IF NOT EXISTS managed_object_store_schema (" + "schema_version TEXT PRIMARY KEY, installed_at DOUBLE PRECISION NOT NULL)" + ) + installed_versions = { + str(row["schema_version"]) for row in await conn.fetch("SELECT schema_version FROM managed_object_store_schema") + } + if not installed_versions: + await conn.execute( + "INSERT INTO managed_object_store_schema (schema_version, installed_at) VALUES ($1, $2)", + _FORMAT_VERSION, + time.time(), + ) + elif installed_versions != {_FORMAT_VERSION}: + raise RuntimeError("Managed-object metadata schema version is incompatible") + await conn.execute( + """ + CREATE TABLE IF NOT EXISTS managed_encrypted_objects ( + object_id TEXT PRIMARY KEY, + tenant_scope TEXT NOT NULL, + owner_scope TEXT NOT NULL, + session_scope TEXT NOT NULL, + session_id TEXT NOT NULL REFERENCES sessions(session_id) ON DELETE RESTRICT, + object_kind TEXT NOT NULL CHECK (object_kind IN ('document', 'workbook', 'evidence')), + content_identity TEXT NOT NULL, + locator TEXT NOT NULL UNIQUE, + plaintext_size BIGINT NOT NULL CHECK (plaintext_size >= 0), + ciphertext_size BIGINT NOT NULL DEFAULT 0 CHECK (ciphertext_size >= 0), + key_id TEXT NOT NULL, + generation BIGINT NOT NULL DEFAULT 1 CHECK (generation >= 1), + state TEXT NOT NULL CHECK ( + state IN ('pending', 'active', 'deleting', 'deleted', 'quarantined') + ), + header_sha256 TEXT NOT NULL DEFAULT '', + ciphertext_sha256 TEXT NOT NULL DEFAULT '', + mutation_id TEXT NOT NULL, + created_at DOUBLE PRECISION NOT NULL, + updated_at DOUBLE PRECISION NOT NULL, + UNIQUE ( + tenant_scope, owner_scope, session_scope, object_kind, content_identity + ) + ); + CREATE INDEX IF NOT EXISTS idx_managed_objects_session + ON managed_encrypted_objects(session_id, state); + CREATE INDEX IF NOT EXISTS idx_managed_objects_key + ON managed_encrypted_objects(key_id, state); + """ + ) + finally: + await conn.close() + + async def _authorize(self, conn: Any, scope: ManagedObjectScope, require_active: bool = True) -> None: + await conn.execute("SELECT pg_advisory_xact_lock(hashtextextended($1, 0))", scope.session_id) + row = await conn.fetchrow( + "SELECT s.owner_user_id, lifecycle.state FROM sessions AS s " + "JOIN session_lifecycle AS lifecycle ON lifecycle.session_id = s.session_id " + "WHERE s.session_id = $1 FOR UPDATE OF s, lifecycle", + scope.session_id, + ) + if ( + row is None + or (require_active and str(row["state"] or "") != "active") + or not hmac.compare_digest(str(row["owner_user_id"] or ""), scope.owner_id) + ): + raise ManagedObjectNotFoundError("managed_object_not_found") + + def put(self, request: _PutRequest, total_budget: int, writer: _Writer) -> _ObjectRecord: + return self._run(self._put(request, total_budget, writer)) + + async def _put(self, request: _PutRequest, total_budget: int, writer: _Writer) -> _ObjectRecord: + conn = await self._connect() + try: + async with conn.transaction(): + await self._authorize(conn, request.scope) + await conn.execute( + "SELECT pg_advisory_xact_lock(hashtextextended($1, 0))", + "managed-object-global-budget", + ) + row = await conn.fetchrow( + "SELECT * FROM managed_encrypted_objects WHERE object_id = $1 FOR UPDATE", + request.object_id, + ) + if row is not None: + record = _mapping_record(row) + if not _record_matches_request(record, request): + raise ManagedObjectConflictError("managed_object_conflict") + if record.state == "active": + return record + if record.state == "deleted": + raise ManagedObjectConflictError("managed_object_tombstoned") + raise ManagedObjectConflictError("managed_object_conflict") + used = int( + await conn.fetchval( + "SELECT COALESCE(SUM(plaintext_size), 0) FROM managed_encrypted_objects WHERE state = ANY($1::text[])", + ["pending", "active", "deleting"], + ) + or 0 + ) + if used + request.plaintext_size > total_budget: + raise ManagedObjectStoreLimitError("managed_object_store_limit") + now = time.time() + row = await conn.fetchrow( + """ + INSERT INTO managed_encrypted_objects ( + object_id, tenant_scope, owner_scope, session_scope, session_id, + object_kind, content_identity, locator, plaintext_size, + ciphertext_size, key_id, generation, state, header_sha256, + ciphertext_sha256, mutation_id, created_at, updated_at + ) VALUES ($1,$2,$3,$4,$5,$6,$7,$8,$9,0,$10,1,'pending','',$11,$12,$13,$13) + RETURNING * + """, + request.object_id, + request.tenant_scope, + request.owner_scope, + request.session_scope, + request.scope.session_id, + request.object_kind, + request.content_identity, + request.locator, + request.plaintext_size, + request.key_id, + "", + request.mutation_id, + now, + ) + pending = _mapping_record(row) + stored = dict(writer(pending)) + row = await conn.fetchrow( + "UPDATE managed_encrypted_objects SET state='active', ciphertext_size=$1, " + "header_sha256=$2, ciphertext_sha256=$3, updated_at=$4 " + "WHERE object_id=$5 AND state='pending' AND mutation_id=$6 RETURNING *", + _strict_nonnegative_int(stored.get("ciphertext_size")), + _strict_sha256(stored.get("header_sha256")), + _strict_sha256(stored.get("ciphertext_sha256")), + time.time(), + request.object_id, + request.mutation_id, + ) + if row is None: + raise ManagedObjectConflictError("managed_object_conflict") + return _mapping_record(row) + finally: + await conn.close() + + def read(self, scope: ManagedObjectScope, bindings: Mapping[str, str], object_id: str, reader: _Reader[_T]) -> _T: + return self._run(self._read(scope, bindings, object_id, reader)) + + async def _read(self, scope: ManagedObjectScope, bindings: Mapping[str, str], object_id: str, reader: _Reader[_T]) -> _T: + conn = await self._connect() + try: + async with conn.transaction(): + await self._authorize(conn, scope) + row = await conn.fetchrow( + "SELECT * FROM managed_encrypted_objects WHERE object_id=$1 AND state='active' FOR SHARE", + object_id, + ) + if row is None: + raise ManagedObjectNotFoundError("managed_object_not_found") + record = _mapping_record(row) + if not _record_matches_scope(record, scope, bindings): + raise ManagedObjectNotFoundError("managed_object_not_found") + return reader(record) + finally: + await conn.close() + + def delete(self, scope: ManagedObjectScope, bindings: Mapping[str, str], object_id: str, deleter: _Deleter) -> bool: + return self._run(self._delete(scope, bindings, object_id, deleter)) + + async def _delete(self, scope: ManagedObjectScope, bindings: Mapping[str, str], object_id: str, deleter: _Deleter) -> bool: + conn = await self._connect() + try: + async with conn.transaction(): + await self._authorize(conn, scope) + row = await conn.fetchrow( + "SELECT * FROM managed_encrypted_objects WHERE object_id=$1 FOR UPDATE", + object_id, + ) + if row is None: + raise ManagedObjectNotFoundError("managed_object_not_found") + record = _mapping_record(row) + if not _record_matches_scope(record, scope, bindings): + raise ManagedObjectNotFoundError("managed_object_not_found") + if record.state == "deleted": + return True + if record.state != "active": + raise ManagedObjectConflictError("managed_object_conflict") + await conn.execute( + "UPDATE managed_encrypted_objects SET state='deleting', updated_at=$1 WHERE object_id=$2", + time.time(), + object_id, + ) + deleter(record) + result = await conn.execute( + "UPDATE managed_encrypted_objects SET state='deleted', ciphertext_size=0, " + "header_sha256='', ciphertext_sha256='', updated_at=$1 " + "WHERE object_id=$2 AND state='deleting'", + time.time(), + object_id, + ) + if not str(result).endswith(" 1"): + raise ManagedObjectConflictError("managed_object_conflict") + return True + finally: + await conn.close() + + def purge_session(self, session_id: str, purger: _SessionPurger) -> dict[str, int]: + return self._run(self._purge_session(session_id, purger)) + + async def _purge_session(self, session_id: str, purger: _SessionPurger) -> dict[str, int]: + normalized = str(session_id or "").strip() + if not normalized: + raise ValueError("session_id is required") + conn = await self._connect() + try: + async with conn.transaction(): + await conn.execute("SELECT pg_advisory_xact_lock(hashtextextended($1, 0))", normalized) + lifecycle = await conn.fetchval( + "SELECT state FROM session_lifecycle WHERE session_id=$1 FOR UPDATE", + normalized, + ) + if str(lifecycle or "") not in _PURGE_LIFECYCLE_STATES: + raise RuntimeError("Managed-object purge requires a lifecycle tombstone") + rows = await conn.fetch( + "SELECT * FROM managed_encrypted_objects WHERE session_id=$1 ORDER BY object_id FOR UPDATE", + normalized, + ) + report = purger([_mapping_record(row) for row in rows]) + if int(report.get("residual_files", -1)) != 0: + raise RuntimeError("Managed-object purge left residual ciphertext") + await conn.execute("DELETE FROM managed_encrypted_objects WHERE session_id=$1", normalized) + if int(await conn.fetchval("SELECT COUNT(*) FROM managed_encrypted_objects WHERE session_id=$1", normalized) or 0): + raise RuntimeError("Managed-object purge inventory remains") + return report + finally: + await conn.close() + + def records(self) -> list[_ObjectRecord]: + return self._run(self._records()) + + async def _records(self) -> list[_ObjectRecord]: + conn = await self._connect() + try: + rows = await conn.fetch("SELECT * FROM managed_encrypted_objects ORDER BY object_id") + return [_mapping_record(row) for row in rows] + finally: + await conn.close() + + def rotate(self, old_key_id: str, new_key_id: str, rotator: _Rotator) -> int: + return self._run(self._rotate(old_key_id, new_key_id, rotator)) + + async def _rotate(self, old_key_id: str, new_key_id: str, rotator: _Rotator) -> int: + conn = await self._connect() + try: + async with conn.transaction(): + session_rows = await conn.fetch( + "SELECT DISTINCT objects.session_id FROM managed_encrypted_objects AS objects " + "JOIN session_lifecycle AS lifecycle " + "ON lifecycle.session_id=objects.session_id " + "WHERE objects.state='active' AND objects.key_id=$1 " + "AND lifecycle.state='active' ORDER BY objects.session_id", + old_key_id, + ) + for session_row in session_rows: + await conn.execute( + "SELECT pg_advisory_xact_lock(hashtextextended($1, 0))", + str(session_row["session_id"]), + ) + rows = await conn.fetch( + "SELECT objects.* FROM managed_encrypted_objects AS objects " + "JOIN session_lifecycle AS lifecycle ON lifecycle.session_id=objects.session_id " + "WHERE objects.state='active' AND objects.key_id=$1 AND lifecycle.state='active' " + "ORDER BY objects.object_id FOR UPDATE OF objects", + old_key_id, + ) + rotated = 0 + for row in rows: + record = _mapping_record(row) + stored = dict(rotator(record)) + result = await conn.execute( + "UPDATE managed_encrypted_objects SET key_id=$1, generation=generation+1, " + "ciphertext_size=$2, header_sha256=$3, ciphertext_sha256=$4, updated_at=$5 " + "WHERE object_id=$6 AND state='active' AND key_id=$7 AND generation=$8", + new_key_id, + _strict_nonnegative_int(stored.get("ciphertext_size")), + _strict_sha256(stored.get("header_sha256")), + _strict_sha256(stored.get("ciphertext_sha256")), + time.time(), + record.object_id, + old_key_id, + record.generation, + ) + if not str(result).endswith(" 1"): + raise ManagedObjectConflictError("managed_object_rotation_conflict") + rotated += 1 + return rotated + finally: + await conn.close() + + def stats(self) -> dict[str, int]: + return self._run(self._stats()) + + async def _stats(self) -> dict[str, int]: + conn = await self._connect() + try: + row = await conn.fetchrow( + "SELECT COUNT(*) AS objects, COALESCE(SUM(plaintext_size),0) AS plaintext_bytes, " + "COALESCE(SUM(ciphertext_size),0) AS ciphertext_bytes " + "FROM managed_encrypted_objects WHERE state='active'" + ) + return {key: int(row[key]) for key in ("objects", "plaintext_bytes", "ciphertext_bytes")} + finally: + await conn.close() + + +class EncryptedManagedObjectStore: + """Bounded, content-addressed AES-GCM object storage.""" + + def __init__( + self, + config: ManagedObjectStoreConfig, + repository: ManagedObjectMetadataRepository, + ) -> None: + if not config.enabled: + raise ManagedObjectStoreDisabledError("managed_object_store_disabled") + self._config = config + self._repository = repository + self._current_key_id = config.current_key_id + self._keys: dict[str, bytes] = { + config.current_key_id: bytes(config.current_key), + **{key_id: bytes(key) for key_id, key in config.previous_keys.items()}, + } + self._address_key = bytes(config.address_key) + self._root = _initialize_secure_root(config.root) + root_info = self._root.lstat() + self._root_identity = (root_info.st_dev, root_info.st_ino) + root_flags = os.O_RDONLY + if hasattr(os, "O_DIRECTORY"): + root_flags |= os.O_DIRECTORY + if hasattr(os, "O_NOFOLLOW"): + root_flags |= os.O_NOFOLLOW + self._root_fd = os.open(self._root, root_flags) + opened_root = os.fstat(self._root_fd) + if (opened_root.st_dev, opened_root.st_ino) != self._root_identity: + os.close(self._root_fd) + raise ManagedObjectIntegrityError("managed_object_root_identity_invalid") + self._repository.initialize() + + @property + def provider_name(self) -> str: + return self._repository.provider_name + + def put_bytes( + self, + scope: ManagedObjectScope, + object_kind: ManagedObjectKind | str, + payload: bytes, + ) -> ManagedObjectReceipt: + if not isinstance(payload, bytes): + raise TypeError("payload must be bytes") + return self.put_stream(scope, object_kind, io.BytesIO(payload)) + + def put_stream( + self, + scope: ManagedObjectScope, + object_kind: ManagedObjectKind | str, + source: BinaryIO, + ) -> ManagedObjectReceipt: + kind = _normalize_kind(object_kind) + if not callable(getattr(source, "seekable", None)) or not source.seekable(): + raise ValueError("Managed-object input must be a seekable binary stream") + try: + start = int(source.tell()) + except (AttributeError, OSError, TypeError, ValueError) as exc: + raise ValueError("Managed-object input must be a seekable binary stream") from exc + plaintext_sha, size = self._hash_source(source, start) + bindings = self._scope_bindings(scope) + identity_payload = _canonical_json_bytes( + { + "kind": kind, + "ownerScope": bindings["owner_scope"], + "plaintextSha256": plaintext_sha, + "schemaVersion": _FORMAT_VERSION, + "sessionScope": bindings["session_scope"], + "tenantScope": bindings["tenant_scope"], + } + ) + content_identity = hmac.new(self._address_key, identity_payload, hashlib.sha256).hexdigest() + object_id = ( + "obj_" + + hmac.new( + self._address_key, + b"managed-object-id\x00" + identity_payload, + hashlib.sha256, + ).hexdigest() + ) + locator = f"{bindings['session_scope'][:32]}/{uuid.uuid4().hex}.mobj" + request = _PutRequest( + scope=scope, + tenant_scope=bindings["tenant_scope"], + owner_scope=bindings["owner_scope"], + session_scope=bindings["session_scope"], + object_kind=kind, + content_identity=content_identity, + object_id=object_id, + plaintext_size=size, + key_id=self._current_key_id, + locator=locator, + mutation_id=uuid.uuid4().hex, + ) + + def writer(record: _ObjectRecord) -> Mapping[str, Any]: + source.seek(start) + return self._encrypt_record(source, record, self._current_key_id, self._keys[self._current_key_id]) + + with self._file_lock(exclusive=True): + record = self._repository.put(request, self._config.total_bytes, writer) + self._verify_record_file_metadata(record) + return _public_receipt(record) + + def get_bytes(self, scope: ManagedObjectScope, object_id: str) -> bytes: + normalized_id = _normalize_object_id(object_id) + bindings = self._scope_bindings(scope) + with self._file_lock(exclusive=False): + return self._repository.read( + scope, + bindings, + normalized_id, + self._decrypt_record, + ) + + def delete( + self, + scope: ManagedObjectScope, + object_id: str, + ) -> ManagedObjectDeletionReceipt: + normalized_id = _normalize_object_id(object_id) + bindings = self._scope_bindings(scope) + with self._file_lock(exclusive=True): + self._repository.delete(scope, bindings, normalized_id, self._delete_record_file) + return ManagedObjectDeletionReceipt() + + def purge_session(self, session_id: str) -> dict[str, int]: + normalized = str(session_id or "").strip() + if not normalized: + raise ValueError("session_id is required") + session_scope = self._scope_hmac("session", normalized) + + def purger(records: list[_ObjectRecord]) -> dict[str, int]: + deleted = 0 + for record in records: + deleted += int(self._delete_locator(record.locator)) + deleted += self._purge_scoped_directory_files("objects", session_scope[:32]) + quarantined_deleted = self._purge_scoped_directory_files( + "quarantine", + session_scope[:32], + ) + return { + "objects_deleted": deleted, + "quarantined_deleted": quarantined_deleted, + "residual_files": 0, + } + + with self._file_lock(exclusive=True): + return self._repository.purge_session(normalized, purger) + + def register_lifecycle_purger(self, coordinator: Any) -> None: + register = getattr(coordinator, "register_purger", None) + if not callable(register): + raise TypeError("Lifecycle coordinator does not support purger registration") + + async def purge_managed_objects(session_id: str) -> dict[str, int]: + return await asyncio.to_thread(self.purge_session, session_id) + + register("managed_encrypted_objects", purge_managed_objects) + + def recover(self) -> dict[str, int]: + with self._file_lock(exclusive=True): + records = self._repository.records() + known = {record.locator for record in records if record.state != "deleted"} + quarantined = 0 + invalid_metadata = 0 + for candidate in _regular_files(self._root / "objects"): + relative = candidate.relative_to(self._root / "objects").as_posix() + if relative in known: + continue + try: + self._quarantine_ciphertext(relative) + quarantined += 1 + except (OSError, ValueError): + invalid_metadata += 1 + missing = sum(1 for record in records if record.state == "active" and not self._locator_exists(record.locator)) + return { + "orphans_quarantined": quarantined, + "active_files_missing": missing, + "recovery_failures": invalid_metadata, + } + + def rotate( + self, + *, + rotation_id: str, + new_key_id: str, + new_key: bytes, + ) -> dict[str, Any]: + normalized_rotation_id = str(rotation_id or "").strip() + if not _ROTATION_ID_PATTERN.fullmatch(normalized_rotation_id): + raise ValueError("rotation_id is invalid") + _validate_key_id(new_key_id) + _validate_32_byte_key(new_key, "new encryption key") + if hmac.compare_digest(new_key, self._address_key): + raise ValueError("Managed-object key roles must be distinct") + with self._file_lock(exclusive=True): + completed = self._load_rotation_receipt(normalized_rotation_id) + if completed is not None: + configured = self._keys.get(new_key_id) + expected_key_id_hash = hashlib.sha256(new_key_id.encode()).hexdigest() + old_key_id = self._key_id_for_hash(str(completed["old_key_id_sha256"])) + configured_old = self._keys[old_key_id] + if ( + configured is None + or not hmac.compare_digest(bytes(new_key), configured) + or old_key_id == new_key_id + or not hmac.compare_digest( + str(completed["old_key_commitment"]), + self._rotation_key_commitment(old_key_id, configured_old), + ) + or not hmac.compare_digest( + str(completed["new_key_commitment"]), + self._rotation_key_commitment(new_key_id, bytes(new_key)), + ) + or not hmac.compare_digest( + str(completed.get("new_key_id_sha256") or ""), + expected_key_id_hash, + ) + ): + raise ManagedObjectConflictError("managed_object_rotation_conflict") + return completed + + pending = self._load_rotation_journal(normalized_rotation_id) + if pending is None: + if new_key_id in self._keys or any( + hmac.compare_digest(new_key, configured) + for configured in self._keys.values() + ): + raise ValueError("Rotation requires distinct old and new key roles") + old_key_id = self._current_key_id + candidates = [record for record in self._repository.records() if record.state == "active" and record.key_id == old_key_id] + target_tokens = sorted(self._rotation_target_token(record.locator) for record in candidates) + pending = self._write_rotation_journal( + normalized_rotation_id, + old_key_id=old_key_id, + old_key=self._keys[old_key_id], + new_key_id=new_key_id, + new_key=bytes(new_key), + target_tokens=target_tokens, + ) + else: + old_key_id = self._key_id_for_hash(str(pending["old_key_id_sha256"])) + if ( + not hmac.compare_digest( + str(pending["new_key_id_sha256"]), + hashlib.sha256(new_key_id.encode()).hexdigest(), + ) + or old_key_id == new_key_id + ): + raise ManagedObjectConflictError("managed_object_rotation_conflict") + target_tokens = list(pending["target_tokens"]) + + old_key = self._keys.get(old_key_id) + configured_new = self._keys.get(new_key_id) + if ( + old_key is None + or hmac.compare_digest(old_key, new_key) + or (configured_new is not None and not hmac.compare_digest(configured_new, new_key)) + or not hmac.compare_digest( + str(pending["old_key_commitment"]), + self._rotation_key_commitment(old_key_id, old_key), + ) + or not hmac.compare_digest( + str(pending["new_key_commitment"]), + self._rotation_key_commitment(new_key_id, bytes(new_key)), + ) + ): + raise ManagedObjectConflictError("managed_object_rotation_conflict") + key_ring = dict(self._keys) + key_ring[new_key_id] = bytes(new_key) + + def rotator(record: _ObjectRecord) -> Mapping[str, Any]: + header = self._read_header(record.locator) + header_key_id = str(header.get("keyId") or "") + if header_key_id == new_key_id: + plaintext = self._decrypt_record( + record, + key_ring=key_ring, + allow_generation=record.generation + 1, + ) + else: + if header_key_id != old_key_id: + raise ManagedObjectIntegrityError("managed_object_integrity_failed") + plaintext = self._decrypt_record(record, key_ring=key_ring) + upgraded = _ObjectRecord( + **{ + **record.__dict__, + "key_id": new_key_id, + "generation": record.generation + 1, + } + ) + if header_key_id == new_key_id: + return self._file_metadata(record.locator) + return self._encrypt_record( + io.BytesIO(plaintext), + upgraded, + new_key_id, + bytes(new_key), + ) + + # The signed pending journal is intentionally left in place on + # every exception so a retry can reconcile an ambiguous replace. + self._repository.rotate(old_key_id, new_key_id, rotator) + target_set = set(target_tokens) + records_after_rotation = self._repository.records() + surviving_targets = { + self._rotation_target_token(record.locator): record + for record in records_after_rotation + if self._rotation_target_token(record.locator) in target_set + } + old_remaining = [record for record in surviving_targets.values() if record.state == "active" and record.key_id == old_key_id] + if old_remaining: + raise ManagedObjectConflictError("managed_object_rotation_incomplete") + objects_rotated = sum(1 for record in surviving_targets.values() if record.state == "active" and record.key_id == new_key_id) + objects_retired = len(target_set) - objects_rotated + required_key_ids = { + record.key_id + for record in records_after_rotation + if record.state == "active" + } + unavailable = required_key_ids - key_ring.keys() + if unavailable: + raise ManagedObjectConflictError("managed_object_rotation_key_unavailable") + retained_keys = {new_key_id: bytes(new_key), old_key_id: old_key} + retained_keys.update( + { + key_id: key_ring[key_id] + for key_id in sorted(required_key_ids) + if key_id not in retained_keys + } + ) + self._keys = retained_keys + self._current_key_id = new_key_id + return self._complete_rotation_receipt( + normalized_rotation_id, + old_key_id=old_key_id, + old_key=old_key, + new_key_id=new_key_id, + new_key=bytes(new_key), + objects_rotated=objects_rotated, + objects_retired=objects_retired, + ) + + def operator_status(self) -> dict[str, Any]: + with self._file_lock(exclusive=False): + stats = self._repository.stats() + records = self._repository.records() + object_files = _regular_files(self._root / "objects") + if any(stat.S_IMODE(path.lstat().st_mode) != 0o600 for path in object_files): + raise ManagedObjectIntegrityError("managed_object_integrity_failed") + actual = {path.relative_to(self._root / "objects").as_posix() for path in object_files} + active_records = [record for record in records if record.state == "active"] + expected = {record.locator for record in active_records} + quarantine_count = len(_regular_files(self._root / "quarantine")) + missing = len(expected - actual) + orphans = len(actual - expected) + unavailable_keys = sum( + 1 for record in active_records if record.key_id not in self._keys + ) + return { + "schema_version": _FORMAT_VERSION, + "enabled": True, + "provider": self.provider_name, + "encryption_algorithm": _ALGORITHM, + "encryption_evidence": "local_rehearsal_only", + "production_encryption_claim": False, + "cloud_provider_contacted": False, + "objects": stats["objects"], + "plaintext_bytes": stats["plaintext_bytes"], + "ciphertext_bytes": stats["ciphertext_bytes"], + "quarantined_ciphertext": quarantine_count, + "missing_ciphertext": missing, + "orphan_ciphertext": orphans, + "unavailable_key_objects": unavailable_keys, + "integrity_status": ( + "passed" + if not missing and not orphans and not unavailable_keys + else "failed" + ), + "current_key_configured": True, + "previous_key_count": max(len(self._keys) - 1, 0), + "root_security_verified": True, + } + + def _hash_source(self, source: BinaryIO, start: int) -> tuple[str, int]: + digest = hashlib.sha256() + size = 0 + source.seek(start) + while True: + chunk = source.read(self._config.chunk_bytes) + if chunk == b"": + break + if not isinstance(chunk, bytes): + raise TypeError("Managed-object streams must return bytes") + size += len(chunk) + if size > self._config.max_object_bytes: + raise ManagedObjectStoreLimitError("managed_object_object_limit") + digest.update(chunk) + source.seek(start) + return digest.hexdigest(), size + + def _scope_bindings(self, scope: ManagedObjectScope) -> dict[str, str]: + return { + "tenant_scope": self._scope_hmac("tenant", scope.tenant_id), + "owner_scope": self._scope_hmac("owner", scope.owner_id), + "session_scope": self._scope_hmac("session", scope.session_id), + } + + def _scope_hmac(self, label: str, value: str) -> str: + return hmac.new( + self._address_key, + f"{_FORMAT_VERSION}\x00{label}\x00{value}".encode(), + hashlib.sha256, + ).hexdigest() + + @contextlib.contextmanager + def _file_lock(self, *, exclusive: bool) -> Iterator[None]: + self._assert_root_identity() + flags = os.O_RDWR | os.O_CREAT + if hasattr(os, "O_NOFOLLOW"): + flags |= os.O_NOFOLLOW + fd = os.open(".store.lock", flags, 0o600, dir_fd=self._root_fd) + try: + os.fchmod(fd, 0o600) + opened = os.fstat(fd) + _require_private_regular_file(opened) + installed = os.stat(".store.lock", dir_fd=self._root_fd, follow_symlinks=False) + _require_private_regular_file(installed) + if (opened.st_dev, opened.st_ino) != (installed.st_dev, installed.st_ino): + raise ManagedObjectIntegrityError("managed_object_integrity_failed") + self._assert_root_identity() + mode = fcntl.LOCK_EX if exclusive else fcntl.LOCK_SH + fcntl.flock(fd, mode) + self._assert_root_identity() + yield + finally: + try: + self._assert_root_identity() + finally: + fcntl.flock(fd, fcntl.LOCK_UN) + os.close(fd) + + def _assert_root_identity(self) -> None: + try: + info = self._root.lstat() + except OSError as exc: + raise ManagedObjectIntegrityError("managed_object_root_identity_invalid") from exc + opened = os.fstat(self._root_fd) + if ( + not stat.S_ISDIR(info.st_mode) + or stat.S_ISLNK(info.st_mode) + or stat.S_IMODE(info.st_mode) != 0o700 + or (info.st_dev, info.st_ino) != self._root_identity + or (opened.st_dev, opened.st_ino) != self._root_identity + ): + raise ManagedObjectIntegrityError("managed_object_root_identity_invalid") + + def _encrypt_record( + self, + source: BinaryIO, + record: _ObjectRecord, + key_id: str, + key: bytes, + ) -> Mapping[str, Any]: + _validate_locator(record.locator) + nonce = os.urandom(_NONCE_BYTES) + header = { + "algorithm": _ALGORITHM, + "contentIdentity": record.content_identity, + "generation": record.generation, + "keyId": key_id, + "nonce": _b64url(nonce), + "objectId": record.object_id, + "objectKind": record.object_kind, + "ownerScope": record.owner_scope, + "plaintextSize": record.plaintext_size, + "schemaVersion": _FORMAT_VERSION, + "sessionScope": record.session_scope, + "tenantScope": record.tenant_scope, + } + header_bytes = _canonical_json_bytes(header) + prefix = _MAGIC + struct.pack(">I", len(header_bytes)) + header_bytes + encryptor = Cipher(algorithms.AES(key), modes.GCM(nonce)).encryptor() + encryptor.authenticate_additional_data(prefix) + temporary = f".{uuid.uuid4().hex}.tmp" + flags = os.O_WRONLY | os.O_CREAT | os.O_EXCL + if hasattr(os, "O_NOFOLLOW"): + flags |= os.O_NOFOLLOW + with self._locator_directory(record.locator, create=True) as (directory_fd, filename): + fd = os.open(temporary, flags, 0o600, dir_fd=directory_fd) + file_digest = hashlib.sha256() + plaintext_digest = hashlib.sha256() + written = 0 + try: + for chunk in (prefix,): + _write_all(fd, chunk) + file_digest.update(chunk) + written += len(chunk) + plaintext_read = 0 + while True: + chunk = source.read(self._config.chunk_bytes) + if chunk == b"": + break + if not isinstance(chunk, bytes): + raise TypeError("Managed-object streams must return bytes") + plaintext_read += len(chunk) + if plaintext_read > record.plaintext_size: + raise ManagedObjectIntegrityError("managed_object_integrity_failed") + plaintext_digest.update(chunk) + encrypted = encryptor.update(chunk) + _write_all(fd, encrypted) + file_digest.update(encrypted) + written += len(encrypted) + if not hmac.compare_digest( + _content_identity_from_header( + self._address_key, + header, + plaintext_digest.hexdigest(), + ), + record.content_identity, + ): + raise ManagedObjectIntegrityError("managed_object_integrity_failed") + final = encryptor.finalize() + if final: + _write_all(fd, final) + file_digest.update(final) + written += len(final) + tag = encryptor.tag + _write_all(fd, tag) + file_digest.update(tag) + written += len(tag) + if plaintext_read != record.plaintext_size: + raise ManagedObjectIntegrityError("managed_object_integrity_failed") + os.fsync(fd) + os.fchmod(fd, 0o600) + except BaseException: + os.close(fd) + try: + os.unlink(temporary, dir_fd=directory_fd) + except FileNotFoundError: + pass + raise + else: + os.close(fd) + os.replace( + temporary, + filename, + src_dir_fd=directory_fd, + dst_dir_fd=directory_fd, + ) + os.fsync(directory_fd) + return { + "ciphertext_size": written, + "header_sha256": hashlib.sha256(header_bytes).hexdigest(), + "ciphertext_sha256": file_digest.hexdigest(), + } + + def _decrypt_record( + self, + record: _ObjectRecord, + *, + key_ring: Mapping[str, bytes] | None = None, + allow_generation: int | None = None, + ) -> bytes: + keys = self._keys if key_ring is None else key_ring + try: + flags = os.O_RDONLY + if hasattr(os, "O_NOFOLLOW"): + flags |= os.O_NOFOLLOW + with self._locator_directory(record.locator, create=False) as (directory_fd, filename): + fd = os.open(filename, flags, dir_fd=directory_fd) + try: + file_stat = os.fstat(fd) + _require_private_regular_file(file_stat) + total_size = file_stat.st_size + if allow_generation is None and total_size != record.ciphertext_size: + raise ManagedObjectIntegrityError("managed_object_integrity_failed") + minimum = len(_MAGIC) + 4 + 2 + _TAG_BYTES + if total_size < minimum or total_size > self._config.max_object_bytes + _MAX_HEADER_BYTES + 64: + raise ManagedObjectIntegrityError("managed_object_integrity_failed") + magic = _read_exact(fd, len(_MAGIC)) + header_length_bytes = _read_exact(fd, 4) + if magic != _MAGIC: + raise ManagedObjectIntegrityError("managed_object_integrity_failed") + header_length = struct.unpack(">I", header_length_bytes)[0] + if not 2 <= header_length <= _MAX_HEADER_BYTES: + raise ManagedObjectIntegrityError("managed_object_integrity_failed") + header_bytes = _read_exact(fd, header_length) + if allow_generation is None and hashlib.sha256(header_bytes).hexdigest() != record.header_sha256: + raise ManagedObjectIntegrityError("managed_object_integrity_failed") + header = _parse_header(header_bytes) + _validate_header(header, record, allow_generation=allow_generation) + key_id = str(header["keyId"]) + key = keys.get(key_id) + if key is None: + raise ManagedObjectIntegrityError("managed_object_integrity_failed") + nonce = _decode_nonce(header["nonce"]) + ciphertext_bytes = total_size - len(_MAGIC) - 4 - header_length - _TAG_BYTES + if ciphertext_bytes < 0: + raise ManagedObjectIntegrityError("managed_object_integrity_failed") + tag = os.pread(fd, _TAG_BYTES, total_size - _TAG_BYTES) + if len(tag) != _TAG_BYTES: + raise ManagedObjectIntegrityError("managed_object_integrity_failed") + prefix = magic + header_length_bytes + header_bytes + decryptor = Cipher(algorithms.AES(key), modes.GCM(nonce, tag)).decryptor() + decryptor.authenticate_additional_data(prefix) + staging_fd = self._unlinked_staging_fd() + plaintext_digest = hashlib.sha256() + plaintext_size = 0 + file_digest = hashlib.sha256(prefix) + remaining = ciphertext_bytes + try: + while remaining: + chunk = os.read(fd, min(self._config.chunk_bytes, remaining)) + if not chunk: + raise ManagedObjectIntegrityError("managed_object_integrity_failed") + remaining -= len(chunk) + file_digest.update(chunk) + decrypted = decryptor.update(chunk) + plaintext_digest.update(decrypted) + plaintext_size += len(decrypted) + _write_all(staging_fd, decrypted) + final = decryptor.finalize() + plaintext_digest.update(final) + plaintext_size += len(final) + _write_all(staging_fd, final) + file_digest.update(tag) + if ( + allow_generation is None + and record.ciphertext_sha256 + and not hmac.compare_digest(file_digest.hexdigest(), record.ciphertext_sha256) + ): + raise ManagedObjectIntegrityError("managed_object_integrity_failed") + if plaintext_size != record.plaintext_size: + raise ManagedObjectIntegrityError("managed_object_integrity_failed") + identity = _content_identity_from_header( + self._address_key, + header, + plaintext_digest.hexdigest(), + ) + if not hmac.compare_digest(identity, record.content_identity): + raise ManagedObjectIntegrityError("managed_object_integrity_failed") + os.lseek(staging_fd, 0, os.SEEK_SET) + return _read_bounded_fd(staging_fd, record.plaintext_size) + finally: + os.close(staging_fd) + finally: + os.close(fd) + except ManagedObjectIntegrityError: + raise + except (InvalidTag, OSError, ValueError, TypeError, KeyError, json.JSONDecodeError) as exc: + raise ManagedObjectIntegrityError("managed_object_integrity_failed") from exc + + def _read_header(self, locator: str) -> dict[str, Any]: + flags = os.O_RDONLY + if hasattr(os, "O_NOFOLLOW"): + flags |= os.O_NOFOLLOW + with self._locator_directory(locator, create=False) as (directory_fd, filename): + fd = os.open(filename, flags, dir_fd=directory_fd) + try: + _require_private_regular_file(os.fstat(fd)) + if _read_exact(fd, len(_MAGIC)) != _MAGIC: + raise ManagedObjectIntegrityError("managed_object_integrity_failed") + length = struct.unpack(">I", _read_exact(fd, 4))[0] + if not 2 <= length <= _MAX_HEADER_BYTES: + raise ManagedObjectIntegrityError("managed_object_integrity_failed") + return _parse_header(_read_exact(fd, length)) + finally: + os.close(fd) + + def _file_metadata(self, locator: str) -> dict[str, Any]: + digest = hashlib.sha256() + flags = os.O_RDONLY + if hasattr(os, "O_NOFOLLOW"): + flags |= os.O_NOFOLLOW + with self._locator_directory(locator, create=False) as (directory_fd, filename): + fd = os.open(filename, flags, dir_fd=directory_fd) + try: + file_stat = os.fstat(fd) + _require_private_regular_file(file_stat) + while True: + chunk = os.read(fd, self._config.chunk_bytes) + if not chunk: + break + digest.update(chunk) + os.lseek(fd, len(_MAGIC), os.SEEK_SET) + header_length = struct.unpack(">I", _read_exact(fd, 4))[0] + header_bytes = _read_exact(fd, header_length) + return { + "ciphertext_size": file_stat.st_size, + "header_sha256": hashlib.sha256(header_bytes).hexdigest(), + "ciphertext_sha256": digest.hexdigest(), + } + finally: + os.close(fd) + + def _verify_record_file_metadata(self, record: _ObjectRecord) -> None: + try: + stored = self._file_metadata(record.locator) + except ManagedObjectIntegrityError: + raise + except OSError as exc: + raise ManagedObjectIntegrityError("managed_object_integrity_failed") from exc + if ( + stored["ciphertext_size"] != record.ciphertext_size + or not hmac.compare_digest(stored["header_sha256"], record.header_sha256) + or not hmac.compare_digest(stored["ciphertext_sha256"], record.ciphertext_sha256) + ): + raise ManagedObjectIntegrityError("managed_object_integrity_failed") + + def _unlinked_staging_fd(self) -> int: + flags = os.O_RDWR | os.O_CREAT | os.O_EXCL + if hasattr(os, "O_NOFOLLOW"): + flags |= os.O_NOFOLLOW + candidate = f".{uuid.uuid4().hex}.plain" + with self._named_directory_fd("staging") as staging_fd: + fd = os.open(candidate, flags, 0o600, dir_fd=staging_fd) + os.fchmod(fd, 0o600) + os.unlink(candidate, dir_fd=staging_fd) + return fd + + @contextlib.contextmanager + def _named_directory_fd(self, name: str) -> Iterator[int]: + if name not in {"objects", "staging", "quarantine", "journals", "receipts"}: + raise ValueError("Managed-object directory name is invalid") + flags = os.O_RDONLY + if hasattr(os, "O_DIRECTORY"): + flags |= os.O_DIRECTORY + if hasattr(os, "O_NOFOLLOW"): + flags |= os.O_NOFOLLOW + fd = os.open(name, flags, dir_fd=self._root_fd) + try: + info = os.fstat(fd) + if not stat.S_ISDIR(info.st_mode) or stat.S_IMODE(info.st_mode) != 0o700: + raise ManagedObjectIntegrityError("managed_object_integrity_failed") + yield fd + finally: + os.close(fd) + + @contextlib.contextmanager + def _scoped_directory_fd( + self, + base_name: str, + directory_name: str, + *, + create: bool, + ) -> Iterator[int]: + if not re.fullmatch(r"[0-9a-f]{32}", directory_name): + raise ManagedObjectIntegrityError("managed_object_metadata_invalid") + flags = os.O_RDONLY + if hasattr(os, "O_DIRECTORY"): + flags |= os.O_DIRECTORY + if hasattr(os, "O_NOFOLLOW"): + flags |= os.O_NOFOLLOW + with self._named_directory_fd(base_name) as base_fd: + try: + fd = os.open(directory_name, flags, dir_fd=base_fd) + except FileNotFoundError: + if not create: + raise + try: + os.mkdir(directory_name, mode=0o700, dir_fd=base_fd) + except FileExistsError: + pass + os.fsync(base_fd) + fd = os.open(directory_name, flags, dir_fd=base_fd) + try: + info = os.fstat(fd) + if not stat.S_ISDIR(info.st_mode) or stat.S_IMODE(info.st_mode) != 0o700: + raise ManagedObjectIntegrityError("managed_object_integrity_failed") + yield fd + finally: + os.close(fd) + + @contextlib.contextmanager + def _locator_directory( + self, + locator: str, + *, + create: bool, + ) -> Iterator[tuple[int, str]]: + _validate_locator(locator) + directory_name, filename = locator.split("/", 1) + with self._scoped_directory_fd( + "objects", + directory_name, + create=create, + ) as directory_fd: + yield directory_fd, filename + + def _locator_exists(self, locator: str) -> bool: + try: + with self._locator_directory(locator, create=False) as (directory_fd, filename): + info = os.stat(filename, dir_fd=directory_fd, follow_symlinks=False) + _require_private_regular_file(info) + return True + except FileNotFoundError: + return False + + def _purge_scoped_directory_files( + self, + base_name: str, + directory_name: str, + ) -> int: + if not re.fullmatch(r"[0-9a-f]{32}", directory_name): + raise ManagedObjectIntegrityError("managed_object_integrity_failed") + flags = os.O_RDONLY + if hasattr(os, "O_DIRECTORY"): + flags |= os.O_DIRECTORY + if hasattr(os, "O_NOFOLLOW"): + flags |= os.O_NOFOLLOW + with self._named_directory_fd(base_name) as base_fd: + _require_private_directory(os.fstat(base_fd)) + try: + expected = os.stat( + directory_name, + dir_fd=base_fd, + follow_symlinks=False, + ) + except FileNotFoundError: + return 0 + _require_private_directory(expected) + directory_fd = os.open(directory_name, flags, dir_fd=base_fd) + try: + opened = os.fstat(directory_fd) + _require_private_directory(opened) + if (opened.st_dev, opened.st_ino) != ( + expected.st_dev, + expected.st_ino, + ): + raise ManagedObjectIntegrityError("managed_object_integrity_failed") + with os.scandir(directory_fd) as entries: + names = sorted(entry.name for entry in entries) + for filename in names: + _secure_unlink_regular_at(directory_fd, filename) + with os.scandir(directory_fd) as entries: + if next(entries, None) is not None: + raise ManagedObjectIntegrityError("managed_object_integrity_failed") + os.fsync(directory_fd) + installed = os.stat( + directory_name, + dir_fd=base_fd, + follow_symlinks=False, + ) + _require_private_directory(installed) + if (opened.st_dev, opened.st_ino) != ( + installed.st_dev, + installed.st_ino, + ): + raise ManagedObjectIntegrityError("managed_object_integrity_failed") + os.rmdir(directory_name, dir_fd=base_fd) + os.fsync(base_fd) + return len(names) + finally: + os.close(directory_fd) + + def _delete_record_file(self, record: _ObjectRecord) -> None: + self._delete_locator(record.locator) + if self._locator_exists(record.locator): + raise RuntimeError("Managed-object ciphertext deletion could not be verified") + + def _delete_locator(self, locator: str) -> bool: + try: + with self._locator_directory(locator, create=False) as (directory_fd, filename): + _secure_unlink_regular_at(directory_fd, filename) + except FileNotFoundError: + return False + return True + + def _quarantine_ciphertext(self, locator: str) -> None: + match = _RECOVERABLE_CIPHERTEXT_PATTERN.fullmatch(str(locator or "")) + if match is None: + raise ValueError("Unsafe orphan object") + directory_name = match.group("directory") + filename = match.group("filename") + destination = f"{uuid.uuid4().hex}.mobj" + with ( + self._scoped_directory_fd("objects", directory_name, create=False) as source_fd, + self._scoped_directory_fd("quarantine", directory_name, create=True) as destination_fd, + ): + source_info = os.stat(filename, dir_fd=source_fd, follow_symlinks=False) + _require_private_regular_file(source_info) + os.replace( + filename, + destination, + src_dir_fd=source_fd, + dst_dir_fd=destination_fd, + ) + installed = os.stat(destination, dir_fd=destination_fd, follow_symlinks=False) + _require_private_regular_file(installed) + if (installed.st_dev, installed.st_ino) != ( + source_info.st_dev, + source_info.st_ino, + ): + raise ManagedObjectIntegrityError("managed_object_integrity_failed") + os.chmod(destination, 0o600, dir_fd=destination_fd, follow_symlinks=False) + os.fsync(source_fd) + os.fsync(destination_fd) + + def _rotation_path(self, rotation_id: str, suffix: str) -> Path: + return self._root / suffix / f"{rotation_id}.json" + + def _write_rotation_journal( + self, + rotation_id: str, + *, + old_key_id: str, + old_key: bytes, + new_key_id: str, + new_key: bytes, + target_tokens: list[str], + ) -> dict[str, Any]: + payload = { + "schema_version": _FORMAT_VERSION, + "rotation_id": rotation_id, + "state": "pending", + "old_key_id_sha256": hashlib.sha256(old_key_id.encode()).hexdigest(), + "new_key_id_sha256": hashlib.sha256(new_key_id.encode()).hexdigest(), + "old_key_commitment": self._rotation_key_commitment(old_key_id, old_key), + "new_key_commitment": self._rotation_key_commitment(new_key_id, new_key), + "target_tokens": list(target_tokens), + "objects_targeted": len(target_tokens), + "started_at": int(time.time()), + } + self._write_signed_json(self._rotation_path(rotation_id, "journals"), payload) + return payload + + def _load_rotation_journal(self, rotation_id: str) -> dict[str, Any] | None: + path = self._rotation_path(rotation_id, "journals") + if not self._signed_json_exists(path): + return None + envelope = self._read_signed_json(path) + payload = envelope.get("payload") + expected = { + "schema_version", + "rotation_id", + "state", + "old_key_id_sha256", + "new_key_id_sha256", + "old_key_commitment", + "new_key_commitment", + "target_tokens", + "objects_targeted", + "started_at", + } + if not isinstance(payload, dict) or frozenset(payload) != expected: + raise ManagedObjectIntegrityError("managed_object_rotation_journal_invalid") + targets = payload.get("target_tokens") + objects_targeted = payload.get("objects_targeted") + started_at = payload.get("started_at") + if ( + payload.get("schema_version") != _FORMAT_VERSION + or payload.get("rotation_id") != rotation_id + or payload.get("state") != "pending" + or not isinstance(targets, list) + or not _is_bounded_nonnegative_int(objects_targeted) + or not _is_bounded_nonnegative_int(started_at) + or len(targets) != objects_targeted + or len(targets) != len(set(targets)) + or any(not isinstance(token, str) or not re.fullmatch(r"[0-9a-f]{64}", token) for token in targets) + ): + raise ManagedObjectIntegrityError("managed_object_rotation_journal_invalid") + try: + old_key_id_hash = _strict_sha256(payload.get("old_key_id_sha256")) + new_key_id_hash = _strict_sha256(payload.get("new_key_id_sha256")) + _strict_sha256(payload.get("old_key_commitment")) + _strict_sha256(payload.get("new_key_commitment")) + except ManagedObjectIntegrityError as exc: + raise ManagedObjectIntegrityError("managed_object_rotation_journal_invalid") from exc + if hmac.compare_digest(old_key_id_hash, new_key_id_hash): + raise ManagedObjectIntegrityError("managed_object_rotation_journal_invalid") + return payload + + def _complete_rotation_receipt( + self, + rotation_id: str, + *, + old_key_id: str, + old_key: bytes, + new_key_id: str, + new_key: bytes, + objects_rotated: int, + objects_retired: int, + ) -> dict[str, Any]: + payload = { + "schema_version": _FORMAT_VERSION, + "rotation_id": rotation_id, + "status": "completed", + "algorithm": _ALGORITHM, + "old_key_id_sha256": hashlib.sha256(old_key_id.encode()).hexdigest(), + "new_key_id_sha256": hashlib.sha256(new_key_id.encode()).hexdigest(), + "old_key_commitment": self._rotation_key_commitment(old_key_id, old_key), + "new_key_commitment": self._rotation_key_commitment(new_key_id, new_key), + "objects_rotated": objects_rotated, + "objects_retired_during_rotation": objects_retired, + "completed_at": int(time.time()), + "production_encryption_claim": False, + } + path = self._rotation_path(rotation_id, "receipts") + envelope = self._write_signed_json(path, payload) + journal = self._rotation_path(rotation_id, "journals") + with self._named_directory_fd("journals") as journal_fd: + try: + os.unlink(journal.name, dir_fd=journal_fd) + except FileNotFoundError: + pass + os.fsync(journal_fd) + return _public_signed_payload(envelope) + + def _load_rotation_receipt(self, rotation_id: str) -> dict[str, Any] | None: + path = self._rotation_path(rotation_id, "receipts") + if not self._signed_json_exists(path): + return None + envelope = self._read_signed_json(path) + payload = envelope.get("payload") + expected = { + "schema_version", + "rotation_id", + "status", + "algorithm", + "old_key_id_sha256", + "new_key_id_sha256", + "old_key_commitment", + "new_key_commitment", + "objects_rotated", + "objects_retired_during_rotation", + "completed_at", + "production_encryption_claim", + } + if ( + not isinstance(payload, dict) + or frozenset(payload) != expected + or payload.get("status") != "completed" + or payload.get("schema_version") != _FORMAT_VERSION + or payload.get("rotation_id") != rotation_id + or payload.get("algorithm") != _ALGORITHM + or not _is_bounded_nonnegative_int(payload.get("objects_rotated")) + or not _is_bounded_nonnegative_int(payload.get("objects_retired_during_rotation")) + or not _is_bounded_nonnegative_int(payload.get("completed_at")) + or payload.get("production_encryption_claim") is not False + ): + raise ManagedObjectIntegrityError("managed_object_rotation_receipt_invalid") + try: + old_key_id_hash = _strict_sha256(payload.get("old_key_id_sha256")) + new_key_id_hash = _strict_sha256(payload.get("new_key_id_sha256")) + _strict_sha256(payload.get("old_key_commitment")) + _strict_sha256(payload.get("new_key_commitment")) + except ManagedObjectIntegrityError as exc: + raise ManagedObjectIntegrityError("managed_object_rotation_receipt_invalid") from exc + if hmac.compare_digest(old_key_id_hash, new_key_id_hash): + raise ManagedObjectIntegrityError("managed_object_rotation_receipt_invalid") + return _public_signed_payload(envelope) + + def _rotation_target_token(self, locator: str) -> str: + return hmac.new( + self._address_key, + b"rotation-target\x00" + locator.encode(), + hashlib.sha256, + ).hexdigest() + + def _rotation_key_commitment(self, key_id: str, key: bytes) -> str: + return hmac.new( + self._address_key, + b"rotation-key-commitment\x00" + key_id.encode() + b"\x00" + key, + hashlib.sha256, + ).hexdigest() + + def _key_id_for_hash(self, expected_hash: str) -> str: + _strict_sha256(expected_hash) + matches = [key_id for key_id in self._keys if hmac.compare_digest(hashlib.sha256(key_id.encode()).hexdigest(), expected_hash)] + if len(matches) != 1: + raise ManagedObjectConflictError("managed_object_rotation_key_unavailable") + return matches[0] + + def _write_signed_json(self, path: Path, payload: Mapping[str, Any]) -> dict[str, Any]: + canonical_payload = _canonical_json_bytes(payload) + envelope = { + "payload": dict(payload), + "signature_algorithm": "HMAC-SHA256", + "signature": hmac.new(self._address_key, canonical_payload, hashlib.sha256).hexdigest(), + "payload_sha256": hashlib.sha256(canonical_payload).hexdigest(), + } + data = _canonical_json_bytes(envelope) + if len(data) > _MAX_JOURNAL_BYTES: + raise ManagedObjectStoreLimitError("managed_object_rotation_journal_limit") + directory_name = path.parent.name + filename = path.name + temporary = f".{uuid.uuid4().hex}.tmp" + flags = os.O_WRONLY | os.O_CREAT | os.O_EXCL + if hasattr(os, "O_NOFOLLOW"): + flags |= os.O_NOFOLLOW + with self._named_directory_fd(directory_name) as directory_fd: + fd = os.open(temporary, flags, 0o600, dir_fd=directory_fd) + try: + _write_all(fd, data) + os.fsync(fd) + os.fchmod(fd, 0o600) + except BaseException: + os.close(fd) + try: + os.unlink(temporary, dir_fd=directory_fd) + except FileNotFoundError: + pass + raise + else: + os.close(fd) + os.replace( + temporary, + filename, + src_dir_fd=directory_fd, + dst_dir_fd=directory_fd, + ) + os.fsync(directory_fd) + return envelope + + def _read_signed_json(self, path: Path) -> dict[str, Any]: + directory_name = path.parent.name + flags = os.O_RDONLY + if hasattr(os, "O_NOFOLLOW"): + flags |= os.O_NOFOLLOW + with self._named_directory_fd(directory_name) as directory_fd: + fd = os.open(path.name, flags, dir_fd=directory_fd) + try: + info = os.fstat(fd) + _require_private_regular_file( + info, + error_code="managed_object_rotation_receipt_invalid", + ) + if info.st_size > _MAX_JOURNAL_BYTES: + raise ManagedObjectIntegrityError("managed_object_rotation_receipt_invalid") + raw = _read_exact(fd, info.st_size) + finally: + os.close(fd) + try: + envelope = json.loads(raw) + except (json.JSONDecodeError, UnicodeDecodeError) as exc: + raise ManagedObjectIntegrityError("managed_object_rotation_receipt_invalid") from exc + if ( + not isinstance(envelope, dict) + or frozenset(envelope) != {"payload", "signature_algorithm", "signature", "payload_sha256"} + or _canonical_json_bytes(envelope) != raw + ): + raise ManagedObjectIntegrityError("managed_object_rotation_receipt_invalid") + payload = envelope.get("payload") + canonical = _canonical_json_bytes(payload) + expected_signature = hmac.new(self._address_key, canonical, hashlib.sha256).hexdigest() + if ( + envelope.get("signature_algorithm") != "HMAC-SHA256" + or not hmac.compare_digest(str(envelope.get("signature") or ""), expected_signature) + or not hmac.compare_digest(str(envelope.get("payload_sha256") or ""), hashlib.sha256(canonical).hexdigest()) + ): + raise ManagedObjectIntegrityError("managed_object_rotation_receipt_invalid") + return envelope + + def _signed_json_exists(self, path: Path) -> bool: + with self._named_directory_fd(path.parent.name) as directory_fd: + try: + info = os.stat(path.name, dir_fd=directory_fd, follow_symlinks=False) + except FileNotFoundError: + return False + _require_private_regular_file( + info, + error_code="managed_object_rotation_receipt_invalid", + ) + return True + + +def _initialize_secure_root(root: Path) -> Path: + normalized = Path(root) + if normalized.exists() or normalized.is_symlink(): + info = normalized.lstat() + if not stat.S_ISDIR(info.st_mode) or normalized.is_symlink(): + raise ValueError("Managed-object root must be a secure directory") + if stat.S_IMODE(info.st_mode) != 0o700: + raise ValueError("Managed-object root must be a secure directory") + else: + normalized.mkdir(parents=True, mode=0o700) + normalized.chmod(0o700) + root_info = normalized.lstat() + if hasattr(os, "getuid") and root_info.st_uid != os.getuid(): + raise ValueError("Managed-object root must be owned by the current process user") + for child in ("objects", "staging", "quarantine", "journals", "receipts"): + _ensure_secure_directory(normalized / child) + _fsync_directory(normalized) + return normalized + + +def _ensure_secure_directory(path: Path) -> None: + if path.exists() or path.is_symlink(): + info = path.lstat() + if not stat.S_ISDIR(info.st_mode) or path.is_symlink() or stat.S_IMODE(info.st_mode) != 0o700: + raise ValueError("Managed-object path must be a secure directory") + else: + path.mkdir(mode=0o700) + path.chmod(0o700) + _fsync_directory(path.parent) + info = path.lstat() + if hasattr(os, "getuid") and info.st_uid != os.getuid(): + raise ValueError("Managed-object directory ownership is invalid") + + +def _sqlite_record(row: sqlite3.Row) -> _ObjectRecord: + return _mapping_record(dict(row)) + + +def _mapping_record(row: Mapping[str, Any]) -> _ObjectRecord: + generation = _strict_positive_int(row["generation"]) + created_at = _strict_finite_nonnegative_number(row["created_at"]) + updated_at = _strict_finite_nonnegative_number(row["updated_at"]) + record = _ObjectRecord( + object_id=str(row["object_id"]), + tenant_scope=_strict_sha256(row["tenant_scope"]), + owner_scope=_strict_sha256(row["owner_scope"]), + session_scope=_strict_sha256(row["session_scope"]), + session_id=str(row["session_id"]), + object_kind=_normalize_kind(str(row["object_kind"])), + content_identity=_strict_sha256(row["content_identity"]), + locator=str(row["locator"]), + plaintext_size=_strict_nonnegative_int(row["plaintext_size"]), + ciphertext_size=_strict_nonnegative_int(row["ciphertext_size"]), + key_id=str(row["key_id"]), + generation=generation, + state=str(row["state"]), + header_sha256=str(row["header_sha256"] or ""), + ciphertext_sha256=str(row["ciphertext_sha256"] or ""), + mutation_id=str(row["mutation_id"]), + created_at=created_at, + updated_at=updated_at, + ) + _normalize_object_id(record.object_id) + _validate_locator(record.locator) + _validate_key_id(record.key_id) + if record.state not in _OBJECT_STATES: + raise ManagedObjectIntegrityError("managed_object_metadata_invalid") + if record.header_sha256: + _strict_sha256(record.header_sha256) + if record.ciphertext_sha256: + _strict_sha256(record.ciphertext_sha256) + return record + + +def _record_matches_request(record: _ObjectRecord, request: _PutRequest) -> bool: + comparisons = ( + (record.tenant_scope, request.tenant_scope), + (record.owner_scope, request.owner_scope), + (record.session_scope, request.session_scope), + (record.session_id, request.scope.session_id), + (record.object_kind, request.object_kind), + (record.content_identity, request.content_identity), + (record.object_id, request.object_id), + ) + return all(hmac.compare_digest(left, right) for left, right in comparisons) and (record.plaintext_size == request.plaintext_size) + + +def _record_matches_scope( + record: _ObjectRecord, + scope: ManagedObjectScope, + bindings: Mapping[str, str], +) -> bool: + return all( + ( + hmac.compare_digest(record.tenant_scope, bindings["tenant_scope"]), + hmac.compare_digest(record.owner_scope, bindings["owner_scope"]), + hmac.compare_digest(record.session_scope, bindings["session_scope"]), + hmac.compare_digest(record.session_id, scope.session_id), + ) + ) + + +def _public_receipt(record: _ObjectRecord) -> ManagedObjectReceipt: + return ManagedObjectReceipt( + object_id=record.object_id, + object_kind=record.object_kind, + plaintext_size=record.plaintext_size, + generation=record.generation, + ) + + +def _validate_header( + header: Mapping[str, Any], + record: _ObjectRecord, + *, + allow_generation: int | None, +) -> None: + expected_keys = { + "algorithm", + "contentIdentity", + "generation", + "keyId", + "nonce", + "objectId", + "objectKind", + "ownerScope", + "plaintextSize", + "schemaVersion", + "sessionScope", + "tenantScope", + } + if frozenset(header) != expected_keys: + raise ManagedObjectIntegrityError("managed_object_integrity_failed") + generation = int(header["generation"]) + allowed_generations = {record.generation} + if allow_generation is not None: + allowed_generations.add(allow_generation) + pairs = [ + (str(header["algorithm"]), _ALGORITHM), + (str(header["contentIdentity"]), record.content_identity), + (str(header["objectId"]), record.object_id), + (str(header["objectKind"]), record.object_kind), + (str(header["ownerScope"]), record.owner_scope), + (str(header["schemaVersion"]), _FORMAT_VERSION), + (str(header["sessionScope"]), record.session_scope), + (str(header["tenantScope"]), record.tenant_scope), + ] + if allow_generation is None: + pairs.append((str(header["keyId"]), record.key_id)) + if ( + any(not hmac.compare_digest(left, right) for left, right in pairs) + or generation not in allowed_generations + or int(header["plaintextSize"]) != record.plaintext_size + ): + raise ManagedObjectIntegrityError("managed_object_integrity_failed") + _validate_key_id(str(header["keyId"])) + _decode_nonce(header["nonce"]) + + +def _content_identity_from_header( + address_key: bytes, + header: Mapping[str, Any], + plaintext_sha256: str, +) -> str: + payload = _canonical_json_bytes( + { + "kind": str(header["objectKind"]), + "ownerScope": str(header["ownerScope"]), + "plaintextSha256": plaintext_sha256, + "schemaVersion": str(header["schemaVersion"]), + "sessionScope": str(header["sessionScope"]), + "tenantScope": str(header["tenantScope"]), + } + ) + return hmac.new(address_key, payload, hashlib.sha256).hexdigest() + + +def _parse_header(header_bytes: bytes) -> dict[str, Any]: + try: + parsed = json.loads(header_bytes) + except (json.JSONDecodeError, UnicodeDecodeError) as exc: + raise ManagedObjectIntegrityError("managed_object_integrity_failed") from exc + if not isinstance(parsed, dict) or _canonical_json_bytes(parsed) != header_bytes: + raise ManagedObjectIntegrityError("managed_object_integrity_failed") + return parsed + + +def _normalize_kind(kind: ManagedObjectKind | str) -> str: + normalized = kind.value if isinstance(kind, ManagedObjectKind) else str(kind or "").strip().lower() + if normalized not in {member.value for member in ManagedObjectKind}: + raise ValueError("Unsupported managed-object kind") + return normalized + + +def _normalize_object_id(object_id: str) -> str: + normalized = str(object_id or "").strip() + if not _OBJECT_ID_PATTERN.fullmatch(normalized): + raise ManagedObjectNotFoundError("managed_object_not_found") + return normalized + + +def _validate_locator(locator: str) -> None: + if not _LOCATOR_PATTERN.fullmatch(str(locator or "")): + raise ManagedObjectIntegrityError("managed_object_metadata_invalid") + + +def _validate_key_id(key_id: str) -> None: + if not _ID_PATTERN.fullmatch(str(key_id or "")): + raise ValueError("Encryption key ID is invalid") + + +def _validate_32_byte_key(key: bytes, label: str) -> None: + if not isinstance(key, bytes) or len(key) != 32: + raise ValueError(f"{label} must be exactly 32-byte key material") + + +def _decode_external_key(raw: Any) -> bytes: + encoded = str(raw or "").strip() + if not encoded: + raise ValueError("Managed-object key material is required") + if re.fullmatch(r"[0-9a-fA-F]{64}", encoded): + return bytes.fromhex(encoded) + if not re.fullmatch(r"[A-Za-z0-9_-]{43}=?", encoded): + raise ValueError("Managed-object key material must be hex or URL-safe base64") + try: + padding = "=" * (-len(encoded) % 4) + decoded = base64.b64decode(encoded + padding, altchars=b"-_", validate=True) + except (ValueError, binascii.Error) as exc: + raise ValueError("Managed-object key material must be hex or URL-safe base64") from exc + _validate_32_byte_key(decoded, "managed-object key") + if not hmac.compare_digest(_b64url(decoded), encoded.rstrip("=")): + raise ValueError("Managed-object key material must be hex or URL-safe base64") + return decoded + + +def _unique_json_object(pairs: list[tuple[str, Any]]) -> dict[str, Any]: + result: dict[str, Any] = {} + for key, value in pairs: + if key in result: + raise ValueError("Duplicate JSON object key") + result[key] = value + return result + + +def _decode_nonce(raw: Any) -> bytes: + try: + encoded = str(raw) + padding = "=" * (-len(encoded) % 4) + nonce = base64.urlsafe_b64decode(encoded + padding) + except (ValueError, binascii.Error) as exc: + raise ManagedObjectIntegrityError("managed_object_integrity_failed") from exc + if len(nonce) != _NONCE_BYTES: + raise ManagedObjectIntegrityError("managed_object_integrity_failed") + return nonce + + +def _b64url(raw: bytes) -> str: + return base64.urlsafe_b64encode(raw).rstrip(b"=").decode("ascii") + + +def _parse_bool(raw: Any) -> bool: + normalized = str(raw or "").strip().lower() + if normalized in {"true", "1", "yes", "on"}: + return True + if normalized in {"false", "0", "no", "off", ""}: + return False + raise ValueError("Boolean managed-object configuration is invalid") + + +def _bounded_env_int(raw: Any, default: int, minimum: int, maximum: int) -> int: + if raw in (None, ""): + return default + try: + value = int(str(raw), 10) + except ValueError as exc: + raise ValueError("Managed-object byte budget must be an integer") from exc + if not minimum <= value <= maximum: + raise ValueError("Managed-object byte budget is outside the supported range") + return value + + +def _strict_nonnegative_int(raw: Any) -> int: + if isinstance(raw, bool) or not isinstance(raw, int) or raw < 0: + raise ManagedObjectIntegrityError("managed_object_metadata_invalid") + return raw + + +def _strict_positive_int(raw: Any) -> int: + if isinstance(raw, bool) or not isinstance(raw, int) or raw < 1: + raise ManagedObjectIntegrityError("managed_object_metadata_invalid") + return raw + + +def _strict_finite_nonnegative_number(raw: Any) -> float: + if ( + isinstance(raw, bool) + or not isinstance(raw, (int, float)) + or not math.isfinite(raw) + or raw < 0 + ): + raise ManagedObjectIntegrityError("managed_object_metadata_invalid") + return float(raw) + + +def _is_bounded_nonnegative_int(raw: Any) -> bool: + return ( + not isinstance(raw, bool) + and isinstance(raw, int) + and 0 <= raw <= 2**63 - 1 + ) + + +def _strict_sha256(raw: Any) -> str: + value = str(raw or "") + if not re.fullmatch(r"[0-9a-f]{64}", value): + raise ManagedObjectIntegrityError("managed_object_metadata_invalid") + return value + + +def _canonical_json_bytes(payload: Any) -> bytes: + return json.dumps(payload, sort_keys=True, separators=(",", ":"), ensure_ascii=True).encode() + + +def _write_all(fd: int, payload: bytes) -> None: + view = memoryview(payload) + while view: + written = os.write(fd, view) + if written <= 0: + raise OSError("short write") + view = view[written:] + + +def _read_exact(fd: int, size: int) -> bytes: + chunks: list[bytes] = [] + remaining = size + while remaining: + chunk = os.read(fd, remaining) + if not chunk: + raise OSError("short read") + chunks.append(chunk) + remaining -= len(chunk) + return b"".join(chunks) + + +def _read_bounded_fd(fd: int, expected_size: int) -> bytes: + result = bytearray() + while len(result) < expected_size: + chunk = os.read(fd, min(256 * 1024, expected_size - len(result))) + if not chunk: + break + result.extend(chunk) + if len(result) != expected_size or os.read(fd, 1): + raise ManagedObjectIntegrityError("managed_object_integrity_failed") + return bytes(result) + + +def _fsync_directory(path: Path) -> None: + flags = os.O_RDONLY + if hasattr(os, "O_DIRECTORY"): + flags |= os.O_DIRECTORY + if hasattr(os, "O_NOFOLLOW"): + flags |= os.O_NOFOLLOW + fd = os.open(path, flags) + try: + os.fsync(fd) + finally: + os.close(fd) + + +def _require_private_regular_file( + info: os.stat_result, + *, + error_code: str = "managed_object_integrity_failed", +) -> None: + owner_invalid = hasattr(os, "getuid") and info.st_uid != os.getuid() + if ( + not stat.S_ISREG(info.st_mode) + or stat.S_IMODE(info.st_mode) != 0o600 + or info.st_nlink != 1 + or owner_invalid + ): + raise ManagedObjectIntegrityError(error_code) + + +def _require_private_directory( + info: os.stat_result, + *, + error_code: str = "managed_object_integrity_failed", +) -> None: + owner_invalid = hasattr(os, "getuid") and info.st_uid != os.getuid() + if ( + not stat.S_ISDIR(info.st_mode) + or stat.S_ISLNK(info.st_mode) + or stat.S_IMODE(info.st_mode) != 0o700 + or owner_invalid + ): + raise ManagedObjectIntegrityError(error_code) + + +def _secure_unlink_regular_at(directory_fd: int, filename: str) -> None: + if not filename or "/" in filename or filename in {".", ".."}: + raise ManagedObjectIntegrityError("managed_object_integrity_failed") + flags = os.O_RDONLY + if hasattr(os, "O_NOFOLLOW"): + flags |= os.O_NOFOLLOW + expected = os.stat(filename, dir_fd=directory_fd, follow_symlinks=False) + _require_private_regular_file(expected) + fd = os.open(filename, flags, dir_fd=directory_fd) + temporary = f".delete-{uuid.uuid4().hex}.tmp" + try: + opened = os.fstat(fd) + _require_private_regular_file(opened) + if (opened.st_dev, opened.st_ino) != (expected.st_dev, expected.st_ino): + raise ManagedObjectIntegrityError("managed_object_integrity_failed") + os.replace( + filename, + temporary, + src_dir_fd=directory_fd, + dst_dir_fd=directory_fd, + ) + installed = os.stat(temporary, dir_fd=directory_fd, follow_symlinks=False) + _require_private_regular_file(installed) + if (opened.st_dev, opened.st_ino) != (installed.st_dev, installed.st_ino): + raise ManagedObjectIntegrityError("managed_object_integrity_failed") + os.unlink(temporary, dir_fd=directory_fd) + os.fsync(directory_fd) + finally: + os.close(fd) + + +def _regular_files(root: Path) -> list[Path]: + if not root.exists(): + return [] + files: list[Path] = [] + + def visit(directory: Path) -> None: + try: + directory_info = directory.lstat() + except FileNotFoundError: + return + if ( + not stat.S_ISDIR(directory_info.st_mode) + or stat.S_ISLNK(directory_info.st_mode) + or stat.S_IMODE(directory_info.st_mode) != 0o700 + ): + raise ManagedObjectIntegrityError("managed_object_integrity_failed") + with os.scandir(directory) as entries: + for entry in entries: + candidate = directory / entry.name + try: + info = entry.stat(follow_symlinks=False) + except FileNotFoundError: + continue + if stat.S_ISLNK(info.st_mode): + raise ManagedObjectIntegrityError("managed_object_integrity_failed") + if stat.S_ISDIR(info.st_mode): + visit(candidate) + elif stat.S_ISREG(info.st_mode): + _require_private_regular_file(info) + files.append(candidate) + else: + raise ManagedObjectIntegrityError("managed_object_integrity_failed") + + visit(root) + return files + + +def _public_signed_payload(envelope: Mapping[str, Any]) -> dict[str, Any]: + payload = envelope.get("payload") + if not isinstance(payload, dict): + raise ManagedObjectIntegrityError("managed_object_rotation_receipt_invalid") + return { + **payload, + "signature_algorithm": str(envelope.get("signature_algorithm") or ""), + "signature": str(envelope.get("signature") or ""), + "receipt_sha256": str(envelope.get("payload_sha256") or ""), + } + + +__all__ = [ + "EncryptedManagedObjectStore", + "ManagedObjectConflictError", + "ManagedObjectDeletionReceipt", + "ManagedObjectIntegrityError", + "ManagedObjectKind", + "ManagedObjectMetadataRepository", + "ManagedObjectNotFoundError", + "ManagedObjectReceipt", + "ManagedObjectScope", + "ManagedObjectStoreConfig", + "ManagedObjectStoreDisabledError", + "ManagedObjectStoreError", + "ManagedObjectStoreLimitError", + "PostgresManagedObjectMetadataRepository", + "SQLiteManagedObjectMetadataRepository", +] diff --git a/backend/storage/migrations.py b/backend/storage/migrations.py index 96eeb74..2641440 100644 --- a/backend/storage/migrations.py +++ b/backend/storage/migrations.py @@ -7,6 +7,7 @@ from __future__ import annotations +import json import sqlite3 import time from dataclasses import dataclass @@ -547,6 +548,604 @@ def _v7_baselines_and_history(conn: sqlite3.Connection) -> None: ) +def _snapshot_integrity(conn: sqlite3.Connection) -> None: + """Add a state_hash column to workbook_snapshots so rollback can verify integrity.""" + existing = {row[1] for row in conn.execute("PRAGMA table_info(workbook_snapshots)")} + if "state_hash" not in existing: + conn.execute("ALTER TABLE workbook_snapshots ADD COLUMN state_hash TEXT NOT NULL DEFAULT ''") + + +def _session_ownership(conn: sqlite3.Connection) -> None: + """Add durable tenant ownership without assigning legacy rows implicitly.""" + session_columns = {row[1] for row in conn.execute("PRAGMA table_info(sessions)")} + if "owner_user_id" not in session_columns: + conn.execute("ALTER TABLE sessions ADD COLUMN owner_user_id TEXT") + job_columns = {row[1] for row in conn.execute("PRAGMA table_info(background_jobs)")} + if "owner_user_id" not in job_columns: + conn.execute("ALTER TABLE background_jobs ADD COLUMN owner_user_id TEXT") + conn.execute( + "CREATE INDEX IF NOT EXISTS idx_sessions_owner_updated " + "ON sessions(owner_user_id, updated_at)" + ) + conn.execute( + "CREATE INDEX IF NOT EXISTS idx_background_jobs_owner_updated " + "ON background_jobs(owner_user_id, updated_at)" + ) + + +def _enterprise_oidc_identity_bindings(conn: sqlite3.Connection) -> None: + """Pin enterprise identities to immutable issuer+subject pairs. + + Legacy active rows with no binding remain usable only by enterprise API + key. Partial or duplicate legacy bindings fail closed for manual repair. + """ + table_exists = conn.execute( + "SELECT 1 FROM sqlite_master WHERE type = 'table' AND name = 'enterprise_users'" + ).fetchone() + if not table_exists: + return + + columns = {row[1] for row in conn.execute("PRAGMA table_info(enterprise_users)")} + if "oidc_issuer" not in columns: + conn.execute("ALTER TABLE enterprise_users ADD COLUMN oidc_issuer TEXT NOT NULL DEFAULT ''") + if "oidc_subject" not in columns: + conn.execute("ALTER TABLE enterprise_users ADD COLUMN oidc_subject TEXT NOT NULL DEFAULT ''") + + rows = conn.execute( + "SELECT user_id, status, metadata_json, oidc_issuer, oidc_subject FROM enterprise_users" + ).fetchall() + normalized: dict[str, dict] = {} + pair_owners: dict[tuple[str, str], list[str]] = {} + for row in rows: + user_id = str(row[0]) + status = str(row[1] or "active").strip().lower() + try: + raw_metadata = json.loads(row[2] or "{}") + except (TypeError, ValueError): + raw_metadata = {} + metadata = raw_metadata if isinstance(raw_metadata, dict) else {} + issuer = str(row[3] or metadata.get("oidc_issuer") or "").strip() + subject = str(row[4] or metadata.get("oidc_subject") or "").strip() + normalized[user_id] = { + "status": status, + "metadata": metadata, + "issuer": issuer, + "subject": subject, + } + if issuer and subject: + pair_owners.setdefault((issuer, subject), []).append(user_id) + + duplicate_users = { + user_id + for owners in pair_owners.values() + if len(owners) > 1 + for user_id in owners + } + for user_id, record in normalized.items(): + metadata = dict(record["metadata"]) + status = str(record["status"]) + issuer = str(record["issuer"]) + subject = str(record["subject"]) + if user_id in duplicate_users or bool(issuer) != bool(subject): + if issuer: + metadata["legacy_oidc_issuer"] = issuer + if subject: + metadata["legacy_oidc_subject"] = subject + metadata["oidc_binding_state"] = "duplicate" if user_id in duplicate_users else "partial" + metadata.pop("oidc_issuer", None) + metadata.pop("oidc_subject", None) + status = "binding_required" + issuer = "" + subject = "" + elif issuer and subject: + metadata["oidc_issuer"] = issuer + metadata["oidc_subject"] = subject + metadata.pop("oidc_binding_state", None) + elif status == "active": + status = "api_key_only" + metadata["oidc_binding_state"] = "legacy_unbound" + + conn.execute( + "UPDATE enterprise_users SET status = ?, metadata_json = ?, oidc_issuer = ?, oidc_subject = ? " + "WHERE user_id = ?", + (status, json.dumps(metadata), issuer, subject, user_id), + ) + + conn.executescript( + """ + CREATE UNIQUE INDEX IF NOT EXISTS idx_enterprise_users_oidc_identity + ON enterprise_users(oidc_issuer, oidc_subject) + WHERE oidc_issuer <> '' AND oidc_subject <> ''; + + CREATE TRIGGER IF NOT EXISTS trg_enterprise_users_oidc_complete_insert + BEFORE INSERT ON enterprise_users + WHEN (TRIM(COALESCE(NEW.oidc_issuer, '')) = '') + <> (TRIM(COALESCE(NEW.oidc_subject, '')) = '') + BEGIN + SELECT RAISE(ABORT, 'OIDC identity binding requires both issuer and subject'); + END; + + CREATE TRIGGER IF NOT EXISTS trg_enterprise_users_oidc_complete_update + BEFORE UPDATE OF oidc_issuer, oidc_subject ON enterprise_users + WHEN (TRIM(COALESCE(NEW.oidc_issuer, '')) = '') + <> (TRIM(COALESCE(NEW.oidc_subject, '')) = '') + BEGIN + SELECT RAISE(ABORT, 'OIDC identity binding requires both issuer and subject'); + END; + + CREATE TRIGGER IF NOT EXISTS trg_enterprise_users_active_binding_insert + BEFORE INSERT ON enterprise_users + WHEN LOWER(TRIM(NEW.status)) = 'active' + AND (TRIM(COALESCE(NEW.oidc_issuer, '')) = '' OR TRIM(COALESCE(NEW.oidc_subject, '')) = '') + BEGIN + SELECT RAISE(ABORT, 'Active enterprise user requires an immutable OIDC issuer and subject'); + END; + + CREATE TRIGGER IF NOT EXISTS trg_enterprise_users_active_binding_update + BEFORE UPDATE OF status, oidc_issuer, oidc_subject ON enterprise_users + WHEN LOWER(TRIM(NEW.status)) = 'active' + AND (TRIM(COALESCE(NEW.oidc_issuer, '')) = '' OR TRIM(COALESCE(NEW.oidc_subject, '')) = '') + BEGIN + SELECT RAISE(ABORT, 'Active enterprise user requires an immutable OIDC issuer and subject'); + END; + + CREATE TRIGGER IF NOT EXISTS trg_enterprise_users_oidc_immutable + BEFORE UPDATE OF oidc_issuer, oidc_subject ON enterprise_users + WHEN TRIM(COALESCE(OLD.oidc_issuer, '')) <> '' + AND ( + NEW.oidc_issuer <> OLD.oidc_issuer + OR NEW.oidc_subject <> OLD.oidc_subject + ) + BEGIN + SELECT RAISE(ABORT, 'OIDC issuer and subject are immutable once bound'); + END; + """ + ) + + +SQLITE_SESSION_LIFECYCLE_TABLES: tuple[str, ...] = ( + "sessions", + "migration_v05_imports", + "migration_v05_payloads", + "conversations", + "task_history", + "operation_plans", + "workbook_snapshots", + "audit_events", + "documents", + "citations", + "apply_attempts", + "document_chunks", + "background_jobs", + "metric_events", + "workbook_maps", + "operation_audits", + "verification_runs", + "agent_runs", + "source_evidence_graphs", + "operation_simulations", + "workbook_invariant_runs", + "workbook_truth_models", + "formula_graph_artifacts", + "operation_simulations_v2", + "rollback_snapshots_v2", + "agent_run_artifacts", + "repair_plans_v2", + "verification_reports_v2", + "agent_run_steps_v3", + "agent_checkpoints_v3", + "operation_diff_previews_v2", + "source_cell_provenance_v2", + "v7_workbook_reviews", + "local_replay_artifacts", + "local_replay_baselines", +) + +# Authoritative child-first deletion order. Tables with non-session foreign +# keys precede their parents: apply attempts before snapshots/plans, snapshots +# and plan artifacts before plans, document chunks before documents, and the +# session row last. Keep this inventory identical to the lifecycle table set. +SQLITE_SESSION_PURGE_ORDER: tuple[str, ...] = ( + "migration_v05_payloads", + "migration_v05_imports", + "local_replay_artifacts", + "local_replay_baselines", + "v7_workbook_reviews", + "verification_reports_v2", + "repair_plans_v2", + "agent_run_artifacts", + "agent_run_steps_v3", + "agent_checkpoints_v3", + "rollback_snapshots_v2", + "operation_diff_previews_v2", + "source_cell_provenance_v2", + "operation_simulations_v2", + "formula_graph_artifacts", + "workbook_truth_models", + "workbook_invariant_runs", + "operation_simulations", + "source_evidence_graphs", + "agent_runs", + "metric_events", + "apply_attempts", + "operation_audits", + "verification_runs", + "citations", + "audit_events", + "workbook_snapshots", + "background_jobs", + "document_chunks", + "conversations", + "task_history", + "workbook_maps", + "operation_plans", + "documents", + "sessions", +) +if ( + len(SQLITE_SESSION_PURGE_ORDER) != len(SQLITE_SESSION_LIFECYCLE_TABLES) + or set(SQLITE_SESSION_PURGE_ORDER) != set(SQLITE_SESSION_LIFECYCLE_TABLES) +): + raise RuntimeError("SQLite lifecycle purge order must cover every session table exactly once") + + +def _install_session_lifecycle_triggers( + conn: sqlite3.Connection, + table_name: str, +) -> None: + table_exists = conn.execute( + "SELECT 1 FROM sqlite_master WHERE type = 'table' AND name = ?", + (table_name,), + ).fetchone() + if not table_exists: + # Later migrations can extend the authoritative lifecycle inventory; + # their create step installs the same triggers once the table exists. + return + trigger_prefix = f"trg_{table_name}_session_lifecycle" + conn.execute( + f""" + CREATE TRIGGER IF NOT EXISTS {trigger_prefix}_insert + BEFORE INSERT ON {table_name} + WHEN TRIM(COALESCE(NEW.session_id, '')) <> '' + BEGIN + SELECT CASE WHEN NOT EXISTS ( + SELECT 1 + FROM session_lifecycle AS lifecycle + WHERE lifecycle.session_id = NEW.session_id + AND lifecycle.state = 'active' + ) THEN RAISE(ABORT, 'session lifecycle is not active') END; + END + """ + ) + conn.execute( + f""" + CREATE TRIGGER IF NOT EXISTS {trigger_prefix}_update + BEFORE UPDATE ON {table_name} + WHEN TRIM(COALESCE(NEW.session_id, '')) <> '' + BEGIN + SELECT CASE WHEN OLD.session_id <> NEW.session_id + THEN RAISE(ABORT, 'session_id is immutable') END; + SELECT CASE WHEN NOT EXISTS ( + SELECT 1 + FROM session_lifecycle AS lifecycle + WHERE lifecycle.session_id = NEW.session_id + AND lifecycle.state = 'active' + ) THEN RAISE(ABORT, 'session lifecycle is not active') END; + END + """ + ) + + +def purge_sqlite_session_rows( + conn: sqlite3.Connection, + session_id: str, +) -> dict[str, int]: + """Delete one session using the authoritative child-first FK order.""" + normalized_session_id = str(session_id or "").strip() + if not normalized_session_id: + raise ValueError("session_id is required") + available_tables = { + str(row[0]) + for row in conn.execute( + "SELECT name FROM sqlite_master WHERE type = 'table'" + ).fetchall() + } + removed: dict[str, int] = {} + if {"plan_revisions", "operation_plans"}.issubset(available_tables): + cursor = conn.execute( + "DELETE FROM plan_revisions WHERE plan_id IN " + "(SELECT plan_id FROM operation_plans WHERE session_id = ?)", + (normalized_session_id,), + ) + removed["plan_revisions"] = max(int(cursor.rowcount), 0) + for table_name in SQLITE_SESSION_PURGE_ORDER: + if table_name not in available_tables: + continue + cursor = conn.execute( + f"DELETE FROM {table_name} WHERE session_id = ?", # noqa: S608 -- static inventory + (normalized_session_id,), + ) + removed[table_name] = max(int(cursor.rowcount), 0) + return removed + + +def assert_sqlite_session_lifecycle_integrity(conn: sqlite3.Connection) -> None: + """Fail closed on FK violations or session payload outside its lifecycle.""" + foreign_key_violations = conn.execute("PRAGMA foreign_key_check").fetchall() + if foreign_key_violations: + violation = foreign_key_violations[0] + raise RuntimeError( + "SQLite foreign-key integrity check failed: " + f"table={violation[0]} rowid={violation[1]} parent={violation[2]}" + ) + + available_tables = { + str(row[0]) + for row in conn.execute( + "SELECT name FROM sqlite_master WHERE type = 'table'" + ).fetchall() + } + invalid_session = conn.execute( + "SELECT session.session_id FROM sessions AS session " + "LEFT JOIN session_lifecycle AS lifecycle " + "ON lifecycle.session_id = session.session_id " + "WHERE lifecycle.session_id IS NULL OR lifecycle.state = 'verified' LIMIT 1" + ).fetchone() + if invalid_session: + raise RuntimeError( + f"Session row violates lifecycle state: {invalid_session[0]}" + ) + for table_name in SQLITE_SESSION_LIFECYCLE_TABLES: + if table_name == "sessions" or table_name not in available_tables: + continue + orphan = conn.execute( + f"SELECT item.session_id FROM {table_name} AS item " # noqa: S608 -- static inventory + "LEFT JOIN sessions AS session ON session.session_id = item.session_id " + "LEFT JOIN session_lifecycle AS lifecycle ON lifecycle.session_id = item.session_id " + "WHERE TRIM(COALESCE(item.session_id, '')) <> '' " + "AND (session.session_id IS NULL OR lifecycle.session_id IS NULL " + "OR lifecycle.state = 'verified') LIMIT 1" + ).fetchone() + if orphan: + raise RuntimeError( + f"Session lifecycle orphan remains in {table_name}: {orphan[0]}" + ) + + +def _session_lifecycle_fencing(conn: sqlite3.Connection) -> None: + """Fence every session-bearing SQLite write with a durable tombstone.""" + conn.executescript( + """ + CREATE TABLE IF NOT EXISTS session_lifecycle ( + session_id TEXT PRIMARY KEY, + state TEXT NOT NULL CHECK (state IN ('active', 'purging', 'verified', 'failed')), + receipt_id TEXT NOT NULL DEFAULT '', + started_at REAL NOT NULL DEFAULT 0, + completed_at REAL NOT NULL DEFAULT 0, + errors_json TEXT NOT NULL DEFAULT '[]', + updated_at REAL NOT NULL + ); + + INSERT INTO session_lifecycle + (session_id, state, receipt_id, started_at, completed_at, errors_json, updated_at) + SELECT session_id, 'active', '', 0, 0, '[]', updated_at + FROM sessions + WHERE TRIM(session_id) <> '' + ON CONFLICT(session_id) DO NOTHING; + + CREATE INDEX IF NOT EXISTS idx_session_lifecycle_state_updated + ON session_lifecycle(state, updated_at); + + CREATE TRIGGER IF NOT EXISTS trg_session_lifecycle_no_reactivation + BEFORE UPDATE OF state ON session_lifecycle + WHEN OLD.state <> 'active' AND NEW.state = 'active' + BEGIN + SELECT RAISE(ABORT, 'session lifecycle tombstone cannot be reactivated'); + END; + """ + ) + + for table_name in SQLITE_SESSION_LIFECYCLE_TABLES: + _install_session_lifecycle_triggers(conn, table_name) + + +def _enterprise_admin_audit_isolation(conn: sqlite3.Connection) -> None: + """Move administrator audits out of attacker-claimable session storage.""" + conn.executescript( + """ + CREATE TABLE IF NOT EXISTS enterprise_admin_audits ( + audit_id TEXT PRIMARY KEY, + event_type TEXT NOT NULL, + payload_json TEXT NOT NULL DEFAULT '{}', + created_at REAL NOT NULL + ); + + CREATE INDEX IF NOT EXISTS idx_enterprise_admin_audits_created + ON enterprise_admin_audits(created_at DESC); + + INSERT INTO enterprise_admin_audits + (audit_id, event_type, payload_json, created_at) + SELECT audit_id, event_type, payload, created_at + FROM audit_events + WHERE event_type LIKE 'enterprise_admin.%' + ON CONFLICT(audit_id) DO NOTHING; + + DELETE FROM audit_events + WHERE event_type LIKE 'enterprise_admin.%'; + + CREATE TRIGGER IF NOT EXISTS trg_audit_events_no_enterprise_admin_insert + BEFORE INSERT ON audit_events + WHEN NEW.event_type LIKE 'enterprise_admin.%' + BEGIN + SELECT RAISE(ABORT, 'enterprise administrator audits require global storage'); + END; + + CREATE TRIGGER IF NOT EXISTS trg_audit_events_no_enterprise_admin_update + BEFORE UPDATE OF event_type ON audit_events + WHEN NEW.event_type LIKE 'enterprise_admin.%' + BEGIN + SELECT RAISE(ABORT, 'enterprise administrator audits require global storage'); + END; + """ + ) + + +def _repair_session_lifecycle_orphans(conn: sqlite3.Connection) -> None: + """Reconcile restored/legacy rows according to their durable lifecycle state.""" + now = time.time() + _session_lifecycle_fencing(conn) + source_tables: dict[str, set[str]] = {} + available_tables = { + str(row[0]) + for row in conn.execute( + "SELECT name FROM sqlite_master WHERE type = 'table'" + ).fetchall() + } + for table_name in SQLITE_SESSION_LIFECYCLE_TABLES: + if table_name not in available_tables: + continue + rows = conn.execute( + f"SELECT DISTINCT session_id FROM {table_name} " # noqa: S608 -- static inventory + "WHERE TRIM(COALESCE(session_id, '')) <> ''" + ).fetchall() + for row in rows: + session_id = str(row[0]).strip() + source_tables.setdefault(session_id, set()).add(table_name) + + for session_id in sorted(source_tables): + tables = source_tables[session_id] + session = conn.execute( + "SELECT 1 FROM sessions WHERE session_id = ?", + (session_id,), + ).fetchone() + lifecycle = conn.execute( + "SELECT state FROM session_lifecycle WHERE session_id = ?", + (session_id,), + ).fetchone() + lifecycle_state = str(lifecycle[0] or "") if lifecycle else "" + if lifecycle_state == "verified": + # VERIFIED is terminal: destroy every restored payload using the + # same child-first order as ordinary lifecycle deletion. + purge_sqlite_session_rows(conn, session_id) + continue + if not lifecycle: + lifecycle_state = "active" + conn.execute( + "INSERT INTO session_lifecycle " + "(session_id, state, receipt_id, started_at, completed_at, errors_json, updated_at) " + "VALUES (?, 'active', '', 0, 0, '[]', ?)", + (session_id, now), + ) + if not session: + _insert_lifecycle_recovery_session( + conn, + session_id=session_id, + lifecycle_state=lifecycle_state, + source_tables=tables, + now=now, + ) + + assert_sqlite_session_lifecycle_integrity(conn) + + +def _insert_lifecycle_recovery_session( + conn: sqlite3.Connection, + *, + session_id: str, + lifecycle_state: str, + source_tables: set[str], + now: float, +) -> None: + """Restore ownerless inventory without reopening non-active reads/writes.""" + metadata = { + "migration": "2026_08_10_0019_session_lifecycle_orphan_repair", + "quarantined_legacy_orphan": True, + "source_tables": sorted(table for table in source_tables if table != "sessions"), + } + bypass_nonactive_insert = lifecycle_state in {"purging", "failed"} + if bypass_nonactive_insert: + metadata["lifecycle_recovery_state"] = lifecycle_state + # The application trigger correctly rejects ordinary resurrection. + # This transactional startup-only repair recreates inventory solely so + # the existing purge receipt can retry and enumerate managed files. + conn.execute("DROP TRIGGER IF EXISTS trg_sessions_session_lifecycle_insert") + try: + conn.execute( + "INSERT INTO sessions " + "(session_id, owner_user_id, created_at, updated_at, metadata) " + "VALUES (?, NULL, ?, ?, ?)", + (session_id, now, now, json.dumps(metadata, sort_keys=True)), + ) + finally: + if bypass_nonactive_insert: + _install_session_lifecycle_triggers(conn, "sessions") + + +def _destroy_deleted_session_orphan_quarantine(conn: sqlite3.Connection) -> None: + """Remove raw payloads retained by the superseded orphan repair migration.""" + conn.execute("DROP TABLE IF EXISTS session_lifecycle_orphan_quarantine") + + +def _v05_neutral_import_store(conn: sqlite3.Connection) -> None: + """Create the one-way, lifecycle-owned neutral v0.5 intake boundary.""" + conn.executescript( + """ + CREATE TABLE IF NOT EXISTS migration_v05_imports ( + import_id TEXT PRIMARY KEY, + session_id TEXT NOT NULL UNIQUE, + schema_version TEXT NOT NULL, + state TEXT NOT NULL CHECK ( + state IN ( + 'validated_pending_import', + 'importing', + 'imported', + 'failed_retryable' + ) + ), + version INTEGER NOT NULL DEFAULT 0 CHECK (version >= 0), + attempt_count INTEGER NOT NULL DEFAULT 0 CHECK (attempt_count >= 0), + binding_sha256 TEXT NOT NULL UNIQUE, + archive_sha256 TEXT NOT NULL, + content_sha256 TEXT NOT NULL UNIQUE, + owner_scope_sha256 TEXT NOT NULL, + manifest_sha256 TEXT NOT NULL, + stage_receipt_sha256 TEXT NOT NULL, + counts_json TEXT NOT NULL, + payload_count INTEGER NOT NULL DEFAULT 0 CHECK (payload_count >= 0), + receipt_json TEXT NOT NULL DEFAULT '', + failure_code TEXT NOT NULL DEFAULT '', + created_at REAL NOT NULL, + updated_at REAL NOT NULL, + FOREIGN KEY (session_id) REFERENCES sessions(session_id) + ); + + CREATE TABLE IF NOT EXISTS migration_v05_payloads ( + import_id TEXT NOT NULL, + session_id TEXT NOT NULL, + relative_path TEXT NOT NULL, + payload_kind TEXT NOT NULL CHECK ( + payload_kind IN ('blob', 'document', 'session', 'workflow_recipe') + ), + payload_sha256 TEXT NOT NULL, + size_bytes INTEGER NOT NULL CHECK (size_bytes >= 0), + payload_bytes BLOB NOT NULL, + PRIMARY KEY (import_id, relative_path), + FOREIGN KEY (import_id) REFERENCES migration_v05_imports(import_id) ON DELETE CASCADE, + FOREIGN KEY (session_id) REFERENCES sessions(session_id) + ); + + CREATE INDEX IF NOT EXISTS idx_migration_v05_imports_state + ON migration_v05_imports(state, updated_at); + CREATE INDEX IF NOT EXISTS idx_migration_v05_imports_session + ON migration_v05_imports(session_id); + CREATE INDEX IF NOT EXISTS idx_migration_v05_payloads_session + ON migration_v05_payloads(session_id, import_id); + """ + ) + _install_session_lifecycle_triggers(conn, "migration_v05_imports") + _install_session_lifecycle_triggers(conn, "migration_v05_payloads") + + MIGRATIONS: tuple[Migration, ...] = ( Migration("2026_05_01_0001_baseline_schema", "Record existing SessionStore baseline schema", _baseline), Migration("2026_05_01_0002_apply_attempts", "Durable two-phase apply attempt tracking", _apply_attempts), @@ -561,6 +1160,38 @@ def _v7_baselines_and_history(conn: sqlite3.Connection) -> None: Migration("2026_05_01_0011_v7_local_review_artifacts", "V7 local workbook reviews and anonymized replay artifacts", _v7_local_review_artifacts), Migration("2026_05_01_0012_v7_default_readiness_reports", "V7 default-prep readiness reports", _v7_default_readiness_reports), Migration("2026_05_01_0013_v7_replay_baselines", "V7 replay baselines and readiness history support", _v7_baselines_and_history), + Migration("2026_05_01_0014_snapshot_integrity", "Snapshot integrity hashes for rollback verification", _snapshot_integrity), + Migration("2026_08_09_0015_session_ownership", "Fail-closed ownership for sessions and workbook jobs", _session_ownership), + Migration( + "2026_08_09_0016_enterprise_oidc_identity", + "Immutable issuer and subject bindings for active enterprise users", + _enterprise_oidc_identity_bindings, + ), + Migration( + "2026_08_09_0017_session_lifecycle_fencing", + "Durable session tombstones and cross-process write fencing", + _session_lifecycle_fencing, + ), + Migration( + "2026_08_10_0018_enterprise_admin_audit_isolation", + "Move enterprise administrator audits outside the user session namespace", + _enterprise_admin_audit_isolation, + ), + Migration( + "2026_08_10_0019_session_lifecycle_orphan_repair", + "Repair live legacy rows and destroy rows behind lifecycle tombstones", + _repair_session_lifecycle_orphans, + ), + Migration( + "2026_08_10_0020_destroy_deleted_session_orphan_quarantine", + "Destroy raw deleted-session payloads retained by legacy orphan quarantine", + _destroy_deleted_session_orphan_quarantine, + ), + Migration( + "2026_08_10_0021_v05_neutral_import_store", + "Lifecycle-owned one-way neutral v0.5 import state and payload ledger", + _v05_neutral_import_store, + ), ) @@ -588,6 +1219,11 @@ def run_migrations(conn: sqlite3.Connection) -> list[str]: (migration.id, migration.description, time.time()), ) newly_applied.append(migration.id) + # Migration ledgers describe schema history, not present data integrity. + # Re-run the idempotent repair after every startup so partial restores and + # trigger bypasses cannot retain deleted payload or strand retry inventory. + _destroy_deleted_session_orphan_quarantine(conn) + _repair_session_lifecycle_orphans(conn) return newly_applied diff --git a/backend/storage/postgres.py b/backend/storage/postgres.py index f027a63..7bfec9b 100644 --- a/backend/storage/postgres.py +++ b/backend/storage/postgres.py @@ -10,18 +10,894 @@ from __future__ import annotations +import asyncio +import hashlib +import hmac import json +import os +import secrets +import threading import time -import asyncio import uuid -from typing import Optional +from collections.abc import AsyncIterator +from contextlib import asynccontextmanager +from typing import Any, Optional import structlog -from backend.storage.protocol import SessionStoreProtocol, ConversationMessage +from backend.api.security.ownership import OwnershipNotFoundError, current_principal +from backend.storage.enterprise_identity import ( + api_key_status_allows, + prepare_enterprise_identity_binding, +) +from backend.storage.protocol import ConversationMessage +from backend.storage.v05_import_store import ( + MATERIALIZATION_MAX_ENTRY_BYTES, + MATERIALIZATION_MAX_PAYLOAD_BYTES, + MATERIALIZATION_MAX_PAYLOADS, + NEUTRAL_IMPORT_STATES, + NeutralV05ImportError, + binding_sha256 as v05_binding_sha256, + build_import_receipt, + canonical_json as v05_canonical_json, + ensure_row_binding, + neutral_import_id, + neutral_session_id, + normalize_import_bundle, + normalize_owner_id, + normalize_sha256, + public_import_status, + sha256_bytes, + sha256_text, + validate_import_receipt, +) logger = structlog.get_logger() +POSTGRES_TYPED_PROJECTION_TABLES = [ + "v7_default_readiness_reports", + "local_replay_artifacts", + "local_replay_baselines", + "agent_run_artifacts", + "verification_reports_v2", + "workbook_benchmark_fixture_results", + "operation_audits", + "workbook_maps", + "source_evidence_graphs", + "operation_simulations", + "workbook_invariant_runs", + "agent_runs", + "workbook_truth_models", + "formula_graph_artifacts", + "operation_simulations_v2", + "v7_workbook_reviews", +] + +POSTGRES_SESSION_LIFECYCLE_TABLES = ( + "sessions", + "migration_v05_imports", + "migration_v05_payloads", + "conversations", + "task_history", + "flagship_artifacts", + "local_replay_artifacts", + "local_replay_baselines", + "agent_run_artifacts", + "verification_reports_v2", + "operation_audits", + "workbook_maps", + "source_evidence_graphs", + "operation_simulations", + "workbook_invariant_runs", + "agent_runs", + "workbook_truth_models", + "formula_graph_artifacts", + "operation_simulations_v2", + "v7_workbook_reviews", +) + +POSTGRES_GLOBAL_ARTIFACT_TYPES = frozenset( + { + "benchmark_run", + "enterprise_api_key", + "enterprise_config", + "enterprise_team", + "enterprise_user", + "v7_default_readiness_report", + "workbook_benchmark_fixture", + } +) + +SESSION_BOUND_ARTIFACT_TYPES = { + "agent_run", + "agent_run_step_artifact_v3", + "apply_attempt", + "brain_artifact_bundle", + "brain_run", + "citation", + "document", + "formula_graph", + "operation_audit", + "operation_diff_preview_v2", + "operation_plan", + "operation_simulation", + "operation_simulation_v2", + "plan_revision", + "repair_plan", + "rollback_snapshot", + "source_cell_provenance_v2", + "source_evidence_graph", + "v7_workbook_review", + "verification_report", + "verification_run", + "workbook_job", + "workbook_invariant", + "workbook_map", + "workbook_snapshot", + "workbook_truth_model", +} + + +def _artifact_type_requires_session(artifact_type: str) -> bool: + normalized = str(artifact_type or "") + return ( + normalized in SESSION_BOUND_ARTIFACT_TYPES + or normalized.startswith("agent_run:") + or normalized.startswith("audit:") + or normalized.startswith("local_replay:") + ) + + +def _artifact_is_session_scoped(artifact_type: str, session_id: str) -> bool: + normalized_session_id = str(session_id or "").strip() + if not normalized_session_id or artifact_type in POSTGRES_GLOBAL_ARTIFACT_TYPES: + return False + return True + +POSTGRES_TYPED_PROJECTION_MIGRATIONS = { + "postgres_v7_readiness_projection_v1": ["v7_default_readiness_reports"], + "postgres_local_replay_projection_v1": ["local_replay_artifacts", "local_replay_baselines"], + "postgres_run_evidence_projection_v1": [ + "agent_run_artifacts", + "verification_reports_v2", + "workbook_benchmark_fixture_results", + "operation_audits", + ], + "postgres_workbook_intelligence_projection_v1": [ + "workbook_maps", + "source_evidence_graphs", + "operation_simulations", + "workbook_invariant_runs", + "agent_runs", + "workbook_truth_models", + "formula_graph_artifacts", + "operation_simulations_v2", + "v7_workbook_reviews", + ], +} + +POSTGRES_TYPED_PROJECTION_BACKFILLS = { + "v7_default_readiness_reports": ( + """ + INSERT INTO v7_default_readiness_reports + (readiness_id, status, default_allowed, score, report_json, created_at, updated_at) + SELECT + artifact_id, + COALESCE(NULLIF(payload->>'status', ''), 'warning'), + CASE + WHEN jsonb_typeof(payload->'defaultAllowed') = 'boolean' THEN (payload->>'defaultAllowed')::boolean + ELSE FALSE + END, + CASE + WHEN jsonb_typeof(payload->'score') = 'number' THEN (payload->>'score')::double precision + ELSE 0 + END, + payload, + created_at, + updated_at + FROM flagship_artifacts + WHERE artifact_type = 'v7_default_readiness_report' + ON CONFLICT(readiness_id) DO NOTHING; + """, + ), + "local_replay_artifacts": ( + """ + INSERT INTO local_replay_artifacts + ( + replay_id, + session_id, + artifact_kind, + schema_version, + v7_gate_status, + accuracy_score, + confidence_gap_count, + checkpoint_reason_count, + formula_failure_count, + truth_schema, + simulation_schema, + replay_json, + created_at, + updated_at + ) + SELECT + artifact_id, + session_id, + regexp_replace(artifact_type, '^local_replay:', ''), + COALESCE(NULLIF(payload->>'schemaVersion', ''), ''), + COALESCE(NULLIF(payload->>'v7GateStatus', ''), ''), + CASE + WHEN jsonb_typeof(payload->'accuracyScore') = 'number' THEN (payload->>'accuracyScore')::double precision + ELSE 0 + END, + CASE + WHEN jsonb_typeof(payload->'confidenceGapCount') = 'number' THEN ((payload->>'confidenceGapCount')::numeric)::integer + ELSE 0 + END, + CASE + WHEN jsonb_typeof(payload->'checkpointReasonCount') = 'number' THEN ((payload->>'checkpointReasonCount')::numeric)::integer + ELSE 0 + END, + CASE + WHEN jsonb_typeof(payload->'formulaFailureCount') = 'number' THEN ((payload->>'formulaFailureCount')::numeric)::integer + ELSE 0 + END, + COALESCE(NULLIF(payload->>'truthSchema', ''), ''), + COALESCE(NULLIF(payload->>'simulationSchema', ''), ''), + payload, + created_at, + updated_at + FROM flagship_artifacts AS backfill_artifact + WHERE artifact_type LIKE 'local_replay:%' + AND EXISTS ( + SELECT 1 FROM sessions AS backfill_session + JOIN session_lifecycle AS backfill_lifecycle + ON backfill_lifecycle.session_id = backfill_session.session_id + WHERE backfill_session.session_id = backfill_artifact.session_id + AND backfill_lifecycle.state = 'active' + ) + ON CONFLICT(replay_id) DO NOTHING; + """, + ), + "local_replay_baselines": ( + """ + INSERT INTO local_replay_baselines + ( + baseline_id, + session_id, + status, + production_evidence, + evidence_class, + replay_count, + deterministic_drift_class, + baseline_json, + created_at, + updated_at + ) + SELECT + artifact_id, + session_id, + COALESCE(NULLIF(payload->>'status', ''), NULLIF(payload->'metadata'->>'status', ''), 'warning'), + CASE + WHEN jsonb_typeof(payload->'productionEvidence') = 'boolean' + THEN (payload->>'productionEvidence')::boolean + WHEN jsonb_typeof(payload->'metadata'->'productionEvidence') = 'boolean' + THEN (payload->'metadata'->>'productionEvidence')::boolean + ELSE FALSE + END, + COALESCE(NULLIF(payload->>'evidenceClass', ''), NULLIF(payload->'metadata'->>'evidenceClass', ''), ''), + CASE + WHEN jsonb_typeof(payload->'replayCount') = 'number' THEN ((payload->>'replayCount')::numeric)::integer + WHEN jsonb_typeof(payload->'metadata'->'replayCount') = 'number' + THEN ((payload->'metadata'->>'replayCount')::numeric)::integer + ELSE 0 + END, + COALESCE( + NULLIF(payload->>'deterministicDriftClass', ''), + NULLIF(payload->'baselineComparison'->>'deterministicDriftClass', ''), + '' + ), + payload, + created_at, + updated_at + FROM flagship_artifacts AS backfill_artifact + WHERE artifact_type = 'local_replay_baseline' + AND EXISTS ( + SELECT 1 FROM sessions AS backfill_session + JOIN session_lifecycle AS backfill_lifecycle + ON backfill_lifecycle.session_id = backfill_session.session_id + WHERE backfill_session.session_id = backfill_artifact.session_id + AND backfill_lifecycle.state = 'active' + ) + ON CONFLICT(baseline_id) DO NOTHING; + """, + ), + "agent_run_artifacts": ( + """ + INSERT INTO agent_run_artifacts + (artifact_id, run_id, session_id, artifact_type, plan_id, artifact_json, created_at, updated_at) + SELECT + artifact_id, + run_id, + session_id, + CASE + WHEN artifact_type LIKE 'agent_run:%' THEN regexp_replace(artifact_type, '^agent_run:', '') + WHEN artifact_type = 'agent_run_step_artifact_v3' + THEN CONCAT('step:', COALESCE(NULLIF(payload->>'artifactType', ''), 'artifact')) + ELSE artifact_type + END, + plan_id, + payload, + created_at, + updated_at + FROM flagship_artifacts AS backfill_artifact + WHERE ( + artifact_type LIKE 'agent_run:%' + OR artifact_type = 'agent_run_step_artifact_v3' + OR artifact_type = 'brain_run' + OR artifact_type = 'brain_artifact_bundle' + ) + AND EXISTS ( + SELECT 1 FROM sessions AS backfill_session + JOIN session_lifecycle AS backfill_lifecycle + ON backfill_lifecycle.session_id = backfill_session.session_id + WHERE backfill_session.session_id = backfill_artifact.session_id + AND backfill_lifecycle.state = 'active' + ) + ON CONFLICT(artifact_id) DO NOTHING; + """, + ), + "verification_reports_v2": ( + """ + INSERT INTO verification_reports_v2 + (verification_id, session_id, run_id, plan_id, status, report_json, created_at, updated_at) + SELECT + artifact_id, + session_id, + run_id, + plan_id, + COALESCE(NULLIF(payload->>'status', ''), 'warning'), + payload, + created_at, + updated_at + FROM flagship_artifacts AS backfill_artifact + WHERE artifact_type = 'verification_report' + AND EXISTS ( + SELECT 1 FROM sessions AS backfill_session + JOIN session_lifecycle AS backfill_lifecycle + ON backfill_lifecycle.session_id = backfill_session.session_id + WHERE backfill_session.session_id = backfill_artifact.session_id + AND backfill_lifecycle.state = 'active' + ) + ON CONFLICT(verification_id) DO NOTHING; + """, + ), + "workbook_benchmark_fixture_results": ( + """ + INSERT INTO workbook_benchmark_fixture_results + (fixture_result_id, run_id, fixture_id, status, result_json, created_at) + SELECT + artifact_id, + run_id, + COALESCE(NULLIF(payload->>'fixtureId', ''), NULLIF(payload->>'fixture_id', ''), ''), + COALESCE(NULLIF(payload->>'status', ''), 'warning'), + payload, + created_at + FROM flagship_artifacts + WHERE artifact_type = 'workbook_benchmark_fixture' + ON CONFLICT(fixture_result_id) DO NOTHING; + """, + ), + "operation_audits": ( + """ + INSERT INTO operation_audits + ( + audit_id, + session_id, + plan_id, + operation_id, + operation_type, + dirty_range_count, + warning_count, + payload_json, + created_at + ) + SELECT + artifact_id, + session_id, + plan_id, + COALESCE(NULLIF(payload->>'operation_id', ''), NULLIF(payload->>'operationId', ''), ''), + COALESCE(NULLIF(payload->>'type', ''), NULLIF(payload->>'operation_type', ''), NULLIF(payload->>'operationType', ''), ''), + CASE + WHEN jsonb_typeof(payload->'dirty_ranges') = 'array' THEN jsonb_array_length(payload->'dirty_ranges') + WHEN jsonb_typeof(payload->'dirtyRanges') = 'array' THEN jsonb_array_length(payload->'dirtyRanges') + ELSE 0 + END, + CASE + WHEN jsonb_typeof(payload->'warnings') = 'array' THEN jsonb_array_length(payload->'warnings') + ELSE 0 + END, + payload, + created_at + FROM flagship_artifacts AS backfill_artifact + WHERE artifact_type = 'operation_audit' + AND EXISTS ( + SELECT 1 FROM sessions AS backfill_session + JOIN session_lifecycle AS backfill_lifecycle + ON backfill_lifecycle.session_id = backfill_session.session_id + WHERE backfill_session.session_id = backfill_artifact.session_id + AND backfill_lifecycle.state = 'active' + ) + ON CONFLICT(audit_id) DO NOTHING; + """, + ), + "workbook_maps": ( + """ + INSERT INTO workbook_maps + ( + map_id, + session_id, + workbook_fingerprint, + workbook_name, + schema_version, + entity_count, + formula_count, + risk_count, + map_json, + created_at, + updated_at + ) + SELECT + artifact_id, + session_id, + COALESCE(NULLIF(payload->>'workbookFingerprint', ''), ''), + COALESCE(NULLIF(payload->>'workbookName', ''), ''), + COALESCE(NULLIF(payload->>'schemaVersion', ''), ''), + CASE + WHEN jsonb_typeof(payload->'entities') = 'array' THEN jsonb_array_length(payload->'entities') + WHEN jsonb_typeof(payload->'semanticEntities') = 'array' THEN jsonb_array_length(payload->'semanticEntities') + ELSE 0 + END, + CASE + WHEN jsonb_typeof(payload->'formulaFacts') = 'array' THEN jsonb_array_length(payload->'formulaFacts') + WHEN jsonb_typeof(payload->'formulas') = 'array' THEN jsonb_array_length(payload->'formulas') + ELSE 0 + END, + CASE + WHEN jsonb_typeof(payload->'risks') = 'array' THEN jsonb_array_length(payload->'risks') + WHEN jsonb_typeof(payload->'requiredInspections') = 'array' THEN jsonb_array_length(payload->'requiredInspections') + WHEN jsonb_typeof(payload->'confidenceGaps') = 'array' THEN jsonb_array_length(payload->'confidenceGaps') + ELSE 0 + END, + payload, + created_at, + updated_at + FROM flagship_artifacts AS backfill_artifact + WHERE artifact_type = 'workbook_map' + AND EXISTS ( + SELECT 1 FROM sessions AS backfill_session + JOIN session_lifecycle AS backfill_lifecycle + ON backfill_lifecycle.session_id = backfill_session.session_id + WHERE backfill_session.session_id = backfill_artifact.session_id + AND backfill_lifecycle.state = 'active' + ) + ON CONFLICT(map_id) DO NOTHING; + """, + ), + "source_evidence_graphs": ( + """ + INSERT INTO source_evidence_graphs + (graph_id, session_id, document_ids, graph_json, created_at, updated_at) + SELECT + artifact_id, + session_id, + CASE + WHEN jsonb_typeof(payload->'documentIds') = 'array' THEN payload->'documentIds' + WHEN jsonb_typeof(payload->'document_ids') = 'array' THEN payload->'document_ids' + ELSE '[]'::jsonb + END, + payload, + created_at, + updated_at + FROM flagship_artifacts AS backfill_artifact + WHERE artifact_type = 'source_evidence_graph' + AND EXISTS ( + SELECT 1 FROM sessions AS backfill_session + JOIN session_lifecycle AS backfill_lifecycle + ON backfill_lifecycle.session_id = backfill_session.session_id + WHERE backfill_session.session_id = backfill_artifact.session_id + AND backfill_lifecycle.state = 'active' + ) + ON CONFLICT(graph_id) DO NOTHING; + """, + ), + "operation_simulations": ( + """ + INSERT INTO operation_simulations + (simulation_id, session_id, plan_id, run_id, status, simulation_json, created_at) + SELECT + artifact_id, + session_id, + plan_id, + run_id, + COALESCE(NULLIF(payload->>'status', ''), NULLIF(payload->>'simulationGate', ''), 'warning'), + payload, + created_at + FROM flagship_artifacts AS backfill_artifact + WHERE artifact_type = 'operation_simulation' + AND EXISTS ( + SELECT 1 FROM sessions AS backfill_session + JOIN session_lifecycle AS backfill_lifecycle + ON backfill_lifecycle.session_id = backfill_session.session_id + WHERE backfill_session.session_id = backfill_artifact.session_id + AND backfill_lifecycle.state = 'active' + ) + ON CONFLICT(simulation_id) DO NOTHING; + """, + ), + "workbook_invariant_runs": ( + """ + INSERT INTO workbook_invariant_runs + (invariant_run_id, session_id, run_id, plan_id, status, report_json, created_at) + SELECT + artifact_id, + session_id, + run_id, + plan_id, + COALESCE(NULLIF(payload->>'status', ''), 'warning'), + payload, + created_at + FROM flagship_artifacts AS backfill_artifact + WHERE artifact_type = 'workbook_invariant' + AND EXISTS ( + SELECT 1 FROM sessions AS backfill_session + JOIN session_lifecycle AS backfill_lifecycle + ON backfill_lifecycle.session_id = backfill_session.session_id + WHERE backfill_session.session_id = backfill_artifact.session_id + AND backfill_lifecycle.state = 'active' + ) + ON CONFLICT(invariant_run_id) DO NOTHING; + """, + ), + "agent_runs": ( + """ + INSERT INTO agent_runs + ( + run_id, + session_id, + status, + schema_version, + plan_id, + semantic_map_id, + simulation_id, + verification_id, + v7_gate_status, + execution_group_count, + checkpoint_count, + prompt, + run_json, + created_at, + updated_at + ) + SELECT + artifact_id, + session_id, + COALESCE(NULLIF(payload->>'status', ''), 'created'), + COALESCE(NULLIF(payload->>'schemaVersion', ''), ''), + plan_id, + COALESCE(NULLIF(payload->>'semanticMapId', ''), ''), + COALESCE(NULLIF(payload->>'simulationId', ''), ''), + COALESCE(NULLIF(payload->>'verificationId', ''), ''), + COALESCE(NULLIF(payload->>'v7GateStatus', ''), NULLIF(payload->'v7QualityGate'->>'status', ''), ''), + CASE + WHEN jsonb_typeof(payload->'executionGroups') = 'array' THEN jsonb_array_length(payload->'executionGroups') + ELSE 0 + END, + CASE + WHEN jsonb_typeof(payload->'checkpointReasons') = 'array' THEN jsonb_array_length(payload->'checkpointReasons') + WHEN jsonb_typeof(payload->'checkpoints') = 'array' THEN jsonb_array_length(payload->'checkpoints') + ELSE 0 + END, + COALESCE(NULLIF(payload->>'prompt', ''), ''), + payload, + created_at, + updated_at + FROM flagship_artifacts AS backfill_artifact + WHERE artifact_type = 'agent_run' + AND EXISTS ( + SELECT 1 FROM sessions AS backfill_session + JOIN session_lifecycle AS backfill_lifecycle + ON backfill_lifecycle.session_id = backfill_session.session_id + WHERE backfill_session.session_id = backfill_artifact.session_id + AND backfill_lifecycle.state = 'active' + ) + ON CONFLICT(run_id) DO NOTHING; + """, + ), + "workbook_truth_models": ( + """ + INSERT INTO workbook_truth_models + ( + truth_model_id, + session_id, + workbook_fingerprint, + schema_version, + semantic_map_id, + formula_graph_id, + accuracy_score, + model_taxonomy, + confidence_gap_count, + semantic_target_decision_count, + truth_json, + created_at, + updated_at + ) + SELECT + artifact_id, + session_id, + COALESCE(NULLIF(payload->>'workbookFingerprint', ''), ''), + COALESCE(NULLIF(payload->>'schemaVersion', ''), ''), + COALESCE(NULLIF(payload->>'semanticMapId', ''), NULLIF(payload->>'mapId', ''), ''), + COALESCE(NULLIF(payload->>'formulaGraphId', ''), ''), + CASE + WHEN jsonb_typeof(payload->'accuracyScore') = 'number' THEN (payload->>'accuracyScore')::double precision + ELSE 0 + END, + COALESCE(NULLIF(payload->>'modelTaxonomy', ''), ''), + CASE + WHEN jsonb_typeof(payload->'confidenceGaps') = 'array' THEN jsonb_array_length(payload->'confidenceGaps') + ELSE 0 + END, + CASE + WHEN jsonb_typeof(payload->'semanticTargetDecisions') = 'array' + THEN jsonb_array_length(payload->'semanticTargetDecisions') + WHEN jsonb_typeof(payload->'targetDecisions') = 'array' THEN jsonb_array_length(payload->'targetDecisions') + ELSE 0 + END, + payload, + created_at, + updated_at + FROM flagship_artifacts AS backfill_artifact + WHERE artifact_type = 'workbook_truth_model' + AND EXISTS ( + SELECT 1 FROM sessions AS backfill_session + JOIN session_lifecycle AS backfill_lifecycle + ON backfill_lifecycle.session_id = backfill_session.session_id + WHERE backfill_session.session_id = backfill_artifact.session_id + AND backfill_lifecycle.state = 'active' + ) + ON CONFLICT(truth_model_id) DO NOTHING; + """, + ), + "formula_graph_artifacts": ( + """ + INSERT INTO formula_graph_artifacts + ( + graph_id, + session_id, + workbook_fingerprint, + schema_version, + engine, + formula_count, + material_failure_count, + pre_apply_simulation_ready, + parser_confidence, + quality_json, + graph_json, + created_at, + updated_at + ) + SELECT + artifact_id, + session_id, + COALESCE(NULLIF(payload->>'workbookFingerprint', ''), ''), + COALESCE(NULLIF(payload->>'schemaVersion', ''), ''), + COALESCE(NULLIF(payload->>'engine', ''), ''), + CASE + WHEN jsonb_typeof(payload->'formulas') = 'array' THEN jsonb_array_length(payload->'formulas') + WHEN jsonb_typeof(payload->'nodes') = 'array' THEN jsonb_array_length(payload->'nodes') + ELSE 0 + END, + CASE + WHEN jsonb_typeof(payload->'materialFormulaFailures') = 'array' + THEN jsonb_array_length(payload->'materialFormulaFailures') + WHEN jsonb_typeof(payload->'materialFailures') = 'array' THEN jsonb_array_length(payload->'materialFailures') + ELSE 0 + END, + CASE + WHEN jsonb_typeof(payload->'preApplySimulationReady') = 'boolean' + THEN (payload->>'preApplySimulationReady')::boolean + WHEN jsonb_typeof(payload->'simulationReady') = 'boolean' THEN (payload->>'simulationReady')::boolean + ELSE FALSE + END, + CASE + WHEN jsonb_typeof(payload->'parserConfidence') = 'number' + THEN (payload->>'parserConfidence')::double precision + WHEN jsonb_typeof(payload->'confidence') = 'number' THEN (payload->>'confidence')::double precision + ELSE 0 + END, + CASE + WHEN jsonb_typeof(payload->'quality') = 'object' THEN payload->'quality' + WHEN jsonb_typeof(payload->'qualityReport') = 'object' THEN payload->'qualityReport' + ELSE '{}'::jsonb + END, + payload, + created_at, + updated_at + FROM flagship_artifacts AS backfill_artifact + WHERE artifact_type = 'formula_graph' + AND EXISTS ( + SELECT 1 FROM sessions AS backfill_session + JOIN session_lifecycle AS backfill_lifecycle + ON backfill_lifecycle.session_id = backfill_session.session_id + WHERE backfill_session.session_id = backfill_artifact.session_id + AND backfill_lifecycle.state = 'active' + ) + ON CONFLICT(graph_id) DO NOTHING; + """, + ), + "operation_simulations_v2": ( + """ + INSERT INTO operation_simulations_v2 + (simulation_id, session_id, truth_model_id, plan_id, run_id, status, simulation_json, created_at, updated_at) + SELECT + artifact_id, + session_id, + COALESCE(NULLIF(payload->>'truthModelId', ''), ''), + plan_id, + run_id, + COALESCE(NULLIF(payload->>'status', ''), NULLIF(payload->>'simulationGate', ''), 'warning'), + payload, + created_at, + updated_at + FROM flagship_artifacts AS backfill_artifact + WHERE artifact_type = 'operation_simulation_v2' + AND EXISTS ( + SELECT 1 FROM sessions AS backfill_session + JOIN session_lifecycle AS backfill_lifecycle + ON backfill_lifecycle.session_id = backfill_session.session_id + WHERE backfill_session.session_id = backfill_artifact.session_id + AND backfill_lifecycle.state = 'active' + ) + ON CONFLICT(simulation_id) DO NOTHING; + """, + ), + "v7_workbook_reviews": ( + """ + INSERT INTO v7_workbook_reviews + ( + review_id, + session_id, + plan_id, + run_id, + workbook_fingerprint, + accuracy_score, + confidence_gap_count, + checkpoint_reason_count, + default_readiness_status, + review_json, + created_at, + updated_at + ) + SELECT + artifact_id, + session_id, + plan_id, + run_id, + COALESCE(NULLIF(payload->>'workbookFingerprint', ''), ''), + CASE + WHEN jsonb_typeof(payload->'accuracyScore') = 'number' THEN (payload->>'accuracyScore')::double precision + ELSE 0 + END, + CASE + WHEN jsonb_typeof(payload->'confidenceGaps') = 'array' THEN jsonb_array_length(payload->'confidenceGaps') + ELSE 0 + END, + CASE + WHEN jsonb_typeof(payload->'checkpointReasons') = 'array' THEN jsonb_array_length(payload->'checkpointReasons') + WHEN jsonb_typeof(payload->'requiredInspections') = 'array' THEN jsonb_array_length(payload->'requiredInspections') + ELSE 0 + END, + COALESCE(NULLIF(payload->'defaultReadiness'->>'status', ''), NULLIF(payload->>'defaultReadinessStatus', ''), ''), + payload, + created_at, + updated_at + FROM flagship_artifacts AS backfill_artifact + WHERE artifact_type = 'v7_workbook_review' + AND EXISTS ( + SELECT 1 FROM sessions AS backfill_session + JOIN session_lifecycle AS backfill_lifecycle + ON backfill_lifecycle.session_id = backfill_session.session_id + WHERE backfill_session.session_id = backfill_artifact.session_id + AND backfill_lifecycle.state = 'active' + ) + ON CONFLICT(review_id) DO NOTHING; + """, + ), +} + +# A pre-atomic writer could commit a newer flagship payload and crash before +# updating its typed projection. Startup treats the flagship row as canonical: +# remove only projections that have a matching flagship row, then rebuild them +# through the normal, lifecycle-aware backfill statements below. +POSTGRES_TYPED_PROJECTION_RECONCILIATION = { + "v7_default_readiness_reports": ( + "readiness_id", + "artifact.artifact_type = 'v7_default_readiness_report'", + ), + "local_replay_artifacts": ("replay_id", "artifact.artifact_type LIKE 'local_replay:%'"), + "local_replay_baselines": ( + "baseline_id", + "artifact.artifact_type = 'local_replay_baseline'", + ), + "agent_run_artifacts": ( + "artifact_id", + "(artifact.artifact_type LIKE 'agent_run:%' " + "OR artifact.artifact_type IN (" + "'agent_run_step_artifact_v3', 'brain_run', 'brain_artifact_bundle'))", + ), + "verification_reports_v2": ( + "verification_id", + "artifact.artifact_type = 'verification_report'", + ), + "workbook_benchmark_fixture_results": ( + "fixture_result_id", + "artifact.artifact_type = 'workbook_benchmark_fixture'", + ), + "operation_audits": ("audit_id", "artifact.artifact_type = 'operation_audit'"), + "workbook_maps": ("map_id", "artifact.artifact_type = 'workbook_map'"), + "source_evidence_graphs": ( + "graph_id", + "artifact.artifact_type = 'source_evidence_graph'", + ), + "operation_simulations": ( + "simulation_id", + "artifact.artifact_type = 'operation_simulation'", + ), + "workbook_invariant_runs": ( + "invariant_run_id", + "artifact.artifact_type = 'workbook_invariant'", + ), + "agent_runs": ("run_id", "artifact.artifact_type = 'agent_run'"), + "workbook_truth_models": ( + "truth_model_id", + "artifact.artifact_type = 'workbook_truth_model'", + ), + "formula_graph_artifacts": ("graph_id", "artifact.artifact_type = 'formula_graph'"), + "operation_simulations_v2": ( + "simulation_id", + "artifact.artifact_type = 'operation_simulation_v2'", + ), + "v7_workbook_reviews": ( + "review_id", + "artifact.artifact_type = 'v7_workbook_review'", + ), +} + + +def _api_key_pepper() -> bytes: + pepper = os.getenv("API_KEY_HMAC_PEPPER", "") + if not pepper: + pepper = "finance-ai-default-pepper-set-API_KEY_HMAC_PEPPER-in-prod" + return pepper.encode("utf-8") + + +def _hash_enterprise_api_key(api_key: str) -> str: + digest = hmac.new(_api_key_pepper(), api_key.encode("utf-8"), hashlib.sha256).hexdigest() + return f"v2${digest}" + + +def _enterprise_api_key_candidates(api_key: str) -> list[str]: + legacy = hashlib.sha256(api_key.encode("utf-8")).hexdigest() + return [_hash_enterprise_api_key(api_key), legacy] + + +def _enterprise_user_allows_api_key(user: Optional[dict], *, require_mfa: bool = False) -> bool: + if not user: + return False + if not api_key_status_allows(user.get("status")): + return False + return not require_mfa or bool(user.get("mfa_enabled")) + class PostgresSessionStore: """PostgreSQL-backed session store with async connection pooling.""" @@ -34,6 +910,10 @@ def __init__(self, dsn: str): """ self.dsn = dsn self._pool = None + self._pool_loop: asyncio.AbstractEventLoop | None = None + self._sync_loop: asyncio.AbstractEventLoop | None = None + self._sync_loop_thread: threading.Thread | None = None + self._sync_loop_lock = threading.Lock() async def _get_pool(self): """Lazy-initialize the connection pool.""" @@ -47,12 +927,13 @@ async def _get_pool(self): command_timeout=30, ) await self._init_schema() + self._pool_loop = asyncio.get_running_loop() logger.info("postgres_pool_created", dsn=self.dsn.split("@")[-1]) - except ImportError: + except ImportError as exc: raise RuntimeError( "asyncpg is required for PostgreSQL support. " "Install it with: pip install asyncpg" - ) + ) from exc return self._pool async def _init_schema(self): @@ -62,11 +943,79 @@ async def _init_schema(self): await conn.execute(""" CREATE TABLE IF NOT EXISTS sessions ( session_id TEXT PRIMARY KEY, + owner_user_id TEXT, created_at DOUBLE PRECISION NOT NULL, updated_at DOUBLE PRECISION NOT NULL, metadata JSONB DEFAULT '{}' ); + ALTER TABLE sessions ADD COLUMN IF NOT EXISTS owner_user_id TEXT; + CREATE INDEX IF NOT EXISTS idx_sessions_owner_updated + ON sessions(owner_user_id, updated_at DESC); + + CREATE TABLE IF NOT EXISTS session_lifecycle ( + session_id TEXT PRIMARY KEY, + state TEXT NOT NULL DEFAULT 'active' + CHECK (state IN ('active', 'purging', 'verified', 'failed')), + receipt_id TEXT NOT NULL DEFAULT '', + started_at DOUBLE PRECISION NOT NULL DEFAULT 0, + completed_at DOUBLE PRECISION NOT NULL DEFAULT 0, + errors JSONB NOT NULL DEFAULT '[]', + updated_at DOUBLE PRECISION NOT NULL + ); + + CREATE INDEX IF NOT EXISTS idx_session_lifecycle_state_updated + ON session_lifecycle(state, updated_at DESC); + + CREATE TABLE IF NOT EXISTS migration_v05_imports ( + import_id TEXT PRIMARY KEY, + session_id TEXT NOT NULL UNIQUE REFERENCES sessions(session_id), + schema_version TEXT NOT NULL, + state TEXT NOT NULL CHECK ( + state IN ( + 'validated_pending_import', + 'importing', + 'imported', + 'failed_retryable' + ) + ), + version BIGINT NOT NULL DEFAULT 0 CHECK (version >= 0), + attempt_count BIGINT NOT NULL DEFAULT 0 CHECK (attempt_count >= 0), + binding_sha256 TEXT NOT NULL UNIQUE, + archive_sha256 TEXT NOT NULL, + content_sha256 TEXT NOT NULL UNIQUE, + owner_scope_sha256 TEXT NOT NULL, + manifest_sha256 TEXT NOT NULL, + stage_receipt_sha256 TEXT NOT NULL, + counts_json JSONB NOT NULL, + payload_count BIGINT NOT NULL DEFAULT 0 CHECK (payload_count >= 0), + receipt_json JSONB NOT NULL DEFAULT '{}'::jsonb, + failure_code TEXT NOT NULL DEFAULT '', + created_at DOUBLE PRECISION NOT NULL, + updated_at DOUBLE PRECISION NOT NULL + ); + + CREATE TABLE IF NOT EXISTS migration_v05_payloads ( + import_id TEXT NOT NULL REFERENCES migration_v05_imports(import_id) + ON DELETE CASCADE, + session_id TEXT NOT NULL REFERENCES sessions(session_id), + relative_path TEXT NOT NULL, + payload_kind TEXT NOT NULL CHECK ( + payload_kind IN ('blob', 'document', 'session', 'workflow_recipe') + ), + payload_sha256 TEXT NOT NULL, + size_bytes BIGINT NOT NULL CHECK (size_bytes >= 0), + payload_bytes BYTEA NOT NULL, + PRIMARY KEY (import_id, relative_path) + ); + + CREATE INDEX IF NOT EXISTS idx_migration_v05_imports_state + ON migration_v05_imports(state, updated_at DESC); + CREATE INDEX IF NOT EXISTS idx_migration_v05_imports_session + ON migration_v05_imports(session_id); + CREATE INDEX IF NOT EXISTS idx_migration_v05_payloads_session + ON migration_v05_payloads(session_id, import_id); + CREATE TABLE IF NOT EXISTS conversations ( id SERIAL PRIMARY KEY, session_id TEXT NOT NULL REFERENCES sessions(session_id), @@ -95,48 +1044,913 @@ async def _init_schema(self): CREATE INDEX IF NOT EXISTS idx_task_session ON task_history(session_id); CREATE INDEX IF NOT EXISTS idx_task_timestamp ON task_history(timestamp DESC); + CREATE TABLE IF NOT EXISTS postgres_schema_migrations ( + migration_id TEXT PRIMARY KEY, + applied_at DOUBLE PRECISION NOT NULL, + details JSONB DEFAULT '{}' + ); + CREATE TABLE IF NOT EXISTS flagship_artifacts ( - artifact_id TEXT PRIMARY KEY, + artifact_id TEXT NOT NULL, artifact_type TEXT NOT NULL, session_id TEXT DEFAULT '', run_id TEXT DEFAULT '', plan_id TEXT DEFAULT '', payload JSONB DEFAULT '{}', created_at DOUBLE PRECISION NOT NULL, - updated_at DOUBLE PRECISION NOT NULL + updated_at DOUBLE PRECISION NOT NULL, + PRIMARY KEY (artifact_id, artifact_type) + ); + + CREATE TABLE IF NOT EXISTS enterprise_admin_audits ( + audit_id TEXT PRIMARY KEY, + event_type TEXT NOT NULL, + payload JSONB NOT NULL DEFAULT '{}', + created_at DOUBLE PRECISION NOT NULL ); + CREATE INDEX IF NOT EXISTS idx_enterprise_admin_audits_created + ON enterprise_admin_audits(created_at DESC); + + INSERT INTO enterprise_admin_audits + (audit_id, event_type, payload, created_at) + SELECT + artifact_id, + COALESCE(NULLIF(payload->>'event_type', ''), + SUBSTRING(artifact_type FROM 7)), + COALESCE(payload->'payload', '{}'::jsonb), + created_at + FROM flagship_artifacts + WHERE artifact_type LIKE 'audit:enterprise_admin.%' + ON CONFLICT(audit_id) DO NOTHING; + + DELETE FROM flagship_artifacts + WHERE artifact_type LIKE 'audit:enterprise_admin.%'; + + DO $$ + DECLARE + old_pk_name TEXT; + composite_pk_name TEXT; + BEGIN + SELECT c.conname + INTO old_pk_name + FROM pg_constraint c + JOIN pg_class t ON t.oid = c.conrelid + WHERE t.relname = 'flagship_artifacts' + AND c.contype = 'p' + AND array_length(c.conkey, 1) = 1 + AND c.conkey[1] = ( + SELECT a.attnum + FROM pg_attribute a + WHERE a.attrelid = t.oid + AND a.attname = 'artifact_id' + ) + LIMIT 1; + + IF old_pk_name IS NOT NULL THEN + EXECUTE format('ALTER TABLE %I DROP CONSTRAINT %I', 'flagship_artifacts', old_pk_name); + END IF; + + SELECT c.conname + INTO composite_pk_name + FROM pg_constraint c + JOIN pg_class t ON t.oid = c.conrelid + WHERE t.relname = 'flagship_artifacts' + AND c.contype = 'p' + AND array_length(c.conkey, 1) = 2 + AND c.conkey @> ARRAY[ + ( + SELECT a.attnum + FROM pg_attribute a + WHERE a.attrelid = t.oid + AND a.attname = 'artifact_id' + ), + ( + SELECT a.attnum + FROM pg_attribute a + WHERE a.attrelid = t.oid + AND a.attname = 'artifact_type' + ) + ]::smallint[] + LIMIT 1; + + IF composite_pk_name IS NULL THEN + ALTER TABLE flagship_artifacts + ADD CONSTRAINT flagship_artifacts_pkey PRIMARY KEY (artifact_id, artifact_type); + END IF; + END $$; + CREATE INDEX IF NOT EXISTS idx_flagship_artifacts_type ON flagship_artifacts(artifact_type, updated_at DESC); CREATE INDEX IF NOT EXISTS idx_flagship_artifacts_run ON flagship_artifacts(run_id, updated_at DESC); CREATE INDEX IF NOT EXISTS idx_flagship_artifacts_session ON flagship_artifacts(session_id, updated_at DESC); + + -- Normalize legacy enterprise-user identity metadata before + -- enabling the immutable issuer+subject constraint. + UPDATE flagship_artifacts + SET payload = payload || jsonb_build_object( + 'oidc_issuer', BTRIM(COALESCE(payload->>'oidc_issuer', payload->'metadata'->>'oidc_issuer', '')), + 'oidc_subject', BTRIM(COALESCE(payload->>'oidc_subject', payload->'metadata'->>'oidc_subject', '')) + ) + WHERE artifact_type = 'enterprise_user'; + + UPDATE flagship_artifacts + SET payload = jsonb_set( + payload - 'oidc_issuer' - 'oidc_subject', + '{metadata}', + ( + COALESCE(payload->'metadata', '{}'::jsonb) - 'oidc_issuer' - 'oidc_subject' + ) || jsonb_build_object( + 'legacy_oidc_issuer', COALESCE(payload->>'oidc_issuer', ''), + 'legacy_oidc_subject', COALESCE(payload->>'oidc_subject', ''), + 'oidc_binding_state', 'partial' + ), + TRUE + ) || jsonb_build_object('status', 'binding_required') + WHERE artifact_type = 'enterprise_user' + AND ((BTRIM(COALESCE(payload->>'oidc_issuer', '')) = '') + <> (BTRIM(COALESCE(payload->>'oidc_subject', '')) = '')); + + WITH duplicate_bindings AS ( + SELECT + payload->>'oidc_issuer' AS issuer, + payload->>'oidc_subject' AS subject + FROM flagship_artifacts + WHERE artifact_type = 'enterprise_user' + AND BTRIM(COALESCE(payload->>'oidc_issuer', '')) <> '' + AND BTRIM(COALESCE(payload->>'oidc_subject', '')) <> '' + GROUP BY payload->>'oidc_issuer', payload->>'oidc_subject' + HAVING COUNT(*) > 1 + ) + UPDATE flagship_artifacts AS artifact + SET payload = jsonb_set( + artifact.payload - 'oidc_issuer' - 'oidc_subject', + '{metadata}', + ( + COALESCE(artifact.payload->'metadata', '{}'::jsonb) - 'oidc_issuer' - 'oidc_subject' + ) || jsonb_build_object( + 'legacy_oidc_issuer', COALESCE(artifact.payload->>'oidc_issuer', ''), + 'legacy_oidc_subject', COALESCE(artifact.payload->>'oidc_subject', ''), + 'oidc_binding_state', 'duplicate' + ), + TRUE + ) || jsonb_build_object('status', 'binding_required') + FROM duplicate_bindings AS duplicate + WHERE artifact.artifact_type = 'enterprise_user' + AND artifact.payload->>'oidc_issuer' = duplicate.issuer + AND artifact.payload->>'oidc_subject' = duplicate.subject; + + UPDATE flagship_artifacts + SET payload = jsonb_set( + payload || jsonb_build_object('status', 'api_key_only'), + '{metadata}', + COALESCE(payload->'metadata', '{}'::jsonb) + || jsonb_build_object('oidc_binding_state', 'legacy_unbound'), + TRUE + ) + WHERE artifact_type = 'enterprise_user' + AND LOWER(COALESCE(payload->>'status', 'active')) = 'active' + AND BTRIM(COALESCE(payload->>'oidc_issuer', '')) = '' + AND BTRIM(COALESCE(payload->>'oidc_subject', '')) = ''; + + UPDATE flagship_artifacts + SET payload = jsonb_set( + payload, + '{metadata}', + ( + COALESCE(payload->'metadata', '{}'::jsonb) - 'oidc_binding_state' + ) || jsonb_build_object( + 'oidc_issuer', payload->>'oidc_issuer', + 'oidc_subject', payload->>'oidc_subject' + ), + TRUE + ) + WHERE artifact_type = 'enterprise_user' + AND BTRIM(COALESCE(payload->>'oidc_issuer', '')) <> '' + AND BTRIM(COALESCE(payload->>'oidc_subject', '')) <> ''; + + CREATE UNIQUE INDEX IF NOT EXISTS idx_flagship_enterprise_oidc_identity + ON flagship_artifacts ( + (payload->>'oidc_issuer'), + (payload->>'oidc_subject') + ) + WHERE artifact_type = 'enterprise_user' + AND BTRIM(COALESCE(payload->>'oidc_issuer', '')) <> '' + AND BTRIM(COALESCE(payload->>'oidc_subject', '')) <> ''; + + CREATE OR REPLACE FUNCTION enforce_enterprise_oidc_identity() + RETURNS TRIGGER AS $$ + DECLARE + new_issuer TEXT := BTRIM(COALESCE(NEW.payload->>'oidc_issuer', '')); + new_subject TEXT := BTRIM(COALESCE(NEW.payload->>'oidc_subject', '')); + old_issuer TEXT := ''; + old_subject TEXT := ''; + BEGIN + IF NEW.artifact_type <> 'enterprise_user' THEN + RETURN NEW; + END IF; + IF (new_issuer = '') <> (new_subject = '') THEN + RAISE EXCEPTION 'OIDC identity binding requires both issuer and subject'; + END IF; + IF TG_OP = 'UPDATE' THEN + old_issuer := BTRIM(COALESCE(OLD.payload->>'oidc_issuer', '')); + old_subject := BTRIM(COALESCE(OLD.payload->>'oidc_subject', '')); + IF old_issuer <> '' + AND (new_issuer <> old_issuer OR new_subject <> old_subject) THEN + RAISE EXCEPTION 'OIDC issuer and subject are immutable once bound'; + END IF; + END IF; + IF LOWER(COALESCE(NEW.payload->>'status', 'active')) = 'active' + AND (new_issuer = '' OR new_subject = '') THEN + RAISE EXCEPTION 'Active enterprise user requires an immutable OIDC issuer and subject'; + END IF; + RETURN NEW; + END; + $$ LANGUAGE plpgsql; + + DROP TRIGGER IF EXISTS trg_flagship_enterprise_oidc_identity ON flagship_artifacts; + CREATE TRIGGER trg_flagship_enterprise_oidc_identity + BEFORE INSERT OR UPDATE ON flagship_artifacts + FOR EACH ROW EXECUTE FUNCTION enforce_enterprise_oidc_identity(); + + CREATE TABLE IF NOT EXISTS v7_default_readiness_reports ( + readiness_id TEXT PRIMARY KEY, + status TEXT NOT NULL DEFAULT 'warning', + default_allowed BOOLEAN NOT NULL DEFAULT FALSE, + score DOUBLE PRECISION NOT NULL DEFAULT 0, + report_json JSONB DEFAULT '{}', + created_at DOUBLE PRECISION NOT NULL, + updated_at DOUBLE PRECISION NOT NULL + ); + + CREATE INDEX IF NOT EXISTS idx_v7_default_readiness_status + ON v7_default_readiness_reports(status, updated_at DESC); + + CREATE TABLE IF NOT EXISTS local_replay_artifacts ( + replay_id TEXT PRIMARY KEY, + session_id TEXT NOT NULL DEFAULT '', + artifact_kind TEXT NOT NULL DEFAULT '', + schema_version TEXT NOT NULL DEFAULT '', + v7_gate_status TEXT NOT NULL DEFAULT '', + accuracy_score DOUBLE PRECISION NOT NULL DEFAULT 0, + confidence_gap_count INTEGER NOT NULL DEFAULT 0, + checkpoint_reason_count INTEGER NOT NULL DEFAULT 0, + formula_failure_count INTEGER NOT NULL DEFAULT 0, + truth_schema TEXT NOT NULL DEFAULT '', + simulation_schema TEXT NOT NULL DEFAULT '', + replay_json JSONB DEFAULT '{}', + created_at DOUBLE PRECISION NOT NULL, + updated_at DOUBLE PRECISION NOT NULL + ); + + CREATE INDEX IF NOT EXISTS idx_local_replay_artifacts_kind + ON local_replay_artifacts(artifact_kind, updated_at DESC); + CREATE INDEX IF NOT EXISTS idx_local_replay_artifacts_session + ON local_replay_artifacts(session_id, updated_at DESC); + CREATE INDEX IF NOT EXISTS idx_local_replay_artifacts_gate + ON local_replay_artifacts(v7_gate_status, updated_at DESC); + + CREATE TABLE IF NOT EXISTS local_replay_baselines ( + baseline_id TEXT PRIMARY KEY, + session_id TEXT NOT NULL DEFAULT '', + status TEXT NOT NULL DEFAULT 'warning', + production_evidence BOOLEAN NOT NULL DEFAULT FALSE, + evidence_class TEXT NOT NULL DEFAULT '', + replay_count INTEGER NOT NULL DEFAULT 0, + deterministic_drift_class TEXT NOT NULL DEFAULT '', + baseline_json JSONB DEFAULT '{}', + created_at DOUBLE PRECISION NOT NULL, + updated_at DOUBLE PRECISION NOT NULL + ); + + CREATE INDEX IF NOT EXISTS idx_local_replay_baselines_status + ON local_replay_baselines(status, updated_at DESC); + CREATE INDEX IF NOT EXISTS idx_local_replay_baselines_production + ON local_replay_baselines(production_evidence, updated_at DESC); + + CREATE TABLE IF NOT EXISTS agent_run_artifacts ( + artifact_id TEXT PRIMARY KEY, + run_id TEXT NOT NULL DEFAULT '', + session_id TEXT NOT NULL DEFAULT '', + artifact_type TEXT NOT NULL DEFAULT '', + plan_id TEXT NOT NULL DEFAULT '', + artifact_json JSONB DEFAULT '{}', + created_at DOUBLE PRECISION NOT NULL, + updated_at DOUBLE PRECISION NOT NULL + ); + + CREATE INDEX IF NOT EXISTS idx_agent_run_artifacts_run + ON agent_run_artifacts(run_id, updated_at DESC); + CREATE INDEX IF NOT EXISTS idx_agent_run_artifacts_session + ON agent_run_artifacts(session_id, updated_at DESC); + CREATE INDEX IF NOT EXISTS idx_agent_run_artifacts_type + ON agent_run_artifacts(artifact_type, updated_at DESC); + + CREATE TABLE IF NOT EXISTS verification_reports_v2 ( + verification_id TEXT PRIMARY KEY, + session_id TEXT NOT NULL DEFAULT '', + run_id TEXT NOT NULL DEFAULT '', + plan_id TEXT NOT NULL DEFAULT '', + status TEXT NOT NULL DEFAULT 'warning', + report_json JSONB DEFAULT '{}', + created_at DOUBLE PRECISION NOT NULL, + updated_at DOUBLE PRECISION NOT NULL + ); + + CREATE INDEX IF NOT EXISTS idx_verification_reports_v2_session + ON verification_reports_v2(session_id, updated_at DESC); + CREATE INDEX IF NOT EXISTS idx_verification_reports_v2_run + ON verification_reports_v2(run_id, updated_at DESC); + + CREATE TABLE IF NOT EXISTS workbook_benchmark_fixture_results ( + fixture_result_id TEXT PRIMARY KEY, + run_id TEXT NOT NULL DEFAULT '', + fixture_id TEXT NOT NULL DEFAULT '', + status TEXT NOT NULL DEFAULT 'warning', + result_json JSONB DEFAULT '{}', + created_at DOUBLE PRECISION NOT NULL + ); + + CREATE INDEX IF NOT EXISTS idx_workbook_fixture_results_run + ON workbook_benchmark_fixture_results(run_id, created_at DESC); + CREATE INDEX IF NOT EXISTS idx_workbook_fixture_results_fixture + ON workbook_benchmark_fixture_results(fixture_id, created_at DESC); + + CREATE TABLE IF NOT EXISTS operation_audits ( + audit_id TEXT PRIMARY KEY, + session_id TEXT NOT NULL DEFAULT '', + plan_id TEXT NOT NULL DEFAULT '', + operation_id TEXT NOT NULL DEFAULT '', + operation_type TEXT NOT NULL DEFAULT '', + dirty_range_count INTEGER NOT NULL DEFAULT 0, + warning_count INTEGER NOT NULL DEFAULT 0, + payload_json JSONB DEFAULT '{}', + created_at DOUBLE PRECISION NOT NULL + ); + + CREATE INDEX IF NOT EXISTS idx_operation_audits_session + ON operation_audits(session_id, created_at DESC); + CREATE INDEX IF NOT EXISTS idx_operation_audits_plan + ON operation_audits(plan_id, created_at DESC); + CREATE INDEX IF NOT EXISTS idx_operation_audits_type + ON operation_audits(operation_type, created_at DESC); + + CREATE TABLE IF NOT EXISTS workbook_maps ( + map_id TEXT PRIMARY KEY, + session_id TEXT NOT NULL DEFAULT '', + workbook_fingerprint TEXT NOT NULL DEFAULT '', + workbook_name TEXT NOT NULL DEFAULT '', + schema_version TEXT NOT NULL DEFAULT '', + entity_count INTEGER NOT NULL DEFAULT 0, + formula_count INTEGER NOT NULL DEFAULT 0, + risk_count INTEGER NOT NULL DEFAULT 0, + map_json JSONB DEFAULT '{}', + created_at DOUBLE PRECISION NOT NULL, + updated_at DOUBLE PRECISION NOT NULL + ); + + CREATE INDEX IF NOT EXISTS idx_workbook_maps_session + ON workbook_maps(session_id, updated_at DESC); + CREATE INDEX IF NOT EXISTS idx_workbook_maps_fingerprint + ON workbook_maps(workbook_fingerprint, updated_at DESC); + + CREATE TABLE IF NOT EXISTS source_evidence_graphs ( + graph_id TEXT PRIMARY KEY, + session_id TEXT NOT NULL DEFAULT '', + document_ids JSONB DEFAULT '[]', + graph_json JSONB DEFAULT '{}', + created_at DOUBLE PRECISION NOT NULL, + updated_at DOUBLE PRECISION NOT NULL + ); + + CREATE INDEX IF NOT EXISTS idx_source_evidence_graphs_session + ON source_evidence_graphs(session_id, updated_at DESC); + + CREATE TABLE IF NOT EXISTS operation_simulations ( + simulation_id TEXT PRIMARY KEY, + session_id TEXT NOT NULL DEFAULT '', + plan_id TEXT NOT NULL DEFAULT '', + run_id TEXT NOT NULL DEFAULT '', + status TEXT NOT NULL DEFAULT 'warning', + simulation_json JSONB DEFAULT '{}', + created_at DOUBLE PRECISION NOT NULL + ); + + CREATE INDEX IF NOT EXISTS idx_operation_simulations_session + ON operation_simulations(session_id, created_at DESC); + CREATE INDEX IF NOT EXISTS idx_operation_simulations_run + ON operation_simulations(run_id, created_at DESC); + + CREATE TABLE IF NOT EXISTS workbook_invariant_runs ( + invariant_run_id TEXT PRIMARY KEY, + session_id TEXT NOT NULL DEFAULT '', + run_id TEXT NOT NULL DEFAULT '', + plan_id TEXT NOT NULL DEFAULT '', + status TEXT NOT NULL DEFAULT 'warning', + report_json JSONB DEFAULT '{}', + created_at DOUBLE PRECISION NOT NULL + ); + + CREATE INDEX IF NOT EXISTS idx_workbook_invariant_runs_session + ON workbook_invariant_runs(session_id, created_at DESC); + CREATE INDEX IF NOT EXISTS idx_workbook_invariant_runs_run + ON workbook_invariant_runs(run_id, created_at DESC); + + CREATE TABLE IF NOT EXISTS agent_runs ( + run_id TEXT PRIMARY KEY, + session_id TEXT NOT NULL DEFAULT '', + status TEXT NOT NULL DEFAULT 'created', + schema_version TEXT NOT NULL DEFAULT '', + plan_id TEXT NOT NULL DEFAULT '', + semantic_map_id TEXT NOT NULL DEFAULT '', + simulation_id TEXT NOT NULL DEFAULT '', + verification_id TEXT NOT NULL DEFAULT '', + v7_gate_status TEXT NOT NULL DEFAULT '', + execution_group_count INTEGER NOT NULL DEFAULT 0, + checkpoint_count INTEGER NOT NULL DEFAULT 0, + prompt TEXT NOT NULL DEFAULT '', + run_json JSONB DEFAULT '{}', + created_at DOUBLE PRECISION NOT NULL, + updated_at DOUBLE PRECISION NOT NULL + ); + + CREATE INDEX IF NOT EXISTS idx_agent_runs_session + ON agent_runs(session_id, updated_at DESC); + CREATE INDEX IF NOT EXISTS idx_agent_runs_status + ON agent_runs(status, updated_at DESC); + + CREATE TABLE IF NOT EXISTS workbook_truth_models ( + truth_model_id TEXT PRIMARY KEY, + session_id TEXT NOT NULL DEFAULT '', + workbook_fingerprint TEXT NOT NULL DEFAULT '', + schema_version TEXT NOT NULL DEFAULT '', + semantic_map_id TEXT NOT NULL DEFAULT '', + formula_graph_id TEXT NOT NULL DEFAULT '', + accuracy_score DOUBLE PRECISION NOT NULL DEFAULT 0, + model_taxonomy TEXT NOT NULL DEFAULT '', + confidence_gap_count INTEGER NOT NULL DEFAULT 0, + semantic_target_decision_count INTEGER NOT NULL DEFAULT 0, + truth_json JSONB DEFAULT '{}', + created_at DOUBLE PRECISION NOT NULL, + updated_at DOUBLE PRECISION NOT NULL + ); + + CREATE INDEX IF NOT EXISTS idx_workbook_truth_models_session + ON workbook_truth_models(session_id, updated_at DESC); + CREATE INDEX IF NOT EXISTS idx_workbook_truth_models_fingerprint + ON workbook_truth_models(workbook_fingerprint, updated_at DESC); + + CREATE TABLE IF NOT EXISTS formula_graph_artifacts ( + graph_id TEXT PRIMARY KEY, + session_id TEXT NOT NULL DEFAULT '', + workbook_fingerprint TEXT NOT NULL DEFAULT '', + schema_version TEXT NOT NULL DEFAULT '', + engine TEXT NOT NULL DEFAULT '', + formula_count INTEGER NOT NULL DEFAULT 0, + material_failure_count INTEGER NOT NULL DEFAULT 0, + pre_apply_simulation_ready BOOLEAN NOT NULL DEFAULT FALSE, + parser_confidence DOUBLE PRECISION NOT NULL DEFAULT 0, + quality_json JSONB DEFAULT '{}', + graph_json JSONB DEFAULT '{}', + created_at DOUBLE PRECISION NOT NULL, + updated_at DOUBLE PRECISION NOT NULL + ); + + CREATE INDEX IF NOT EXISTS idx_formula_graph_artifacts_session + ON formula_graph_artifacts(session_id, updated_at DESC); + CREATE INDEX IF NOT EXISTS idx_formula_graph_artifacts_fingerprint + ON formula_graph_artifacts(workbook_fingerprint, updated_at DESC); + + CREATE TABLE IF NOT EXISTS operation_simulations_v2 ( + simulation_id TEXT PRIMARY KEY, + session_id TEXT NOT NULL DEFAULT '', + truth_model_id TEXT NOT NULL DEFAULT '', + plan_id TEXT NOT NULL DEFAULT '', + run_id TEXT NOT NULL DEFAULT '', + status TEXT NOT NULL DEFAULT 'warning', + simulation_json JSONB DEFAULT '{}', + created_at DOUBLE PRECISION NOT NULL, + updated_at DOUBLE PRECISION NOT NULL + ); + + CREATE INDEX IF NOT EXISTS idx_operation_simulations_v2_session + ON operation_simulations_v2(session_id, updated_at DESC); + CREATE INDEX IF NOT EXISTS idx_operation_simulations_v2_run + ON operation_simulations_v2(run_id, updated_at DESC); + + CREATE TABLE IF NOT EXISTS v7_workbook_reviews ( + review_id TEXT PRIMARY KEY, + session_id TEXT NOT NULL DEFAULT '', + plan_id TEXT NOT NULL DEFAULT '', + run_id TEXT NOT NULL DEFAULT '', + workbook_fingerprint TEXT NOT NULL DEFAULT '', + accuracy_score DOUBLE PRECISION NOT NULL DEFAULT 0, + confidence_gap_count INTEGER NOT NULL DEFAULT 0, + checkpoint_reason_count INTEGER NOT NULL DEFAULT 0, + default_readiness_status TEXT NOT NULL DEFAULT '', + review_json JSONB DEFAULT '{}', + created_at DOUBLE PRECISION NOT NULL, + updated_at DOUBLE PRECISION NOT NULL + ); + + CREATE INDEX IF NOT EXISTS idx_v7_workbook_reviews_session + ON v7_workbook_reviews(session_id, updated_at DESC); + CREATE INDEX IF NOT EXISTS idx_v7_workbook_reviews_run + ON v7_workbook_reviews(run_id, updated_at DESC); + CREATE INDEX IF NOT EXISTS idx_v7_workbook_reviews_fingerprint + ON v7_workbook_reviews(workbook_fingerprint, updated_at DESC); + + -- Older builds could commit a flagship or typed projection + -- without first materializing its local-mode session. Preserve + -- those rows by attaching them to an ownerless legacy session; + -- future low-level writes must use an explicit session boundary. + -- Seed lifecycle rows before sessions so an already-installed + -- sessions trigger can validate legacy repair inserts during + -- every subsequent process restart. + WITH legacy_session_ids AS ( + SELECT session_id + FROM flagship_artifacts + WHERE BTRIM(COALESCE(session_id, '')) <> '' + AND artifact_type NOT IN ( + 'benchmark_run', + 'enterprise_api_key', + 'enterprise_config', + 'enterprise_team', + 'enterprise_user', + 'v7_default_readiness_report', + 'workbook_benchmark_fixture' + ) + UNION SELECT session_id FROM local_replay_artifacts WHERE BTRIM(session_id) <> '' + UNION SELECT session_id FROM local_replay_baselines WHERE BTRIM(session_id) <> '' + UNION SELECT session_id FROM agent_run_artifacts WHERE BTRIM(session_id) <> '' + UNION SELECT session_id FROM verification_reports_v2 WHERE BTRIM(session_id) <> '' + UNION SELECT session_id FROM operation_audits WHERE BTRIM(session_id) <> '' + UNION SELECT session_id FROM workbook_maps WHERE BTRIM(session_id) <> '' + UNION SELECT session_id FROM source_evidence_graphs WHERE BTRIM(session_id) <> '' + UNION SELECT session_id FROM operation_simulations WHERE BTRIM(session_id) <> '' + UNION SELECT session_id FROM workbook_invariant_runs WHERE BTRIM(session_id) <> '' + UNION SELECT session_id FROM agent_runs WHERE BTRIM(session_id) <> '' + UNION SELECT session_id FROM workbook_truth_models WHERE BTRIM(session_id) <> '' + UNION SELECT session_id FROM formula_graph_artifacts WHERE BTRIM(session_id) <> '' + UNION SELECT session_id FROM operation_simulations_v2 WHERE BTRIM(session_id) <> '' + UNION SELECT session_id FROM v7_workbook_reviews WHERE BTRIM(session_id) <> '' + ) + INSERT INTO session_lifecycle + (session_id, state, receipt_id, started_at, completed_at, errors, updated_at) + SELECT + session_id, + 'active', + '', + 0, + 0, + '[]'::jsonb, + EXTRACT(EPOCH FROM NOW()) + FROM legacy_session_ids + ON CONFLICT(session_id) DO NOTHING; + + WITH legacy_session_ids AS ( + SELECT session_id + FROM flagship_artifacts + WHERE BTRIM(COALESCE(session_id, '')) <> '' + AND artifact_type NOT IN ( + 'benchmark_run', + 'enterprise_api_key', + 'enterprise_config', + 'enterprise_team', + 'enterprise_user', + 'v7_default_readiness_report', + 'workbook_benchmark_fixture' + ) + UNION SELECT session_id FROM local_replay_artifacts WHERE BTRIM(session_id) <> '' + UNION SELECT session_id FROM local_replay_baselines WHERE BTRIM(session_id) <> '' + UNION SELECT session_id FROM agent_run_artifacts WHERE BTRIM(session_id) <> '' + UNION SELECT session_id FROM verification_reports_v2 WHERE BTRIM(session_id) <> '' + UNION SELECT session_id FROM operation_audits WHERE BTRIM(session_id) <> '' + UNION SELECT session_id FROM workbook_maps WHERE BTRIM(session_id) <> '' + UNION SELECT session_id FROM source_evidence_graphs WHERE BTRIM(session_id) <> '' + UNION SELECT session_id FROM operation_simulations WHERE BTRIM(session_id) <> '' + UNION SELECT session_id FROM workbook_invariant_runs WHERE BTRIM(session_id) <> '' + UNION SELECT session_id FROM agent_runs WHERE BTRIM(session_id) <> '' + UNION SELECT session_id FROM workbook_truth_models WHERE BTRIM(session_id) <> '' + UNION SELECT session_id FROM formula_graph_artifacts WHERE BTRIM(session_id) <> '' + UNION SELECT session_id FROM operation_simulations_v2 WHERE BTRIM(session_id) <> '' + UNION SELECT session_id FROM v7_workbook_reviews WHERE BTRIM(session_id) <> '' + ) + INSERT INTO sessions (session_id, owner_user_id, created_at, updated_at, metadata) + SELECT + legacy.session_id, + NULL, + EXTRACT(EPOCH FROM NOW()), + EXTRACT(EPOCH FROM NOW()), + '{"migration":"postgres_session_lifecycle_fencing_v1"}'::jsonb + FROM legacy_session_ids AS legacy + JOIN session_lifecycle AS lifecycle + ON lifecycle.session_id = legacy.session_id + AND lifecycle.state = 'active' + ON CONFLICT(session_id) DO NOTHING; + + INSERT INTO session_lifecycle + (session_id, state, receipt_id, started_at, completed_at, errors, updated_at) + SELECT + session_id, + 'active', + '', + 0, + 0, + '[]'::jsonb, + EXTRACT(EPOCH FROM NOW()) + FROM sessions + ON CONFLICT(session_id) DO NOTHING; + + -- Global administrative/configuration artifacts are not owned + -- by a session. Normalize legacy rows before ownership becomes + -- immutable so future updates cannot smuggle session scope in. + UPDATE flagship_artifacts + SET session_id = '' + WHERE artifact_type IN ( + 'benchmark_run', + 'enterprise_api_key', + 'enterprise_config', + 'enterprise_team', + 'enterprise_user', + 'v7_default_readiness_report', + 'workbook_benchmark_fixture' + ) + AND BTRIM(COALESCE(session_id, '')) <> ''; + + CREATE OR REPLACE FUNCTION enforce_session_lifecycle_write() + RETURNS TRIGGER AS $$ + DECLARE + lifecycle_state TEXT; + BEGIN + IF TG_OP = 'UPDATE' + AND BTRIM(COALESCE(OLD.session_id, '')) + IS DISTINCT FROM BTRIM(COALESCE(NEW.session_id, '')) THEN + RAISE EXCEPTION 'session ownership is immutable for durable artifacts' + USING ERRCODE = '23514'; + END IF; + IF BTRIM(COALESCE(NEW.session_id, '')) = '' THEN + RETURN NEW; + END IF; + IF TG_TABLE_NAME = 'flagship_artifacts' + AND ( + (to_jsonb(NEW)->>'artifact_type') IN ( + 'benchmark_run', + 'enterprise_api_key', + 'enterprise_config', + 'enterprise_team', + 'enterprise_user', + 'v7_default_readiness_report', + 'workbook_benchmark_fixture' + ) + ) THEN + RETURN NEW; + END IF; + + PERFORM pg_advisory_xact_lock(hashtextextended(NEW.session_id, 0)); + SELECT state + INTO lifecycle_state + FROM session_lifecycle + WHERE session_id = NEW.session_id; + IF lifecycle_state IS DISTINCT FROM 'active' + OR ( + TG_TABLE_NAME <> 'sessions' + AND NOT EXISTS ( + SELECT 1 FROM sessions WHERE session_id = NEW.session_id + ) + ) THEN + RAISE EXCEPTION 'session lifecycle rejected durable write for %', NEW.session_id + USING ERRCODE = '23514'; + END IF; + RETURN NEW; + END; + $$ LANGUAGE plpgsql; + + DO $$ + DECLARE + lifecycle_table TEXT; + BEGIN + FOREACH lifecycle_table IN ARRAY ARRAY[ + 'sessions', + 'migration_v05_imports', + 'migration_v05_payloads', + 'conversations', + 'task_history', + 'flagship_artifacts', + 'local_replay_artifacts', + 'local_replay_baselines', + 'agent_run_artifacts', + 'verification_reports_v2', + 'operation_audits', + 'workbook_maps', + 'source_evidence_graphs', + 'operation_simulations', + 'workbook_invariant_runs', + 'agent_runs', + 'workbook_truth_models', + 'formula_graph_artifacts', + 'operation_simulations_v2', + 'v7_workbook_reviews' + ] + LOOP + EXECUTE format( + 'DROP TRIGGER IF EXISTS trg_session_lifecycle_write ON %I', + lifecycle_table + ); + EXECUTE format( + 'CREATE TRIGGER trg_session_lifecycle_write ' + 'BEFORE INSERT OR UPDATE ON %I ' + 'FOR EACH ROW EXECUTE FUNCTION enforce_session_lifecycle_write()', + lifecycle_table + ); + END LOOP; + END $$; + """) + await self._async_backfill_typed_projections(conn) + await conn.execute(""" + INSERT INTO postgres_schema_migrations (migration_id, applied_at, details) + VALUES + ('postgres_artifact_parity_v1', EXTRACT(EPOCH FROM NOW()), '{"schema":"flagship_artifacts"}'::jsonb), + ('postgres_artifact_identity_v2', EXTRACT(EPOCH FROM NOW()), '{"identity":"artifact_id+artifact_type"}'::jsonb), + ('postgres_session_lifecycle_fencing_v1', EXTRACT(EPOCH FROM NOW()), '{"states":["active","purging","verified","failed"],"cross_process":true}'::jsonb), + ('postgres_v05_neutral_import_v1', EXTRACT(EPOCH FROM NOW()), '{"schema":"migration_v05_imports+migration_v05_payloads","one_way":true,"runtime_workbook":false}'::jsonb), + ('postgres_enterprise_oidc_identity_v1', EXTRACT(EPOCH FROM NOW()), '{"identity":"issuer+subject","legacy_unbound":"api_key_only"}'::jsonb), + ('postgres_v7_readiness_projection_v1', EXTRACT(EPOCH FROM NOW()), '{"schema":"v7_default_readiness_reports","backfill":true}'::jsonb), + ('postgres_local_replay_projection_v1', EXTRACT(EPOCH FROM NOW()), '{"schema":"local_replay_artifacts+local_replay_baselines","backfill":true}'::jsonb), + ('postgres_run_evidence_projection_v1', EXTRACT(EPOCH FROM NOW()), '{"schema":"agent_run_artifacts+verification_reports_v2+workbook_benchmark_fixture_results+operation_audits","backfill":true}'::jsonb), + ('postgres_workbook_intelligence_projection_v1', EXTRACT(EPOCH FROM NOW()), '{"schema":"workbook_maps+source_evidence_graphs+operation_simulations+workbook_invariant_runs+agent_runs+workbook_truth_models+formula_graph_artifacts+operation_simulations_v2+v7_workbook_reviews","backfill":true}'::jsonb) + ON CONFLICT (migration_id) DO NOTHING; """) + async def _async_backfill_typed_projections(self, conn) -> None: + async with conn.transaction(): + await conn.execute( + "SELECT pg_advisory_xact_lock(hashtextextended(" + "'postgres-typed-projection-reconciliation', 0))" + ) + for table_name, (id_column, artifact_predicate) in ( + POSTGRES_TYPED_PROJECTION_RECONCILIATION.items() + ): + await conn.execute( + f"DELETE FROM {table_name} AS projection " # noqa: S608 -- static registry + "USING flagship_artifacts AS artifact " + f"WHERE projection.{id_column} = artifact.artifact_id " # noqa: S608 + f"AND ({artifact_predicate})" # noqa: S608 + ) + for statements in POSTGRES_TYPED_PROJECTION_BACKFILLS.values(): + for statement in statements: + await conn.execute(statement) + async def close(self): """Close the connection pool.""" if self._pool: - await self._pool.close() + pool = self._pool self._pool = None + pool_loop = self._pool_loop + self._pool_loop = None + running_loop = asyncio.get_running_loop() + if pool_loop and pool_loop.is_running() and pool_loop is not running_loop: + await asyncio.wrap_future(asyncio.run_coroutine_threadsafe(pool.close(), pool_loop)) + else: + await pool.close() + self._stop_sync_loop() # ── Sync wrappers (for compatibility with existing code) ───────── - def get_or_create_session(self, session_id: str) -> dict: - return asyncio.get_event_loop().run_until_complete( + def _ensure_sync_loop(self) -> asyncio.AbstractEventLoop: + """Return a dedicated loop for sync wrappers used inside async routes.""" + if self._sync_loop and self._sync_loop.is_running(): + return self._sync_loop + + with self._sync_loop_lock: + if self._sync_loop and self._sync_loop.is_running(): + return self._sync_loop + + ready = threading.Event() + + def _run_loop(): + loop = asyncio.new_event_loop() + asyncio.set_event_loop(loop) + self._sync_loop = loop + ready.set() + loop.run_forever() + pending = asyncio.all_tasks(loop) + for task in pending: + task.cancel() + if pending: + loop.run_until_complete(asyncio.gather(*pending, return_exceptions=True)) + loop.close() + + thread = threading.Thread( + target=_run_loop, + name="PostgresSessionStoreSyncLoop", + daemon=True, + ) + self._sync_loop_thread = thread + thread.start() + ready.wait(timeout=5) + if not self._sync_loop: + raise RuntimeError("Postgres sync compatibility loop failed to start") + return self._sync_loop + + def _run(self, coroutine): + """Run an async store method from sync API surfaces without nesting loops.""" + loop = self._ensure_sync_loop() + try: + running_loop = asyncio.get_running_loop() + except RuntimeError: + running_loop = None + if running_loop is loop: + raise RuntimeError("Postgres sync wrapper cannot be called from its own compatibility loop") + future = asyncio.run_coroutine_threadsafe(coroutine, loop) + return future.result() + + def _stop_sync_loop(self): + loop = self._sync_loop + thread = self._sync_loop_thread + self._sync_loop = None + self._sync_loop_thread = None + if loop and loop.is_running(): + loop.call_soon_threadsafe(loop.stop) + if thread and thread.is_alive() and thread is not threading.current_thread(): + thread.join(timeout=5) + + def get_or_create_session(self, session_id: str, owner_user_id: Optional[str] = None) -> dict: + principal = current_principal() + if principal.authenticated and not principal.is_admin: + owner_user_id = principal.user_id + elif owner_user_id is None and principal.authenticated: + owner_user_id = principal.user_id or None + coroutine = ( self._async_get_or_create_session(session_id) + if owner_user_id is None + else self._async_get_or_create_session(session_id, owner_user_id) + ) + return self._run(coroutine) + + def get_session_owner(self, session_id: str) -> Optional[str]: + return self._run(self._async_get_session_owner(session_id)) + + def session_exists(self, session_id: str) -> bool: + return self._run(self._async_session_exists(session_id)) + + def get_session_lifecycle_state(self, session_id: str) -> Optional[str]: + return self._run(self._async_get_session_lifecycle_state(session_id)) + + def get_session_lifecycle_record(self, session_id: str) -> Optional[dict]: + return self._run(self._async_get_session_lifecycle_record(session_id)) + + def set_session_lifecycle_state( + self, + session_id: str, + state: str, + *, + receipt_id: str, + started_at: float, + completed_at: float = 0.0, + errors: Optional[list[str]] = None, + ) -> None: + self._run( + self._async_set_session_lifecycle_state( + session_id, + state, + receipt_id=receipt_id, + started_at=started_at, + completed_at=completed_at, + errors=errors, + ) ) - def list_sessions(self, limit: int = 20) -> list[dict]: - return asyncio.get_event_loop().run_until_complete( - self._async_list_sessions(limit) + def session_deletion_verified(self, session_id: str) -> bool: + return self._run(self._async_session_deletion_verified(session_id)) + + def list_session_ids_before(self, cutoff_timestamp: float) -> list[str]: + return self._run(self._async_list_session_ids_before(cutoff_timestamp)) + + def list_document_file_paths(self, session_id: str) -> list[str]: + return self._run(self._async_list_document_file_paths(session_id)) + + def list_sessions( + self, + limit: int = 20, + owner_user_id: Optional[str] = None, + ) -> list[dict]: + principal = current_principal() + if principal.authenticated and not principal.is_admin: + if not principal.user_id: + return [] + owner_user_id = principal.user_id + return self._run( + self._async_list_sessions(limit, owner_user_id) ) def add_message(self, session_id: str, role: str, content: str, metadata: Optional[dict] = None): - asyncio.get_event_loop().run_until_complete( + self._run( self._async_add_message(session_id, role, content, metadata) ) def get_conversation(self, session_id: str, limit: int = 50) -> list[ConversationMessage]: - return asyncio.get_event_loop().run_until_complete( + return self._run( self._async_get_conversation(session_id, limit) ) @@ -156,7 +1970,7 @@ def record_task(self, session_id: str, task_id: str, prompt: str, mode: str, operations: list, summary: str, plan: Optional[list] = None, provider: str = "", model: str = "", tokens: Optional[dict] = None): - asyncio.get_event_loop().run_until_complete( + self._run( self._async_record_task( session_id, task_id, prompt, mode, operations, summary, plan, provider, model, tokens @@ -164,217 +1978,1102 @@ def record_task(self, session_id: str, task_id: str, prompt: str, ) def get_task_history(self, session_id: str, limit: int = 20) -> list[dict]: - return asyncio.get_event_loop().run_until_complete( + return self._run( self._async_get_task_history(session_id, limit) ) def get_total_tokens(self, session_id: Optional[str] = None) -> dict: - return asyncio.get_event_loop().run_until_complete( + return self._run( self._async_get_total_tokens(session_id) ) def get_migration_status(self) -> dict: return { "engine": "postgres", - "schema": "flagship_artifacts", + "schema": "flagship_artifacts+typed_projections", "pending": [], - "applied": ["postgres_artifact_parity_v1"], + "applied": [ + "postgres_artifact_parity_v1", + "postgres_artifact_identity_v2", + "postgres_session_lifecycle_fencing_v1", + "postgres_v05_neutral_import_v1", + *POSTGRES_TYPED_PROJECTION_MIGRATIONS, + ], + "identity": ["artifact_id", "artifact_type"], + "ledger": "postgres_schema_migrations", + "typed_projections": list(POSTGRES_TYPED_PROJECTION_TABLES), + "projection_migrations": { + migration_id: list(tables) + for migration_id, tables in POSTGRES_TYPED_PROJECTION_MIGRATIONS.items() + }, + "typed_projection_backfills": { + table: len(statements) + for table, statements in POSTGRES_TYPED_PROJECTION_BACKFILLS.items() + }, } - def save_workbook_map(self, semantic_map: dict) -> str: - artifact_id = semantic_map.get("mapId") or semantic_map.get("map_id") or f"wmap_{uuid.uuid4().hex[:12]}" - return asyncio.get_event_loop().run_until_complete( - self._async_save_artifact(artifact_id, "workbook_map", semantic_map.get("sessionId", ""), "", "", semantic_map) - ) - - def save_source_evidence_graph(self, graph: dict) -> str: - artifact_id = graph.get("graphId") or graph.get("graph_id") or f"sev_{uuid.uuid4().hex[:12]}" - return asyncio.get_event_loop().run_until_complete( - self._async_save_artifact(artifact_id, "source_evidence_graph", graph.get("sessionId", ""), "", "", graph) - ) - - def save_operation_simulation(self, session_id: str, simulation: dict, *, plan_id: str = "", run_id: str = "") -> str: - artifact_id = simulation.get("simulationId") or f"sim_{uuid.uuid4().hex[:12]}" - return asyncio.get_event_loop().run_until_complete( - self._async_save_artifact(artifact_id, "operation_simulation", session_id, run_id, plan_id, simulation) - ) + # ── One-way v0.5 neutral migration intake ───────────────────── - def save_workbook_invariant_run(self, session_id: str, report: dict, *, run_id: str = "", plan_id: str = "") -> str: - artifact_id = report.get("reportId") or f"inv_{uuid.uuid4().hex[:12]}" - return asyncio.get_event_loop().run_until_complete( - self._async_save_artifact(artifact_id, "workbook_invariant", session_id, run_id, plan_id, report) - ) + def consume_v05_import(self, request: dict, payloads: list[dict]) -> dict: + normalized, normalized_payloads = normalize_import_bundle(request, payloads) + principal = current_principal() + if ( + principal.authenticated + and not principal.is_admin + and principal.user_id != normalized["ownerId"] + ): + raise NeutralV05ImportError("owner_scope_forbidden") + from backend.services.lifecycle import get_lifecycle_coordinator - def save_agent_run(self, run: dict) -> str: - artifact_id = run.get("runId") or run.get("run_id") or f"run_{uuid.uuid4().hex[:12]}" - return asyncio.get_event_loop().run_until_complete( - self._async_save_artifact(artifact_id, "agent_run", run.get("sessionId", ""), artifact_id, run.get("planId", ""), run) - ) + if get_lifecycle_coordinator().writes_revoked(str(normalized["targetSessionId"])): + raise NeutralV05ImportError("session_fenced") + return self._run(self._async_consume_v05_import(normalized, normalized_payloads)) - def get_agent_run(self, run_id: str) -> Optional[dict]: - return asyncio.get_event_loop().run_until_complete(self._async_get_artifact(run_id, "agent_run")) + def get_v05_import(self, import_id: str) -> Optional[dict]: + normalized_import_id = str(import_id or "").strip() + prefix = "v05-import-" + if not normalized_import_id.startswith(prefix): + return None + try: + if neutral_import_id(normalized_import_id.removeprefix(prefix)) != normalized_import_id: + return None + except NeutralV05ImportError: + return None + return self._run(self._async_get_v05_import(normalized_import_id)) - def save_workbook_truth_model(self, truth_model: dict) -> str: - artifact_id = truth_model.get("truthModelId") or f"truth_{uuid.uuid4().hex[:12]}" - return asyncio.get_event_loop().run_until_complete( - self._async_save_artifact(artifact_id, "workbook_truth_model", truth_model.get("sessionId", ""), "", "", truth_model) + def read_v05_import_bundle( + self, + import_id: str, + *, + expected_binding_sha256: str, + expected_content_sha256: str, + expected_owner_id: str, + expected_owner_scope_sha256: str, + expected_target_session_id: str, + ) -> Optional[dict]: + normalized_import_id = str(import_id or "").strip() + prefix = "v05-import-" + if not normalized_import_id.startswith(prefix): + return None + try: + if neutral_import_id(normalized_import_id.removeprefix(prefix)) != normalized_import_id: + return None + expected_binding = normalize_sha256(expected_binding_sha256) + expected_content = normalize_sha256(expected_content_sha256) + expected_owner = normalize_owner_id(expected_owner_id) + expected_owner_scope = normalize_sha256(expected_owner_scope_sha256) + expected_session = str(expected_target_session_id or "").strip() + if ( + normalized_import_id != neutral_import_id(expected_content) + or expected_session != neutral_session_id(expected_content) + or not hmac.compare_digest(expected_owner_scope, sha256_text(expected_owner)) + ): + return None + except NeutralV05ImportError: + return None + return self._run( + self._async_read_v05_import_bundle( + normalized_import_id, + expected_binding_sha256=expected_binding, + expected_content_sha256=expected_content, + expected_owner_id=expected_owner, + expected_owner_scope_sha256=expected_owner_scope, + expected_target_session_id=expected_session, + ) ) - def get_workbook_truth_model(self, truth_model_id: str) -> Optional[dict]: - return asyncio.get_event_loop().run_until_complete(self._async_get_artifact(truth_model_id, "workbook_truth_model")) - - def save_formula_graph_artifact(self, graph: dict) -> str: - artifact_id = graph.get("graphId") or f"fgraph_{uuid.uuid4().hex[:12]}" - return asyncio.get_event_loop().run_until_complete( - self._async_save_artifact(artifact_id, "formula_graph", graph.get("sessionId", ""), "", "", graph) - ) + async def _async_consume_v05_import( + self, + request: dict, + payloads: tuple[dict, ...], + ) -> dict: + claim_version: int | None = None + try: + prepared = await self._async_prepare_v05_import(request, payloads) + if prepared["state"] == "imported": + return prepared + claimed = await self._async_claim_v05_import(request, payloads) + if claimed["state"] == "imported": + return claimed + claim_version = int(claimed["version"]) + return await self._async_complete_v05_import( + request, + payloads, + expected_version=claim_version, + ) + except NeutralV05ImportError: + raise + except Exception as exc: + if claim_version is not None: + try: + await self._async_mark_v05_import_retryable_failure( + request, + expected_version=claim_version, + failure_code="storage_write_failed", + ) + except Exception: + pass + raise NeutralV05ImportError("storage_write_failed") from exc + + async def _async_prepare_v05_import( + self, + request: dict, + payloads: tuple[dict, ...], + ) -> dict: + pool = await self._get_pool() + session_id = str(request["targetSessionId"]) + now = time.time() + async with pool.acquire() as conn: + async with conn.transaction(): + await conn.execute( + "SELECT pg_advisory_xact_lock(hashtextextended($1, 0))", + session_id, + ) + lifecycle = await conn.fetchrow( + "SELECT state FROM session_lifecycle WHERE session_id = $1 FOR UPDATE", + session_id, + ) + session = await conn.fetchrow( + "SELECT owner_user_id FROM sessions WHERE session_id = $1 FOR UPDATE", + session_id, + ) + if lifecycle and str(lifecycle["state"] or "") != "active": + raise NeutralV05ImportError("session_fenced") + if session: + if lifecycle is None: + raise NeutralV05ImportError("corrupt_import_state") + if str(session["owner_user_id"] or "") != request["ownerId"]: + raise NeutralV05ImportError("owner_session_conflict") + else: + if lifecycle is not None: + raise NeutralV05ImportError("session_fenced") + await conn.execute( + "INSERT INTO session_lifecycle " + "(session_id, state, receipt_id, started_at, completed_at, errors, updated_at) " + "VALUES ($1, 'active', '', 0, 0, '[]'::jsonb, $2)", + session_id, + now, + ) + await conn.execute( + "INSERT INTO sessions " + "(session_id, owner_user_id, created_at, updated_at, metadata) " + "VALUES ($1, $2, $3, $4, $5::jsonb)", + session_id, + request["ownerId"], + now, + now, + json.dumps( + { + "contentSha256": request["contentSha256"], + "neutralV05Import": True, + }, + sort_keys=True, + ), + ) + + row = await conn.fetchrow( + "SELECT * FROM migration_v05_imports WHERE import_id = $1 FOR UPDATE", + request["importId"], + ) + if row: + return await self._async_v05_status_in_conn( + conn, + dict(row), + request, + payloads=payloads, + ) + await conn.execute( + """ + INSERT INTO migration_v05_imports + (import_id, session_id, schema_version, state, version, attempt_count, + binding_sha256, archive_sha256, content_sha256, owner_scope_sha256, + manifest_sha256, stage_receipt_sha256, counts_json, payload_count, + receipt_json, failure_code, created_at, updated_at) + VALUES + ($1, $2, $3, 'validated_pending_import', 0, 0, + $4, $5, $6, $7, $8, $9, $10::jsonb, 0, '{}'::jsonb, '', $11, $12) + """, + request["importId"], + session_id, + request["schemaVersion"], + request["bindingSha256"], + request["archiveSha256"], + request["contentSha256"], + request["ownerScopeSha256"], + request["manifestSha256"], + request["stageReceiptSha256"], + json.dumps(request["counts"], sort_keys=True), + now, + now, + ) + inserted = await conn.fetchrow( + "SELECT * FROM migration_v05_imports WHERE import_id = $1 FOR UPDATE", + request["importId"], + ) + if not inserted: + raise NeutralV05ImportError("storage_write_failed") + return await self._async_v05_status_in_conn( + conn, + dict(inserted), + request, + payloads=payloads, + ) + + async def _async_claim_v05_import( + self, + request: dict, + payloads: tuple[dict, ...], + ) -> dict: + pool = await self._get_pool() + async with pool.acquire() as conn: + async with conn.transaction(): + await self._async_require_v05_import_session(conn, request) + row = await conn.fetchrow( + "SELECT * FROM migration_v05_imports WHERE import_id = $1 FOR UPDATE", + request["importId"], + ) + if not row: + raise NeutralV05ImportError("corrupt_import_state") + row_dict = dict(row) + ensure_row_binding(row_dict, request) + state = str(row_dict["state"] or "") + if state in {"imported", "importing"}: + return await self._async_v05_status_in_conn( + conn, + row_dict, + request, + payloads=payloads, + ) + if state not in {"validated_pending_import", "failed_retryable"}: + raise NeutralV05ImportError("corrupt_import_state") + version = int(row_dict["version"]) + result = await conn.execute( + "UPDATE migration_v05_imports " + "SET state = 'importing', version = version + 1, " + "attempt_count = attempt_count + 1, failure_code = '', updated_at = $1 " + "WHERE import_id = $2 AND binding_sha256 = $3 AND version = $4 AND state = $5", + time.time(), + request["importId"], + request["bindingSha256"], + version, + state, + ) + if self._row_count(result) != 1: + raise NeutralV05ImportError("import_cas_conflict") + claimed = await conn.fetchrow( + "SELECT * FROM migration_v05_imports WHERE import_id = $1 FOR UPDATE", + request["importId"], + ) + if not claimed: + raise NeutralV05ImportError("corrupt_import_state") + return await self._async_v05_status_in_conn( + conn, + dict(claimed), + request, + payloads=payloads, + ) + + async def _async_complete_v05_import( + self, + request: dict, + payloads: tuple[dict, ...], + *, + expected_version: int, + ) -> dict: + pool = await self._get_pool() + async with pool.acquire() as conn: + async with conn.transaction(): + await self._async_require_v05_import_session(conn, request) + row = await conn.fetchrow( + "SELECT * FROM migration_v05_imports WHERE import_id = $1 FOR UPDATE", + request["importId"], + ) + if not row: + raise NeutralV05ImportError("corrupt_import_state") + row_dict = dict(row) + ensure_row_binding(row_dict, request) + if str(row_dict["state"] or "") == "imported": + return await self._async_v05_status_in_conn( + conn, + row_dict, + request, + payloads=payloads, + ) + if ( + str(row_dict["state"] or "") != "importing" + or int(row_dict["version"]) != expected_version + ): + raise NeutralV05ImportError("import_cas_conflict") + existing = await conn.fetchval( + "SELECT COUNT(*) FROM migration_v05_payloads WHERE import_id = $1", + request["importId"], + ) + if int(existing or 0) != 0: + raise NeutralV05ImportError("corrupt_import_state") + for payload in payloads: + await conn.execute( + """ + INSERT INTO migration_v05_payloads + (import_id, session_id, relative_path, payload_kind, + payload_sha256, size_bytes, payload_bytes) + VALUES ($1, $2, $3, $4, $5, $6, $7) + """, + request["importId"], + request["targetSessionId"], + payload["path"], + payload["kind"], + payload["sha256"], + payload["sizeBytes"], + payload["content"], + ) + await self._async_verify_v05_payload_inventory( + conn, + request, + payloads=payloads, + require_complete=True, + ) + receipt = build_import_receipt(request, stored_payloads=len(payloads)) + result = await conn.execute( + "UPDATE migration_v05_imports " + "SET state = 'imported', version = version + 1, payload_count = $1, " + "receipt_json = $2::jsonb, failure_code = '', updated_at = $3 " + "WHERE import_id = $4 AND binding_sha256 = $5 " + "AND version = $6 AND state = 'importing'", + len(payloads), + v05_canonical_json(receipt).decode("utf-8"), + time.time(), + request["importId"], + request["bindingSha256"], + expected_version, + ) + if self._row_count(result) != 1: + raise NeutralV05ImportError("import_cas_conflict") + completed = await conn.fetchrow( + "SELECT * FROM migration_v05_imports WHERE import_id = $1 FOR UPDATE", + request["importId"], + ) + if not completed: + raise NeutralV05ImportError("corrupt_import_state") + return await self._async_v05_status_in_conn( + conn, + dict(completed), + request, + payloads=payloads, + ) + + async def _async_mark_v05_import_retryable_failure( + self, + request: dict, + *, + expected_version: int, + failure_code: str, + ) -> None: + pool = await self._get_pool() + async with pool.acquire() as conn: + async with conn.transaction(): + await conn.execute( + "UPDATE migration_v05_imports " + "SET state = 'failed_retryable', version = version + 1, " + "failure_code = $1, updated_at = $2 " + "WHERE import_id = $3 AND binding_sha256 = $4 " + "AND state = 'importing' AND version = $5", + failure_code, + time.time(), + request["importId"], + request["bindingSha256"], + expected_version, + ) + + async def _async_get_v05_import(self, import_id: str) -> Optional[dict]: + pool = await self._get_pool() + async with pool.acquire() as conn: + async with conn.transaction(isolation="repeatable_read"): + preflight = await conn.fetchrow( + "SELECT session_id FROM migration_v05_imports WHERE import_id = $1", + import_id, + ) + if not preflight: + return None + session_id = str(preflight["session_id"] or "") + if not await self._async_session_visible_in_transaction(conn, session_id): + return None + row = await conn.fetchrow( + "SELECT * FROM migration_v05_imports WHERE import_id = $1 FOR SHARE", + import_id, + ) + if not row: + return None + row_dict = dict(row) + if str(row_dict["session_id"] or "") != session_id: + raise NeutralV05ImportError("corrupt_import_state") + request = self._postgres_v05_request_from_row(row_dict) + return await self._async_v05_status_in_conn(conn, row_dict, request) + + async def _async_read_v05_import_bundle( + self, + import_id: str, + *, + expected_binding_sha256: str, + expected_content_sha256: str, + expected_owner_id: str, + expected_owner_scope_sha256: str, + expected_target_session_id: str, + ) -> Optional[dict]: + pool = await self._get_pool() + async with pool.acquire() as conn: + async with conn.transaction(isolation="repeatable_read"): + if not await self._async_session_visible_in_transaction( + conn, + expected_target_session_id, + ): + return None + row = await conn.fetchrow( + "SELECT * FROM migration_v05_imports WHERE import_id = $1 FOR SHARE", + import_id, + ) + if not row: + return None + row_dict = dict(row) + session_id = str(row_dict["session_id"]) + if session_id != expected_target_session_id: + return None + owner_row = await conn.fetchrow( + "SELECT owner_user_id FROM sessions WHERE session_id = $1 FOR SHARE", + session_id, + ) + if not owner_row: + return None + try: + stored_owner = normalize_owner_id(owner_row["owner_user_id"]) + except NeutralV05ImportError as exc: + raise NeutralV05ImportError("corrupt_import_state") from exc + if ( + stored_owner != expected_owner_id + or not hmac.compare_digest( + sha256_text(stored_owner), + expected_owner_scope_sha256, + ) + ): + return None + request = self._postgres_v05_request_from_row(row_dict) + if ( + not hmac.compare_digest( + request["bindingSha256"], + expected_binding_sha256, + ) + or not hmac.compare_digest( + request["contentSha256"], + expected_content_sha256, + ) + or not hmac.compare_digest( + request["ownerScopeSha256"], + expected_owner_scope_sha256, + ) + or request["targetSessionId"] != expected_target_session_id + ): + return None + request["ownerId"] = stored_owner + if str(row_dict.get("state") or "") != "imported": + raise NeutralV05ImportError("import_not_ready") + await self._async_enforce_v05_materialization_inventory_budget( + conn, + import_id, + ) + rows = await conn.fetch( + "SELECT relative_path, payload_kind, payload_sha256, size_bytes, " + "payload_bytes FROM migration_v05_payloads " + "WHERE import_id = $1 ORDER BY relative_path", + import_id, + ) + payloads = tuple( + { + "content": payload["payload_bytes"], + "kind": payload["payload_kind"], + "path": payload["relative_path"], + "sha256": payload["payload_sha256"], + "sizeBytes": payload["size_bytes"], + } + for payload in rows + ) + await self._async_verify_v05_payload_inventory( + conn, + request, + payloads=payloads, + require_complete=True, + ) + status = await self._async_v05_status_in_conn( + conn, + row_dict, + request, + payloads=payloads, + ) + normalized_request, normalized_payloads = normalize_import_bundle( + request, + list(payloads), + ) + normalized_receipt = validate_import_receipt( + status["receipt"], + normalized_request, + stored_payloads=len(normalized_payloads), + ) + return { + "payloads": normalized_payloads, + "receipt": normalized_receipt, + "request": normalized_request, + } + + @staticmethod + async def _async_enforce_v05_materialization_inventory_budget( + conn, + import_id: str, + ) -> None: + inventory = await conn.fetchrow( + "SELECT COUNT(*)::bigint AS payload_count, " + "COALESCE(SUM(size_bytes), 0)::bigint AS declared_bytes, " + "COALESCE(MAX(size_bytes), 0)::bigint AS max_declared_bytes, " + "COALESCE(SUM(octet_length(payload_bytes)), 0)::bigint AS actual_bytes, " + "COALESCE(MAX(octet_length(payload_bytes)), 0)::bigint AS max_actual_bytes " + "FROM migration_v05_payloads WHERE import_id = $1", + import_id, + ) + if not inventory: + raise NeutralV05ImportError("corrupt_import_payloads") + values = { + key: inventory[key] + for key in ( + "payload_count", + "declared_bytes", + "max_declared_bytes", + "actual_bytes", + "max_actual_bytes", + ) + } + if any( + not isinstance(value, int) or isinstance(value, bool) or value < 0 + for value in values.values() + ): + raise NeutralV05ImportError("corrupt_import_payloads") + if values["declared_bytes"] != values["actual_bytes"]: + raise NeutralV05ImportError("corrupt_import_payloads") + if values["max_declared_bytes"] != values["max_actual_bytes"]: + raise NeutralV05ImportError("corrupt_import_payloads") + if values["payload_count"] == 0: + raise NeutralV05ImportError("corrupt_import_payloads") + if ( + values["payload_count"] > MATERIALIZATION_MAX_PAYLOADS + or values["actual_bytes"] > MATERIALIZATION_MAX_PAYLOAD_BYTES + or values["max_actual_bytes"] > MATERIALIZATION_MAX_ENTRY_BYTES + ): + raise NeutralV05ImportError("materialization_budget_exceeded") + + async def _async_require_v05_import_session(self, conn, request: dict) -> None: + session_id = str(request["targetSessionId"]) + await conn.execute( + "SELECT pg_advisory_xact_lock(hashtextextended($1, 0))", + session_id, + ) + row = await conn.fetchrow( + "SELECT session.owner_user_id, lifecycle.state " + "FROM sessions AS session " + "JOIN session_lifecycle AS lifecycle ON lifecycle.session_id = session.session_id " + "WHERE session.session_id = $1 FOR UPDATE OF session, lifecycle", + session_id, + ) + if not row or str(row["state"] or "") != "active": + raise NeutralV05ImportError("session_fenced") + if str(row["owner_user_id"] or "") != request["ownerId"]: + raise NeutralV05ImportError("owner_session_conflict") + principal = current_principal() + if ( + principal.authenticated + and not principal.is_admin + and principal.user_id != request["ownerId"] + ): + raise NeutralV05ImportError("owner_scope_forbidden") + + async def _async_v05_status_in_conn( + self, + conn, + row: dict, + request: dict, + *, + payloads: tuple[dict, ...] | None = None, + ) -> dict: + state = str(row.get("state") or "") + if state not in NEUTRAL_IMPORT_STATES: + raise NeutralV05ImportError("corrupt_import_state") + stored_payloads = await self._async_verify_v05_payload_inventory( + conn, + request, + payloads=payloads if state == "imported" else None, + require_complete=state == "imported", + ) + if state != "imported" and stored_payloads != 0: + raise NeutralV05ImportError("corrupt_import_state") + payload_count = row.get("payload_count") + if ( + not isinstance(payload_count, int) + or isinstance(payload_count, bool) + or payload_count < 0 + ): + raise NeutralV05ImportError("corrupt_import_state") + if state == "imported" and payload_count != stored_payloads: + raise NeutralV05ImportError("corrupt_import_state") + return public_import_status( + row, + request, + stored_payloads=stored_payloads, + ) + + @staticmethod + async def _async_verify_v05_payload_inventory( + conn, + request: dict, + *, + payloads: tuple[dict, ...] | None, + require_complete: bool = False, + ) -> int: + rows = await conn.fetch( + "SELECT session_id, relative_path, payload_kind, payload_sha256, " + "size_bytes, payload_bytes FROM migration_v05_payloads " + "WHERE import_id = $1 ORDER BY relative_path", + request["importId"], + ) + if payloads is not None and len(rows) != len(payloads): + raise NeutralV05ImportError("corrupt_import_payloads") + if not rows: + if require_complete: + raise NeutralV05ImportError("corrupt_import_payloads") + return 0 + descriptors: list[dict] = [] + actual_counts = {"documents": 0, "sessions": 0, "workflows": 0} + for index, raw_row in enumerate(rows): + row = dict(raw_row) + content = row.get("payload_bytes") + size_bytes = row.get("size_bytes") + session_id = row.get("session_id") + payload_kind = row.get("payload_kind") + relative_path = row.get("relative_path") + payload_sha256 = row.get("payload_sha256") + if ( + not isinstance(content, bytes) + or not isinstance(size_bytes, int) + or isinstance(size_bytes, bool) + or not isinstance(session_id, str) + or not isinstance(payload_kind, str) + or payload_kind not in {"blob", "document", "session", "workflow_recipe"} + or not isinstance(relative_path, str) + or not relative_path + or not isinstance(payload_sha256, str) + or session_id != request["targetSessionId"] + or size_bytes != len(content) + or payload_sha256 != sha256_bytes(content) + ): + raise NeutralV05ImportError("corrupt_import_payloads") + descriptor = { + "kind": payload_kind, + "path": relative_path, + "sha256": payload_sha256, + "size_bytes": size_bytes, + } + descriptors.append(descriptor) + if descriptor["kind"] == "document": + actual_counts["documents"] += 1 + elif descriptor["kind"] == "session": + actual_counts["sessions"] += 1 + elif descriptor["kind"] == "workflow_recipe": + actual_counts["workflows"] += 1 + if payloads is not None: + expected = payloads[index] + if ( + descriptor["path"] != expected["path"] + or descriptor["kind"] != expected["kind"] + or descriptor["sha256"] != expected["sha256"] + or descriptor["size_bytes"] != expected["sizeBytes"] + or content != expected["content"] + ): + raise NeutralV05ImportError("corrupt_import_payloads") + if sha256_bytes(v05_canonical_json(descriptors, newline=True)) != request[ + "contentSha256" + ] or actual_counts != request["counts"]: + raise NeutralV05ImportError("corrupt_import_payloads") + return len(rows) + + @staticmethod + def _postgres_v05_request_from_row(row: dict) -> dict: + counts: Any = row.get("counts_json") + if isinstance(counts, str): + try: + counts = json.loads(counts) + except (TypeError, ValueError) as exc: + raise NeutralV05ImportError("corrupt_import_state") from exc + if not isinstance(counts, dict): + raise NeutralV05ImportError("corrupt_import_state") + request: dict[str, Any] = { + "archiveSha256": str(row.get("archive_sha256") or ""), + "bindingSha256": str(row.get("binding_sha256") or ""), + "contentSha256": str(row.get("content_sha256") or ""), + "counts": counts, + "importId": str(row.get("import_id") or ""), + "manifestSha256": str(row.get("manifest_sha256") or ""), + "ownerScopeSha256": str(row.get("owner_scope_sha256") or ""), + "schemaVersion": str(row.get("schema_version") or ""), + "stageReceiptSha256": str(row.get("stage_receipt_sha256") or ""), + "targetSessionId": str(row.get("session_id") or ""), + } + if ( + request["importId"] != neutral_import_id(request["contentSha256"]) + or request["targetSessionId"] != neutral_session_id(request["contentSha256"]) + ): + raise NeutralV05ImportError("corrupt_import_state") + if v05_binding_sha256(request) != request["bindingSha256"]: + raise NeutralV05ImportError("corrupt_import_state") + ensure_row_binding(row, request) + return request + + def save_workbook_job(self, job: dict) -> str: + run_id = str(job["run_id"]) + record = dict(job) + principal = current_principal() + if principal.authenticated and not principal.is_admin: + session_id = str(record.get("session_id") or "") + if not principal.user_id or not session_id: + raise OwnershipNotFoundError("Workbook job not found") + record["owner_user_id"] = principal.user_id + record["account_id"] = principal.user_id + elif principal.authenticated: + record.setdefault("owner_user_id", principal.user_id) + else: + record.setdefault("owner_user_id", "") + return self._run( + self._async_save_artifact( + run_id, + "workbook_job", + str(record.get("session_id") or ""), + run_id, + str(record.get("plan_id") or ""), + record, + ) + ) + + def get_workbook_job(self, run_id: str) -> Optional[dict]: + return self._run(self._async_get_artifact(run_id, "workbook_job")) + + def list_workbook_jobs( + self, + limit: int = 50, + status: Optional[str] = None, + session_id: Optional[str] = None, + account_id: Optional[str] = None, + ) -> list[dict]: + return self._run(self._async_list_workbook_jobs(limit, status, session_id, account_id)) + + def save_workbook_map(self, semantic_map: dict) -> str: + artifact_id = semantic_map.get("mapId") or semantic_map.get("map_id") or f"wmap_{uuid.uuid4().hex[:12]}" + return self._run( + self._async_save_workbook_map(artifact_id, semantic_map) + ) + + def get_workbook_map(self, map_id: str) -> Optional[dict]: + typed = self._run(self._async_get_typed_json("workbook_maps", "map_json", "map_id", map_id)) + if typed: + return typed + return self._run(self._async_get_artifact(map_id, "workbook_map")) + + def save_source_evidence_graph(self, graph: dict) -> str: + artifact_id = graph.get("graphId") or graph.get("graph_id") or f"sev_{uuid.uuid4().hex[:12]}" + return self._run( + self._async_save_source_evidence_graph(artifact_id, graph) + ) + + def save_operation_simulation(self, session_id: str, simulation: dict, *, plan_id: str = "", run_id: str = "") -> str: + artifact_id = simulation.get("simulationId") or f"sim_{uuid.uuid4().hex[:12]}" + return self._run( + self._async_save_operation_simulation(artifact_id, session_id, simulation, plan_id=plan_id, run_id=run_id) + ) + + def save_workbook_invariant_run(self, session_id: str, report: dict, *, run_id: str = "", plan_id: str = "") -> str: + artifact_id = report.get("reportId") or f"inv_{uuid.uuid4().hex[:12]}" + return self._run( + self._async_save_workbook_invariant_run(artifact_id, session_id, report, run_id=run_id, plan_id=plan_id) + ) + + def save_agent_run(self, run: dict) -> str: + artifact_id = run.get("runId") or run.get("run_id") or f"run_{uuid.uuid4().hex[:12]}" + return self._run( + self._async_save_agent_run(artifact_id, run) + ) + + def get_agent_run(self, run_id: str) -> Optional[dict]: + typed = self._run(self._async_get_typed_json("agent_runs", "run_json", "run_id", run_id)) + if typed: + return typed + return self._run(self._async_get_artifact(run_id, "agent_run")) + + def append_agent_run_event( + self, + run_id: str, + *, + expected_sequence: int, + event_type: str, + event_payload: dict, + updated_run: dict, + idempotency_key: str = "", + ) -> dict: + return self._run( + self._async_append_agent_run_event( + run_id, + expected_sequence=expected_sequence, + event_type=event_type, + event_payload=event_payload, + updated_run=updated_run, + idempotency_key=idempotency_key, + ) + ) + + def list_agent_run_events( + self, + run_id: str, + after_sequence: int = 0, + limit: int = 100, + ) -> dict: + return self._run( + self._async_list_agent_run_events( + run_id, + after_sequence=after_sequence, + limit=limit, + ) + ) + + def save_agent_egress_preview(self, session_id: str, preview: dict) -> str: + return self._run(self._async_save_agent_egress_preview(session_id, preview)) + + def consume_agent_egress_approval( + self, + session_id: str, + *, + approval_id: str, + approved_digest: str, + request_digest: str, + consumption_id: str, + now: Optional[float] = None, + ) -> dict: + return self._run( + self._async_consume_agent_egress_approval( + session_id, + approval_id=approval_id, + approved_digest=approved_digest, + request_digest=request_digest, + consumption_id=consumption_id, + now=now, + ) + ) + + def save_workbook_truth_model(self, truth_model: dict) -> str: + artifact_id = truth_model.get("truthModelId") or f"truth_{uuid.uuid4().hex[:12]}" + return self._run( + self._async_save_workbook_truth_model(artifact_id, truth_model) + ) + + def get_workbook_truth_model(self, truth_model_id: str) -> Optional[dict]: + typed = self._run( + self._async_get_typed_json("workbook_truth_models", "truth_json", "truth_model_id", truth_model_id) + ) + if typed: + return typed + return self._run(self._async_get_artifact(truth_model_id, "workbook_truth_model")) + + def save_formula_graph_artifact(self, graph: dict) -> str: + artifact_id = graph.get("graphId") or f"fgraph_{uuid.uuid4().hex[:12]}" + return self._run( + self._async_save_formula_graph_artifact(artifact_id, graph) + ) def get_formula_graph_artifact(self, graph_id: str) -> Optional[dict]: - return asyncio.get_event_loop().run_until_complete(self._async_get_artifact(graph_id, "formula_graph")) + typed = self._run(self._async_get_typed_json("formula_graph_artifacts", "graph_json", "graph_id", graph_id)) + if typed: + return typed + return self._run(self._async_get_artifact(graph_id, "formula_graph")) def save_operation_simulation_v2(self, session_id: str, simulation: dict, *, plan_id: str = "", run_id: str = "") -> str: artifact_id = simulation.get("simulationId") or f"sim_{uuid.uuid4().hex[:12]}" - return asyncio.get_event_loop().run_until_complete( - self._async_save_artifact(artifact_id, "operation_simulation_v2", session_id, run_id, plan_id, simulation) + return self._run( + self._async_save_operation_simulation_v2(artifact_id, session_id, simulation, plan_id=plan_id, run_id=run_id) ) def save_v7_workbook_review(self, review: dict) -> str: artifact_id = review.get("reviewId") or f"v7review_{uuid.uuid4().hex[:12]}" - return asyncio.get_event_loop().run_until_complete( - self._async_save_artifact( - artifact_id, - "v7_workbook_review", - review.get("sessionId", ""), - review.get("runId", ""), - review.get("planId", ""), - review, - ) + return self._run( + self._async_save_v7_workbook_review(artifact_id, review) ) def get_v7_workbook_review(self, review_id: str) -> Optional[dict]: - return asyncio.get_event_loop().run_until_complete(self._async_get_artifact(review_id, "v7_workbook_review")) + typed = self._run(self._async_get_typed_json("v7_workbook_reviews", "review_json", "review_id", review_id)) + if typed: + return typed + return self._run(self._async_get_artifact(review_id, "v7_workbook_review")) def save_local_replay_artifact(self, session_id: str, artifact_type: str, payload: dict) -> str: artifact_id = f"replay_{uuid.uuid4().hex[:12]}" replay_payload = {**payload, "replayId": artifact_id, "artifactType": artifact_type} - return asyncio.get_event_loop().run_until_complete( - self._async_save_artifact(artifact_id, f"local_replay:{artifact_type}", session_id, "", "", replay_payload) - ) + return self._run(self._async_save_local_replay_artifact(artifact_id, session_id, artifact_type, replay_payload)) def list_local_replay_artifacts(self, limit: int = 100) -> list[dict]: - return asyncio.get_event_loop().run_until_complete(self._async_list_artifacts_by_prefix("local_replay:", limit)) + rows = self._run(self._async_list_local_replay_artifacts(limit)) + if rows: + return rows + return self._run(self._async_list_artifacts_by_prefix("local_replay:", limit)) def clear_local_replay_artifacts(self, session_id: str) -> int: - return asyncio.get_event_loop().run_until_complete(self._async_delete_artifacts_by_prefix("local_replay:", session_id)) + return self._run(self._async_clear_local_replay_artifacts(session_id)) def save_local_replay_baseline(self, baseline: dict) -> str: artifact_id = baseline.get("baselineId") or f"replaybase_{uuid.uuid4().hex[:12]}" baseline = {**baseline, "baselineId": artifact_id} - return asyncio.get_event_loop().run_until_complete( - self._async_save_artifact( - artifact_id, - "local_replay_baseline", - baseline.get("sessionId", "local-replay-baseline"), - "", - "", - baseline, - ) - ) + return self._run(self._async_save_local_replay_baseline(baseline)) def list_local_replay_baselines(self, limit: int = 20) -> list[dict]: - rows = asyncio.get_event_loop().run_until_complete(self._async_list_artifacts_by_prefix("local_replay_baseline", limit)) + rows = self._run(self._async_list_local_replay_baselines(limit)) + if rows: + return rows + rows = self._run(self._async_list_artifacts_by_prefix("local_replay_baseline", limit)) return [row.get("artifact") or row.get("payload") or row for row in rows] def save_v7_default_readiness_report(self, report: dict) -> str: artifact_id = report.get("readinessId") or f"v7ready_{uuid.uuid4().hex[:12]}" - return asyncio.get_event_loop().run_until_complete( - self._async_save_artifact(artifact_id, "v7_default_readiness_report", "", "", "", report) - ) + payload = {**report, "readinessId": artifact_id} + return self._run(self._async_save_v7_default_readiness_report(payload)) def get_latest_v7_default_readiness_report(self) -> Optional[dict]: - reports = asyncio.get_event_loop().run_until_complete(self._async_list_artifacts_by_prefix("v7_default_readiness_report", 1)) - if not reports: - return None - return reports[0].get("artifact") or reports[0].get("payload") or reports[0] + reports = self._run(self._async_list_v7_default_readiness_reports(1)) + if reports: + return reports[0] + reports = self._run(self._async_list_artifacts_by_prefix("v7_default_readiness_report", 1)) + return (reports[0].get("artifact") or reports[0].get("payload") or reports[0]) if reports else None def list_v7_default_readiness_reports(self, limit: int = 20) -> list[dict]: - reports = asyncio.get_event_loop().run_until_complete(self._async_list_artifacts_by_prefix("v7_default_readiness_report", limit)) + reports = self._run(self._async_list_v7_default_readiness_reports(limit)) + if reports: + return reports + reports = self._run(self._async_list_artifacts_by_prefix("v7_default_readiness_report", limit)) return [row.get("artifact") or row.get("payload") or row for row in reports] def save_rollback_snapshot_v2(self, snapshot: dict) -> str: artifact_id = snapshot.get("snapshotId") or f"rollback_{uuid.uuid4().hex[:12]}" - return asyncio.get_event_loop().run_until_complete( + return self._run( self._async_save_artifact(artifact_id, "rollback_snapshot", snapshot.get("sessionId", ""), snapshot.get("runId", ""), snapshot.get("planId", ""), snapshot) ) def save_brain_run(self, run: dict) -> str: run_id = run.get("runId") or f"brainrun_{uuid.uuid4().hex[:12]}" - return asyncio.get_event_loop().run_until_complete( - self._async_save_artifact(run_id, "brain_run", run.get("sessionId", ""), run_id, run.get("metadata", {}).get("planId", ""), run) + return self._run( + self._async_save_agent_run_artifact( + run_id, + run_id, + run.get("sessionId", ""), + "brain_run", + run.get("metadata", {}).get("planId", ""), + run, + generic_artifact_type="brain_run", + ) ) def get_brain_run(self, run_id: str) -> Optional[dict]: - return asyncio.get_event_loop().run_until_complete(self._async_get_artifact(run_id, "brain_run")) + typed = self._run(self._async_get_agent_run_artifact(run_id, "brain_run")) + if typed: + return typed + return self._run(self._async_get_artifact(run_id, "brain_run")) def list_brain_runs(self, session_id: str, limit: int = 20) -> list[dict]: - return asyncio.get_event_loop().run_until_complete(self._async_list_brain_runs(session_id, limit)) + return self._run(self._async_list_brain_runs(session_id, limit)) def save_brain_artifact_bundle(self, bundle: dict) -> str: bundle_id = bundle.get("bundleId") or f"brainbundle_{uuid.uuid4().hex[:12]}" run_id = (bundle.get("metadata") or {}).get("runId") or f"brain_{bundle.get('planId', '')}" - return asyncio.get_event_loop().run_until_complete( - self._async_save_artifact(bundle_id, "brain_artifact_bundle", bundle.get("sessionId", ""), run_id, bundle.get("planId", ""), bundle) + return self._run( + self._async_save_agent_run_artifact( + bundle_id, + run_id, + bundle.get("sessionId", ""), + "brain_artifact_bundle", + bundle.get("planId", ""), + bundle, + generic_artifact_type="brain_artifact_bundle", + ) ) def get_brain_artifact_bundle(self, bundle_id: str) -> Optional[dict]: - return asyncio.get_event_loop().run_until_complete(self._async_get_artifact(bundle_id, "brain_artifact_bundle")) + typed = self._run(self._async_get_agent_run_artifact(bundle_id, "brain_artifact_bundle")) + if typed: + return typed + return self._run(self._async_get_artifact(bundle_id, "brain_artifact_bundle")) def save_agent_run_artifact(self, run_id: str, session_id: str, artifact_type: str, artifact: dict) -> str: artifact_id = artifact.get("artifactId") or f"artifact_{artifact_type}_{uuid.uuid4().hex[:12]}" - return asyncio.get_event_loop().run_until_complete( - self._async_save_artifact(artifact_id, f"agent_run:{artifact_type}", session_id, run_id, artifact.get("planId", ""), artifact) + return self._run( + self._async_save_agent_run_artifact( + artifact_id, + run_id, + session_id, + artifact_type, + artifact.get("planId", ""), + artifact, + generic_artifact_type=f"agent_run:{artifact_type}", + ) ) def list_agent_run_artifacts(self, run_id: str, limit: int = 100) -> list[dict]: - return asyncio.get_event_loop().run_until_complete(self._async_list_run_artifacts(run_id, limit)) + rows = self._run(self._async_list_agent_run_artifacts(run_id, limit)) + if rows: + return rows + return self._run(self._async_list_run_artifacts(run_id, limit)) def save_repair_plan_v2(self, session_id: str, repair_plan: dict) -> str: artifact_id = repair_plan.get("repairId") or f"repair_{uuid.uuid4().hex[:12]}" - return asyncio.get_event_loop().run_until_complete( + return self._run( self._async_save_artifact(artifact_id, "repair_plan", session_id, repair_plan.get("runId", ""), repair_plan.get("planId", ""), repair_plan) ) def get_repair_plan_v2(self, repair_id: str) -> Optional[dict]: - return asyncio.get_event_loop().run_until_complete(self._async_get_artifact(repair_id, "repair_plan")) + return self._run(self._async_get_artifact(repair_id, "repair_plan")) def save_verification_report_v2(self, session_id: str, verification_report: dict) -> str: artifact_id = verification_report.get("verificationId") or verification_report.get("reportId") or f"verify_{uuid.uuid4().hex[:12]}" - return asyncio.get_event_loop().run_until_complete( - self._async_save_artifact(artifact_id, "verification_report", session_id, verification_report.get("runId", ""), verification_report.get("planId", ""), verification_report) + return self._run( + self._async_save_verification_report_v2(session_id, artifact_id, verification_report) ) def save_workbook_benchmark_fixture_result(self, run_id: str, result: dict) -> str: artifact_id = f"fixture_{result.get('fixtureId', 'unknown')}_{uuid.uuid4().hex[:8]}" - return asyncio.get_event_loop().run_until_complete( - self._async_save_artifact(artifact_id, "workbook_benchmark_fixture", "", run_id, "", result) + return self._run( + self._async_save_workbook_benchmark_fixture_result(run_id, artifact_id, result) ) def save_agent_run_step_artifact_v3(self, run_id: str, session_id: str, step_id: str, artifact: dict) -> str: artifact_id = artifact.get("artifactId") or f"stepartifact_{uuid.uuid4().hex[:12]}" artifact = {**artifact, "artifactId": artifact_id, "runId": run_id, "stepId": step_id} - return asyncio.get_event_loop().run_until_complete( - self._async_save_artifact(artifact_id, "agent_run_step_artifact_v3", session_id, run_id, artifact.get("planId", ""), artifact) + artifact_type = f"step:{artifact.get('artifactType', 'artifact')}" + return self._run( + self._async_save_agent_run_artifact( + artifact_id, + run_id, + session_id, + artifact_type, + artifact.get("planId", ""), + artifact, + generic_artifact_type="agent_run_step_artifact_v3", + ) ) def save_operation_diff_preview_v2(self, session_id: str, diff_preview: dict) -> str: artifact_id = diff_preview.get("diffPreviewId") or diff_preview.get("operationId") or f"diff_{uuid.uuid4().hex[:12]}" - return asyncio.get_event_loop().run_until_complete( + return self._run( self._async_save_artifact(artifact_id, "operation_diff_preview_v2", session_id, diff_preview.get("runId", ""), diff_preview.get("planId", ""), diff_preview) ) def save_source_cell_provenance_v2(self, session_id: str, provenance: dict) -> str: artifact_id = provenance.get("provenanceId") or f"prov_{uuid.uuid4().hex[:12]}" - return asyncio.get_event_loop().run_until_complete( + return self._run( self._async_save_artifact(artifact_id, "source_cell_provenance_v2", session_id, "", "", provenance) ) @@ -383,12 +3082,26 @@ def save_source_cell_provenance_v2(self, session_id: str, provenance: dict) -> s def save_operation_plan(self, plan: dict) -> None: plan_id = plan.get("id") or plan.get("plan_id") or f"plan_{uuid.uuid4().hex[:12]}" plan = {**plan, "id": plan_id} - asyncio.get_event_loop().run_until_complete( + self._run( self._async_save_artifact(plan_id, "operation_plan", plan.get("session_id", ""), "", plan_id, plan) ) + def compare_and_swap_operation_plan( + self, + plan_id: str, + expected_plan: dict, + updated_plan: dict, + ) -> bool: + return self._run( + self._async_compare_and_swap_operation_plan( + plan_id, + expected_plan, + updated_plan, + ) + ) + def get_operation_plan(self, plan_id: str) -> Optional[dict]: - artifact = asyncio.get_event_loop().run_until_complete(self._async_get_artifact(plan_id, "operation_plan")) + artifact = self._run(self._async_get_artifact(plan_id, "operation_plan")) if not artifact: return None return artifact.get("artifact") or artifact.get("payload") or artifact @@ -410,24 +3123,40 @@ def record_snapshot( label: str = "", ) -> str: snapshot_id = f"snap_{int(time.time() * 1000)}_{uuid.uuid4().hex[:6]}" + state = workbook_state or {} + state_hash = hashlib.sha256( + json.dumps(state, sort_keys=True, separators=(",", ":")).encode("utf-8") + ).hexdigest() payload = { "snapshot_id": snapshot_id, "session_id": session_id, "plan_id": plan_id, "label": label, - "workbook_state": workbook_state or {}, + "workbook_state": state, + "state_hash": state_hash, "created_at": time.time(), } - asyncio.get_event_loop().run_until_complete( + self._run( self._async_save_artifact(snapshot_id, "workbook_snapshot", session_id, "", plan_id or "", payload) ) return snapshot_id def get_snapshot(self, snapshot_id: str) -> Optional[dict]: - artifact = asyncio.get_event_loop().run_until_complete(self._async_get_artifact(snapshot_id, "workbook_snapshot")) + artifact = self._run(self._async_get_artifact(snapshot_id, "workbook_snapshot")) if not artifact: return None - return artifact.get("artifact") or artifact.get("payload") or artifact + data = artifact.get("artifact") or artifact.get("payload") or artifact + if not isinstance(data, dict): + return data + stored_hash = str(data.get("state_hash") or "") + if stored_hash: + recomputed = hashlib.sha256( + json.dumps(data.get("workbook_state") or {}, sort_keys=True, separators=(",", ":")).encode("utf-8") + ).hexdigest() + data["state_verified"] = stored_hash == recomputed + else: + data["state_verified"] = None + return data def record_audit_event( self, @@ -447,41 +3176,92 @@ def record_audit_event( "snapshot_id": snapshot_id, "created_at": time.time(), } - asyncio.get_event_loop().run_until_complete( + self._run( self._async_save_artifact(audit_id, f"audit:{event_type}", session_id, "", plan_id or "", record) ) return audit_id + def record_enterprise_admin_audit( + self, + event_type: str, + payload: Optional[dict] = None, + ) -> str: + principal = current_principal() + if principal.authenticated and not principal.is_admin: + raise OwnershipNotFoundError("Audit stream not found") + normalized_type = str(event_type or "").strip() + if not normalized_type.startswith("enterprise_admin."): + raise ValueError("Enterprise administrator audit type is required") + return self._run( + self._async_record_enterprise_admin_audit(normalized_type, payload or {}) + ) + + def list_enterprise_admin_audits(self, limit: int = 1000) -> list[dict]: + principal = current_principal() + if principal.authenticated and not principal.is_admin: + raise OwnershipNotFoundError("Audit stream not found") + return self._run(self._async_list_enterprise_admin_audits(limit)) + def export_audit_csv(self, session_id: str) -> str: + import csv + import io + # Lightweight exporter — sufficient parity for the CSV download route. - rows = asyncio.get_event_loop().run_until_complete( + rows = self._run( self._async_list_artifacts_for_session(session_id, limit=1000) ) - out = ["audit_id,event_type,plan_id,snapshot_id,created_at"] + out = io.StringIO() + writer = csv.writer(out) + writer.writerow([ + "audit_id", + "event_type", + "plan_id", + "snapshot_id", + "created_at", + "actor_user_id", + "actor_email", + "actor_role", + "actor_auth_source", + ]) for row in rows: artifact = row.get("artifact") or row.get("payload") or {} atype = str(row.get("artifact_type") or "") if not atype.startswith("audit:"): continue - out.append( - ",".join( - str(artifact.get(field, "")).replace(",", " ") - for field in ("audit_id", "event_type", "plan_id", "snapshot_id", "created_at") - ) - ) - return "\n".join(out) + "\n" + payload = artifact.get("payload", {}) if isinstance(artifact, dict) else {} + actor = payload.get("actor", {}) if isinstance(payload, dict) else {} + writer.writerow([ + artifact.get("audit_id", ""), + artifact.get("event_type", ""), + artifact.get("plan_id", ""), + artifact.get("snapshot_id", ""), + artifact.get("created_at", ""), + actor.get("user_id", ""), + actor.get("email", ""), + actor.get("role", ""), + actor.get("auth_source", ""), + ]) + return out.getvalue() def record_apply_attempt( self, *, session_id: str, plan_id: str, - snapshot_id: Optional[str], token_hash: str, - workbook_fingerprint: str, - status: str, + snapshot_id: str, + workbook_fingerprint: str = "", payload: Optional[dict] = None, + status: str = "approved", ) -> str: + latest = self.get_latest_apply_attempt(plan_id) + if ( + latest + and latest.get("token_hash") == token_hash + and latest.get("status") == status + and not latest.get("completed_at") + ): + return str(latest["attempt_id"]) attempt_id = f"apply_{uuid.uuid4().hex[:14]}" record = { "attempt_id": attempt_id, @@ -493,168 +3273,2476 @@ def record_apply_attempt( "status": status, "payload": payload or {}, "created_at": time.time(), + "completed_at": None, } - asyncio.get_event_loop().run_until_complete( + self._run( self._async_save_artifact(attempt_id, "apply_attempt", session_id, "", plan_id, record) ) return attempt_id - async def _async_list_artifacts_for_session(self, session_id: str, limit: int = 100) -> list[dict]: - pool = await self._get_pool() - async with pool.acquire() as conn: - rows = await conn.fetch( - "SELECT artifact_id, artifact_type, session_id, run_id, plan_id, payload " - "FROM flagship_artifacts WHERE session_id = $1 ORDER BY updated_at DESC LIMIT $2", - session_id, limit, + def get_latest_apply_attempt(self, plan_id: str) -> Optional[dict]: + row = self._run( + self._async_get_latest_artifact_by_plan("apply_attempt", plan_id) + ) + if not row: + return None + return row["payload"] + + def complete_apply_attempt( + self, + *, + plan_id: str, + token_hash: str, + status: str, + payload: Optional[dict] = None, + ) -> Optional[dict]: + row = self._run( + self._async_get_latest_artifact_by_plan("apply_attempt", plan_id) + ) + if not row: + return None + attempt = row["payload"] + if attempt.get("token_hash") != token_hash: + return None + if attempt.get("completed_at"): + attempt["token_consumed"] = True + return attempt + merged_payload = attempt.get("payload") or {} + merged_payload.update(payload or {}) + completed_at = time.time() + attempt.update( + { + "status": status, + "payload": merged_payload, + "completed_at": completed_at, + "token_consumed": True, + } + ) + self._run( + self._async_save_artifact( + row["artifact_id"], + "apply_attempt", + attempt.get("session_id", ""), + "", + plan_id, + attempt, ) - return [dict(r) for r in rows] + ) + return attempt - # ── Async implementations ──────────────────────────────────────── + def record_apply_result( + self, + session_id: str, + plan_id: str, + status: str, + payload: Optional[dict] = None, + ) -> str: + updated = self.update_operation_plan_status(plan_id, status) + audit_id = self.record_audit_event( + session_id=session_id, + plan_id=plan_id, + event_type=f"plan_{status}", + payload=payload or {}, + ) + if not updated: + logger.warning("apply_result_without_plan", plan_id=plan_id, status=status) + return audit_id - async def _async_get_or_create_session(self, session_id: str) -> dict: - pool = await self._get_pool() - now = time.time() - async with pool.acquire() as conn: - row = await conn.fetchrow( - "SELECT * FROM sessions WHERE session_id = $1", session_id - ) - if row: - await conn.execute( - "UPDATE sessions SET updated_at = $1 WHERE session_id = $2", - now, session_id, - ) - return dict(row) - else: - await conn.execute( - "INSERT INTO sessions (session_id, created_at, updated_at) VALUES ($1, $2, $3)", - session_id, now, now, - ) - return {"session_id": session_id, "created_at": now, "updated_at": now} + def get_timeline(self, session_id: str, limit: int = 50) -> dict: + rows = self._run( + self._async_list_artifacts_for_session(session_id, limit=max(int(limit), 0)) + ) + artifacts = [ + { + **row, + "payload": self._decode_payload(row.get("payload")), + } + for row in rows + ] - async def _async_list_sessions(self, limit: int = 20) -> list[dict]: - pool = await self._get_pool() - async with pool.acquire() as conn: - rows = await conn.fetch( - "SELECT s.*, COUNT(c.id) as message_count FROM sessions s " - "LEFT JOIN conversations c ON s.session_id = c.session_id " - "GROUP BY s.session_id ORDER BY s.updated_at DESC LIMIT $1", - limit, - ) - return [dict(r) for r in rows] + def by_type(artifact_type: str) -> list[dict]: + return [row["payload"] for row in artifacts if row.get("artifact_type") == artifact_type] - async def _async_add_message(self, session_id: str, role: str, content: str, - metadata: Optional[dict] = None): - await self._async_get_or_create_session(session_id) - pool = await self._get_pool() - now = time.time() - async with pool.acquire() as conn: - await conn.execute( - "INSERT INTO conversations (session_id, role, content, timestamp, metadata) " - "VALUES ($1, $2, $3, $4, $5)", - session_id, role, content, now, json.dumps(metadata or {}), - ) + audit_events = [ + row["payload"] + for row in artifacts + if str(row.get("artifact_type") or "").startswith("audit:") + ] + return { + "session_id": session_id, + "messages": [ + {"role": m.role, "content": m.content, "timestamp": m.timestamp, "metadata": m.metadata} + for m in self.get_conversation(session_id, 50) + ], + "plans": by_type("operation_plan"), + "snapshots": by_type("workbook_snapshot"), + "audit_events": audit_events, + "workbook_maps": by_type("workbook_map"), + "verification_runs": by_type("verification_run"), + "agent_runs": by_type("agent_run"), + "source_evidence_graphs": by_type("source_evidence_graph"), + "operation_simulations": by_type("operation_simulation") + by_type("operation_simulation_v2"), + "documents": by_type("document"), + "citations": self.get_citations(session_id), + } - async def _async_get_conversation(self, session_id: str, limit: int = 50) -> list[ConversationMessage]: + def delete_session(self, session_id: str) -> bool: + return self._run(self._async_delete_session(session_id)) + + def save_document( + self, + session_id: str, + filename: str, + content_type: str = "", + size_bytes: int = 0, + file_path: str = "", + summary: Optional[dict] = None, + parsed: Optional[dict] = None, + document_id: Optional[str] = None, + ) -> str: + doc_id = document_id or f"doc_{uuid.uuid4().hex[:12]}" + record = { + "document_id": doc_id, + "session_id": session_id, + "filename": filename, + "content_type": content_type, + "size_bytes": size_bytes, + "file_path": file_path, + "summary": summary or {}, + "parsed": parsed or {}, + "created_at": time.time(), + } + return self._run( + self._async_save_artifact(doc_id, "document", session_id, "", "", record) + ) + + def get_document(self, document_id: str) -> Optional[dict]: + return self._run(self._async_get_artifact(document_id, "document")) + + def list_documents(self, session_id: str, limit: int = 20) -> list[dict]: + return self._run(self._async_list_documents(session_id, limit)) + + def save_citations( + self, + session_id: str, + citations_or_plan_id=None, + citations: Optional[list[dict]] = None, + *, + plan_id: Optional[str] = None, + ) -> int: + if citations is None and isinstance(citations_or_plan_id, list): + citation_rows = citations_or_plan_id + resolved_plan_id = plan_id or "" + else: + citation_rows = citations or [] + resolved_plan_id = str(plan_id or citations_or_plan_id or "") + if not citation_rows: + return 0 + return self._run( + self._async_save_citations(session_id, resolved_plan_id, citation_rows) + ) + + def get_citations( + self, + session_id: str, + plan_id: Optional[str] = None, + limit: int = 200, + ) -> list[dict]: + return self._run( + self._async_list_citations(session_id, plan_id, limit) + ) + + def save_benchmark_run(self, run: dict) -> str: + run_id = run.get("run_id") or run.get("runId") or f"benchmark_{uuid.uuid4().hex[:12]}" + payload = {**run, "run_id": run_id} + return self._run( + self._async_save_artifact(run_id, "benchmark_run", "", run_id, "", payload) + ) + + def list_benchmark_runs(self, limit: int = 20) -> list[dict]: + rows = self._run(self._async_list_artifacts_by_prefix("benchmark_run", limit)) + return [row.get("payload") or row.get("artifact") or row for row in rows] + + # ── Enterprise admin parity ────────────────────────────────────── + + @staticmethod + def _enterprise_user_artifact_id(user_id: str) -> str: + return f"enterprise_user:{user_id}" + + @staticmethod + def _enterprise_team_artifact_id(team_id: str) -> str: + return f"enterprise_team:{team_id}" + + @staticmethod + def _enterprise_api_key_artifact_id(key_id: str) -> str: + return f"enterprise_api_key:{key_id}" + + @staticmethod + def _enterprise_config_artifact_id(key: str) -> str: + return f"enterprise_config:{key}" + + def upsert_enterprise_user( + self, + user_id: str, + email: str, + name: str, + role: str, + status: Optional[str] = None, + mfa_enabled: Optional[bool] = None, + metadata: Optional[dict] = None, + oidc_issuer: Optional[str] = None, + oidc_subject: Optional[str] = None, + ) -> dict: + try: + return self._run( + self._async_upsert_enterprise_user( + user_id=user_id, + email=email, + name=name, + role=role, + status=status, + mfa_enabled=mfa_enabled, + metadata=metadata, + oidc_issuer=oidc_issuer, + oidc_subject=oidc_subject, + ) + ) + except Exception as exc: + if "idx_flagship_enterprise_oidc_identity" in str(exc) or exc.__class__.__name__ == "UniqueViolationError": + raise ValueError("OIDC identity is already bound to another enterprise user") from exc + raise + + def get_enterprise_user(self, user_id: str) -> Optional[dict]: + return self._run(self._async_get_artifact(self._enterprise_user_artifact_id(user_id), "enterprise_user")) + + def get_enterprise_user_by_email(self, email: str) -> Optional[dict]: + target = email.lower() + for user in self.list_enterprise_users(limit=10000): + if str(user.get("email", "")).lower() == target: + return user + return None + + def get_enterprise_user_by_oidc_identity(self, issuer: str, subject: str) -> Optional[dict]: + return self._run( + self._async_get_enterprise_user_by_oidc_identity( + str(issuer).strip(), + str(subject).strip(), + ) + ) + + def list_enterprise_users(self, limit: int = 100) -> list[dict]: + rows = self._run(self._async_list_artifacts_by_prefix("enterprise_user", limit)) + return [row.get("payload") or row.get("artifact") or row for row in rows] + + def create_enterprise_team(self, name: str, metadata: Optional[dict] = None, team_id: Optional[str] = None) -> dict: + now = time.time() + tid = team_id or f"team_{uuid.uuid4().hex[:10]}" + existing = self.get_enterprise_team(tid) or {} + payload = { + **existing, + "team_id": tid, + "name": name, + "metadata": metadata or {}, + "members": existing.get("members", []), + "created_at": existing.get("created_at", now), + "updated_at": now, + } + self._run(self._async_save_artifact(self._enterprise_team_artifact_id(tid), "enterprise_team", "", "", "", payload)) + return payload + + def get_enterprise_team(self, team_id: str) -> Optional[dict]: + team = self._run(self._async_get_artifact(self._enterprise_team_artifact_id(team_id), "enterprise_team")) + if not team: + return None + enriched_members = [] + for member in team.get("members", []): + user = self.get_enterprise_user(member.get("user_id", "")) or {} + enriched_members.append( + { + **member, + "email": user.get("email"), + "name": user.get("name"), + "status": user.get("status"), + } + ) + return {**team, "members": enriched_members} + + def list_enterprise_teams(self, limit: int = 100) -> list[dict]: + rows = self._run(self._async_list_artifacts_by_prefix("enterprise_team", limit)) + teams = [row.get("payload") or row.get("artifact") or row for row in rows] + return [self.get_enterprise_team(team.get("team_id", "")) or team for team in teams] + + def add_enterprise_team_member(self, team_id: str, user_id: str, role: str = "member") -> bool: + team = self.get_enterprise_team(team_id) + user = self.get_enterprise_user(user_id) + if not team or not user: + return False + members = [ + member + for member in team.get("members", []) + if member.get("user_id") != user_id + ] + members.append({"team_id": team_id, "user_id": user_id, "role": role, "joined_at": time.time()}) + payload = { + **team, + "members": members, + "updated_at": time.time(), + } + self._run(self._async_save_artifact(self._enterprise_team_artifact_id(team_id), "enterprise_team", "", "", "", payload)) + return True + + def create_enterprise_api_key(self, user_id: str, label: str = "") -> dict: + if not self.get_enterprise_user(user_id): + raise ValueError(f"Unknown enterprise user: {user_id}") + raw_key = f"fai_{secrets.token_urlsafe(24)}" + key_id = f"key_{uuid.uuid4().hex[:10]}" + payload = { + "key_id": key_id, + "user_id": user_id, + "key_hash": _hash_enterprise_api_key(raw_key), + "label": label, + "active": True, + "created_at": time.time(), + "last_used_at": None, + } + self._run(self._async_save_artifact(self._enterprise_api_key_artifact_id(key_id), "enterprise_api_key", "", "", "", payload)) + return {"key_id": key_id, "user_id": user_id, "api_key": raw_key, "label": label} + + def revoke_enterprise_api_key(self, key_id: str) -> bool: + rows = self._run(self._async_list_artifacts_by_prefix("enterprise_api_key", 10000)) + for row in rows: + payload = row.get("payload") or row.get("artifact") or row + if payload.get("key_id") != key_id: + continue + updated = {**payload, "active": False, "revoked_at": time.time()} + self._run( + self._async_save_artifact( + self._enterprise_api_key_artifact_id(key_id), + "enterprise_api_key", + "", + "", + "", + updated, + ) + ) + return True + return False + + def get_enterprise_user_by_api_key(self, api_key: str) -> Optional[dict]: + candidates = _enterprise_api_key_candidates(api_key) + preferred_hash = candidates[0] + for row in self._run(self._async_list_artifacts_by_prefix("enterprise_api_key", 10000)): + payload = row.get("payload") or row.get("artifact") or row + if not payload.get("active", True) or payload.get("key_hash") not in candidates: + continue + user = self.get_enterprise_user(payload.get("user_id", "")) + if not _enterprise_user_allows_api_key(user, require_mfa=self._enterprise_mfa_required()): + return None + updated = {**payload, "key_hash": preferred_hash, "last_used_at": time.time()} + self._run( + self._async_save_artifact( + self._enterprise_api_key_artifact_id(updated["key_id"]), + "enterprise_api_key", + "", + "", + "", + updated, + ) + ) + return user + return None + + def list_enterprise_api_keys(self, user_id: Optional[str] = None) -> list[dict]: + rows = self._run(self._async_list_artifacts_by_prefix("enterprise_api_key", 10000)) + keys = [] + for row in rows: + payload = row.get("payload") or row.get("artifact") or row + if user_id and payload.get("user_id") != user_id: + continue + keys.append( + { + "key_id": payload.get("key_id"), + "user_id": payload.get("user_id"), + "label": payload.get("label", ""), + "active": bool(payload.get("active", True)), + "created_at": payload.get("created_at"), + "last_used_at": payload.get("last_used_at"), + } + ) + return keys + + def set_enterprise_config(self, key: str, value: dict) -> dict: + payload = {"config_key": key, "config": value, "updated_at": time.time()} + self._run(self._async_save_artifact(self._enterprise_config_artifact_id(key), "enterprise_config", "", "", "", payload)) + return payload + + def get_enterprise_config(self, key: str) -> dict: + return self._run(self._async_get_artifact(self._enterprise_config_artifact_id(key), "enterprise_config")) or {} + + def _enterprise_mfa_required(self) -> bool: + for key in ("security_policy", "sso", "provider_policy"): + stored = self.get_enterprise_config(key) + config = stored.get("config") if isinstance(stored, dict) else {} + if isinstance(config, dict) and (config.get("require_mfa") or config.get("requireMfa")): + return True + return False + + def enterprise_dashboard(self) -> dict: + return { + "users": self._run(self._async_count_artifacts_by_prefix("enterprise_user")), + "teams": self._run(self._async_count_artifacts_by_prefix("enterprise_team")), + "documents": self._run(self._async_count_artifacts_by_prefix("document")), + "operation_plans": self._run(self._async_count_artifacts_by_prefix("operation_plan")), + "audit_events": self._run(self._async_count_artifacts_by_prefix("audit:")), + "benchmark_runs": self._run(self._async_count_artifacts_by_prefix("benchmark_run")), + } + + def purge_sessions_before(self, cutoff_timestamp: float) -> int: + return self._run(self._async_purge_sessions_before(cutoff_timestamp)) + + def save_plan_revision( + self, + plan_id: str, + revision_type: str, + reason: str, + plan: dict, + ) -> str: + resolved_session_id = str(plan.get("session_id") or plan.get("sessionId") or "") + resolved_plan_id = str(plan_id) + payload: dict[str, Any] = { + "plan_id": resolved_plan_id, + "revision_type": revision_type, + "reason": reason, + "plan": plan, + } + revision_id = str(payload.get("revision_id") or payload.get("revisionId") or f"rev_{uuid.uuid4().hex[:12]}") + payload = {**payload, "revision_id": revision_id, "created_at": time.time()} + return self._run( + self._async_save_artifact(revision_id, "plan_revision", resolved_session_id, "", resolved_plan_id, payload) + ) + + def save_verification_run(self, session_id: str, plan_id: str, status: str, report: dict) -> str: + verification_id = report.get("verification_id") or report.get("verificationId") or f"verify_{uuid.uuid4().hex[:12]}" + payload = { + "verification_id": verification_id, + "session_id": session_id, + "plan_id": plan_id, + "status": status, + "report": report, + "created_at": time.time(), + } + return self._run( + self._async_save_artifact(verification_id, "verification_run", session_id, "", plan_id, payload) + ) + + def save_operation_audits(self, session_id: str, plan_id: str, rows: list[dict]) -> int: + if not rows: + return 0 + return self._run( + self._async_save_operation_audits(session_id, plan_id, rows) + ) + + async def _async_list_artifacts_for_session(self, session_id: str, limit: int = 100) -> list[dict]: + pool = await self._get_pool() + async with pool.acquire() as conn: + async with conn.transaction(): + if not await self._async_session_visible_in_transaction(conn, session_id): + return [] + rows = await conn.fetch( + "SELECT artifact_id, artifact_type, session_id, run_id, plan_id, payload " + "FROM flagship_artifacts WHERE session_id = $1 " + "ORDER BY updated_at DESC LIMIT $2", + session_id, limit, + ) + return [dict(r) for r in rows] + + # ── Async implementations ──────────────────────────────────────── + + async def _async_get_or_create_session( + self, + session_id: str, + owner_user_id: Optional[str] = None, + ) -> dict: + from backend.services.lifecycle import get_lifecycle_coordinator + + if get_lifecycle_coordinator().writes_revoked(session_id): + raise OwnershipNotFoundError("Session not found") + pool = await self._get_pool() + principal = current_principal() + if principal.authenticated and not principal.is_admin: + if not principal.user_id: + raise OwnershipNotFoundError("Session not found") + owner_user_id = principal.user_id + elif owner_user_id is None and principal.authenticated: + owner_user_id = principal.user_id or None + now = time.time() + async with pool.acquire() as conn: + async with conn.transaction(): + await conn.execute( + "SELECT pg_advisory_xact_lock(hashtextextended($1, 0))", + session_id, + ) + lifecycle_row = await conn.fetchrow( + "SELECT state FROM session_lifecycle WHERE session_id = $1 FOR UPDATE", + session_id, + ) + if lifecycle_row and str(lifecycle_row["state"] or "") != "active": + raise OwnershipNotFoundError("Session not found") + + row = await conn.fetchrow( + "SELECT * FROM sessions WHERE session_id = $1 FOR UPDATE", + session_id, + ) + if row: + existing_owner = str(row["owner_user_id"] or "") + if principal.authenticated and not principal.is_admin and existing_owner != principal.user_id: + raise OwnershipNotFoundError("Session not found") + if owner_user_id and existing_owner and existing_owner != owner_user_id and not principal.is_admin: + raise OwnershipNotFoundError("Session not found") + if not lifecycle_row: + await conn.execute( + "INSERT INTO session_lifecycle " + "(session_id, state, receipt_id, started_at, completed_at, errors, updated_at) " + "VALUES ($1, 'active', '', 0, 0, '[]'::jsonb, $2)", + session_id, + now, + ) + await conn.execute( + "UPDATE sessions SET updated_at = $1 WHERE session_id = $2", + now, + session_id, + ) + return {**dict(row), "updated_at": now} + + if lifecycle_row: + # An ACTIVE row without its session can only be a partial + # legacy migration. Explicit creation repairs it under the + # same per-session transaction fence. + await conn.execute( + "UPDATE session_lifecycle SET updated_at = $1 WHERE session_id = $2", + now, + session_id, + ) + else: + await conn.execute( + "INSERT INTO session_lifecycle " + "(session_id, state, receipt_id, started_at, completed_at, errors, updated_at) " + "VALUES ($1, 'active', '', 0, 0, '[]'::jsonb, $2)", + session_id, + now, + ) + await conn.execute( + "INSERT INTO sessions (session_id, owner_user_id, created_at, updated_at) " + "VALUES ($1, $2, $3, $4)", + session_id, + owner_user_id, + now, + now, + ) + return { + "session_id": session_id, + "owner_user_id": owner_user_id, + "created_at": now, + "updated_at": now, + } + + async def _async_get_session_owner(self, session_id: str) -> Optional[str]: + pool = await self._get_pool() + async with pool.acquire() as conn: + row = await conn.fetchrow( + "SELECT owner_user_id FROM sessions WHERE session_id = $1", + session_id, + ) + return (str(row["owner_user_id"] or "") or None) if row else None + + async def _async_session_exists(self, session_id: str) -> bool: + pool = await self._get_pool() + async with pool.acquire() as conn: + value = await conn.fetchval("SELECT 1 FROM sessions WHERE session_id = $1", session_id) + return value is not None + + async def _async_get_session_lifecycle_state(self, session_id: str) -> Optional[str]: + pool = await self._get_pool() + async with pool.acquire() as conn: + value = await conn.fetchval( + "SELECT state FROM session_lifecycle WHERE session_id = $1", + session_id, + ) + return str(value) if value is not None else None + + async def _async_get_session_lifecycle_record(self, session_id: str) -> Optional[dict]: + pool = await self._get_pool() + async with pool.acquire() as conn: + row = await conn.fetchrow( + "SELECT session_id, state, receipt_id, started_at, completed_at, errors, updated_at " + "FROM session_lifecycle WHERE session_id = $1", + session_id, + ) + if not row: + return None + record = dict(row) + raw_errors = record.get("errors") + record["errors"] = json.loads(raw_errors) if isinstance(raw_errors, str) else list(raw_errors or []) + return record + + async def _async_session_deletion_verified(self, session_id: str) -> bool: + pool = await self._get_pool() + async with pool.acquire() as conn: + return not await self._async_session_has_durable_rows(conn, session_id) + + async def _async_set_session_lifecycle_state( + self, + session_id: str, + state: str, + *, + receipt_id: str, + started_at: float, + completed_at: float = 0.0, + errors: Optional[list[str]] = None, + ) -> None: + normalized_state = str(state or "").strip().lower() + if normalized_state not in {"purging", "verified", "failed"}: + raise ValueError("Lifecycle state must be purging, verified, or failed") + normalized_session_id = str(session_id or "").strip() + normalized_receipt_id = str(receipt_id or "").strip() + if not normalized_session_id or not normalized_receipt_id: + raise ValueError("session_id and receipt_id are required") + + pool = await self._get_pool() + now = time.time() + async with pool.acquire() as conn: + async with conn.transaction(): + await conn.execute( + "SELECT pg_advisory_xact_lock(hashtextextended($1, 0))", + normalized_session_id, + ) + current = await conn.fetchrow( + "SELECT state, receipt_id FROM session_lifecycle " + "WHERE session_id = $1 FOR UPDATE", + normalized_session_id, + ) + current_state = str(current["state"] or "") if current else "" + current_receipt_id = str(current["receipt_id"] or "") if current else "" + + if normalized_state == "purging": + if current_state == "verified": + raise OwnershipNotFoundError("Session not found") + if ( + current_state == "purging" + and current_receipt_id + and current_receipt_id != normalized_receipt_id + ): + raise RuntimeError("Session purge is already in progress") + session_present = await conn.fetchval( + "SELECT 1 FROM sessions WHERE session_id = $1 FOR UPDATE", + normalized_session_id, + ) + if session_present is None and not current: + raise OwnershipNotFoundError("Session not found") + await conn.execute( + """ + INSERT INTO session_lifecycle + (session_id, state, receipt_id, started_at, completed_at, errors, updated_at) + VALUES ($1, 'purging', $2, $3, 0, '[]'::jsonb, $4) + ON CONFLICT(session_id) DO UPDATE SET + state = 'purging', + receipt_id = EXCLUDED.receipt_id, + started_at = EXCLUDED.started_at, + completed_at = 0, + errors = '[]'::jsonb, + updated_at = EXCLUDED.updated_at + """, + normalized_session_id, + normalized_receipt_id, + float(started_at), + now, + ) + return + + if current_state != "purging" or current_receipt_id != normalized_receipt_id: + raise RuntimeError("Lifecycle completion does not match the active purge receipt") + if normalized_state == "verified" and await self._async_session_has_durable_rows( + conn, + normalized_session_id, + ): + raise RuntimeError("Cannot verify deletion while session-bound rows remain") + result = await conn.execute( + "UPDATE session_lifecycle SET state = $1, completed_at = $2, " + "errors = $3::jsonb, updated_at = $4 " + "WHERE session_id = $5 AND state = 'purging' AND receipt_id = $6", + normalized_state, + float(completed_at), + json.dumps(errors or []), + now, + normalized_session_id, + normalized_receipt_id, + ) + if self._row_count(result) != 1: + raise RuntimeError("Lifecycle completion compare-and-set failed") + + async def _async_session_has_durable_rows(self, conn, session_id: str) -> bool: + global_artifact_types = sorted(POSTGRES_GLOBAL_ARTIFACT_TYPES) + clauses = [ + "SELECT 1 FROM sessions WHERE session_id = $1", + "SELECT 1 FROM conversations WHERE session_id = $1", + "SELECT 1 FROM task_history WHERE session_id = $1", + "SELECT 1 FROM flagship_artifacts WHERE session_id = $1 " + "AND artifact_type <> ALL($2::text[])", + ] + clauses.extend( + f"SELECT 1 FROM {table_name} WHERE session_id = $1" + for table_name in POSTGRES_SESSION_LIFECYCLE_TABLES + if table_name not in {"sessions", "conversations", "task_history", "flagship_artifacts"} + ) + return bool( + await conn.fetchval( + "SELECT EXISTS (" + " UNION ALL ".join(clauses) + ")", + session_id, + global_artifact_types, + ) + ) + + async def _async_require_session_write(self, conn, session_id: str) -> None: + await conn.execute( + "SELECT pg_advisory_xact_lock(hashtextextended($1, 0))", + session_id, + ) + row = await conn.fetchrow( + "SELECT s.owner_user_id, lifecycle.state AS lifecycle_state " + "FROM sessions AS s " + "JOIN session_lifecycle AS lifecycle ON lifecycle.session_id = s.session_id " + "WHERE s.session_id = $1 FOR UPDATE OF s, lifecycle", + session_id, + ) + if not row or str(row["lifecycle_state"] or "") != "active": + raise OwnershipNotFoundError("Resource not found") + principal = current_principal() + existing_owner = str(row["owner_user_id"] or "") + if ( + principal.authenticated + and not principal.is_admin + and (not principal.user_id or existing_owner != principal.user_id) + ): + raise OwnershipNotFoundError("Resource not found") + + async def _async_list_session_ids_before(self, cutoff_timestamp: float) -> list[str]: + pool = await self._get_pool() + async with pool.acquire() as conn: + rows = await conn.fetch( + "SELECT session_id FROM sessions WHERE updated_at < $1 ORDER BY updated_at", + cutoff_timestamp, + ) + return [str(row["session_id"]) for row in rows] + + async def _async_list_document_file_paths(self, session_id: str) -> list[str]: + pool = await self._get_pool() + async with pool.acquire() as conn: + rows = await conn.fetch( + "SELECT payload->>'file_path' AS file_path FROM flagship_artifacts " + "WHERE artifact_type = 'document' AND session_id = $1", + session_id, + ) + return [str(row["file_path"]) for row in rows if row["file_path"]] + + async def _async_session_is_visible(self, session_id: str) -> bool: + pool = await self._get_pool() + async with pool.acquire() as conn: + async with conn.transaction(): + return await self._async_session_visible_in_transaction(conn, session_id) + + async def _async_session_visible_in_transaction(self, conn, session_id: str) -> bool: + from backend.services.lifecycle import get_lifecycle_coordinator + + if get_lifecycle_coordinator().reads_revoked(session_id): + return False + await conn.execute( + "SELECT pg_advisory_xact_lock(hashtextextended($1, 0))", + session_id, + ) + row = await conn.fetchrow( + "SELECT s.owner_user_id, lifecycle.state AS lifecycle_state " + "FROM sessions AS s JOIN session_lifecycle AS lifecycle " + "ON lifecycle.session_id = s.session_id WHERE s.session_id = $1 " + "FOR SHARE OF s, lifecycle", + session_id, + ) + if not row or str(row["lifecycle_state"] or "") != "active": + return False + principal = current_principal() + if not principal.authenticated or principal.is_admin: + return True + if not principal.user_id: + return False + return str(row["owner_user_id"] or "") == principal.user_id + + async def _async_list_sessions( + self, + limit: int = 20, + owner_user_id: Optional[str] = None, + ) -> list[dict]: + pool = await self._get_pool() + async with pool.acquire() as conn: + if owner_user_id: + rows = await conn.fetch( + "SELECT s.*, COUNT(c.id) as message_count FROM sessions s " + "JOIN session_lifecycle lifecycle ON lifecycle.session_id = s.session_id " + "LEFT JOIN conversations c ON s.session_id = c.session_id " + "WHERE s.owner_user_id = $1 AND lifecycle.state = 'active' " + "GROUP BY s.session_id ORDER BY s.updated_at DESC LIMIT $2", + owner_user_id, + limit, + ) + else: + rows = await conn.fetch( + "SELECT s.*, COUNT(c.id) as message_count FROM sessions s " + "JOIN session_lifecycle lifecycle ON lifecycle.session_id = s.session_id " + "LEFT JOIN conversations c ON s.session_id = c.session_id " + "WHERE lifecycle.state = 'active' " + "GROUP BY s.session_id ORDER BY s.updated_at DESC LIMIT $1", + limit, + ) + return [dict(r) for r in rows] + + async def _async_add_message(self, session_id: str, role: str, content: str, + metadata: Optional[dict] = None): + pool = await self._get_pool() + now = time.time() + async with pool.acquire() as conn: + async with conn.transaction(): + await self._async_require_session_write(conn, session_id) + await conn.execute( + "INSERT INTO conversations (session_id, role, content, timestamp, metadata) " + "VALUES ($1, $2, $3, $4, $5)", + session_id, + role, + content, + now, + json.dumps(metadata or {}), + ) + + async def _async_get_conversation(self, session_id: str, limit: int = 50) -> list[ConversationMessage]: + pool = await self._get_pool() + async with pool.acquire() as conn: + async with conn.transaction(): + if not await self._async_session_visible_in_transaction(conn, session_id): + return [] + rows = await conn.fetch( + "SELECT * FROM conversations WHERE session_id = $1 " + "ORDER BY timestamp DESC LIMIT $2", + session_id, limit, + ) + rows = list(reversed(rows)) + return [ + ConversationMessage( + role=r["role"], + content=r["content"], + timestamp=r["timestamp"], + metadata=json.loads(r["metadata"] or "{}") if isinstance(r["metadata"], str) else (r["metadata"] or {}), + ) + for r in rows + ] + + async def _async_record_task(self, session_id, task_id, prompt, mode, + operations, summary, plan, provider, model, tokens): + pool = await self._get_pool() + now = time.time() + async with pool.acquire() as conn: + async with conn.transaction(): + await self._async_require_session_write(conn, session_id) + await conn.execute( + "INSERT INTO task_history " + "(session_id, task_id, prompt, mode, operations, summary, plan, provider, model, tokens_used, timestamp) " + "VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11)", + session_id, + task_id, + prompt, + mode, + json.dumps(operations), + summary, + json.dumps(plan or []), + provider, + model, + json.dumps(tokens or {}), + now, + ) + + async def _async_get_task_history(self, session_id: str, limit: int = 20) -> list[dict]: + pool = await self._get_pool() + async with pool.acquire() as conn: + async with conn.transaction(): + if not await self._async_session_visible_in_transaction(conn, session_id): + return [] + rows = await conn.fetch( + "SELECT * FROM task_history WHERE session_id = $1 " + "ORDER BY timestamp DESC LIMIT $2", + session_id, limit, + ) + results = [] + for r in rows: + d = dict(r) + for field in ("operations", "plan", "tokens_used"): + if isinstance(d.get(field), str): + d[field] = json.loads(d[field]) + results.append(d) + return results + + async def _async_get_total_tokens(self, session_id: Optional[str] = None) -> dict: + pool = await self._get_pool() + async with pool.acquire() as conn: + if session_id: + async with conn.transaction(): + if not await self._async_session_visible_in_transaction(conn, session_id): + return {"prompt_tokens": 0, "completion_tokens": 0, "total_tokens": 0} + rows = await conn.fetch( + "SELECT tokens_used FROM task_history WHERE session_id = $1", + session_id, + ) + else: + principal = current_principal() + if principal.authenticated and not principal.is_admin: + if not principal.user_id: + rows = [] + else: + rows = await conn.fetch( + "SELECT history.tokens_used FROM task_history history " + "JOIN sessions session ON session.session_id = history.session_id " + "JOIN session_lifecycle lifecycle ON lifecycle.session_id = history.session_id " + "WHERE lifecycle.state = 'active' AND session.owner_user_id = $1", + principal.user_id, + ) + else: + rows = await conn.fetch( + "SELECT history.tokens_used FROM task_history history " + "JOIN session_lifecycle lifecycle ON lifecycle.session_id = history.session_id " + "WHERE lifecycle.state = 'active'" + ) + + total = {"prompt_tokens": 0, "completion_tokens": 0} + for r in rows: + tokens = r["tokens_used"] + if isinstance(tokens, str): + tokens = json.loads(tokens) + total["prompt_tokens"] += tokens.get("prompt_tokens", 0) + total["completion_tokens"] += tokens.get("completion_tokens", 0) + total["total_tokens"] = total["prompt_tokens"] + total["completion_tokens"] + return total + + async def _async_compare_and_swap_operation_plan( + self, + plan_id: str, + expected_plan: dict, + updated_plan: dict, + ) -> bool: + """Replace one exact plan revision under the cross-process session fence.""" + expected_id = str(expected_plan.get("id") or expected_plan.get("plan_id") or "") + updated_id = str(updated_plan.get("id") or updated_plan.get("plan_id") or "") + expected_session_id = str(expected_plan.get("session_id") or "") + updated_session_id = str(updated_plan.get("session_id") or "") + if expected_id != plan_id or updated_id != plan_id: + raise ValueError("Operation plan identity is immutable") + if not expected_session_id or updated_session_id != expected_session_id: + raise ValueError("Operation plan session ownership is immutable") + + from backend.services.lifecycle import get_lifecycle_coordinator + + if get_lifecycle_coordinator().writes_revoked(expected_session_id): + raise OwnershipNotFoundError("Resource not found") + pool = await self._get_pool() + async with pool.acquire() as conn: + async with conn.transaction(): + await self._async_require_session_write(conn, expected_session_id) + row = await conn.fetchrow( + "SELECT session_id, payload FROM flagship_artifacts " + "WHERE artifact_id = $1 AND artifact_type = 'operation_plan' FOR UPDATE", + plan_id, + ) + if not row: + return False + if str(row["session_id"] or "") != expected_session_id: + raise ValueError("Operation plan session ownership is immutable") + current_payload = row["payload"] + if isinstance(current_payload, str): + current_payload = json.loads(current_payload) + if current_payload != expected_plan: + return False + updated = await conn.fetchrow( + "UPDATE flagship_artifacts SET payload = $1::jsonb, updated_at = $2 " + "WHERE artifact_id = $3 AND artifact_type = 'operation_plan' " + "AND session_id = $4 AND payload = $5::jsonb RETURNING artifact_id", + json.dumps(updated_plan), + time.time(), + plan_id, + expected_session_id, + json.dumps(expected_plan), + ) + return updated is not None + + async def _async_save_artifact( + self, + artifact_id: str, + artifact_type: str, + session_id: str, + run_id: str, + plan_id: str, + payload: dict, + ) -> str: + pool = await self._get_pool() + async with pool.acquire() as conn: + async with conn.transaction(): + await self._async_save_artifact_in_transaction( + conn, + artifact_id, + artifact_type, + session_id, + run_id, + plan_id, + payload, + ) + return artifact_id + + async def _async_save_artifact_in_transaction( + self, + conn, + artifact_id: str, + artifact_type: str, + session_id: str, + run_id: str, + plan_id: str, + payload: dict, + *, + now: Optional[float] = None, + ) -> str: + if not session_id and _artifact_type_requires_session(artifact_type): + raise OwnershipNotFoundError("Resource not found") + is_session_scoped = _artifact_is_session_scoped(artifact_type, session_id) + if is_session_scoped: + from backend.services.lifecycle import get_lifecycle_coordinator + + if get_lifecycle_coordinator().writes_revoked(session_id): + raise OwnershipNotFoundError("Resource not found") + await self._async_require_session_write(conn, session_id) + existing = await conn.fetchrow( + "SELECT session_id FROM flagship_artifacts " + "WHERE artifact_id = $1 AND artifact_type = $2 FOR UPDATE", + artifact_id, + artifact_type, + ) + existing_session_id = str(existing["session_id"] or "") if existing else "" + if existing_session_id and existing_session_id != str(session_id or ""): + raise ValueError("Artifact session ownership is immutable") + timestamp = float(now if now is not None else time.time()) + await conn.execute( + """ + INSERT INTO flagship_artifacts + (artifact_id, artifact_type, session_id, run_id, plan_id, + payload, created_at, updated_at) + VALUES ($1, $2, $3, $4, $5, $6, $7, $8) + ON CONFLICT(artifact_id, artifact_type) DO UPDATE SET + session_id = EXCLUDED.session_id, + run_id = EXCLUDED.run_id, + plan_id = EXCLUDED.plan_id, + payload = EXCLUDED.payload, + updated_at = EXCLUDED.updated_at + """, + artifact_id, + artifact_type, + session_id, + run_id, + plan_id, + json.dumps(payload), + timestamp, + timestamp, + ) + return artifact_id + + @asynccontextmanager + async def _async_projected_artifact_transaction( + self, + artifact_id: str, + artifact_type: str, + session_id: str, + run_id: str, + plan_id: str, + payload: dict, + *, + now: Optional[float] = None, + ) -> AsyncIterator[Any]: + """Fence a flagship row and its typed projection in one transaction.""" + + pool = await self._get_pool() + async with pool.acquire() as conn: + async with conn.transaction(): + await self._async_save_artifact_in_transaction( + conn, + artifact_id, + artifact_type, + session_id, + run_id, + plan_id, + payload, + now=now, + ) + yield conn + + async def _async_get_artifact(self, artifact_id: str, artifact_type: str) -> Optional[dict]: + pool = await self._get_pool() + async with pool.acquire() as conn: + async with conn.transaction(): + row = await conn.fetchrow( + "SELECT session_id, payload FROM flagship_artifacts " + "WHERE artifact_id = $1 AND artifact_type = $2", + artifact_id, + artifact_type, + ) + if not row: + return None + session_id = str(row["session_id"] or "") + is_session_bound = _artifact_is_session_scoped(artifact_type, session_id) + if is_session_bound and not await self._async_session_visible_in_transaction( + conn, session_id + ): + return None + payload = row["payload"] + return json.loads(payload) if isinstance(payload, str) else payload + + async def _async_get_enterprise_user_by_oidc_identity( + self, + issuer: str, + subject: str, + ) -> Optional[dict]: + if not issuer or not subject: + return None + pool = await self._get_pool() + async with pool.acquire() as conn: + row = await conn.fetchrow( + "SELECT payload FROM flagship_artifacts " + "WHERE artifact_type = 'enterprise_user' " + "AND payload->>'oidc_issuer' = $1 " + "AND payload->>'oidc_subject' = $2 " + "AND LOWER(COALESCE(payload->>'status', '')) = 'active' " + "LIMIT 1", + issuer, + subject, + ) + if not row: + return None + payload = row["payload"] + decoded = json.loads(payload) if isinstance(payload, str) else payload + return decoded if isinstance(decoded, dict) else None + + async def _async_record_enterprise_admin_audit( + self, + event_type: str, + payload: dict, + ) -> str: + principal = current_principal() + if principal.authenticated and not principal.is_admin: + raise OwnershipNotFoundError("Audit stream not found") + audit_id = f"admin_audit_{uuid.uuid4().hex[:18]}" + pool = await self._get_pool() + async with pool.acquire() as conn: + await conn.execute( + "INSERT INTO enterprise_admin_audits " + "(audit_id, event_type, payload, created_at) VALUES ($1, $2, $3, $4)", + audit_id, + event_type, + json.dumps(payload), + time.time(), + ) + return audit_id + + async def _async_list_enterprise_admin_audits(self, limit: int = 1000) -> list[dict]: + principal = current_principal() + if principal.authenticated and not principal.is_admin: + raise OwnershipNotFoundError("Audit stream not found") + pool = await self._get_pool() + async with pool.acquire() as conn: + rows = await conn.fetch( + "SELECT audit_id, event_type, payload, created_at " + "FROM enterprise_admin_audits ORDER BY created_at DESC LIMIT $1", + max(int(limit), 0), + ) + results = [] + for row in rows: + payload = row["payload"] + results.append( + { + "audit_id": row["audit_id"], + "event_type": row["event_type"], + "payload": json.loads(payload) if isinstance(payload, str) else payload, + "created_at": row["created_at"], + } + ) + return results + + async def _async_upsert_enterprise_user( + self, + *, + user_id: str, + email: str, + name: str, + role: str, + status: Optional[str], + mfa_enabled: Optional[bool], + metadata: Optional[dict], + oidc_issuer: Optional[str], + oidc_subject: Optional[str], + ) -> dict: + """Atomically merge profile and security state for one user.""" + pool = await self._get_pool() + artifact_id = self._enterprise_user_artifact_id(user_id) + now = time.time() + async with pool.acquire() as conn: + async with conn.transaction(): + await conn.execute( + "SELECT pg_advisory_xact_lock(hashtextextended($1, 0))", + f"enterprise-user:{user_id}", + ) + row = await conn.fetchrow( + "SELECT payload FROM flagship_artifacts " + "WHERE artifact_id = $1 AND artifact_type = 'enterprise_user' FOR UPDATE", + artifact_id, + ) + existing: dict = {} + if row: + raw_existing = row["payload"] + decoded = json.loads(raw_existing) if isinstance(raw_existing, str) else raw_existing + existing = decoded if isinstance(decoded, dict) else {} + binding = prepare_enterprise_identity_binding( + status=status if status is not None else str(existing.get("status") or "active"), + metadata=metadata, + oidc_issuer=oidc_issuer, + oidc_subject=oidc_subject, + existing=existing, + ) + effective_mfa = ( + bool(existing.get("mfa_enabled")) + if mfa_enabled is None + else bool(mfa_enabled) + ) + payload = { + **existing, + "user_id": user_id, + "email": str(email).strip().lower(), + "name": name, + "role": role, + "status": binding.status, + "mfa_enabled": effective_mfa, + "metadata": binding.metadata, + "oidc_issuer": binding.issuer, + "oidc_subject": binding.subject, + "created_at": existing.get("created_at", now), + "updated_at": now, + } + await conn.execute( + """ + INSERT INTO flagship_artifacts + (artifact_id, artifact_type, session_id, run_id, plan_id, + payload, created_at, updated_at) + VALUES ($1, 'enterprise_user', '', '', '', $2, $3, $3) + ON CONFLICT(artifact_id, artifact_type) DO UPDATE SET + payload = EXCLUDED.payload, + updated_at = EXCLUDED.updated_at + """, + artifact_id, + json.dumps(payload), + now, + ) + return payload + + async def _async_list_documents(self, session_id: str, limit: int) -> list[dict]: + pool = await self._get_pool() + async with pool.acquire() as conn: + async with conn.transaction(): + if not await self._async_session_visible_in_transaction(conn, session_id): + return [] + rows = await conn.fetch( + "SELECT payload FROM flagship_artifacts " + "WHERE artifact_type = 'document' AND session_id = $1 " + "ORDER BY updated_at DESC LIMIT $2", + session_id, + max(int(limit), 0), + ) + documents = [] + for row in rows: + payload = row["payload"] + decoded = json.loads(payload) if isinstance(payload, str) else payload + if isinstance(decoded, dict): + documents.append(decoded) + return documents + + async def _async_list_workbook_jobs( + self, + limit: int = 50, + status: Optional[str] = None, + session_id: Optional[str] = None, + account_id: Optional[str] = None, + ) -> list[dict]: + pool = await self._get_pool() + clauses = ["artifact_type = $1"] + params: list[object] = ["workbook_job"] + principal = current_principal() + if principal.authenticated and not principal.is_admin: + if not principal.user_id: + return [] + params.append(principal.user_id) + clauses.append( + "EXISTS (SELECT 1 FROM sessions s WHERE s.session_id = flagship_artifacts.session_id " + f"AND s.owner_user_id = ${len(params)})" + ) + account_id = None + clauses.append( + "EXISTS (SELECT 1 FROM sessions lifecycle_session " + "JOIN session_lifecycle lifecycle ON lifecycle.session_id = lifecycle_session.session_id " + "WHERE lifecycle_session.session_id = flagship_artifacts.session_id " + "AND lifecycle.state = 'active')" + ) + if status: + params.append(status) + clauses.append(f"payload->>'status' = ${len(params)}") + if session_id: + params.append(session_id) + clauses.append(f"session_id = ${len(params)}") + if account_id: + params.append(account_id) + clauses.append(f"payload->>'account_id' = ${len(params)}") + params.append(max(limit, 0)) + async with pool.acquire() as conn: + rows = await conn.fetch( + f""" + SELECT payload + FROM flagship_artifacts + WHERE {' AND '.join(clauses)} + ORDER BY created_at DESC + LIMIT ${len(params)} + """, + *params, + ) + return [self._decode_payload(row["payload"]) for row in rows] + + @staticmethod + def _decode_payload(payload): + if isinstance(payload, str): + return json.loads(payload or "{}") + return payload or {} + + async def _async_get_typed_json( + self, + table_name: str, + json_column: str, + id_column: str, + artifact_id: str, + ) -> Optional[dict]: + pool = await self._get_pool() + async with pool.acquire() as conn: + async with conn.transaction(): + row = await conn.fetchrow( + f"SELECT session_id, {json_column} FROM {table_name} " # noqa: S608 -- static callers + f"WHERE {id_column} = $1", # noqa: S608 -- static callers + artifact_id, + ) + if not row or not await self._async_session_visible_in_transaction( + conn, str(row["session_id"] or "") + ): + return None + return self._decode_payload(row[json_column]) + + async def _async_save_workbook_map(self, map_id: str, semantic_map: dict) -> str: + payload = {**semantic_map, "mapId": map_id} + session_id = str(payload.get("sessionId") or "") + now = time.time() + entities = payload.get("entities") or payload.get("semanticEntities") or [] + formula_facts = payload.get("formulaFacts") or payload.get("formulas") or [] + risks = payload.get("risks") or payload.get("requiredInspections") or payload.get("confidenceGaps") or [] + async with self._async_projected_artifact_transaction( + map_id, + "workbook_map", + session_id, + "", + "", + payload, + now=now, + ) as conn: + await conn.execute( + """ + INSERT INTO workbook_maps + ( + map_id, + session_id, + workbook_fingerprint, + workbook_name, + schema_version, + entity_count, + formula_count, + risk_count, + map_json, + created_at, + updated_at + ) + VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11) + ON CONFLICT(map_id) DO UPDATE SET + session_id = EXCLUDED.session_id, + workbook_fingerprint = EXCLUDED.workbook_fingerprint, + workbook_name = EXCLUDED.workbook_name, + schema_version = EXCLUDED.schema_version, + entity_count = EXCLUDED.entity_count, + formula_count = EXCLUDED.formula_count, + risk_count = EXCLUDED.risk_count, + map_json = EXCLUDED.map_json, + updated_at = EXCLUDED.updated_at + """, + map_id, + session_id, + str(payload.get("workbookFingerprint") or ""), + str(payload.get("workbookName") or ""), + str(payload.get("schemaVersion") or ""), + len(entities) if isinstance(entities, list) else 0, + len(formula_facts) if isinstance(formula_facts, list) else 0, + len(risks) if isinstance(risks, list) else 0, + json.dumps(payload), + now, + now, + ) + return map_id + + async def _async_save_source_evidence_graph(self, graph_id: str, graph: dict) -> str: + payload = {**graph, "graphId": graph_id} + session_id = str(payload.get("sessionId") or "") + now = time.time() + document_ids = payload.get("documentIds") or payload.get("document_ids") or [] + async with self._async_projected_artifact_transaction( + graph_id, + "source_evidence_graph", + session_id, + "", + "", + payload, + now=now, + ) as conn: + await conn.execute( + """ + INSERT INTO source_evidence_graphs + (graph_id, session_id, document_ids, graph_json, created_at, updated_at) + VALUES ($1, $2, $3, $4, $5, $6) + ON CONFLICT(graph_id) DO UPDATE SET + session_id = EXCLUDED.session_id, + document_ids = EXCLUDED.document_ids, + graph_json = EXCLUDED.graph_json, + updated_at = EXCLUDED.updated_at + """, + graph_id, + session_id, + json.dumps(document_ids if isinstance(document_ids, list) else []), + json.dumps(payload), + now, + now, + ) + return graph_id + + async def _async_save_operation_simulation( + self, + simulation_id: str, + session_id: str, + simulation: dict, + *, + plan_id: str = "", + run_id: str = "", + ) -> str: + payload = {**simulation, "simulationId": simulation_id} + now = time.time() + async with self._async_projected_artifact_transaction( + simulation_id, + "operation_simulation", + session_id, + run_id, + plan_id, + payload, + now=now, + ) as conn: + await conn.execute( + """ + INSERT INTO operation_simulations + (simulation_id, session_id, plan_id, run_id, status, simulation_json, created_at) + VALUES ($1, $2, $3, $4, $5, $6, $7) + ON CONFLICT(simulation_id) DO UPDATE SET + session_id = EXCLUDED.session_id, + plan_id = EXCLUDED.plan_id, + run_id = EXCLUDED.run_id, + status = EXCLUDED.status, + simulation_json = EXCLUDED.simulation_json, + created_at = EXCLUDED.created_at + """, + simulation_id, + session_id, + plan_id, + run_id, + str(payload.get("status") or payload.get("simulationGate") or "warning"), + json.dumps(payload), + now, + ) + return simulation_id + + async def _async_save_workbook_invariant_run( + self, + invariant_run_id: str, + session_id: str, + report: dict, + *, + run_id: str = "", + plan_id: str = "", + ) -> str: + payload = {**report, "reportId": invariant_run_id} + now = time.time() + async with self._async_projected_artifact_transaction( + invariant_run_id, + "workbook_invariant", + session_id, + run_id, + plan_id, + payload, + now=now, + ) as conn: + await conn.execute( + """ + INSERT INTO workbook_invariant_runs + (invariant_run_id, session_id, run_id, plan_id, status, report_json, created_at) + VALUES ($1, $2, $3, $4, $5, $6, $7) + ON CONFLICT(invariant_run_id) DO UPDATE SET + session_id = EXCLUDED.session_id, + run_id = EXCLUDED.run_id, + plan_id = EXCLUDED.plan_id, + status = EXCLUDED.status, + report_json = EXCLUDED.report_json, + created_at = EXCLUDED.created_at + """, + invariant_run_id, + session_id, + run_id, + plan_id, + str(payload.get("status") or "warning"), + json.dumps(payload), + now, + ) + return invariant_run_id + + async def _async_save_agent_run(self, run_id: str, run: dict) -> str: + payload = {**run, "runId": run_id} + session_id = str(payload.get("sessionId") or payload.get("session_id") or "") + plan_id = str(payload.get("planId") or payload.get("plan_id") or "") + now = time.time() + execution_groups = payload.get("executionGroups") or [] + checkpoints = payload.get("checkpoints") or payload.get("checkpointReasons") or [] + v7_quality_gate = payload.get("v7QualityGate") + v7_quality_gate = v7_quality_gate if isinstance(v7_quality_gate, dict) else {} + async with self._async_projected_artifact_transaction( + run_id, + "agent_run", + session_id, + run_id, + plan_id, + payload, + now=now, + ) as conn: + await conn.execute( + """ + INSERT INTO agent_runs + ( + run_id, + session_id, + status, + schema_version, + plan_id, + semantic_map_id, + simulation_id, + verification_id, + v7_gate_status, + execution_group_count, + checkpoint_count, + prompt, + run_json, + created_at, + updated_at + ) + VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, $12, $13, $14, $15) + ON CONFLICT(run_id) DO UPDATE SET + session_id = EXCLUDED.session_id, + status = EXCLUDED.status, + schema_version = EXCLUDED.schema_version, + plan_id = EXCLUDED.plan_id, + semantic_map_id = EXCLUDED.semantic_map_id, + simulation_id = EXCLUDED.simulation_id, + verification_id = EXCLUDED.verification_id, + v7_gate_status = EXCLUDED.v7_gate_status, + execution_group_count = EXCLUDED.execution_group_count, + checkpoint_count = EXCLUDED.checkpoint_count, + prompt = EXCLUDED.prompt, + run_json = EXCLUDED.run_json, + updated_at = EXCLUDED.updated_at + """, + run_id, + session_id, + str(payload.get("status") or "created"), + str(payload.get("schemaVersion") or ""), + plan_id, + str(payload.get("semanticMapId") or ""), + str(payload.get("simulationId") or ""), + str(payload.get("verificationId") or ""), + str(payload.get("v7GateStatus") or v7_quality_gate.get("status") or ""), + len(execution_groups) if isinstance(execution_groups, list) else 0, + len(checkpoints) if isinstance(checkpoints, list) else 0, + str(payload.get("prompt") or ""), + json.dumps(payload), + now, + now, + ) + return run_id + + async def _async_append_agent_run_event( + self, + run_id: str, + *, + expected_sequence: int, + event_type: str, + event_payload: dict, + updated_run: dict, + idempotency_key: str = "", + ) -> dict: + """Compare-and-append a run event under the durable session fence.""" + normalized_run_id = str(run_id or "").strip() + normalized_type = str(event_type or "").strip() + normalized_key = str(idempotency_key or "").strip() + if not normalized_run_id or not normalized_type: + raise ValueError("run_id and event_type are required") + if int(expected_sequence) < 0: + raise ValueError("expected_sequence must be non-negative") + pool = await self._get_pool() + async with pool.acquire() as conn: + async with conn.transaction(): + owner_row = await conn.fetchrow( + "SELECT session_id FROM agent_runs WHERE run_id = $1", + normalized_run_id, + ) + if not owner_row: + return {"outcome": "not_found", "run": None, "event": None} + session_id = str(owner_row["session_id"] or "") + from backend.services.lifecycle import get_lifecycle_coordinator + + if get_lifecycle_coordinator().writes_revoked(session_id): + raise OwnershipNotFoundError("Resource not found") + await self._async_require_session_write(conn, session_id) + row = await conn.fetchrow( + "SELECT session_id, run_json FROM agent_runs " + "WHERE run_id = $1 FOR UPDATE", + normalized_run_id, + ) + if not row: + return {"outcome": "not_found", "run": None, "event": None} + current_run = self._decode_payload(row["run_json"]) + event_rows = await conn.fetch( + "SELECT artifact_json FROM agent_run_artifacts " + "WHERE run_id = $1 AND artifact_type = 'control_event'", + normalized_run_id, + ) + events = [self._decode_payload(item["artifact_json"]) for item in event_rows] + if normalized_key: + replay = next( + ( + item + for item in events + if str(item.get("idempotencyKey") or "") == normalized_key + ), + None, + ) + if replay: + if str(replay.get("eventType") or "") != normalized_type: + return { + "outcome": "idempotency_conflict", + "run": current_run, + "event": replay, + } + return {"outcome": "replayed", "run": current_run, "event": replay} + + metadata = current_run.get("metadata") + metadata = dict(metadata) if isinstance(metadata, dict) else {} + control = metadata.get("controlPlane") + control = dict(control) if isinstance(control, dict) else {} + stored_sequence = max( + int(control.get("eventSequence") or 0), + max((int(item.get("sequence") or 0) for item in events), default=0), + ) + if stored_sequence != int(expected_sequence): + return { + "outcome": "sequence_conflict", + "run": current_run, + "event": None, + "currentSequence": stored_sequence, + } + if str(current_run.get("status") or "") in { + "completed", + "failed", + "cancelled", + }: + return { + "outcome": "terminal_fence", + "run": current_run, + "event": None, + "currentSequence": stored_sequence, + } + + next_run = json.loads(json.dumps(updated_run, default=str)) + next_run_id = str(next_run.get("runId") or next_run.get("run_id") or "") + next_session_id = str( + next_run.get("sessionId") or next_run.get("session_id") or "" + ) + if next_run_id != normalized_run_id or next_session_id != session_id: + raise ValueError("Agent run identity and session ownership are immutable") + sequence = stored_sequence + 1 + occurred_at = time.time() + control.update( + { + "schemaVersion": "2026-08-agent-run-control-v1", + "eventSequence": sequence, + "cursor": str(sequence), + "updatedAt": occurred_at, + } + ) + next_metadata = next_run.get("metadata") + next_metadata = dict(next_metadata) if isinstance(next_metadata, dict) else {} + next_metadata["controlPlane"] = control + next_run["metadata"] = next_metadata + next_run["updatedAt"] = occurred_at + event_id = "run_event_" + hashlib.sha256( + f"{normalized_run_id}:{sequence}:{normalized_type}:{normalized_key}".encode( + "utf-8" + ) + ).hexdigest()[:24] + event = { + "schemaVersion": "2026-08-agent-run-event-v1", + "eventId": event_id, + "runId": normalized_run_id, + "sequence": sequence, + "cursor": str(sequence), + "eventType": normalized_type, + "fromStatus": str(current_run.get("status") or ""), + "toStatus": str(next_run.get("status") or ""), + "payload": dict(event_payload or {}), + "idempotencyKey": normalized_key, + "occurredAt": occurred_at, + } + await conn.execute( + "UPDATE agent_runs SET status = $1, prompt = $2, run_json = $3, " + "updated_at = $4 WHERE run_id = $5 AND session_id = $6", + str(next_run.get("status") or "created"), + str(next_run.get("prompt") or ""), + json.dumps(next_run), + occurred_at, + normalized_run_id, + session_id, + ) + await conn.execute( + "INSERT INTO flagship_artifacts " + "(artifact_id, artifact_type, session_id, run_id, plan_id, payload, created_at, updated_at) " + "VALUES ($1, 'agent_run', $2, $1, $3, $4, $5, $5) " + "ON CONFLICT(artifact_id, artifact_type) DO UPDATE SET " + "payload = EXCLUDED.payload, updated_at = EXCLUDED.updated_at", + normalized_run_id, + session_id, + str(next_run.get("planId") or next_run.get("plan_id") or ""), + json.dumps(next_run), + occurred_at, + ) + await conn.execute( + "INSERT INTO agent_run_artifacts " + "(artifact_id, run_id, session_id, artifact_type, plan_id, artifact_json, created_at, updated_at) " + "VALUES ($1, $2, $3, 'control_event', $4, $5, $6, $6)", + event_id, + normalized_run_id, + session_id, + str(next_run.get("planId") or next_run.get("plan_id") or ""), + json.dumps(event), + occurred_at, + ) + return {"outcome": "appended", "run": next_run, "event": event} + + async def _async_list_agent_run_events( + self, + run_id: str, + after_sequence: int = 0, + limit: int = 100, + ) -> dict: + after = max(int(after_sequence), 0) + bounded_limit = min(max(int(limit), 1), 100) + pool = await self._get_pool() + async with pool.acquire() as conn: + async with conn.transaction(): + run = await conn.fetchrow( + "SELECT session_id FROM agent_runs WHERE run_id = $1", + str(run_id or "").strip(), + ) + if not run or not await self._async_session_visible_in_transaction( + conn, str(run["session_id"] or "") + ): + return { + "events": [], + "nextCursor": str(after), + "latestSequence": after, + "hasMore": False, + } + rows = await conn.fetch( + "SELECT artifact_json FROM agent_run_artifacts " + "WHERE run_id = $1 AND artifact_type = 'control_event' " + "AND (artifact_json->>'sequence')::bigint > $2 " + "ORDER BY (artifact_json->>'sequence')::bigint ASC LIMIT $3", + str(run_id or "").strip(), + after, + bounded_limit + 1, + ) + latest = await conn.fetchval( + "SELECT COALESCE(MAX((artifact_json->>'sequence')::bigint), 0) " + "FROM agent_run_artifacts " + "WHERE run_id = $1 AND artifact_type = 'control_event'", + str(run_id or "").strip(), + ) + decoded = [self._decode_payload(row["artifact_json"]) for row in rows] + page = decoded[:bounded_limit] + next_cursor = str(int(page[-1].get("sequence") or after)) if page else str(after) + return { + "events": page, + "nextCursor": next_cursor, + "latestSequence": max(int(latest or 0), after), + "hasMore": len(decoded) > len(page), + } + + async def _async_save_agent_egress_preview( + self, + session_id: str, + preview: dict, + ) -> str: + approval_id = str(preview.get("approvalId") or "").strip() + preview_session_id = str(preview.get("sessionId") or "").strip() + if not approval_id or preview_session_id != str(session_id or "").strip(): + raise ValueError("Egress preview identity and session are required") + pool = await self._get_pool() + async with pool.acquire() as conn: + async with conn.transaction(): + await self._async_require_session_write(conn, session_id) + existing = await conn.fetchrow( + "SELECT artifact_type, artifact_json FROM agent_run_artifacts " + "WHERE artifact_id = $1 FOR UPDATE", + approval_id, + ) + if existing: + current = self._decode_payload(existing["artifact_json"]) + if str(existing["artifact_type"] or "") != "egress_preview" or current != preview: + raise ValueError("Egress approval identity is immutable") + return approval_id + issued_at = float(preview.get("issuedAt") or time.time()) + await conn.execute( + "INSERT INTO agent_run_artifacts " + "(artifact_id, run_id, session_id, artifact_type, plan_id, artifact_json, created_at, updated_at) " + "VALUES ($1, $2, $3, 'egress_preview', '', $4, $5, $5)", + approval_id, + f"egress:{approval_id}", + session_id, + json.dumps(preview), + issued_at, + ) + return approval_id + + async def _async_consume_agent_egress_approval( + self, + session_id: str, + *, + approval_id: str, + approved_digest: str, + request_digest: str, + consumption_id: str, + now: Optional[float] = None, + ) -> dict: + consumed_at = float(time.time() if now is None else now) + pool = await self._get_pool() + async with pool.acquire() as conn: + async with conn.transaction(): + await self._async_require_session_write(conn, session_id) + row = await conn.fetchrow( + "SELECT artifact_json FROM agent_run_artifacts " + "WHERE artifact_id = $1 AND session_id = $2 " + "AND artifact_type = 'egress_preview' FOR UPDATE", + approval_id, + session_id, + ) + if not row: + return {"outcome": "not_found", "preview": None} + preview = self._decode_payload(row["artifact_json"]) + if not hmac.compare_digest( + str(preview.get("requestDigest") or ""), + str(request_digest or ""), + ) or not hmac.compare_digest( + str(preview.get("approvalDigest") or ""), + str(approved_digest or ""), + ): + return {"outcome": "digest_mismatch", "preview": preview} + if consumed_at > float(preview.get("expiresAt") or 0): + return {"outcome": "expired", "preview": preview} + if preview.get("consumedAt") is not None: + return {"outcome": "already_consumed", "preview": preview} + preview["consumedAt"] = consumed_at + preview["consumptionId"] = str(consumption_id or "") + await conn.execute( + "UPDATE agent_run_artifacts SET artifact_json = $1, updated_at = $2 " + "WHERE artifact_id = $3 AND session_id = $4 " + "AND artifact_type = 'egress_preview'", + json.dumps(preview), + consumed_at, + approval_id, + session_id, + ) + return {"outcome": "consumed", "preview": preview} + + async def _async_save_workbook_truth_model(self, truth_model_id: str, truth_model: dict) -> str: + payload = {**truth_model, "truthModelId": truth_model_id} + session_id = str(payload.get("sessionId") or "") + now = time.time() + confidence_gaps = payload.get("confidenceGaps") or [] + semantic_decisions = payload.get("semanticTargetDecisions") or payload.get("targetDecisions") or [] + model_taxonomy = payload.get("modelTaxonomy") or payload.get("taxonomy") or "" + async with self._async_projected_artifact_transaction( + truth_model_id, + "workbook_truth_model", + session_id, + "", + "", + payload, + now=now, + ) as conn: + await conn.execute( + """ + INSERT INTO workbook_truth_models + ( + truth_model_id, + session_id, + workbook_fingerprint, + schema_version, + semantic_map_id, + formula_graph_id, + accuracy_score, + model_taxonomy, + confidence_gap_count, + semantic_target_decision_count, + truth_json, + created_at, + updated_at + ) + VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, $12, $13) + ON CONFLICT(truth_model_id) DO UPDATE SET + session_id = EXCLUDED.session_id, + workbook_fingerprint = EXCLUDED.workbook_fingerprint, + schema_version = EXCLUDED.schema_version, + semantic_map_id = EXCLUDED.semantic_map_id, + formula_graph_id = EXCLUDED.formula_graph_id, + accuracy_score = EXCLUDED.accuracy_score, + model_taxonomy = EXCLUDED.model_taxonomy, + confidence_gap_count = EXCLUDED.confidence_gap_count, + semantic_target_decision_count = EXCLUDED.semantic_target_decision_count, + truth_json = EXCLUDED.truth_json, + updated_at = EXCLUDED.updated_at + """, + truth_model_id, + session_id, + str(payload.get("workbookFingerprint") or ""), + str(payload.get("schemaVersion") or ""), + str(payload.get("semanticMapId") or payload.get("mapId") or ""), + str(payload.get("formulaGraphId") or ""), + float(payload.get("accuracyScore") or 0), + json.dumps(model_taxonomy) if isinstance(model_taxonomy, (dict, list)) else str(model_taxonomy or ""), + len(confidence_gaps) if isinstance(confidence_gaps, list) else 0, + len(semantic_decisions) if isinstance(semantic_decisions, list) else 0, + json.dumps(payload), + now, + now, + ) + return truth_model_id + + async def _async_save_formula_graph_artifact(self, graph_id: str, graph: dict) -> str: + payload = {**graph, "graphId": graph_id} + session_id = str(payload.get("sessionId") or "") + now = time.time() + formulas = payload.get("formulas") or payload.get("nodes") or [] + material_failures = payload.get("materialFormulaFailures") or payload.get("materialFailures") or [] + quality = payload.get("quality") or payload.get("qualityReport") or {} + async with self._async_projected_artifact_transaction( + graph_id, + "formula_graph", + session_id, + "", + "", + payload, + now=now, + ) as conn: + await conn.execute( + """ + INSERT INTO formula_graph_artifacts + ( + graph_id, + session_id, + workbook_fingerprint, + schema_version, + engine, + formula_count, + material_failure_count, + pre_apply_simulation_ready, + parser_confidence, + quality_json, + graph_json, + created_at, + updated_at + ) + VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, $12, $13) + ON CONFLICT(graph_id) DO UPDATE SET + session_id = EXCLUDED.session_id, + workbook_fingerprint = EXCLUDED.workbook_fingerprint, + schema_version = EXCLUDED.schema_version, + engine = EXCLUDED.engine, + formula_count = EXCLUDED.formula_count, + material_failure_count = EXCLUDED.material_failure_count, + pre_apply_simulation_ready = EXCLUDED.pre_apply_simulation_ready, + parser_confidence = EXCLUDED.parser_confidence, + quality_json = EXCLUDED.quality_json, + graph_json = EXCLUDED.graph_json, + updated_at = EXCLUDED.updated_at + """, + graph_id, + session_id, + str(payload.get("workbookFingerprint") or ""), + str(payload.get("schemaVersion") or ""), + str(payload.get("engine") or ""), + len(formulas) if isinstance(formulas, list) else 0, + len(material_failures) if isinstance(material_failures, list) else 0, + bool(payload.get("preApplySimulationReady") or payload.get("simulationReady")), + float(payload.get("parserConfidence") or payload.get("confidence") or 0), + json.dumps(quality if isinstance(quality, dict) else {}), + json.dumps(payload), + now, + now, + ) + return graph_id + + async def _async_save_operation_simulation_v2( + self, + simulation_id: str, + session_id: str, + simulation: dict, + *, + plan_id: str = "", + run_id: str = "", + ) -> str: + payload = {**simulation, "simulationId": simulation_id} + now = time.time() + async with self._async_projected_artifact_transaction( + simulation_id, + "operation_simulation_v2", + session_id, + run_id, + plan_id, + payload, + now=now, + ) as conn: + await conn.execute( + """ + INSERT INTO operation_simulations_v2 + (simulation_id, session_id, truth_model_id, plan_id, run_id, status, simulation_json, created_at, updated_at) + VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9) + ON CONFLICT(simulation_id) DO UPDATE SET + session_id = EXCLUDED.session_id, + truth_model_id = EXCLUDED.truth_model_id, + plan_id = EXCLUDED.plan_id, + run_id = EXCLUDED.run_id, + status = EXCLUDED.status, + simulation_json = EXCLUDED.simulation_json, + updated_at = EXCLUDED.updated_at + """, + simulation_id, + session_id, + str(payload.get("truthModelId") or ""), + plan_id, + run_id, + str(payload.get("status") or payload.get("simulationGate") or "warning"), + json.dumps(payload), + now, + now, + ) + return simulation_id + + async def _async_save_v7_workbook_review(self, review_id: str, review: dict) -> str: + payload = {**review, "reviewId": review_id} + session_id = str(payload.get("sessionId") or "") + run_id = str(payload.get("runId") or "") + plan_id = str(payload.get("planId") or "") + now = time.time() + confidence_gaps = payload.get("confidenceGaps") or [] + checkpoint_reasons = payload.get("checkpointReasons") or payload.get("requiredInspections") or [] + readiness = payload.get("defaultReadiness") or payload.get("defaultReadinessSnapshot") or {} + readiness = readiness if isinstance(readiness, dict) else {} + async with self._async_projected_artifact_transaction( + review_id, + "v7_workbook_review", + session_id, + run_id, + plan_id, + payload, + now=now, + ) as conn: + await conn.execute( + """ + INSERT INTO v7_workbook_reviews + ( + review_id, + session_id, + plan_id, + run_id, + workbook_fingerprint, + accuracy_score, + confidence_gap_count, + checkpoint_reason_count, + default_readiness_status, + review_json, + created_at, + updated_at + ) + VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, $12) + ON CONFLICT(review_id) DO UPDATE SET + session_id = EXCLUDED.session_id, + plan_id = EXCLUDED.plan_id, + run_id = EXCLUDED.run_id, + workbook_fingerprint = EXCLUDED.workbook_fingerprint, + accuracy_score = EXCLUDED.accuracy_score, + confidence_gap_count = EXCLUDED.confidence_gap_count, + checkpoint_reason_count = EXCLUDED.checkpoint_reason_count, + default_readiness_status = EXCLUDED.default_readiness_status, + review_json = EXCLUDED.review_json, + updated_at = EXCLUDED.updated_at + """, + review_id, + session_id, + plan_id, + run_id, + str(payload.get("workbookFingerprint") or ""), + float(payload.get("accuracyScore") or 0), + len(confidence_gaps) if isinstance(confidence_gaps, list) else 0, + len(checkpoint_reasons) if isinstance(checkpoint_reasons, list) else 0, + str(readiness.get("status") or payload.get("defaultReadinessStatus") or ""), + json.dumps(payload), + now, + now, + ) + return review_id + + async def _async_save_v7_default_readiness_report(self, report: dict) -> str: + readiness_id = str(report.get("readinessId") or f"v7ready_{uuid.uuid4().hex[:12]}") + payload = {**report, "readinessId": readiness_id} + now = time.time() + async with self._async_projected_artifact_transaction( + readiness_id, + "v7_default_readiness_report", + "", + "", + "", + payload, + now=now, + ) as conn: + await conn.execute( + """ + INSERT INTO v7_default_readiness_reports + (readiness_id, status, default_allowed, score, report_json, created_at, updated_at) + VALUES ($1, $2, $3, $4, $5, $6, $7) + ON CONFLICT(readiness_id) DO UPDATE SET + status = EXCLUDED.status, + default_allowed = EXCLUDED.default_allowed, + score = EXCLUDED.score, + report_json = EXCLUDED.report_json, + updated_at = EXCLUDED.updated_at + """, + readiness_id, + str(payload.get("status") or "warning"), + bool(payload.get("defaultAllowed")), + float(payload.get("score") or 0), + json.dumps(payload), + now, + now, + ) + return readiness_id + + async def _async_list_v7_default_readiness_reports(self, limit: int = 20) -> list[dict]: pool = await self._get_pool() async with pool.acquire() as conn: rows = await conn.fetch( - "SELECT * FROM conversations WHERE session_id = $1 " - "ORDER BY timestamp DESC LIMIT $2", - session_id, limit, + """ + SELECT report_json + FROM v7_default_readiness_reports + ORDER BY updated_at DESC + LIMIT $1 + """, + max(limit, 0), ) - rows = list(reversed(rows)) - return [ - ConversationMessage( - role=r["role"], - content=r["content"], - timestamp=r["timestamp"], - metadata=json.loads(r["metadata"] or "{}") if isinstance(r["metadata"], str) else (r["metadata"] or {}), - ) - for r in rows - ] + return [self._decode_payload(row["report_json"]) for row in rows] - async def _async_record_task(self, session_id, task_id, prompt, mode, - operations, summary, plan, provider, model, tokens): - await self._async_get_or_create_session(session_id) - pool = await self._get_pool() + async def _async_save_local_replay_artifact( + self, + replay_id: str, + session_id: str, + artifact_type: str, + payload: dict, + ) -> str: now = time.time() - async with pool.acquire() as conn: + async with self._async_projected_artifact_transaction( + replay_id, + f"local_replay:{artifact_type}", + session_id, + "", + "", + payload, + now=now, + ) as conn: await conn.execute( - "INSERT INTO task_history " - "(session_id, task_id, prompt, mode, operations, summary, plan, provider, model, tokens_used, timestamp) " - "VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11)", - session_id, task_id, prompt, mode, - json.dumps(operations), summary, json.dumps(plan or []), - provider, model, json.dumps(tokens or {}), now, + """ + INSERT INTO local_replay_artifacts + ( + replay_id, + session_id, + artifact_kind, + schema_version, + v7_gate_status, + accuracy_score, + confidence_gap_count, + checkpoint_reason_count, + formula_failure_count, + truth_schema, + simulation_schema, + replay_json, + created_at, + updated_at + ) + VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, $12, $13, $14) + ON CONFLICT(replay_id) DO UPDATE SET + session_id = EXCLUDED.session_id, + artifact_kind = EXCLUDED.artifact_kind, + schema_version = EXCLUDED.schema_version, + v7_gate_status = EXCLUDED.v7_gate_status, + accuracy_score = EXCLUDED.accuracy_score, + confidence_gap_count = EXCLUDED.confidence_gap_count, + checkpoint_reason_count = EXCLUDED.checkpoint_reason_count, + formula_failure_count = EXCLUDED.formula_failure_count, + truth_schema = EXCLUDED.truth_schema, + simulation_schema = EXCLUDED.simulation_schema, + replay_json = EXCLUDED.replay_json, + updated_at = EXCLUDED.updated_at + """, + replay_id, + session_id, + artifact_type, + str(payload.get("schemaVersion") or ""), + str(payload.get("v7GateStatus") or ""), + float(payload.get("accuracyScore") or 0), + int(payload.get("confidenceGapCount") or 0), + int(payload.get("checkpointReasonCount") or 0), + int(payload.get("formulaFailureCount") or 0), + str(payload.get("truthSchema") or ""), + str(payload.get("simulationSchema") or ""), + json.dumps(payload), + now, + now, ) + return replay_id - async def _async_get_task_history(self, session_id: str, limit: int = 20) -> list[dict]: + async def _async_list_local_replay_artifacts(self, limit: int = 100) -> list[dict]: pool = await self._get_pool() + principal = current_principal() + params: list[object] = [] + owner_clause = "" + if principal.authenticated and not principal.is_admin: + if not principal.user_id: + return [] + params.append(principal.user_id) + owner_clause = f"AND session.owner_user_id = ${len(params)}" + params.append(max(limit, 0)) async with pool.acquire() as conn: rows = await conn.fetch( - "SELECT * FROM task_history WHERE session_id = $1 " - "ORDER BY timestamp DESC LIMIT $2", - session_id, limit, + f""" + SELECT replay.replay_id, replay.session_id, replay.artifact_kind, + replay.replay_json, replay.created_at + FROM local_replay_artifacts AS replay + JOIN sessions AS session ON session.session_id = replay.session_id + JOIN session_lifecycle AS lifecycle ON lifecycle.session_id = replay.session_id + WHERE lifecycle.state = 'active' + {owner_clause} + ORDER BY replay.updated_at DESC + LIMIT ${len(params)} + """, + *params, ) - results = [] - for r in rows: - d = dict(r) - for field in ("operations", "plan", "tokens_used"): - if isinstance(d.get(field), str): - d[field] = json.loads(d[field]) - results.append(d) - return results + results = [] + for row in rows: + results.append( + { + "artifact_id": row["replay_id"], + "artifact_type": f"local_replay:{row['artifact_kind']}", + "session_id": row["session_id"], + "payload": self._decode_payload(row["replay_json"]), + "created_at": row["created_at"], + } + ) + return results - async def _async_get_total_tokens(self, session_id: Optional[str] = None) -> dict: + async def _async_clear_local_replay_artifacts(self, session_id: str) -> int: pool = await self._get_pool() async with pool.acquire() as conn: - if session_id: - rows = await conn.fetch( - "SELECT tokens_used FROM task_history WHERE session_id = $1", + async with conn.transaction(): + await self._async_require_session_write(conn, session_id) + generic_result = await conn.execute( + "DELETE FROM flagship_artifacts " + "WHERE artifact_type LIKE 'local_replay:%' AND session_id = $1", session_id, ) - else: - rows = await conn.fetch("SELECT tokens_used FROM task_history") + typed_result = await conn.execute( + "DELETE FROM local_replay_artifacts WHERE session_id = $1", + session_id, + ) + return max(self._row_count(generic_result), self._row_count(typed_result)) - total = {"prompt_tokens": 0, "completion_tokens": 0} - for r in rows: - tokens = r["tokens_used"] - if isinstance(tokens, str): - tokens = json.loads(tokens) - total["prompt_tokens"] += tokens.get("prompt_tokens", 0) - total["completion_tokens"] += tokens.get("completion_tokens", 0) - total["total_tokens"] = total["prompt_tokens"] + total["completion_tokens"] - return total + async def _async_save_local_replay_baseline(self, baseline: dict) -> str: + baseline_id = str(baseline.get("baselineId") or f"replaybase_{uuid.uuid4().hex[:12]}") + payload = {**baseline, "baselineId": baseline_id} + session_id = str(payload.get("sessionId") or "local-replay-baseline") + metadata = payload.get("metadata") + metadata = metadata if isinstance(metadata, dict) else {} + baseline_comparison = payload.get("baselineComparison") + baseline_comparison = baseline_comparison if isinstance(baseline_comparison, dict) else {} + now = time.time() + async with self._async_projected_artifact_transaction( + baseline_id, + "local_replay_baseline", + session_id, + "", + "", + payload, + now=now, + ) as conn: + await conn.execute( + """ + INSERT INTO local_replay_baselines + ( + baseline_id, + session_id, + status, + production_evidence, + evidence_class, + replay_count, + deterministic_drift_class, + baseline_json, + created_at, + updated_at + ) + VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10) + ON CONFLICT(baseline_id) DO UPDATE SET + session_id = EXCLUDED.session_id, + status = EXCLUDED.status, + production_evidence = EXCLUDED.production_evidence, + evidence_class = EXCLUDED.evidence_class, + replay_count = EXCLUDED.replay_count, + deterministic_drift_class = EXCLUDED.deterministic_drift_class, + baseline_json = EXCLUDED.baseline_json, + updated_at = EXCLUDED.updated_at + """, + baseline_id, + session_id, + str(payload.get("status") or metadata.get("status") or "warning"), + bool(payload.get("productionEvidence") or metadata.get("productionEvidence")), + str(payload.get("evidenceClass") or metadata.get("evidenceClass") or ""), + int(payload.get("replayCount") or metadata.get("replayCount") or 0), + str( + payload.get("deterministicDriftClass") + or baseline_comparison.get("deterministicDriftClass") + or "" + ), + json.dumps(payload), + now, + now, + ) + return baseline_id - async def _async_save_artifact( + async def _async_list_local_replay_baselines(self, limit: int = 20) -> list[dict]: + pool = await self._get_pool() + principal = current_principal() + params: list[object] = [] + owner_clause = "" + if principal.authenticated and not principal.is_admin: + if not principal.user_id: + return [] + params.append(principal.user_id) + owner_clause = f"AND session.owner_user_id = ${len(params)}" + params.append(max(limit, 0)) + async with pool.acquire() as conn: + rows = await conn.fetch( + f""" + SELECT baseline.session_id, baseline.baseline_json + FROM local_replay_baselines AS baseline + JOIN sessions AS session ON session.session_id = baseline.session_id + JOIN session_lifecycle AS lifecycle ON lifecycle.session_id = baseline.session_id + WHERE lifecycle.state = 'active' + {owner_clause} + ORDER BY baseline.updated_at DESC + LIMIT ${len(params)} + """, + *params, + ) + return [ + self._decode_payload(row["baseline_json"]) + for row in rows + ] + + async def _async_save_agent_run_artifact( self, artifact_id: str, - artifact_type: str, - session_id: str, run_id: str, + session_id: str, + artifact_type: str, plan_id: str, - payload: dict, + artifact: dict, + *, + generic_artifact_type: str, ) -> str: - pool = await self._get_pool() + payload = { + **artifact, + "artifactId": artifact_id, + "runId": run_id, + } now = time.time() - async with pool.acquire() as conn: + async with self._async_projected_artifact_transaction( + artifact_id, + generic_artifact_type, + session_id, + run_id, + plan_id, + payload, + now=now, + ) as conn: await conn.execute( """ - INSERT INTO flagship_artifacts - (artifact_id, artifact_type, session_id, run_id, plan_id, payload, created_at, updated_at) + INSERT INTO agent_run_artifacts + (artifact_id, run_id, session_id, artifact_type, plan_id, artifact_json, created_at, updated_at) VALUES ($1, $2, $3, $4, $5, $6, $7, $8) ON CONFLICT(artifact_id) DO UPDATE SET - artifact_type = EXCLUDED.artifact_type, - session_id = EXCLUDED.session_id, run_id = EXCLUDED.run_id, + session_id = EXCLUDED.session_id, + artifact_type = EXCLUDED.artifact_type, plan_id = EXCLUDED.plan_id, - payload = EXCLUDED.payload, + artifact_json = EXCLUDED.artifact_json, updated_at = EXCLUDED.updated_at """, artifact_id, - artifact_type, - session_id, run_id, + session_id, + artifact_type, plan_id, json.dumps(payload), now, @@ -662,41 +5750,427 @@ async def _async_save_artifact( ) return artifact_id - async def _async_get_artifact(self, artifact_id: str, artifact_type: str) -> Optional[dict]: + async def _async_get_agent_run_artifact(self, artifact_id: str, artifact_type: str) -> Optional[dict]: pool = await self._get_pool() async with pool.acquire() as conn: - row = await conn.fetchrow( - "SELECT payload FROM flagship_artifacts WHERE artifact_id = $1 AND artifact_type = $2", - artifact_id, - artifact_type, - ) - if not row: - return None - payload = row["payload"] - return json.loads(payload) if isinstance(payload, str) else payload + async with conn.transaction(): + row = await conn.fetchrow( + """ + SELECT session_id, artifact_json + FROM agent_run_artifacts + WHERE artifact_id = $1 AND artifact_type = $2 + """, + artifact_id, + artifact_type, + ) + if not row or not await self._async_session_visible_in_transaction( + conn, str(row["session_id"] or "") + ): + return None + return self._decode_payload(row["artifact_json"]) - async def _async_list_run_artifacts(self, run_id: str, limit: int = 100) -> list[dict]: + async def _async_list_agent_run_artifacts(self, run_id: str, limit: int = 100) -> list[dict]: pool = await self._get_pool() + principal = current_principal() + owner_clause = "" + params: list[object] = [run_id] + if principal.authenticated and not principal.is_admin: + if not principal.user_id: + return [] + params.append(principal.user_id) + owner_clause = ( + "AND EXISTS (SELECT 1 FROM sessions s WHERE s.session_id = agent_run_artifacts.session_id " + f"AND s.owner_user_id = ${len(params)}) " + ) + params.append(max(limit, 0)) async with pool.acquire() as conn: rows = await conn.fetch( - """ - SELECT artifact_id, artifact_type, payload, created_at - FROM flagship_artifacts + f""" + SELECT artifact_id, artifact_type, session_id, artifact_json, created_at + FROM agent_run_artifacts WHERE run_id = $1 + AND EXISTS ( + SELECT 1 FROM session_lifecycle lifecycle + WHERE lifecycle.session_id = agent_run_artifacts.session_id + AND lifecycle.state = 'active' + ) + {owner_clause} ORDER BY updated_at DESC - LIMIT $2 + LIMIT ${len(params)} + """, + *params, + ) + return [ + { + "artifact_id": row["artifact_id"], + "artifact_type": row["artifact_type"], + "artifact": self._decode_payload(row["artifact_json"]), + "created_at": row["created_at"], + } + for row in rows + ] + + async def _async_save_verification_report_v2( + self, + session_id: str, + verification_id: str, + verification_report: dict, + ) -> str: + payload = {**verification_report, "verificationId": verification_id} + run_id = str(payload.get("runId") or "") + plan_id = str(payload.get("planId") or "") + now = time.time() + async with self._async_projected_artifact_transaction( + verification_id, + "verification_report", + session_id, + run_id, + plan_id, + payload, + now=now, + ) as conn: + await conn.execute( + """ + INSERT INTO verification_reports_v2 + (verification_id, session_id, run_id, plan_id, status, report_json, created_at, updated_at) + VALUES ($1, $2, $3, $4, $5, $6, $7, $8) + ON CONFLICT(verification_id) DO UPDATE SET + session_id = EXCLUDED.session_id, + run_id = EXCLUDED.run_id, + plan_id = EXCLUDED.plan_id, + status = EXCLUDED.status, + report_json = EXCLUDED.report_json, + updated_at = EXCLUDED.updated_at + """, + verification_id, + session_id, + run_id, + plan_id, + str(payload.get("status") or "warning"), + json.dumps(payload), + now, + now, + ) + return verification_id + + async def _async_save_workbook_benchmark_fixture_result( + self, + run_id: str, + fixture_result_id: str, + result: dict, + ) -> str: + payload = {**result, "fixtureResultId": fixture_result_id, "runId": run_id} + now = time.time() + async with self._async_projected_artifact_transaction( + fixture_result_id, + "workbook_benchmark_fixture", + "", + run_id, + "", + payload, + now=now, + ) as conn: + await conn.execute( + """ + INSERT INTO workbook_benchmark_fixture_results + (fixture_result_id, run_id, fixture_id, status, result_json, created_at) + VALUES ($1, $2, $3, $4, $5, $6) + ON CONFLICT(fixture_result_id) DO UPDATE SET + run_id = EXCLUDED.run_id, + fixture_id = EXCLUDED.fixture_id, + status = EXCLUDED.status, + result_json = EXCLUDED.result_json, + created_at = EXCLUDED.created_at """, + fixture_result_id, run_id, - limit, + str(payload.get("fixtureId") or payload.get("fixture_id") or ""), + str(payload.get("status") or "warning"), + json.dumps(payload), + now, + ) + return fixture_result_id + + async def _async_save_operation_audit_projection( + self, + conn, + audit_id: str, + session_id: str, + plan_id: str, + payload: dict, + created_at: float, + ) -> None: + dirty_ranges = payload.get("dirty_ranges") or payload.get("dirtyRanges") or [] + warnings = payload.get("warnings") or [] + await conn.execute( + """ + INSERT INTO operation_audits + ( + audit_id, + session_id, + plan_id, + operation_id, + operation_type, + dirty_range_count, + warning_count, + payload_json, + created_at + ) + VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9) + ON CONFLICT(audit_id) DO UPDATE SET + session_id = EXCLUDED.session_id, + plan_id = EXCLUDED.plan_id, + operation_id = EXCLUDED.operation_id, + operation_type = EXCLUDED.operation_type, + dirty_range_count = EXCLUDED.dirty_range_count, + warning_count = EXCLUDED.warning_count, + payload_json = EXCLUDED.payload_json, + created_at = EXCLUDED.created_at + """, + audit_id, + session_id, + plan_id, + str(payload.get("operation_id") or payload.get("operationId") or ""), + str(payload.get("type") or payload.get("operation_type") or payload.get("operationType") or ""), + len(dirty_ranges) if isinstance(dirty_ranges, list) else 0, + len(warnings) if isinstance(warnings, list) else 0, + json.dumps(payload), + created_at, + ) + + @staticmethod + def _row_count(result: str) -> int: + try: + return int(str(result).split()[-1]) + except (ValueError, IndexError): + return 0 + + async def _async_get_latest_artifact_by_plan(self, artifact_type: str, plan_id: str) -> Optional[dict]: + pool = await self._get_pool() + async with pool.acquire() as conn: + async with conn.transaction(): + row = await conn.fetchrow( + """ + SELECT artifact_id, artifact_type, session_id, run_id, plan_id, + payload, created_at, updated_at + FROM flagship_artifacts + WHERE artifact_type = $1 AND plan_id = $2 + ORDER BY updated_at DESC + LIMIT 1 + """, + artifact_type, + plan_id, + ) + if not row: + return None + session_id = str(row["session_id"] or "") + if _artifact_is_session_scoped( + artifact_type, session_id + ) and not await self._async_session_visible_in_transaction(conn, session_id): + return None + data = dict(row) + data["payload"] = self._decode_payload(data.get("payload")) + return data + + async def _async_save_citations(self, session_id: str, plan_id: str, citations: list[dict]) -> int: + now = time.time() + for citation in citations: + artifact_id = f"citation_{uuid.uuid4().hex[:12]}" + payload = { + "citation_id": artifact_id, + "session_id": session_id, + "plan_id": plan_id, + "operation_id": citation.get("operation_id") or citation.get("operationId"), + "cell": citation.get("cell", ""), + "citation": citation, + "created_at": now, + } + await self._async_save_artifact(artifact_id, "citation", session_id, "", plan_id, payload) + return len(citations) + + async def _async_list_citations( + self, + session_id: str, + plan_id: Optional[str] = None, + limit: int = 200, + ) -> list[dict]: + pool = await self._get_pool() + async with pool.acquire() as conn: + async with conn.transaction(): + if not await self._async_session_visible_in_transaction(conn, session_id): + return [] + if plan_id: + rows = await conn.fetch( + """ + SELECT artifact_id, payload, created_at + FROM flagship_artifacts + WHERE artifact_type = 'citation' AND session_id = $1 AND plan_id = $2 + ORDER BY updated_at DESC + LIMIT $3 + """, + session_id, + plan_id, + limit, + ) + else: + rows = await conn.fetch( + """ + SELECT artifact_id, payload, created_at + FROM flagship_artifacts + WHERE artifact_type = 'citation' AND session_id = $1 + ORDER BY updated_at DESC + LIMIT $2 + """, + session_id, + limit, + ) + results = [] + for row in rows: + payload = self._decode_payload(row["payload"]) + results.append({**payload, "citation_id": row["artifact_id"], "created_at": row["created_at"]}) + return results + + async def _async_save_operation_audits(self, session_id: str, plan_id: str, rows: list[dict]) -> int: + now = time.time() + pool = await self._get_pool() + async with pool.acquire() as conn: + async with conn.transaction(): + for audit in rows: + artifact_id = ( + audit.get("audit_id") + or audit.get("auditId") + or f"opaudit_{uuid.uuid4().hex[:12]}" + ) + payload = { + **audit, + "audit_id": artifact_id, + "session_id": session_id, + "plan_id": plan_id, + "created_at": now, + } + await self._async_save_artifact_in_transaction( + conn, + artifact_id, + "operation_audit", + session_id, + "", + plan_id, + payload, + now=now, + ) + await self._async_save_operation_audit_projection( + conn, + artifact_id, + session_id, + plan_id, + payload, + now, + ) + return len(rows) + + async def _async_delete_session(self, session_id: str) -> bool: + pool = await self._get_pool() + async with pool.acquire() as conn: + async with conn.transaction(): + await conn.execute( + "SELECT pg_advisory_xact_lock(hashtextextended($1, 0))", + session_id, + ) + lifecycle_row = await conn.fetchrow( + "SELECT state FROM session_lifecycle WHERE session_id = $1 FOR UPDATE", + session_id, + ) + if not lifecycle_row or str(lifecycle_row["state"] or "") != "purging": + raise RuntimeError("Session deletion requires a durable PURGING tombstone") + session_row = await conn.fetchrow( + "SELECT 1 FROM sessions WHERE session_id = $1 FOR UPDATE", + session_id, + ) + typed_results = [] + for table_name in POSTGRES_SESSION_LIFECYCLE_TABLES: + if table_name in {"sessions", "conversations", "task_history", "flagship_artifacts"}: + continue + typed_results.append( + await conn.execute( + f"DELETE FROM {table_name} WHERE session_id = $1", # noqa: S608 -- static table inventory + session_id, + ) + ) + artifact_result = await conn.execute( + "DELETE FROM flagship_artifacts WHERE session_id = $1 " + "AND artifact_type <> ALL($2::text[])", + session_id, + sorted(POSTGRES_GLOBAL_ARTIFACT_TYPES), + ) + task_result = await conn.execute( + "DELETE FROM task_history WHERE session_id = $1", + session_id, + ) + conversation_result = await conn.execute( + "DELETE FROM conversations WHERE session_id = $1", + session_id, + ) + session_result = await conn.execute( + "DELETE FROM sessions WHERE session_id = $1", + session_id, + ) + remaining = await self._async_session_has_durable_rows(conn, session_id) + if remaining: + raise RuntimeError("PostgreSQL session deletion verification failed") + deleted = any( + self._row_count(result) > 0 + for result in (*typed_results, artifact_result, task_result, conversation_result, session_result) + ) + # A PURGING tombstone with an empty inventory is a recoverable + # post-commit state, not a deletion no-op. + return deleted or session_row is None + + async def _async_list_run_artifacts(self, run_id: str, limit: int = 100) -> list[dict]: + pool = await self._get_pool() + principal = current_principal() + params: list[object] = [run_id, sorted(POSTGRES_GLOBAL_ARTIFACT_TYPES)] + owner_clause = "" + if principal.authenticated and not principal.is_admin: + if not principal.user_id: + return [] + params.append(principal.user_id) + owner_clause = f"AND session.owner_user_id = ${len(params)}" + params.append(limit) + async with pool.acquire() as conn: + rows = await conn.fetch( + f""" + SELECT artifact.artifact_id, artifact.artifact_type, + artifact.session_id, artifact.payload, artifact.created_at + FROM flagship_artifacts AS artifact + LEFT JOIN sessions AS session ON session.session_id = artifact.session_id + LEFT JOIN session_lifecycle AS lifecycle + ON lifecycle.session_id = artifact.session_id + WHERE artifact.run_id = $1 + AND ( + BTRIM(COALESCE(artifact.session_id, '')) = '' + OR artifact.artifact_type = ANY($2::text[]) + OR (lifecycle.state = 'active' {owner_clause}) + ) + ORDER BY artifact.updated_at DESC + LIMIT ${len(params)} + """, + *params, ) results = [] for row in rows: + artifact_type = str(row["artifact_type"] or "") payload = row["payload"] + artifact = json.loads(payload) if isinstance(payload, str) else payload + if artifact_type.startswith("agent_run:"): + artifact_type = artifact_type[len("agent_run:"):] + elif artifact_type == "agent_run_step_artifact_v3": + artifact_payload = artifact if isinstance(artifact, dict) else {} + artifact_type = f"step:{artifact_payload.get('artifactType', 'artifact')}" results.append( { "artifact_id": row["artifact_id"], - "artifact_type": row["artifact_type"], - "artifact": json.loads(payload) if isinstance(payload, str) else payload, + "artifact_type": artifact_type, + "artifact": artifact, "created_at": row["created_at"], } ) @@ -705,36 +6179,70 @@ async def _async_list_run_artifacts(self, run_id: str, limit: int = 100) -> list async def _async_list_brain_runs(self, session_id: str, limit: int = 20) -> list[dict]: pool = await self._get_pool() async with pool.acquire() as conn: - rows = await conn.fetch( - """ - SELECT payload - FROM flagship_artifacts - WHERE session_id = $1 AND artifact_type = 'brain_run' - ORDER BY updated_at DESC - LIMIT $2 - """, - session_id, - limit, - ) + async with conn.transaction(): + if not await self._async_session_visible_in_transaction(conn, session_id): + return [] + rows = await conn.fetch( + """ + SELECT artifact_json + FROM agent_run_artifacts + WHERE session_id = $1 AND artifact_type = 'brain_run' + ORDER BY updated_at DESC + LIMIT $2 + """, + session_id, + limit, + ) + if not rows: + rows = await conn.fetch( + """ + SELECT payload AS artifact_json + FROM flagship_artifacts + WHERE session_id = $1 AND artifact_type = 'brain_run' + ORDER BY updated_at DESC + LIMIT $2 + """, + session_id, + limit, + ) results = [] for row in rows: - payload = row["payload"] - results.append(json.loads(payload) if isinstance(payload, str) else payload) + results.append(self._decode_payload(row["artifact_json"])) return results async def _async_list_artifacts_by_prefix(self, artifact_type_prefix: str, limit: int = 100) -> list[dict]: pool = await self._get_pool() + principal = current_principal() + params: list[object] = [ + f"{artifact_type_prefix}%", + sorted(POSTGRES_GLOBAL_ARTIFACT_TYPES), + ] + owner_clause = "" + if principal.authenticated and not principal.is_admin: + if not principal.user_id: + return [] + params.append(principal.user_id) + owner_clause = f"AND session.owner_user_id = ${len(params)}" + params.append(limit) async with pool.acquire() as conn: rows = await conn.fetch( - """ - SELECT artifact_id, artifact_type, session_id, payload, created_at - FROM flagship_artifacts - WHERE artifact_type LIKE $1 - ORDER BY updated_at DESC - LIMIT $2 + f""" + SELECT artifact.artifact_id, artifact.artifact_type, + artifact.session_id, artifact.payload, artifact.created_at + FROM flagship_artifacts AS artifact + LEFT JOIN sessions AS session ON session.session_id = artifact.session_id + LEFT JOIN session_lifecycle AS lifecycle + ON lifecycle.session_id = artifact.session_id + WHERE artifact.artifact_type LIKE $1 + AND ( + BTRIM(COALESCE(artifact.session_id, '')) = '' + OR artifact.artifact_type = ANY($2::text[]) + OR (lifecycle.state = 'active' {owner_clause}) + ) + ORDER BY artifact.updated_at DESC + LIMIT ${len(params)} """, - f"{artifact_type_prefix}%", - limit, + *params, ) results = [] for row in rows: @@ -750,6 +6258,40 @@ async def _async_list_artifacts_by_prefix(self, artifact_type_prefix: str, limit ) return results + async def _async_count_artifacts_by_prefix(self, artifact_type_prefix: str) -> int: + pool = await self._get_pool() + principal = current_principal() + params: list[object] = [f"{artifact_type_prefix}%", sorted(POSTGRES_GLOBAL_ARTIFACT_TYPES)] + owner_clause = "" + if principal.authenticated and not principal.is_admin: + if not principal.user_id: + return 0 + params.append(principal.user_id) + owner_clause = f"AND session.owner_user_id = ${len(params)}" + async with pool.acquire() as conn: + row = await conn.fetchrow( + f""" + SELECT COUNT(*) AS count + FROM flagship_artifacts AS artifact + WHERE artifact.artifact_type LIKE $1 + AND ( + BTRIM(COALESCE(artifact.session_id, '')) = '' + OR artifact.artifact_type = ANY($2::text[]) + OR EXISTS ( + SELECT 1 + FROM sessions AS session + JOIN session_lifecycle AS lifecycle + ON lifecycle.session_id = session.session_id + WHERE session.session_id = artifact.session_id + AND lifecycle.state = 'active' + {owner_clause} + ) + ) + """, + *params, + ) + return int(row["count"]) if row else 0 + async def _async_delete_artifacts_by_prefix(self, artifact_type_prefix: str, session_id: str) -> int: pool = await self._get_pool() async with pool.acquire() as conn: @@ -766,3 +6308,9 @@ async def _async_delete_artifacts_by_prefix(self, artifact_type_prefix: str, ses return int(str(result).split()[-1]) except (ValueError, IndexError): return 0 + + async def _async_purge_sessions_before(self, cutoff_timestamp: float) -> int: + _ = cutoff_timestamp + raise RuntimeError( + "Direct PostgreSQL retention deletion is disabled; use LifecycleCoordinator.purge_before" + ) diff --git a/backend/storage/postgres_migration_validation.py b/backend/storage/postgres_migration_validation.py new file mode 100644 index 0000000..429bf19 --- /dev/null +++ b/backend/storage/postgres_migration_validation.py @@ -0,0 +1,291 @@ +"""Opt-in live Postgres validation for typed projection migrations.""" + +from __future__ import annotations + +import argparse +import asyncio +import json +import os +import sys +import uuid +from dataclasses import dataclass +from typing import Any + +from backend.storage.postgres import ( + POSTGRES_TYPED_PROJECTION_MIGRATIONS, + POSTGRES_TYPED_PROJECTION_TABLES, + PostgresSessionStore, +) + + +SEED_ARTIFACTS: dict[str, tuple[str, dict[str, Any]]] = { + "v7_default_readiness_reports": ( + "v7_default_readiness_report", + {"status": "ready", "defaultAllowed": False, "score": 0.91}, + ), + "local_replay_artifacts": ( + "local_replay:truth", + { + "schemaVersion": "2026-06-validation", + "v7GateStatus": "passed", + "accuracyScore": 0.9, + "confidenceGapCount": 1, + "checkpointReasonCount": 1, + "formulaFailureCount": 0, + "truthSchema": "truth-v5", + "simulationSchema": "simulation-v6", + }, + ), + "local_replay_baselines": ( + "local_replay_baseline", + { + "status": "ready", + "productionEvidence": True, + "evidenceClass": "validation", + "replayCount": 1, + "deterministicDriftClass": "stable", + }, + ), + "agent_run_artifacts": ("agent_run:summary", {"artifactType": "summary", "status": "ready"}), + "verification_reports_v2": ("verification_report", {"status": "passed"}), + "workbook_benchmark_fixture_results": ( + "workbook_benchmark_fixture", + {"fixtureId": "postgres_validation_fixture", "status": "passed"}, + ), + "operation_audits": ( + "operation_audit", + {"operationId": "op_validation", "type": "set_range_values", "dirtyRanges": ["A1"], "warnings": []}, + ), + "workbook_maps": ( + "workbook_map", + { + "workbookFingerprint": "fp_validation", + "workbookName": "Validation Workbook", + "schemaVersion": "map-v1", + "entities": [{"id": "entity_1"}], + "formulas": [{"address": "A1"}], + "risks": [{"id": "risk_1"}], + }, + ), + "source_evidence_graphs": ( + "source_evidence_graph", + {"documentIds": ["doc_validation"], "evidence": [{"id": "evidence_1"}]}, + ), + "operation_simulations": ("operation_simulation", {"status": "passed"}), + "workbook_invariant_runs": ("workbook_invariant", {"status": "passed"}), + "agent_runs": ( + "agent_run", + { + "status": "completed", + "schemaVersion": "agent-run-v4", + "semanticMapId": "map_validation", + "simulationId": "sim_validation", + "verificationId": "verify_validation", + "v7GateStatus": "passed", + "executionGroups": [{"id": "group_1"}], + "checkpointReasons": [{"id": "checkpoint_1"}], + "prompt": "Validate Postgres migrations", + }, + ), + "workbook_truth_models": ( + "workbook_truth_model", + { + "workbookFingerprint": "fp_validation", + "schemaVersion": "truth-v5", + "semanticMapId": "map_validation", + "formulaGraphId": "graph_validation", + "accuracyScore": 0.93, + "modelTaxonomy": "dcf", + "confidenceGaps": [{"id": "gap_1"}], + "semanticTargetDecisions": [{"id": "decision_1"}], + }, + ), + "formula_graph_artifacts": ( + "formula_graph", + { + "workbookFingerprint": "fp_validation", + "schemaVersion": "formula-graph-v5", + "engine": "hyperformula", + "formulas": [{"address": "A1"}], + "materialFormulaFailures": [], + "preApplySimulationReady": True, + "parserConfidence": 0.98, + "quality": {"status": "passed"}, + }, + ), + "operation_simulations_v2": ( + "operation_simulation_v2", + {"truthModelId": "truth_validation", "status": "passed"}, + ), + "v7_workbook_reviews": ( + "v7_workbook_review", + { + "workbookFingerprint": "fp_validation", + "accuracyScore": 0.94, + "confidenceGaps": [{"id": "gap_1"}], + "checkpointReasons": [{"id": "checkpoint_1"}], + "defaultReadiness": {"status": "blocked"}, + }, + ), +} + + +@dataclass(frozen=True) +class PostgresMigrationValidationResult: + schema: str + typed_table_counts: dict[str, int] + applied_migrations: list[str] + reader_checks: dict[str, int] + + def to_dict(self) -> dict[str, Any]: + return { + "status": "passed", + "schema": self.schema, + "typedTableCounts": self.typed_table_counts, + "appliedMigrations": self.applied_migrations, + "readerChecks": self.reader_checks, + } + + +async def validate_postgres_migration_backfill( + dsn: str, + *, + keep_schema: bool = False, +) -> PostgresMigrationValidationResult: + try: + import asyncpg + except ImportError as exc: # pragma: no cover - depends on optional install + raise RuntimeError("asyncpg is required for live Postgres migration validation") from exc + + schema = f"excel_ai_validation_{uuid.uuid4().hex[:12]}" + quoted_schema = _quote_ident(schema) + admin_conn = await asyncpg.connect(dsn=dsn) + pool = None + try: + await admin_conn.execute(f"CREATE SCHEMA {quoted_schema}") + + async def setup_connection(conn): + await conn.execute(f"SET search_path TO {quoted_schema}") + + pool = await asyncpg.create_pool( + dsn=dsn, + min_size=1, + max_size=2, + command_timeout=30, + setup=setup_connection, + ) + store = PostgresSessionStore(dsn) + store._pool = pool + store._pool_loop = asyncio.get_running_loop() + + await store._init_schema() + await _seed_legacy_flagship_artifacts(pool) + await store._init_schema() + counts = await _typed_table_counts(pool) + _assert_all_typed_tables_populated(counts) + applied = await _applied_migrations(pool) + _assert_all_projection_migrations_applied(applied) + reader_checks = await _reader_checks(store) + + await store._init_schema() + second_counts = await _typed_table_counts(pool) + if second_counts != counts: + raise AssertionError(f"Postgres typed projection backfill is not idempotent: {counts} -> {second_counts}") + + return PostgresMigrationValidationResult( + schema=schema, + typed_table_counts=counts, + applied_migrations=applied, + reader_checks=reader_checks, + ) + finally: + if pool is not None: + await pool.close() + if not keep_schema: + await admin_conn.execute(f"DROP SCHEMA IF EXISTS {quoted_schema} CASCADE") + await admin_conn.close() + + +async def _seed_legacy_flagship_artifacts(pool) -> None: + now = 1_720_000_000.0 + rows = [] + for index, (table, (artifact_type, payload)) in enumerate(SEED_ARTIFACTS.items(), start=1): + rows.append( + ( + f"{table}_validation", + artifact_type, + "postgres_validation_session", + "postgres_validation_run", + "postgres_validation_plan", + json.dumps(payload), + now + index, + now + index, + ) + ) + async with pool.acquire() as conn: + await conn.executemany( + """ + INSERT INTO flagship_artifacts + (artifact_id, artifact_type, session_id, run_id, plan_id, payload, created_at, updated_at) + VALUES ($1, $2, $3, $4, $5, $6::jsonb, $7, $8) + ON CONFLICT(artifact_id, artifact_type) DO UPDATE SET + payload = EXCLUDED.payload, + updated_at = EXCLUDED.updated_at + """, + rows, + ) + + +async def _typed_table_counts(pool) -> dict[str, int]: + counts: dict[str, int] = {} + async with pool.acquire() as conn: + for table in POSTGRES_TYPED_PROJECTION_TABLES: + row = await conn.fetchrow(f"SELECT COUNT(*)::int AS count FROM {_quote_ident(table)}") + counts[table] = int(row["count"]) + return counts + + +async def _applied_migrations(pool) -> list[str]: + async with pool.acquire() as conn: + rows = await conn.fetch("SELECT migration_id FROM postgres_schema_migrations ORDER BY migration_id") + return [str(row["migration_id"]) for row in rows] + + +async def _reader_checks(store: PostgresSessionStore) -> dict[str, int]: + readiness = await store._async_list_v7_default_readiness_reports(limit=5) + replays = await store._async_list_local_replay_artifacts(limit=5) + baselines = await store._async_list_local_replay_baselines(limit=5) + return { + "v7DefaultReadinessReports": len(readiness), + "localReplayArtifacts": len(replays), + "localReplayBaselines": len(baselines), + } + + +def _assert_all_typed_tables_populated(counts: dict[str, int]) -> None: + missing = [table for table, count in counts.items() if count < 1] + if missing: + raise AssertionError(f"Typed projection backfill did not populate tables: {', '.join(missing)}") + + +def _assert_all_projection_migrations_applied(applied: list[str]) -> None: + missing = sorted(set(POSTGRES_TYPED_PROJECTION_MIGRATIONS) - set(applied)) + if missing: + raise AssertionError(f"Postgres projection migrations missing from ledger: {', '.join(missing)}") + + +def _quote_ident(identifier: str) -> str: + return '"' + identifier.replace('"', '""') + '"' + + +def main(argv: list[str] | None = None) -> int: + parser = argparse.ArgumentParser(description="Validate Postgres typed projection migrations against a live database.") + parser.add_argument("--dsn", default=os.getenv("POSTGRES_VALIDATION_DATABASE_URL", "")) + parser.add_argument("--keep-schema", action="store_true", help="Do not drop the temporary validation schema.") + args = parser.parse_args(argv) + if not args.dsn: + print("POSTGRES_VALIDATION_DATABASE_URL or --dsn is required for live Postgres validation.", file=sys.stderr) + return 2 + result = asyncio.run(validate_postgres_migration_backfill(args.dsn, keep_schema=args.keep_schema)) + print(json.dumps(result.to_dict(), indent=2, sort_keys=True)) + return 0 diff --git a/backend/storage/protocol.py b/backend/storage/protocol.py index 1b5bd0a..7709086 100644 --- a/backend/storage/protocol.py +++ b/backend/storage/protocol.py @@ -22,9 +22,35 @@ class ConversationMessage: class SessionStoreProtocol(Protocol): """Protocol for session stores — implemented by SQLite and PostgreSQL.""" - def get_or_create_session(self, session_id: str) -> dict: ... + def get_or_create_session( + self, + session_id: str, + owner_user_id: Optional[str] = None, + ) -> dict: ... - def list_sessions(self, limit: int = 20) -> list[dict]: ... + def get_session_owner(self, session_id: str) -> Optional[str]: ... + def session_exists(self, session_id: str) -> bool: ... + def get_session_lifecycle_state(self, session_id: str) -> Optional[str]: ... + def get_session_lifecycle_record(self, session_id: str) -> Optional[dict]: ... + def set_session_lifecycle_state( + self, + session_id: str, + state: str, + *, + receipt_id: str, + started_at: float, + completed_at: float = 0.0, + errors: Optional[list[str]] = None, + ) -> None: ... + def session_deletion_verified(self, session_id: str) -> bool: ... + def list_session_ids_before(self, cutoff_timestamp: float) -> list[str]: ... + def list_document_file_paths(self, session_id: str) -> list[str]: ... + + def list_sessions( + self, + limit: int = 20, + owner_user_id: Optional[str] = None, + ) -> list[dict]: ... def add_message( self, session_id: str, role: str, content: str, @@ -54,10 +80,32 @@ def get_total_tokens( self, session_id: Optional[str] = None, ) -> dict: ... def get_migration_status(self) -> dict: ... + def consume_v05_import( + self, + request: dict, + payloads: list[dict], + ) -> dict: ... + def get_v05_import(self, import_id: str) -> Optional[dict]: ... + def read_v05_import_bundle( + self, + import_id: str, + *, + expected_binding_sha256: str, + expected_content_sha256: str, + expected_owner_id: str, + expected_owner_scope_sha256: str, + expected_target_session_id: str, + ) -> Optional[dict]: ... # Route-called core workflow persistence. Both SQLite and Postgres must # implement these before DB_ENGINE=postgres can be production-default safe. def save_operation_plan(self, plan: dict) -> None: ... + def compare_and_swap_operation_plan( + self, + plan_id: str, + expected_plan: dict, + updated_plan: dict, + ) -> bool: ... def get_operation_plan(self, plan_id: str) -> Optional[dict]: ... def update_operation_plan_status(self, plan_id: str, status: str) -> Optional[dict]: ... def record_snapshot( @@ -72,6 +120,12 @@ def record_audit_event( plan_id: Optional[str] = None, snapshot_id: Optional[str] = None, ) -> str: ... + def record_enterprise_admin_audit( + self, + event_type: str, + payload: Optional[dict] = None, + ) -> str: ... + def list_enterprise_admin_audits(self, limit: int = 1000) -> list[dict]: ... def export_audit_csv(self, session_id: str) -> str: ... def record_apply_attempt( self, @@ -88,7 +142,7 @@ def complete_apply_attempt( self, *, plan_id: str, token_hash: str, status: str, payload: Optional[dict] = None, ) -> Optional[dict]: ... def record_apply_result(self, session_id: str, plan_id: str, status: str, payload: Optional[dict] = None) -> str: ... - def get_timeline(self, session_id: str) -> dict: ... + def get_timeline(self, session_id: str, limit: int = 50) -> dict: ... def delete_session(self, session_id: str) -> bool: ... # Document, citation, and benchmark persistence used by public routes. @@ -104,9 +158,56 @@ def save_document( document_id: Optional[str] = None, ) -> str: ... def get_document(self, document_id: str) -> Optional[dict]: ... - def save_citations(self, session_id: str, plan_id: str, citations: list[dict]) -> int: ... + def list_documents(self, session_id: str, limit: int = 20) -> list[dict]: ... + def save_citations( + self, + session_id: str, + citations: list[dict], + plan_id: Optional[str] = None, + ) -> int: ... def get_citations(self, session_id: str, plan_id: Optional[str] = None, limit: int = 200) -> list[dict]: ... def save_benchmark_run(self, run: dict) -> str: ... + def list_benchmark_runs(self, limit: int = 20) -> list[dict]: ... + def save_workbook_job(self, job: dict) -> str: ... + def get_workbook_job(self, run_id: str) -> Optional[dict]: ... + def list_workbook_jobs( + self, + limit: int = 50, + status: Optional[str] = None, + session_id: Optional[str] = None, + account_id: Optional[str] = None, + ) -> list[dict]: ... + + # Enterprise administration and API-key auth persistence used by + # enterprise routes and middleware. + def upsert_enterprise_user( + self, + user_id: str, + email: str, + name: str, + role: str, + status: Optional[str] = None, + mfa_enabled: Optional[bool] = None, + metadata: Optional[dict] = None, + oidc_issuer: Optional[str] = None, + oidc_subject: Optional[str] = None, + ) -> dict: ... + def get_enterprise_user(self, user_id: str) -> Optional[dict]: ... + def get_enterprise_user_by_email(self, email: str) -> Optional[dict]: ... + def get_enterprise_user_by_oidc_identity(self, issuer: str, subject: str) -> Optional[dict]: ... + def list_enterprise_users(self, limit: int = 100) -> list[dict]: ... + def create_enterprise_team(self, name: str, metadata: Optional[dict] = None, team_id: Optional[str] = None) -> dict: ... + def get_enterprise_team(self, team_id: str) -> Optional[dict]: ... + def list_enterprise_teams(self, limit: int = 100) -> list[dict]: ... + def add_enterprise_team_member(self, team_id: str, user_id: str, role: str = "member") -> bool: ... + def create_enterprise_api_key(self, user_id: str, label: str = "") -> dict: ... + def revoke_enterprise_api_key(self, key_id: str) -> bool: ... + def get_enterprise_user_by_api_key(self, api_key: str) -> Optional[dict]: ... + def list_enterprise_api_keys(self, user_id: Optional[str] = None) -> list[dict]: ... + def set_enterprise_config(self, key: str, value: dict) -> dict: ... + def get_enterprise_config(self, key: str) -> dict: ... + def enterprise_dashboard(self) -> dict: ... + def purge_sessions_before(self, cutoff_timestamp: float) -> int: ... # V3/V4 workbook-accuracy artifact methods. SQLite implements these today; # Postgres should reach parity before server deployment becomes the default. @@ -117,6 +218,33 @@ def save_operation_simulation(self, session_id: str, simulation: dict, *, plan_i def save_workbook_invariant_run(self, session_id: str, report: dict, *, run_id: str = "", plan_id: str = "") -> str: ... def save_agent_run(self, run: dict) -> str: ... def get_agent_run(self, run_id: str) -> Optional[dict]: ... + def append_agent_run_event( + self, + run_id: str, + *, + expected_sequence: int, + event_type: str, + event_payload: dict, + updated_run: dict, + idempotency_key: str = "", + ) -> dict: ... + def list_agent_run_events( + self, + run_id: str, + after_sequence: int = 0, + limit: int = 100, + ) -> dict: ... + def save_agent_egress_preview(self, session_id: str, preview: dict) -> str: ... + def consume_agent_egress_approval( + self, + session_id: str, + *, + approval_id: str, + approved_digest: str, + request_digest: str, + consumption_id: str, + now: Optional[float] = None, + ) -> dict: ... def save_workbook_truth_model(self, truth_model: dict) -> str: ... def get_workbook_truth_model(self, truth_model_id: str) -> Optional[dict]: ... def save_formula_graph_artifact(self, graph: dict) -> str: ... @@ -149,6 +277,12 @@ def save_workbook_benchmark_fixture_result(self, run_id: str, result: dict) -> s def save_agent_run_step_artifact_v3(self, run_id: str, session_id: str, step_id: str, artifact: dict) -> str: ... def save_operation_diff_preview_v2(self, session_id: str, diff_preview: dict) -> str: ... def save_source_cell_provenance_v2(self, session_id: str, provenance: dict) -> str: ... - def save_plan_revision(self, session_id: str, plan_id: str, revision: dict) -> str: ... + def save_plan_revision( + self, + plan_id: str, + revision_type: str, + reason: str, + plan: dict, + ) -> str: ... def save_verification_run(self, session_id: str, plan_id: str, status: str, report: dict) -> str: ... def save_operation_audits(self, session_id: str, plan_id: str, rows: list[dict]) -> int: ... diff --git a/backend/storage/v05_import_store.py b/backend/storage/v05_import_store.py new file mode 100644 index 0000000..c2ee56f --- /dev/null +++ b/backend/storage/v05_import_store.py @@ -0,0 +1,414 @@ +"""Typed, privacy-safe contract for the neutral v0.5 import store. + +This module deliberately does not create a runtime workbook or translate +legacy records into current domain models. It authenticates the exact staged +payload ledger and gives both storage engines one deterministic state machine +and receipt format for a later, separately reviewed consumer. +""" + +from __future__ import annotations + +import hashlib +import hmac +import json +import re +from typing import Any, Mapping + + +STAGE_SCHEMA_VERSION = "com.excelagent.migration.v05.v1" +NEUTRAL_RECEIPT_SCHEMA_VERSION = "com.excelagent.migration.v05.consume.v1" +NEUTRAL_IMPORT_STATES = frozenset( + {"validated_pending_import", "importing", "imported", "failed_retryable"} +) +PAYLOAD_KINDS = frozenset({"blob", "document", "session", "workflow_recipe"}) + +_HEX_SHA256 = re.compile(r"[0-9a-f]{64}") +_FAILURE_CODE = re.compile(r"[a-z][a-z0-9_]{0,63}") +_REQUEST_KEYS = frozenset( + { + "archiveSha256", + "bindingSha256", + "contentSha256", + "counts", + "importId", + "manifestSha256", + "ownerId", + "ownerScopeSha256", + "schemaVersion", + "stageReceiptSha256", + "targetSessionId", + } +) +_PAYLOAD_KEYS = frozenset({"content", "kind", "path", "sha256", "sizeBytes"}) +_COUNT_KEYS = frozenset({"documents", "sessions", "workflows"}) +_RECEIPT_BODY_KEYS = frozenset( + { + "archiveSha256", + "contentSha256", + "counts", + "importId", + "legacyStorageMutated", + "manifestSha256", + "neutralStorageOnly", + "ownerScopeSha256", + "reverseConversionAvailable", + "runtimeWorkbookCreated", + "schemaVersion", + "stageReceiptSha256", + "status", + "storedPayloads", + "targetSessionSha256", + } +) +_MAX_PAYLOADS = 8192 +_MAX_PAYLOAD_BYTES = 512 * 1024 * 1024 +_MAX_ENTRY_BYTES = 64 * 1024 * 1024 + +# Materialization is deliberately tighter than neutral archival intake. The +# reviewed reader returns exact bytes in one bounded dictionary, so it refuses +# large archival bundles rather than transiently allocating the 512 MiB intake +# ceiling. A future streaming/callback protocol may raise these limits after +# its lifecycle snapshot and backpressure semantics are independently reviewed. +MATERIALIZATION_MAX_PAYLOADS = 2048 +MATERIALIZATION_MAX_PAYLOAD_BYTES = 64 * 1024 * 1024 +MATERIALIZATION_MAX_ENTRY_BYTES = 16 * 1024 * 1024 + + +class NeutralV05ImportError(RuntimeError): + """Categorical failure that is safe for logs and operator JSON.""" + + def __init__(self, code: str) -> None: + normalized = str(code or "").strip().lower() + if _FAILURE_CODE.fullmatch(normalized) is None: + normalized = "neutral_import_failed" + self.code = normalized + super().__init__(normalized) + + +def canonical_json(value: Any, *, newline: bool = False) -> bytes: + try: + encoded = json.dumps( + value, + allow_nan=False, + ensure_ascii=False, + separators=(",", ":"), + sort_keys=True, + ).encode("utf-8") + except (TypeError, ValueError) as exc: + raise NeutralV05ImportError("invalid_import_contract") from exc + return encoded + (b"\n" if newline else b"") + + +def sha256_bytes(value: bytes) -> str: + return hashlib.sha256(value).hexdigest() + + +def sha256_text(value: str) -> str: + return sha256_bytes(value.encode("utf-8")) + + +def normalize_sha256(value: Any, *, code: str = "invalid_import_contract") -> str: + if not isinstance(value, str): + raise NeutralV05ImportError(code) + normalized = value.strip().lower() + if _HEX_SHA256.fullmatch(normalized) is None: + raise NeutralV05ImportError(code) + return normalized + + +def normalize_owner_id(value: Any) -> str: + if not isinstance(value, str): + raise NeutralV05ImportError("invalid_owner_scope") + normalized = value.strip() + if ( + not normalized + or len(normalized) > 512 + or any(ord(character) < 32 for character in normalized) + ): + raise NeutralV05ImportError("invalid_owner_scope") + return normalized + + +def neutral_import_id(content_sha256: str) -> str: + return f"v05-import-{normalize_sha256(content_sha256)}" + + +def neutral_session_id(content_sha256: str) -> str: + return f"v05-neutral-{normalize_sha256(content_sha256)}" + + +def _normalized_counts(value: Any) -> dict[str, int]: + if not isinstance(value, dict) or frozenset(value) != _COUNT_KEYS: + raise NeutralV05ImportError("invalid_import_contract") + counts: dict[str, int] = {} + for key in sorted(_COUNT_KEYS): + count = value.get(key) + if not isinstance(count, int) or isinstance(count, bool) or count < 0: + raise NeutralV05ImportError("invalid_import_contract") + counts[key] = count + return counts + + +def import_binding_fields(request: Mapping[str, Any]) -> dict[str, Any]: + return { + "archiveSha256": request["archiveSha256"], + "contentSha256": request["contentSha256"], + "counts": request["counts"], + "manifestSha256": request["manifestSha256"], + "ownerScopeSha256": request["ownerScopeSha256"], + "schemaVersion": request["schemaVersion"], + "stageReceiptSha256": request["stageReceiptSha256"], + } + + +def binding_sha256(request: Mapping[str, Any]) -> str: + return sha256_bytes(canonical_json(import_binding_fields(request))) + + +def normalize_import_bundle( + request: Mapping[str, Any], + payloads: list[dict[str, Any]], +) -> tuple[dict[str, Any], tuple[dict[str, Any], ...]]: + """Validate the complete contract again at the storage trust boundary.""" + + if not isinstance(request, Mapping) or frozenset(request) != _REQUEST_KEYS: + raise NeutralV05ImportError("invalid_import_contract") + if not isinstance(payloads, list) or not payloads or len(payloads) > _MAX_PAYLOADS: + raise NeutralV05ImportError("invalid_payload_ledger") + + owner_id = normalize_owner_id(request.get("ownerId")) + normalized: dict[str, Any] = { + "archiveSha256": normalize_sha256(request.get("archiveSha256")), + "contentSha256": normalize_sha256(request.get("contentSha256")), + "counts": _normalized_counts(request.get("counts")), + "manifestSha256": normalize_sha256(request.get("manifestSha256")), + "ownerId": owner_id, + "ownerScopeSha256": normalize_sha256(request.get("ownerScopeSha256")), + "schemaVersion": str(request.get("schemaVersion") or ""), + "stageReceiptSha256": normalize_sha256(request.get("stageReceiptSha256")), + } + if normalized["schemaVersion"] != STAGE_SCHEMA_VERSION: + raise NeutralV05ImportError("unsupported_import_schema") + if not hmac.compare_digest(normalized["ownerScopeSha256"], sha256_text(owner_id)): + raise NeutralV05ImportError("owner_scope_mismatch") + + normalized["importId"] = str(request.get("importId") or "") + normalized["targetSessionId"] = str(request.get("targetSessionId") or "") + if normalized["importId"] != neutral_import_id(normalized["contentSha256"]): + raise NeutralV05ImportError("invalid_import_contract") + if normalized["targetSessionId"] != neutral_session_id(normalized["contentSha256"]): + raise NeutralV05ImportError("invalid_import_contract") + + normalized_payloads: list[dict[str, Any]] = [] + seen_paths: set[str] = set() + total_bytes = 0 + for raw in payloads: + if not isinstance(raw, dict) or frozenset(raw) != _PAYLOAD_KEYS: + raise NeutralV05ImportError("invalid_payload_ledger") + path = raw.get("path") + kind = raw.get("kind") + content = raw.get("content") + size = raw.get("sizeBytes") + if ( + not isinstance(path, str) + or not path + or len(path) > 1024 + or path in seen_paths + or kind not in PAYLOAD_KINDS + or not isinstance(content, bytes) + or not isinstance(size, int) + or isinstance(size, bool) + or size < 0 + or size > _MAX_ENTRY_BYTES + or size != len(content) + ): + raise NeutralV05ImportError("invalid_payload_ledger") + digest = normalize_sha256(raw.get("sha256"), code="invalid_payload_ledger") + if not hmac.compare_digest(digest, sha256_bytes(content)): + raise NeutralV05ImportError("invalid_payload_ledger") + seen_paths.add(path) + total_bytes += size + if total_bytes > _MAX_PAYLOAD_BYTES: + raise NeutralV05ImportError("invalid_payload_ledger") + normalized_payloads.append( + { + "content": content, + "kind": str(kind), + "path": path, + "sha256": digest, + "sizeBytes": size, + } + ) + normalized_payloads.sort(key=lambda item: item["path"]) + if [item["path"] for item in normalized_payloads] != [item["path"] for item in payloads]: + raise NeutralV05ImportError("invalid_payload_ledger") + + descriptors = [ + { + "kind": item["kind"], + "path": item["path"], + "sha256": item["sha256"], + "size_bytes": item["sizeBytes"], + } + for item in normalized_payloads + ] + ledger_digest = sha256_bytes(canonical_json(descriptors, newline=True)) + if not hmac.compare_digest(ledger_digest, normalized["contentSha256"]): + raise NeutralV05ImportError("invalid_payload_ledger") + actual_counts = { + "documents": sum(item["kind"] == "document" for item in normalized_payloads), + "sessions": sum(item["kind"] == "session" for item in normalized_payloads), + "workflows": sum(item["kind"] == "workflow_recipe" for item in normalized_payloads), + } + if actual_counts != normalized["counts"]: + raise NeutralV05ImportError("invalid_payload_ledger") + + expected_binding = binding_sha256(normalized) + supplied_binding = normalize_sha256(request.get("bindingSha256")) + if not hmac.compare_digest(expected_binding, supplied_binding): + raise NeutralV05ImportError("invalid_import_contract") + normalized["bindingSha256"] = expected_binding + return normalized, tuple(normalized_payloads) + + +def build_import_receipt( + request: Mapping[str, Any], + *, + stored_payloads: int, +) -> dict[str, Any]: + body = { + "archiveSha256": request["archiveSha256"], + "contentSha256": request["contentSha256"], + "counts": dict(request["counts"]), + "importId": request["importId"], + "legacyStorageMutated": False, + "manifestSha256": request["manifestSha256"], + "neutralStorageOnly": True, + "ownerScopeSha256": request["ownerScopeSha256"], + "reverseConversionAvailable": False, + "runtimeWorkbookCreated": False, + "schemaVersion": NEUTRAL_RECEIPT_SCHEMA_VERSION, + "stageReceiptSha256": request["stageReceiptSha256"], + "status": "imported", + "storedPayloads": stored_payloads, + "targetSessionSha256": sha256_text(str(request["targetSessionId"])), + } + return { + "receipt": body, + "receiptSha256": sha256_bytes(canonical_json(body)), + } + + +def validate_import_receipt( + value: Any, + request: Mapping[str, Any], + *, + stored_payloads: int, +) -> dict[str, Any]: + if not isinstance(value, dict) or frozenset(value) != {"receipt", "receiptSha256"}: + raise NeutralV05ImportError("corrupt_import_receipt") + body = value.get("receipt") + if not isinstance(body, dict) or frozenset(body) != _RECEIPT_BODY_KEYS: + raise NeutralV05ImportError("corrupt_import_receipt") + expected = build_import_receipt(request, stored_payloads=stored_payloads) + if not hmac.compare_digest( + canonical_json(value), + canonical_json(expected), + ): + raise NeutralV05ImportError("corrupt_import_receipt") + return expected + + +def parse_receipt_json( + raw: Any, + request: Mapping[str, Any], + *, + stored_payloads: int, +) -> dict[str, Any]: + if isinstance(raw, dict): + if len(canonical_json(raw)) > 64 * 1024: + raise NeutralV05ImportError("corrupt_import_receipt") + return validate_import_receipt(raw, request, stored_payloads=stored_payloads) + if not isinstance(raw, str) or len(raw.encode("utf-8")) > 64 * 1024: + raise NeutralV05ImportError("corrupt_import_receipt") + try: + value = json.loads(raw, object_pairs_hook=_reject_duplicate_keys) + except (TypeError, ValueError) as exc: + raise NeutralV05ImportError("corrupt_import_receipt") from exc + return validate_import_receipt(value, request, stored_payloads=stored_payloads) + + +def _reject_duplicate_keys(pairs: list[tuple[str, Any]]) -> dict[str, Any]: + value: dict[str, Any] = {} + for key, item in pairs: + if key in value: + raise ValueError("duplicate key") + value[key] = item + return value + + +def ensure_row_binding(row: Mapping[str, Any], request: Mapping[str, Any]) -> None: + expected = { + "archive_sha256": request["archiveSha256"], + "binding_sha256": request["bindingSha256"], + "content_sha256": request["contentSha256"], + "import_id": request["importId"], + "manifest_sha256": request["manifestSha256"], + "owner_scope_sha256": request["ownerScopeSha256"], + "schema_version": request["schemaVersion"], + "session_id": request["targetSessionId"], + "stage_receipt_sha256": request["stageReceiptSha256"], + } + for key, expected_value in expected.items(): + actual = str(row.get(key) or "") + if not hmac.compare_digest(actual, str(expected_value)): + raise NeutralV05ImportError("import_binding_conflict") + counts_raw = row.get("counts_json") + if isinstance(counts_raw, str): + try: + counts_raw = json.loads(counts_raw, object_pairs_hook=_reject_duplicate_keys) + except (TypeError, ValueError) as exc: + raise NeutralV05ImportError("import_binding_conflict") from exc + if counts_raw != request["counts"]: + raise NeutralV05ImportError("import_binding_conflict") + + +def public_import_status( + row: Mapping[str, Any], + request: Mapping[str, Any], + *, + stored_payloads: int, +) -> dict[str, Any]: + ensure_row_binding(row, request) + state = str(row.get("state") or "") + if state not in NEUTRAL_IMPORT_STATES: + raise NeutralV05ImportError("corrupt_import_state") + version = row.get("version") + attempt_count = row.get("attempt_count") + if ( + not isinstance(version, int) + or isinstance(version, bool) + or version < 0 + or not isinstance(attempt_count, int) + or isinstance(attempt_count, bool) + or attempt_count < 0 + ): + raise NeutralV05ImportError("corrupt_import_state") + result: dict[str, Any] = { + "attemptCount": attempt_count, + "importId": request["importId"], + "state": state, + "version": version, + } + if state == "imported": + result["receipt"] = parse_receipt_json( + row.get("receipt_json"), + request, + stored_payloads=stored_payloads, + ) + elif state == "failed_retryable": + failure_code = str(row.get("failure_code") or "") + if _FAILURE_CODE.fullmatch(failure_code) is None: + raise NeutralV05ImportError("corrupt_import_state") + result["failureCode"] = failure_code + return result diff --git a/backend/templates/composer.py b/backend/templates/composer.py index 96b6203..2e9832f 100644 --- a/backend/templates/composer.py +++ b/backend/templates/composer.py @@ -10,11 +10,11 @@ from __future__ import annotations from dataclasses import dataclass, field -from typing import Optional +from typing import Any, Optional import structlog -from backend.templates.registry import TemplateRegistry, get_registry +from backend.templates.registry import generate_template logger = structlog.get_logger() @@ -41,7 +41,7 @@ class TemplateOutput: # Maps template_id → list of named outputs (relative to start_cell) # These are the "plugs" that other templates can connect to. -TEMPLATE_OUTPUTS = { +TEMPLATE_OUTPUTS: dict[str, list[TemplateOutput]] = { "dcf_model": [ TemplateOutput("enterprise_value", "B28", "Implied Enterprise Value", "number"), TemplateOutput("equity_value", "B29", "Implied Equity Value", "number"), @@ -76,7 +76,7 @@ class TemplateOutput: # ── Pre-defined Compositions ───────────────────────────────────────────── -COMPOSITIONS = { +COMPOSITIONS: dict[str, dict[str, Any]] = { "full_valuation": { "name": "Full Valuation Package", "description": "3-Statement → DCF → Sensitivity — fully linked", @@ -125,8 +125,7 @@ class TemplateOutput: class TemplateComposer: """Combines multiple templates into inter-linked workbook models.""" - def __init__(self, registry: Optional[TemplateRegistry] = None): - self.registry = registry or get_registry() + def __init__(self): self._outputs = TEMPLATE_OUTPUTS self._compositions = COMPOSITIONS @@ -171,21 +170,18 @@ def compose( comp = self._compositions[composition_id] params = params or {} - all_operations = [] - output_cell_map = {} # "template.output_name" → "Sheet!Cell" + all_operations: list[dict[str, Any]] = [] + output_cell_map: dict[str, str] = {} # "template.output_name" → "Sheet!Cell" # ── Step 1: Generate operations for each template ──────────── for placement in comp["templates"]: template_params = params.get(placement.template_id, placement.params) - # Generate template operations via registry - template = self.registry.get_template(placement.template_id) - if not template: + try: + ops = generate_template(placement.template_id, template_params) + except KeyError: logger.warning("template_not_found", id=placement.template_id) continue - - try: - ops = template.generate(template_params) except Exception as e: logger.error("template_generate_failed", id=placement.template_id, error=str(e)) continue @@ -228,9 +224,9 @@ def compose( def compose_custom( self, - placements: list[dict], - links: list[dict] = None, - ) -> list[dict]: + placements: list[dict[str, Any]], + links: list[dict[str, str]] | None = None, + ) -> list[dict[str, Any]]: """Generate a custom composition from arbitrary template placements. Args: diff --git a/backend/templates/finance/amortization.py b/backend/templates/finance/amortization.py index cb380a8..1fa9d06 100644 --- a/backend/templates/finance/amortization.py +++ b/backend/templates/finance/amortization.py @@ -7,6 +7,8 @@ - Summary statistics """ +from typing import Any + TEMPLATE_META = { "id": "amortization", "name": "Loan Amortization Schedule", @@ -18,7 +20,7 @@ } -def generate(params: dict = None) -> list[dict]: +def generate(params: dict[str, Any] | None = None) -> list[dict[str, Any]]: """Generate a loan amortization schedule.""" p = params or {} principal = p.get("principal", 500_000) @@ -27,8 +29,7 @@ def generate(params: dict = None) -> list[dict]: start_date = p.get("start_date", "2026-01-01") months = years * 12 - monthly_rate = annual_rate / 12 - ops = [] + ops: list[dict[str, Any]] = [] cur = '$#,##0.00;($#,##0.00);"-"' # ── Title ──────────────────────────────────────────────────── diff --git a/backend/templates/finance/comps_analysis.py b/backend/templates/finance/comps_analysis.py index 64ddd09..8b27396 100644 --- a/backend/templates/finance/comps_analysis.py +++ b/backend/templates/finance/comps_analysis.py @@ -13,7 +13,7 @@ from __future__ import annotations from dataclasses import dataclass -from typing import Optional +from typing import Any, Optional import structlog @@ -31,7 +31,7 @@ } -def generate(params: dict = None) -> list[dict]: +def generate(params: dict[str, Any] | None = None) -> list[dict[str, Any]]: """Registry-compatible entry point.""" return generate_comps_analysis(**(params or {})) @@ -126,8 +126,7 @@ def add(cell, value, type="input", fmt=None, formula=None): r += 1 # ── Peer data rows ─────────────────────────────────────────────── - peer_start_row = r - multiples = {"ev_rev": [], "ev_ebitda": [], "pe": []} + multiples: dict[str, list[float]] = {"ev_rev": [], "ev_ebitda": [], "pe": []} for peer in peers: market_cap = peer.share_price * peer.shares_outstanding diff --git a/backend/templates/finance/dcf_model.py b/backend/templates/finance/dcf_model.py index 807922c..0a0132f 100644 --- a/backend/templates/finance/dcf_model.py +++ b/backend/templates/finance/dcf_model.py @@ -8,6 +8,8 @@ - Enterprise Value → Equity Value bridge """ +from typing import Any + TEMPLATE_META = { "id": "dcf_model", "name": "Discounted Cash Flow (DCF)", @@ -19,7 +21,7 @@ } -def generate(params: dict = None) -> list[dict]: +def generate(params: dict[str, Any] | None = None) -> list[dict[str, Any]]: """Generate all cell operations for a DCF model. Params (all optional): @@ -50,7 +52,7 @@ def generate(params: dict = None) -> list[dict]: net_debt = p.get("net_debt", 50_000_000) shares = p.get("shares_outstanding", 10_000_000) - ops = [] + ops: list[dict[str, Any]] = [] # ── Section: Title ─────────────────────────────────────────── ops.append({"cell": "A1", "value": "Discounted Cash Flow Analysis", "type": "label", diff --git a/backend/templates/finance/lbo_model.py b/backend/templates/finance/lbo_model.py index 7c5d5df..2ff4672 100644 --- a/backend/templates/finance/lbo_model.py +++ b/backend/templates/finance/lbo_model.py @@ -7,6 +7,8 @@ - Returns analysis (IRR, MOIC) """ +from typing import Any + TEMPLATE_META = { "id": "lbo_model", "name": "Leveraged Buyout (LBO)", @@ -18,7 +20,7 @@ } -def generate(params: dict = None) -> list[dict]: +def generate(params: dict[str, Any] | None = None) -> list[dict[str, Any]]: """Generate all cell operations for an LBO model.""" p = params or {} ebitda = p.get("entry_ebitda", 50_000_000) @@ -31,7 +33,7 @@ def generate(params: dict = None) -> list[dict]: tax_rate = p.get("tax_rate", 0.25) da_pct = p.get("da_pct_ebitda", 0.15) - ops = [] + ops: list[dict[str, Any]] = [] cur = '$#,##0;($#,##0);"-"' pct = "0.0%" mult = "0.0x" diff --git a/backend/templates/finance/merger_model.py b/backend/templates/finance/merger_model.py index 3877541..06a2e23 100644 --- a/backend/templates/finance/merger_model.py +++ b/backend/templates/finance/merger_model.py @@ -14,9 +14,9 @@ from __future__ import annotations -from typing import Optional import structlog +from typing import Any logger = structlog.get_logger() @@ -32,7 +32,7 @@ } -def generate(params: dict = None) -> list[dict]: +def generate(params: dict[str, Any] | None = None) -> list[dict[str, Any]]: """Registry-compatible entry point.""" allowed = generate_merger_model.__code__.co_varnames[:generate_merger_model.__code__.co_argcount] filtered = {key: value for key, value in (params or {}).items() if key in allowed} @@ -91,7 +91,6 @@ def add(cell, value, type="input", fmt=None, formula=None): HEADER_FMT = {"bold": True, "bg_color": "#1B3A5C", "font_color": "#FFFFFF", "font_size": 12} SECTION_FMT = {"bold": True, "bg_color": "#D9E2F3", "border_bottom": "medium"} - LABEL_FMT = {} INPUT_FMT = {"font_color": "#0000FF", "number_format": "$#,##0"} PCT_FMT = {"font_color": "#0000FF", "number_format": "0.0%"} CALC_FMT = {"number_format": "$#,##0", "font_color": "#000000"} diff --git a/backend/templates/finance/sensitivity.py b/backend/templates/finance/sensitivity.py index 0663973..f62166e 100644 --- a/backend/templates/finance/sensitivity.py +++ b/backend/templates/finance/sensitivity.py @@ -6,6 +6,8 @@ - Conditional formatting indicators """ +from typing import Any + TEMPLATE_META = { "id": "sensitivity", "name": "Sensitivity Analysis Table", @@ -17,7 +19,7 @@ } -def generate(params: dict = None) -> list[dict]: +def generate(params: dict[str, Any] | None = None) -> list[dict[str, Any]]: """Generate a 2D sensitivity table.""" p = params or {} title = p.get("title", "Sensitivity Analysis — Implied Share Price") @@ -31,7 +33,7 @@ def generate(params: dict = None) -> list[dict]: col_step = p.get("col_step", 0.005) base_value = p.get("base_value", 25.00) - ops = [] + ops: list[dict[str, Any]] = [] cur = "$#,##0.00" pct = "0.0%" @@ -86,9 +88,9 @@ def generate(params: dict = None) -> list[dict]: row_mid = (row_start + row_end) / 2 col_mid = (col_start + col_end) / 2 - for i, rv in enumerate(row_values): + for i, _rv in enumerate(row_values): row = 7 + i - for j, cv in enumerate(col_values): + for j, _cv in enumerate(col_values): col = chr(67 + j) # C, D, E, ... # Reference to row/col input cells for DATA TABLE functionality row_ref = f"$B${row}" diff --git a/backend/templates/finance/three_statement.py b/backend/templates/finance/three_statement.py index 6276075..2c24c52 100644 --- a/backend/templates/finance/three_statement.py +++ b/backend/templates/finance/three_statement.py @@ -10,6 +10,8 @@ - Working Capital Schedule """ +from typing import Any + TEMPLATE_META = { "id": "three_statement", "name": "3-Statement Financial Model", @@ -21,7 +23,7 @@ } -def generate(params: dict = None) -> list[dict]: +def generate(params: dict[str, Any] | None = None) -> list[dict[str, Any]]: """Generate all cell operations for a 3-statement model.""" p = params or {} base_rev = p.get("base_revenue", 100_000_000) @@ -40,7 +42,7 @@ def generate(params: dict = None) -> list[dict]: initial_debt = p.get("initial_debt", 40_000_000) initial_cash = p.get("initial_cash", 15_000_000) - ops = [] + ops: list[dict[str, Any]] = [] cur = '$#,##0;($#,##0);"-"' pct = "0.0%" diff --git a/backend/templates/registry.py b/backend/templates/registry.py index 6e44835..8b265b5 100644 --- a/backend/templates/registry.py +++ b/backend/templates/registry.py @@ -11,6 +11,7 @@ import importlib import pkgutil from pathlib import Path +from types import ModuleType from typing import Optional import structlog @@ -18,7 +19,225 @@ logger = structlog.get_logger() # Registry: template_id → module -_registry: dict[str, object] = {} +_registry: dict[str, ModuleType] = {} + +GUIDED_FINANCE_WORKFLOWS: dict[str, dict] = { + "dcf_model": { + "workflow_id": "guided_dcf_valuation", + "title": "Guided DCF Valuation", + "use_case": "Build an intrinsic valuation from historical financials and explicit operating assumptions.", + "automation_prompt": "Build a DCF using historical revenue, margin, capex, NWC, WACC, terminal growth, net debt, and diluted shares.", + "required_inputs": [ + "base_revenue", + "revenue_growth", + "ebit_margin_or_cost_structure", + "tax_rate", + "wacc", + "terminal_growth", + "net_debt", + "shares_outstanding", + ], + "setup_steps": [ + "Load or paste historical income statement and cash flow drivers.", + "Confirm projection assumptions for revenue, margins, capex, working capital, WACC, and terminal growth.", + "Generate the DCF model and review enterprise value, equity value, and implied share price.", + ], + "key_outputs": [ + "enterprise_value", + "equity_value", + "terminal_value", + "implied_price_per_share", + ], + "quality_checks": [ + "Terminal growth remains below WACC.", + "Free cash flow sign and margin trends are directionally plausible.", + "Net debt and share count are populated before relying on equity value.", + ], + }, + "lbo_model": { + "workflow_id": "guided_lbo_returns", + "title": "Guided LBO Returns", + "use_case": "Evaluate private-equity entry, financing, deleveraging, and exit returns.", + "automation_prompt": "Build an LBO with purchase multiple, leverage, interest rate, EBITDA growth, capex, taxes, and exit multiple.", + "required_inputs": [ + "entry_ebitda", + "entry_multiple", + "leverage_turns", + "interest_rate", + "ebitda_growth", + "exit_multiple", + ], + "setup_steps": [ + "Set entry valuation and debt funding assumptions.", + "Project EBITDA, cash flow available for debt paydown, and ending debt balances.", + "Review MOIC and IRR across hold periods and exit assumptions.", + ], + "key_outputs": [ + "entry_enterprise_value", + "equity_contribution", + "ending_debt", + "moic", + "irr", + ], + "quality_checks": [ + "Sources equal uses before reviewing returns.", + "Debt paydown does not exceed available free cash flow.", + "Exit multiple and entry multiple assumptions are clearly visible.", + ], + }, + "three_statement": { + "workflow_id": "guided_three_statement_model", + "title": "Guided 3-Statement Model", + "use_case": "Create an integrated income statement, balance sheet, and cash flow forecast.", + "automation_prompt": "Build a linked 3-statement model with revenue growth, cost structure, working capital, capex, debt, interest, and tax assumptions.", + "required_inputs": [ + "base_revenue", + "revenue_growth", + "cost_structure", + "working_capital_days", + "capex_pct", + "initial_debt", + "initial_cash", + ], + "setup_steps": [ + "Populate the base year P&L, working capital days, debt, cash, and capex assumptions.", + "Generate linked income statement, balance sheet, and cash flow sections.", + "Review balance checks and cash flow linkage before exporting or using as an input to valuation templates.", + ], + "key_outputs": [ + "revenue", + "ebitda", + "net_income", + "total_assets", + "cash_flow_from_operations", + "balance_check", + ], + "quality_checks": [ + "Balance sheet check is zero for every projected year.", + "Cash flow statement links back to net income and working capital movements.", + "Interest expense and debt assumptions are consistent.", + ], + }, + "comps_analysis": { + "workflow_id": "guided_comps_analysis", + "title": "Guided Trading Comps", + "use_case": "Benchmark a subject company against public peers using trading multiples.", + "automation_prompt": "Build a comparable company analysis with peer prices, shares, net debt, revenue, EBITDA, net income, multiples, and implied valuation range.", + "required_inputs": [ + "subject_revenue", + "subject_ebitda", + "subject_net_income", + "subject_net_debt", + "subject_shares", + "peer_group", + ], + "setup_steps": [ + "Select the peer group and load market data, share counts, net debt, and financial metrics.", + "Generate EV/Revenue, EV/EBITDA, P/E, margin, and growth statistics.", + "Review implied enterprise value, equity value, and share price ranges.", + ], + "key_outputs": [ + "median_ev_revenue", + "median_ev_ebitda", + "median_pe", + "implied_enterprise_value", + "implied_share_price", + ], + "quality_checks": [ + "Peer financial metrics use the same period basis.", + "Outlier multiples are reviewed before using the median range.", + "Subject net debt and shares reconcile to the valuation bridge.", + ], + }, + "merger_model": { + "workflow_id": "guided_merger_accretion_dilution", + "title": "Guided Merger Accretion/Dilution", + "use_case": "Assess acquisition consideration, pro forma earnings, goodwill, synergies, and EPS impact.", + "automation_prompt": "Build a merger model with acquirer and target share prices, shares, net income, premium, cash-stock mix, financing cost, tax rate, and synergies.", + "required_inputs": [ + "acquirer_price", + "acquirer_shares", + "acquirer_net_income", + "target_price", + "target_shares", + "target_net_income", + "premium_pct", + "pct_cash", + "pct_stock", + "synergies", + ], + "setup_steps": [ + "Load acquirer and target market data, earnings, consideration mix, and premium assumptions.", + "Generate transaction summary, sources and uses, pro forma income statement, and synergy schedule.", + "Review accretion or dilution, goodwill, and financing assumptions.", + ], + "key_outputs": [ + "offer_price", + "equity_value", + "pro_forma_eps", + "accretion_dilution_pct", + "goodwill", + ], + "quality_checks": [ + "Cash and stock consideration add to total equity value.", + "Pro forma share count reflects stock consideration.", + "Synergy and financing assumptions are explicit before interpreting accretion.", + ], + }, + "sensitivity": { + "workflow_id": "guided_sensitivity_table", + "title": "Guided Sensitivity Table", + "use_case": "Create a two-variable sensitivity table around valuation or returns drivers.", + "automation_prompt": "Build a sensitivity table using a base case output, row variable, column variable, ranges, and step sizes.", + "required_inputs": [ + "base_value", + "row_label", + "row_start", + "row_end", + "row_step", + "col_label", + "col_start", + "col_end", + "col_step", + ], + "setup_steps": [ + "Choose the base case output cell and the two variables to sensitize.", + "Set row and column ranges with practical step sizes.", + "Generate the table and review the highlighted base case against the source model.", + ], + "key_outputs": [ + "base_case", + "row_axis", + "column_axis", + "sensitivity_matrix", + ], + "quality_checks": [ + "Base case ties to the source model output.", + "Axis ranges bracket the base case assumption.", + "Step sizes produce a readable table without hiding key downside or upside cases.", + ], + }, +} + + +def _template_payload(template_id: str, meta: dict) -> dict: + """Return API-facing metadata for a template.""" + payload = { + "id": template_id, + "name": meta.get("name", template_id), + "category": meta.get("category", "General"), + "description": meta.get("description", ""), + "complexity": meta.get("complexity", "medium"), + "estimated_cells": meta.get("estimated_cells", 0), + "tags": meta.get("tags", []), + } + + guided_workflow = meta.get("guided_workflow") or GUIDED_FINANCE_WORKFLOWS.get(template_id) + if guided_workflow: + payload["workflow_id"] = guided_workflow["workflow_id"] + payload["guided_workflow"] = guided_workflow + + return payload def _discover_templates(): @@ -33,7 +252,7 @@ def _discover_templates(): package_name = "backend.templates.finance" - for finder, name, is_pkg in pkgutil.iter_modules([str(finance_dir)]): + for _, name, _ in pkgutil.iter_modules([str(finance_dir)]): if name.startswith("_"): continue try: @@ -60,15 +279,7 @@ def list_templates() -> list[dict]: templates = [] for tid, module in _registry.items(): meta = getattr(module, "TEMPLATE_META", {}) - templates.append({ - "id": tid, - "name": meta.get("name", tid), - "category": meta.get("category", "General"), - "description": meta.get("description", ""), - "complexity": meta.get("complexity", "medium"), - "estimated_cells": meta.get("estimated_cells", 0), - "tags": meta.get("tags", []), - }) + templates.append(_template_payload(tid, meta)) return sorted(templates, key=lambda t: t["name"]) @@ -83,15 +294,7 @@ def get_template(template_id: str) -> Optional[dict]: return None meta = getattr(module, "TEMPLATE_META", {}) - return { - "id": template_id, - "name": meta.get("name", template_id), - "category": meta.get("category", "General"), - "description": meta.get("description", ""), - "complexity": meta.get("complexity", "medium"), - "estimated_cells": meta.get("estimated_cells", 0), - "tags": meta.get("tags", []), - } + return _template_payload(template_id, meta) def generate_template(template_id: str, params: Optional[dict] = None) -> list[dict]: @@ -114,7 +317,7 @@ def generate_template(template_id: str, params: Optional[dict] = None) -> list[d if not module: raise KeyError(f"Template '{template_id}' not found. Available: {list(_registry.keys())}") - gen_fn = getattr(module, "generate") + gen_fn = module.generate ops = gen_fn(params or {}) logger.info("template_generated", template=template_id, ops=len(ops), diff --git a/backend/tests/benchmarks.py b/backend/tests/benchmarks.py index c588951..c641842 100644 --- a/backend/tests/benchmarks.py +++ b/backend/tests/benchmarks.py @@ -14,7 +14,6 @@ from __future__ import annotations import json -import math import re import time from dataclasses import asdict, dataclass, field @@ -314,7 +313,7 @@ async def _run_case(self, case: BenchmarkCase, pipeline_fn) -> BenchmarkResult: """Execute a single benchmark case.""" start = time.time() errors = [] - details = [] + details: list[dict] = [] # Execute through the pipeline actual_operations = [] diff --git a/backend/tests/conftest.py b/backend/tests/conftest.py new file mode 100644 index 0000000..b29333b --- /dev/null +++ b/backend/tests/conftest.py @@ -0,0 +1,8 @@ +from __future__ import annotations + +import asyncio +import sys + + +if sys.platform == "win32" and hasattr(asyncio, "WindowsSelectorEventLoopPolicy"): + asyncio.set_event_loop_policy(asyncio.WindowsSelectorEventLoopPolicy()) diff --git a/backend/tests/contract/test_dependency_risk_policy.py b/backend/tests/contract/test_dependency_risk_policy.py index 4248b2f..7f6cdaa 100644 --- a/backend/tests/contract/test_dependency_risk_policy.py +++ b/backend/tests/contract/test_dependency_risk_policy.py @@ -2,18 +2,275 @@ from datetime import date from pathlib import Path +from backend.benchmarks import v7_readiness +from scripts import check_dependency_risks +from backend.benchmarks.v7_readiness import ( + _dependency_risk, + _hyperformula_license_risk_status, + _sheetjs_xlsx_risk_status, +) + ROOT = Path(__file__).resolve().parents[3] -def test_xlsx_dependency_risk_is_allowlisted_with_owner_mitigation_and_expiry(): +def test_resolved_xlsx_dependency_no_longer_needs_an_allowlist_exception(): payload = json.loads((ROOT / "security" / "dependency-risk-allowlist.json").read_text(encoding="utf-8")) - risks = payload["acceptedRisks"] - xlsx = next(item for item in risks if item["package"] == "xlsx") - - assert xlsx["schemaVersion"] if "schemaVersion" in xlsx else payload["schemaVersion"] - assert set(xlsx["advisoryIds"]) == {"GHSA-4r6h-8v6p-xvw6", "GHSA-5pgg-2g8v-p4x9"} - assert xlsx["owner"] - assert xlsx["mitigations"] - assert date.fromisoformat(xlsx["expiresOn"]) >= date.today() - assert "workbook-worker/package-lock.json" in xlsx["paths"] + worker = json.loads((ROOT / "workbook-worker" / "package-lock.json").read_text(encoding="utf-8")) + locked_xlsx = worker["packages"]["node_modules/xlsx"] + + assert payload["acceptedRisks"] == [] + assert locked_xlsx["version"] == "0.20.3" + assert locked_xlsx["resolved"].startswith("https://cdn.sheetjs.com/xlsx-0.20.3/") + + +def test_dependency_risk_checker_reports_precise_xlsx_expiry_countdown(monkeypatch, capsys): + monkeypatch.setattr(check_dependency_risks, "_load_allowlist", _xlsx_allowlist) + monkeypatch.setattr(check_dependency_risks, "_npm_audit_findings", _xlsx_audit_findings) + + assert check_dependency_risks.main(today=date(2026, 8, 1)) == 0 + + captured = capsys.readouterr() + assert "Dependency risk policy passed" in captured.out + assert "workbook-worker/package-lock.json: accepted xlsx GHSA-4R6H-8V6P-XVW6, GHSA-5PGG-2G8V-P4X9" in captured.out + assert "expiresOn=2026-08-31" in captured.out + assert "daysUntilExpiry=30" in captured.out + assert "releaseBlocker=expires_on_or_before_release" in captured.out + + +def test_dependency_risk_checker_blocks_xlsx_on_expiry_day(monkeypatch, capsys): + monkeypatch.setattr(check_dependency_risks, "_load_allowlist", _xlsx_allowlist) + monkeypatch.setattr(check_dependency_risks, "_npm_audit_findings", _xlsx_audit_findings) + + assert check_dependency_risks.main(today=date(2026, 8, 31)) == 1 + + captured = capsys.readouterr() + assert "Dependency risk policy passed" not in captured.out + assert "dependency risk allowlist release blocker for xlsx" in captured.err + assert "expiresOn=2026-08-31" in captured.err + assert "daysUntilExpiry=0" in captured.err + assert "releaseBlocker=expires_on_or_before_release" in captured.err + + +def test_v7_dependency_risk_reports_pinned_sheetjs_as_remediated(tmp_path, monkeypatch): + risk = _dependency_risk(corpus_path=_matched_corpus_artifact(tmp_path)) + + assert risk["status"] == "passed" + assert risk["sheetjsAdvisoryStatus"] == "remediated" + assert risk["hyperformulaLicenseStatus"]["status"] == "not_required" + assert risk["tracked"]["sheetjsAcceptedRisk"] is True + assert risk["tracked"]["hyperformulaLicenseApproved"] is True + assert risk["tracked"]["hyperformulaLicenseKey"] is True + assert risk["tracked"]["sheetjsParserComparisonAdapter"] is True + assert risk["tracked"]["sheetjsParserComparisonCorpus"] is True + assert risk["tracked"]["sheetjsWorkerHardening"] is True + assert risk["parserComparisonCorpus"]["status"] == "matched" + assert "sheetjs_xlsx_high_severity_advisory_no_npm_fix" not in risk["unresolved"] + + +def test_sheetjs_remediation_supersedes_an_expired_allowlist_entry(): + status = _sheetjs_xlsx_risk_status( + {"acceptedRisks": _xlsx_allowlist()}, + "MAX_PAYLOAD_BYTES ALLOWED_EXTENSIONS assertOoxmlMagic MAX_OOXML_FILE_COUNT ZIP_ENCRYPTED_FLAG ZIP_DATA_DESCRIPTOR_FLAG", + "ExcelJS parserComparisonSnapshot parserComparisonCorpus comparisonOnly replacementReady: false", + {"status": "matched", "fixtureCount": 1, "materialMismatchCount": 0, "failed": 0}, + today=date(2026, 8, 31), + remediated=True, + ) + + assert status["status"] == "remediated" + assert status["releaseBlocker"] is False + + +def test_v7_dependency_risk_blocks_hyperformula_without_explicit_license_evidence(tmp_path, monkeypatch): + monkeypatch.setattr(v7_readiness, "HYPERFORMULA_ENABLED", True) + monkeypatch.setattr(v7_readiness, "HYPERFORMULA_DISTRIBUTED", True) + monkeypatch.setattr(v7_readiness, "HYPERFORMULA_LICENSE_APPROVED", False) + monkeypatch.setattr(v7_readiness, "HYPERFORMULA_LICENSE_KEY", "") + + risk = _hyperformula_license_risk_status({"hyperformula": "3.2.0"}) + + assert risk["status"] == "blocked_unapproved" + assert "hyperformula_license_approval_missing" in risk["unresolved"] + assert "hyperformula_commercial_license_key_missing_or_invalid" in risk["unresolved"] + assert risk["releaseBlocker"] is True + + +def test_v7_dependency_risk_rejects_gpl_sentinel_as_commercial_evidence(monkeypatch): + monkeypatch.setattr(v7_readiness, "HYPERFORMULA_ENABLED", True) + monkeypatch.setattr(v7_readiness, "HYPERFORMULA_DISTRIBUTED", True) + monkeypatch.setattr(v7_readiness, "HYPERFORMULA_LICENSE_APPROVED", True) + monkeypatch.setattr(v7_readiness, "HYPERFORMULA_LICENSE_KEY", "gpl-v3") + + risk = _hyperformula_license_risk_status({"hyperformula": "3.2.0"}) + + assert risk["status"] == "blocked_unapproved" + assert risk["keyEvidence"] is False + assert "hyperformula_commercial_license_key_missing_or_invalid" in risk["unresolved"] + + +def test_v7_dependency_risk_blocks_sheetjs_when_acceptance_expires(): + worker = """ +MAX_PAYLOAD_BYTES +ALLOWED_EXTENSIONS +assertOoxmlMagic +MAX_OOXML_FILE_COUNT +ZIP_ENCRYPTED_FLAG +ZIP_DATA_DESCRIPTOR_FLAG +""" + parser = """ +ExcelJS +parserComparisonSnapshot +comparisonOnly +replacementReady: false +""" + + status = _sheetjs_xlsx_risk_status( + {"acceptedRisks": _xlsx_allowlist()}, + worker, + parser, + {"status": "matched", "fixtureCount": 2, "materialMismatchCount": 0, "failed": 0}, + today=date(2026, 8, 31), + ) + + assert status["status"] == "expired" + assert status["daysUntilExpiry"] == 0 + + +def test_v7_dependency_risk_blocks_sheetjs_when_corpus_is_missing(): + worker = """ +MAX_PAYLOAD_BYTES +ALLOWED_EXTENSIONS +assertOoxmlMagic +MAX_OOXML_FILE_COUNT +ZIP_ENCRYPTED_FLAG +ZIP_DATA_DESCRIPTOR_FLAG +""" + parser = """ +ExcelJS +parserComparisonSnapshot +parserComparisonCorpus +comparisonOnly +replacementReady: false +""" + + status = _sheetjs_xlsx_risk_status( + {"acceptedRisks": _xlsx_allowlist()}, + worker, + parser, + {"status": "missing", "fixtureCount": 0, "materialMismatchCount": 0, "failed": 0}, + today=date(2026, 8, 1), + ) + + assert status["status"] == "missing_or_mismatched_parser_comparison_corpus" + assert status["comparisonCorpusEvidence"] is False + + +def test_v7_readiness_cli_uses_explicit_parser_corpus_and_fails_warnings(tmp_path, monkeypatch, capsys): + corpus = tmp_path / "parser-corpus.json" + seen: dict[str, object] = {} + + class DummyReport: + status = "warning" + + def model_dump(self, **_kwargs): + return {"status": self.status} + + async def fake_run_v7_readiness(**kwargs): + seen.update(kwargs) + print("incidental benchmark log") + return DummyReport() + + monkeypatch.setattr(v7_readiness, "run_v7_readiness", fake_run_v7_readiness) + + exit_code = v7_readiness.main( + [ + "--replay-limit", + "0", + "--parser-comparison-corpus", + str(corpus), + "--require-private-fixtures", + "--fail-on-warning", + ] + ) + + assert exit_code == 1 + assert seen["persist"] is False + assert seen["replay_limit"] == 0 + assert seen["parser_corpus_path"] == corpus + assert seen["replay_baseline_id"] is None + assert seen["require_private_fixtures"] is True + captured = capsys.readouterr() + assert json.loads(captured.out)["status"] == "warning" + assert "incidental benchmark log" in captured.err + + +def test_v7_readiness_private_fixture_gate_is_default_off() -> None: + snapshot = { + "status": "missing", + "ready": False, + "required": False, + "fixtureCount": 0, + "issueCodes": ["workbook_path_missing"], + } + optional_gate = v7_readiness._private_fixture_gate(snapshot) + required_gate = v7_readiness._private_fixture_gate({**snapshot, "required": True}) + + assert optional_gate["status"] == "passed" + assert optional_gate["privateStatus"] == "missing" + assert optional_gate["ready"] is False + assert required_gate["status"] == "failed" + + +def _matched_corpus_artifact(tmp_path): + path = tmp_path / "parser-comparison-corpus-latest.json" + path.write_text( + json.dumps( + { + "ok": True, + "result": { + "status": "matched", + "fixtureCount": 2, + "summary": { + "matched": 2, + "mismatched": 0, + "failed": 0, + "materialMismatchCount": 0, + }, + }, + } + ), + encoding="utf-8", + ) + return path + + +def _xlsx_allowlist(): + return [ + { + "ecosystem": "npm", + "package": "xlsx", + "paths": ["workbook-worker/package-lock.json"], + "advisoryIds": ["GHSA-4r6h-8v6p-xvw6", "GHSA-5pgg-2g8v-p4x9"], + "severity": "high", + "owner": "release-readiness", + "expiresOn": "2026-08-31", + "releaseBlocker": check_dependency_risks.DEFAULT_RELEASE_BLOCKER, + "status": "accepted_pending_parser_replacement", + "mitigations": ["Worker hardening remains in place while parser replacement is pending."], + } + ] + + +def _xlsx_audit_findings(target): + if target.name != "workbook-worker": + return [] + return [ + check_dependency_risks.Finding( + ecosystem="npm", + package="xlsx", + severity="high", + advisory_ids=("GHSA-4R6H-8V6P-XVW6", "GHSA-5PGG-2G8V-P4X9"), + package_lock="workbook-worker/package-lock.json", + ) + ] diff --git a/backend/tests/contract/test_external_verification_status.py b/backend/tests/contract/test_external_verification_status.py new file mode 100644 index 0000000..067c6c5 --- /dev/null +++ b/backend/tests/contract/test_external_verification_status.py @@ -0,0 +1,83 @@ +from __future__ import annotations + +import importlib.util +import subprocess +import sys +from pathlib import Path + +import pytest + + +ROOT = Path(__file__).resolve().parents[3] + + +def _module(): + path = ROOT / "scripts" / "publish_verification_status.py" + spec = importlib.util.spec_from_file_location("external_status_contract", path) + assert spec is not None and spec.loader is not None + module = importlib.util.module_from_spec(spec) + sys.modules[spec.name] = module + spec.loader.exec_module(module) + return module + + +def test_status_payload_requires_matching_signed_outcome_and_immutable_url() -> None: + module = _module() + source_sha = "a" * 40 + report = {"sourceSha": source_sha, "profile": "full", "result": "passed"} + payload = module.build_status_payload( + verification_report=report, + state="success", + evidence_url=f"https://evidence.example.test/excel-agent/{source_sha}/receipt", + context="excel-agent/self-hosted/full", + expected_sha=source_sha, + ) + assert payload == { + "state": "success", + "target_url": f"https://evidence.example.test/excel-agent/{source_sha}/receipt", + "description": "Signed full verification passed", + "context": "excel-agent/self-hosted/full", + } + + with pytest.raises(ValueError, match="cannot publish success"): + module.build_status_payload( + verification_report={**report, "result": "failed"}, + state="success", + evidence_url=f"https://evidence.example.test/{source_sha}", + context="excel-agent/self-hosted/full", + expected_sha=source_sha, + ) + with pytest.raises(ValueError, match="exact source SHA"): + module.build_status_payload( + verification_report=report, + state="success", + evidence_url="https://evidence.example.test/latest/receipt", + context="excel-agent/self-hosted/full", + expected_sha=source_sha, + ) + with pytest.raises(ValueError, match="credentials"): + module.build_status_payload( + verification_report=report, + state="success", + evidence_url=f"https://token@evidence.example.test/{source_sha}", + context="excel-agent/self-hosted/full", + expected_sha=source_sha, + ) + + +def test_status_cli_is_dry_run_unless_publish_is_explicit() -> None: + script = ROOT / "scripts" / "publish_verification_status.py" + completed = subprocess.run( + [sys.executable, str(script), "--help"], + cwd=ROOT, + check=True, + capture_output=True, + text=True, + ) + assert "--publish" in completed.stdout + assert "--token-env" in completed.stdout + assert "--evidence-url" in completed.stdout + source = script.read_text(encoding="utf-8") + assert "if args.publish:" in source + assert "os.getenv(args.token_env" in source + assert '"Authorization": f"Bearer {token}"' in source diff --git a/backend/tests/contract/test_fixture_inventory_audit.py b/backend/tests/contract/test_fixture_inventory_audit.py new file mode 100644 index 0000000..a65de91 --- /dev/null +++ b/backend/tests/contract/test_fixture_inventory_audit.py @@ -0,0 +1,136 @@ +from scripts import run_fixture_inventory_audit as audit + + +def test_fixture_honesty_audit_fails_missing_provenance_policy_and_claim_evidence(): + manifest = { + "fixtures": [ + { + "fixtureId": "chart_claim", + "name": "Chart Source Update", + } + ] + } + result = { + "status": "passed", + "inventoryCoverage": {"features": {"charts": []}}, + "results": [ + { + "fixtureId": "chart_claim", + "actualObjects": {"charts": [], "tables": [], "pivots": [], "comments": []}, + "packageFeatures": {}, + } + ], + } + + honesty = audit.evaluate_fixture_honesty(manifest, result) + + assert honesty["status"] == "failed" + assert {issue["code"] for issue in honesty["issues"]} >= { + "provenance_missing", + "required_feature_evidence_policy_missing", + "misleading_fixture_claim", + } + + +def test_fixture_honesty_audit_accepts_generated_evidence_but_reports_real_workbook_gap(): + manifest = { + "fixtureProvenance": { + "kind": "generated", + "source": "unit-test generator", + "authoringTool": "ExcelJS", + "excelAuthored": False, + "privateExternal": False, + }, + "fixtureInventoryPolicy": {"requiredFeatureEvidence": ["charts", "tables"]}, + "fixtures": [ + {"fixtureId": "chart_case", "name": "Chart Source Update"}, + {"fixtureId": "table_case", "name": "Table Source Intersection"}, + ], + } + result = { + "status": "passed", + "inventoryCoverage": { + "features": { + "charts": [{"fixtureId": "chart_case", "count": 1}], + "tables": [{"fixtureId": "table_case", "count": 1}], + } + }, + "results": [ + { + "fixtureId": "chart_case", + "actualObjects": {"charts": [{"path": "xl/charts/chart1.xml"}]}, + "packageFeatures": {}, + }, + { + "fixtureId": "table_case", + "actualObjects": {"tables": [{"name": "RevenueTable"}]}, + "packageFeatures": {}, + }, + ], + } + + honesty = audit.evaluate_fixture_honesty(manifest, result) + + assert honesty["status"] == "warning" + assert honesty["provenanceSummary"] == {"generated": 2, "excel_authored": 0, "private_external": 0, "unknown": 0} + assert {issue["code"] for issue in honesty["issues"]} == {"real_excel_authored_fixture_gap"} + + +def test_fixture_honesty_audit_accepts_private_external_provenance_without_real_gap(): + manifest = { + "fixtureProvenance": { + "kind": "private_external", + "privateExternal": True, + }, + "fixtureInventoryPolicy": {"requiredFeatureEvidence": ["macros"]}, + "fixtures": [{"fixtureId": "private_macro", "name": "Macro Source Checkpoint"}], + } + result = { + "status": "passed", + "inventoryCoverage": {"features": {"macros": [{"fixtureId": "private_macro", "count": 1}]}}, + "results": [ + { + "fixtureId": "private_macro", + "actualObjects": {"charts": [], "tables": [], "pivots": [], "comments": []}, + "packageFeatures": {"macros": True}, + } + ], + } + + honesty = audit.evaluate_fixture_honesty(manifest, result) + + assert honesty["status"] == "passed" + assert honesty["realWorkbookFixtureGap"]["present"] is False + assert honesty["provenanceSummary"]["private_external"] == 1 + + +def test_apply_fixture_honesty_audit_elevates_worker_status(): + envelope = { + "result": { + "status": "passed", + "inventoryCoverage": {"features": {"tables": []}}, + "results": [ + { + "fixtureId": "table_claim", + "actualObjects": {"charts": [], "tables": [], "pivots": [], "comments": []}, + "packageFeatures": {}, + } + ], + } + } + manifest = { + "fixtureProvenance": { + "kind": "generated", + "source": "unit-test generator", + "excelAuthored": False, + "privateExternal": False, + }, + "fixtureInventoryPolicy": {"requiredFeatureEvidence": ["tables"]}, + "fixtures": [{"fixtureId": "table_claim", "name": "Table Coverage Case"}], + } + + updated = audit.apply_fixture_honesty_audit(envelope, manifest) + + assert updated["result"]["status"] == "failed" + assert updated["result"]["provenanceAudit"]["status"] == "failed" + assert "provenanceAudit" in updated["result"] diff --git a/backend/tests/contract/test_flagship_contracts.py b/backend/tests/contract/test_flagship_contracts.py index 38e4f81..73facc5 100644 --- a/backend/tests/contract/test_flagship_contracts.py +++ b/backend/tests/contract/test_flagship_contracts.py @@ -7,7 +7,6 @@ from backend.services.operation_registry import get_operation_registry -client = TestClient(app) headers = {"X-API-Key": API_KEY} @@ -28,6 +27,11 @@ def _context() -> dict: } +def _post(path: str, payload: dict): + with TestClient(app) as client: + return client.post(path, headers=headers, json=payload) + + def test_v2_and_v3_plan_routes_remain_mounted(): payload = { "prompt": "Create a simple DCF shell", @@ -35,12 +39,12 @@ def test_v2_and_v3_plan_routes_remain_mounted(): "context": _context(), "accuracy_mode": "fast", } - v2 = client.post("/api/v1/tasks/plan/v2", headers=headers, json=payload) + v2 = _post("/api/v1/tasks/plan/v2", payload) assert v2.status_code == 200 assert "operations" in v2.json() assert (v2.json().get("schemaVersion") or v2.json().get("schema_version")) == "2026-05-operation-plan-v2" - v3 = client.post("/api/v1/tasks/plan/v3", headers=headers, json=payload) + v3 = _post("/api/v1/tasks/plan/v3", payload) assert v3.status_code == 200 assert "operations" in v3.json() assert (v3.json().get("schemaVersion") or v3.json().get("schema_version")) == "2026-05-operation-plan-v3" @@ -48,10 +52,9 @@ def test_v2_and_v3_plan_routes_remain_mounted(): def test_flagship_v5_additive_response_shapes_are_stable(): context = _context() - truth = client.post( + truth = _post( "/api/v1/workbooks/truth-model", - headers=headers, - json={"session_id": "contract-v5", "workbook_context": context, "workbook_fingerprint": "contract"}, + {"session_id": "contract-v5", "workbook_context": context, "workbook_fingerprint": "contract"}, ) assert truth.status_code == 200 truth_payload = truth.json() @@ -66,10 +69,9 @@ def test_flagship_v5_additive_response_shapes_are_stable(): ]: assert key in truth_payload - graph = client.post( + graph = _post( "/api/v1/formulas/graph", - headers=headers, - json={"session_id": "contract-v5", "workbook_context": context, "workbook_fingerprint": "contract"}, + {"session_id": "contract-v5", "workbook_context": context, "workbook_fingerprint": "contract"}, ) assert graph.status_code == 200 graph_payload = graph.json() @@ -77,10 +79,9 @@ def test_flagship_v5_additive_response_shapes_are_stable(): for key in ["formulaImpactReport", "formulaClusters", "outputRankings", "centrality", "parserConfidence"]: assert key in graph_payload - simulation = client.post( + simulation = _post( "/api/v1/operations/simulate", - headers=headers, - json={ + { "session_id": "contract-v5", "truth_model_id": truth_payload["truthModelId"], "workbook_context": context, diff --git a/backend/tests/contract/test_local_replay_corpus.py b/backend/tests/contract/test_local_replay_corpus.py new file mode 100644 index 0000000..9c36a70 --- /dev/null +++ b/backend/tests/contract/test_local_replay_corpus.py @@ -0,0 +1,340 @@ +import json +from pathlib import Path +from types import SimpleNamespace + +import pytest + +from backend.benchmarks import local_replay, local_replay_corpus, v7_readiness +from backend.session_store import SessionStore + + +SYNTHETIC_CORPUS = ( + Path(__file__).resolve().parents[2] + / "benchmarks" + / "fixtures" + / "replays" + / "synthetic_aggregate_replay_corpus.json" +) + + +def test_local_replay_corpus_importer_pins_privacy_safe_production_baseline(tmp_path, monkeypatch): + store = SessionStore(db_path=tmp_path / "sessions.db") + monkeypatch.setattr(local_replay, "get_session_store", lambda: store) + monkeypatch.setattr(local_replay_corpus, "get_session_store", lambda: store) + corpus_path = tmp_path / "aggregate-replays.json" + corpus_path.write_text(json.dumps(production_corpus_payload(aggregate_replay_records())), encoding="utf-8") + + result = local_replay_corpus.import_local_replay_corpus( + corpus_path, + baseline_id="replaybase_safe_corpus", + production_evidence=True, + evidence_class="anonymized_user_acceptance_aggregate", + minimum_production_replays=2, + ) + evaluation = local_replay.evaluate_local_replays(limit=100, baseline_id="replaybase_safe_corpus") + + assert result["status"] == "passed" + assert result["productionEvidence"] is True + assert result["baseline"]["metadata"]["productionEvidence"] is True + assert result["baseline"]["metadata"]["evidenceClass"] == "anonymized_user_acceptance_aggregate" + assert result["baseline"]["metadata"]["importedArtifactCount"] == 2 + assert result["baseline"]["metadata"]["productionCorpusEnvelope"]["sourceCohortHash"].startswith("sha256:") + assert result["baseline"]["metadata"]["productionCorpusEnvelope"]["recordCount"] == 2 + assert evaluation.status == "passed" + assert evaluation.deterministic_drift_class == "stable" + assert evaluation.privacy_scan["status"] == "passed" + assert all("payloadHash" in item["payload"] for item in store.list_local_replay_artifacts(100)) + + +def test_local_replay_corpus_rejects_raw_workbook_content_fields(tmp_path, monkeypatch): + store = SessionStore(db_path=tmp_path / "sessions.db") + monkeypatch.setattr(local_replay_corpus, "get_session_store", lambda: store) + raw = aggregate_replay_records()[0] + raw["payload"]["formula"] = "=CustomerRevenue!B4" + raw["payload"]["workbookName"] = "Customer Acme Renewal.xlsx" + corpus_path = tmp_path / "raw-replay.json" + corpus_path.write_text(json.dumps([raw]), encoding="utf-8") + + with pytest.raises(ValueError, match="restricted raw-content keys"): + local_replay_corpus.import_local_replay_corpus(corpus_path, production_evidence=False) + + assert store.list_local_replay_artifacts(100) == [] + + +def test_local_replay_corpus_rejects_raw_content_values_in_allowed_fields(tmp_path, monkeypatch): + store = SessionStore(db_path=tmp_path / "sessions.db") + monkeypatch.setattr(local_replay_corpus, "get_session_store", lambda: store) + raw = aggregate_replay_records()[0] + raw["payload"]["sourceProject"] = "Customer Acme Renewal.xlsx" + raw["payload"]["replaySignature"]["operationTypes"] = ["Sheet1!B4"] + corpus_path = tmp_path / "raw-allowed-value.json" + corpus_path.write_text(json.dumps([raw]), encoding="utf-8") + + with pytest.raises(ValueError, match="restricted raw-content value"): + local_replay_corpus.import_local_replay_corpus(corpus_path, production_evidence=False) + + assert store.list_local_replay_artifacts(100) == [] + + +@pytest.mark.parametrize( + ("field", "value", "message"), + [ + ("operationCount", True, "integer"), + ("accuracyScore", "NaN", "finite number"), + ("rollbackAvailable", "false", "boolean"), + ("objectCounts", {"charts": True}, "numeric count"), + ], +) +def test_local_replay_corpus_rejects_coercive_or_nonfinite_aggregate_values( + field, + value, + message, +): + raw = aggregate_replay_records()[0] + raw["payload"][field] = value + + with pytest.raises(ValueError, match=message): + local_replay_corpus.sanitize_corpus_record(raw, index=1) + + +def test_synthetic_non_production_replay_fixture_imports_and_stays_default_blocked(tmp_path, monkeypatch): + store = SessionStore(db_path=tmp_path / "sessions.db") + monkeypatch.setattr(local_replay, "get_session_store", lambda: store) + monkeypatch.setattr(local_replay_corpus, "get_session_store", lambda: store) + + result = local_replay_corpus.import_local_replay_corpus( + SYNTHETIC_CORPUS, + baseline_id="replaybase_synthetic_contract", + production_evidence=False, + evidence_class="synthetic_replay_contract", + ) + evaluation = local_replay.evaluate_local_replays(limit=100, baseline_id="replaybase_synthetic_contract") + baseline_status = v7_readiness._baseline_status(evaluation, result["baseline"]["metadata"]) + blockers = v7_readiness._default_switch_blockers( + [], + {"status": "passed"}, + {"status": "passed"}, + {"status": "passed"}, + baseline_status, + ) + + assert result["status"] == "passed" + assert result["productionEvidence"] is False + assert result["baseline"]["metadata"]["evidenceClass"] == "synthetic_replay_contract" + assert result["baseline"]["metadata"]["importedArtifactCount"] == 3 + assert evaluation.status == "passed" + assert evaluation.deterministic_drift_class == "stable" + assert baseline_status["evidenceTier"] == "synthetic_contract" + assert "synthetic contract evidence" in " ".join(blockers) + + +def test_v7_default_blocker_distinguishes_sanitized_non_production_corpus(): + replay = SimpleNamespace( + baseline_comparison={}, + baseline_id="replaybase_non_prod_corpus", + deterministic_drift_class="stable", + ) + + baseline_status = v7_readiness._baseline_status( + replay, + { + "productionEvidence": False, + "evidenceClass": "synthetic_replay_contract", + }, + ) + synthetic_blockers = v7_readiness._default_switch_blockers( + [], + {"status": "passed"}, + {"status": "passed"}, + {"status": "passed"}, + baseline_status, + ) + assert baseline_status["evidenceTier"] == "synthetic_contract" + assert "synthetic contract evidence" in " ".join(synthetic_blockers) + + baseline_status = v7_readiness._baseline_status( + replay, + { + "productionEvidence": False, + "evidenceClass": "anonymized_local_replay_corpus", + }, + ) + corpus_blockers = v7_readiness._default_switch_blockers( + [], + {"status": "passed"}, + {"status": "passed"}, + {"status": "passed"}, + baseline_status, + ) + assert baseline_status["evidenceTier"] == "sanitized_non_production_corpus" + assert "sanitized non-production corpus evidence" in " ".join(corpus_blockers) + + +def test_local_replay_corpus_requires_minimum_count_for_production_evidence(tmp_path, monkeypatch): + store = SessionStore(db_path=tmp_path / "sessions.db") + monkeypatch.setattr(local_replay_corpus, "get_session_store", lambda: store) + corpus_path = tmp_path / "too-small.json" + corpus_path.write_text(json.dumps(production_corpus_payload([aggregate_replay_records()[0]])), encoding="utf-8") + + with pytest.raises(ValueError, match="requires at least 2 sanitized replay artifacts"): + local_replay_corpus.import_local_replay_corpus( + corpus_path, + production_evidence=True, + evidence_class="anonymized_user_acceptance_aggregate", + minimum_production_replays=2, + ) + + assert store.list_local_replay_artifacts(100) == [] + + +def test_local_replay_corpus_requires_production_envelope_for_production_evidence(tmp_path, monkeypatch): + store = SessionStore(db_path=tmp_path / "sessions.db") + monkeypatch.setattr(local_replay_corpus, "get_session_store", lambda: store) + corpus_path = tmp_path / "bare-production-list.json" + corpus_path.write_text(json.dumps(aggregate_replay_records()), encoding="utf-8") + + with pytest.raises(ValueError, match="requires a JSON envelope object"): + local_replay_corpus.import_local_replay_corpus( + corpus_path, + production_evidence=True, + evidence_class="anonymized_user_acceptance_aggregate", + minimum_production_replays=2, + ) + + assert store.list_local_replay_artifacts(100) == [] + + +def test_local_replay_corpus_rejects_invalid_production_envelope_metadata(tmp_path, monkeypatch): + store = SessionStore(db_path=tmp_path / "sessions.db") + monkeypatch.setattr(local_replay_corpus, "get_session_store", lambda: store) + corpus_path = tmp_path / "bad-envelope.json" + payload = production_corpus_payload(aggregate_replay_records()) + payload["sourceCohortHash"] = "customer-cohort-a" + corpus_path.write_text(json.dumps(payload), encoding="utf-8") + + with pytest.raises(ValueError, match="sourceCohortHash must be a sha256 digest"): + local_replay_corpus.import_local_replay_corpus( + corpus_path, + production_evidence=True, + evidence_class="anonymized_user_acceptance_aggregate", + minimum_production_replays=2, + ) + + assert store.list_local_replay_artifacts(100) == [] + + +def test_local_replay_corpus_requires_isolated_store_for_production_evidence(tmp_path, monkeypatch): + store = SessionStore(db_path=tmp_path / "sessions.db") + monkeypatch.setattr(local_replay, "get_session_store", lambda: store) + monkeypatch.setattr(local_replay_corpus, "get_session_store", lambda: store) + store.get_or_create_session("foreign-session") + store.save_local_replay_artifact("foreign-session", "v7_plan", aggregate_replay_records()[0]["payload"]) + corpus_path = tmp_path / "safe-but-dirty-store.json" + corpus_path.write_text(json.dumps(production_corpus_payload(aggregate_replay_records())), encoding="utf-8") + + with pytest.raises(ValueError, match="isolated local replay store"): + local_replay_corpus.import_local_replay_corpus( + corpus_path, + baseline_id="replaybase_dirty_store", + production_evidence=True, + evidence_class="anonymized_user_acceptance_aggregate", + minimum_production_replays=2, + ) + + assert all((item.get("baselineId") or "") != "replaybase_dirty_store" for item in store.list_local_replay_baselines(10)) + + +def production_corpus_payload(records: list[dict], *, production_evidence: bool = True) -> dict: + return { + "schemaVersion": local_replay_corpus.CORPUS_SCHEMA_VERSION, + "generatedAt": "2026-06-15T12:00:00Z", + "anonymizationPolicyVersion": "anon-policy-v1", + "sourceCohortHash": "sha256:" + "a" * 64, + "exporterVersion": "replay-exporter-v1", + "recordCount": len(records), + "productionEvidence": production_evidence, + "artifacts": records, + } + + +def aggregate_replay_records() -> list[dict]: + return [ + { + "artifactType": "v7_plan", + "payload": { + "schemaVersion": "2026-05-local-replay-v1", + "artifactKind": "aggregate_user_acceptance_plan", + "operationTypes": ["write_cell"], + "operationCount": 1, + "accuracyScore": 0.99, + "confidenceGapCount": 0, + "checkpointReasonCount": 0, + "formulaFailureCount": 0, + "truthSchema": "2026-05-workbook-truth-model-v5", + "formulaSchema": "2026-05-formula-graph-artifact-v5", + "simulationSchema": "2026-05-operation-simulation-v6", + "v7GateStatus": "passed", + "dirtyRangeCount": 1, + "rollbackAvailable": True, + "fixtureIdHash": "fx_a8b553a4", + "category": "writeback", + "sourceProject": "uat-batch-a", + "sheetCount": 4, + "formulaCount": 128, + "objectCounts": {"tables": 2, "charts": 1, "pivots": 0}, + "warningCategoryCounts": {}, + "semanticMatchCounts": {"matched": 12, "missing": 0}, + "replaySignature": { + "schemaVersion": "2026-05-local-replay-v1", + "operationTypes": ["write_cell"], + "operationCount": 1, + "truthSchema": "2026-05-workbook-truth-model-v5", + "formulaSchema": "2026-05-formula-graph-artifact-v5", + "simulationSchema": "2026-05-operation-simulation-v6", + "v7GateStatus": "passed", + "formulaFailureCount": 0, + "confidenceGapCount": 0, + "checkpointReasonCount": 0, + }, + }, + }, + { + "artifactType": "v7_apply_verification", + "payload": { + "schemaVersion": "2026-05-local-replay-v1", + "artifactKind": "aggregate_user_acceptance_checkpoint", + "operationTypes": ["create_chart"], + "operationCount": 1, + "accuracyScore": 0.94, + "confidenceGapCount": 0, + "checkpointReasonCount": 1, + "formulaFailureCount": 0, + "truthSchema": "2026-05-workbook-truth-model-v5", + "formulaSchema": "2026-05-formula-graph-artifact-v5", + "simulationSchema": "2026-05-operation-simulation-v6", + "v7GateStatus": "checkpoint_required", + "dirtyRangeCount": 0, + "rollbackAvailable": False, + "fixtureIdHash": "fx_b10d2402", + "category": "object_mutation", + "sourceProject": "uat-batch-a", + "sheetCount": 2, + "formulaCount": 33, + "objectCounts": {"tables": 1, "charts": 2, "pivots": 0}, + "warningCategoryCounts": {"checkpoint_required": 1}, + "semanticMatchCounts": {"matched": 8, "missing": 1}, + "replaySignature": { + "schemaVersion": "2026-05-local-replay-v1", + "operationTypes": ["create_chart"], + "operationCount": 1, + "truthSchema": "2026-05-workbook-truth-model-v5", + "formulaSchema": "2026-05-formula-graph-artifact-v5", + "simulationSchema": "2026-05-operation-simulation-v6", + "v7GateStatus": "checkpoint_required", + "formulaFailureCount": 0, + "confidenceGapCount": 0, + "checkpointReasonCount": 1, + }, + }, + }, + ] diff --git a/backend/tests/contract/test_local_replay_seed.py b/backend/tests/contract/test_local_replay_seed.py new file mode 100644 index 0000000..5d36bee --- /dev/null +++ b/backend/tests/contract/test_local_replay_seed.py @@ -0,0 +1,192 @@ +import asyncio + +from backend.benchmarks import local_replay, local_replay_seed, v7_readiness +from backend.benchmarks.local_replay_seed import SEED_BASELINE_ID, seed_local_replay_evidence +from backend.session_store import SessionStore + + +def test_local_replay_seed_pins_stable_privacy_safe_baseline(tmp_path, monkeypatch): + store = SessionStore(db_path=tmp_path / "sessions.db") + monkeypatch.setattr(local_replay, "get_session_store", lambda: store) + monkeypatch.setattr(local_replay_seed, "get_session_store", lambda: store) + + result = seed_local_replay_evidence(replay_limit=100) + evaluation = local_replay.evaluate_local_replays(limit=100, baseline_id=SEED_BASELINE_ID) + + assert result["status"] == "passed" + assert result["baselineId"] == SEED_BASELINE_ID + assert result["seedReplayIds"] + assert result["baseline"]["metadata"]["productionEvidence"] is False + assert result["baseline"]["metadata"]["evidenceClass"] == "deterministic_plumbing_seed" + assert evaluation.status == "passed" + assert evaluation.capture_count >= 2 + assert evaluation.deterministic_drift_class == "stable" + assert evaluation.privacy_scan["status"] == "passed" + + +def test_v7_readiness_accepts_seeded_replay_baseline_without_replay_warnings(tmp_path, monkeypatch): + store = SessionStore(db_path=tmp_path / "sessions.db") + monkeypatch.setattr(local_replay, "get_session_store", lambda: store) + monkeypatch.setattr(local_replay_seed, "get_session_store", lambda: store) + monkeypatch.setattr(v7_readiness, "get_session_store", lambda: store) + seed_local_replay_evidence(replay_limit=100) + + async def fake_local_accuracy(**_kwargs): + return { + "status": "passed", + "workbookAware": { + "gate": { + "status": "passed", + "score": 100, + "total": 30, + "minimumFixtureTarget": 30, + "fixtureCoverageStatus": "target_met", + }, + "results": [{"status": "passed", "score": 100} for _ in range(30)], + }, + } + + monkeypatch.setattr(v7_readiness, "run_local_accuracy", fake_local_accuracy) + monkeypatch.setattr(v7_readiness, "_formula_readiness", lambda: {"status": "passed", "simulationGate": "passed"}) + monkeypatch.setattr( + v7_readiness, + "_registry_coverage", + lambda: { + "status": "passed", + "coverageRatio": 1, + "readinessBreakdown": {"applyReadyOperations": ["write_cell"]}, + "safeGatedOperations": [], + }, + ) + monkeypatch.setattr(v7_readiness, "_rollback_readiness", lambda: {"status": "passed", "rangeSnapshots": True, "proofClasses": True}) + monkeypatch.setattr(v7_readiness, "_storage_parity", lambda: {"status": "passed", "missingMethods": []}) + monkeypatch.setattr(v7_readiness, "_worker_hardening", lambda: {"status": "passed"}) + monkeypatch.setattr(v7_readiness, "_office_mock_coverage", lambda: {"status": "passed", "requiredSignals": [], "presentSignals": []}) + monkeypatch.setattr(v7_readiness, "_dependency_risk", lambda _path=None: {"status": "passed"}) + monkeypatch.setattr(v7_readiness, "_taskpane_coverage", lambda: {"status": "passed"}) + monkeypatch.setattr(v7_readiness, "_generated_types_freshness", lambda: {"status": "passed"}) + + report = asyncio.run(v7_readiness.run_v7_readiness(replay_baseline_id=SEED_BASELINE_ID)) + + assert report.gates["localReplay"]["status"] == "passed" + assert report.gates["replayBaseline"]["status"] == "passed" + assert report.baseline_status["deterministicDriftClass"] == "stable" + assert report.baseline_status["productionEvidence"] is False + assert report.default_allowed is False + assert "Replay baseline is deterministic plumbing evidence" in " ".join(report.metadata["defaultSwitchBlockers"]) + assert "No anonymized local replay artifacts have been captured yet." not in report.warnings + assert "localReplay gate is warning." not in report.warnings + assert "replayBaseline gate is warning." not in report.warnings + + +def test_v7_release_readiness_blocks_seeded_replay_baseline_without_production_evidence(tmp_path, monkeypatch): + store = SessionStore(db_path=tmp_path / "sessions.db") + monkeypatch.setattr(local_replay, "get_session_store", lambda: store) + monkeypatch.setattr(local_replay_seed, "get_session_store", lambda: store) + monkeypatch.setattr(v7_readiness, "get_session_store", lambda: store) + seed_local_replay_evidence(replay_limit=100) + + async def fake_local_accuracy(**_kwargs): + return { + "status": "passed", + "workbookAware": { + "gate": { + "status": "passed", + "score": 100, + "total": 30, + "minimumFixtureTarget": 30, + "fixtureCoverageStatus": "target_met", + }, + "results": [{"status": "passed", "score": 100} for _ in range(30)], + }, + } + + monkeypatch.setattr(v7_readiness, "run_local_accuracy", fake_local_accuracy) + monkeypatch.setattr(v7_readiness, "_formula_readiness", lambda: {"status": "passed", "simulationGate": "passed"}) + monkeypatch.setattr( + v7_readiness, + "_registry_coverage", + lambda: { + "status": "passed", + "coverageRatio": 1, + "readinessBreakdown": {"applyReadyOperations": ["write_cell"]}, + "safeGatedOperations": [], + }, + ) + monkeypatch.setattr(v7_readiness, "_rollback_readiness", lambda: {"status": "passed", "rangeSnapshots": True, "proofClasses": True}) + monkeypatch.setattr(v7_readiness, "_storage_parity", lambda: {"status": "passed", "missingMethods": []}) + monkeypatch.setattr(v7_readiness, "_worker_hardening", lambda: {"status": "passed"}) + monkeypatch.setattr(v7_readiness, "_office_mock_coverage", lambda: {"status": "passed", "requiredSignals": [], "presentSignals": []}) + monkeypatch.setattr(v7_readiness, "_dependency_risk", lambda _path=None: {"status": "passed"}) + monkeypatch.setattr(v7_readiness, "_taskpane_coverage", lambda: {"status": "passed"}) + monkeypatch.setattr(v7_readiness, "_generated_types_freshness", lambda: {"status": "passed"}) + + report = asyncio.run( + v7_readiness.run_v7_readiness( + replay_baseline_id=SEED_BASELINE_ID, + require_production_replay=True, + ) + ) + + assert report.status == "blocked" + assert report.default_allowed is False + assert report.baseline_status["productionEvidence"] is False + assert "Production replay evidence is required" in " ".join(report.blockers) + assert report.metadata["requireProductionReplay"] is True + + +def test_fixed_id_replay_baseline_upsert_refreshes_latest_lookup(tmp_path): + store = SessionStore(db_path=tmp_path / "sessions.db") + store.get_or_create_session("old") + store.get_or_create_session("seed") + + for index in range(60): + store.save_local_replay_baseline( + { + "baselineId": f"replaybase_old_{index}", + "schemaVersion": "2026-05-local-replay-baseline", + "name": "Old Baseline", + "sessionId": "old", + "evaluationId": "old", + "replayCount": 0, + "driftSeverity": "none", + "signatureBuckets": {}, + "createdAt": 1000 + index, + "metadata": {}, + } + ) + + store.save_local_replay_baseline( + { + "baselineId": SEED_BASELINE_ID, + "schemaVersion": "2026-05-local-replay-baseline", + "name": "Initial Seed Baseline", + "sessionId": "seed", + "evaluationId": "seed-old", + "replayCount": 1, + "driftSeverity": "none", + "signatureBuckets": {}, + "createdAt": 900, + "metadata": {"productionEvidence": False}, + } + ) + assert all((item.get("baselineId") or "") != SEED_BASELINE_ID for item in store.list_local_replay_baselines(50)) + + store.save_local_replay_baseline( + { + "baselineId": SEED_BASELINE_ID, + "schemaVersion": "2026-05-local-replay-baseline", + "name": "Refreshed Seed Baseline", + "sessionId": "seed", + "evaluationId": "seed-new", + "replayCount": 2, + "driftSeverity": "none", + "signatureBuckets": {}, + "createdAt": 2000, + "metadata": {"productionEvidence": False}, + } + ) + + latest = store.list_local_replay_baselines(50) + assert latest[0]["baselineId"] == SEED_BASELINE_ID + assert latest[0]["evaluationId"] == "seed-new" diff --git a/backend/tests/contract/test_local_verification_plane.py b/backend/tests/contract/test_local_verification_plane.py new file mode 100644 index 0000000..2fa9391 --- /dev/null +++ b/backend/tests/contract/test_local_verification_plane.py @@ -0,0 +1,183 @@ +from __future__ import annotations + +import importlib.util +import re +import subprocess +import sys +from pathlib import Path + + +ROOT = Path(__file__).resolve().parents[3] +WORKFLOW_ROOT = ROOT / ".github" / "workflows" + + +def _load_toolchain_doctor(): + path = ROOT / "scripts" / "toolchain_doctor.py" + spec = importlib.util.spec_from_file_location("toolchain_doctor_contract", path) + assert spec is not None and spec.loader is not None + module = importlib.util.module_from_spec(spec) + sys.modules[spec.name] = module + spec.loader.exec_module(module) + return module + + +def test_workflows_use_only_self_hosted_runners() -> None: + workflows = sorted(WORKFLOW_ROOT.glob("*.yml")) + assert workflows + for workflow in workflows: + text = workflow.read_text(encoding="utf-8") + assert "ubuntu-latest" not in text + assert "windows-latest" not in text + declarations = re.findall(r"^\s*runs-on:\s*(.+)$", text, flags=re.MULTILINE) + assert declarations, workflow + assert all("self-hosted" in declaration for declaration in declarations) + if "pull_request:" in text: + assert "github.event.pull_request.head.repo.full_name == github.repository" in text + + v7 = (WORKFLOW_ROOT / "v7-readiness.yml").read_text(encoding="utf-8") + assert "[self-hosted, windows, x64, excel-agent-excel]" in v7 + assert v7.count("[self-hosted, linux, x64, excel-agent]") >= 6 + + +def test_verify_local_shell_contract() -> None: + script = ROOT / "scripts" / "verify-local.sh" + sdk_examples = ROOT / "vnext" / "scripts" / "verify-sdk-examples.sh" + subprocess.run(["sh", "-n", str(script)], cwd=ROOT, check=True) + subprocess.run(["sh", "-n", str(sdk_examples)], cwd=ROOT, check=True) + result = subprocess.run( + ["sh", str(script), "--help"], + cwd=ROOT, + check=True, + capture_output=True, + text=True, + ) + assert "fast" in result.stdout + assert "release" in result.stdout + assert "strict external release-evidence prerequisites" in result.stdout + script_text = script.read_text(encoding="utf-8") + assert "GitHub-hosted" in script_text + assert "./vnext/scripts/verify-sdk-examples.sh" in script_text + + +def test_toolchain_doctor_report_is_machine_readable(monkeypatch) -> None: + module = _load_toolchain_doctor() + monkeypatch.setenv("GITHUB_ACTIONS", "true") + monkeypatch.setenv("RUNNER_ENVIRONMENT", "self-hosted") + report = module.collect_toolchain_report( + "fast", + require_self_hosted=True, + specs=(), + ) + assert report["schema_version"] == "excel-agent-toolchain-doctor-v1" + assert report["passed"] is True + assert report["self_hosted_ok"] is True + assert report["tools"] == [] + + monkeypatch.setenv("RUNNER_ENVIRONMENT", "github-hosted") + rejected = module.collect_toolchain_report( + "fast", + require_self_hosted=True, + specs=(), + ) + assert rejected["passed"] is False + assert rejected["failures"] == ["runner_environment"] + + +def test_dependabot_covers_vnext_ecosystems() -> None: + text = (ROOT / ".github" / "dependabot.yml").read_text(encoding="utf-8") + for ecosystem in ("npm", "pip", "cargo"): + pattern = ( + rf'package-ecosystem:\s*"{ecosystem}"' + rf'[\s\S]{{0,180}}directory:\s*"/vnext"' + ) + assert re.search(pattern, text), ecosystem + + +def test_github_community_contracts_are_present() -> None: + required = ( + ROOT / "CONTRIBUTING.md", + ROOT / "CODE_OF_CONDUCT.md", + ROOT / ".github" / "PULL_REQUEST_TEMPLATE.md", + ROOT / ".github" / "ISSUE_TEMPLATE" / "config.yml", + ROOT / ".github" / "ISSUE_TEMPLATE" / "bug.yml", + ROOT / ".github" / "ISSUE_TEMPLATE" / "refinement.yml", + ROOT / ".github" / "ISSUE_TEMPLATE" / "release-evidence.yml", + ROOT / ".github" / "actionlint.yaml", + ROOT / ".github" / "CODEOWNERS", + ROOT / "docs" / "SELF_HOSTED_VERIFICATION.md", + ROOT / "docs" / "RELEASE_PROMOTION.md", + ROOT / "docs" / "IMPLEMENTATION_STATUS.md", + ) + for path in required: + assert path.is_file(), path + assert path.read_text(encoding="utf-8").strip(), path + + pull_request_template = required[2].read_text(encoding="utf-8") + assert "No GitHub-hosted runner minutes are required" in pull_request_template + assert "External/native evidence attached" in pull_request_template + + codeowners = (ROOT / ".github" / "CODEOWNERS").read_text(encoding="utf-8") + assert "* @stag175" in codeowners + status = (ROOT / "docs" / "IMPLEMENTATION_STATUS.md").read_text(encoding="utf-8") + for label in ("Implemented", "Experimental", "Designed", "Blocked"): + assert f"**{label}**" in status + assert "does not imply server availability" in status + assert "strict verifier must remain red" in status + + promotion = (ROOT / "docs" / "RELEASE_PROMOTION.md").read_text(encoding="utf-8") + assert "build_release_promotion_bundle.py" in promotion + assert "verify_release_promotion_bundle.py" in promotion + assert "two distinct" in promotion + + +def test_toolchain_manifest_and_offline_cache_contract() -> None: + verifier = ROOT / "scripts" / "verify_toolchain_manifest.py" + completed = subprocess.run( + [sys.executable, str(verifier), "--json"], + cwd=ROOT, + check=True, + capture_output=True, + text=True, + ) + assert '"passed": true' in completed.stdout + assert '"rustToolchain": "1.97.1"' in completed.stdout + assert '"rustMinimumSupported": "1.85"' in completed.stdout + assert '"rustComponents": [' in completed.stdout + assert '"rustTargets": [' in completed.stdout + assert '"rustToolsImages": {' in completed.stdout + assert '"rustToolsDockerfileSha256":' in completed.stdout + assert '"rustWasmPackages": [' in completed.stdout + assert '"rustNativeOnlyPackages": [' in completed.stdout + assert '"rustWorkspacePackages": [' in completed.stdout + + exporter = ROOT / "scripts" / "export_dependency_cache.sh" + subprocess.run(["sh", "-n", str(exporter)], cwd=ROOT, check=True) + help_result = subprocess.run( + ["sh", str(exporter), "--help"], + cwd=ROOT, + check=True, + capture_output=True, + text=True, + ) + assert "outside the repository" in help_result.stdout + exporter_text = exporter.read_text(encoding="utf-8") + assert "pip download" in exporter_text + assert "manylinux_2_28_x86_64" in exporter_text + assert "npm --prefix" in exporter_text and " ci " in exporter_text + assert "cargo vendor --locked" in exporter_text + assert "docker image save" in exporter_text + assert "rust-tools-image.tar" in exporter_text + assert "scripts/rust_toolchain.py" in exporter_text + assert "shasum -a 256 -c SHA256SUMS" in exporter_text + + forbidden_output = ROOT / ".dependency-cache-contract-test" + rejected = subprocess.run( + ["sh", str(exporter), "--output", str(forbidden_output)], + cwd=ROOT, + check=False, + capture_output=True, + text=True, + ) + assert rejected.returncode == 2 + assert "outside the repository" in rejected.stderr + assert not forbidden_output.exists() diff --git a/backend/tests/contract/test_package_integrity_policy.py b/backend/tests/contract/test_package_integrity_policy.py new file mode 100644 index 0000000..318468e --- /dev/null +++ b/backend/tests/contract/test_package_integrity_policy.py @@ -0,0 +1,324 @@ +import json + +from scripts import check_package_integrity + + +def test_package_integrity_policy_accepts_valid_project_and_requirements(tmp_path): + project = _write_npm_project(tmp_path / "plugin") + requirements = tmp_path / "requirements.txt" + requirements.write_text("fastapi>=0.115.0\npydantic[email]>=2.10.0\n", encoding="utf-8") + lock = tmp_path / "requirements.lock.txt" + lock.write_text( + _hashed_lock("fastapi==0.141.1", "pydantic[email]==2.13.4"), + encoding="utf-8", + ) + + assert check_package_integrity._validate_npm_project(project / "package.json", project / "package-lock.json") == [] + assert check_package_integrity._validate_python_requirements(requirements) == [] + assert check_package_integrity._validate_python_lock(requirements, lock) == [] + + +def test_package_integrity_policy_blocks_missing_or_inexact_python_lock(tmp_path): + requirements = tmp_path / "requirements.txt" + requirements.write_text("fastapi>=0.115.0\npydantic[email]>=2.10.0\n", encoding="utf-8") + lock = tmp_path / "requirements.lock.txt" + lock.write_text(_hashed_lock("fastapi>=0.141.1"), encoding="utf-8") + + failures = check_package_integrity._validate_python_lock(requirements, lock) + + assert any("must use one exact version" in failure for failure in failures) + assert any("pydantic is missing from the lock" in failure for failure in failures) + + +def test_package_integrity_policy_requires_valid_hashes_for_every_python_pin(tmp_path): + requirements = tmp_path / "requirements.txt" + requirements.write_text("fastapi>=0.115.0\n", encoding="utf-8") + lock = tmp_path / "requirements.lock.txt" + lock.write_text("fastapi==0.141.1\n", encoding="utf-8") + + failures = check_package_integrity._validate_python_lock(requirements, lock) + + assert any("must include a SHA-256 hash" in failure for failure in failures) + + lock.write_text( + "fastapi==0.141.1 \\\n --hash=sha256:not-a-digest\n", + encoding="utf-8", + ) + malformed = check_package_integrity._validate_python_lock(requirements, lock) + assert any("malformed or unsupported hash" in failure for failure in malformed) + + +def test_package_integrity_policy_blocks_npm_lock_drift(tmp_path): + project = _write_npm_project(tmp_path / "plugin") + package_lock_path = project / "package-lock.json" + payload = json.loads(package_lock_path.read_text(encoding="utf-8")) + payload["packages"][""]["dependencies"]["left-pad"] = "^1.1.0" + package_lock_path.write_text(json.dumps(payload), encoding="utf-8") + + failures = check_package_integrity._validate_npm_project(project / "package.json", package_lock_path) + + assert any("root dependencies drift" in failure for failure in failures) + + +def test_package_integrity_policy_blocks_registry_package_without_integrity(tmp_path): + project = _write_npm_project(tmp_path / "plugin") + package_lock_path = project / "package-lock.json" + payload = json.loads(package_lock_path.read_text(encoding="utf-8")) + del payload["packages"]["node_modules/left-pad"]["integrity"] + package_lock_path.write_text(json.dumps(payload), encoding="utf-8") + + failures = check_package_integrity._validate_npm_project(project / "package.json", package_lock_path) + + assert any("node_modules/left-pad missing integrity hash" in failure for failure in failures) + + +def test_package_integrity_policy_includes_vnext_npm_project() -> None: + targets = { + (package_json.relative_to(check_package_integrity.ROOT).as_posix(), package_lock.relative_to(check_package_integrity.ROOT).as_posix()) + for package_json, package_lock in check_package_integrity.NPM_TARGETS + } + + assert ("vnext/package.json", "vnext/package-lock.json") in targets + + +def test_standard_proprietary_build_physically_omits_hyperformula_runtime() -> None: + root = check_package_integrity.ROOT + plugin_package = json.loads((root / "plugin" / "package.json").read_text(encoding="utf-8")) + plugin_lock = json.loads((root / "plugin" / "package-lock.json").read_text(encoding="utf-8")) + worker_package = json.loads((root / "workbook-worker" / "package.json").read_text(encoding="utf-8")) + worker_lock = json.loads((root / "workbook-worker" / "package-lock.json").read_text(encoding="utf-8")) + calc_worker = (root / "plugin" / "src" / "spreadsheet" / "calc-worker.ts").read_text(encoding="utf-8") + backend_dockerfile = (root / "backend" / "Dockerfile").read_text(encoding="utf-8") + + assert "hyperformula" not in (plugin_package.get("dependencies") or {}) + assert "node_modules/hyperformula" not in (plugin_lock.get("packages") or {}) + assert "hyperformula" not in (worker_package.get("dependencies") or {}) + assert (worker_package.get("devDependencies") or {}).get("hyperformula") == "3.2.0" + assert worker_lock["packages"]["node_modules/hyperformula"]["dev"] is True + assert 'from "hyperformula"' not in calc_worker + assert "calculation_engine_unavailable" in calc_worker + assert "npm ci --omit=dev" in backend_dockerfile + + +def test_package_integrity_policy_accepts_exact_locked_cargo_workspace(tmp_path): + manifest, lock = _write_cargo_workspace(tmp_path) + + assert check_package_integrity._validate_cargo_workspace(manifest, lock) == [] + + +def test_package_integrity_policy_rejects_inexact_or_mismatched_cargo_workspace(tmp_path): + manifest, lock = _write_cargo_workspace( + tmp_path, + prost_spec="0.14", + ) + + failures = check_package_integrity._validate_cargo_workspace(manifest, lock) + + assert any("must use an exact =version" in failure for failure in failures) + + +def test_package_integrity_policy_rejects_cargo_lock_version_mismatch(tmp_path): + manifest, lock = _write_cargo_workspace( + tmp_path, + locked_prost_version="0.13.5", + ) + + failures = check_package_integrity._validate_cargo_workspace(manifest, lock) + + assert any("does not contain prost==0.14.4" in failure for failure in failures) + + +def test_package_integrity_policy_rejects_missing_cargo_lock(tmp_path): + manifest, lock = _write_cargo_workspace(tmp_path) + lock.unlink() + + failures = check_package_integrity._validate_cargo_workspace(manifest, lock) + + assert failures == [f"{lock}: missing Cargo dependency lock"] + + +def test_package_integrity_policy_blocks_lock_package_without_resolved_url(tmp_path): + project = _write_npm_project(tmp_path / "plugin") + package_lock_path = project / "package-lock.json" + payload = json.loads(package_lock_path.read_text(encoding="utf-8")) + del payload["packages"]["node_modules/left-pad"]["resolved"] + package_lock_path.write_text(json.dumps(payload), encoding="utf-8") + + failures = check_package_integrity._validate_npm_project(project / "package.json", package_lock_path) + + assert any("node_modules/left-pad missing resolved URL" in failure for failure in failures) + + +def test_package_integrity_policy_blocks_unbounded_or_external_backend_requirements(tmp_path): + requirements = tmp_path / "requirements.txt" + requirements.write_text( + "\n".join( + [ + "fastapi", + "internal-tool @ https://example.com/internal-tool.whl", + ] + ), + encoding="utf-8", + ) + + failures = check_package_integrity._validate_python_requirements(requirements) + + assert any("requirement must include an explicit version specifier" in failure for failure in failures) + assert any("unsupported requirement source" in failure for failure in failures) + + +def test_package_integrity_policy_keeps_office_manifest_versions_aligned(tmp_path): + matching = tmp_path / "manifest.xml" + stale = tmp_path / "manifest.production.xml.template" + matching.write_text(_office_manifest("0.5.0.0"), encoding="utf-8") + stale.write_text(_office_manifest("0.5.1.0"), encoding="utf-8") + + failures = check_package_integrity._validate_manifest_versions( + (matching, stale), + ) + + assert len(failures) == 1 + assert "Office manifest versions must match" in failures[0] + + +def test_package_integrity_policy_accepts_zero_major_four_part_office_version(tmp_path): + manifest = tmp_path / "manifest.xml" + manifest.write_text(_office_manifest("0.5.0.0"), encoding="utf-8") + + failures = check_package_integrity._validate_manifest_versions((manifest,)) + + assert failures == [] + + +def test_package_integrity_policy_rejects_invalid_office_version_parts(tmp_path): + manifest = tmp_path / "manifest.xml" + manifest.write_text(_office_manifest("0.5.0.65536"), encoding="utf-8") + + failures = check_package_integrity._validate_manifest_versions((manifest,)) + + assert failures == [f"{manifest}: invalid Office manifest version '0.5.0.65536'"] + + +def test_package_integrity_main_reports_failures(monkeypatch, tmp_path, capsys): + project = _write_npm_project(tmp_path / "plugin") + package_lock_path = project / "package-lock.json" + payload = json.loads(package_lock_path.read_text(encoding="utf-8")) + payload["packages"]["node_modules/left-pad"]["resolved"] = "http://registry.npmjs.org/left-pad/-/left-pad-1.3.0.tgz" + package_lock_path.write_text(json.dumps(payload), encoding="utf-8") + requirements = tmp_path / "requirements.txt" + requirements.write_text("fastapi>=0.115.0\n", encoding="utf-8") + lock = tmp_path / "requirements.lock.txt" + lock.write_text(_hashed_lock("fastapi==0.141.1"), encoding="utf-8") + + monkeypatch.setattr(check_package_integrity, "NPM_TARGETS", [(project / "package.json", package_lock_path)]) + monkeypatch.setattr(check_package_integrity, "BACKEND_REQUIREMENTS", requirements) + monkeypatch.setattr(check_package_integrity, "BACKEND_REQUIREMENTS_LOCK", lock) + + assert check_package_integrity.main() == 1 + + captured = capsys.readouterr() + assert "uses non-HTTPS resolved URL" in captured.err + assert "Package integrity policy passed" not in captured.out + + +def _write_npm_project(project): + project.mkdir(parents=True) + package_json = { + "name": "finance-ai-excel-plugin", + "version": "0.4.0", + "private": True, + "dependencies": {"left-pad": "^1.3.0"}, + "devDependencies": {"vitest": "^1.0.0"}, + } + package_lock = { + "name": "finance-ai-excel-plugin", + "version": "0.4.0", + "lockfileVersion": 3, + "requires": True, + "packages": { + "": { + "name": "finance-ai-excel-plugin", + "version": "0.4.0", + "dependencies": {"left-pad": "^1.3.0"}, + "devDependencies": {"vitest": "^1.0.0"}, + }, + "node_modules/left-pad": { + "version": "1.3.0", + "resolved": "https://registry.npmjs.org/left-pad/-/left-pad-1.3.0.tgz", + "integrity": "sha512-test", + }, + "node_modules/vitest": { + "version": "1.0.0", + "resolved": "https://registry.npmjs.org/vitest/-/vitest-1.0.0.tgz", + "integrity": "sha512-test", + "dev": True, + }, + }, + } + (project / "package.json").write_text(json.dumps(package_json), encoding="utf-8") + (project / "package-lock.json").write_text(json.dumps(package_lock), encoding="utf-8") + return project + + +def _hashed_lock(*requirements: str) -> str: + digest = "a" * 64 + return "".join( + f"{requirement} \\\n --hash=sha256:{digest}\n" for requirement in requirements + ) + + +def _write_cargo_workspace( + project, + *, + prost_spec: str = "=0.14.4", + locked_prost_version: str = "0.14.4", +): + project.mkdir(parents=True, exist_ok=True) + manifest = project / "Cargo.toml" + lock = project / "Cargo.lock" + manifest.write_text( + "\n".join( + [ + "[workspace]", + 'members = ["crates/protocol"]', + "resolver = \"2\"", + "", + "[workspace.dependencies]", + f'prost = "{prost_spec}"', + "", + ] + ), + encoding="utf-8", + ) + member = project / "crates" / "protocol" + member.mkdir(parents=True) + (member / "Cargo.toml").write_text( + "[package]\nname = \"protocol\"\nversion = \"0.1.0\"\n", + encoding="utf-8", + ) + lock.write_text( + "\n".join( + [ + "# This file is automatically @generated by Cargo.", + "version = 4", + "", + "[[package]]", + 'name = "prost"', + f'version = "{locked_prost_version}"', + 'source = "registry+https://github.com/rust-lang/crates.io-index"', + 'checksum = "0123456789abcdef"', + "", + ] + ), + encoding="utf-8", + ) + return manifest, lock + + +def _office_manifest(version: str) -> str: + return ( + '\n' + '\n' + f" {version}\n" + "\n" + ) diff --git a/backend/tests/contract/test_private_collaboration_execution_receipt.py b/backend/tests/contract/test_private_collaboration_execution_receipt.py new file mode 100644 index 0000000..4451d50 --- /dev/null +++ b/backend/tests/contract/test_private_collaboration_execution_receipt.py @@ -0,0 +1,319 @@ +from __future__ import annotations + +import base64 +import hashlib +import json +from pathlib import Path +from typing import Any, Callable + +import pytest +from cryptography.hazmat.primitives import serialization +from cryptography.hazmat.primitives.asymmetric.ed25519 import Ed25519PrivateKey + +from backend.benchmarks.private_collaboration_schedules import ( + PRIVATE_RELEASE_MIN_SCHEDULE_COUNT, + REQUIRED_SCENARIOS, + SCHEMA_VERSION as MANIFEST_SCHEMA, + SCHEDULE_SCHEMA_VERSION, +) +from scripts import validate_private_collaboration_execution_receipt as execution + + +RELEASE_SHA = "a" * 40 +KEY_ID = "collaboration-runner-2026-08" + + +def _schedule(index: int) -> dict[str, Any]: + actors = ["actor_00000001", "actor_00000002"] + events = [] + for sequence, kind in enumerate( + ( + "bridge_acquire", + "crdt_update", + "partition_start", + "crash", + "restart", + "partition_end", + "compact", + "undo", + "bridge_release", + ), + start=1, + ): + payload = f"private-collaboration-event-{index}-{sequence}".encode() + events.append( + { + "sequence": sequence, + "actorIdHash": actors[(sequence - 1) % 2], + "logicalTime": sequence, + "kind": kind, + "payloadBase64": base64.b64encode(payload).decode("ascii"), + "payloadSha256": f"sha256:{hashlib.sha256(payload).hexdigest()}", + "expectedDisposition": "accepted", + } + ) + return { + "schemaVersion": SCHEDULE_SCHEMA_VERSION, + "scheduleIdHash": f"schedule_{index:08x}", + "seedHash": "sha256:" + f"{index + 10_000:064x}", + "actors": actors, + "scenarios": sorted(REQUIRED_SCENARIOS), + "events": events, + "expected": { + "finalStateVectorHash": "sha256:" + f"{index + 20_000:064x}", + "conflictCount": index % 3, + "acknowledgedUpdateCount": 2, + "lostAcknowledgedUpdateCount": 0, + "authorityMode": "app", + }, + } + + +def _corpus( + root: Path, +) -> tuple[Path, list[tuple[dict[str, Any], dict[str, Any]]], bytes]: + private = root / "private-collaboration" + schedules = private / "schedules" + schedules.mkdir(parents=True) + entries: list[dict[str, Any]] = [] + payloads: list[dict[str, Any]] = [] + for index in range(PRIVATE_RELEASE_MIN_SCHEDULE_COUNT): + payload = _schedule(index) + raw = json.dumps(payload, indent=2, sort_keys=True).encode() + path = schedules / f"schedule_{index:08x}.json" + path.write_bytes(raw) + payloads.append(payload) + entries.append( + { + "scheduleIdHash": payload["scheduleIdHash"], + "schedulePath": f"schedules/{path.name}", + "scheduleSha256": f"sha256:{hashlib.sha256(raw).hexdigest()}", + "provenance": { + "kind": "randomized", + "generatorHash": "sha256:" + "b" * 64, + "sourceReceiptHash": "sha256:" + f"{index + 30_000:064x}", + }, + } + ) + manifest = json.dumps( + {"schemaVersion": MANIFEST_SCHEMA, "schedules": entries}, + indent=2, + sort_keys=True, + ).encode() + (private / "manifest.json").write_bytes(manifest) + return private, [*zip(payloads, entries, strict=True)], manifest + + +def _signed_execution( + root: Path, +) -> tuple[ + Path, + list[tuple[dict[str, Any], dict[str, Any]]], + bytes, + Path, + Path, + Path, + dict[str, Any], + Ed25519PrivateKey, +]: + private, inventory, manifest = _corpus(root) + key = Ed25519PrivateKey.generate() + trusted = root / "trusted" + trusted.mkdir() + key_path = trusted / "collaboration-public.pem" + key_path.write_bytes( + key.public_key().public_bytes( + serialization.Encoding.PEM, + serialization.PublicFormat.SubjectPublicKeyInfo, + ) + ) + results = [] + for index, (schedule, entry) in enumerate(inventory): + expected = schedule["expected"] + results.append( + { + "scheduleIdHash": schedule["scheduleIdHash"], + "scheduleSha256": entry["scheduleSha256"], + "status": "passed", + "deterministicReplayVerified": True, + "exactExpectedOutcomeVerified": True, + "observedFinalStateVectorHash": expected["finalStateVectorHash"], + "observedConflictCount": expected["conflictCount"], + "observedAcknowledgedUpdateCount": expected[ + "acknowledgedUpdateCount" + ], + "observedLostAcknowledgedUpdateCount": 0, + "observedAuthorityMode": expected["authorityMode"], + "bridgeFenceVerified": True, + "artifactSha256": "sha256:" + f"{index + 40_000:064x}", + } + ) + payload = { + "schemaVersion": execution.SCHEMA_VERSION, + "releaseSha": RELEASE_SHA, + "signingKeyId": KEY_ID, + "runner": { + "runtimeKind": execution.RUNTIME_KIND, + "implementationSha256": "sha256:" + "c" * 64, + "runnerImageSha256": "sha256:" + "d" * 64, + "executorSha256": "sha256:" + "e" * 64, + "deterministicReplayCount": 2, + }, + "corpus": { + "manifestSha256": f"sha256:{hashlib.sha256(manifest).hexdigest()}", + "scheduleCount": len(results), + "failedCount": 0, + "results": results, + }, + } + artifact = root / "artifact" + artifact.mkdir() + receipt = artifact / "collaboration-execution.json" + signature = artifact / "collaboration-execution.ed25519" + _write_signed(receipt, signature, payload, key) + return private, inventory, manifest, receipt, signature, key_path, payload, key + + +def _write_signed( + receipt: Path, + signature: Path, + payload: dict[str, Any], + key: Ed25519PrivateKey, +) -> None: + raw = json.dumps(payload, indent=2, sort_keys=True).encode() + receipt.write_bytes(raw) + signature.write_bytes(key.sign(raw)) + + +def _validate(root: Path) -> tuple[dict[str, Any], tuple[Any, ...]]: + fixture = _signed_execution(root) + private, _inventory, _manifest, receipt, signature, key_path, _payload, _key = fixture + report = execution.validate_private_collaboration_execution_receipt( + receipt_path=receipt, + signature_path=signature, + trusted_public_key_path=key_path, + trusted_signing_key_id=KEY_ID, + expected_release_sha=RELEASE_SHA, + private_root=private, + ) + return report, fixture + + +def test_signed_collaboration_execution_binds_all_100_schedules(tmp_path: Path) -> None: + report, fixture = _validate(tmp_path) + private = fixture[0] + encoded = json.dumps(report, sort_keys=True) + + assert report["status"] == "ready" + assert report["signatureVerified"] is True + assert report["runtimeKind"] == execution.RUNTIME_KIND + assert report["scheduleCount"] == 100 + assert report["failedCount"] == 0 + assert report["deterministicReplayCount"] == 2 + assert report["eventCount"] == 900 + assert report["acknowledgedUpdateCount"] == 200 + assert report["lostAcknowledgedUpdateCount"] == 0 + assert report["requiredScenarios"] == sorted(REQUIRED_SCENARIOS) + assert str(private) not in encoded + assert "private-collaboration-event" not in encoded + + +def test_collaboration_execution_rejects_signed_outcome_mismatch(tmp_path: Path) -> None: + _report, fixture = _validate(tmp_path) + private, _inventory, _manifest, receipt, signature, key_path, payload, key = fixture + payload["corpus"]["results"][0]["observedFinalStateVectorHash"] = ( + "sha256:" + "0" * 64 + ) + _write_signed(receipt, signature, payload, key) + + with pytest.raises( + execution.PrivateCollaborationExecutionError, + match="exactly match", + ): + execution.validate_private_collaboration_execution_receipt( + receipt_path=receipt, + signature_path=signature, + trusted_public_key_path=key_path, + trusted_signing_key_id=KEY_ID, + expected_release_sha=RELEASE_SHA, + private_root=private, + ) + + +@pytest.mark.parametrize( + "mutate, message", + [ + ( + lambda payload: payload["runner"].__setitem__( + "deterministicReplayCount", 1 + ), + "at least twice", + ), + ( + lambda payload: payload["corpus"].__setitem__("failedCount", False), + "non-negative integer", + ), + ( + lambda payload: payload["corpus"]["results"][0].__setitem__( + "bridgeFenceVerified", False + ), + "exactly match", + ), + ], +) +def test_collaboration_execution_rejects_incomplete_runtime_proof( + tmp_path: Path, + mutate: Callable[[dict[str, Any]], None], + message: str, +) -> None: + _report, fixture = _validate(tmp_path) + private, _inventory, _manifest, receipt, signature, key_path, payload, key = fixture + mutate(payload) + _write_signed(receipt, signature, payload, key) + + with pytest.raises(execution.PrivateCollaborationExecutionError, match=message): + execution.validate_private_collaboration_execution_receipt( + receipt_path=receipt, + signature_path=signature, + trusted_public_key_path=key_path, + trusted_signing_key_id=KEY_ID, + expected_release_sha=RELEASE_SHA, + private_root=private, + ) + + +def test_collaboration_execution_rejects_tampered_receipt_and_embedded_key( + tmp_path: Path, +) -> None: + _report, fixture = _validate(tmp_path) + private, _inventory, _manifest, receipt, signature, key_path, _payload, _key = fixture + receipt.write_bytes(receipt.read_bytes().replace(b'"failedCount": 0', b'"failedCount": 1')) + with pytest.raises( + execution.PrivateCollaborationExecutionError, + match="signature verification", + ): + execution.validate_private_collaboration_execution_receipt( + receipt_path=receipt, + signature_path=signature, + trusted_public_key_path=key_path, + trusted_signing_key_id=KEY_ID, + expected_release_sha=RELEASE_SHA, + private_root=private, + ) + + _report, fixture = _validate(tmp_path / "second") + private, _inventory, _manifest, receipt, signature, key_path, _payload, _key = fixture + embedded = receipt.parent / "embedded.pem" + embedded.write_bytes(key_path.read_bytes()) + with pytest.raises( + execution.PrivateCollaborationExecutionError, + match="artifact root", + ): + execution.validate_private_collaboration_execution_receipt( + receipt_path=receipt, + signature_path=signature, + trusted_public_key_path=embedded, + trusted_signing_key_id=KEY_ID, + expected_release_sha=RELEASE_SHA, + private_root=private, + ) diff --git a/backend/tests/contract/test_private_collaboration_schedules.py b/backend/tests/contract/test_private_collaboration_schedules.py new file mode 100644 index 0000000..79f47d3 --- /dev/null +++ b/backend/tests/contract/test_private_collaboration_schedules.py @@ -0,0 +1,255 @@ +from __future__ import annotations + +import base64 +import hashlib +import json +from pathlib import Path +from typing import Any + +from backend.benchmarks.private_collaboration_schedules import ( + PRIVATE_RELEASE_MIN_SCHEDULE_COUNT, + REQUIRED_SCENARIOS, + SCHEMA_VERSION, + SCHEDULE_SCHEMA_VERSION, + load_private_collaboration_schedule_manifest, + main, + private_collaboration_schedule_policy, + private_collaboration_schedule_policy_snapshot, +) + + +def _schedule_payload(index: int) -> dict[str, Any]: + actors = ["actor_00000001", "actor_00000002"] + event_kinds = [ + "bridge_acquire", + "crdt_update", + "partition_start", + "crash", + "restart", + "partition_end", + "compact", + "undo", + "bridge_release", + ] + events = [] + for sequence, kind in enumerate(event_kinds, start=1): + payload = f"private-operation-{index}-{sequence}".encode() + events.append( + { + "sequence": sequence, + "actorIdHash": actors[(sequence - 1) % len(actors)], + "logicalTime": sequence, + "kind": kind, + "payloadBase64": base64.b64encode(payload).decode("ascii"), + "payloadSha256": f"sha256:{hashlib.sha256(payload).hexdigest()}", + "expectedDisposition": "accepted", + } + ) + return { + "schemaVersion": SCHEDULE_SCHEMA_VERSION, + "scheduleIdHash": f"schedule_{index:08x}", + "seedHash": "sha256:" + f"{index + 50_000:064x}", + "actors": actors, + "scenarios": sorted(REQUIRED_SCENARIOS), + "events": events, + "expected": { + "finalStateVectorHash": "sha256:" + f"{index + 60_000:064x}", + "conflictCount": 0, + "acknowledgedUpdateCount": 2, + "lostAcknowledgedUpdateCount": 0, + "authorityMode": "app", + }, + } + + +def _write_corpus(root: Path, count: int = PRIVATE_RELEASE_MIN_SCHEDULE_COUNT): + schedules_dir = root / "schedules" + schedules_dir.mkdir(parents=True) + entries = [] + paths = [] + for index in range(count): + payload = _schedule_payload(index) + raw = json.dumps(payload, indent=2, sort_keys=True).encode() + path = schedules_dir / f"schedule_{index:08x}.json" + path.write_bytes(raw) + paths.append(path) + entries.append( + { + "scheduleIdHash": payload["scheduleIdHash"], + "schedulePath": f"schedules/{path.name}", + "scheduleSha256": f"sha256:{hashlib.sha256(raw).hexdigest()}", + "provenance": { + "kind": "randomized", + "generatorHash": "sha256:" + "a" * 64, + "sourceReceiptHash": "sha256:" + f"{index + 70_000:064x}", + }, + } + ) + manifest = root / "manifest.json" + manifest.write_text( + json.dumps({"schemaVersion": SCHEMA_VERSION, "schedules": entries}, indent=2, sort_keys=True), + encoding="utf-8", + ) + return manifest, paths, entries + + +def test_release_scale_collaboration_schedule_manifest_is_ready_and_safe(tmp_path: Path) -> None: + _write_corpus(tmp_path) + + validation = load_private_collaboration_schedule_manifest(tmp_path) + policy = private_collaboration_schedule_policy(validation) + encoded = json.dumps(validation.to_safe_dict(), sort_keys=True) + + assert validation.status == "ready" + assert validation.schedule_count == 100 + assert policy["ready"] is True + assert policy["minimumScheduleCount"] == 100 + assert policy["missingScenarios"] == [] + assert validation.schedules[0].actor_count == 2 + assert validation.schedules[0].event_count == 9 + assert validation.schedules[0].expected_acknowledged_update_count == 2 + assert str(tmp_path) not in encoded + assert "private-operation" not in encoded + assert "payloadBase64" not in encoded + + +def test_collaboration_schedule_manifest_requires_100_schedules(tmp_path: Path) -> None: + _write_corpus(tmp_path, count=99) + + snapshot = private_collaboration_schedule_policy_snapshot(tmp_path) + + assert snapshot["status"] == "insufficient" + assert snapshot["ready"] is False + assert snapshot["scheduleCount"] == 99 + assert snapshot["releasePolicy"]["minimumScheduleCount"] == 100 + assert "private_collaboration_minimum_count_missing" in snapshot["issueCodes"] + + +def test_collaboration_schedule_payload_tamper_fails_without_leaking_bytes(tmp_path: Path) -> None: + _manifest, paths, entries = _write_corpus(tmp_path, count=1) + payload = json.loads(paths[0].read_text(encoding="utf-8")) + payload["events"][0]["payloadBase64"] = base64.b64encode( + b"Customer Alpha private formula =SUM(Secret!A1:A9)" + ).decode("ascii") + raw = json.dumps(payload, indent=2, sort_keys=True).encode() + paths[0].write_bytes(raw) + entries[0]["scheduleSha256"] = f"sha256:{hashlib.sha256(raw).hexdigest()}" + (tmp_path / "manifest.json").write_text( + json.dumps({"schemaVersion": SCHEMA_VERSION, "schedules": entries}), + encoding="utf-8", + ) + + validation = load_private_collaboration_schedule_manifest(tmp_path) + encoded = json.dumps(validation.to_safe_dict(), sort_keys=True) + + assert validation.status == "invalid" + assert "schedule_payload_invalid" in encoded + assert "Customer Alpha" not in encoded + assert "Secret" not in encoded + + +def test_collaboration_schedule_rejects_symlinked_payload(tmp_path: Path) -> None: + _manifest, paths, entries = _write_corpus(tmp_path, count=1) + outside = tmp_path.parent / f"{tmp_path.name}-outside-schedule.json" + outside.write_bytes(paths[0].read_bytes()) + paths[0].unlink() + try: + paths[0].symlink_to(outside) + except OSError: + return + + validation = load_private_collaboration_schedule_manifest(tmp_path) + + assert validation.status == "invalid" + assert {issue.code for issue in validation.issues} == {"schedule_payload_invalid"} + + +def test_collaboration_schedule_rejects_lost_acknowledged_update(tmp_path: Path) -> None: + _manifest, paths, entries = _write_corpus(tmp_path, count=1) + payload = json.loads(paths[0].read_text(encoding="utf-8")) + payload["expected"]["lostAcknowledgedUpdateCount"] = 1 + raw = json.dumps(payload, indent=2, sort_keys=True).encode() + paths[0].write_bytes(raw) + entries[0]["scheduleSha256"] = f"sha256:{hashlib.sha256(raw).hexdigest()}" + (tmp_path / "manifest.json").write_text( + json.dumps({"schemaVersion": SCHEMA_VERSION, "schedules": entries}), + encoding="utf-8", + ) + + validation = load_private_collaboration_schedule_manifest(tmp_path) + + assert validation.status == "invalid" + assert {issue.code for issue in validation.issues} == {"schedule_payload_invalid"} + + +def test_collaboration_schedule_rejects_duplicate_random_seed(tmp_path: Path) -> None: + _manifest, paths, entries = _write_corpus(tmp_path, count=2) + first = json.loads(paths[0].read_text(encoding="utf-8")) + second = json.loads(paths[1].read_text(encoding="utf-8")) + second["seedHash"] = first["seedHash"] + raw = json.dumps(second, indent=2, sort_keys=True).encode() + paths[1].write_bytes(raw) + entries[1]["scheduleSha256"] = f"sha256:{hashlib.sha256(raw).hexdigest()}" + (tmp_path / "manifest.json").write_text( + json.dumps({"schemaVersion": SCHEMA_VERSION, "schedules": entries}), + encoding="utf-8", + ) + + validation = load_private_collaboration_schedule_manifest(tmp_path) + + assert validation.status == "invalid" + assert {issue.code for issue in validation.issues} == {"schedule_seed_duplicate"} + + +def test_collaboration_manifest_example_uses_current_schema() -> None: + example = ( + Path(__file__).resolve().parents[2] + / "benchmarks" + / "fixtures" + / "private-collaboration" + / "manifest.example.json" + ) + payload = json.loads(example.read_text(encoding="utf-8")) + + assert payload["schemaVersion"] == SCHEMA_VERSION + assert set(payload["schedules"][0]) == { + "scheduleIdHash", + "schedulePath", + "scheduleSha256", + "provenance", + } + + +def test_collaboration_schedule_cli_fails_closed_without_private_corpus( + tmp_path: Path, + capsys, +) -> None: + assert main(["--private-root", str(tmp_path), "--require-release-scale", "--json"]) == 1 + report = json.loads(capsys.readouterr().out) + assert report["status"] == "missing" + assert report["ready"] is False + assert str(tmp_path) not in json.dumps(report) + + +def test_collaboration_schedule_json_schemas_match_runtime_versions() -> None: + schema_root = ( + Path(__file__).resolve().parents[2] + / "benchmarks" + / "fixtures" + / "private-collaboration" + ) + manifest_schema = json.loads( + (schema_root / "manifest.schema.json").read_text(encoding="utf-8") + ) + schedule_schema = json.loads( + (schema_root / "schedule.schema.json").read_text(encoding="utf-8") + ) + + assert manifest_schema["properties"]["schemaVersion"]["const"] == SCHEMA_VERSION + assert ( + schedule_schema["properties"]["schemaVersion"]["const"] + == SCHEDULE_SCHEMA_VERSION + ) + assert schedule_schema["properties"]["expected"]["properties"][ + "lostAcknowledgedUpdateCount" + ] == {"const": 0} diff --git a/backend/tests/contract/test_private_corpus_execution_receipt.py b/backend/tests/contract/test_private_corpus_execution_receipt.py new file mode 100644 index 0000000..9aad890 --- /dev/null +++ b/backend/tests/contract/test_private_corpus_execution_receipt.py @@ -0,0 +1,263 @@ +from __future__ import annotations + +import hashlib +import json +import zipfile +from pathlib import Path +from typing import Any + +import pytest +from cryptography.hazmat.primitives import serialization +from cryptography.hazmat.primitives.asymmetric.ed25519 import Ed25519PrivateKey + +from backend.benchmarks.private_golden_fixtures import SCHEMA_VERSION as MANIFEST_SCHEMA +from scripts import validate_private_corpus_execution_receipt as execution + + +RELEASE_SHA = "a" * 40 +KEY_ID = "excel-corpus-windows-2026-08" +CATEGORIES = ( + "financial_model", + "protected_workbook", + "messy_formulas_data", + "charts_tables_pivots_macros", +) + + +def _fixture_root(root: Path, count: int = 250) -> tuple[Path, list[dict[str, Any]], bytes]: + private = root / "private" + workbooks = private / "workbooks" + workbooks.mkdir(parents=True) + fixtures = [] + for index in range(count): + fixture_id = f"fx_{index:08x}" + workbook = workbooks / f"{fixture_id}.xlsx" + with zipfile.ZipFile(workbook, "w") as archive: + archive.writestr( + "[Content_Types].xml", + "" + "" + "" + "", + ) + archive.writestr("xl/workbook.xml", f"") + workbook_sha = f"sha256:{hashlib.sha256(workbook.read_bytes()).hexdigest()}" + fixtures.append( + { + "fixtureIdHash": fixture_id, + "category": CATEGORIES[index % len(CATEGORIES)], + "workbookPath": f"workbooks/{fixture_id}.xlsx", + "workbookSha256": workbook_sha, + "provenance": { + "kind": "excel_authored", + "excelBuildHash": "sha256:" + "b" * 64, + "sourceReceiptHash": "sha256:" + f"{index + 1:064x}", + }, + "expectedTruthTargets": ["truth"], + "expectedFormulaFacts": [{"formulaHash": "sha256:" + "c" * 64}], + "expectedSimulationGates": ["checkpoint_required"], + "rollbackExpectations": { + "required": True, + "dirtyRanges": ["Model!A1"], + }, + "repairOutcomes": [{"kind": "formula_repair"}], + "auditExpectations": {"requiredEvidence": ["truth_model"]}, + } + ) + manifest = json.dumps( + {"schemaVersion": MANIFEST_SCHEMA, "fixtures": fixtures}, + indent=2, + sort_keys=True, + ).encode() + (private / "manifest.json").write_bytes(manifest) + return private, fixtures, manifest + + +def _signed_receipt( + root: Path, + fixtures: list[dict[str, Any]], + manifest: bytes, +) -> tuple[Path, Path, Path, dict[str, Any], Ed25519PrivateKey]: + key = Ed25519PrivateKey.generate() + trusted_root = root / "trusted" + trusted_root.mkdir() + key_path = trusted_root / "trusted-key.pem" + key_path.write_bytes( + key.public_key().public_bytes( + serialization.Encoding.PEM, + serialization.PublicFormat.SubjectPublicKeyInfo, + ) + ) + results = [ + { + "fixtureIdHash": fixture["fixtureIdHash"], + "workbookSha256": fixture["workbookSha256"], + "status": "passed", + "openVerified": True, + "recalculationVerified": True, + "saveVerified": True, + "repairDialogDetected": False, + "assertionCount": 7, + "artifactSha256": "sha256:" + f"{index + 10_000:064x}", + } + for index, fixture in enumerate(fixtures) + ] + payload = { + "schemaVersion": execution.SCHEMA_VERSION, + "releaseSha": RELEASE_SHA, + "signingKeyId": KEY_ID, + "runner": { + "hostPlatform": "windows", + "nativeExcel": True, + "excelBuildHash": "sha256:" + "d" * 64, + "runnerImageHash": "sha256:" + "e" * 64, + "executorHash": "sha256:" + "f" * 64, + }, + "corpus": { + "manifestSha256": f"sha256:{hashlib.sha256(manifest).hexdigest()}", + "workbookCount": len(results), + "failedCount": 0, + "results": results, + }, + } + artifact_root = root / "artifact" + artifact_root.mkdir() + receipt_path = artifact_root / "execution-receipt.json" + signature_path = artifact_root / "execution-receipt.ed25519" + raw = json.dumps(payload, indent=2, sort_keys=True).encode() + receipt_path.write_bytes(raw) + signature_path.write_bytes(key.sign(raw)) + return receipt_path, signature_path, key_path, payload, key + + +def _validate(root: Path, private: Path, fixtures: list[dict[str, Any]], manifest: bytes): + receipt, signature, key_path, payload, key = _signed_receipt(root, fixtures, manifest) + report = execution.validate_private_corpus_execution_receipt( + receipt_path=receipt, + signature_path=signature, + trusted_public_key_path=key_path, + trusted_signing_key_id=KEY_ID, + expected_release_sha=RELEASE_SHA, + private_root=private, + ) + return report, receipt, signature, payload, key + + +def test_signed_native_execution_receipt_binds_all_250_private_workbooks(tmp_path: Path) -> None: + private, fixtures, manifest = _fixture_root(tmp_path) + report, *_ = _validate(tmp_path, private, fixtures, manifest) + encoded = json.dumps(report, sort_keys=True) + + assert report["status"] == "ready" + assert report["signatureVerified"] is True + assert report["releaseSha"] == RELEASE_SHA + assert report["hostPlatform"] == "windows" + assert report["nativeExcelExecuted"] is True + assert report["workbookCount"] == 250 + assert report["failedCount"] == 0 + assert report["assertionCount"] == 1750 + assert str(private) not in encoded + assert "workbooks/" not in encoded + + +def test_native_execution_receipt_rejects_tamper_and_wrong_inventory(tmp_path: Path) -> None: + private, fixtures, manifest = _fixture_root(tmp_path) + _report, receipt, signature, payload, key = _validate(tmp_path, private, fixtures, manifest) + receipt.write_bytes(receipt.read_bytes().replace(b'"failedCount": 0', b'"failedCount": 1')) + with pytest.raises(execution.PrivateCorpusExecutionError, match="signature verification"): + execution.validate_private_corpus_execution_receipt( + receipt_path=receipt, + signature_path=signature, + trusted_public_key_path=tmp_path / "trusted" / "trusted-key.pem", + trusted_signing_key_id=KEY_ID, + expected_release_sha=RELEASE_SHA, + private_root=private, + ) + + payload["corpus"]["results"][0]["workbookSha256"] = "sha256:" + "0" * 64 + raw = json.dumps(payload, indent=2, sort_keys=True).encode() + receipt.write_bytes(raw) + signature.write_bytes(key.sign(raw)) + with pytest.raises(execution.PrivateCorpusExecutionError, match="inventory"): + execution.validate_private_corpus_execution_receipt( + receipt_path=receipt, + signature_path=signature, + trusted_public_key_path=tmp_path / "trusted" / "trusted-key.pem", + trusted_signing_key_id=KEY_ID, + expected_release_sha=RELEASE_SHA, + private_root=private, + ) + + +def test_native_execution_receipt_rejects_non_native_or_incomplete_results(tmp_path: Path) -> None: + private, fixtures, manifest = _fixture_root(tmp_path) + _report, receipt, signature, payload, key = _validate(tmp_path, private, fixtures, manifest) + payload["runner"]["nativeExcel"] = False + raw = json.dumps(payload, indent=2, sort_keys=True).encode() + receipt.write_bytes(raw) + signature.write_bytes(key.sign(raw)) + with pytest.raises(execution.PrivateCorpusExecutionError, match="native Excel"): + execution.validate_private_corpus_execution_receipt( + receipt_path=receipt, + signature_path=signature, + trusted_public_key_path=tmp_path / "trusted" / "trusted-key.pem", + trusted_signing_key_id=KEY_ID, + expected_release_sha=RELEASE_SHA, + private_root=private, + ) + + payload["runner"]["nativeExcel"] = True + payload["corpus"]["results"][0]["repairDialogDetected"] = True + raw = json.dumps(payload, indent=2, sort_keys=True).encode() + receipt.write_bytes(raw) + signature.write_bytes(key.sign(raw)) + with pytest.raises(execution.PrivateCorpusExecutionError, match="pass all host checks"): + execution.validate_private_corpus_execution_receipt( + receipt_path=receipt, + signature_path=signature, + trusted_public_key_path=tmp_path / "trusted" / "trusted-key.pem", + trusted_signing_key_id=KEY_ID, + expected_release_sha=RELEASE_SHA, + private_root=private, + ) + + +def test_native_execution_receipt_rejects_key_from_artifact_root(tmp_path: Path) -> None: + private, fixtures, manifest = _fixture_root(tmp_path) + _report, receipt, signature, _payload, _key = _validate( + tmp_path, private, fixtures, manifest + ) + embedded_key = receipt.parent / "embedded-key.pem" + embedded_key.write_bytes((tmp_path / "trusted" / "trusted-key.pem").read_bytes()) + + with pytest.raises(execution.PrivateCorpusExecutionError, match="artifact root"): + execution.validate_private_corpus_execution_receipt( + receipt_path=receipt, + signature_path=signature, + trusted_public_key_path=embedded_key, + trusted_signing_key_id=KEY_ID, + expected_release_sha=RELEASE_SHA, + private_root=private, + ) + + +def test_native_execution_receipt_rejects_boolean_count_alias(tmp_path: Path) -> None: + private, fixtures, manifest = _fixture_root(tmp_path) + _report, receipt, signature, payload, key = _validate( + tmp_path, private, fixtures, manifest + ) + payload["corpus"]["failedCount"] = False + raw = json.dumps(payload, indent=2, sort_keys=True).encode() + receipt.write_bytes(raw) + signature.write_bytes(key.sign(raw)) + + with pytest.raises(execution.PrivateCorpusExecutionError, match="counts"): + execution.validate_private_corpus_execution_receipt( + receipt_path=receipt, + signature_path=signature, + trusted_public_key_path=tmp_path / "trusted" / "trusted-key.pem", + trusted_signing_key_id=KEY_ID, + expected_release_sha=RELEASE_SHA, + private_root=private, + ) diff --git a/backend/tests/contract/test_private_golden_fixtures.py b/backend/tests/contract/test_private_golden_fixtures.py new file mode 100644 index 0000000..3153d7e --- /dev/null +++ b/backend/tests/contract/test_private_golden_fixtures.py @@ -0,0 +1,693 @@ +import hashlib +import json +import zipfile +from dataclasses import replace +from pathlib import Path +from types import SimpleNamespace + +from backend.benchmarks import v7_golden_cases +from backend.benchmarks.private_golden_fixtures import ( + PRIVATE_RELEASE_MIN_FIXTURE_COUNT, + SCHEMA_VERSION, + PrivateGoldenFixture, + PrivateGoldenFixtureValidation, + load_private_golden_fixture_manifest, + private_fixture_release_policy, + private_golden_fixture_policy_snapshot, +) +from scripts import validate_release_evidence_index as release_evidence +from scripts.run_private_corpus_preflight import run_private_corpus_preflight + + +def test_missing_private_manifest_returns_missing(tmp_path): + result = load_private_golden_fixture_manifest(tmp_path) + + assert result.status == "missing" + assert result.fixture_count == 0 + assert result.fixtures == () + assert result.issues == () + + +def test_private_fixture_policy_snapshot_is_required_safe_and_actionable(tmp_path): + snapshot = private_golden_fixture_policy_snapshot(tmp_path, require_private_fixtures=True) + safe_payload = json.dumps(snapshot, sort_keys=True) + + assert snapshot["status"] == "missing" + assert snapshot["ready"] is False + assert snapshot["required"] is True + assert snapshot["fixtureCount"] == 0 + assert snapshot["issueCodes"] == [] + assert snapshot["requiredActions"] + assert str(tmp_path) not in safe_payload + assert "manifest.json" not in safe_payload + + +def test_invalid_private_manifest_does_not_leak_private_values(tmp_path): + manifest = tmp_path / "manifest.json" + manifest.write_text( + json.dumps( + { + "schemaVersion": SCHEMA_VERSION, + "fixtures": [ + { + "fixtureIdHash": "fx_customer-secret", + "category": "customer_like_model", + "workbookPath": "workbooks/Customer MegaCorp Renewal.xlsx", + "expectedTruthTargets": ["MegaCorp renewal revenue"], + "expectedFormulaFacts": [{"formula": "='Secret Customers'!B7"}], + "expectedSimulationGates": ["checkpoint_required"], + "rollbackExpectations": {"required": True, "dirtyRanges": ["Secret!B7"]}, + "repairOutcomes": ["repair_plan_expected"], + "auditExpectations": {"requiredEvidence": ["truth_model"]}, + } + ], + } + ), + encoding="utf-8", + ) + + result = load_private_golden_fixture_manifest(tmp_path) + safe_payload = json.dumps(result.to_safe_dict(), sort_keys=True) + + assert result.status == "invalid" + assert result.fixtures == () + assert "MegaCorp" not in safe_payload + assert "Secret Customers" not in safe_payload + assert "renewal revenue" not in safe_payload + assert {issue.code for issue in result.issues} >= {"fixture_id_hash_required", "workbook_path_missing"} + + +def test_ready_private_manifest_requires_existing_workbook_and_metadata(tmp_path): + workbook = tmp_path / "workbooks" / "fx_a8b553a4.xlsx" + workbook.parent.mkdir() + _write_minimal_workbook_package(workbook) + (tmp_path / "manifest.json").write_text( + json.dumps(_manifest_payload(workbook=workbook)), encoding="utf-8" + ) + + result = load_private_golden_fixture_manifest(tmp_path) + + assert result.status == "ready" + assert result.fixture_count == 1 + assert result.issues == () + assert result.fixtures[0].fixture_id_hash == "fx_a8b553a4" + assert result.fixtures[0].workbook_path == "workbooks/fx_a8b553a4.xlsx" + safe_fixture = result.to_safe_dict()["fixtures"][0] + assert safe_fixture["expectedFormulaFactCount"] == 1 + assert safe_fixture["workbookExtension"] == ".xlsx" + assert safe_fixture["workbookSizeClass"] == "small" + + +def test_private_fixture_policy_requires_release_scale_and_category_coverage(tmp_path): + workbook = tmp_path / "workbooks" / "fx_a8b553a4.xlsx" + workbook.parent.mkdir() + _write_minimal_workbook_package(workbook) + (tmp_path / "manifest.json").write_text( + json.dumps(_manifest_payload(workbook=workbook)), encoding="utf-8" + ) + + snapshot = private_golden_fixture_policy_snapshot(tmp_path, require_private_fixtures=True) + + assert snapshot["status"] == "insufficient" + assert snapshot["ready"] is False + assert snapshot["required"] is True + assert snapshot["fixtureCount"] == 1 + assert "private_fixture_minimum_count_missing" in snapshot["issueCodes"] + assert "private_fixture_required_categories_missing" in snapshot["issueCodes"] + assert snapshot["releasePolicy"]["minimumFixtureCount"] == 250 + assert set(snapshot["releasePolicy"]["missingCategories"]) >= { + "charts_tables_pivots_macros", + "financial_model", + "messy_formulas_data", + "protected_workbook", + } + + +def test_synthetic_private_corpus_shape_manifest_is_ready_and_safe(tmp_path): + raw_formulas = [ + "=SUM(Revenue!B12:B24)", + "='Protected Model'!$F$42*(1+Scenario!B7)", + "=XLOOKUP(A2,RawDump!A:A,RawDump!G:G)", + "=GETPIVOTDATA(\"Revenue\",Pivot!$A$3)", + ] + workbook_paths = [ + "workbooks/fx_10000001.xlsx", + "workbooks/fx_10000002.xlsx", + "workbooks/fx_10000003.xlsx", + "workbooks/fx_10000004.xlsm", + ] + for workbook_path in workbook_paths: + workbook = tmp_path / workbook_path + workbook.parent.mkdir(exist_ok=True) + _write_minimal_workbook_package(workbook) + + (tmp_path / "manifest.json").write_text( + json.dumps( + { + "schemaVersion": SCHEMA_VERSION, + "fixtures": [ + _private_corpus_fixture( + fixture_id_hash="fx_10000001", + category="financial_model", + workbook_path=workbook_paths[0], + workbook=tmp_path / workbook_paths[0], + truth_targets=["three_statement_bridge", "debt_schedule_rollforward"], + formula_hash="sha256:" + "1" * 64, + dependency_hashes=["sha256:" + "2" * 64], + audit_evidence=["truth_model", "formula_graph", "rollback_snapshot"], + ), + _private_corpus_fixture( + fixture_id_hash="fx_10000002", + category="protected_workbook", + workbook_path=workbook_paths[1], + workbook=tmp_path / workbook_paths[1], + truth_targets=["protected_formula_integrity", "locked_range_detection"], + formula_hash="sha256:" + "3" * 64, + dependency_hashes=["sha256:" + "4" * 64], + audit_evidence=["protection_state", "simulation_gate"], + ), + _private_corpus_fixture( + fixture_id_hash="fx_10000003", + category="messy_formulas_data", + workbook_path=workbook_paths[2], + workbook=tmp_path / workbook_paths[2], + truth_targets=["messy_source_normalization", "table_reference_repair"], + formula_hash="sha256:" + "5" * 64, + dependency_hashes=["sha256:" + "6" * 64, "sha256:" + "7" * 64], + audit_evidence=["source_evidence", "repair_trace"], + ), + _private_corpus_fixture( + fixture_id_hash="fx_10000004", + category="charts_tables_pivots_macros", + workbook_path=workbook_paths[3], + workbook=tmp_path / workbook_paths[3], + truth_targets=["chart_source_preserved", "pivot_refresh_guard", "macro_surface_detected"], + formula_hash="sha256:" + "8" * 64, + dependency_hashes=["sha256:" + "9" * 64], + audit_evidence=["object_fidelity", "macro_inventory", "pivot_metadata"], + ), + ], + } + ), + encoding="utf-8", + ) + + result = load_private_golden_fixture_manifest(tmp_path) + safe_payload = json.dumps(result.to_safe_dict(), sort_keys=True) + + assert result.status == "ready" + assert result.fixture_count == 4 + assert result.issues == () + assert {fixture.category for fixture in result.fixtures} == { + "financial_model", + "protected_workbook", + "messy_formulas_data", + "charts_tables_pivots_macros", + } + assert [fixture.workbook_path for fixture in result.fixtures] == workbook_paths + assert all(fixture.expected_formula_facts for fixture in result.fixtures) + assert str(tmp_path) not in safe_payload + assert "workbooks/" not in safe_payload + assert "formulaHash" not in safe_payload + for raw_formula in raw_formulas: + assert raw_formula not in safe_payload + + snapshot = private_golden_fixture_policy_snapshot(tmp_path, require_private_fixtures=True) + assert snapshot["status"] == "insufficient" + assert snapshot["ready"] is False + assert snapshot["releasePolicy"]["missingCategories"] == [] + assert "private_fixture_minimum_count_missing" in snapshot["issueCodes"] + + +def test_release_policy_requires_full_250_workbook_corpus() -> None: + template = PrivateGoldenFixture( + fixture_id_hash="fx_00000000", + category="financial_model", + workbook_path="workbooks/fx_00000000.xlsx", + workbook_sha256="sha256:" + "a" * 64, + workbook_extension=".xlsx", + workbook_size_class="small", + provenance_kind="excel_authored", + expected_truth_targets=("truth",), + expected_formula_facts=({"formulaHash": "sha256:" + "0" * 64},), + expected_simulation_gates=("checkpoint_required",), + rollback_expectations={"required": True}, + repair_outcomes=({"kind": "formula_repair"},), + audit_expectations={"requiredEvidence": ["truth_model"]}, + ) + categories = ( + "financial_model", + "protected_workbook", + "messy_formulas_data", + "charts_tables_pivots_macros", + ) + fixtures = tuple( + replace( + template, + fixture_id_hash=f"fx_{index:08x}", + category=categories[index % len(categories)], + workbook_path=f"workbooks/fx_{index:08x}.xlsx", + ) + for index in range(PRIVATE_RELEASE_MIN_FIXTURE_COUNT) + ) + validation = PrivateGoldenFixtureValidation( + status="ready", + schema_version=SCHEMA_VERSION, + fixture_count=len(fixtures), + fixtures=fixtures, + ) + + policy = private_fixture_release_policy(validation) + + assert policy["ready"] is True + assert policy["minimumFixtureCount"] == 250 + assert policy["missingCategories"] == [] + assert release_evidence.PRIVATE_CORPUS_MIN_WORKBOOKS == PRIVATE_RELEASE_MIN_FIXTURE_COUNT + + +def test_private_manifest_rejects_raw_formula_string_facts(tmp_path): + workbook = tmp_path / "workbooks" / "fx_a8b553a4.xlsx" + workbook.parent.mkdir() + _write_minimal_workbook_package(workbook) + payload = _manifest_payload(workbook=workbook) + payload["fixtures"][0]["expectedFormulaFacts"] = ["='Sensitive Model'!B20"] + (tmp_path / "manifest.json").write_text(json.dumps(payload), encoding="utf-8") + + result = load_private_golden_fixture_manifest(tmp_path) + safe_payload = json.dumps(result.to_safe_dict(), sort_keys=True) + + assert result.status == "invalid" + assert "formula_fact_must_be_object" in {issue.code for issue in result.issues} + assert "Sensitive Model" not in safe_payload + + +def test_private_manifest_rejects_formula_fact_objects_with_raw_formula_text(tmp_path): + workbook = tmp_path / "workbooks" / "fx_a8b553a4.xlsx" + workbook.parent.mkdir() + _write_minimal_workbook_package(workbook) + payload = _manifest_payload(workbook=workbook) + payload["fixtures"][0]["expectedFormulaFacts"][0]["formula"] = "='Sensitive Model'!B20" + (tmp_path / "manifest.json").write_text(json.dumps(payload), encoding="utf-8") + + result = load_private_golden_fixture_manifest(tmp_path) + safe_payload = json.dumps(result.to_safe_dict(), sort_keys=True) + + assert result.status == "invalid" + assert "formula_fact_raw_formula_forbidden" in {issue.code for issue in result.issues} + assert "Sensitive Model" not in safe_payload + + +def test_private_manifest_rejects_fake_workbook_bytes_without_leaking_path(tmp_path): + workbook = tmp_path / "workbooks" / "fx_a8b553a4.xlsx" + workbook.parent.mkdir() + workbook.write_bytes(b"fake workbook bytes") + (tmp_path / "manifest.json").write_text( + json.dumps(_manifest_payload(workbook=workbook)), encoding="utf-8" + ) + + result = load_private_golden_fixture_manifest(tmp_path) + safe_payload = json.dumps(result.to_safe_dict(), sort_keys=True) + + assert result.status == "invalid" + assert "workbook_package_signature_invalid" in {issue.code for issue in result.issues} + assert "fx_a8b553a4.xlsx" not in safe_payload + + +def test_private_manifest_rejects_paths_outside_private_folder(tmp_path): + outside = tmp_path.parent / "fx_a8b553a4.xlsx" + _write_minimal_workbook_package(outside) + payload = _manifest_payload( + workbook=outside, + workbook_path="../fx_a8b553a4.xlsx", + ) + (tmp_path / "manifest.json").write_text(json.dumps(payload), encoding="utf-8") + + result = load_private_golden_fixture_manifest(tmp_path) + + assert result.status == "invalid" + assert "workbook_path_not_private_relative" in {issue.code for issue in result.issues} + + +def test_private_manifest_binds_workbook_digest_and_hashed_excel_provenance(tmp_path): + workbook = tmp_path / "workbooks" / "fx_a8b553a4.xlsx" + workbook.parent.mkdir() + _write_minimal_workbook_package(workbook) + payload = _manifest_payload(workbook=workbook) + payload["fixtures"][0]["workbookSha256"] = "sha256:" + "9" * 64 + payload["fixtures"][0]["provenance"]["excelBuildHash"] = "Excel 16.0" + (tmp_path / "manifest.json").write_text(json.dumps(payload), encoding="utf-8") + + result = load_private_golden_fixture_manifest(tmp_path) + + assert result.status == "invalid" + assert {issue.code for issue in result.issues} >= { + "workbook_sha256_mismatch", + "workbook_provenance_hash_invalid", + } + + +def test_private_manifest_rejects_duplicate_ids_and_case_colliding_paths(tmp_path): + workbook = tmp_path / "workbooks" / "fx_a8b553a4.xlsx" + workbook.parent.mkdir() + _write_minimal_workbook_package(workbook) + duplicate = tmp_path / "workbooks" / "FX_A8B553A4.xlsx" + duplicate.write_bytes(workbook.read_bytes()) + payload = _manifest_payload(workbook=workbook) + second = json.loads(json.dumps(payload["fixtures"][0])) + second["workbookPath"] = "workbooks/FX_A8B553A4.xlsx" + second["workbookSha256"] = _workbook_hash(duplicate) + payload["fixtures"].append(second) + (tmp_path / "manifest.json").write_text(json.dumps(payload), encoding="utf-8") + + result = load_private_golden_fixture_manifest(tmp_path) + + assert result.status == "invalid" + assert {issue.code for issue in result.issues} >= { + "fixture_id_duplicate", + "workbook_path_duplicate", + } + + +def test_private_manifest_rejects_symlink_and_unsafe_zip_members(tmp_path): + outside = tmp_path.parent / "private-source.xlsx" + _write_minimal_workbook_package(outside) + workbook = tmp_path / "workbooks" / "fx_a8b553a4.xlsx" + workbook.parent.mkdir() + try: + workbook.symlink_to(outside) + except OSError: + return + (tmp_path / "manifest.json").write_text( + json.dumps(_manifest_payload(workbook=outside)), encoding="utf-8" + ) + result = load_private_golden_fixture_manifest(tmp_path) + assert "workbook_path_symlink_forbidden" in {issue.code for issue in result.issues} + + workbook.unlink() + with zipfile.ZipFile(workbook, "w") as workbook_zip: + workbook_zip.writestr("[Content_Types].xml", "") + workbook_zip.writestr("xl/workbook.xml", "") + workbook_zip.writestr("../escaped.xml", "secret") + (tmp_path / "manifest.json").write_text( + json.dumps(_manifest_payload(workbook=workbook)), encoding="utf-8" + ) + result = load_private_golden_fixture_manifest(tmp_path) + assert "workbook_package_member_path_invalid" in { + issue.code for issue in result.issues + } + + +def test_private_manifest_rejects_case_colliding_zip_members(tmp_path): + workbook = tmp_path / "workbooks" / "fx_a8b553a4.xlsx" + workbook.parent.mkdir() + with zipfile.ZipFile(workbook, "w") as workbook_zip: + workbook_zip.writestr("[Content_Types].xml", "") + workbook_zip.writestr("[content_types].xml", "") + workbook_zip.writestr("xl/workbook.xml", "") + (tmp_path / "manifest.json").write_text( + json.dumps(_manifest_payload(workbook=workbook)), encoding="utf-8" + ) + + result = load_private_golden_fixture_manifest(tmp_path) + + assert "workbook_package_member_duplicate" in {issue.code for issue in result.issues} + + +def test_manifest_example_documents_current_schema(): + example_path = ( + __import__("pathlib").Path(__file__).resolve().parents[2] + / "benchmarks" + / "fixtures" + / "private" + / "manifest.example.json" + ) + payload = json.loads(example_path.read_text(encoding="utf-8")) + + assert payload["schemaVersion"] == SCHEMA_VERSION + assert set(payload["fixtures"][0]) >= { + "fixtureIdHash", + "category", + "workbookPath", + "workbookSha256", + "provenance", + "expectedTruthTargets", + "expectedFormulaFacts", + "expectedSimulationGates", + "rollbackExpectations", + "repairOutcomes", + "auditExpectations", + } + + +def test_private_corpus_worker_preflight_is_bounded_safe_and_not_native_evidence(tmp_path): + workbook = tmp_path / "workbooks" / "fx_a8b553a4.xlsx" + workbook.parent.mkdir() + _write_minimal_workbook_package(workbook) + (tmp_path / "manifest.json").write_text( + json.dumps(_manifest_payload(workbook=workbook)), encoding="utf-8" + ) + calls = [] + + def summarize(content: bytes, filename: str) -> dict: + calls.append((content, filename)) + return { + "engine": "sheetjs", + "sheets": [{"name": "Private Customer Model"}], + "formula_count": 1, + "has_macros": False, + "formulas": ["='Do Not Leak'!A1"], + } + + report = run_private_corpus_preflight(tmp_path, summarize=summarize) + encoded = json.dumps(report, sort_keys=True) + + assert report["status"] == "passed" + assert report["workbookCount"] == 1 + assert report["passedCount"] == 1 + assert report["failedCount"] == 0 + assert report["releaseScaleReady"] is False + assert report["nativeExcelExecuted"] is False + assert report["releaseEvidenceEligible"] is False + assert report["results"] == [ + { + "fixtureIdHash": "fx_a8b553a4", + "status": "passed", + "issueCodes": [], + "processor": "bounded_workbook_worker", + "sheetCount": 1, + "formulaCount": 1, + "macroDetected": False, + } + ] + assert calls and calls[0][1] == "private-fixture.xlsx" + assert calls[0][0] == workbook.read_bytes() + assert str(tmp_path) not in encoded + assert "Private Customer Model" not in encoded + assert "Do Not Leak" not in encoded + + +def test_private_corpus_worker_preflight_reduces_worker_errors_to_safe_codes(tmp_path): + workbook = tmp_path / "workbooks" / "fx_a8b553a4.xlsx" + workbook.parent.mkdir() + _write_minimal_workbook_package(workbook) + (tmp_path / "manifest.json").write_text( + json.dumps(_manifest_payload(workbook=workbook)), encoding="utf-8" + ) + + def reject(_content: bytes, _filename: str) -> dict: + raise RuntimeError("Customer Name and /private/source/path must not escape") + + report = run_private_corpus_preflight(tmp_path, summarize=reject) + encoded = json.dumps(report, sort_keys=True) + + assert report["status"] == "failed" + assert report["issueCodes"] == ["worker_rejected"] + assert report["results"][0]["issueCodes"] == ["worker_rejected"] + assert "Customer Name" not in encoded + assert "/private/source/path" not in encoded + + +def test_private_corpus_worker_preflight_blocks_before_worker_on_invalid_manifest(tmp_path): + calls = [] + + def summarize(_content: bytes, _filename: str) -> dict: + calls.append(True) + return {} + + report = run_private_corpus_preflight(tmp_path, summarize=summarize) + + assert report["status"] == "blocked" + assert report["manifestStatus"] == "missing" + assert report["workbookCount"] == 0 + assert report["nativeExcelExecuted"] is False + assert report["releaseEvidenceEligible"] is False + assert calls == [] + + +def test_golden_case_evaluation_reports_private_fixture_status_without_details(monkeypatch): + class Runner: + def run_all(self): + return [ + SimpleNamespace( + status="passed", + score=100, + fixture_id="public_case", + failures=[], + model_dump=lambda **_kwargs: {"fixtureId": "public_case", "status": "passed", "score": 100}, + ) + ] + + private_validation = SimpleNamespace( + to_safe_dict=lambda: { + "status": "invalid", + "schemaVersion": SCHEMA_VERSION, + "fixtureCount": 0, + "issues": [ + { + "code": "workbook_path_missing", + "location": "fixtures[0].workbookPath", + "message": "workbookPath must point to an existing private workbook.", + } + ], + "fixtures": [], + } + ) + monkeypatch.setattr(v7_golden_cases, "get_workbook_fixture_benchmark_runner", lambda: Runner()) + monkeypatch.setattr(v7_golden_cases, "validate_private_golden_fixture_manifest", lambda: private_validation) + + evaluation = v7_golden_cases.evaluate_v7_golden_cases() + private_status = evaluation.metadata["privateFixtureValidation"] + safe_payload = json.dumps(evaluation.model_dump(mode="json", by_alias=True), sort_keys=True) + + assert private_status == { + "status": "invalid", + "schemaVersion": SCHEMA_VERSION, + "fixtureCount": 0, + "issueCodes": ["workbook_path_missing"], + } + assert "fixtures[0].workbookPath" not in safe_payload + + +def test_golden_case_release_gate_requires_ready_private_manifest_without_leaking_details(monkeypatch): + class Runner: + def run_all(self): + return [ + SimpleNamespace( + status="passed", + score=100, + fixture_id=f"public_case_{index}", + failures=[], + model_dump=lambda **_kwargs: {"fixtureId": "public_case", "status": "passed", "score": 100}, + ) + for index in range(30) + ] + + private_validation = SimpleNamespace( + to_safe_dict=lambda: { + "status": "missing", + "schemaVersion": None, + "fixtureCount": 0, + "issues": [], + "fixtures": [], + } + ) + monkeypatch.setattr(v7_golden_cases, "get_workbook_fixture_benchmark_runner", lambda: Runner()) + monkeypatch.setattr(v7_golden_cases, "validate_private_golden_fixture_manifest", lambda: private_validation) + + evaluation = v7_golden_cases.evaluate_v7_golden_cases(require_private_fixtures=True) + safe_payload = json.dumps(evaluation.model_dump(mode="json", by_alias=True), sort_keys=True) + + assert evaluation.status == "failed" + assert evaluation.metadata["requirePrivateFixtures"] is True + assert evaluation.metadata["privateFixtureValidation"] == { + "status": "missing", + "schemaVersion": None, + "fixtureCount": 0, + "issueCodes": [], + } + assert "private golden fixture manifest is required" in " ".join(evaluation.failures) + assert "workbookPath" not in safe_payload + + +def _manifest_payload( + *, + workbook: Path, + workbook_path: str = "workbooks/fx_a8b553a4.xlsx", +) -> dict: + return { + "schemaVersion": SCHEMA_VERSION, + "fixtures": [ + { + "fixtureIdHash": "fx_a8b553a4", + "category": "customer_like_model", + "workbookPath": workbook_path, + "workbookSha256": _workbook_hash(workbook), + "provenance": _provenance(), + "expectedTruthTargets": ["primary_driver_identified"], + "expectedFormulaFacts": [{"address": "B20", "formulaHash": "sha256:" + "0" * 64}], + "expectedSimulationGates": ["checkpoint_required"], + "rollbackExpectations": {"required": True, "dirtyRanges": ["Model!B20"]}, + "repairOutcomes": [{"kind": "formula_repair", "expectedStatus": "repair_plan_expected"}], + "auditExpectations": {"requiredEvidence": ["truth_model", "formula_graph"]}, + } + ], + } + + +def _private_corpus_fixture( + *, + fixture_id_hash: str, + category: str, + workbook_path: str, + workbook: Path, + truth_targets: list[str], + formula_hash: str, + dependency_hashes: list[str], + audit_evidence: list[str], +) -> dict: + return { + "fixtureIdHash": fixture_id_hash, + "category": category, + "workbookPath": workbook_path, + "workbookSha256": _workbook_hash(workbook), + "provenance": _provenance(), + "expectedTruthTargets": truth_targets, + "expectedFormulaFacts": [ + { + "sheet": "Model", + "address": "B20", + "formulaHash": formula_hash, + "dependencyHashes": dependency_hashes, + } + ], + "expectedSimulationGates": ["checkpoint_required", "rollback_snapshot_required"], + "rollbackExpectations": {"required": True, "dirtyRanges": ["Model!B20"], "snapshotRequired": True}, + "repairOutcomes": [{"kind": "formula_repair", "expectedStatus": "repair_plan_expected"}], + "auditExpectations": {"requiredEvidence": audit_evidence, "exportRequired": True}, + } + + +def _workbook_hash(path) -> str: + return "sha256:" + hashlib.sha256(path.read_bytes()).hexdigest() + + +def _provenance() -> dict[str, str]: + return { + "kind": "excel_authored", + "excelBuildHash": "sha256:" + "e" * 64, + "sourceReceiptHash": "sha256:" + "f" * 64, + } + + +def _write_minimal_workbook_package(path): + with zipfile.ZipFile(path, "w") as workbook_zip: + workbook_zip.writestr( + "[Content_Types].xml", + """ + + + + +""", + ) + workbook_zip.writestr("xl/workbook.xml", "") diff --git a/backend/tests/contract/test_production_replay_execution_receipt.py b/backend/tests/contract/test_production_replay_execution_receipt.py new file mode 100644 index 0000000..f816104 --- /dev/null +++ b/backend/tests/contract/test_production_replay_execution_receipt.py @@ -0,0 +1,296 @@ +from __future__ import annotations + +import copy +import hashlib +import json +from pathlib import Path +from typing import Any + +import pytest +from cryptography.hazmat.primitives import serialization +from cryptography.hazmat.primitives.asymmetric.ed25519 import Ed25519PrivateKey + +from backend.benchmarks.local_replay_corpus import sanitize_corpus_record +from scripts.validate_production_replay_execution_receipt import ( + ProductionReplayExecutionError, + validate_production_replay_execution_receipt, +) + + +RELEASE_SHA = "a" * 40 +KEY_ID = "production-replay-evidence-2026" + + +def test_signed_production_replay_receipt_binds_every_sanitized_record(tmp_path: Path) -> None: + fixture = _fixture(tmp_path) + + report = validate_production_replay_execution_receipt( + receipt_path=fixture["receipt_path"], + signature_path=fixture["signature_path"], + trusted_public_key_path=fixture["public_key_path"], + trusted_signing_key_id=KEY_ID, + expected_release_sha=RELEASE_SHA, + corpus_path=fixture["corpus_path"], + ) + + assert report["ready"] is True + assert report["recordCount"] == 25 + assert report["failedCount"] == 0 + assert report["signatureVerified"] is True + assert report["productionEvidence"] is True + assert report["modelProvidersContacted"] is False + assert str(tmp_path) not in json.dumps(report, sort_keys=True) + + +def test_production_replay_receipt_rejects_signature_tampering(tmp_path: Path) -> None: + fixture = _fixture(tmp_path) + raw = fixture["receipt_path"].read_bytes() + fixture["receipt_path"].write_bytes(raw.replace(b'"status":"passed"', b'"status":"failed"', 1)) + + with pytest.raises(ProductionReplayExecutionError, match="signature verification"): + _validate(fixture) + + +@pytest.mark.parametrize( + ("mutation", "message"), + [ + (lambda value: value["corpus"]["records"].pop(), "complete replay corpus"), + ( + lambda value: value["corpus"]["records"][0].__setitem__( + "deterministicReplayCount", 1 + ), + "integer from 2", + ), + ( + lambda value: value["evaluation"].__setitem__("modelProvidersContacted", True), + "private, passing production replay", + ), + ( + lambda value: value["evaluation"].__setitem__( + "evidenceClass", "deterministic_plumbing_seed" + ), + "production evidence class", + ), + ( + lambda value: value["corpus"]["records"][1].__setitem__( + "artifactSha256", value["corpus"]["records"][0]["artifactSha256"] + ), + "unique", + ), + ], +) +def test_production_replay_receipt_rejects_incomplete_or_false_execution_claims( + tmp_path: Path, + mutation, + message: str, +) -> None: + fixture = _fixture(tmp_path) + mutation(fixture["receipt"]) + _write_signed_receipt(fixture) + + with pytest.raises(ProductionReplayExecutionError, match=message): + _validate(fixture) + + +def test_production_replay_receipt_rejects_raw_content_and_duplicate_padding( + tmp_path: Path, +) -> None: + raw_fixture = _fixture(tmp_path / "raw") + corpus = raw_fixture["corpus"] + corpus["artifacts"][0]["payload"]["formula"] = "=PrivateRevenue!B4" + _write_json(raw_fixture["corpus_path"], corpus) + + with pytest.raises(ProductionReplayExecutionError, match="privacy validation"): + _validate(raw_fixture) + + duplicate_fixture = _fixture(tmp_path / "duplicate") + duplicate_corpus = duplicate_fixture["corpus"] + duplicate_corpus["artifacts"][1] = copy.deepcopy(duplicate_corpus["artifacts"][0]) + _write_json(duplicate_fixture["corpus_path"], duplicate_corpus) + + with pytest.raises(ProductionReplayExecutionError, match="duplicate sanitized"): + _validate(duplicate_fixture) + + +def test_production_replay_receipt_requires_out_of_artifact_trust_key(tmp_path: Path) -> None: + fixture = _fixture(tmp_path) + embedded_key = fixture["receipt_path"].parent / "embedded-public-key.pem" + embedded_key.write_bytes(fixture["public_key_path"].read_bytes()) + + with pytest.raises(ProductionReplayExecutionError, match="outside the repository"): + validate_production_replay_execution_receipt( + receipt_path=fixture["receipt_path"], + signature_path=fixture["signature_path"], + trusted_public_key_path=embedded_key, + trusted_signing_key_id=KEY_ID, + expected_release_sha=RELEASE_SHA, + corpus_path=fixture["corpus_path"], + ) + + +def _validate(fixture: dict[str, Any]) -> dict[str, Any]: + return validate_production_replay_execution_receipt( + receipt_path=fixture["receipt_path"], + signature_path=fixture["signature_path"], + trusted_public_key_path=fixture["public_key_path"], + trusted_signing_key_id=KEY_ID, + expected_release_sha=RELEASE_SHA, + corpus_path=fixture["corpus_path"], + ) + + +def _fixture(root: Path) -> dict[str, Any]: + artifact_root = root / "artifacts" + corpus_root = root / "corpus" + trust_root = root / "trust" + artifact_root.mkdir(parents=True) + corpus_root.mkdir(parents=True) + trust_root.mkdir(parents=True) + + records = [_record(index) for index in range(1, 26)] + corpus = { + "schemaVersion": "2026-06-local-replay-corpus-v1", + "generatedAt": "2026-08-10T12:00:00Z", + "anonymizationPolicyVersion": "anon-policy-v2", + "sourceCohortHash": _sha("source-cohort"), + "exporterVersion": "replay-exporter-v2", + "recordCount": len(records), + "productionEvidence": True, + "artifacts": records, + } + corpus_path = corpus_root / "production-replay.json" + _write_json(corpus_path, corpus) + raw_corpus = corpus_path.read_bytes() + sanitized = [ + sanitize_corpus_record(record, index=index) + for index, record in enumerate(records, start=1) + ] + + receipt = { + "schemaVersion": "2026-08-production-replay-execution-v1", + "releaseSha": RELEASE_SHA, + "signingKeyId": KEY_ID, + "runner": { + "runtimeKind": "excelagent-v7-local-replay-evaluator", + "implementationHash": _sha("implementation"), + "executorHash": _sha("executor"), + "runnerImageHash": _sha("runner-image"), + "toolchainManifestHash": _sha("toolchain"), + }, + "corpus": { + "corpusSha256": "sha256:" + hashlib.sha256(raw_corpus).hexdigest(), + "sourceCohortHash": corpus["sourceCohortHash"], + "recordCount": len(records), + "generatedAt": corpus["generatedAt"], + "anonymizationPolicyVersion": corpus["anonymizationPolicyVersion"], + "exporterVersion": corpus["exporterVersion"], + "records": [ + { + "index": index, + "artifactType": item["artifactType"], + "payloadHash": "sha256:" + item["payload"]["payloadHash"], + "status": "passed", + "deterministicReplayCount": 2, + "artifactSha256": _sha(f"artifact-{index}"), + } + for index, item in enumerate(sanitized, start=1) + ], + }, + "evaluation": { + "status": "passed", + "productionEvidence": True, + "evidenceClass": "anonymized_prod_uat_v1", + "captureCount": len(records), + "failureCount": 0, + "privacyScanStatus": "passed", + "driftSeverity": "none", + "deterministicDriftClass": "stable", + "baselineDigest": _sha("baseline"), + "artifactSha256": _sha("evaluation-artifact"), + "modelProvidersContacted": False, + }, + } + + private_key = Ed25519PrivateKey.generate() + public_key_path = trust_root / "public-key.pem" + public_key_path.write_bytes( + private_key.public_key().public_bytes( + serialization.Encoding.PEM, + serialization.PublicFormat.SubjectPublicKeyInfo, + ) + ) + fixture = { + "corpus": corpus, + "corpus_path": corpus_path, + "receipt": receipt, + "receipt_path": artifact_root / "production-replay-receipt.json", + "signature_path": artifact_root / "production-replay-receipt.ed25519", + "public_key_path": public_key_path, + "private_key": private_key, + } + _write_signed_receipt(fixture) + return fixture + + +def _write_signed_receipt(fixture: dict[str, Any]) -> None: + raw = json.dumps( + fixture["receipt"], + sort_keys=True, + separators=(",", ":"), + allow_nan=False, + ).encode("utf-8") + fixture["receipt_path"].write_bytes(raw) + fixture["signature_path"].write_bytes(fixture["private_key"].sign(raw)) + + +def _write_json(path: Path, value: dict[str, Any]) -> None: + path.write_text( + json.dumps(value, sort_keys=True, separators=(",", ":"), allow_nan=False), + encoding="utf-8", + ) + + +def _record(index: int) -> dict[str, Any]: + return { + "artifactType": "v7_plan", + "payload": { + "schemaVersion": "2026-05-local-replay-v1", + "artifactKind": f"aggregate_prod_plan_{index:02d}", + "operationTypes": ["write_cell"], + "operationCount": 1, + "accuracyScore": 0.99, + "confidenceGapCount": 0, + "checkpointReasonCount": 0, + "formulaFailureCount": 0, + "truthSchema": "truth-v5", + "formulaSchema": "formula-v5", + "simulationSchema": "simulation-v6", + "v7GateStatus": "passed", + "dirtyRangeCount": 1, + "rollbackAvailable": True, + "fixtureIdHash": f"fx_{index:08x}", + "category": "writeback", + "sourceProject": "prod-batch-a", + "sheetCount": 4, + "formulaCount": 128, + "objectCounts": {"tables": 2, "charts": 1, "pivots": 0}, + "warningCategoryCounts": {}, + "semanticMatchCounts": {"matched": 12, "missing": 0}, + "replaySignature": { + "schemaVersion": "2026-05-local-replay-v1", + "operationTypes": ["write_cell"], + "operationCount": 1, + "truthSchema": "truth-v5", + "formulaSchema": "formula-v5", + "simulationSchema": "simulation-v6", + "v7GateStatus": "passed", + "formulaFailureCount": 0, + "confidenceGapCount": 0, + "checkpointReasonCount": 0, + }, + }, + } + + +def _sha(value: str) -> str: + return "sha256:" + hashlib.sha256(value.encode("utf-8")).hexdigest() diff --git a/backend/tests/contract/test_python_dependency_policy.py b/backend/tests/contract/test_python_dependency_policy.py new file mode 100644 index 0000000..7bcd769 --- /dev/null +++ b/backend/tests/contract/test_python_dependency_policy.py @@ -0,0 +1,78 @@ +import subprocess + +from scripts import check_python_dependency_conflicts + + +def test_python_dependency_policy_ignores_non_project_conflicts(monkeypatch, capsys): + monkeypatch.setattr(check_python_dependency_conflicts, "_load_backend_requirement_roots", lambda: {"fastapi": set()}) + monkeypatch.setattr(check_python_dependency_conflicts, "_dependency_closure", lambda roots: {"fastapi", "starlette"}) + monkeypatch.setattr( + check_python_dependency_conflicts, + "_run_pip_check", + lambda: subprocess.CompletedProcess( + ["pip", "check"], + 1, + stdout="streamlit 1.57.0 has requirement starlette>=0.40.0, but you have starlette 0.38.6.\n", + stderr="", + ), + ) + + assert check_python_dependency_conflicts.main() == 0 + + captured = capsys.readouterr() + assert "Ignoring non-project pip check conflict: streamlit" in captured.out + assert "Python dependency policy passed" in captured.out + + +def test_python_dependency_policy_blocks_project_conflicts(monkeypatch, capsys): + monkeypatch.setattr(check_python_dependency_conflicts, "_load_backend_requirement_roots", lambda: {"fastapi": set()}) + monkeypatch.setattr(check_python_dependency_conflicts, "_dependency_closure", lambda roots: {"fastapi", "starlette"}) + monkeypatch.setattr( + check_python_dependency_conflicts, + "_run_pip_check", + lambda: subprocess.CompletedProcess( + ["pip", "check"], + 1, + stdout="fastapi 0.115.0 has requirement starlette>=0.40.0, but you have starlette 0.38.6.\n", + stderr="", + ), + ) + + assert check_python_dependency_conflicts.main() == 1 + + captured = capsys.readouterr() + assert "Project Python dependency conflict: fastapi" in captured.err + + +def test_python_dependency_policy_passes_clean_pip_check(monkeypatch, capsys): + monkeypatch.setattr(check_python_dependency_conflicts, "_load_backend_requirement_roots", lambda: {"fastapi": set()}) + monkeypatch.setattr(check_python_dependency_conflicts, "_dependency_closure", lambda roots: {"fastapi", "starlette"}) + monkeypatch.setattr( + check_python_dependency_conflicts, + "_run_pip_check", + lambda: subprocess.CompletedProcess(["pip", "check"], 0, stdout="No broken requirements found.\n", stderr=""), + ) + + assert check_python_dependency_conflicts.main() == 0 + + captured = capsys.readouterr() + assert "No broken requirements found." in captured.out + assert "Python dependency policy passed" in captured.out + + +def test_python_dependency_policy_fails_when_pip_check_cannot_run(monkeypatch, capsys): + monkeypatch.setattr(check_python_dependency_conflicts, "_load_backend_requirement_roots", lambda: {"fastapi": set()}) + monkeypatch.setattr(check_python_dependency_conflicts, "_dependency_closure", lambda roots: {"fastapi", "starlette"}) + monkeypatch.setattr( + check_python_dependency_conflicts, + "_run_pip_check", + lambda: subprocess.CompletedProcess( + ["python", "-m", "pip", "check"], + 1, + stdout="", + stderr="python: No module named pip\n", + ), + ) + + assert check_python_dependency_conflicts.main() == 1 + assert "pip check infrastructure error" in capsys.readouterr().err diff --git a/backend/tests/contract/test_python_launcher_contract.py b/backend/tests/contract/test_python_launcher_contract.py new file mode 100644 index 0000000..2c57510 --- /dev/null +++ b/backend/tests/contract/test_python_launcher_contract.py @@ -0,0 +1,55 @@ +from __future__ import annotations + +import os +import subprocess +import sys +from pathlib import Path + + +ROOT = Path(__file__).resolve().parents[3] +LAUNCHER = ROOT / "scripts" / "run-python.mjs" + + +def test_python_launcher_honors_explicit_interpreter_and_arguments(tmp_path: Path) -> None: + child = tmp_path / "child.py" + child.write_text( + "import json, sys\nprint(json.dumps(sys.argv[1:]))\n", + encoding="utf-8", + ) + env = {**os.environ, "PYTHON": sys.executable} + + completed = subprocess.run( + ["node", str(LAUNCHER), str(child), "alpha", "two words"], + cwd=ROOT, + env=env, + text=True, + stdout=subprocess.PIPE, + stderr=subprocess.PIPE, + check=False, + ) + + assert completed.returncode == 0, completed.stderr + assert completed.stdout.strip() == '["alpha", "two words"]' + + +def test_python_launcher_propagates_child_exit_status(tmp_path: Path) -> None: + child = tmp_path / "exit.py" + child.write_text("raise SystemExit(7)\n", encoding="utf-8") + + completed = subprocess.run( + ["node", str(LAUNCHER), str(child)], + cwd=ROOT, + env={**os.environ, "PYTHON": sys.executable}, + check=False, + ) + + assert completed.returncode == 7 + + +def test_python_launcher_prefers_the_repository_virtualenv_before_system_python() -> None: + source = LAUNCHER.read_text(encoding="utf-8") + + assert source.index('resolve(repoRoot, ".venv", "bin", "python3")') < source.index( + '{ command: "python3", prefix: [] }' + ) + assert 'resolve(repoRoot, ".venv", "Scripts", "python.exe")' in source diff --git a/backend/tests/contract/test_release_evidence_index.py b/backend/tests/contract/test_release_evidence_index.py new file mode 100644 index 0000000..f68e9a2 --- /dev/null +++ b/backend/tests/contract/test_release_evidence_index.py @@ -0,0 +1,784 @@ +from __future__ import annotations + +import hashlib +import json +import os +from pathlib import Path +from typing import Any + +import pytest +from cryptography.hazmat.primitives import serialization +from cryptography.hazmat.primitives.asymmetric.ed25519 import Ed25519PrivateKey +from cryptography.hazmat.primitives.asymmetric.rsa import generate_private_key + +from scripts import validate_release_evidence_index as evidence + +RELEASE_SHA = "a" * 40 +SIGNING_KEY_ID = "closure-release-evidence-2026-08" +SIGNING_PRIVATE_KEY = Ed25519PrivateKey.from_private_bytes(hashlib.sha256(b"release-evidence-test-key").digest()) +TRUSTED_PUBLIC_KEY_FILENAME = "trusted-release-evidence-key.pem" + + +def _details(item_id: str) -> dict[str, Any]: + details: dict[str, Any] = {"successful": True} + if item_id == "signed_production_manifest": + details["signatureVerified"] = True + if item_id == "production_manifest_windows": + details.update(host="windows", signedManifestVerified=True) + if item_id == "production_manifest_macos": + details.update(host="macos", signedManifestVerified=True) + if item_id == "production_manifest_excel_web": + details.update(host="excel_web", signedManifestVerified=True) + if item_id == "private_corpus_execution": + details.update( + excelAuthored=True, + nativeExcelExecuted=True, + executionReceiptVerified=True, + hostPlatform="windows", + excelBuildHash="sha256:" + "1" * 64, + corpusManifestSha256="sha256:" + "2" * 64, + executionReceiptSha256="sha256:" + "3" * 64, + workbookCount=250, + failedCount=0, + ) + if item_id == "private_corpus_inventory": + details.update(workbookCount=250, uninventoriedCount=0) + if item_id == "production_replay": + details.update( + productionEvidence=True, + executionReceiptVerified=True, + privacyScanStatus="passed", + modelProvidersContacted=False, + failedCount=0, + recordCount=25, + deterministicReplayCount=2, + corpusSha256="sha256:" + "4" * 64, + sourceCohortHash="sha256:" + "5" * 64, + executionReceiptSha256="sha256:" + "6" * 64, + baselineDigest="sha256:" + "7" * 64, + evaluationArtifactSha256="sha256:" + "8" * 64, + ) + if item_id == "parser_corpus": + details.update(fixtureCount=250, materialMismatchCount=0) + if item_id.startswith("postgres_"): + details["liveDatabase"] = True + if item_id == "postgres_backup_restore": + details["restoreVerified"] = True + if item_id == "hyperformula_posture": + details.update( + posture="disabled", + enabled=False, + distributed=False, + runtimePresent=False, + gplSentinelAbsent=True, + ) + if item_id == "sbom": + details.update(format="cyclonedx", verified=True) + if item_id in {"trust_report", "dependency_review"}: + details["verified"] = True + if item_id == "container_scan": + details.update(criticalCount=0, highCount=0) + if item_id in {"container_signature", "provenance_verification"}: + details["signatureVerified"] = True + if item_id == "provenance_verification": + details["subjectReleaseSha"] = RELEASE_SHA + if item_id == "independent_pentest": + details.update( + independent=True, + openCriticalCount=0, + openHighCount=0, + undispositionedMediumCount=0, + ) + if item_id == "release_signoff": + details.update( + subjectReleaseSha=RELEASE_SHA, + signatures=[ + { + "role": role, + "signerId": f"{role}-owner", + "signatureReference": f"sigstore://release/{role}", + "signatureVerified": True, + } + for role in ("release", "security", "workbook", "client", "legal") + ], + ) + return details + + +def _write_artifact(root: Path) -> tuple[Path, dict[str, Any]]: + items: list[dict[str, Any]] = [] + for item_id in evidence.REQUIRED_EVIDENCE: + relative_path = f"data/release-trust/{item_id}.json" + artifact_path = root / relative_path + artifact_path.parent.mkdir(parents=True, exist_ok=True) + artifact_bytes = json.dumps({"evidence": item_id, "releaseSha": RELEASE_SHA}).encode() + artifact_path.write_bytes(artifact_bytes) + items.append( + { + "id": item_id, + "releaseSha": RELEASE_SHA, + "status": "passed", + "path": relative_path, + "sha256": hashlib.sha256(artifact_bytes).hexdigest(), + "size": len(artifact_bytes), + "details": _details(item_id), + } + ) + payload = { + "schemaVersion": evidence.INDEX_SCHEMA_VERSION, + "releaseSha": RELEASE_SHA, + "signingKeyId": SIGNING_KEY_ID, + "generatedAt": "2026-08-09T23:59:00Z", + "producer": "controlled-release-workflow", + "items": items, + } + index_path = root / evidence.INDEX_FILENAME + _rewrite_index(index_path, payload) + trusted_public_key_path = root.parent / TRUSTED_PUBLIC_KEY_FILENAME + trusted_public_key_path.write_bytes( + SIGNING_PRIVATE_KEY.public_key().public_bytes( + encoding=serialization.Encoding.PEM, + format=serialization.PublicFormat.SubjectPublicKeyInfo, + ) + ) + return index_path, payload + + +def _rewrite_index( + index_path: Path, + payload: dict[str, Any], + *, + signing_key: Ed25519PrivateKey = SIGNING_PRIVATE_KEY, +) -> None: + raw_index = json.dumps(payload, indent=2, sort_keys=True).encode() + index_path.write_bytes(raw_index) + (index_path.parent / evidence.SIGNATURE_FILENAME).write_bytes(signing_key.sign(raw_index)) + + +def _validation_kwargs(artifact_root: Path) -> dict[str, Any]: + return { + "signature_path": artifact_root / evidence.SIGNATURE_FILENAME, + "trusted_public_key_path": artifact_root.parent / TRUSTED_PUBLIC_KEY_FILENAME, + "trusted_signing_key_id": SIGNING_KEY_ID, + } + + +def test_exact_sha_evidence_index_validates_and_restores_only_indexed_files(tmp_path: Path) -> None: + artifact_root = tmp_path / "artifact" + restore_root = tmp_path / "checkout" + artifact_root.mkdir() + restore_root.mkdir() + index_path, payload = _write_artifact(artifact_root) + + report = evidence.validate_release_evidence_index( + index_path=index_path, + artifact_root=artifact_root, + expected_release_sha=RELEASE_SHA, + restore_root=restore_root, + **_validation_kwargs(artifact_root), + ) + + assert report["status"] == "ready" + assert report["releaseSha"] == RELEASE_SHA + assert report["itemCount"] == len(evidence.REQUIRED_EVIDENCE) + assert report["restoredPathCount"] == len(evidence.REQUIRED_EVIDENCE) + for item in payload["items"]: + assert (restore_root / item["path"]).read_bytes() == (artifact_root / item["path"]).read_bytes() + assert not (restore_root / evidence.INDEX_FILENAME).exists() + + +def test_evidence_index_accepts_workflow_relative_paths(tmp_path: Path, monkeypatch) -> None: + artifact_root = tmp_path / ".release-evidence-input" + checkout = tmp_path / "checkout" + artifact_root.mkdir() + checkout.mkdir() + _write_artifact(artifact_root) + monkeypatch.chdir(tmp_path) + + report = evidence.validate_release_evidence_index( + index_path=Path(".release-evidence-input/release-evidence-index.json"), + artifact_root=Path(".release-evidence-input"), + expected_release_sha=RELEASE_SHA, + restore_root=checkout, + **_validation_kwargs(artifact_root), + ) + + assert report["ready"] is True + + +def test_evidence_index_rejects_missing_required_evidence(tmp_path: Path) -> None: + artifact_root = tmp_path / "artifact" + artifact_root.mkdir() + index_path, payload = _write_artifact(artifact_root) + removed = payload["items"].pop() + (artifact_root / removed["path"]).unlink() + _rewrite_index(index_path, payload) + + with pytest.raises(evidence.EvidenceIndexError, match="missing required successful evidence"): + evidence.validate_release_evidence_index( + index_path=index_path, + artifact_root=artifact_root, + expected_release_sha=RELEASE_SHA, + **_validation_kwargs(artifact_root), + ) + + +def test_evidence_index_rejects_stale_item_sha_even_when_index_sha_matches(tmp_path: Path) -> None: + artifact_root = tmp_path / "artifact" + artifact_root.mkdir() + index_path, payload = _write_artifact(artifact_root) + payload["items"][0]["releaseSha"] = "b" * 40 + _rewrite_index(index_path, payload) + + with pytest.raises(evidence.EvidenceIndexError, match=r"not a{40}"): + evidence.validate_release_evidence_index( + index_path=index_path, + artifact_root=artifact_root, + expected_release_sha=RELEASE_SHA, + **_validation_kwargs(artifact_root), + ) + + +@pytest.mark.parametrize("mutation", ["digest", "size"]) +def test_evidence_index_rejects_tampered_digest_or_size_without_partial_restore( + tmp_path: Path, + mutation: str, +) -> None: + artifact_root = tmp_path / "artifact" + restore_root = tmp_path / "checkout" + artifact_root.mkdir() + restore_root.mkdir() + index_path, payload = _write_artifact(artifact_root) + target = payload["items"][-1] + if mutation == "digest": + (artifact_root / target["path"]).write_bytes(b"x" * target["size"]) + expected_error = "SHA-256 mismatch" + else: + target["size"] += 1 + _rewrite_index(index_path, payload) + expected_error = "size mismatch" + + with pytest.raises(evidence.EvidenceIndexError, match=expected_error): + evidence.validate_release_evidence_index( + index_path=index_path, + artifact_root=artifact_root, + expected_release_sha=RELEASE_SHA, + restore_root=restore_root, + **_validation_kwargs(artifact_root), + ) + + assert not any(path.is_file() for path in restore_root.rglob("*")) + + +def test_evidence_index_rejects_traversal_and_unindexed_files(tmp_path: Path) -> None: + artifact_root = tmp_path / "artifact" + artifact_root.mkdir() + index_path, payload = _write_artifact(artifact_root) + payload["items"][0]["path"] = "../outside.json" + _rewrite_index(index_path, payload) + + with pytest.raises(evidence.EvidenceIndexError, match=r"not canonical|Unsafe evidence path"): + evidence.validate_release_evidence_index( + index_path=index_path, + artifact_root=artifact_root, + expected_release_sha=RELEASE_SHA, + **_validation_kwargs(artifact_root), + ) + + index_path, _payload = _write_artifact(artifact_root) + (artifact_root / "unindexed.txt").write_text("not in the index", encoding="utf-8") + with pytest.raises(evidence.EvidenceIndexError, match="unindexed files"): + evidence.validate_release_evidence_index( + index_path=index_path, + artifact_root=artifact_root, + expected_release_sha=RELEASE_SHA, + **_validation_kwargs(artifact_root), + ) + + +@pytest.mark.parametrize( + "unsafe_path", + [ + "data/release-trust/report.json:alternate-stream", + "data/release-trust/CON.json", + "data/release-trust/trailing-space ", + ], +) +def test_evidence_index_rejects_nonportable_release_paths(tmp_path: Path, unsafe_path: str) -> None: + artifact_root = tmp_path / "artifact" + artifact_root.mkdir() + index_path, payload = _write_artifact(artifact_root) + payload["items"][0]["path"] = unsafe_path + _rewrite_index(index_path, payload) + + with pytest.raises(evidence.EvidenceIndexError, match="not portable"): + evidence.validate_release_evidence_index( + index_path=index_path, + artifact_root=artifact_root, + expected_release_sha=RELEASE_SHA, + **_validation_kwargs(artifact_root), + ) + + +def test_evidence_index_rejects_symlinks_and_duplicate_json_keys(tmp_path: Path) -> None: + artifact_root = tmp_path / "artifact" + artifact_root.mkdir() + index_path, payload = _write_artifact(artifact_root) + first = payload["items"][0] + source = artifact_root / first["path"] + real_source = source.with_suffix(".real.json") + source.replace(real_source) + try: + source.symlink_to(real_source.name) + except OSError: + pytest.skip("Symlink creation is unavailable on this host") + with pytest.raises(evidence.EvidenceIndexError, match="symlink"): + evidence.validate_release_evidence_index( + index_path=index_path, + artifact_root=artifact_root, + expected_release_sha=RELEASE_SHA, + **_validation_kwargs(artifact_root), + ) + + source.unlink() + real_source.replace(source) + raw = index_path.read_text(encoding="utf-8") + duplicate_key_index = raw.replace( + f'"schemaVersion": "{evidence.INDEX_SCHEMA_VERSION}"', + f'"schemaVersion": "{evidence.INDEX_SCHEMA_VERSION}", "schemaVersion": "duplicate"', + 1, + ) + index_path.write_text(duplicate_key_index, encoding="utf-8") + (artifact_root / evidence.SIGNATURE_FILENAME).write_bytes(SIGNING_PRIVATE_KEY.sign(duplicate_key_index.encode())) + with pytest.raises(evidence.EvidenceIndexError, match="duplicate JSON key"): + evidence.validate_release_evidence_index( + index_path=index_path, + artifact_root=artifact_root, + expected_release_sha=RELEASE_SHA, + **_validation_kwargs(artifact_root), + ) + + +def test_evidence_index_refuses_restore_through_destination_symlink(tmp_path: Path) -> None: + artifact_root = tmp_path / "artifact" + restore_root = tmp_path / "checkout" + outside = tmp_path / "outside" + artifact_root.mkdir() + restore_root.mkdir() + outside.mkdir() + index_path, _payload = _write_artifact(artifact_root) + try: + (restore_root / "data").symlink_to(outside, target_is_directory=True) + except OSError: + pytest.skip("Symlink creation is unavailable on this host") + + with pytest.raises(evidence.EvidenceIndexError, match="Symlinks are prohibited"): + evidence.validate_release_evidence_index( + index_path=index_path, + artifact_root=artifact_root, + expected_release_sha=RELEASE_SHA, + restore_root=restore_root, + **_validation_kwargs(artifact_root), + ) + + assert not any(path.is_file() for path in outside.rglob("*")) + + +def test_evidence_index_rejects_unproven_hyperformula_and_pentest_postures(tmp_path: Path) -> None: + artifact_root = tmp_path / "artifact" + artifact_root.mkdir() + index_path, payload = _write_artifact(artifact_root) + hyperformula = next(item for item in payload["items"] if item["id"] == "hyperformula_posture") + hyperformula["details"]["runtimePresent"] = True + _rewrite_index(index_path, payload) + with pytest.raises(evidence.EvidenceIndexError, match="Disabled HyperFormula posture"): + evidence.validate_release_evidence_index( + index_path=index_path, + artifact_root=artifact_root, + expected_release_sha=RELEASE_SHA, + **_validation_kwargs(artifact_root), + ) + + index_path, payload = _write_artifact(artifact_root) + pentest = next(item for item in payload["items"] if item["id"] == "independent_pentest") + pentest["details"]["undispositionedMediumCount"] = 1 + _rewrite_index(index_path, payload) + with pytest.raises(evidence.EvidenceIndexError, match="zero undispositioned medium"): + evidence.validate_release_evidence_index( + index_path=index_path, + artifact_root=artifact_root, + expected_release_sha=RELEASE_SHA, + **_validation_kwargs(artifact_root), + ) + + +@pytest.mark.parametrize("item_id", ["private_corpus_execution", "private_corpus_inventory"]) +def test_evidence_index_requires_250_workbook_private_corpus( + tmp_path: Path, + item_id: str, +) -> None: + artifact_root = tmp_path / "artifact" + artifact_root.mkdir() + index_path, payload = _write_artifact(artifact_root) + private_item = next(item for item in payload["items"] if item["id"] == item_id) + private_item["details"]["workbookCount"] = 249 + _rewrite_index(index_path, payload) + + with pytest.raises(evidence.EvidenceIndexError, match="at least 250"): + evidence.validate_release_evidence_index( + index_path=index_path, + artifact_root=artifact_root, + expected_release_sha=RELEASE_SHA, + **_validation_kwargs(artifact_root), + ) + + +@pytest.mark.parametrize( + ("field", "value", "message"), + [ + ("nativeExcelExecuted", False, "native-host"), + ("executionReceiptVerified", False, "signed-receipt"), + ("hostPlatform", "synthetic", "host platform"), + ("excelBuildHash", "Excel 16.0", "excelBuildHash"), + ("corpusManifestSha256", "sha256:1234", "corpusManifestSha256"), + ("executionReceiptSha256", None, "executionReceiptSha256"), + ], +) +def test_evidence_index_requires_native_signed_private_corpus_proof( + tmp_path: Path, + field: str, + value: Any, + message: str, +) -> None: + artifact_root = tmp_path / "artifact" + artifact_root.mkdir() + index_path, payload = _write_artifact(artifact_root) + private_item = next( + item for item in payload["items"] if item["id"] == "private_corpus_execution" + ) + private_item["details"][field] = value + _rewrite_index(index_path, payload) + + with pytest.raises(evidence.EvidenceIndexError, match=message): + evidence.validate_release_evidence_index( + index_path=index_path, + artifact_root=artifact_root, + expected_release_sha=RELEASE_SHA, + **_validation_kwargs(artifact_root), + ) + + +@pytest.mark.parametrize( + ("field", "value", "message"), + [ + ("executionReceiptVerified", False, "verified, privacy-safe"), + ("privacyScanStatus", "failed", "verified, privacy-safe"), + ("modelProvidersContacted", True, "provider-free"), + ("failedCount", 1, "no failures"), + ("recordCount", 24, "at least 25"), + ("deterministicReplayCount", 1, "deterministic reruns"), + ("corpusSha256", "sha256:1234", "corpusSha256"), + ("executionReceiptSha256", None, "executionReceiptSha256"), + ], +) +def test_evidence_index_requires_signed_production_replay_execution_proof( + tmp_path: Path, + field: str, + value: Any, + message: str, +) -> None: + artifact_root = tmp_path / "artifact" + artifact_root.mkdir() + index_path, payload = _write_artifact(artifact_root) + replay_item = next( + item for item in payload["items"] if item["id"] == "production_replay" + ) + replay_item["details"][field] = value + _rewrite_index(index_path, payload) + + with pytest.raises(evidence.EvidenceIndexError, match=message): + evidence.validate_release_evidence_index( + index_path=index_path, + artifact_root=artifact_root, + expected_release_sha=RELEASE_SHA, + **_validation_kwargs(artifact_root), + ) + + +def test_evidence_index_accepts_complete_commercial_hyperformula_posture(tmp_path: Path) -> None: + artifact_root = tmp_path / "artifact" + artifact_root.mkdir() + index_path, payload = _write_artifact(artifact_root) + hyperformula = next(item for item in payload["items"] if item["id"] == "hyperformula_posture") + hyperformula["details"] = { + "successful": True, + "posture": "commercial", + "enabled": True, + "distributed": True, + "runtimePresent": True, + "licenseApproved": True, + "commercialKeyInjected": True, + "gplSentinelAbsent": True, + } + _rewrite_index(index_path, payload) + + report = evidence.validate_release_evidence_index( + index_path=index_path, + artifact_root=artifact_root, + expected_release_sha=RELEASE_SHA, + **_validation_kwargs(artifact_root), + ) + + assert report["ready"] is True + + +def test_evidence_index_rejects_nonstandard_json_numbers(tmp_path: Path) -> None: + artifact_root = tmp_path / "artifact" + artifact_root.mkdir() + index_path, payload = _write_artifact(artifact_root) + payload["items"][0]["details"]["notJson"] = float("nan") + _rewrite_index(index_path, payload) + + with pytest.raises(evidence.EvidenceIndexError, match="non-standard JSON number"): + evidence.validate_release_evidence_index( + index_path=index_path, + artifact_root=artifact_root, + expected_release_sha=RELEASE_SHA, + **_validation_kwargs(artifact_root), + ) + + +def test_evidence_index_requires_detached_signature_and_authenticates_raw_details(tmp_path: Path) -> None: + artifact_root = tmp_path / "artifact" + artifact_root.mkdir() + index_path, payload = _write_artifact(artifact_root) + signature_path = artifact_root / evidence.SIGNATURE_FILENAME + signature_path.unlink() + + with pytest.raises(evidence.EvidenceIndexError, match="signature.*exactly"): + evidence.validate_release_evidence_index( + index_path=index_path, + artifact_root=artifact_root, + expected_release_sha=RELEASE_SHA, + **_validation_kwargs(artifact_root), + ) + + _rewrite_index(index_path, payload) + raw_index = index_path.read_bytes() + index_path.write_bytes(raw_index.replace(b'"successful": true', b'"successful": false', 1)) + with pytest.raises(evidence.EvidenceIndexError, match="signature verification failed"): + evidence.validate_release_evidence_index( + index_path=index_path, + artifact_root=artifact_root, + expected_release_sha=RELEASE_SHA, + **_validation_kwargs(artifact_root), + ) + + +def test_evidence_index_rejects_wrong_public_key_and_key_id(tmp_path: Path) -> None: + artifact_root = tmp_path / "artifact" + artifact_root.mkdir() + index_path, _payload = _write_artifact(artifact_root) + wrong_key_path = tmp_path / "wrong-release-key.pem" + wrong_key_path.write_bytes( + Ed25519PrivateKey.generate() + .public_key() + .public_bytes( + encoding=serialization.Encoding.PEM, + format=serialization.PublicFormat.SubjectPublicKeyInfo, + ) + ) + + with pytest.raises(evidence.EvidenceIndexError, match="signature verification failed"): + evidence.validate_release_evidence_index( + index_path=index_path, + artifact_root=artifact_root, + expected_release_sha=RELEASE_SHA, + signature_path=artifact_root / evidence.SIGNATURE_FILENAME, + trusted_public_key_path=wrong_key_path, + trusted_signing_key_id=SIGNING_KEY_ID, + ) + + with pytest.raises(evidence.EvidenceIndexError, match="signingKeyId.*trusted key id"): + evidence.validate_release_evidence_index( + index_path=index_path, + artifact_root=artifact_root, + expected_release_sha=RELEASE_SHA, + signature_path=artifact_root / evidence.SIGNATURE_FILENAME, + trusted_public_key_path=tmp_path / TRUSTED_PUBLIC_KEY_FILENAME, + trusted_signing_key_id="rotated-release-key", + ) + + +@pytest.mark.parametrize("raw_signature", [b"x" * 63, b"x" * 65, b"eA=="]) +def test_evidence_index_rejects_non_raw_or_wrong_length_signature(tmp_path: Path, raw_signature: bytes) -> None: + artifact_root = tmp_path / "artifact" + artifact_root.mkdir() + index_path, _payload = _write_artifact(artifact_root) + (artifact_root / evidence.SIGNATURE_FILENAME).write_bytes(raw_signature) + + with pytest.raises(evidence.EvidenceIndexError, match="raw 64-byte Ed25519 signature"): + evidence.validate_release_evidence_index( + index_path=index_path, + artifact_root=artifact_root, + expected_release_sha=RELEASE_SHA, + **_validation_kwargs(artifact_root), + ) + + +def test_evidence_index_rejects_non_ed25519_trusted_key(tmp_path: Path) -> None: + artifact_root = tmp_path / "artifact" + artifact_root.mkdir() + index_path, _payload = _write_artifact(artifact_root) + rsa_key_path = tmp_path / "rsa-release-key.pem" + rsa_key_path.write_bytes( + generate_private_key(public_exponent=65537, key_size=2048) + .public_key() + .public_bytes( + encoding=serialization.Encoding.PEM, + format=serialization.PublicFormat.SubjectPublicKeyInfo, + ) + ) + + with pytest.raises(evidence.EvidenceIndexError, match="must be an Ed25519 public key"): + evidence.validate_release_evidence_index( + index_path=index_path, + artifact_root=artifact_root, + expected_release_sha=RELEASE_SHA, + signature_path=artifact_root / evidence.SIGNATURE_FILENAME, + trusted_public_key_path=rsa_key_path, + trusted_signing_key_id=SIGNING_KEY_ID, + ) + + +def test_evidence_index_rejects_trusted_key_from_untrusted_artifact_or_repository( + tmp_path: Path, + monkeypatch, +) -> None: + artifact_root = tmp_path / "artifact" + artifact_root.mkdir() + index_path, _payload = _write_artifact(artifact_root) + artifact_key_path = artifact_root / "claimed-trusted-key.pem" + artifact_key_path.write_bytes((tmp_path / TRUSTED_PUBLIC_KEY_FILENAME).read_bytes()) + + with pytest.raises(evidence.EvidenceIndexError, match="out of band, outside the evidence artifact"): + evidence.validate_release_evidence_index( + index_path=index_path, + artifact_root=artifact_root, + expected_release_sha=RELEASE_SHA, + signature_path=artifact_root / evidence.SIGNATURE_FILENAME, + trusted_public_key_path=artifact_key_path, + trusted_signing_key_id=SIGNING_KEY_ID, + ) + + monkeypatch.setattr(evidence, "REPOSITORY_ROOT", tmp_path) + with pytest.raises(evidence.EvidenceIndexError, match="out of band, outside the repository"): + evidence.validate_release_evidence_index( + index_path=index_path, + artifact_root=artifact_root, + expected_release_sha=RELEASE_SHA, + signature_path=artifact_root / evidence.SIGNATURE_FILENAME, + trusted_public_key_path=tmp_path / TRUSTED_PUBLIC_KEY_FILENAME, + trusted_signing_key_id=SIGNING_KEY_ID, + ) + + +def test_release_signoff_requires_verified_unique_owner_roles(tmp_path: Path) -> None: + artifact_root = tmp_path / "artifact" + artifact_root.mkdir() + index_path, payload = _write_artifact(artifact_root) + signoff = next(item for item in payload["items"] if item["id"] == "release_signoff") + signoff["details"]["signatures"][-1]["role"] = "release" + _rewrite_index(index_path, payload) + + with pytest.raises(evidence.EvidenceIndexError, match="release, security, workbook, client, and legal"): + evidence.validate_release_evidence_index( + index_path=index_path, + artifact_root=artifact_root, + expected_release_sha=RELEASE_SHA, + **_validation_kwargs(artifact_root), + ) + + +def test_restore_transaction_rolls_back_replacements_and_new_files_on_later_failure( + tmp_path: Path, + monkeypatch, +) -> None: + artifact_root = tmp_path / "artifact" + restore_root = tmp_path / "checkout" + artifact_root.mkdir() + restore_root.mkdir() + index_path, payload = _write_artifact(artifact_root) + first, second = payload["items"][:2] + first_destination = restore_root / first["path"] + second_destination = restore_root / second["path"] + second_destination.parent.mkdir(parents=True, exist_ok=True) + second_destination.write_bytes(b"original-second") + real_replace = os.replace + failure_injected = False + + def fail_second_destination(source, destination): + nonlocal failure_injected + if Path(destination) == second_destination and not failure_injected: + failure_injected = True + raise OSError("injected second-destination replacement failure") + return real_replace(source, destination) + + monkeypatch.setattr(evidence.os, "replace", fail_second_destination) + with pytest.raises(evidence.EvidenceIndexError, match="all changes were rolled back"): + evidence.validate_release_evidence_index( + index_path=index_path, + artifact_root=artifact_root, + expected_release_sha=RELEASE_SHA, + restore_root=restore_root, + **_validation_kwargs(artifact_root), + ) + + assert failure_injected is True + assert not first_destination.exists() + assert second_destination.read_bytes() == b"original-second" + assert not list(restore_root.rglob("*.tmp")) + assert not list(restore_root.rglob("*.backup")) + + +def test_restore_transaction_surfaces_and_preserves_backup_on_rollback_failure( + tmp_path: Path, + monkeypatch, +) -> None: + artifact_root = tmp_path / "artifact" + restore_root = tmp_path / "checkout" + artifact_root.mkdir() + restore_root.mkdir() + index_path, payload = _write_artifact(artifact_root) + first, second = payload["items"][:2] + first_destination = restore_root / first["path"] + second_destination = restore_root / second["path"] + first_destination.parent.mkdir(parents=True, exist_ok=True) + first_destination.write_bytes(b"original-first") + second_destination.write_bytes(b"original-second") + real_replace = os.replace + commit_failure_injected = False + + def fail_commit_and_rollback(source, destination): + nonlocal commit_failure_injected + source_path = Path(source) + destination_path = Path(destination) + if destination_path == second_destination and not commit_failure_injected: + commit_failure_injected = True + raise OSError("injected second-destination failure") + if destination_path == first_destination and source_path.suffix == ".backup": + raise OSError("injected rollback failure") + return real_replace(source, destination) + + monkeypatch.setattr(evidence.os, "replace", fail_commit_and_rollback) + with pytest.raises(evidence.EvidenceIndexError, match=r"rollback failed: .*injected rollback failure"): + evidence.validate_release_evidence_index( + index_path=index_path, + artifact_root=artifact_root, + expected_release_sha=RELEASE_SHA, + restore_root=restore_root, + **_validation_kwargs(artifact_root), + ) + + assert commit_failure_injected is True + assert list(first_destination.parent.glob(f".{first_destination.name}.*.backup")) diff --git a/backend/tests/contract/test_release_promotion_bundle.py b/backend/tests/contract/test_release_promotion_bundle.py new file mode 100644 index 0000000..d398705 --- /dev/null +++ b/backend/tests/contract/test_release_promotion_bundle.py @@ -0,0 +1,415 @@ +from __future__ import annotations + +import hashlib +import json +import subprocess +import sys +from pathlib import Path +from typing import Any, cast + +import pytest +from cryptography.hazmat.primitives import serialization +from cryptography.hazmat.primitives.asymmetric.ed25519 import Ed25519PrivateKey + +from scripts import build_release_promotion_bundle as builder +from scripts import validate_release_evidence_index as evidence +from scripts import verify_release_promotion_bundle as verifier +from scripts.run_verification_receipt import SCHEMA_VERSION as VERIFICATION_SCHEMA +from scripts.run_verification_receipt import canonical_receipt_bytes + + +def _private_key(path: Path) -> tuple[Ed25519PrivateKey, Path]: + key = Ed25519PrivateKey.generate() + path.write_bytes( + key.private_bytes( + serialization.Encoding.PEM, + serialization.PrivateFormat.PKCS8, + serialization.NoEncryption(), + ) + ) + public_path = path.with_name(f"{path.stem}-public.pem") + public_path.write_bytes( + key.public_key().public_bytes( + serialization.Encoding.PEM, + serialization.PublicFormat.SubjectPublicKeyInfo, + ) + ) + return key, public_path + + +def _repository(root: Path) -> str: + (root / "security").mkdir(parents=True) + (root / "docs").mkdir() + (root / "toolchain.lock.json").write_text( + '{"schemaVersion":1}\n', encoding="utf-8" + ) + (root / "security" / "sbom.cdx.json").write_text( + '{"bomFormat":"CycloneDX"}\n', encoding="utf-8" + ) + (root / "docs" / "THIRD_PARTY_NOTICES.md").write_text( + "# Notices\n", encoding="utf-8" + ) + subprocess.run(["git", "init", "-q"], cwd=root, check=True) + subprocess.run( + ["git", "config", "user.email", "release@example.invalid"], + cwd=root, + check=True, + ) + subprocess.run( + ["git", "config", "user.name", "Release Test"], cwd=root, check=True + ) + subprocess.run( + ["git", "remote", "add", "origin", "https://example.invalid/excel-agent.git"], + cwd=root, + check=True, + ) + subprocess.run(["git", "add", "."], cwd=root, check=True) + subprocess.run(["git", "commit", "-qm", "release"], cwd=root, check=True) + return subprocess.run( + ["git", "rev-parse", "HEAD"], + cwd=root, + check=True, + capture_output=True, + text=True, + ).stdout.strip() + + +def _verification_evidence( + root: Path, + release_sha: str, + signing_key: Ed25519PrivateKey, + key_id: str, +) -> None: + root.mkdir() + log_bytes = b"all verification gates passed\n" + toolchain_bytes = b'{"schemaVersion":1}\n' + receipt: dict[str, object] = { + "schemaVersion": VERIFICATION_SCHEMA, + "signingKeyId": key_id, + "source": { + "commitSha": release_sha, + "branch": "release", + "remote": "https://example.invalid/excel-agent.git", + }, + "profile": "release", + "result": {"status": "passed", "exitCode": 0}, + "log": { + "path": "verification.log", + "sha256": hashlib.sha256(log_bytes).hexdigest(), + "sizeBytes": len(log_bytes), + }, + "toolchainManifest": { + "path": "toolchain.lock.json", + "sha256": hashlib.sha256(toolchain_bytes).hexdigest(), + "sizeBytes": len(toolchain_bytes), + }, + } + receipt_bytes = canonical_receipt_bytes(receipt) + (root / "verification-receipt.json").write_bytes(receipt_bytes) + (root / "verification-receipt.ed25519").write_bytes( + signing_key.sign(receipt_bytes) + ) + (root / "verification.log").write_bytes(log_bytes) + (root / "toolchain.lock.json").write_bytes(toolchain_bytes) + + +def _evidence_details(item_id: str, release_sha: str) -> dict[str, Any]: + details: dict[str, Any] = {"successful": True} + if item_id == "signed_production_manifest": + details["signatureVerified"] = True + host_map = { + "production_manifest_windows": "windows", + "production_manifest_macos": "macos", + "production_manifest_excel_web": "excel_web", + } + if item_id in host_map: + details.update(host=host_map[item_id], signedManifestVerified=True) + if item_id == "private_corpus_execution": + details.update( + excelAuthored=True, + nativeExcelExecuted=True, + executionReceiptVerified=True, + hostPlatform="windows", + excelBuildHash="sha256:" + "1" * 64, + corpusManifestSha256="sha256:" + "2" * 64, + executionReceiptSha256="sha256:" + "3" * 64, + workbookCount=250, + failedCount=0, + ) + if item_id == "private_corpus_inventory": + details.update(workbookCount=250, uninventoriedCount=0) + if item_id == "production_replay": + details.update( + productionEvidence=True, + executionReceiptVerified=True, + privacyScanStatus="passed", + modelProvidersContacted=False, + failedCount=0, + recordCount=25, + deterministicReplayCount=2, + corpusSha256="sha256:" + "4" * 64, + sourceCohortHash="sha256:" + "5" * 64, + executionReceiptSha256="sha256:" + "6" * 64, + baselineDigest="sha256:" + "7" * 64, + evaluationArtifactSha256="sha256:" + "8" * 64, + ) + if item_id == "parser_corpus": + details.update(fixtureCount=250, materialMismatchCount=0) + if item_id.startswith("postgres_"): + details["liveDatabase"] = True + if item_id == "postgres_backup_restore": + details["restoreVerified"] = True + if item_id == "hyperformula_posture": + details.update( + posture="disabled", + enabled=False, + distributed=False, + runtimePresent=False, + gplSentinelAbsent=True, + ) + if item_id == "sbom": + details.update(format="cyclonedx", verified=True) + if item_id in {"trust_report", "dependency_review"}: + details["verified"] = True + if item_id == "container_scan": + details.update(criticalCount=0, highCount=0) + if item_id in {"container_signature", "provenance_verification"}: + details["signatureVerified"] = True + if item_id == "provenance_verification": + details["subjectReleaseSha"] = release_sha + if item_id == "independent_pentest": + details.update( + independent=True, + openCriticalCount=0, + openHighCount=0, + undispositionedMediumCount=0, + ) + if item_id == "release_signoff": + details.update( + subjectReleaseSha=release_sha, + signatures=[ + { + "role": role, + "signerId": f"{role}-owner", + "signatureReference": f"sigstore://release/{role}", + "signatureVerified": True, + } + for role in ("release", "security", "workbook", "client", "legal") + ], + ) + return details + + +def _release_evidence( + root: Path, + release_sha: str, + signing_key: Ed25519PrivateKey, + key_id: str, +) -> tuple[Path, Path]: + root.mkdir() + items: list[dict[str, Any]] = [] + for item_id in evidence.REQUIRED_EVIDENCE: + relative = f"data/release-trust/{item_id}.json" + payload = json.dumps( + {"evidence": item_id, "releaseSha": release_sha}, sort_keys=True + ).encode() + path = root / relative + path.parent.mkdir(parents=True, exist_ok=True) + path.write_bytes(payload) + items.append( + { + "id": item_id, + "releaseSha": release_sha, + "status": "passed", + "path": relative, + "sha256": hashlib.sha256(payload).hexdigest(), + "size": len(payload), + "details": _evidence_details(item_id, release_sha), + } + ) + index = { + "schemaVersion": evidence.INDEX_SCHEMA_VERSION, + "releaseSha": release_sha, + "signingKeyId": key_id, + "items": items, + } + raw_index = json.dumps(index, indent=2, sort_keys=True).encode() + index_path = root / evidence.INDEX_FILENAME + signature_path = root / evidence.SIGNATURE_FILENAME + index_path.write_bytes(raw_index) + signature_path.write_bytes(signing_key.sign(raw_index)) + return index_path, signature_path + + +def _artifacts(root: Path) -> list[tuple[str, Path]]: + root.mkdir() + artifacts = [] + for kind in sorted(builder.REQUIRED_ARTIFACT_KINDS): + path = root / f"{kind}.bin" + path.write_bytes(f"release artifact: {kind}\n".encode()) + artifacts.append((kind, path)) + return artifacts + + +def _fixture(tmp_path: Path) -> dict[str, object]: + repository = tmp_path / "repo" + repository.mkdir() + release_sha = _repository(repository) + verification_key, verification_public = _private_key( + tmp_path / "verification-private.pem" + ) + evidence_key, evidence_public = _private_key(tmp_path / "evidence-private.pem") + release_key, release_public = _private_key(tmp_path / "release-private.pem") + security_key, security_public = _private_key(tmp_path / "security-private.pem") + verification_root = tmp_path / "verification" + evidence_root = tmp_path / "release-evidence" + _verification_evidence( + verification_root, release_sha, verification_key, "verification-k1" + ) + evidence_index, evidence_signature = _release_evidence( + evidence_root, release_sha, evidence_key, "evidence-k1" + ) + return { + "repository": repository, + "release_sha": release_sha, + "verification_root": verification_root, + "verification_public": verification_public, + "evidence_root": evidence_root, + "evidence_index": evidence_index, + "evidence_signature": evidence_signature, + "evidence_public": evidence_public, + "release_key": tmp_path / "release-private.pem", + "release_public": release_public, + "security_key": tmp_path / "security-private.pem", + "security_public": security_public, + "release_private_object": release_key, + "security_private_object": security_key, + "artifacts": _artifacts(tmp_path / "artifacts"), + } + + +def _build(fixture: dict[str, object], output: Path) -> dict[str, object]: + return builder.build_release_promotion_bundle( + repository_root=fixture["repository"], # type: ignore[arg-type] + output_directory=output, + release_sha=fixture["release_sha"], # type: ignore[arg-type] + verification_evidence=fixture["verification_root"], # type: ignore[arg-type] + verification_public_key=fixture["verification_public"], # type: ignore[arg-type] + verification_key_id="verification-k1", + release_evidence_root=fixture["evidence_root"], # type: ignore[arg-type] + release_evidence_index=fixture["evidence_index"], # type: ignore[arg-type] + release_evidence_signature=fixture["evidence_signature"], # type: ignore[arg-type] + release_evidence_public_key=fixture["evidence_public"], # type: ignore[arg-type] + release_evidence_key_id="evidence-k1", + release_private_key=fixture["release_key"], # type: ignore[arg-type] + release_key_id="release-k1", + security_private_key=fixture["security_key"], # type: ignore[arg-type] + security_key_id="security-k1", + artifacts=fixture["artifacts"], # type: ignore[arg-type] + ) + + +def _verify(fixture: dict[str, object], bundle: Path) -> dict[str, object]: + return verifier.verify_release_promotion_bundle( + bundle_directory=bundle, + expected_sha=fixture["release_sha"], # type: ignore[arg-type] + release_public_key=fixture["release_public"], # type: ignore[arg-type] + release_key_id="release-k1", + security_public_key=fixture["security_public"], # type: ignore[arg-type] + security_key_id="security-k1", + verification_public_key=fixture["verification_public"], # type: ignore[arg-type] + release_evidence_public_key=fixture["evidence_public"], # type: ignore[arg-type] + ) + + +def test_release_promotion_bundle_binds_all_bytes_and_two_distinct_approvals( + tmp_path: Path, +) -> None: + fixture = _fixture(tmp_path) + bundle = tmp_path / "promotion" + + manifest = _build(fixture, bundle) + report = _verify(fixture, bundle) + + assert manifest["releaseSha"] == fixture["release_sha"] + assert report["passed"] is True + assert set(cast(list[str], report["artifactKinds"])) == builder.REQUIRED_ARTIFACT_KINDS + assert not (bundle / "release-public.pem").exists() + assert not list(tmp_path.glob(".promotion.*")) + + +def test_release_promotion_bundle_rejects_tamper_extra_files_and_same_signer( + tmp_path: Path, +) -> None: + fixture = _fixture(tmp_path) + bundle = tmp_path / "promotion" + _build(fixture, bundle) + target = next((bundle / "artifacts").rglob("*.bin")) + target.write_bytes(target.read_bytes() + b"tampered") + with pytest.raises(ValueError, match="size does not match|digest mismatch"): + _verify(fixture, bundle) + + target.write_bytes(target.read_bytes()[: -len(b"tampered")]) + (bundle / "unexpected.txt").write_text("not indexed", encoding="utf-8") + with pytest.raises(ValueError, match="path inventory mismatch"): + _verify(fixture, bundle) + + second_output = tmp_path / "same-signer" + with pytest.raises(ValueError, match="distinct keys"): + builder.build_release_promotion_bundle( + repository_root=fixture["repository"], # type: ignore[arg-type] + output_directory=second_output, + release_sha=fixture["release_sha"], # type: ignore[arg-type] + verification_evidence=fixture["verification_root"], # type: ignore[arg-type] + verification_public_key=fixture["verification_public"], # type: ignore[arg-type] + verification_key_id="verification-k1", + release_evidence_root=fixture["evidence_root"], # type: ignore[arg-type] + release_evidence_index=fixture["evidence_index"], # type: ignore[arg-type] + release_evidence_signature=fixture["evidence_signature"], # type: ignore[arg-type] + release_evidence_public_key=fixture["evidence_public"], # type: ignore[arg-type] + release_evidence_key_id="evidence-k1", + release_private_key=fixture["release_key"], # type: ignore[arg-type] + release_key_id="release-k1", + security_private_key=fixture["release_key"], # type: ignore[arg-type] + security_key_id="security-k1", + artifacts=fixture["artifacts"], # type: ignore[arg-type] + ) + + +def test_release_promotion_bundle_requires_complete_artifact_set(tmp_path: Path) -> None: + fixture = _fixture(tmp_path) + fixture["artifacts"] = [ + item + for item in cast(list[tuple[str, Path]], fixture["artifacts"]) + if item[0] != "backend_image" + ] + with pytest.raises(ValueError, match="missing required artifact kinds: backend_image"): + _build(fixture, tmp_path / "promotion") + + +def test_release_promotion_cli_contracts() -> None: + builder_help = subprocess.run( + [sys.executable, str(builder.ROOT / "scripts/build_release_promotion_bundle.py"), "--help"], + cwd=builder.ROOT, + check=True, + capture_output=True, + text=True, + ).stdout + assert "--release-private-key" in builder_help + assert "--security-private-key" in builder_help + assert "--artifact KIND=PATH" in builder_help + + verifier_help = subprocess.run( + [ + sys.executable, + str(builder.ROOT / "scripts/verify_release_promotion_bundle.py"), + "--help", + ], + cwd=builder.ROOT, + check=True, + capture_output=True, + text=True, + ).stdout + assert "--release-public-key" in verifier_help + assert "--security-public-key" in verifier_help + assert "--expected-sha" in verifier_help diff --git a/backend/tests/contract/test_release_workflow_contracts.py b/backend/tests/contract/test_release_workflow_contracts.py new file mode 100644 index 0000000..60647ad --- /dev/null +++ b/backend/tests/contract/test_release_workflow_contracts.py @@ -0,0 +1,83 @@ +import re +from pathlib import Path + + +ROOT = Path(__file__).resolve().parents[3] +WORKFLOW = ROOT / ".github" / "workflows" / "v7-readiness.yml" +WORKFLOWS = ROOT / ".github" / "workflows" + + +def test_release_evidence_workflow_passes_strict_artifacts_to_trust_bundle() -> None: + text = WORKFLOW.read_text(encoding="utf-8") + + assert "python scripts/validate_windows_excel_smoke.py --require-existing-artifacts --json" in text + assert "data/release-trust/windows-excel-smoke-validation.json" in text + assert "python -m backend.benchmarks.v7_readiness --fail-on-warning --require-production-replay --require-private-fixtures" in text + assert "data/release-trust/v7-readiness.json" in text + assert "--smoke-artifact-validation-json data/release-trust/windows-excel-smoke-validation.json" in text + assert "--v7-readiness-json data/release-trust/v7-readiness.json" in text + assert "--require-private-fixtures" in text + + +def test_release_workflow_directly_gates_version_quality_sbom_and_dependencies() -> None: + text = WORKFLOW.read_text(encoding="utf-8") + + assert "python scripts/check_version_sync.py" in text + assert "python scripts/check_package_integrity.py" in text + assert "python scripts/generate_sbom.py --check" in text + assert "python -m ruff check backend scripts --select F,E9,B,ASYNC" in text + assert "python -m mypy backend --ignore-missing-imports" in text + assert "actions/dependency-review-action@2031cfc080254a8a887f58cffee85186f0e49e48" in text + + +def test_release_workflow_builds_and_scans_the_production_edge() -> None: + text = WORKFLOW.read_text(encoding="utf-8") + + assert "docker build -f infra/nginx/Dockerfile -t finance-ai-excel-plugin-edge:ci ." in text + assert "40-render-production-manifest.sh" in text + assert "image-ref: finance-ai-excel-plugin-edge:ci" in text + assert "aquasecurity/trivy-action@ed142fd0673e97e23eac54620cfb913e5ce36c25" in text + assert text.count("version: v0.73.0") == 2 + assert "version: latest" not in text + + +def test_all_workflow_actions_are_immutable_sha_pins() -> None: + action_reference = re.compile(r"^\s*-?\s*uses:\s*[^\s#]+@([^\s#]+)", re.MULTILINE) + references: list[tuple[str, str]] = [] + for workflow in sorted(WORKFLOWS.glob("*.yml")): + for revision in action_reference.findall(workflow.read_text(encoding="utf-8")): + references.append((workflow.name, revision)) + + assert references + assert all(re.fullmatch(r"[0-9a-f]{40}", revision) for _, revision in references), references + + +def test_strict_release_workflow_validates_exact_sha_index_before_safe_restore() -> None: + text = WORKFLOW.read_text(encoding="utf-8") + + validator = "python scripts/validate_release_evidence_index.py" + prereq = "python scripts/verify_release_evidence_prereqs.py" + current_gates = "python scripts/validate_windows_excel_smoke.py --require-existing-artifacts --json" + assert validator in text + assert "--index .release-evidence-input/release-evidence-index.json" in text + assert "--signature .release-evidence-input/release-evidence-index.ed25519" in text + assert "--artifact-root .release-evidence-input" in text + assert "--release-sha $env:GITHUB_SHA" in text + assert "--trusted-public-key $trustedKeyPath" in text + assert "--trusted-signing-key-id $env:RELEASE_EVIDENCE_SIGNING_KEY_ID" in text + assert "--restore-root $env:GITHUB_WORKSPACE" in text + assert "--evidence-index .release-evidence-input/release-evidence-index.json" in text + assert "--evidence-signature .release-evidence-input/release-evidence-index.ed25519" in text + assert "--evidence-root .release-evidence-input" in text + release_job = text[text.index(" release-evidence:") :] + assert release_job.index( + "python -m pip install --require-hashes -r backend/requirements.lock.txt" + ) < release_job.index(validator) + assert "RELEASE_EVIDENCE_ED25519_PUBLIC_KEY_B64: ${{ secrets.RELEASE_EVIDENCE_ED25519_PUBLIC_KEY_B64 }}" in release_job + assert "RELEASE_EVIDENCE_SIGNING_KEY_ID: ${{ vars.RELEASE_EVIDENCE_SIGNING_KEY_ID }}" in release_job + assert "[string]::IsNullOrWhiteSpace($env:RELEASE_EVIDENCE_ED25519_PUBLIC_KEY_B64)" in release_job + assert "[string]::IsNullOrWhiteSpace($env:RELEASE_EVIDENCE_SIGNING_KEY_ID)" in release_job + assert text.index(validator) < text.index(current_gates) + assert text.index(validator) < text.index(prereq) + assert "StartsWith($evidenceCommit" not in text + assert "Copy-Item -LiteralPath $sourceItem.FullName" not in text diff --git a/backend/tests/contract/test_sbom_generation.py b/backend/tests/contract/test_sbom_generation.py new file mode 100644 index 0000000..bb399c9 --- /dev/null +++ b/backend/tests/contract/test_sbom_generation.py @@ -0,0 +1,137 @@ +import pytest + +from scripts import generate_sbom + + +def test_sbom_is_deterministic_and_covers_each_locked_ecosystem() -> None: + first = generate_sbom.build_sbom() + second = generate_sbom.build_sbom() + + assert first == second + assert first["bomFormat"] == "CycloneDX" + assert first["specVersion"] == "1.6" + assert first["metadata"]["component"]["version"] == "0.5.0" + refs = [component["bom-ref"] for component in first["components"]] + assert len(refs) == len(set(refs)) + assert any(ref.startswith("pkg:pypi/") for ref in refs) + assert any(ref.startswith("pkg:npm/") for ref in refs) + assert any(ref.startswith("pkg:cargo/") for ref in refs) + + source_locks = { + component["bom-ref"]: next(prop["value"] for prop in component["properties"] if prop["name"] == "excel-agent:source-locks") + for component in first["components"] + } + assert source_locks["pkg:pypi/protobuf@7.35.1"] == "vnext/pyproject.toml" + assert source_locks["pkg:pypi/setuptools@82.0.1"] == "vnext/pyproject.toml" + assert source_locks["pkg:npm/%40bufbuild/protobuf@2.13.0"] == "vnext/package-lock.json" + assert source_locks["pkg:cargo/prost@0.14.4"] == "vnext/Cargo.lock" + + components = {component["bom-ref"]: component for component in first["components"]} + expected_cargo_licenses = { + "pkg:cargo/adler2@2.0.1": "0BSD OR MIT OR Apache-2.0", + "pkg:cargo/arc-swap@1.9.2": "MIT OR Apache-2.0", + "pkg:cargo/async-lock@3.4.2": "Apache-2.0 OR MIT", + "pkg:cargo/async-trait@0.1.92": "MIT OR Apache-2.0", + "pkg:cargo/bitflags@2.13.1": "MIT OR Apache-2.0", + "pkg:cargo/bumpalo@3.20.3": "MIT OR Apache-2.0", + "pkg:cargo/crc32fast@1.5.0": "MIT OR Apache-2.0", + "pkg:cargo/crossbeam-utils@0.8.22": "MIT OR Apache-2.0", + "pkg:cargo/dashmap@6.2.1": "MIT", + "pkg:cargo/equivalent@1.0.2": "Apache-2.0 OR MIT", + "pkg:cargo/event-listener@5.4.2": "Apache-2.0 OR MIT", + "pkg:cargo/event-listener-strategy@0.5.4": "Apache-2.0 OR MIT", + "pkg:cargo/fastrand@2.5.0": "Apache-2.0 OR MIT", + "pkg:cargo/flate2@1.1.9": "MIT OR Apache-2.0", + "pkg:cargo/getrandom@0.4.3": "MIT OR Apache-2.0", + "pkg:cargo/hashbrown@0.14.5": "MIT OR Apache-2.0", + "pkg:cargo/hashbrown@0.17.1": "MIT OR Apache-2.0", + "pkg:cargo/indexmap@2.14.0": "Apache-2.0 OR MIT", + "pkg:cargo/itoa@1.0.18": "MIT OR Apache-2.0", + "pkg:cargo/js-sys@0.3.104": "MIT OR Apache-2.0", + "pkg:cargo/lock_api@0.4.14": "MIT OR Apache-2.0", + "pkg:cargo/memchr@2.8.3": "Unlicense OR MIT", + "pkg:cargo/miniz_oxide@0.8.9": "MIT OR Zlib OR Apache-2.0", + "pkg:cargo/once_cell@1.21.4": "MIT OR Apache-2.0", + "pkg:cargo/parking@2.2.1": "Apache-2.0 OR MIT", + "pkg:cargo/parking_lot_core@0.9.12": "MIT OR Apache-2.0", + "pkg:cargo/pin-project-lite@0.2.17": "Apache-2.0 OR MIT", + "pkg:cargo/quick-xml@0.41.0": "MIT", + "pkg:cargo/r-efi@6.0.0": "MIT OR Apache-2.0 OR LGPL-2.1-or-later", + "pkg:cargo/redox_syscall@0.5.18": "MIT", + "pkg:cargo/rustversion@1.0.23": "MIT OR Apache-2.0", + "pkg:cargo/scopeguard@1.2.0": "MIT OR Apache-2.0", + "pkg:cargo/serde@1.0.229": "MIT OR Apache-2.0", + "pkg:cargo/serde_core@1.0.229": "MIT OR Apache-2.0", + "pkg:cargo/serde_derive@1.0.229": "MIT OR Apache-2.0", + "pkg:cargo/serde_json@1.0.151": "MIT OR Apache-2.0", + "pkg:cargo/simd-adler32@0.3.10": "MIT", + "pkg:cargo/smallstr@0.3.1": "MIT OR Apache-2.0", + "pkg:cargo/smallvec@1.15.2": "MIT OR Apache-2.0", + "pkg:cargo/syn@3.0.3": "MIT OR Apache-2.0", + "pkg:cargo/thiserror@2.0.20": "MIT OR Apache-2.0", + "pkg:cargo/thiserror-impl@2.0.20": "MIT OR Apache-2.0", + "pkg:cargo/typed-path@0.12.3": "MIT OR Apache-2.0", + "pkg:cargo/unicode-casefold@0.2.0": "MIT/Apache-2.0", + "pkg:cargo/unicode-normalization@0.1.25": "MIT OR Apache-2.0", + "pkg:cargo/tinyvec@1.12.0": "Zlib OR Apache-2.0 OR MIT", + "pkg:cargo/tinyvec_macros@0.1.1": "MIT OR Apache-2.0 OR Zlib", + "pkg:cargo/wasm-bindgen@0.2.127": "MIT OR Apache-2.0", + "pkg:cargo/wasm-bindgen-macro@0.2.127": "MIT OR Apache-2.0", + "pkg:cargo/wasm-bindgen-macro-support@0.2.127": "MIT OR Apache-2.0", + "pkg:cargo/wasm-bindgen-shared@0.2.127": "MIT OR Apache-2.0", + "pkg:cargo/windows-link@0.2.1": "MIT OR Apache-2.0", + "pkg:cargo/yrs@0.27.3": "MIT", + "pkg:cargo/zip@7.2.0": "MIT", + "pkg:cargo/zlib-rs@0.6.7": "Zlib", + "pkg:cargo/zmij@1.0.23": "MIT", + } + for ref, license_name in expected_cargo_licenses.items(): + assert components[ref]["licenses"] == [{"license": {"name": license_name}}] + assert source_locks[ref] == "vnext/Cargo.lock" + + cargo_components = [component for ref, component in components.items() if ref.startswith("pkg:cargo/")] + assert cargo_components + assert all(component.get("licenses") for component in cargo_components) + + +def test_committed_sbom_matches_dependency_locks() -> None: + assert generate_sbom.main(["--check"]) == 0 + + +def test_python_sbom_components_retain_artifact_hashes(tmp_path) -> None: + digest = "a" * 64 + lock = tmp_path / "requirements.lock.txt" + lock.write_text( + f"fastapi==0.141.1 \\\n --hash=sha256:{digest}\n", + encoding="utf-8", + ) + + components = generate_sbom._python_components(lock) + + assert components == [ + { + "type": "library", + "bom-ref": "pkg:pypi/fastapi@0.141.1", + "name": "fastapi", + "version": "0.141.1", + "purl": "pkg:pypi/fastapi@0.141.1", + "hashes": [{"alg": "SHA-256", "content": digest}], + } + ] + + lock.write_text("fastapi==0.141.1\n", encoding="utf-8") + with pytest.raises(ValueError, match="missing hashes"): + generate_sbom._python_components(lock) + + +def test_cargo_sbom_rejects_unreviewed_license_mapping(tmp_path) -> None: + lock_path = tmp_path / "Cargo.lock" + lock_path.write_text( + 'version = 4\n\n[[package]]\nname = "unreviewed"\nversion = "1.0.0"\n' + 'source = "registry+https://github.com/rust-lang/crates.io-index"\n' + f'checksum = "{"0" * 64}"\n', + encoding="utf-8", + ) + + with pytest.raises(ValueError, match="has no reviewed Cargo license mapping"): + generate_sbom._cargo_components(lock_path) diff --git a/backend/tests/contract/test_stack_readiness_contracts.py b/backend/tests/contract/test_stack_readiness_contracts.py index 3b8ae66..a647b78 100644 --- a/backend/tests/contract/test_stack_readiness_contracts.py +++ b/backend/tests/contract/test_stack_readiness_contracts.py @@ -3,6 +3,7 @@ from __future__ import annotations import subprocess +import sys from pathlib import Path from backend.services.operation_registry import get_operation_registry_v3, operation_metadata @@ -37,6 +38,7 @@ def test_source_data_modules_are_not_hidden_by_runtime_data_ignore_rule(): def test_env_example_matches_v7_safe_default_configuration(): env = _env_example() + assert env["HOST"] == "127.0.0.1" assert env["PLAN_ENGINE_VERSION"] == "v2" assert env["AGENT_RUNNER_VERSION"] == "disabled" assert env["SEMANTIC_MAP_VERSION"] == "v1" @@ -46,11 +48,43 @@ def test_env_example_matches_v7_safe_default_configuration(): assert env["ROLLBACK_MODE"] == "range_v1" assert env["SOURCE_EVIDENCE_MODE"] == "enabled" assert env["OPERATION_REGISTRY_VERSION"] == "v1" + assert env["ANTHROPIC_MODEL_MEDIUM"] == "claude-haiku-4-5-20251001" + assert env["ANTHROPIC_MODEL_COMPLEX"] == "claude-sonnet-4-6" + assert env["HYPERFORMULA_ENABLED"] == "false" + assert env["FORMULA_ENGINE"] == "univer" + assert env["HYPERFORMULA_DISTRIBUTED"] == "false" + assert env["HYPERFORMULA_LICENSE_APPROVED"] == "false" + assert env["HYPERFORMULA_LICENSE_KEY"] == "" + assert env["PDF_PARSE_TIMEOUT_SECONDS"] == "30" + assert env["PDF_PARSE_CPU_SECONDS"] == "20" + assert env["PDF_PARSE_MEMORY_BYTES"] == "805306368" + assert env["PDF_PARSE_MAX_OUTPUT_BYTES"] == "16777216" + assert env["PDF_PARSE_MAX_CONCURRENCY"] == "2" + assert env["PDF_PARSE_MAX_QUEUE"] == "16" + assert env["WORK_ADMISSION_MAX_CONCURRENCY"] == "4" + assert env["WORK_ADMISSION_MAX_QUEUE"] == "64" + assert env["WORK_ADMISSION_MAX_QUEUE_PER_SESSION"] == "8" + assert env["WORK_ADMISSION_QUEUE_TIMEOUT_SECONDS"] == "5" + assert env["WORKBOOK_WORKER_MAX_CONCURRENCY"] == "2" + assert env["WORKBOOK_WORKER_MAX_QUEUE"] == "32" + assert env["MANAGED_OBJECT_STORE_ENABLED"] == "false" + assert env["MANAGED_OBJECT_STORE_CURRENT_KEY"] == "" + assert env["MANAGED_OBJECT_STORE_ADDRESS_KEY"] == "" + assert env["MANAGED_OBJECT_STORE_PREVIOUS_KEYS"] == "{}" + assert env["MANAGED_OBJECT_STORE_MAX_OBJECT_BYTES"] == "67108864" + assert env["MANAGED_OBJECT_STORE_TOTAL_BYTES"] == "1073741824" + assert env["MANAGED_OBJECT_STORE_CHUNK_BYTES"] == "262144" + assert env["MANAGED_OBJECT_STORE_NEW_KEY"] == "" + + +def test_backend_direct_execution_defaults_to_loopback(): + config_source = (ROOT / "backend" / "config.py").read_text(encoding="utf-8") + assert 'HOST = os.getenv("HOST", "127.0.0.1")' in config_source def test_generated_taskpane_api_types_are_in_sync_with_openapi_schema(): result = subprocess.run( - ["python", "scripts/generate_openapi_types.py", "--check"], + [sys.executable, "scripts/generate_openapi_types.py", "--check"], cwd=ROOT, text=True, stdout=subprocess.PIPE, @@ -85,6 +119,9 @@ def test_v3_registry_exposes_default_switch_readiness_metadata_for_every_operati assert metadata["rollbackProofClass"] in { "not_required", "range_restore_operations", + "chart_restore_operations", + "table_restore_operations", + "comment_restore_operations", "created_object_pointer", "manual", } @@ -103,6 +140,22 @@ def test_v3_registry_exposes_default_switch_readiness_metadata_for_every_operati assert unknown["handlerCoverage"]["failClosed"] is True assert unknown["defaultSwitchBlockers"] + assert registry["create_chart"]["mockCoverageStatus"] == "behavioral" + assert registry["create_chart"]["rollbackProofClass"] == "chart_restore_operations" + assert registry["create_chart"]["rollbackProof"]["warningsRequired"] is False + assert registry["update_chart"]["rollbackProof"]["strategy"] == "chart_restore_operations" + assert registry["delete_chart"]["rollbackProof"]["strategy"] == "chart_restore_operations" + assert registry["create_table"]["mockCoverageStatus"] == "behavioral" + assert registry["create_table"]["rollbackProofClass"] == "table_restore_operations" + assert registry["create_table"]["rollbackProof"]["warningsRequired"] is False + assert registry["update_table"]["rollbackProof"]["strategy"] == "table_restore_operations" + assert registry["delete_table"]["rollbackProof"]["strategy"] == "table_restore_operations" + assert registry["add_comment"]["mockCoverageStatus"] == "behavioral" + assert registry["add_comment"]["rollbackProofClass"] == "comment_restore_operations" + assert registry["add_comment"]["rollbackProof"]["warningsRequired"] is False + assert registry["update_comment"]["rollbackProof"]["strategy"] == "comment_restore_operations" + assert registry["delete_comment"]["rollbackProof"]["strategy"] == "comment_restore_operations" + def test_stack_readiness_exposes_product_workflow_rollout_and_proof_contract(): from fastapi.testclient import TestClient @@ -117,6 +170,11 @@ def test_stack_readiness_exposes_product_workflow_rollout_and_proof_contract(): assert payload["categories"]["product"]["signals"]["schemaVersion"] == "2026-05-product-trust-workspace-readiness" assert payload["categories"]["brain"]["signals"]["schemaVersion"] == "2026-05-brain-readiness" + plugin_signals = payload["categories"]["plugin"]["signals"] + assert plugin_signals["officeMockCoverage"]["status"] == "passed" + assert plugin_signals["rollbackCoverage"]["automaticObjectRollback"] == {"charts": True, "tables": True, "comments": True, "pivots": False} + assert plugin_signals["rollbackCoverage"]["objectRollback"] == "chart_table_comment_automatic_pivot_manual" + assert payload["executableProofs"]["proofs"]["rollbackRestore"] == "passed" assert payload["brainReadiness"]["applyEligibilityCoverage"]["serverAuthoritative"] is True assert payload["brainReadiness"]["artifactLineageCoverage"]["status"] == "passed" assert payload["trustWorkflow"]["policy"] == "understand_inspect_simulate_checkpoint_apply_verify_rollback_repair" diff --git a/backend/tests/contract/test_verification_receipt.py b/backend/tests/contract/test_verification_receipt.py new file mode 100644 index 0000000..104e1ed --- /dev/null +++ b/backend/tests/contract/test_verification_receipt.py @@ -0,0 +1,238 @@ +from __future__ import annotations + +import importlib.util +import hashlib +import json +import subprocess +import sys +from pathlib import Path + +import pytest +from cryptography.exceptions import InvalidSignature +from cryptography.hazmat.primitives.asymmetric.ed25519 import Ed25519PrivateKey +from cryptography.hazmat.primitives.serialization import ( + Encoding, + NoEncryption, + PrivateFormat, + PublicFormat, +) + + +ROOT = Path(__file__).resolve().parents[3] + + +def _module(): + path = ROOT / "scripts" / "run_verification_receipt.py" + spec = importlib.util.spec_from_file_location("verification_receipt_contract", path) + assert spec is not None and spec.loader is not None + module = importlib.util.module_from_spec(spec) + sys.modules[spec.name] = module + spec.loader.exec_module(module) + return module + + +def test_canonical_receipt_is_signed_and_tamper_evident() -> None: + module = _module() + key = Ed25519PrivateKey.generate() + receipt = { + "schemaVersion": module.SCHEMA_VERSION, + "source": {"commitSha": "a" * 40}, + "result": {"status": "passed", "exitCode": 0}, + } + encoded = module.canonical_receipt_bytes(receipt) + assert encoded.endswith(b"\n") + assert encoded == module.canonical_receipt_bytes(dict(reversed(list(receipt.items())))) + signature = key.sign(encoded) + key.public_key().verify(signature, encoded) + with pytest.raises(InvalidSignature): + key.public_key().verify(signature, encoded.replace(b"passed", b"failed")) + + +def test_clean_repository_identity_rejects_dirty_tree(tmp_path: Path) -> None: + module = _module() + repository = tmp_path / "repo" + repository.mkdir() + subprocess.run(["git", "init", "-q"], cwd=repository, check=True) + subprocess.run(["git", "config", "user.email", "test@example.invalid"], cwd=repository, check=True) + subprocess.run(["git", "config", "user.name", "Test"], cwd=repository, check=True) + subprocess.run(["git", "remote", "add", "origin", "https://example.invalid/repo.git"], cwd=repository, check=True) + (repository / "tracked.txt").write_text("clean\n", encoding="utf-8") + subprocess.run(["git", "add", "tracked.txt"], cwd=repository, check=True) + subprocess.run(["git", "commit", "-qm", "baseline"], cwd=repository, check=True) + + identity = module.clean_repository_identity(repository) + assert len(identity["commitSha"]) == 40 + assert identity["remote"] == "https://example.invalid/repo.git" + + subprocess.run( + [ + "git", + "remote", + "set-url", + "origin", + "https://secret-token@example.invalid/repo.git?credential=hidden", + ], + cwd=repository, + check=True, + ) + sanitized = module.clean_repository_identity(repository) + assert sanitized["remote"] == "https://example.invalid/repo.git" + assert "secret-token" not in sanitized["remote"] + + (repository / "untracked.txt").write_text("dirty\n", encoding="utf-8") + with pytest.raises(ValueError, match="clean working tree"): + module.clean_repository_identity(repository) + + +def test_signing_key_and_output_must_stay_outside_repository(tmp_path: Path) -> None: + module = _module() + repository = tmp_path / "repo" + repository.mkdir() + key = Ed25519PrivateKey.generate() + key_path = repository / "release-key.pem" + key_path.write_bytes( + key.private_bytes(Encoding.PEM, PrivateFormat.PKCS8, NoEncryption()) + ) + with pytest.raises(ValueError, match="outside the repository"): + module._regular_private_key(key_path, repository) + with pytest.raises(ValueError, match="outside the repository"): + module._outside_repository(repository / "evidence", repository, "receipt output") + + external_key = tmp_path / "external-key.pem" + external_key.write_bytes( + key.private_bytes(Encoding.PEM, PrivateFormat.PKCS8, NoEncryption()) + ) + key_link = tmp_path / "key-link.pem" + key_link.symlink_to(external_key) + with pytest.raises(ValueError, match="non-symlink"): + module._regular_private_key(key_link, repository) + + +def test_verification_receipt_binds_clean_sha_log_and_signature(tmp_path: Path) -> None: + module = _module() + repository = tmp_path / "repo" + scripts = repository / "scripts" + scripts.mkdir(parents=True) + (repository / "toolchain.lock.json").write_text( + '{"schemaVersion":1}\n', encoding="utf-8" + ) + (scripts / "toolchain_doctor.py").write_text( + "import json\n" + "print(json.dumps({'schema_version':'doctor-v1','passed':True,'tools':" + "[{'name':'python','command':['/private/python'],'error':'','passed':True}]}))\n", + encoding="utf-8", + ) + (scripts / "verify-local.sh").write_text( + "#!/bin/sh\nprintf 'verified %s %s\\n' \"$1\" \"$2\"\n", + encoding="utf-8", + ) + subprocess.run(["git", "init", "-q"], cwd=repository, check=True) + subprocess.run(["git", "config", "user.email", "test@example.invalid"], cwd=repository, check=True) + subprocess.run(["git", "config", "user.name", "Test"], cwd=repository, check=True) + subprocess.run(["git", "remote", "add", "origin", "https://example.invalid/repo.git"], cwd=repository, check=True) + subprocess.run(["git", "add", "."], cwd=repository, check=True) + subprocess.run(["git", "commit", "-qm", "baseline"], cwd=repository, check=True) + commit_sha = subprocess.run( + ["git", "rev-parse", "HEAD"], + cwd=repository, + check=True, + capture_output=True, + text=True, + ).stdout.strip() + + key = Ed25519PrivateKey.generate() + key_path = tmp_path / "verification-key.pem" + key_path.write_bytes( + key.private_bytes(Encoding.PEM, PrivateFormat.PKCS8, NoEncryption()) + ) + output = tmp_path / "evidence" + result = module.run_verification_receipt( + repository_root=repository, + output_directory=output, + profile="fast", + signing_key_path=key_path, + signing_key_id="test-k1", + ) + assert result == 0 + receipt_bytes = (output / "verification-receipt.json").read_bytes() + signature = (output / "verification-receipt.ed25519").read_bytes() + key.public_key().verify(signature, receipt_bytes) + receipt = json.loads(receipt_bytes) + log_bytes = (output / "verification.log").read_bytes() + assert receipt["source"]["commitSha"] == commit_sha + assert receipt["result"] == {"status": "passed", "exitCode": 0} + assert receipt["log"]["sha256"] == hashlib.sha256(log_bytes).hexdigest() + assert receipt["log"]["sizeBytes"] == len(log_bytes) + assert receipt["toolchain"]["tools"][0].get("command") is None + assert str(tmp_path) not in receipt_bytes.decode() + assert not list(tmp_path.glob(".evidence.*")) + + public_key_path = tmp_path / "verification-public.pem" + public_key_path.write_bytes( + key.public_key().public_bytes(Encoding.PEM, PublicFormat.SubjectPublicKeyInfo) + ) + verifier_path = ROOT / "scripts" / "verify_verification_receipt.py" + verification_command = [ + sys.executable, + str(verifier_path), + "--evidence", + str(output), + "--trusted-public-key", + str(public_key_path), + "--expected-sha", + commit_sha, + "--expected-key-id", + "test-k1", + "--require-passed", + "--json", + ] + verified = subprocess.run( + verification_command, + cwd=ROOT, + check=True, + capture_output=True, + text=True, + ) + report = json.loads(verified.stdout) + assert report["passed"] is True + assert report["sourceSha"] == commit_sha + + (output / "verification.log").write_bytes(log_bytes + b"tampered\n") + rejected = subprocess.run( + verification_command, + cwd=ROOT, + check=False, + capture_output=True, + text=True, + ) + assert rejected.returncode == 1 + assert "log digest" in rejected.stderr + + +def test_verification_receipt_cli_contract() -> None: + script = ROOT / "scripts" / "run_verification_receipt.py" + completed = subprocess.run( + [sys.executable, str(script), "--help"], + cwd=ROOT, + check=True, + capture_output=True, + text=True, + ) + assert "--signing-key" in completed.stdout + assert "--signing-key-id" in completed.stdout + assert "--profile" in completed.stdout + + verifier = ROOT / "scripts" / "verify_verification_receipt.py" + verifier_help = subprocess.run( + [sys.executable, str(verifier), "--help"], + cwd=ROOT, + check=True, + capture_output=True, + text=True, + ) + assert "--expected-sha" in verifier_help.stdout + assert "--trusted-public-key" in verifier_help.stdout + assert "--require-passed" in verifier_help.stdout + + manifest = json.loads((ROOT / "toolchain.lock.json").read_text(encoding="utf-8")) + assert manifest["schemaVersion"] == 1 diff --git a/backend/tests/contract/test_version_sync.py b/backend/tests/contract/test_version_sync.py new file mode 100644 index 0000000..35b97de --- /dev/null +++ b/backend/tests/contract/test_version_sync.py @@ -0,0 +1,67 @@ +import json + +from scripts import check_version_sync + + +def test_version_sync_normalizes_product_version_for_both_office_manifests( + tmp_path, + monkeypatch, +) -> None: + version = tmp_path / "VERSION" + config = tmp_path / "config.py" + package = tmp_path / "package.json" + package_lock = tmp_path / "package-lock.json" + development_manifest = tmp_path / "manifest.xml" + production_manifest = tmp_path / "manifest.production.xml.template" + version.write_text("0.5.0\n", encoding="utf-8") + config.write_text('VERSION = "0.5.0"\n', encoding="utf-8") + package.write_text(json.dumps({"version": "0.5.0"}), encoding="utf-8") + package_lock.write_text( + json.dumps({"version": "0.5.0", "packages": {"": {"version": "0.5.0"}}}), + encoding="utf-8", + ) + for manifest in (development_manifest, production_manifest): + manifest.write_text("0.5.0.0", encoding="utf-8") + + monkeypatch.setattr(check_version_sync, "ROOT", tmp_path) + monkeypatch.setattr(check_version_sync, "VERSION_FILE", version) + monkeypatch.setattr(check_version_sync, "BACKEND_CONFIG", config) + monkeypatch.setattr(check_version_sync, "PACKAGE_FILES", (package, package_lock)) + monkeypatch.setattr( + check_version_sync, + "OFFICE_MANIFESTS", + (development_manifest, production_manifest), + ) + + assert check_version_sync.office_version_for_product("0.5.0") == "0.5.0.0" + assert check_version_sync.main() == 0 + + +def test_version_sync_fails_when_production_manifest_drifts(tmp_path, monkeypatch) -> None: + version = tmp_path / "VERSION" + config = tmp_path / "config.py" + package = tmp_path / "package.json" + package_lock = tmp_path / "package-lock.json" + development_manifest = tmp_path / "manifest.xml" + production_manifest = tmp_path / "manifest.production.xml.template" + version.write_text("0.5.0\n", encoding="utf-8") + config.write_text('VERSION = "0.5.0"\n', encoding="utf-8") + package.write_text(json.dumps({"version": "0.5.0"}), encoding="utf-8") + package_lock.write_text( + json.dumps({"version": "0.5.0", "packages": {"": {"version": "0.5.0"}}}), + encoding="utf-8", + ) + development_manifest.write_text("0.5.0.0", encoding="utf-8") + production_manifest.write_text("1.0.0.0", encoding="utf-8") + + monkeypatch.setattr(check_version_sync, "ROOT", tmp_path) + monkeypatch.setattr(check_version_sync, "VERSION_FILE", version) + monkeypatch.setattr(check_version_sync, "BACKEND_CONFIG", config) + monkeypatch.setattr(check_version_sync, "PACKAGE_FILES", (package, package_lock)) + monkeypatch.setattr( + check_version_sync, + "OFFICE_MANIFESTS", + (development_manifest, production_manifest), + ) + + assert check_version_sync.main() == 1 diff --git a/backend/tests/contract/test_windows_excel_smoke_evidence.py b/backend/tests/contract/test_windows_excel_smoke_evidence.py new file mode 100644 index 0000000..17915eb --- /dev/null +++ b/backend/tests/contract/test_windows_excel_smoke_evidence.py @@ -0,0 +1,208 @@ +"""Strict Windows Excel smoke evidence contract tests.""" + +from __future__ import annotations + +import json +from pathlib import Path + +from scripts import generate_release_trust_bundle as trust +from scripts import scaffold_windows_excel_smoke as smoke_scaffold +from scripts import validate_windows_excel_smoke as smoke_validator + + +def test_strict_validator_rejects_completed_rows_without_attachment_references(tmp_path: Path) -> None: + artifacts = write_smoke_artifacts(tmp_path) + smoke_doc = tmp_path / "WINDOWS_EXCEL_MANUAL_SMOKE.md" + smoke_doc.write_text( + complete_smoke_doc(tmp_path, artifacts).replace( + "| EXCEL-ENV | Excel Desktop environment | Metadata captured | Pass | evidence/env.png |", + "| EXCEL-ENV | Excel Desktop environment | Metadata captured | Pass | environment captured in notes |", + ), + encoding="utf-8", + ) + + report = smoke_validator.validate_smoke_doc(smoke_doc, require_existing_artifacts=True) + + assert report["ready"] is False + assert report["status"] == "incomplete" + assert any(gap["kind"] == "checklist_attachment" for gap in report["artifactValidation"]["missing"]) + + +def test_strict_validator_rejects_remote_only_completed_row_evidence(tmp_path: Path) -> None: + artifacts = write_smoke_artifacts(tmp_path) + smoke_doc = tmp_path / "WINDOWS_EXCEL_MANUAL_SMOKE.md" + smoke_doc.write_text( + complete_smoke_doc(tmp_path, artifacts).replace( + "| EXCEL-ENV | Excel Desktop environment | Metadata captured | Pass | evidence/env.png |", + "| EXCEL-ENV | Excel Desktop environment | Metadata captured | Pass | https://evidence.example.com/env.png |", + ), + encoding="utf-8", + ) + + report = smoke_validator.validate_smoke_doc(smoke_doc, require_existing_artifacts=True) + + assert report["ready"] is False + assert report["status"] == "incomplete" + assert any( + gap["kind"] == "checklist_evidence" and "local non-empty artifact" in gap["reason"] + for gap in report["artifactValidation"]["missing"] + ) + + +def test_strict_validator_rejects_stale_preflight_trust_bundle(tmp_path: Path) -> None: + artifacts = write_smoke_artifacts(tmp_path, preflight_generated_at="2026-06-14T23:59:00Z") + smoke_doc = tmp_path / "WINDOWS_EXCEL_MANUAL_SMOKE.md" + smoke_doc.write_text(complete_smoke_doc(tmp_path, artifacts), encoding="utf-8") + + report = smoke_validator.validate_smoke_doc(smoke_doc, require_existing_artifacts=True) + + assert report["ready"] is False + assert report["status"] == "incomplete" + assert report["preflightValidation"]["status"] == "invalid" + assert any("older than the smoke run date" in gap["reason"] for gap in report["preflightValidation"]["gaps"]) + + +def test_scaffold_run_pack_validation_rejects_manifest_slot_drift(tmp_path: Path) -> None: + report = smoke_scaffold.scaffold_smoke_run( + output_dir=tmp_path, + label="smoke", + template=trust.SMOKE_DOC, + tester="Analyst", + excel_version="Microsoft Excel 2405 Build 17628", + ) + smoke_doc = Path(report["smokeDoc"]) + manifest_path = Path(report["evidenceManifest"]) + manifest = json.loads(manifest_path.read_text(encoding="utf-8")) + manifest["slotCount"] = 0 + manifest["slots"] = [] + manifest_path.write_text(json.dumps(manifest, indent=2), encoding="utf-8") + + validation = smoke_validator.validate_smoke_doc(smoke_doc, require_existing_artifacts=True) + + assert validation["ready"] is False + assert validation["runPackValidation"]["status"] == "invalid" + assert any(gap["field"] == "evidence-slot-manifest.slotCount" for gap in validation["runPackValidation"]["gaps"]) + + +def write_smoke_artifacts( + base_dir: Path, + *, + preflight_generated_at: str = "2026-06-15T12:00:00Z", + git_head: str = "a1b2c3d", +) -> dict[str, Path]: + evidence_dir = base_dir / "evidence" + paths = { + "preflight_log": evidence_dir / "preflight.log", + "preflight_bundle": evidence_dir / "preflight-trust-bundle.json", + "env": evidence_dir / "env.png", + "simple": evidence_dir / "simple.mp4", + "sideload": evidence_dir / "sideload.png", + "sync": evidence_dir / "sync.mp4", + "drift": evidence_dir / "drift.mp4", + "protected": evidence_dir / "protected.png", + "export": evidence_dir / "export.xlsx", + "screenshots": evidence_dir / "screenshots" / "taskpane.png", + "excel_log": evidence_dir / "logs" / "excel.log", + } + for path in paths.values(): + path.parent.mkdir(parents=True, exist_ok=True) + if path.suffix == ".json": + path.write_text( + json.dumps( + { + "bundleId": "windows-excel-preflight-20260615T120000Z", + "generatedAt": preflight_generated_at, + "git": {"head": git_head}, + }, + indent=2, + ), + encoding="utf-8", + ) + else: + path.write_text("captured evidence\n", encoding="utf-8") + return paths + + +def complete_smoke_doc(base_dir: Path, artifacts: dict[str, Path]) -> str: + metadata_values = { + "Date": "2026-06-15", + "Tester": "Analyst", + "Git commit": "a1b2c3d", + "Windows version": "Windows 11 23H2", + "Excel version/build": "Microsoft Excel 2405 Build 17628", + "Add-in manifest": "plugin/manifest.xml", + "Backend command": "python -m uvicorn backend.main:app", + "Plugin command": "npm run dev-http -- --host 127.0.0.1", + "Worker command": "npm --prefix workbook-worker test", + "Test fixture set": "backend/benchmarks/fixtures/workbooks/manifest.json", + "Evidence bundle path": str(base_dir), + "Screenshot/video directory": str(artifacts["screenshots"].parent), + "Excel prompt/log evidence": str(artifacts["excel_log"]), + "Preflight trust bundle ID": str(artifacts["preflight_bundle"]), + "Release owner signoff": "Analyst", + "Overall result": "Pass", + } + metadata_rows = "\n".join(f"| {field} | {metadata_values[field]} |" for field in trust.SMOKE_REQUIRED_METADATA_FIELDS) + return f"""# Windows Excel Manual Smoke Checklist + +## Run Metadata + +| Field | Value | +| --- | --- | +{metadata_rows} + +## Automated Preflight + +| Gate | Command | Result | Evidence / Notes | +| --- | --- | --- | --- | +| Full stack preflight | `cmd` | Pass | evidence/preflight.log | +| Preflight trust bundle snapshot | `cmd` | Pass | evidence/preflight-trust-bundle.json | + +## Required Evidence Coverage + +| ID | Evidence Area | Required Capture | Status | Evidence / Notes | +| --- | --- | --- | --- | --- | +| EXCEL-ENV | Excel Desktop environment | Metadata captured | Pass | evidence/env.png | + +## Fixture Matrix + +| Fixture | Contains | Open | Link | Edit | Sync | Export | Reopen | Save/Reopen | Result | Evidence | +| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | +| Simple `.xlsx` | Values/formulas/styles | Pass | Pass | Pass | Pass | Pass | Pass | Pass | Pass | evidence/simple.mp4 | + +## Sideload And Link + +| Step | Expected Result | Status | Evidence / Notes | +| --- | --- | --- | --- | +| Run `npm run sideload` from `plugin` | Add-in sideloads | Pass | evidence/sideload.png | + +## Browser Edit To Excel Sync + +| Patch Type | Action | Expected Result | Status | Evidence / Notes | +| --- | --- | --- | --- | --- | +| Value | Edit one value | Excel receives value | Pass | evidence/sync.mp4 | + +## Host Drift And Reconcile + +| Step | Expected Result | Status | Evidence / Notes | +| --- | --- | --- | --- | +| Make an external Excel edit | Browser detects host drift | Pass | evidence/drift.mp4 | + +## Failure And Recovery + +| Step | Expected Result | Status | Evidence / Notes | +| --- | --- | --- | --- | +| Attempt write to protected sheet | Sync pauses | Blocked | evidence/protected.png | + +## Export Fidelity + +| Step | Expected Result | Status | Evidence / Notes | +| --- | --- | --- | --- | +| Export preserved workbook | Primary export uses preserved OOXML | Pass | evidence/export.xlsx | + +## Blockers And Follow-Up + +| ID | Severity | Area | Description | Owner | Target Fix | +| --- | --- | --- | --- | --- | --- | +| | | | | | | +""" diff --git a/backend/tests/integration/test_postgres_migration_backfill_contract.py b/backend/tests/integration/test_postgres_migration_backfill_contract.py new file mode 100644 index 0000000..d15344c --- /dev/null +++ b/backend/tests/integration/test_postgres_migration_backfill_contract.py @@ -0,0 +1,29 @@ +"""Opt-in live Postgres migration/backfill validation. + +Set POSTGRES_VALIDATION_DATABASE_URL to run against a disposable validation +database. The test creates and drops an isolated schema inside that database. +""" + +from __future__ import annotations + +import asyncio +import os + +import pytest + +from backend.storage.postgres_migration_validation import validate_postgres_migration_backfill +from backend.storage.postgres import POSTGRES_TYPED_PROJECTION_TABLES + + +@pytest.mark.skipif( + not os.getenv("POSTGRES_VALIDATION_DATABASE_URL"), + reason="POSTGRES_VALIDATION_DATABASE_URL is required for live Postgres validation.", +) +def test_postgres_typed_projection_migrations_backfill_against_live_database(): + result = asyncio.run(validate_postgres_migration_backfill(os.environ["POSTGRES_VALIDATION_DATABASE_URL"])) + + assert set(result.typed_table_counts) == set(POSTGRES_TYPED_PROJECTION_TABLES) + assert all(count >= 1 for count in result.typed_table_counts.values()) + assert result.reader_checks["v7DefaultReadinessReports"] >= 1 + assert result.reader_checks["localReplayArtifacts"] >= 1 + assert result.reader_checks["localReplayBaselines"] >= 1 diff --git a/backend/tests/integration/test_postgres_release_rehearsal_pg16.py b/backend/tests/integration/test_postgres_release_rehearsal_pg16.py new file mode 100644 index 0000000..b897d3c --- /dev/null +++ b/backend/tests/integration/test_postgres_release_rehearsal_pg16.py @@ -0,0 +1,260 @@ +"""Opt-in, local-only PostgreSQL 16 backup/restore rehearsal. + +This gate intentionally requires an explicit environment opt-in and an already +present Docker image. It never pulls an image or consumes hosted CI minutes. +""" + +from __future__ import annotations + +import asyncio +import hashlib +import os +import shutil +import subprocess +import time +import uuid +from pathlib import Path +from typing import cast + +import pytest + +from backend.operations.postgres_release_rehearsal import ( + AsyncpgRehearsalDatabase, + ToolVersions, + run_postgres_release_rehearsal, +) + + +pytestmark = pytest.mark.skipif( + os.getenv("RUN_POSTGRES_RELEASE_REHEARSAL_DOCKER") != "1", + reason="Set RUN_POSTGRES_RELEASE_REHEARSAL_DOCKER=1 for the local PG16 rehearsal.", +) + +_IMAGE = "postgres:16-alpine" +_PASSWORD = "disposable-wave3c-only" + + +def _docker(args: list[str], *, timeout: int = 30) -> subprocess.CompletedProcess[bytes]: + return subprocess.run( # noqa: S603 -- explicit local Docker integration gate + ["docker", *args], # noqa: S607 -- resolved by the opt-in preflight + check=False, + timeout=timeout, + stdout=subprocess.PIPE, + stderr=subprocess.PIPE, + ) + + +def _require_local_docker_image() -> None: + if shutil.which("docker") is None: + pytest.skip("Docker is unavailable") + info = _docker(["info", "--format", "{{.ServerVersion}}"]) + if info.returncode != 0: + pytest.skip("Docker daemon is unavailable") + image = _docker(["image", "inspect", _IMAGE, "--format", "{{.Id}}"]) + if image.returncode != 0: + pytest.skip(f"{_IMAGE} is not present locally; this gate never pulls images") + + +class _DockerPostgresTools: + def __init__(self, container: str) -> None: + self._container = container + + def versions(self, *, timeout_seconds: int) -> ToolVersions: + versions: dict[str, str] = {} + digests: dict[str, str] = {} + for tool in ("pg_dump", "pg_restore"): + version = _docker( + ["exec", self._container, tool, "--version"], + timeout=timeout_seconds, + ) + assert version.returncode == 0 + versions[tool] = version.stdout.decode().strip().rsplit(" ", 1)[1] + resolved = _docker( + ["exec", self._container, "which", tool], + timeout=timeout_seconds, + ) + assert resolved.returncode == 0 + tool_path = resolved.stdout.decode().strip() + digest = _docker( + ["exec", self._container, "sha256sum", tool_path], + timeout=timeout_seconds, + ) + assert digest.returncode == 0 + digests[tool] = digest.stdout.decode().split()[0] + return ToolVersions( + pg_dump=versions["pg_dump"], + pg_restore=versions["pg_restore"], + pg_dump_sha256=digests["pg_dump"], + pg_restore_sha256=digests["pg_restore"], + ) + + def dump( + self, + dsn: str, + artifact_path: Path, + *, + timeout_seconds: int, + workspace: Path, + ) -> None: + del dsn, workspace + container_path = "/tmp/source.dump" + dump = _docker( + [ + "exec", + self._container, + "pg_dump", + "-U", + "postgres", + "-d", + "excel_agent_source_wave3c", + "-Fc", + "--no-owner", + "--no-acl", + "--schema=public", + "-f", + container_path, + ], + timeout=timeout_seconds, + ) + assert dump.returncode == 0 + copied = _docker( + ["cp", f"{self._container}:{container_path}", str(artifact_path)], + timeout=timeout_seconds, + ) + assert copied.returncode == 0 + + def restore( + self, + dsn: str, + artifact_path: Path, + *, + expected_sha256: str, + timeout_seconds: int, + workspace: Path, + ) -> None: + del dsn, workspace + assert hashlib.sha256(artifact_path.read_bytes()).hexdigest() == expected_sha256 + container_path = "/tmp/restore.dump" + copied = _docker( + ["cp", str(artifact_path), f"{self._container}:{container_path}"], + timeout=timeout_seconds, + ) + assert copied.returncode == 0 + restored = _docker( + [ + "exec", + self._container, + "pg_restore", + "-U", + "postgres", + "-d", + "excel_agent_rehearsal_wave3c", + "--exit-on-error", + "--no-owner", + "--no-acl", + container_path, + ], + timeout=timeout_seconds, + ) + assert restored.returncode == 0 + + +async def _prepare_databases(base_dsn: str) -> None: + import asyncpg + + conn = await asyncpg.connect(base_dsn + "/postgres") + try: + await conn.execute('CREATE DATABASE "excel_agent_source_wave3c"') + await conn.execute('CREATE DATABASE "excel_agent_rehearsal_wave3c"') + finally: + await conn.close() + + +async def _seed_lifecycle_posture(source_dsn: str) -> None: + import asyncpg + + conn = await asyncpg.connect(source_dsn) + try: + await conn.execute( + "INSERT INTO session_lifecycle " + "(session_id, state, receipt_id, started_at, completed_at, errors, updated_at) " + "VALUES ('active-fixture', 'active', '', 0, 0, '[]'::jsonb, 1), " + "('deleted-fixture', 'verified', 'receipt-fixture', 1, 2, '[]'::jsonb, 2)" + ) + await conn.execute( + "INSERT INTO sessions " + "(session_id, owner_user_id, created_at, updated_at, metadata) " + "VALUES ('active-fixture', 'owner-fixture', 1, 1, '{}'::jsonb)" + ) + await conn.execute( + "INSERT INTO enterprise_admin_audits " + "(audit_id, event_type, payload, created_at) " + "VALUES ('audit-fixture', 'release_rehearsal', " + "'{\"api_key\":\"[REDACTED]\"}'::jsonb, 1)" + ) + finally: + await conn.close() + + +def test_disposable_pg16_release_rehearsal_end_to_end() -> None: + _require_local_docker_image() + container = f"excel-agent-rehearsal-{uuid.uuid4().hex[:12]}" + started = _docker( + [ + "run", + "--name", + container, + "--rm", + "-d", + "-e", + f"POSTGRES_PASSWORD={_PASSWORD}", + "-e", + "POSTGRES_DB=postgres", + "-p", + "127.0.0.1::5432", + _IMAGE, + ], + timeout=30, + ) + assert started.returncode == 0 + try: + ready = False + for _ in range(120): + probe = _docker( + ["exec", container, "pg_isready", "-U", "postgres", "-d", "postgres"], + timeout=5, + ) + if probe.returncode == 0: + ready = True + break + time.sleep(0.25) + assert ready + port_result = _docker(["port", container, "5432/tcp"]) + assert port_result.returncode == 0 + port = int(port_result.stdout.decode().strip().rsplit(":", 1)[1]) + base_dsn = f"postgresql://postgres:{_PASSWORD}@127.0.0.1:{port}" + source_dsn = base_dsn + "/excel_agent_source_wave3c" + restore_dsn = base_dsn + "/excel_agent_rehearsal_wave3c" + asyncio.run(_prepare_databases(base_dsn)) + database = AsyncpgRehearsalDatabase() + database.run_current_migrations(source_dsn, timeout_seconds=60) + asyncio.run(_seed_lifecycle_posture(source_dsn)) + + report = run_postgres_release_rehearsal( + source_dsn=source_dsn, + restore_dsn=restore_dsn, + target_marker="wave3c", + release_sha="a" * 40, + timeout_seconds=180, + database=database, + tools=_DockerPostgresTools(container), + ) + + assert report["status"] == "passed" + assert report["liveDatabase"] is True + checks = cast(dict[str, object], report["checks"]) + assert checks["logicalEquality"] == "verified" + assert checks["lifecycleTombstones"] == "verified" + assert checks["restoreCleanup"] == "verified" + finally: + _docker(["rm", "-f", container], timeout=30) diff --git a/backend/tests/test_admin_bootstrap.py b/backend/tests/test_admin_bootstrap.py new file mode 100644 index 0000000..89fde4e --- /dev/null +++ b/backend/tests/test_admin_bootstrap.py @@ -0,0 +1,58 @@ +from __future__ import annotations + +import json +import stat + +import pytest + +from backend.admin_bootstrap import bootstrap_admin, write_credentials_once +from backend.session_store import SessionStore + + +def test_bootstrap_admin_mints_one_time_key_and_refuses_implicit_rotation(tmp_path, monkeypatch): + monkeypatch.setenv("API_KEY_HMAC_PEPPER", "bootstrap-test-pepper-with-more-than-32-characters") + store = SessionStore(tmp_path / "bootstrap.db") + + credential = bootstrap_admin( + store, + user_id="first-admin", + email="admin@example.com", + name="First Admin", + label="initial", + ) + + assert credential["api_key"].startswith("fai_") + assert store.get_enterprise_user_by_api_key(credential["api_key"])["user_id"] == "first-admin" + with pytest.raises(RuntimeError, match="--rotate"): + bootstrap_admin( + store, + user_id="first-admin", + email="admin@example.com", + name="First Admin", + label="unexpected", + ) + + rotated = bootstrap_admin( + store, + user_id="first-admin", + email="admin@example.com", + name="First Admin", + label="rotated", + rotate=True, + ) + + assert credential["key_id"] in rotated["revoked_key_ids"] + assert store.get_enterprise_user_by_api_key(credential["api_key"]) is None + assert store.get_enterprise_user_by_api_key(rotated["api_key"])["user_id"] == "first-admin" + + +def test_bootstrap_credential_file_is_exclusive_and_private(tmp_path): + output = tmp_path / "secrets" / "bootstrap.json" + payload = {"api_key": "fai_secret", "key_id": "key_1"} + + write_credentials_once(output, payload) + + assert json.loads(output.read_text(encoding="utf-8")) == payload + assert stat.S_IMODE(output.stat().st_mode) == 0o600 + with pytest.raises(FileExistsError): + write_credentials_once(output, payload) diff --git a/backend/tests/test_agent_closure_fail_closed.py b/backend/tests/test_agent_closure_fail_closed.py new file mode 100644 index 0000000..9b5fec9 --- /dev/null +++ b/backend/tests/test_agent_closure_fail_closed.py @@ -0,0 +1,633 @@ +from __future__ import annotations + +import asyncio +import builtins +import sys +from types import SimpleNamespace +from typing import Any + +import pytest +from fastapi import HTTPException + +from backend.agents import pipeline +from backend.api.routes import agent_runs +from backend.api.routes import experiments +from backend.agents.react_engine import ReActEngine +from backend.agents.tools import ToolCall, execute_tool_call +from backend.documents import rag +from backend.excel import citations +from backend.llm.base import LLMResponse +from backend.llm.base import LLMMessage +from backend.llm.providers.openai import OpenAIProvider +from backend.services.agent_runner_v3 import _idempotency_key + + +def _response(payload: str) -> LLMResponse: + return LLMResponse(content=payload, model="local-test", provider="test", usage={}) + + +def test_pipeline_json_decoder_rejects_non_object_payloads() -> None: + with pytest.raises(ValueError, match="JSON object"): + pipeline._extract_json('[{"operations": []}]') + + +def test_react_provider_failure_never_promotes_partial_operations() -> None: + calls = 0 + + async def generate(_system: str, _user: str) -> LLMResponse: + nonlocal calls + calls += 1 + if calls == 1: + return _response( + '{"step_type":"act","operations":[{"type":"write_cell","range":"A1","value":1}]}' + ) + raise RuntimeError("provider unavailable") + + trace = asyncio.run(ReActEngine(generate, validator=lambda _ops, _ctx: {"errors": []}).execute("write A1")) + + assert trace.success is False + assert trace.failure_code == "provider_error" + assert trace.total_operations + assert trace.partial_operations is True + + +def test_react_validator_exception_is_a_terminal_failure() -> None: + async def generate(_system: str, _user: str) -> LLMResponse: + return _response( + '{"step_type":"act","operations":[{"type":"write_cell","range":"A1","value":1}]}' + ) + + def broken_validator(_operations: list[dict], _context: dict | None) -> dict: + raise RuntimeError("validator unavailable") + + trace = asyncio.run(ReActEngine(generate, validator=broken_validator).execute("write A1")) + + assert trace.success is False + assert trace.failure_code == "validator_error" + assert trace.total_operations == [] + + +def test_react_requires_explicit_completion_before_success() -> None: + responses = iter( + [ + _response('{"step_type":"act","operations":[{"type":"write_cell","range":"A1","value":1}]}'), + _response('{"step_type":"complete","summary":"done"}'), + ] + ) + + async def generate(_system: str, _user: str) -> LLMResponse: + return next(responses) + + trace = asyncio.run( + ReActEngine(generate, validator=lambda _ops, _ctx: {"errors": []}).execute("write A1") + ) + + assert trace.success is True + assert trace.completed is True + assert trace.failure_code == "" + + +def test_react_executes_tool_calls_and_records_observations() -> None: + responses = iter( + [ + _response( + '{"step_type":"act","tool_calls":[{"id":"call-1","tool":"write_cells",' + '"args":{"cells":[{"cell":"B2","value":42}]}}]}' + ), + _response('{"step_type":"complete","summary":"done"}'), + ] + ) + + async def generate(_system: str, _user: str) -> LLMResponse: + return next(responses) + + trace = asyncio.run( + ReActEngine( + generate, + validator=lambda _ops, _ctx: {"errors": []}, + tool_executor=execute_tool_call, + ).execute("write B2", tool_context={"cell_snapshot": {"B2": {"value": 1}}}) + ) + + assert trace.success is True + assert trace.total_operations[0]["range"] == "B2" + assert trace.steps[0].tool_results[0]["success"] is True + assert trace.steps[0].tool_results[0]["call_id"] == "call-1" + + +def test_react_rejects_tool_operations_that_exceed_the_step_budget() -> None: + responses = iter( + [ + _response( + '{"step_type":"act","tool_calls":[{"id":"call-1","tool":"bulk",' + '"args":{}}]}' + ), + _response('{"step_type":"complete","summary":"done"}'), + ] + ) + + async def generate(_system: str, _user: str) -> LLMResponse: + return next(responses) + + async def oversized_tool(_call: ToolCall, _context: dict[str, Any]) -> SimpleNamespace: + operations = [ + {"type": "write_cell", "range": f"A{row}", "value": row} + for row in range(1, 32) + ] + return SimpleNamespace(success=True, result={"operations": operations}, error=None) + + trace = asyncio.run( + ReActEngine( + generate, + validator=lambda _ops, _ctx: {"errors": []}, + tool_executor=oversized_tool, + max_operations_per_step=30, + ).execute("write a bounded batch") + ) + + assert trace.success is False + assert trace.failure_code == "operation_limit_exceeded" + assert trace.total_operations == [] + + +def test_read_cells_tool_is_bounded_to_supplied_snapshot() -> None: + result = asyncio.run( + execute_tool_call( + ToolCall(tool_name="read_cells", arguments={"range": "A1:B2"}), + { + "active_sheet": "Model", + "cell_snapshot_complete": True, + "cell_snapshot": { + "A1": {"value": "Revenue"}, + "B2": {"value": 100}, + "C3": {"value": "outside"}, + }, + }, + ) + ) + + assert result.success is True + assert result.result["source"] == "workbook_context_snapshot" # type: ignore[index] + assert set(result.result["cells"]) == {"Model!A1", "Model!B2"} # type: ignore[index] + + +def test_tool_result_with_error_is_never_reported_as_success() -> None: + result = asyncio.run(execute_tool_call(ToolCall(tool_name="read_cells", arguments={"range": "A1"}), {})) + + assert result.success is False + assert result.error == "Workbook context has no cell snapshot" + + +def test_write_cells_tool_rejects_partially_invalid_batches() -> None: + result = asyncio.run( + execute_tool_call( + ToolCall( + tool_name="write_cells", + arguments={ + "cells": [ + {"cell": "A1", "value": 1}, + {"cell": "not-a-cell", "value": 2}, + ] + }, + ) + ) + ) + + assert result.success is False + assert "invalid cell" in (result.error or "").lower() + + +@pytest.mark.parametrize("tool_name", ["validate_range", "lookup_template"]) +def test_tool_dependency_import_failure_is_fail_closed( + tool_name: str, + monkeypatch: pytest.MonkeyPatch, +) -> None: + blocked_module = ( + "backend.excel.formula_parser" + if tool_name == "validate_range" + else "backend.templates.composer" + ) + original_import = builtins.__import__ + + def blocked_import(name: str, *args: Any, **kwargs: Any) -> Any: + if name == blocked_module: + raise ImportError(f"blocked {name}") + return original_import(name, *args, **kwargs) + + monkeypatch.setattr(builtins, "__import__", blocked_import) + arguments: dict[str, Any] = ( + {"operations": []} + if tool_name == "validate_range" + else {"template_id": "dcf_model"} + ) + + result = asyncio.run(execute_tool_call(ToolCall(tool_name=tool_name, arguments=arguments))) + + assert result.success is False + assert result.error + + +def test_get_cell_ranges_tool_requires_a_real_target() -> None: + result = asyncio.run( + execute_tool_call(ToolCall(tool_name="get_cell_ranges", arguments={})) + ) + + assert result.success is False + assert "range" in (result.error or "").lower() + + +def test_agent_run_idempotency_ignores_runtime_timestamps_and_ids() -> None: + request = { + "session_id": "session-1", + "prompt": "Build a DCF", + "provider": "ollama", + "model": "approved-local-model", + "accuracy_mode": "deep", + "approval_policy": "checkpoint_high_risk", + } + first = _idempotency_key({"runId": "run-a", "createdAt": 1.0}, request) + second = _idempotency_key({"runId": "run-b", "createdAt": 99.0}, request) + + assert first == second + + +def test_agent_step_retry_rejects_malformed_durable_operations() -> None: + artifacts = {"v3_plan": {"operations": ["not-an-operation"]}} + + with pytest.raises(ValueError, match=r"operation\[0\]"): + agent_runs._recompute_v7_simulation(artifacts, {"sessionId": "session-1"}) + + +def test_agent_step_retry_rejects_empty_simulation_input() -> None: + payload: dict[str, Any] = {"artifacts": {"v3_plan": {"operations": []}}} + + with pytest.raises(ValueError, match="no executable operations"): + agent_runs._artifact_for_recomputed_step(payload, "simulate") + + +def test_openai_cloud_request_disables_provider_storage(monkeypatch: pytest.MonkeyPatch) -> None: + captured: dict = {} + + class Completions: + async def create(self, **kwargs): + captured.update(kwargs) + return SimpleNamespace( + choices=[SimpleNamespace(message=SimpleNamespace(content="{}"))], + usage=SimpleNamespace(prompt_tokens=1, completion_tokens=1), + ) + + class AsyncOpenAI: + def __init__(self, **_kwargs): + self.chat = SimpleNamespace(completions=Completions()) + + monkeypatch.setitem(sys.modules, "openai", SimpleNamespace(AsyncOpenAI=AsyncOpenAI)) + + response = asyncio.run( + OpenAIProvider(api_key="test-key").generate( + [LLMMessage(role="user", content="Return JSON")], + json_mode=True, + ) + ) + + assert response.content == "{}" + assert captured["store"] is False + + +def test_experimental_routes_are_unavailable_in_production( + monkeypatch: pytest.MonkeyPatch, +) -> None: + monkeypatch.setattr("backend.config.PRODUCTION_MODE", True) + monkeypatch.setattr("backend.config.EXPERIMENTAL_ROUTES_ENABLED", True, raising=False) + + with pytest.raises(HTTPException) as exc_info: + experiments.require_experimental_routes_enabled() + + assert exc_info.value.status_code == 404 + + +def test_rag_pipeline_is_unavailable_in_production(monkeypatch: pytest.MonkeyPatch) -> None: + monkeypatch.setattr("backend.config.PRODUCTION_MODE", True) + monkeypatch.setattr("backend.config.RAG_ENABLED", True, raising=False) + + with pytest.raises(rag.RAGDisabledError): + rag.get_rag_pipeline() + + +def test_rag_pipeline_evicts_session_owned_chunks() -> None: + pipeline = rag.RAGPipeline() + pipeline._available = False + pipeline.index_document("doc-a", "Alpha", {"session_id": "session-a"}) + pipeline.index_document("doc-b", "Beta", {"session_id": "session-b"}) + + receipt = pipeline.purge_session("session-a") + + assert receipt == {"documents_evicted": 1, "chunks_evicted": 1} + assert pipeline.get_stats()["documents"] == 1 + + +def test_planner_parse_failure_is_not_forwarded_to_executor(monkeypatch: pytest.MonkeyPatch) -> None: + async def generate(**_kwargs) -> LLMResponse: + return _response("not json") + + monkeypatch.setattr(pipeline.router, "generate", generate) + monkeypatch.setattr(pipeline, "classify_complexity", lambda _prompt: pipeline.Complexity.MEDIUM) + state = pipeline.AgentState( + user_prompt="Build a model", + workbook_context={}, + provider_override=None, + model_override=None, + mode="action", + plan=None, + plan_summary="", + operations=[], + execution_summary="", + programmatic_issues=[], + llm_verification=None, + is_verified=False, + retry_count=0, + final_operations=[], + final_summary="", + metadata={}, + error=None, + on_progress=None, + conversation_history=[], + ) + + result = asyncio.run(pipeline.run_planner(state)) + + assert result["error"] + assert result["plan"] is None + + +def test_verifier_parse_failure_is_fail_closed(monkeypatch: pytest.MonkeyPatch) -> None: + async def generate(**_kwargs) -> LLMResponse: + return _response("not json") + + monkeypatch.setattr(pipeline.router, "generate", generate) + monkeypatch.setattr(pipeline, "classify_complexity", lambda _prompt: pipeline.Complexity.MEDIUM) + monkeypatch.setattr(pipeline, "validate_operations", lambda *_args, **_kwargs: SimpleNamespace(is_valid=True, issues=[])) + monkeypatch.setattr(pipeline, "run_advanced_validation", lambda *_args, **_kwargs: []) + monkeypatch.setattr(pipeline, "detect_chart_opportunities", lambda *_args, **_kwargs: []) + monkeypatch.setattr(pipeline, "get_syntax_agent", lambda: SimpleNamespace(validate_operations=lambda _ops: [])) + monkeypatch.setattr(pipeline, "get_logic_agent", lambda: SimpleNamespace(verify=lambda _ops, _ctx: [])) + monkeypatch.setattr(pipeline, "get_performance_agent", lambda: SimpleNamespace(optimize=lambda _ops: [])) + monkeypatch.setattr(pipeline, "get_qa_agent", lambda: SimpleNamespace(review=lambda _ops, _ctx: [])) + state = pipeline.AgentState( + user_prompt="Build a model", + workbook_context={}, + provider_override=None, + model_override=None, + mode="action", + plan=[{"step": "write"}], + plan_summary="", + operations=[{"type": "write_cell", "range": "A1", "value": 1}], + execution_summary="", + programmatic_issues=[], + llm_verification=None, + is_verified=False, + retry_count=0, + final_operations=[], + final_summary="", + metadata={}, + error=None, + on_progress=None, + conversation_history=[], + ) + + result = asyncio.run(pipeline.run_verifier(state)) + + assert result["is_verified"] is False + assert result["error"] == "Verifier returned invalid JSON" + + +def test_pipeline_retry_clears_only_transient_verifier_error( + monkeypatch: pytest.MonkeyPatch, +) -> None: + executor_calls = 0 + validation_calls = 0 + + async def generate(**_kwargs) -> LLMResponse: + nonlocal executor_calls + executor_calls += 1 + return _response( + '{"operations":[{"type":"write_cell","range":"A1","value":1}],"summary":"done"}' + ) + + def validate(*_args, **_kwargs): + nonlocal validation_calls + validation_calls += 1 + if validation_calls == 1: + issue = SimpleNamespace( + cell="A1", + issue_type="temporary", + description="retry", + suggestion="retry", + ) + return SimpleNamespace(is_valid=False, issues=[issue]) + return SimpleNamespace(is_valid=True, issues=[]) + + monkeypatch.setattr(pipeline.router, "generate", generate) + monkeypatch.setattr(pipeline, "classify_complexity", lambda _prompt: pipeline.Complexity.SIMPLE) + monkeypatch.setattr(pipeline, "try_template_route", lambda *_args, **_kwargs: None) + monkeypatch.setattr(pipeline, "validate_operations", validate) + monkeypatch.setattr(pipeline, "run_advanced_validation", lambda _ops: []) + monkeypatch.setattr(pipeline, "detect_chart_opportunities", lambda *_args: []) + monkeypatch.setattr(pipeline, "get_syntax_agent", lambda: SimpleNamespace(validate_operations=lambda _ops: [])) + monkeypatch.setattr(pipeline, "get_logic_agent", lambda: SimpleNamespace(verify=lambda _ops, _ctx: [])) + monkeypatch.setattr(pipeline, "get_performance_agent", lambda: SimpleNamespace(optimize=lambda _ops: [])) + monkeypatch.setattr(pipeline, "get_qa_agent", lambda: SimpleNamespace(review=lambda _ops, _ctx: [])) + monkeypatch.setattr(pipeline, "post_process_operations", lambda operations: operations) + + result = asyncio.run(pipeline.run_pipeline("Perform a custom write")) + + assert result["error"] is None + assert result["metadata"]["verified"] is True + assert result["metadata"]["retry_count"] == 1 + assert len(result["operations"]) == 1 + assert executor_calls == 2 + + +class _AgentRunRecoveryStore: + def __init__(self) -> None: + operation = { + "id": "restore-1", + "type": "rollback_restore", + "sheet": "Sheet1", + "range": "A1", + "values": [[1]], + "formulas": [[""]], + "numberFormats": [["General"]], + } + operation_digest = agent_runs.hashlib.sha256( + agent_runs.json.dumps([operation], sort_keys=True, separators=(",", ":"), default=str).encode("utf-8") + ).hexdigest() + recovery_policy_hash = agent_runs.hashlib.sha256( + agent_runs.json.dumps( + { + "recoveryKind": "rollback", + "operationRegistryHash": operation_digest, + "requiresCompletePreHostRead": True, + "requiresCompletePostHostRead": True, + }, + sort_keys=True, + separators=(",", ":"), + ).encode("utf-8") + ).hexdigest() + self.run: dict[str, Any] = { + "runId": "run-1", + "sessionId": "session-1", + "planId": "plan-1", + "metadata": { + "recoveryAttempts": [ + { + "attemptId": "recovery-1", + "recoveryKind": "rollback", + "status": "rollback_pending", + "expectedOperationIds": ["restore-1"], + "approvedOperations": [operation], + "operationDigest": operation_digest, + "operationRegistryHash": operation_digest, + "recoveryPolicyHash": recovery_policy_hash, + } + ] + }, + "logs": [], + } + self.audits: list[dict] = [] + + def get_agent_run(self, run_id: str) -> dict | None: + return self.run if run_id == "run-1" else None + + def save_agent_run(self, payload: dict) -> None: + self.run = payload + + def record_audit_event(self, **kwargs) -> str: + self.audits.append(kwargs) + return "audit-1" + + +def test_agent_run_recovery_result_requires_explicit_office_success( + monkeypatch: pytest.MonkeyPatch, +) -> None: + store = _AgentRunRecoveryStore() + monkeypatch.setattr(agent_runs, "get_session_store", lambda: store) + request = agent_runs.AgentRunRecoveryResultRequest( + recovery_kind="rollback", + attempt_id="recovery-1", + status="rolled_back", + office_apply_result={ + "applied": 1, + "results": [{"operationId": "restore-1", "status": "success"}], + }, + pre_apply_workbook_state={ + "active_sheet": "Sheet1", + "cell_snapshot": { + "Sheet1!A1": { + "sheet": "Sheet1", + "address": "A1", + "value": 42, + "formula": None, + "format": "General", + } + }, + "rollback_snapshot": {"complete": True, "errors": []}, + }, + pre_apply_workbook_fingerprint="pre-fingerprint", + pre_host_snapshot={"complete": True, "truncated": False, "errors": [], "fingerprint": "pre-host"}, + post_apply_workbook_state={ + "active_sheet": "Sheet1", + "cell_snapshot": { + "Sheet1!A1": { + "sheet": "Sheet1", + "address": "A1", + "value": 1, + "formula": None, + "format": "General", + } + }, + "rollback_snapshot": {"complete": True, "errors": []}, + }, + post_apply_workbook_fingerprint="post-fingerprint", + post_host_snapshot={"complete": True, "truncated": False, "errors": [], "fingerprint": "post-host"}, + ) + + response = asyncio.run(agent_runs.record_agent_run_recovery_result("run-1", request)) + + assert response.status == "rolled_back" + assert response.verification["status"] == "verified" + assert store.run["metadata"]["recoveryAttempts"][0]["status"] == "rolled_back" + assert store.audits[0]["event_type"] == "agent_run_rollback_rolled_back" + + +def test_agent_run_recovery_rejects_incomplete_success_ack(monkeypatch: pytest.MonkeyPatch) -> None: + store = _AgentRunRecoveryStore() + monkeypatch.setattr(agent_runs, "get_session_store", lambda: store) + request = agent_runs.AgentRunRecoveryResultRequest( + recovery_kind="rollback", + attempt_id="recovery-1", + status="rolled_back", + office_apply_result={"applied": 0, "results": []}, + ) + + with pytest.raises(HTTPException) as exc_info: + asyncio.run(agent_runs.record_agent_run_recovery_result("run-1", request)) + + assert exc_info.value.status_code == 409 + assert store.run["metadata"]["recoveryAttempts"][0]["status"] == "rollback_pending" + + +def test_agent_run_recovery_rejects_success_ids_without_exact_post_target_proof(monkeypatch: pytest.MonkeyPatch) -> None: + store = _AgentRunRecoveryStore() + monkeypatch.setattr(agent_runs, "get_session_store", lambda: store) + request = agent_runs.AgentRunRecoveryResultRequest( + recovery_kind="rollback", + attempt_id="recovery-1", + status="rolled_back", + office_apply_result={ + "applied": 1, + "results": [{"operationId": "restore-1", "status": "success"}], + }, + pre_apply_workbook_state={ + "active_sheet": "Sheet1", + "cell_snapshot": {"Sheet1!A1": {"sheet": "Sheet1", "address": "A1", "value": 42}}, + "rollback_snapshot": {"complete": True, "errors": []}, + }, + pre_apply_workbook_fingerprint="pre-fingerprint", + pre_host_snapshot={"complete": True, "truncated": False, "errors": [], "fingerprint": "pre-host"}, + post_apply_workbook_state={ + "active_sheet": "Sheet1", + "cell_snapshot": {"Sheet1!XFD1048576": {"sheet": "Sheet1", "address": "XFD1048576", "value": 1}}, + "rollback_snapshot": {"complete": True, "errors": []}, + }, + post_apply_workbook_fingerprint="post-fingerprint", + post_host_snapshot={"complete": True, "truncated": False, "errors": [], "fingerprint": "post-host"}, + ) + + with pytest.raises(HTTPException) as exc_info: + asyncio.run(agent_runs.record_agent_run_recovery_result("run-1", request)) + + assert exc_info.value.status_code == 409 + assert store.run["metadata"]["recoveryAttempts"][0]["status"] == "rollback_pending" + + +def test_citation_tracker_is_session_scoped_and_purgeable() -> None: + citations.purge_citation_tracker("session-a") + citations.purge_citation_tracker("session-b") + + session_a = citations.get_citation_tracker("session-a") + session_a.track_from_document( + cell="A1", + doc_id="private-a.pdf", + source_text="session A private financial data", + ) + + session_b = citations.get_citation_tracker("session-b") + assert citations.get_citation_tracker("session-a") is session_a + assert len(session_a.generate_cell_comments([])) == 1 + assert session_b.generate_cell_comments([]) == [] + assert session_b.get_index().citations == [] + assert session_b.get_index().documents == {} + + purge_result = citations.purge_citation_tracker("session-a") + assert purge_result == {"trackers_evicted": 1, "citations_evicted": 1, "documents_evicted": 0} + assert citations.get_citation_tracker("session-a").get_index().citations == [] diff --git a/backend/tests/test_agent_run_control_plane.py b/backend/tests/test_agent_run_control_plane.py new file mode 100644 index 0000000..ee3e07e --- /dev/null +++ b/backend/tests/test_agent_run_control_plane.py @@ -0,0 +1,485 @@ +from __future__ import annotations + +import json +import threading +import time +from pathlib import Path + +import pytest +from fastapi.testclient import TestClient + +from backend.api.security.ownership import OwnershipNotFoundError, ownership_actor_scope +from backend.services.agent_run_control import ( + build_egress_preview, + build_egress_receipt, + egress_request_digest, + redact_egress_payload, +) +from backend.session_store import SessionStore + + +def _run(session_id: str = "session-control", run_id: str = "run-control") -> dict: + return { + "runId": run_id, + "sessionId": session_id, + "prompt": "Build a DCF", + "status": "paused", + "metadata": {}, + "logs": [], + } + + +def _store(tmp_path: Path) -> SessionStore: + store = SessionStore(tmp_path / "sessions.db") + store.get_or_create_session("session-control") + store.save_agent_run(_run()) + return store + + +def _append( + store: SessionStore, + *, + expected: int, + key: str, + status: str = "ready", + event_type: str = "run_resumed", +) -> dict: + payload = store.get_agent_run("run-control") + assert payload is not None + payload["status"] = status + return store.append_agent_run_event( + "run-control", + expected_sequence=expected, + event_type=event_type, + event_payload={"toStatus": status}, + updated_run=payload, + idempotency_key=key, + ) + + +def test_agent_run_compare_and_append_allows_one_concurrent_winner(tmp_path: Path) -> None: + store = _store(tmp_path) + barrier = threading.Barrier(3) + outcomes: list[str] = [] + + def compete(key: str, status: str) -> None: + payload = _run() + payload["status"] = status + barrier.wait() + result = store.append_agent_run_event( + "run-control", + expected_sequence=0, + event_type=f"run_{status}", + event_payload={"toStatus": status}, + updated_run=payload, + idempotency_key=key, + ) + outcomes.append(str(result["outcome"])) + + threads = [ + threading.Thread(target=compete, args=("resume-a", "ready")), + threading.Thread(target=compete, args=("cancel-b", "cancelled")), + ] + for thread in threads: + thread.start() + barrier.wait() + for thread in threads: + thread.join(timeout=2) + + assert sorted(outcomes) == ["appended", "sequence_conflict"] + replay = store.list_agent_run_events("run-control", after_sequence=0, limit=10) + assert [event["sequence"] for event in replay["events"]] == [1] + + +def test_duplicate_resume_key_replays_without_duplicate_event(tmp_path: Path) -> None: + store = _store(tmp_path) + + first = _append(store, expected=0, key="resume-once") + duplicate = _append(store, expected=0, key="resume-once") + + assert first["outcome"] == "appended" + assert duplicate["outcome"] == "replayed" + assert duplicate["event"]["eventId"] == first["event"]["eventId"] + assert duplicate["run"]["metadata"]["controlPlane"]["eventSequence"] == 1 + assert len(store.list_agent_run_events("run-control", limit=10)["events"]) == 1 + + +def test_event_sequence_cursor_and_replay_window_are_monotonic_and_bounded(tmp_path: Path) -> None: + store = _store(tmp_path) + for sequence in range(5): + result = _append(store, expected=sequence, key=f"event-{sequence}") + assert result["outcome"] == "appended" + + first_page = store.list_agent_run_events("run-control", after_sequence=2, limit=2) + second_page = store.list_agent_run_events( + "run-control", + after_sequence=int(first_page["nextCursor"]), + limit=2, + ) + + assert [event["sequence"] for event in first_page["events"]] == [3, 4] + assert first_page["nextCursor"] == "4" + assert first_page["hasMore"] is True + assert [event["sequence"] for event in second_page["events"]] == [5] + assert second_page["nextCursor"] == "5" + assert second_page["hasMore"] is False + + +def test_cancelled_run_fences_later_resume_even_with_current_sequence(tmp_path: Path) -> None: + store = _store(tmp_path) + cancelled = _append( + store, + expected=0, + key="cancel", + status="cancelled", + event_type="run_cancelled", + ) + + resumed = _append(store, expected=1, key="resume-after-cancel") + + assert cancelled["outcome"] == "appended" + assert resumed["outcome"] == "terminal_fence" + assert store.get_agent_run("run-control")["status"] == "cancelled" # type: ignore[index] + + +def test_session_purge_fence_rejects_new_run_events(tmp_path: Path) -> None: + store = _store(tmp_path) + store.set_session_lifecycle_state( + "session-control", + "purging", + receipt_id="receipt-control", + started_at=time.time(), + ) + + with pytest.raises(OwnershipNotFoundError): + store.append_agent_run_event( + "run-control", + expected_sequence=0, + event_type="run_resumed", + event_payload={"toStatus": "ready"}, + updated_run={**_run(), "status": "ready"}, + idempotency_key="too-late", + ) + + +def test_egress_approval_is_digest_bound_expiring_and_one_use(tmp_path: Path) -> None: + store = _store(tmp_path) + request_digest = "a" * 64 + preview = build_egress_preview( + session_id="session-control", + provider="openai", + model="gpt-test", + request_digest=request_digest, + data_classes=["user_prompt", "workbook_values"], + workbook_ranges=["Model!A1:B9"], + redactions=["workbook_context.api_key"], + estimated_input_tokens=120, + now=100.0, + ttl_seconds=60, + ) + store.save_agent_egress_preview("session-control", preview) + + mismatch = store.consume_agent_egress_approval( + "session-control", + approval_id=preview["approvalId"], + approved_digest="b" * 64, + request_digest=request_digest, + consumption_id="attempt-mismatch", + now=120.0, + ) + consumed = store.consume_agent_egress_approval( + "session-control", + approval_id=preview["approvalId"], + approved_digest=preview["approvalDigest"], + request_digest=request_digest, + consumption_id="attempt-1", + now=120.0, + ) + reused = store.consume_agent_egress_approval( + "session-control", + approval_id=preview["approvalId"], + approved_digest=preview["approvalDigest"], + request_digest=request_digest, + consumption_id="attempt-2", + now=121.0, + ) + + assert mismatch["outcome"] == "digest_mismatch" + assert consumed["outcome"] == "consumed" + assert reused["outcome"] == "already_consumed" + + expired = build_egress_preview( + session_id="session-control", + provider="openai", + model="gpt-test", + request_digest=request_digest, + data_classes=["user_prompt"], + now=200.0, + ttl_seconds=30, + ) + store.save_agent_egress_preview("session-control", expired) + result = store.consume_agent_egress_approval( + "session-control", + approval_id=expired["approvalId"], + approved_digest=expired["approvalDigest"], + request_digest=request_digest, + consumption_id="attempt-expired", + now=231.0, + ) + assert result["outcome"] == "expired" + + +def test_concurrent_egress_consumers_admit_exactly_one(tmp_path: Path) -> None: + store = _store(tmp_path) + preview = build_egress_preview( + session_id="session-control", + provider="openai", + model="gpt-test", + request_digest="e" * 64, + data_classes=["user_prompt"], + ) + store.save_agent_egress_preview("session-control", preview) + barrier = threading.Barrier(3) + outcomes: list[str] = [] + + def consume(name: str) -> None: + barrier.wait() + result = store.consume_agent_egress_approval( + "session-control", + approval_id=preview["approvalId"], + approved_digest=preview["approvalDigest"], + request_digest="e" * 64, + consumption_id=name, + ) + outcomes.append(str(result["outcome"])) + + threads = [threading.Thread(target=consume, args=(f"consumer-{index}",)) for index in range(2)] + for thread in threads: + thread.start() + barrier.wait() + for thread in threads: + thread.join(timeout=2) + + assert sorted(outcomes) == ["already_consumed", "consumed"] + + +def test_egress_digest_changes_with_material_context_and_receipt_is_fail_closed() -> None: + base = { + "session_id": "session-control", + "prompt": "Build a DCF", + "provider": "openai", + "model": "gpt-test", + "workbook_context": {"Model!A1": 1}, + "documents": [], + } + changed = {**base, "workbook_context": {"Model!A1": 2}} + + assert egress_request_digest(base) != egress_request_digest(changed) + receipt = build_egress_receipt( + provider="openai", + model="gpt-test", + request_digest=egress_request_digest(base), + approval_id="approval-1", + approval_digest="c" * 64, + now=123.0, + ) + assert receipt["fallbackAllowed"] is False + assert receipt["cloudFallbackAllowed"] is False + assert receipt["providerStorageDisabled"] is True + assert receipt["providerStorageControl"] == "store:false" + redacted = redact_egress_payload( + {"api_key": "secret", "nested": {"password": "secret", "value": 42}} + ) + assert redacted == { + "api_key": "[REDACTED]", + "nested": {"password": "[REDACTED]", "value": 42}, + } + + +def test_owner_isolation_hides_run_events_and_egress_preview(tmp_path: Path) -> None: + store = SessionStore(tmp_path / "sessions.db") + owner = {"user_id": "owner-a", "role": "analyst"} + foreign = {"user_id": "owner-b", "role": "analyst"} + with ownership_actor_scope(owner): + store.get_or_create_session("session-control") + store.save_agent_run(_run()) + _append(store, expected=0, key="owner-event") + preview = build_egress_preview( + session_id="session-control", + provider="openai", + model="gpt-test", + request_digest="d" * 64, + data_classes=["user_prompt"], + ) + store.save_agent_egress_preview("session-control", preview) + + with ownership_actor_scope(foreign): + assert store.list_agent_run_events("run-control", limit=10)["events"] == [] + with pytest.raises(OwnershipNotFoundError): + store.consume_agent_egress_approval( + "session-control", + approval_id=preview["approvalId"], + approved_digest=preview["approvalDigest"], + request_digest="d" * 64, + consumption_id="foreign", + ) + + +def test_control_plane_metrics_bound_labels_and_never_accept_owner_ids( + monkeypatch: pytest.MonkeyPatch, +) -> None: + from backend import observability + + observed: list[dict[str, str]] = [] + + class Metric: + def labels(self, **labels): + observed.append(labels) + return self + + def inc(self, _amount=1) -> None: + return None + + monkeypatch.setattr(observability, "PROMETHEUS_AVAILABLE", True) + monkeypatch.setattr(observability, "AGENT_RUN_CONTROL_EVENTS", Metric()) + monkeypatch.setattr(observability, "AGENT_EGRESS_APPROVALS", Metric()) + + observability.record_agent_run_control_event("run-owner-a-secret", "tenant-owner-a") + observability.record_agent_egress_approval("tenant-owner-a", "approval-owner-a") + + assert observed == [ + {"event": "other", "outcome": "rejected"}, + {"scope": "unknown", "outcome": "rejected"}, + ] + + +def test_sse_replay_honors_last_event_id_and_limit(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None: + from backend.api.routes import agent_runs + from backend.main import app + + store = _store(tmp_path) + for sequence in range(3): + _append(store, expected=sequence, key=f"sse-{sequence}") + monkeypatch.setattr(agent_runs, "get_session_store", lambda: store) + + response = TestClient(app).get( + "/api/v1/agent-runs/run-control/events?limit=1", + headers={"Last-Event-ID": "1"}, + ) + + assert response.status_code == 200 + assert response.headers["content-type"].startswith("text/event-stream") + assert "id: 2\n" in response.text + assert "id: 3\n" not in response.text + data_line = next(line for line in response.text.splitlines() if line.startswith("data: ")) + assert json.loads(data_line.removeprefix("data: "))["sequence"] == 2 + + +def test_v4_cloud_route_requires_and_consumes_exact_preview_without_fallback( + tmp_path: Path, + monkeypatch: pytest.MonkeyPatch, +) -> None: + from backend.api.routes import agent_runs, tasks + from backend.main import app + + store = SessionStore(tmp_path / "route-sessions.db") + monkeypatch.setattr(agent_runs, "get_session_store", lambda: store) + monkeypatch.setattr(tasks, "get_session_store", lambda: store) + monkeypatch.setattr( + "backend.api.security.ai_policy._load_enterprise_provider_policy", + lambda: {}, + ) + monkeypatch.setenv("AI_ALLOWED_PROVIDERS", "lmstudio,ollama,anthropic,openai") + monkeypatch.setenv("AI_EXTERNAL_SENDING_ALLOWED", "true") + request_payload = { + "session_id": "session-cloud-route", + "prompt": "Build a deterministic DCF template.", + "provider": "openai", + "model": "gpt-test", + "template_id": "dcf_model", + "template_params": { + "base_revenue": 100, + "wacc": 0.1, + "terminal_growth": 0.025, + }, + "workbook_context": {"workbook_name": "Approval Fixture"}, + } + client = TestClient(app) + + missing = client.post("/api/v1/agent-runs/v4", json=request_payload) + preview_response = client.post( + "/api/v1/agent-runs/v4/egress-preview", + json=request_payload, + ) + assert missing.status_code == 409 + assert missing.json()["detail"]["code"] == "egress_approval_required" + assert preview_response.status_code == 200 + preview = preview_response.json() + + approved_payload = { + **request_payload, + "egress_approval": { + "approval_id": preview["approvalId"], + "approved_digest": preview["approvalDigest"], + }, + } + created = client.post("/api/v1/agent-runs/v4", json=approved_payload) + reused = client.post("/api/v1/agent-runs/v4", json=approved_payload) + + assert created.status_code == 200 + receipt = created.json()["artifacts"]["egress_receipt_v1"] + assert receipt["scope"] == "cloud" + assert receipt["fallbackAllowed"] is False + assert receipt["cloudFallbackAllowed"] is False + assert receipt["providerStorageControl"] == "store:false" + assert receipt["providerCallAttempted"] is False + assert receipt["egressPerformed"] is False + assert created.json()["metadata"]["controlPlane"]["eventSequence"] == 1 + assert reused.status_code == 409 + assert reused.json()["detail"]["code"] == "egress_approval_already_consumed" + + +def test_resume_pause_cancel_routes_use_one_monotonic_event_stream( + tmp_path: Path, + monkeypatch: pytest.MonkeyPatch, +) -> None: + from backend.api.routes import agent_runs + from backend.main import app + + store = _store(tmp_path) + monkeypatch.setattr(agent_runs, "get_session_store", lambda: store) + + async def no_terminal_purge(_session_id: str, _status: str) -> None: + return None + + monkeypatch.setattr( + "backend.services.retention.purge_terminal_session_if_immediate", + no_terminal_purge, + ) + client = TestClient(app) + + resumed = client.post( + "/api/v1/agent-runs/run-control/resume", + headers={"Idempotency-Key": "resume-route-once"}, + ) + duplicate = client.post( + "/api/v1/agent-runs/run-control/resume", + headers={"Idempotency-Key": "resume-route-once"}, + ) + paused = client.post("/api/v1/agent-runs/run-control/pause") + cancelled = client.post("/api/v1/agent-runs/run-control/cancel") + fenced = client.post("/api/v1/agent-runs/run-control/resume") + + assert resumed.status_code == 200 + assert duplicate.status_code == 200 + assert paused.status_code == 200 + assert cancelled.status_code == 200 + assert fenced.status_code == 409 + assert fenced.json()["detail"]["code"] == "agent_run_resume_invalid_state" + replay = store.list_agent_run_events("run-control", limit=10) + assert [event["eventType"] for event in replay["events"]] == [ + "run_resumed", + "run_paused", + "run_cancelled", + ] + assert [event["sequence"] for event in replay["events"]] == [1, 2, 3] diff --git a/backend/tests/test_ai_egress_policy.py b/backend/tests/test_ai_egress_policy.py new file mode 100644 index 0000000..e0a36ca --- /dev/null +++ b/backend/tests/test_ai_egress_policy.py @@ -0,0 +1,244 @@ +from __future__ import annotations + +import asyncio +from types import SimpleNamespace + +import pytest +from fastapi.testclient import TestClient + +from backend.api.security.ai_policy import AiPolicyViolation +from backend.llm.base import BaseLLMProvider, LLMMessage, LLMResponse +from backend.llm.router import Complexity, LLMRouter + + +class _FakeProvider(BaseLLMProvider): + def __init__(self, name: str) -> None: + self.name = name + self.generate_calls = 0 + self.stream_calls = 0 + + async def health_check(self) -> bool: + return True + + async def generate( + self, + messages, + model=None, + temperature=0.1, + max_tokens=4096, + json_mode=False, + ) -> LLMResponse: + self.generate_calls += 1 + return LLMResponse(content=f"response:{self.name}", model=model or "", provider=self.name) + + async def generate_stream( + self, + messages, + model=None, + temperature=0.1, + max_tokens=4096, + json_mode=False, + ): + self.stream_calls += 1 + yield f"stream:{self.name}" + + +def _router_with_fake_providers() -> tuple[LLMRouter, dict[str, _FakeProvider]]: + router = LLMRouter() + providers = {name: _FakeProvider(name) for name in ("lmstudio", "ollama", "anthropic", "openai")} + router.providers = {name: provider for name, provider in providers.items()} + return router, providers + + +def _messages() -> list[LLMMessage]: + return [ + LLMMessage(role="system", content="security regression system prompt"), + LLMMessage(role="user", content="security regression user prompt"), + ] + + +@pytest.fixture(autouse=True) +def _local_only_ai_policy(monkeypatch): + monkeypatch.setenv("AI_ALLOWED_PROVIDERS", "lmstudio,ollama,anthropic,openai") + monkeypatch.setenv("AI_EXTERNAL_SENDING_ALLOWED", "false") + monkeypatch.setenv("AI_EXTERNAL_DOCUMENTS_ALLOWED", "true") + monkeypatch.setattr( + "backend.api.security.ai_policy._load_enterprise_provider_policy", + lambda: {}, + ) + + +def test_providerless_complex_routing_never_crosses_external_boundary(): + router, providers = _router_with_fake_providers() + + response = asyncio.run(router.generate(_messages(), complexity=Complexity.COMPLEX)) + + assert response.provider == "lmstudio" + assert providers["anthropic"].generate_calls == 0 + assert providers["openai"].generate_calls == 0 + assert providers["lmstudio"].generate_calls == 1 + + +def test_providerless_streaming_never_crosses_external_boundary(): + router, providers = _router_with_fake_providers() + + response = asyncio.run(router.generate_stream(_messages(), complexity=Complexity.COMPLEX)) + + assert response.provider == "lmstudio" + assert providers["anthropic"].stream_calls == 0 + assert providers["openai"].stream_calls == 0 + assert providers["lmstudio"].stream_calls == 1 + + +def test_providerless_routing_fails_closed_when_no_route_is_policy_approved(): + router, providers = _router_with_fake_providers() + router.routing_table[Complexity.COMPLEX] = [ + ("anthropic", "cloud-a"), + ("openai", "cloud-b"), + ] + + with pytest.raises(AiPolicyViolation) as exc_info: + asyncio.run(router.generate(_messages(), complexity=Complexity.COMPLEX)) + + assert exc_info.value.code == "no_approved_ai_provider" + assert providers["anthropic"].generate_calls == 0 + assert providers["openai"].generate_calls == 0 + + +def test_explicit_allowed_local_provider_remains_supported(): + router, providers = _router_with_fake_providers() + + response = asyncio.run( + router.generate( + _messages(), + complexity=Complexity.COMPLEX, + provider_override="ollama", + model_override="approved-local-model", + ) + ) + + assert response.provider == "ollama" + assert response.model == "approved-local-model" + assert providers["ollama"].generate_calls == 1 + + +def test_verifier_propagates_the_selected_provider_and_model(monkeypatch): + from backend.agents import pipeline + + captured: dict = {} + + async def fake_generate(**kwargs): + captured.update(kwargs) + return LLMResponse( + content='{"status":"pass","issues":[]}', + model="approved-local-model", + provider="ollama", + ) + + empty_agent = SimpleNamespace( + validate_operations=lambda _operations: [], + verify=lambda _operations, _context: [], + optimize=lambda _operations: [], + review=lambda _operations, _context: [], + ) + monkeypatch.setattr(pipeline, "validate_operations", lambda *_args, **_kwargs: SimpleNamespace(is_valid=True, issues=[])) + monkeypatch.setattr(pipeline, "run_advanced_validation", lambda _operations: []) + monkeypatch.setattr(pipeline, "detect_chart_opportunities", lambda *_args: []) + monkeypatch.setattr(pipeline, "get_syntax_agent", lambda: empty_agent) + monkeypatch.setattr(pipeline, "get_logic_agent", lambda: empty_agent) + monkeypatch.setattr(pipeline, "get_performance_agent", lambda: empty_agent) + monkeypatch.setattr(pipeline, "get_qa_agent", lambda: empty_agent) + monkeypatch.setattr(pipeline, "classify_complexity", lambda _prompt: Complexity.MEDIUM) + monkeypatch.setattr(pipeline.router, "generate", fake_generate) + + state = { + "user_prompt": "Build a complex local-only model", + "workbook_context": {}, + "provider_override": "ollama", + "model_override": "approved-local-model", + "operations": [{"type": "input", "cell": "A1", "value": 1}], + "execution_summary": "", + "metadata": {}, + "on_progress": None, + } + + result = asyncio.run(pipeline.run_verifier(state)) + + assert result["is_verified"] is True + assert captured["provider_override"] == "ollama" + assert captured["model_override"] == "approved-local-model" + + +@pytest.mark.parametrize( + "path", + [ + "/api/v1/documents/doc_policy/plan", + "/api/v1/documents/doc_policy/plan/v2", + ], +) +def test_document_plan_routes_enforce_source_data_policy_before_planning(path, monkeypatch): + from backend.main import app + + class Store: + def get_document(self, document_id: str): + assert document_id == "doc_policy" + return { + "document_id": document_id, + "session_id": "session-policy", + "filename": "sensitive.xlsx", + "summary": {}, + "parsed": {"sensitive": "source data"}, + } + + async def unexpected_plan(**_kwargs): + raise AssertionError("document policy must run before the planner") + + monkeypatch.setenv("AI_EXTERNAL_DOCUMENTS_ALLOWED", "false") + monkeypatch.setattr("backend.api.routes.documents.get_session_store", lambda: Store()) + monkeypatch.setattr("backend.services.operation_planner.build_operation_plan", unexpected_plan) + + response = TestClient(app, raise_server_exceptions=False).post( + path, + json={"session_id": "session-policy", "prompt": "Analyze the uploaded source"}, + ) + + assert response.status_code == 403 + assert response.json()["detail"]["code"] == "external_documents_disabled" + + +@pytest.mark.parametrize( + ("path", "service_getter"), + [ + ("/api/v1/agent-runs", "backend.api.routes.agent_runs.get_agent_run_service"), + ("/api/v1/agent-runs/v3", "backend.services.agent_runner_v3.get_agent_run_v3_service"), + ("/api/v1/agent-runs/v4", "backend.services.agent_runner_v4.get_agent_run_v4_service"), + ], +) +def test_agent_run_routes_reject_disallowed_cloud_provider_before_agent_execution( + path, + service_getter, + monkeypatch, +): + from backend.main import app + + class Store: + pass + + class UnexpectedService: + async def create_run(self, **_kwargs): + raise AssertionError("agent policy must run before agent execution") + + monkeypatch.setattr("backend.api.routes.agent_runs.get_session_store", lambda: Store()) + monkeypatch.setattr(service_getter, lambda: UnexpectedService()) + + response = TestClient(app, raise_server_exceptions=False).post( + path, + json={ + "prompt": "Build a model", + "session_id": "session-policy", + "provider": "openai", + }, + ) + + assert response.status_code == 403 + assert response.json()["detail"]["code"] == "external_sending_disabled" diff --git a/backend/tests/test_apply_state_machine.py b/backend/tests/test_apply_state_machine.py index e673ccf..d4998f4 100644 --- a/backend/tests/test_apply_state_machine.py +++ b/backend/tests/test_apply_state_machine.py @@ -50,7 +50,7 @@ def test_legacy_apply_final_status_shortcut_is_disabled_by_default(): ) assert response.status_code == 400 - assert "final_status shortcut is disabled" in response.json()["detail"] + assert "final_status shortcut cannot prove an Office mutation" in response.json()["detail"] def test_apply_result_cannot_mark_office_errors_as_applied(): diff --git a/backend/tests/test_brain_kernel_contracts.py b/backend/tests/test_brain_kernel_contracts.py index 6d2b32f..57630e0 100644 --- a/backend/tests/test_brain_kernel_contracts.py +++ b/backend/tests/test_brain_kernel_contracts.py @@ -37,7 +37,9 @@ def _seed_brain_plan(metadata: dict | None = None) -> tuple[str, str, WriteCellO summary="Update one KPI cell", metadata={**(metadata or {}), "v2_plan": v2_plan.model_dump(mode="json", by_alias=True)}, ) - get_session_store().save_operation_plan(dump_plan(plan)) + store = get_session_store() + store.get_or_create_session(session_id) + store.save_operation_plan(dump_plan(plan)) return session_id, plan_id, operation @@ -94,6 +96,7 @@ def test_brain_apply_eligibility_is_mandatory_for_v7_metadata_without_client_fla def test_brain_apply_eligibility_binds_simulation_hash_and_allows_matching_apply(): session_id, plan_id, operation = _seed_brain_plan() + source_artifact_ids = ["source_graph_1", "truth_1", "formula_graph_1"] eligibility_response = client.post( "/api/v1/experiments/v7/brain/apply-eligibility", @@ -103,7 +106,7 @@ def test_brain_apply_eligibility_binds_simulation_hash_and_allows_matching_apply "planId": plan_id, "workbookFingerprint": "brain-fingerprint", "simulation": _simulation(operation), - "sourceArtifactIds": ["truth_1", "graph_1"], + "sourceArtifactIds": source_artifact_ids, "rollbackProofClass": "range_restore_operations", }, ) @@ -112,6 +115,7 @@ def test_brain_apply_eligibility_binds_simulation_hash_and_allows_matching_apply assert eligibility["schemaVersion"] == "2026-05-apply-eligibility" assert eligibility["allowed"] is True assert eligibility["operationHash"] + assert eligibility["sourceArtifactIds"] == source_artifact_ids assert eligibility["rollbackProofClass"] == "range_restore_operations" assert eligibility["metadata"]["brainRunId"] == f"brain_{plan_id}" assert eligibility["metadata"]["brainBundleId"] == f"brainbundle_{plan_id}" @@ -121,8 +125,13 @@ def test_brain_apply_eligibility_binds_simulation_hash_and_allows_matching_apply assert [step["type"] for step in run.json()["trace"]] == ["apply_eligibility_created"] bundle = client.get(f"/api/v1/experiments/v7/brain/bundles/{eligibility['metadata']['brainBundleId']}", headers=headers) assert bundle.status_code == 200 - assert bundle.json()["applyEligibilityId"] == eligibility["eligibilityId"] - assert bundle.json()["artifactHashes"]["applyEligibility"] + bundle_payload = bundle.json() + assert bundle_payload["applyEligibilityId"] == eligibility["eligibilityId"] + assert bundle_payload["sourceEvidenceId"] == "source_graph_1" + assert bundle_payload["truthModelId"] == "truth_1" + assert bundle_payload["formulaGraphId"] == "formula_graph_1" + assert bundle_payload["metadata"]["sourceArtifactIds"] == source_artifact_ids + assert bundle_payload["artifactHashes"]["applyEligibility"] listed = client.get(f"/api/v1/experiments/v7/brain/runs?sessionId={session_id}", headers=headers) assert listed.status_code == 200 @@ -155,6 +164,8 @@ def test_brain_apply_eligibility_binds_simulation_hash_and_allows_matching_apply approved_bundle = client.get(f"/api/v1/experiments/v7/brain/bundles/{eligibility['metadata']['brainBundleId']}", headers=headers).json() assert payload["rollback_snapshot_id"] in approved_bundle["rollbackSnapshotIds"] assert approved_bundle["metadata"]["applyAttemptId"] == payload["apply_attempt_id"] + assert approved_bundle["sourceEvidenceId"] == "source_graph_1" + assert approved_bundle["metadata"]["sourceArtifactIds"] == source_artifact_ids office_result = { "applied": 1, @@ -245,6 +256,7 @@ def test_brain_readiness_is_additive_on_stack_readiness(): def test_brain_storage_methods_exist_for_sqlite_and_postgres(tmp_path): sqlite = SessionStore(db_path=tmp_path / "brain.db") + sqlite.get_or_create_session("brain-store-session") run = { "runId": "brain_store_run", "schemaVersion": "2026-05-brain-run", diff --git a/backend/tests/test_data_connectors.py b/backend/tests/test_data_connectors.py new file mode 100644 index 0000000..b1cbae4 --- /dev/null +++ b/backend/tests/test_data_connectors.py @@ -0,0 +1,657 @@ +"""Offline tests for data connector readiness and EDGAR adapter behavior.""" + +from __future__ import annotations + +import pytest +from fastapi.testclient import TestClient + +from backend.agents import workflows as workflows_module +from backend.agents.workflows import SavedWorkflow, WorkflowEngine, WorkflowStep +from backend.config import API_KEY +from backend.data.connectors import ( + CSVConnector, + DataSourceRegistry, + EDGARDataConnector, + XBRLConnector, + YahooFinanceConnector, +) +from backend.data.edgar import ( + DEFAULT_SEC_USER_AGENT, + EDGAR_COMPANY_TICKERS, + EDGARConnector, + FinancialStatement, + SECFiling, +) +from backend.main import app + + +def _headers(): + return {"X-API-Key": API_KEY} + + +@pytest.mark.unit +async def test_csv_connector_enforces_bounded_rectangular_unique_schema(monkeypatch): + connector = CSVConnector() + + result = await connector.fetch("Account,Value\nCash,100\nDebt,50") + + assert result.headers == ["Account", "Value"] + assert result.row_count == 2 + assert result.metadata["bounded"] is True + + with pytest.raises(ValueError, match="unique"): + await connector.fetch("Account,account\nCash,100") + with pytest.raises(ValueError, match="same width"): + await connector.fetch("Account,Value\nCash") + + monkeypatch.setenv("CSV_CONNECTOR_MAX_CELL_CHARACTERS", "4") + with pytest.raises(ValueError, match="character budget"): + await connector.fetch("Name\nPrivateCustomer") + + +@pytest.mark.unit +async def test_csv_connector_rejects_byte_row_cell_and_delimiter_abuse(monkeypatch): + connector = CSVConnector() + + monkeypatch.setenv("CSV_CONNECTOR_MAX_INPUT_BYTES", "1024") + with pytest.raises(ValueError, match="byte budget"): + await connector.fetch("A\n" + "x" * 1024) + + monkeypatch.setenv("CSV_CONNECTOR_MAX_INPUT_BYTES", "8192") + monkeypatch.setenv("CSV_CONNECTOR_MAX_ROWS", "1") + with pytest.raises(ValueError, match="row budget"): + await connector.fetch("A\n1\n2") + with pytest.raises(ValueError, match="delimiter"): + await connector.fetch("A||B", {"delimiter": "||"}) + with pytest.raises(ValueError, match="has_header"): + await connector.fetch("A\n1", {"has_header": "yes"}) + + +@pytest.mark.unit +async def test_xbrl_connector_enforces_fact_and_numeric_boundaries(monkeypatch): + connector = XBRLConnector() + fact = ( + '1,250.5' + ) + + result = await connector.fetch(f"{fact}") + + assert result.row_count == 1 + assert result.data[0]["value"] == 1250.5 + assert result.metadata == { + "concepts_found": 1, + "format": "ixbrl", + "bounded": True, + } + + monkeypatch.setenv("XBRL_CONNECTOR_MAX_FACTS", "1") + with pytest.raises(ValueError, match="fact budget"): + await connector.fetch(f"{fact}{fact}") + with pytest.raises(ValueError, match="finite"): + await connector.fetch( + 'NaN' + ) + with pytest.raises(ValueError, match="format"): + await connector.fetch(fact, {"format": "html"}) + + +@pytest.mark.unit +async def test_xbrl_connector_applies_bounded_inline_scale_and_sign(): + connector = XBRLConnector() + scaled = ( + '1.25' + ) + + result = await connector.fetch(scaled) + + assert result.data[0]["value"] == -1_250_000.0 + + with pytest.raises(ValueError, match="scale"): + await connector.fetch( + '1' + ) + with pytest.raises(ValueError, match="sign"): + await connector.fetch( + '1' + ) + + +@pytest.mark.unit +async def test_xbrl_connector_parses_nested_inline_text_and_excludes_ix_exclude(): + connector = XBRLConnector() + nested = ( + '' + 'Excel Agent' + 'private footnote marker' + '' + ) + + result = await connector.fetch(nested) + + assert result.data[0]["value"] == "Excel Agent" + + with pytest.raises(ValueError, match="unterminated"): + await connector.fetch( + 'Excel Agent' + ) + + +@pytest.mark.unit +async def test_xbrl_connector_rejects_oversized_and_null_inputs(monkeypatch): + connector = XBRLConnector() + monkeypatch.setenv("XBRL_CONNECTOR_MAX_INPUT_BYTES", "1024") + + with pytest.raises(ValueError, match="byte budget"): + await connector.fetch("" + "x" * 1024 + "") + with pytest.raises(ValueError, match="null byte"): + await connector.fetch("\x00") + + +@pytest.mark.unit +async def test_xbrl_connector_parses_bounded_xml_instances_and_rejects_entities(monkeypatch): + connector = XBRLConnector() + instance = """ + + + + 1250.5 + """ + + result = await connector.fetch(instance, {"format": "xbrl"}) + + assert result.data == [ + { + "concept": "us-gaap:Revenues", + "label": "Revenue", + "value": 1250.5, + "period": "FY2025", + "unit": "USD", + } + ] + + monkeypatch.setenv("XBRL_CONNECTOR_MAX_XML_ELEMENTS", "1") + with pytest.raises(ValueError, match="element budget"): + await connector.fetch(instance, {"format": "xbrl"}) + + entity_instance = """]> + """ + with pytest.raises(ValueError, match="document type|entity"): + await connector.fetch(entity_instance, {"format": "xbrl"}) + + +@pytest.mark.unit +def test_registry_exposes_edgar_connector_readiness_metadata(): + registry = DataSourceRegistry() + connectors = {connector["id"]: connector for connector in registry.list_connectors()} + + assert {"csv", "xlsx", "xbrl", "edgar", "yahoo_finance"}.issubset(connectors) + edgar = connectors["edgar"] + assert edgar["requires_credentials"] is False + assert edgar["requires_network"] is True + assert edgar["offline_testable"] is True + assert edgar["readiness"]["schema_version"] == "2026-06-data-connector-readiness" + assert {operation["id"] for operation in edgar["operations"]} == { + "company_lookup", + "filings", + "financials", + "search", + } + assert edgar["config"]["user_agent_env"] == "SEC_EDGAR_USER_AGENT" + yahoo = connectors["yahoo_finance"] + assert yahoo["requires_credentials"] is False + assert yahoo["requires_network"] is True + assert yahoo["offline_testable"] is True + assert yahoo["readiness"]["schema_version"] == "2026-06-data-connector-readiness" + assert yahoo["readiness"]["contracted_feed"] is False + assert {operation["id"] for operation in yahoo["operations"]} == {"quote"} + assert yahoo["config"]["live_requests_env"] == "YAHOO_FINANCE_ENABLE_LIVE_REQUESTS" + + +@pytest.mark.unit +async def test_edgar_data_connector_fetches_mocked_filings_without_credentials(): + fake = _FakeEDGAR() + connector = EDGARDataConnector(fake) + + result = await connector.fetch("MSFT", {"operation": "filings", "form_type": "10-K", "count": 1}) + + assert result.source == "edgar" + assert result.row_count == 1 + assert result.metadata["requires_credentials"] is False + assert result.metadata["offline_testable"] is True + assert result.data[0]["ticker"] == "MSFT" + assert result.data[0]["filing_url"].endswith("/msft-2025-10k.htm") + assert fake.filing_requests == [("MSFT", "10-K", 1)] + + +@pytest.mark.unit +async def test_edgar_data_connector_flattens_mocked_financials(): + fake = _FakeEDGAR() + connector = EDGARDataConnector(fake) + + result = await connector.fetch("MSFT", {"operation": "financials", "form_type": "10-K"}) + + assert result.row_count == 2 + assert result.headers == [ + "period", + "statement_type", + "line_item", + "value", + "currency", + "scale", + "concept", + "taxonomy", + "unit", + "filed", + "period_end", + "fiscal_year", + "fiscal_period", + "form", + "accession_number", + "source_url", + ] + assert {row["line_item"] for row in result.data} == {"revenue", "net_income"} + assert result.metadata["statements"] == 1 + revenue = next(row for row in result.data if row["line_item"] == "revenue") + assert revenue["concept"] == "Revenues" + assert revenue["period_end"] == "2025-06-30" + assert revenue["source_url"].endswith("/us-gaap/Revenues.json") + + +@pytest.mark.unit +async def test_yahoo_finance_connector_fails_closed_until_live_requests_enabled(): + connector = YahooFinanceConnector(live_requests_enabled=False) + + result = await connector.fetch("MSFT") + + assert result.source == "yahoo_finance" + assert result.row_count == 0 + assert result.metadata["live_requests_enabled"] is False + assert result.metadata["contracted_feed"] is False + assert "YAHOO_FINANCE_ENABLE_LIVE_REQUESTS" in result.metadata["warnings"][0] + + +@pytest.mark.unit +async def test_yahoo_finance_connector_flattens_mocked_chart_quote(): + connector = YahooFinanceConnector(live_requests_enabled=True, max_symbols=3) + calls = [] + + async def fake_get_json(url: str): + calls.append(url) + symbol = url.split("/chart/", 1)[1].split("?", 1)[0] + return { + "chart": { + "result": [ + { + "meta": { + "symbol": symbol, + "regularMarketPrice": 501.23, + "previousClose": 497.1, + "currency": "USD", + "fullExchangeName": "NasdaqGS", + "instrumentType": "EQUITY", + "regularMarketTime": 1780000000, + "exchangeTimezoneName": "America/New_York", + } + } + ] + } + } + + connector._get_json = fake_get_json + + result = await connector.fetch("msft, bad_symbol!, aapl, msft", {"operation": "quote"}) + + assert [row["symbol"] for row in result.data] == ["MSFT", "AAPL"] + assert result.headers == [ + "symbol", + "price", + "previous_close", + "currency", + "exchange", + "instrument_type", + "regular_market_time", + "timezone", + "source", + ] + assert result.data[0]["price"] == 501.23 + assert result.data[0]["source"] == "yahoo_finance_chart" + assert result.metadata["symbols"] == ["MSFT", "AAPL"] + assert len(calls) == 2 + + +@pytest.mark.unit +async def test_saved_workflow_data_fetch_uses_registered_connector(monkeypatch): + fake = _FakeEDGAR() + registry = DataSourceRegistry() + registry.register("edgar", EDGARDataConnector(fake)) + monkeypatch.setattr("backend.data.connectors._registry", registry) + + engine = WorkflowEngine() + workflow = SavedWorkflow( + id="unit_fetch", + name="Unit Fetch", + description="Fetch data through a connector", + steps=[ + WorkflowStep( + step_type="data_fetch", + config={"source": "edgar", "operation": "filings", "form": "10-K", "count": 1}, + ) + ], + created_at="2026-06-15T00:00:00Z", + updated_at="2026-06-15T00:00:00Z", + variables={"ticker": "Ticker"}, + ) + engine._workflows = {"unit_fetch": workflow} + engine.save_workflow = lambda _workflow: None + + result = await engine.execute_workflow("unit_fetch", {"ticker": "MSFT"}) + + step = result["results"][0] + assert step["status"] == "complete" + assert step["source"] == "edgar" + assert step["query"] == "MSFT" + assert step["row_count"] == 1 + assert step["metadata"]["operation"] == "filings" + assert step["data_preview"][0]["ticker"] == "MSFT" + assert step["provenance_preview"] == [{"accession_number": "0000789019-25-000001"}] + + +@pytest.mark.unit +async def test_default_monthly_pnl_workflow_fetches_financial_facts_with_provenance(tmp_path, monkeypatch): + fake = _FakeEDGAR() + registry = DataSourceRegistry() + registry.register("edgar", EDGARDataConnector(fake)) + monkeypatch.setattr("backend.data.connectors._registry", registry) + monkeypatch.setattr(workflows_module, "WORKFLOWS_DIR", tmp_path) + + engine = WorkflowEngine() + engine.save_workflow = lambda _workflow: None + + workflow = engine.get_workflow("monthly_pnl_update") + assert workflow is not None + assert workflow.steps[0].config == { + "source": "edgar", + "operation": "financials", + "form": "10-Q", + "preview_limit": 10, + } + + result = await engine.execute_workflow("monthly_pnl_update", {"ticker": "MSFT"}) + + step = result["results"][0] + assert step["status"] == "complete" + assert step["metadata"]["operation"] == "financials" + assert step["data_preview"][0]["line_item"] == "revenue" + assert step["provenance_preview"][0]["concept"] == "Revenues" + assert step["provenance_preview"][0]["source_url"].endswith("/us-gaap/Revenues.json") + + +@pytest.mark.unit +def test_generic_connector_fetch_route_exposes_registered_operations(monkeypatch): + fake = _FakeEDGAR() + registry = DataSourceRegistry() + registry.register("edgar", EDGARDataConnector(fake)) + monkeypatch.setattr("backend.data.connectors._registry", registry) + + client = TestClient(app) + response = client.post( + "/api/v1/data/connectors/edgar/fetch", + headers=_headers(), + json={"query": "msft", "params": {"operation": "company_lookup"}}, + ) + + assert response.status_code == 200 + body = response.json() + assert body["source"] == "edgar" + assert body["row_count"] == 1 + assert body["metadata"]["operation"] == "company_lookup" + assert body["data"][0]["ticker"] == "MSFT" + + +@pytest.mark.unit +def test_edgar_readiness_is_no_network_and_flags_placeholder_user_agent(): + connector = EDGARConnector(user_agent=DEFAULT_SEC_USER_AGENT, live_requests_enabled=False) + + readiness = connector.readiness() + + assert readiness["requires_credentials"] is False + assert readiness["offline_testable"] is True + assert readiness["live_requests_enabled"] is False + assert readiness["user_agent_configured"] is False + assert any("SEC_EDGAR_USER_AGENT" in warning for warning in readiness["warnings"]) + + +@pytest.mark.unit +async def test_edgar_get_fails_closed_for_placeholder_user_agent(): + connector = EDGARConnector(user_agent=DEFAULT_SEC_USER_AGENT, live_requests_enabled=True) + + assert await connector._get("https://example.invalid/no-network-should-happen") is None + + +@pytest.mark.unit +async def test_edgar_lookup_uses_company_tickers_payload_without_search_api(): + connector = EDGARConnector(user_agent="UnitTest/1.0 contact@example.com", live_requests_enabled=True) + calls = [] + + async def fake_get(url: str): + calls.append(url) + return { + "0": { + "ticker": "MSFT", + "title": "MICROSOFT CORP", + "cik_str": 789019, + } + } + + connector._get = fake_get + + company = await connector.lookup_company("msft") + + assert calls == [EDGAR_COMPANY_TICKERS] + assert company == { + "ticker": "MSFT", + "company_name": "MICROSOFT CORP", + "cik": "0000789019", + } + + +@pytest.mark.unit +async def test_edgar_filings_validate_form_and_cap_count_without_live_network(): + connector = EDGARConnector( + user_agent="UnitTest/1.0 contact@example.com", + max_filings=2, + live_requests_enabled=True, + ) + lookup_calls = [] + request_calls = [] + + async def fake_lookup(ticker: str): + lookup_calls.append(ticker) + return {"ticker": ticker, "company_name": "Microsoft Corp", "cik": "0000789019"} + + async def fake_get(url: str): + request_calls.append(url) + return { + "filings": { + "recent": { + "form": ["10-K", "10-K", "10-K"], + "filingDate": ["2025-07-30", "2024-07-30", "2023-07-30"], + "accessionNumber": ["0000789019-25-000001", "0000789019-24-000001", "0000789019-23-000001"], + "primaryDocument": ["msft-2025-10k.htm", "msft-2024-10k.htm", "msft-2023-10k.htm"], + "reportDate": ["2025-06-30", "2024-06-30", "2023-06-30"], + } + } + } + + connector.lookup_company = fake_lookup + connector._get = fake_get + + unsupported = await connector.get_filings("MSFT", "S-8", count=4) + filings = await connector.get_filings("MSFT", "10-K", count=99) + + assert unsupported == [] + assert lookup_calls == ["MSFT"] + assert len(request_calls) == 1 + assert len(filings) == 2 + assert [filing.primary_document for filing in filings] == ["msft-2025-10k.htm", "msft-2024-10k.htm"] + + +@pytest.mark.unit +async def test_edgar_financials_pick_latest_fact_deterministically_with_citations(): + connector = EDGARConnector( + user_agent="UnitTest/1.0 contact@example.com", + live_requests_enabled=True, + ) + request_calls = [] + + async def fake_lookup(ticker: str): + return {"ticker": ticker, "company_name": "Microsoft Corp", "cik": "0000789019"} + + async def fake_get(url: str): + request_calls.append(url) + if url.endswith("/Revenues.json"): + return { + "units": { + "USD": [ + { + "form": "10-K", + "val": 100, + "filed": "2024-07-30", + "end": "2024-06-30", + "fy": 2024, + "fp": "FY", + "accn": "0000789019-24-000001", + }, + { + "form": "10-K", + "val": 200, + "filed": "2025-07-30", + "end": "2025-06-30", + "fy": 2025, + "fp": "FY", + "accn": "0000789019-25-000001", + }, + ] + } + } + if url.endswith("/EarningsPerShareDiluted.json"): + return { + "units": { + "USD/shares": [ + { + "form": "10-K", + "val": 12.34, + "filed": "2025-07-30", + "end": "2025-06-30", + "fy": 2025, + "fp": "FY", + "accn": "0000789019-25-000001", + } + ] + } + } + return {"units": {"USD": []}} + + connector.lookup_company = fake_lookup + connector._get = fake_get + + statements = await connector.get_xbrl_financials("MSFT", "10-K") + + income = next(statement for statement in statements if statement.statement_type == "income_statement") + assert income.period == "2025-06-30" + assert income.line_items["revenue"] == 200 + assert income.line_items["eps_diluted"] == 12.34 + assert income.line_item_metadata["revenue"] == { + "concept": "Revenues", + "taxonomy": "us-gaap", + "unit": "USD", + "filed": "2025-07-30", + "period_end": "2025-06-30", + "fiscal_year": 2025, + "fiscal_period": "FY", + "form": "10-K", + "accession_number": "0000789019-25-000001", + "source_url": "https://data.sec.gov/api/xbrl/companyconcept/CIK789019/us-gaap/Revenues.json", + } + assert any(url.endswith("/EarningsPerShareDiluted.json") for url in request_calls) + + +class _FakeEDGAR: + def __init__(self): + self.filing_requests = [] + + def readiness(self) -> dict: + return { + "schema_version": "2026-06-data-connector-readiness", + "configured": True, + "requires_credentials": False, + "requires_network": True, + "offline_testable": True, + "live_requests_enabled": True, + "user_agent_configured": True, + "supported_forms": ["10-K", "10-Q", "8-K"], + "max_filings": 10, + "timeout_seconds": 15, + "warnings": [], + } + + async def lookup_company(self, ticker: str) -> dict: + return {"ticker": ticker.upper(), "company_name": "Microsoft Corp", "cik": "0000789019"} + + async def get_filings(self, ticker: str, form_type: str = "10-K", count: int = 3) -> list[SECFiling]: + self.filing_requests.append((ticker, form_type, count)) + return [ + SECFiling( + ticker=ticker, + company_name="Microsoft Corp", + cik="0000789019", + form_type=form_type, + filing_date="2025-07-30", + period_of_report="2025-06-30", + accession_number="0000789019-25-000001", + primary_document="msft-2025-10k.htm", + filing_url="https://www.sec.gov/Archives/edgar/data/789019/000078901925000001/msft-2025-10k.htm", + ) + ][:count] + + async def get_xbrl_financials(self, ticker: str, form_type: str = "10-K") -> list[FinancialStatement]: + return [ + FinancialStatement( + period="2025-06-30", + statement_type="income_statement", + line_items={"revenue": 245_000_000_000, "net_income": 88_000_000_000}, + line_item_metadata={ + "revenue": { + "concept": "Revenues", + "taxonomy": "us-gaap", + "unit": "USD", + "filed": "2025-07-30", + "period_end": "2025-06-30", + "fiscal_year": 2025, + "fiscal_period": "FY", + "form": form_type, + "accession_number": "0000789019-25-000001", + "source_url": "https://data.sec.gov/api/xbrl/companyconcept/CIK789019/us-gaap/Revenues.json", + }, + "net_income": { + "concept": "NetIncomeLoss", + "taxonomy": "us-gaap", + "unit": "USD", + "filed": "2025-07-30", + "period_end": "2025-06-30", + "fiscal_year": 2025, + "fiscal_period": "FY", + "form": form_type, + "accession_number": "0000789019-25-000001", + "source_url": "https://data.sec.gov/api/xbrl/companyconcept/CIK789019/us-gaap/NetIncomeLoss.json", + }, + }, + ) + ] + + async def search_filings(self, query: str, form_type: str | None = None, date_start: str | None = None, date_end: str | None = None): + return [{"company": "Microsoft Corp", "form": form_type or "10-K", "date": date_start or "2025-07-30", "description": query}] diff --git a/backend/tests/test_e2e.py b/backend/tests/test_e2e.py index c663717..53a0925 100644 --- a/backend/tests/test_e2e.py +++ b/backend/tests/test_e2e.py @@ -31,6 +31,29 @@ def auth_headers(): return {"X-API-Key": API_KEY, "Content-Type": "application/json"} +def _create_enterprise_api_key(client: TestClient, auth_headers: dict, *, user_id: str, role: str) -> str: + user = client.post( + "/api/v1/enterprise/users", + headers=auth_headers, + json={ + "user_id": user_id, + "email": f"{user_id}@example.com", + "name": user_id.replace("-", " ").title(), + "role": role, + "status": "api_key_only", + "mfa_enabled": True, + }, + ) + assert user.status_code == 200 + key = client.post( + f"/api/v1/enterprise/users/{user_id}/api-keys", + headers=auth_headers, + json={"label": f"{role}-route-test"}, + ) + assert key.status_code == 200 + return key.json()["api_key"] + + # ── Health ─────────────────────────────────────────────────────────────── class TestHealthEndpoint: @@ -38,18 +61,18 @@ def test_health_returns_200(self, client, auth_headers): resp = client.get("/health", headers=auth_headers) assert resp.status_code == 200 data = resp.json() - assert "status" in data + assert data["status"] == "alive" assert "version" in data - assert "providers" in data + assert "providers" not in data def test_health_has_diagnostics(self, client, auth_headers): - resp = client.get("/health", headers=auth_headers) + resp = client.get("/health/diagnostics", headers=auth_headers) data = resp.json() assert "diagnostics" in data assert data["diagnostics"]["database"] == "ok" def test_health_has_cache_stats(self, client, auth_headers): - resp = client.get("/health", headers=auth_headers) + resp = client.get("/health/diagnostics", headers=auth_headers) data = resp.json() assert "cache" in data @@ -186,6 +209,7 @@ def test_list_providers(self, client, auth_headers): assert resp.status_code == 200 data = resp.json() assert "providers" in data + assert "aiPolicy" in data assert "lmstudio" in data["providers"] assert "ollama" in data["providers"] assert "anthropic" in data["providers"] @@ -194,9 +218,199 @@ def test_list_providers(self, client, auth_headers): def test_providers_have_cost(self, client, auth_headers): resp = client.get("/api/v1/providers", headers=auth_headers) data = resp.json() - for name, info in data["providers"].items(): + for _name, info in data["providers"].items(): assert "cost" in info assert "available" in info + assert "policy" in info + + +class TestAiPolicyAPI: + def test_effective_ai_policy_contract(self, client, auth_headers, monkeypatch): + monkeypatch.setenv("AI_ALLOWED_PROVIDERS", "ollama,lmstudio") + monkeypatch.setenv("AI_ALLOWED_MODES", "ask,preview") + monkeypatch.setenv("AI_EXTERNAL_SENDING_ALLOWED", "false") + monkeypatch.setenv("AI_EXTERNAL_DOCUMENTS_ALLOWED", "false") + monkeypatch.setenv("AI_SOURCE_RETENTION_HOURS", "24") + monkeypatch.setenv("AI_EXPORT_REGENERATED_FALLBACK_ALLOWED", "false") + monkeypatch.setenv("AI_EXPORT_MACRO_REGENERATED_FALLBACK_ALLOWED", "true") + + resp = client.get("/api/v1/security/ai-policy", headers=auth_headers) + + assert resp.status_code == 200 + data = resp.json() + assert data["schemaVersion"] == "2026-06-effective-ai-policy-v1" + assert data["allowedProviders"] == ["ollama", "lmstudio"] + assert data["allowedModes"] == ["ask", "preview"] + assert data["externalSendingAllowed"] is False + assert data["externalDocumentsAllowed"] is False + assert data["sourceRetention"] == {"hours": 24, "mode": "time_bound"} + assert data["exportFallback"] == {"regeneratedAllowed": False, "macroRegeneratedAllowed": True} + + def test_providers_include_effective_policy(self, client, auth_headers, monkeypatch): + monkeypatch.setenv("AI_ALLOWED_PROVIDERS", "ollama") + monkeypatch.setenv("AI_EXTERNAL_SENDING_ALLOWED", "false") + + resp = client.get("/api/v1/providers", headers=auth_headers) + + assert resp.status_code == 200 + data = resp.json() + assert data["aiPolicy"]["allowedProviders"] == ["ollama"] + assert data["providers"]["ollama"]["policy"]["allowed"] is True + assert data["providers"]["openai"]["policy"]["allowed"] is False + + def test_plan_request_rejects_policy_disabled_cloud_provider(self, client, auth_headers, monkeypatch): + monkeypatch.setenv("AI_EXTERNAL_SENDING_ALLOWED", "false") + + class EmptyStore: + def get_enterprise_config(self, _key): + return {} + + monkeypatch.setattr("backend.storage.get_session_store", lambda: EmptyStore()) + + resp = client.post( + "/api/v1/tasks/plan", + headers=auth_headers, + json={"prompt": "Build a DCF", "provider": "openai"}, + ) + + assert resp.status_code == 403 + assert resp.json()["detail"]["code"] == "external_sending_disabled" + + def test_plan_request_rejects_policy_disabled_source_documents(self, client, auth_headers, monkeypatch): + monkeypatch.setenv("AI_EXTERNAL_DOCUMENTS_ALLOWED", "false") + + resp = client.post( + "/api/v1/tasks/plan/v3", + headers=auth_headers, + json={"prompt": "Build a DCF", "source_document_ids": ["doc_external"]}, + ) + + assert resp.status_code == 403 + assert resp.json()["detail"]["code"] == "external_documents_disabled" + + def test_task_request_rejects_policy_disabled_mode(self, client, auth_headers, monkeypatch): + monkeypatch.setenv("AI_ALLOWED_MODES", "ask") + + resp = client.post( + "/api/v1/tasks", + headers=auth_headers, + json={"prompt": "Build a DCF", "mode": "action"}, + ) + + assert resp.status_code == 403 + assert resp.json()["detail"]["code"] == "ai_mode_disabled" + + def test_stored_enterprise_provider_policy_restricts_effective_policy(self, client, auth_headers, monkeypatch): + monkeypatch.setenv("AI_ALLOWED_PROVIDERS", "lmstudio,ollama,anthropic,openai") + monkeypatch.setenv("AI_ALLOWED_MODES", "ask,preview,action") + monkeypatch.setenv("AI_EXTERNAL_SENDING_ALLOWED", "true") + monkeypatch.setenv("AI_EXTERNAL_DOCUMENTS_ALLOWED", "true") + monkeypatch.setenv("BYOK_ENABLED", "false") + + class FakeStore: + def get_enterprise_config(self, key): + assert key == "provider_policy" + return { + "config_key": "provider_policy", + "config": { + "allow_cloud": False, + "allow_local": True, + "allowed_providers": ["lmstudio", "ollama", "openai"], + "allowed_modes": ["ask", "preview"], + "external_documents_allowed": False, + "allow_export_regenerated_fallback": False, + "allow_macro_regenerated_fallback": False, + }, + } + + monkeypatch.setattr("backend.storage.get_session_store", lambda: FakeStore()) + + resp = client.get("/api/v1/security/ai-policy", headers=auth_headers) + + assert resp.status_code == 200 + data = resp.json() + assert data["policySource"] == "enterprise_config" + assert data["allowedProviders"] == ["lmstudio", "ollama"] + assert data["allowedModes"] == ["ask", "preview"] + assert data["externalSendingAllowed"] is False + assert data["externalDocumentsAllowed"] is False + assert data["exportFallback"] == {"regeneratedAllowed": False, "macroRegeneratedAllowed": False} + + providers = client.get("/api/v1/providers", headers=auth_headers).json()["providers"] + assert providers["lmstudio"]["policy"]["allowed"] is True + assert providers["openai"]["policy"]["allowed"] is False + + def test_ai_policy_enforces_role_mode_entitlements(self): + from backend.api.security.ai_policy import AiPolicyViolation, EffectiveAiPolicy + + policy = EffectiveAiPolicy() + policy.require_request(mode="ask", actor={"user_id": "viewer-1", "role": "viewer"}) + policy.require_request(mode="preview", actor={"user_id": "analyst-1", "role": "analyst"}) + policy.require_request(mode="action", actor={"user_id": "admin-1", "role": "admin"}) + + with pytest.raises(AiPolicyViolation) as exc: + policy.require_request(mode="preview", actor={"user_id": "viewer-1", "role": "viewer"}) + assert exc.value.code == "ai_role_not_permitted" + + def test_enterprise_key_role_entitlements_apply_at_routes(self, client, auth_headers, monkeypatch): + async def fake_ask_mode(*_args, **_kwargs): + return "viewer answer" + + monkeypatch.setattr("backend.api.routes.tasks._handle_ask_mode", fake_ask_mode) + viewer_key = _create_enterprise_api_key( + client, + auth_headers, + user_id="viewer-route-entitlements", + role="viewer", + ) + viewer_headers = {"X-API-Key": viewer_key, "Content-Type": "application/json"} + + policy = client.get("/api/v1/security/ai-policy", headers=viewer_headers) + assert policy.status_code == 200 + entitlements = policy.json()["requesterEntitlements"] + assert entitlements["role"] == "viewer" + assert entitlements["userId"] == "viewer-route-entitlements" + assert entitlements["allowedModes"] == ["ask"] + + ask = client.post( + "/api/v1/tasks/ask", + headers=viewer_headers, + json={"prompt": "Summarize revenue"}, + ) + assert ask.status_code == 200 + assert ask.json()["answer"] == "viewer answer" + + plan = client.post( + "/api/v1/tasks/plan", + headers=viewer_headers, + json={"prompt": "Build a DCF"}, + ) + assert plan.status_code == 403 + assert plan.json()["detail"]["code"] == "ai_role_not_permitted" + + export = client.post( + "/api/v1/spreadsheet/export", + headers=viewer_headers, + json={"session": {"sheets": []}, "allowRegeneratedFallback": False}, + ) + assert export.status_code == 403 + assert export.json()["detail"]["code"] == "ai_role_not_permitted" + + apply = client.post( + "/api/v1/tasks/apply", + headers=viewer_headers, + json={"plan_id": "plan-viewer-denied", "session_id": "session-viewer-denied"}, + ) + assert apply.status_code == 403 + assert apply.json()["detail"]["code"] == "ai_role_not_permitted" + + apply_result = client.post( + "/api/v1/tasks/apply/result", + headers=viewer_headers, + json={"plan_id": "plan-viewer-denied", "session_id": "session-viewer-denied", "status": "applied"}, + ) + assert apply_result.status_code == 403 + assert apply_result.json()["detail"]["code"] == "ai_role_not_permitted" # ── Prompt Management ──────────────────────────────────────────────────── diff --git a/backend/tests/test_encrypted_store_integration.py b/backend/tests/test_encrypted_store_integration.py new file mode 100644 index 0000000..5937831 --- /dev/null +++ b/backend/tests/test_encrypted_store_integration.py @@ -0,0 +1,209 @@ +from __future__ import annotations + +import sqlite3 +from contextlib import contextmanager +from pathlib import Path +from typing import Any + +import pytest + +from backend.api.security.soc2 import SOC2Auditor +from backend.session_store import SessionStore +from backend.storage import encrypted_store + + +class _CipherCursor: + def fetchone(self): + return ("4.6.1 community",) + + +class _CipherProbeConnection: + def execute(self, statement: str): + assert statement == "PRAGMA cipher_version" + return _CipherCursor() + + def close(self) -> None: + return None + + +class _CipherDriver: + @staticmethod + def connect(_path: str) -> _CipherProbeConnection: + return _CipherProbeConnection() + + +def test_plain_sqlite_does_not_masquerade_as_sqlcipher(tmp_path: Path) -> None: + provider = encrypted_store.EncryptedSessionStore(db_path=str(tmp_path / "plain.db")) + + assert provider._driver_has_cipher(sqlite3) is False + assert provider._driver_has_cipher(_CipherDriver) is True + + +def test_sqlcipher_key_pragma_accepts_only_a_raw_256_bit_hex_key() -> None: + key = "ab" * 32 + + assert encrypted_store._sqlcipher_key_pragma(key) == f'PRAGMA key = "x\'{key}\'"' + with pytest.raises(RuntimeError, match="64 lowercase hexadecimal"): + encrypted_store._sqlcipher_key_pragma("a' ; ATTACH DATABASE '/tmp/leak' AS x; --") + + +def test_primary_session_store_uses_the_encryption_connection_provider( + tmp_path: Path, + monkeypatch: pytest.MonkeyPatch, +) -> None: + calls: list[Path] = [] + original = encrypted_store.EncryptedSessionStore.connection + + @contextmanager + def tracking_connection(self: encrypted_store.EncryptedSessionStore): + calls.append(self.db_path) + with original(self) as conn: + yield conn + + monkeypatch.setattr( + encrypted_store.EncryptedSessionStore, + "connection", + tracking_connection, + ) + db_path = tmp_path / "primary.db" + + SessionStore(db_path) + + assert calls + assert set(calls) == {db_path} + + +def test_encryption_configuration_fails_closed_without_sqlcipher( + tmp_path: Path, + monkeypatch: pytest.MonkeyPatch, +) -> None: + monkeypatch.setenv("DB_ENCRYPT", "true") + monkeypatch.setenv("DB_ENCRYPT_KEY", "a" * 64) + monkeypatch.setattr( + encrypted_store.EncryptedSessionStore, + "_check_sqlcipher", + lambda _self: False, + ) + + with pytest.raises(RuntimeError, match="refusing plaintext access"): + SessionStore(tmp_path / "must-not-be-plaintext.db") + + +def test_encryption_configuration_rejects_weak_or_padded_key( + tmp_path: Path, + monkeypatch: pytest.MonkeyPatch, +) -> None: + monkeypatch.setenv("DB_ENCRYPT", "true") + monkeypatch.setenv("DB_ENCRYPT_KEY", " short ") + monkeypatch.setattr( + encrypted_store.EncryptedSessionStore, + "_check_sqlcipher", + lambda _self: True, + ) + + with pytest.raises(RuntimeError, match="64 lowercase hexadecimal"): + SessionStore(tmp_path / "weak-key.db") + + +def test_encryption_status_is_bounded_and_does_not_claim_disabled_storage( + tmp_path: Path, +) -> None: + provider = encrypted_store.EncryptedSessionStore(db_path=str(tmp_path / "secret-path.db")) + + status = provider.get_encryption_status() + + assert status == { + "status": "disabled", + "configured": False, + "driverAvailable": False, + "driver": None, + "keyConfigured": False, + "keyValid": False, + "active": False, + "primaryStoreProtected": False, + "cipher": None, + "kdf": None, + "reasonCode": "not_configured", + } + assert str(tmp_path) not in str(status) + + +def test_postgres_encryption_requires_external_attestation( + monkeypatch: pytest.MonkeyPatch, +) -> None: + import backend.config as config + + monkeypatch.setattr(config, "DB_ENGINE", "postgres") + + status = encrypted_store.get_primary_storage_encryption_status() + + assert status["status"] == "external_attestation_required" + assert status["primaryStoreProtected"] is False + assert status["active"] is False + assert status["reasonCode"] == "postgres_encryption_at_rest_not_runtime_verifiable" + + +@pytest.mark.parametrize("protected", [False, True]) +def test_soc2_encryption_control_uses_primary_store_runtime_proof( + protected: bool, + monkeypatch: pytest.MonkeyPatch, +) -> None: + posture: dict[str, Any] = { + "status": "active" if protected else "blocked", + "primaryStoreProtected": protected, + "reasonCode": None if protected else "driver_unavailable", + } + monkeypatch.setattr( + encrypted_store, + "get_primary_storage_encryption_status", + lambda: posture, + ) + + result = SOC2Auditor().check_encryption() + + assert result["passed"] is protected + assert result["primary_store_protected"] is protected + assert result["status"] == posture["status"] + + +def test_soc2_audit_downgrades_failed_automated_controls( + monkeypatch: pytest.MonkeyPatch, +) -> None: + auditor = SOC2Auditor() + monkeypatch.setattr( + auditor, + "check_encryption", + lambda: {"passed": False, "reason_code": "driver_unavailable"}, + ) + + report = auditor.run_audit() + control = next(item for item in report["controls"] if item["id"] == "CC5.3") + + assert control["declared_status"] == "partial" + assert control["status"] == "blocked" + assert control["automated_check"]["passed"] is False + assert report["summary"]["blocked"] >= 1 + assert report["by_criteria"]["Security"]["blocked"] >= 1 + + +def test_soc2_audit_does_not_return_automated_exception_text( + monkeypatch: pytest.MonkeyPatch, +) -> None: + auditor = SOC2Auditor() + + def explode() -> dict[str, Any]: + raise RuntimeError("/private/customer/path secret-token") + + monkeypatch.setattr(auditor, "check_encryption", explode) + + report = auditor.run_audit() + control = next(item for item in report["controls"] if item["id"] == "CC5.3") + encoded = str(control) + + assert control["status"] == "blocked" + assert control["automated_check"] == { + "passed": False, + "error_type": "RuntimeError", + } + assert "private/customer" not in encoded + assert "secret-token" not in encoded diff --git a/backend/tests/test_enterprise_admin_audit_isolation.py b/backend/tests/test_enterprise_admin_audit_isolation.py new file mode 100644 index 0000000..4b6906b --- /dev/null +++ b/backend/tests/test_enterprise_admin_audit_isolation.py @@ -0,0 +1,80 @@ +"""Enterprise administrator audits must never share a user session namespace.""" + +from __future__ import annotations + +import json +import time + +import pytest + +from backend.api.security.ownership import OwnershipNotFoundError, ownership_actor_scope +from backend.services import lifecycle as lifecycle_module +from backend.services.lifecycle import LifecycleCoordinator +from backend.session_store import SessionStore + + +def _actor(user_id: str, role: str) -> dict: + return {"user_id": user_id, "email": f"{user_id}@example.com", "role": role} + + +@pytest.mark.unit +def test_attacker_owned_predictable_session_cannot_read_global_admin_audit(tmp_path, monkeypatch): + store = SessionStore(tmp_path / "sessions.db") + monkeypatch.setattr( + lifecycle_module, + "_coordinator", + LifecycleCoordinator(store=store, receipts_dir=tmp_path / "receipts"), + ) + with ownership_actor_scope(_actor("attacker", "analyst")): + store.get_or_create_session("enterprise-admin") + + with ownership_actor_scope(_actor("real-admin", "admin")): + audit_id = store.record_enterprise_admin_audit( + "enterprise_admin.secret_rotated", + {"actor": {"email": "real-admin@example.com"}, "secret": "rotated"}, + ) + admin_rows = store.list_enterprise_admin_audits() + + assert admin_rows[0]["audit_id"] == audit_id + with ownership_actor_scope(_actor("attacker", "analyst")): + session_export = store.export_audit_csv("enterprise-admin") + assert "secret_rotated" not in session_export + assert "real-admin@example.com" not in session_export + with pytest.raises(OwnershipNotFoundError): + store.list_enterprise_admin_audits() + + +@pytest.mark.unit +def test_sqlite_startup_quarantines_legacy_admin_audit_out_of_sessions(tmp_path): + db_path = tmp_path / "sessions.db" + store = SessionStore(db_path) + store.get_or_create_session("attacker-session") + payload = {"actor": {"email": "legacy-admin@example.com"}, "action": "secret_rotated"} + with store._conn() as conn: + conn.execute("DROP TRIGGER trg_audit_events_no_enterprise_admin_insert") + conn.execute("DROP TRIGGER trg_audit_events_no_enterprise_admin_update") + conn.execute( + "DELETE FROM schema_migrations " + "WHERE migration_id = '2026_08_10_0018_enterprise_admin_audit_isolation'" + ) + conn.execute( + """ + INSERT INTO audit_events + (audit_id, session_id, event_type, payload, created_at) + VALUES (?, ?, ?, ?, ?) + """, + ( + "legacy-admin-audit", + "attacker-session", + "enterprise_admin.secret_rotated", + json.dumps(payload), + time.time(), + ), + ) + + restarted = SessionStore(db_path) + assert "secret_rotated" not in restarted.export_audit_csv("attacker-session") + with ownership_actor_scope(_actor("real-admin", "admin")): + rows = restarted.list_enterprise_admin_audits() + assert rows[0]["audit_id"] == "legacy-admin-audit" + assert rows[0]["payload"] == payload diff --git a/backend/tests/test_enterprise_metadata_concurrency.py b/backend/tests/test_enterprise_metadata_concurrency.py new file mode 100644 index 0000000..c2390f4 --- /dev/null +++ b/backend/tests/test_enterprise_metadata_concurrency.py @@ -0,0 +1,139 @@ +"""Concurrent enterprise identity/profile update regressions.""" + +from __future__ import annotations + +import threading +import time +from concurrent.futures import ThreadPoolExecutor + +import pytest + +import backend.session_store as session_store_module +from backend.api.routes.enterprise import UserRequest +from backend.session_store import SessionStore + + +def _seed_active_user(store: SessionStore, user_id: str) -> None: + store.upsert_enterprise_user( + user_id=user_id, + email=f"{user_id}@example.com", + name="Original", + role="analyst", + status="active", + mfa_enabled=True, + metadata={"base": "preserve", "security": "baseline"}, + oidc_issuer="https://issuer.example", + oidc_subject=f"subject-{user_id}", + ) + + +def _barrier_first_user_read(store: SessionStore, monkeypatch): + original_get = store.get_enterprise_user + barrier = threading.Barrier(2) + local = threading.local() + + def synchronized_get(user_id: str): + result = original_get(user_id) + if not getattr(local, "synchronized", False): + local.synchronized = True + barrier.wait(timeout=5) + return result + + monkeypatch.setattr(store, "get_enterprise_user", synchronized_get) + return original_get + + +@pytest.mark.unit +def test_sqlite_concurrent_metadata_updates_merge_disjoint_keys(tmp_path, monkeypatch): + store = SessionStore(tmp_path / "sessions.db") + user_id = "metadata-merge-user" + _seed_active_user(store, user_id) + read_final = _barrier_first_user_read(store, monkeypatch) + + def update(key: str): + return store.upsert_enterprise_user( + user_id=user_id, + email=f"{user_id}@example.com", + name="Original", + role="analyst", + status=None, + mfa_enabled=None, + metadata={key: True}, + ) + + with ThreadPoolExecutor(max_workers=2) as executor: + left = executor.submit(update, "left") + right = executor.submit(update, "right") + left.result(timeout=10) + right.result(timeout=10) + + final = read_final(user_id) + assert final is not None + assert final["metadata"] == { + "base": "preserve", + "security": "baseline", + "oidc_issuer": "https://issuer.example", + "oidc_subject": f"subject-{user_id}", + "left": True, + "right": True, + } + assert final["mfa_enabled"] is True + + +@pytest.mark.unit +def test_sqlite_profile_patch_cannot_reactivate_concurrent_disabled_user(tmp_path, monkeypatch): + store = SessionStore(tmp_path / "sessions.db") + user_id = "disable-race-user" + _seed_active_user(store, user_id) + read_final = _barrier_first_user_read(store, monkeypatch) + original_prepare = session_store_module.prepare_enterprise_identity_binding + + def delayed_prepare(**kwargs): + binding = original_prepare(**kwargs) + if binding.status == "active": + time.sleep(0.25) + return binding + + monkeypatch.setattr(session_store_module, "prepare_enterprise_identity_binding", delayed_prepare) + + def disable(): + return store.upsert_enterprise_user( + user_id=user_id, + email=f"{user_id}@example.com", + name="Original", + role="analyst", + status="disabled", + mfa_enabled=True, + metadata={"security_disabled_by": "admin"}, + ) + + def patch_profile(): + return store.upsert_enterprise_user( + user_id=user_id, + email=f"{user_id}@example.com", + name="Updated profile", + role="analyst", + status=None, + mfa_enabled=None, + metadata={"profile_refresh": True}, + ) + + with ThreadPoolExecutor(max_workers=2) as executor: + disabled = executor.submit(disable) + profile = executor.submit(patch_profile) + disabled.result(timeout=10) + profile.result(timeout=10) + + final = read_final(user_id) + assert final is not None + assert final["status"] == "disabled" + assert final["mfa_enabled"] is True + assert final["metadata"]["security_disabled_by"] == "admin" + assert final["metadata"]["profile_refresh"] is True + + +@pytest.mark.unit +def test_enterprise_user_request_uses_patch_semantics_for_security_fields(): + fields = UserRequest.model_fields + assert fields["status"].default is None + assert fields["mfa_enabled"].default is None diff --git a/backend/tests/test_flagship_v6.py b/backend/tests/test_flagship_v6.py index 3f9d411..4bedabc 100644 --- a/backend/tests/test_flagship_v6.py +++ b/backend/tests/test_flagship_v6.py @@ -41,7 +41,9 @@ def test_v6_services_return_strict_accuracy_contracts(): graph = get_formula_graph_v4_service().build(session_id="v6-svc", workbook_context=context) assert isinstance(graph, FormulaGraphArtifactV4) assert graph.schema_version == "2026-05-formula-graph-artifact-v4" - assert graph.pre_apply_simulation_ready is True + assert graph.pre_apply_simulation_ready is False + assert graph.calculation_engine_metadata["hyperformulaAvailable"] is False + assert "Formula graph used fallback engine: univer" in graph.fallback_confidence_gaps assert graph.output_cell_rankings is not None truth = get_workbook_truth_model_v4_service().build(session_id="v6-svc", workbook_context=context) @@ -94,7 +96,9 @@ def test_v6_public_api_surfaces_are_additive_and_persisted(): ) assert graph.status_code == 200 assert graph.json()["schemaVersion"] == "2026-05-formula-graph-artifact-v4" - assert graph.json()["preApplySimulationReady"] is True + assert graph.json()["preApplySimulationReady"] is False + assert graph.json()["calculationEngineMetadata"]["hyperformulaAvailable"] is False + assert "Formula graph used fallback engine: univer" in graph.json()["fallbackConfidenceGaps"] truth = client.post( "/api/v1/workbooks/truth-model/v4", diff --git a/backend/tests/test_flagship_v7.py b/backend/tests/test_flagship_v7.py index b71b044..77d74a2 100644 --- a/backend/tests/test_flagship_v7.py +++ b/backend/tests/test_flagship_v7.py @@ -120,7 +120,12 @@ def test_v7_public_api_surfaces_are_additive(): assert category in stack_payload["categories"] assert stack_payload["brainReadiness"]["schemaVersion"] == "2026-05-brain-readiness" assert stack_payload["brainReadiness"]["artifactLineageCoverage"]["status"] == "passed" - assert stack_payload["categories"]["dependency"]["signals"]["sheetjsAdvisoryStatus"] in {"warning", "passed"} + assert stack_payload["categories"]["dependency"]["signals"]["sheetjsAdvisoryStatus"] in { + "warning", + "passed", + "accepted", + "remediated", + } assert stack_payload["trustWorkflow"]["schemaVersion"] == "2026-05-trust-workflow-state-machine" assert [state["id"] for state in stack_payload["trustWorkflow"]["states"]] == [ "understand", @@ -251,7 +256,8 @@ def test_agent_run_v4_routes_expose_execution_groups_and_audit_export(): assert run.schema_version == "2026-05-agent-run-v4" assert run.execution_groups assert run.audit_export_id - assert run.v7_quality_gate["noSilentFormulaFallback"] is True + assert run.v7_quality_gate["status"] == "blocked" + assert run.v7_quality_gate["noSilentFormulaFallback"] is False groups = client.get(f"/api/v1/agent-runs/{run.run_id}/execution-groups", headers=headers) assert groups.status_code == 200 @@ -381,7 +387,8 @@ def test_v7_default_prep_readiness_and_replay_routes_are_additive(): report = V7DefaultReadinessReport.model_validate(readiness.json()) assert report.schema_version == "2026-05-v7-default-readiness" assert report.status in {"ready", "warning", "blocked"} - assert report.default_allowed is (report.status == "ready") + default_switch_blockers = report.metadata.get("defaultSwitchBlockers") or [] + assert report.default_allowed is (report.status == "ready" and not default_switch_blockers) assert report.opt_in_allowed is (report.status in {"ready", "warning"}) assert report.gates["planEngineDefault"]["value"] == "v2" assert report.plan_comparison is not None @@ -414,6 +421,10 @@ def test_v7_default_prep_readiness_and_replay_routes_are_additive(): assert registry.status_code == 200 operations = registry.json()["operations"] assert operations["screenshot_range"]["applyReady"] is True - assert operations["create_sparkline"]["applyReady"] is True + sparkline = operations["create_sparkline"] + assert sparkline["applyReady"] is False + assert sparkline["rollbackProofClass"] == "manual" + assert sparkline["rollbackProof"] == {"proofClass": "manual", "strategy": "manual", "warningsRequired": True} + assert "Operation is not V7 apply-ready." in sparkline["defaultSwitchBlockers"] assert operations["create_pivot_table"]["safeGated"] is True assert operations["write_cell"]["rollbackProofClass"] == "range_restore_operations" diff --git a/backend/tests/test_formula_tracer.py b/backend/tests/test_formula_tracer.py new file mode 100644 index 0000000..b8c693d --- /dev/null +++ b/backend/tests/test_formula_tracer.py @@ -0,0 +1,53 @@ +from backend.excel.formula_tracer import build_reverse_trace, trace_formula, trace_to_ui + + +def test_formula_tracer_captures_cross_sheet_forward_and_reverse_dependencies(): + operations = [ + {"sheet": "Assumptions", "cell": "A1", "type": "input", "value": 100}, + {"sheet": "Assumptions", "cell": "B2", "type": "formula", "formula": "=A1+1", "value": 101}, + {"sheet": "Model", "cell": "B2", "type": "formula", "formula": "='Assumptions'!A1*2", "value": 200}, + ] + + trace = trace_formula("Model!B2", operations) + ui = trace_to_ui(trace) + reverse = build_reverse_trace("Assumptions!A1", operations) + + assert trace.root_cell == "Model!B2" + assert trace.nodes["Model!B2"].children == ["Assumptions!A1"] + assert ui["root"]["cell"] == "Model!B2" + assert ui["root"]["inputs"][0]["cell"] == "Assumptions!A1" + assert {item["cell"] for item in reverse["affected_cells"]} == {"Assumptions!B2", "Model!B2"} + + +def test_formula_tracer_keeps_same_address_cells_on_different_sheets_separate(): + operations = [ + {"sheet": "Assumptions", "cell": "A1", "type": "input", "value": 10}, + {"sheet": "Model", "cell": "A1", "type": "input", "value": 20}, + {"sheet": "Assumptions", "cell": "B1", "type": "formula", "formula": "=A1+1", "value": 11}, + {"sheet": "Model", "cell": "C1", "type": "formula", "formula": "=A1+1", "value": 21}, + ] + + assumptions_reverse = build_reverse_trace("Assumptions!A1", operations) + model_reverse = build_reverse_trace("Model!A1", operations) + bare_trace = trace_formula("C1", operations) + + assert [item["cell"] for item in assumptions_reverse["affected_cells"]] == ["Assumptions!B1"] + assert [item["cell"] for item in model_reverse["affected_cells"]] == ["Model!C1"] + assert bare_trace.root_cell == "Model!C1" + assert bare_trace.nodes["Model!C1"].children == ["Model!A1"] + + +def test_formula_tracer_preserves_legacy_sheetless_operation_shape(): + operations = [ + {"cell": "A1", "type": "input", "value": 5}, + {"cell": "B1", "type": "formula", "formula": "=A1*2", "value": 10}, + ] + + trace = trace_formula("B1", operations) + reverse = build_reverse_trace("A1", operations) + + assert trace.root_cell == "B1" + assert trace.nodes["B1"].children == ["A1"] + assert trace_to_ui(trace)["root"]["inputs"][0]["cell"] == "A1" + assert reverse["target"] == "A1" + assert reverse["affected_cells"][0]["cell"] == "B1" diff --git a/backend/tests/test_graceful_drain.py b/backend/tests/test_graceful_drain.py new file mode 100644 index 0000000..ff96f5e --- /dev/null +++ b/backend/tests/test_graceful_drain.py @@ -0,0 +1,142 @@ +from __future__ import annotations + +import asyncio +import threading +from concurrent.futures import ThreadPoolExecutor + +import pytest +from fastapi import FastAPI +from fastapi.testclient import TestClient +from starlette.websockets import WebSocketDisconnect + +from backend.api.middleware.drain import DrainController, DrainMiddleware, drain_controller +from backend.main import app + + +def test_controller_flips_state_before_rejecting_new_work_and_joins_in_flight() -> None: + controller = DrainController() + assert controller.try_acquire("/api/v1/tasks") is True + assert controller.snapshot().in_flight == 1 + + assert controller.begin_drain() is True + assert controller.snapshot().draining is True + assert controller.try_acquire("/api/v1/sessions") is False + assert controller.try_acquire("/health/ready") is True + + assert asyncio.run(controller.wait_for_idle(0.01)) is False + controller.release() + assert asyncio.run(controller.wait_for_idle(0.25)) is True + + +def test_drain_middleware_rejects_non_probe_work_with_stable_error() -> None: + controller = DrainController() + mini_app = FastAPI() + + @mini_app.post("/api/v1/work") + async def work() -> dict[str, bool]: + return {"ok": True} + + @mini_app.get("/health/ready") + async def ready() -> dict[str, bool]: + return {"ok": True} + + mini_app.add_middleware(DrainMiddleware, controller=controller) + client = TestClient(mini_app) + controller.begin_drain() + + rejected = client.post("/api/v1/work") + probe = client.get("/health/ready") + + assert rejected.status_code == 503 + assert rejected.json() == { + "error": { + "code": "SERVICE_DRAINING", + "message": "Service is draining; retry on another instance.", + } + } + assert rejected.headers["Retry-After"] == "1" + assert probe.status_code == 200 + + +def test_drain_middleware_joins_an_admitted_concurrent_request() -> None: + controller = DrainController() + started = threading.Event() + release = threading.Event() + mini_app = FastAPI() + + @mini_app.post("/api/v1/slow-work") + async def slow_work() -> dict[str, bool]: + started.set() + await asyncio.to_thread(release.wait, 2) + return {"ok": True} + + mini_app.add_middleware(DrainMiddleware, controller=controller) + client = TestClient(mini_app) + + with ThreadPoolExecutor(max_workers=1) as executor: + admitted = executor.submit(client.post, "/api/v1/slow-work") + assert started.wait(1) + controller.begin_drain() + assert client.post("/api/v1/slow-work").status_code == 503 + assert asyncio.run(controller.wait_for_idle(0.01)) is False + release.set() + assert admitted.result(timeout=2).status_code == 200 + + assert asyncio.run(controller.wait_for_idle(0.25)) is True + + +def test_drain_middleware_rejects_new_websocket_work() -> None: + controller = DrainController() + mini_app = FastAPI() + + @mini_app.websocket("/ws/tasks/{session_id}") + async def websocket_work() -> None: + raise AssertionError("draining websocket must never reach its handler") + + mini_app.add_middleware(DrainMiddleware, controller=controller) + controller.begin_drain() + + with pytest.raises(WebSocketDisconnect) as exc_info: + with TestClient(mini_app).websocket_connect("/ws/tasks/private-session"): + pass + + assert exc_info.value.code == 1013 + + +def test_readiness_flips_immediately_without_touching_storage(monkeypatch) -> None: + def storage_must_not_run() -> None: + raise AssertionError("draining readiness must not probe storage") + + monkeypatch.setattr("backend.main._storage_readiness", storage_must_not_run) + drain_controller.begin_drain() + try: + response = TestClient(app).get("/health/ready") + finally: + drain_controller.start_accepting() + + assert response.status_code == 503 + assert response.json() == { + "status": "not_ready", + "version": "0.5.0", + "checks": { + "drain": "draining", + "storage": "not_checked", + "migrations": "not_checked", + }, + } + + +def test_controller_publishes_drain_and_in_flight_metrics(monkeypatch) -> None: + states: list[bool] = [] + counts: list[int] = [] + monkeypatch.setattr("backend.observability.set_service_draining", states.append) + monkeypatch.setattr("backend.observability.set_in_flight_requests", counts.append) + controller = DrainController() + + controller.start_accepting() + assert controller.try_acquire("/api/v1/tasks") is True + controller.begin_drain() + controller.release() + + assert states == [False, True] + assert counts == [0, 1, 1, 0] diff --git a/backend/tests/test_lifecycle_active_tasks.py b/backend/tests/test_lifecycle_active_tasks.py new file mode 100644 index 0000000..5cd1b3f --- /dev/null +++ b/backend/tests/test_lifecycle_active_tasks.py @@ -0,0 +1,238 @@ +"""Active AI work and non-durable cache lifecycle regressions.""" + +from __future__ import annotations + +import asyncio +import uuid +from types import SimpleNamespace + +import pytest + +from backend.agents import workflows as workflows_module +from backend.api.routes import tasks as tasks_module +from backend.excel import citations as citations_module +from backend.excel import diff_engine as diff_module +from backend.services import lifecycle as lifecycle_module +from backend.services import retention as retention_module +from backend.services.lifecycle import LifecycleCoordinator, LifecycleState +from backend.session_store import SessionStore + + +RECEIPT_KEY = b"active_task_lifecycle_regression_key_material" * 2 + + +class _Jobs: + @staticmethod + def purge_session(_session_id: str): + return {"jobs_revoked": 0, "threads_pending": 0} + + +class _Workflows: + @staticmethod + def purge_session(_session_id: str): + return 0 + + @staticmethod + def count_session_workflows(_session_id: str): + return 0 + + +def _coordinator(tmp_path, store: SessionStore, monkeypatch) -> LifecycleCoordinator: + monkeypatch.setattr(workflows_module, "get_workflow_engine", lambda: _Workflows()) + coordinator = LifecycleCoordinator( + store=store, + job_manager=_Jobs(), + receipts_dir=tmp_path / "receipts", + receipt_signing_keys={"current": RECEIPT_KEY}, + current_receipt_key_id="current", + ) + coordinator.register_purger("diff_engine", diff_module.purge_diff_engine) + monkeypatch.setattr(lifecycle_module, "_coordinator", coordinator) + monkeypatch.setattr(tasks_module, "get_session_store", lambda: store) + monkeypatch.setattr(tasks_module, "_enforce_ai_policy", lambda *_args, **_kwargs: None) + return coordinator + + +def _request_state(): + return SimpleNamespace(state=SimpleNamespace()) + + +@pytest.mark.unit +@pytest.mark.asyncio +async def test_rest_pipeline_is_canceled_and_cannot_recreate_diff_cache_after_purge( + tmp_path, + monkeypatch, +): + store = SessionStore(tmp_path / "sessions.db") + session_id = f"rest-active-{uuid.uuid4().hex}" + coordinator = _coordinator(tmp_path, store, monkeypatch) + entered = asyncio.Event() + provider_canceled = asyncio.Event() + + async def paused_pipeline(**_kwargs): + entered.set() + try: + await asyncio.Future() + except asyncio.CancelledError: + provider_canceled.set() + raise + + monkeypatch.setattr(tasks_module, "run_pipeline", paused_pipeline) + route_task = asyncio.create_task( + tasks_module.create_task( + tasks_module.TaskRequest( + prompt="private prompt", + mode="action", + session_id=session_id, + ), + _request_state(), + ) + ) + await asyncio.wait_for(entered.wait(), timeout=2) + + receipt = await asyncio.wait_for(coordinator.purge_session(session_id), timeout=5) + + assert receipt.state is LifecycleState.VERIFIED + assert provider_canceled.is_set() + with pytest.raises(asyncio.CancelledError): + await route_task + assert diff_module.purge_diff_engine(session_id) == {"engines_evicted": 0} + + +class _PausedWebSocket: + def __init__(self): + self.received = 0 + self.sent: list[dict] = [] + + async def accept(self): + return None + + async def receive_json(self): + self.received += 1 + if self.received == 1: + return {"prompt": "private websocket prompt", "mode": "action"} + await asyncio.Future() + + async def send_json(self, payload): + self.sent.append(payload) + + +@pytest.mark.unit +@pytest.mark.asyncio +async def test_websocket_pipeline_is_canceled_and_cannot_recreate_diff_cache_after_purge( + tmp_path, + monkeypatch, +): + store = SessionStore(tmp_path / "sessions.db") + session_id = f"ws-active-{uuid.uuid4().hex}" + store.get_or_create_session(session_id) + coordinator = _coordinator(tmp_path, store, monkeypatch) + entered = asyncio.Event() + provider_canceled = asyncio.Event() + + async def paused_pipeline(**_kwargs): + entered.set() + try: + await asyncio.Future() + except asyncio.CancelledError: + provider_canceled.set() + raise + + monkeypatch.setattr(tasks_module, "run_pipeline", paused_pipeline) + handler_task = asyncio.create_task( + tasks_module.websocket_task_handler(_PausedWebSocket(), session_id) + ) + await asyncio.wait_for(entered.wait(), timeout=2) + + receipt = await asyncio.wait_for(coordinator.purge_session(session_id), timeout=5) + + assert receipt.state is LifecycleState.VERIFIED + assert provider_canceled.is_set() + with pytest.raises(asyncio.CancelledError): + await handler_task + assert diff_module.purge_diff_engine(session_id) == {"engines_evicted": 0} + + +@pytest.mark.unit +@pytest.mark.asyncio +async def test_immediate_retention_purge_does_not_cancel_its_own_ask_request( + tmp_path, + monkeypatch, +): + store = SessionStore(tmp_path / "sessions.db") + session_id = f"ask-immediate-{uuid.uuid4().hex}" + coordinator = _coordinator(tmp_path, store, monkeypatch) + + async def answer(*_args, **_kwargs): + return "private answer" + + async def purge_immediately(checked_session_id: str, terminal_status: str): + assert checked_session_id == session_id + assert terminal_status == "completed" + return await coordinator.purge_session(checked_session_id) + + monkeypatch.setattr(tasks_module, "_handle_ask_mode", answer) + monkeypatch.setattr( + retention_module, + "purge_terminal_session_if_immediate", + purge_immediately, + ) + + response = await asyncio.wait_for( + tasks_module.ask_task( + tasks_module.AskRequest(prompt="private question", session_id=session_id), + _request_state(), + ), + timeout=5, + ) + + assert response.answer == "private answer" + assert store.get_session_lifecycle_state(session_id) == "verified" + + +@pytest.mark.unit +@pytest.mark.asyncio +async def test_verified_tombstone_fences_and_zeroizes_diff_and_citation_trackers( + tmp_path, + monkeypatch, +): + store = SessionStore(tmp_path / "sessions.db") + session_id = f"cache-fence-{uuid.uuid4().hex}" + store.get_or_create_session(session_id) + coordinator = _coordinator(tmp_path, store, monkeypatch) + diff_engine = diff_module.get_diff_engine(session_id) + diff_engine.push( + diff_engine.create_changeset( + [{"cell": "A1", "value": "SECRET"}], + prompt="private prompt", + ) + ) + tracker = citations_module.get_citation_tracker(session_id) + tracker.track_from_document("A1", "private.pdf", source_text="private financial data") + detached_index = tracker.get_index() + detached_index.add( + citations_module.Citation( + cell="B2", + source_document="caller-owned.pdf", + source_text="must not re-enter the session cache", + ) + ) + assert tracker.get_stats()["total_citations"] == 1 + + receipt = await coordinator.purge_session(session_id) + + assert receipt.state is LifecycleState.VERIFIED + with pytest.raises(RuntimeError, match="lifecycle is not active"): + diff_engine.get_history() + with pytest.raises(RuntimeError, match="lifecycle is not active"): + tracker.get_stats() + with pytest.raises(RuntimeError, match="lifecycle is not active"): + diff_module.get_diff_engine(session_id) + with pytest.raises(RuntimeError, match="lifecycle is not active"): + citations_module.get_citation_tracker(session_id) + assert diff_module.purge_diff_engine(session_id) == {"engines_evicted": 0} + assert citations_module.purge_citation_tracker(session_id) == { + "trackers_evicted": 0, + "citations_evicted": 0, + "documents_evicted": 0, + } diff --git a/backend/tests/test_lifecycle_receipt_key_rotation.py b/backend/tests/test_lifecycle_receipt_key_rotation.py new file mode 100644 index 0000000..ad2e593 --- /dev/null +++ b/backend/tests/test_lifecycle_receipt_key_rotation.py @@ -0,0 +1,109 @@ +"""Rolling deletion-receipt key rotation contract.""" + +from __future__ import annotations + +import json +import uuid + +import pytest + +from backend.agents import workflows as workflows_module +from backend.services.lifecycle import LifecycleCoordinator, LifecycleState +from backend.session_store import SessionStore + + +K1 = b"1" * 48 +K2 = b"2" * 48 + + +class _Jobs: + @staticmethod + def purge_session(_session_id: str): + return {"jobs_revoked": 0, "threads_pending": 0} + + +class _Workflows: + @staticmethod + def purge_session(_session_id: str): + return 0 + + @staticmethod + def count_session_workflows(_session_id: str): + return 0 + + +def _coordinator(store, receipts_dir, *, current: str, keys: dict[str, bytes]): + return LifecycleCoordinator( + store=store, + job_manager=_Jobs(), + receipts_dir=receipts_dir, + current_receipt_key_id=current, + receipt_signing_keys=keys, + ) + + +@pytest.mark.unit +@pytest.mark.asyncio +async def test_two_phase_fleet_rotation_verifies_both_generations_before_retirement( + tmp_path, + monkeypatch, +): + monkeypatch.setattr(workflows_module, "get_workflow_engine", lambda: _Workflows()) + store = SessionStore(tmp_path / "sessions.db") + receipts_dir = tmp_path / "receipts" + old_session = f"rotation-old-{uuid.uuid4().hex}" + overlap_session = f"rotation-overlap-{uuid.uuid4().hex}" + new_session = f"rotation-new-{uuid.uuid4().hex}" + for session_id in (old_session, overlap_session, new_session): + store.get_or_create_session(session_id) + + k1_only = _coordinator(store, receipts_dir, current="k1", keys={"k1": K1}) + old_receipt = await k1_only.purge_session(old_session) + + # Phase 1: K2 is verification-only on every K1-signing replica. + predistributed = _coordinator( + store, + receipts_dir, + current="k1", + keys={"k1": K1, "k2": K2}, + ) + assert (await predistributed.purge_session(old_session)).receipt_id == old_receipt.receipt_id + overlap_receipt = await predistributed.purge_session(overlap_session) + overlap_envelope = json.loads( + (receipts_dir / f"{overlap_receipt.receipt_id}.json").read_text(encoding="utf-8") + ) + assert overlap_envelope["key_id"] == "k1" + + # Phase 2: K2 signs while K1 remains available for verification. + promoted = _coordinator( + store, + receipts_dir, + current="k2", + keys={"k2": K2, "k1": K1}, + ) + assert (await promoted.purge_session(old_session)).receipt_id == old_receipt.receipt_id + assert ( + await promoted.purge_session(overlap_session) + ).receipt_id == overlap_receipt.receipt_id + new_receipt = await promoted.purge_session(new_session) + new_envelope = json.loads( + (receipts_dir / f"{new_receipt.receipt_id}.json").read_text(encoding="utf-8") + ) + assert new_envelope["key_id"] == "k2" + + restarted_overlap = _coordinator( + store, + receipts_dir, + current="k2", + keys={"k2": K2, "k1": K1}, + ) + assert (await restarted_overlap.purge_session(old_session)).state is LifecycleState.VERIFIED + assert (await restarted_overlap.purge_session(new_session)).state is LifecycleState.VERIFIED + + # Phase 3 is safe only after K1's receipt horizon. Removing it sooner is + # deliberately fail-closed, while K2 receipts continue to verify. + retired = _coordinator(store, receipts_dir, current="k2", keys={"k2": K2}) + assert retired.state_for(old_session) is LifecycleState.FAILED + with pytest.raises(RuntimeError, match="no matching trusted receipt"): + await retired.purge_session(old_session) + assert (await retired.purge_session(new_session)).receipt_id == new_receipt.receipt_id diff --git a/backend/tests/test_lifecycle_receipt_security.py b/backend/tests/test_lifecycle_receipt_security.py new file mode 100644 index 0000000..053923d --- /dev/null +++ b/backend/tests/test_lifecycle_receipt_security.py @@ -0,0 +1,593 @@ +"""Authenticated deletion receipt and crash-recovery regressions.""" + +from __future__ import annotations + +import asyncio +import base64 +import hashlib +import json +import threading +import time +import uuid + +import pytest + +from backend.agents import workflows as workflows_module +from backend.services import lifecycle as lifecycle_module +from backend.services.lifecycle import DeletionReceipt, LifecycleCoordinator, LifecycleState +from backend.session_store import SessionStore + + +CURRENT_KEY = b"c" * 48 +PREVIOUS_KEY = b"p" * 48 +NEXT_KEY = b"n" * 48 + + +class _Jobs: + @staticmethod + def purge_session(_session_id: str): + return {"jobs_revoked": 0, "threads_pending": 0} + + +class _Workflows: + @staticmethod + def purge_session(_session_id: str): + return 0 + + @staticmethod + def count_session_workflows(_session_id: str): + return 0 + + +@pytest.fixture(autouse=True) +def _workflow_purger(monkeypatch): + monkeypatch.setattr(workflows_module, "get_workflow_engine", lambda: _Workflows()) + + +def _session_id(prefix: str) -> str: + return f"{prefix}-{uuid.uuid4().hex}" + + +def _coordinator(store, receipts_dir, *, keys, current): + return LifecycleCoordinator( + store=store, + job_manager=_Jobs(), + receipts_dir=receipts_dir, + receipt_signing_keys=keys, + current_receipt_key_id=current, + ) + + +@pytest.mark.unit +@pytest.mark.asyncio +async def test_deletion_receipt_is_signed_and_verified_purge_is_idempotent(tmp_path): + store = SessionStore(tmp_path / "sessions.db") + session_id = _session_id("signed") + store.get_or_create_session(session_id) + receipts_dir = tmp_path / "receipts" + coordinator = _coordinator(store, receipts_dir, keys={"current": CURRENT_KEY}, current="current") + + first = await coordinator.purge_session(session_id) + envelope = json.loads(next(receipts_dir.glob("del_*.json")).read_text(encoding="utf-8")) + + assert first.state is LifecycleState.VERIFIED + assert envelope["schema_version"] == "2026-08-deletion-receipt-v1" + assert envelope["algorithm"] == "HMAC-SHA256" + assert envelope["key_id"] == "current" + assert len(envelope["signature"]) == 64 + assert session_id not in json.dumps(envelope) + + restarted = _coordinator(store, receipts_dir, keys={"current": CURRENT_KEY}, current="current") + second = await restarted.purge_session(session_id) + assert second.to_dict() == first.to_dict() + assert len(list(receipts_dir.glob("del_*.json"))) == 1 + + +@pytest.mark.unit +@pytest.mark.asyncio +async def test_tampered_or_unknown_key_receipts_are_never_trusted(tmp_path): + store = SessionStore(tmp_path / "sessions.db") + session_id = _session_id("tamper") + store.get_or_create_session(session_id) + receipts_dir = tmp_path / "receipts" + coordinator = _coordinator(store, receipts_dir, keys={"current": CURRENT_KEY}, current="current") + await coordinator.purge_session(session_id) + receipt_path = next(receipts_dir.glob("del_*.json")) + original = json.loads(receipt_path.read_text(encoding="utf-8")) + + tampered = json.loads(json.dumps(original)) + tampered["receipt"]["database_deleted"] = False + receipt_path.write_text(json.dumps(tampered), encoding="utf-8") + tampered_restart = _coordinator( + store, + receipts_dir, + keys={"current": CURRENT_KEY}, + current="current", + ) + assert tampered_restart.state_for(session_id) is LifecycleState.FAILED + with pytest.raises(RuntimeError, match="no matching trusted receipt"): + await tampered_restart.purge_session(session_id) + + receipt_path.write_text(json.dumps(original), encoding="utf-8") + unknown_restart = _coordinator( + store, + receipts_dir, + keys={"different": NEXT_KEY}, + current="different", + ) + assert unknown_restart.state_for(session_id) is LifecycleState.FAILED + with pytest.raises(RuntimeError, match="no matching trusted receipt"): + await unknown_restart.purge_session(session_id) + + +@pytest.mark.unit +@pytest.mark.asyncio +async def test_receipt_key_rotation_verifies_previous_and_signs_new_receipts(tmp_path): + store = SessionStore(tmp_path / "sessions.db") + old_session = _session_id("old-key") + new_session = _session_id("new-key") + store.get_or_create_session(old_session) + receipts_dir = tmp_path / "receipts" + old = _coordinator(store, receipts_dir, keys={"previous": PREVIOUS_KEY}, current="previous") + old_receipt = await old.purge_session(old_session) + + rotated = _coordinator( + store, + receipts_dir, + keys={"next": NEXT_KEY, "previous": PREVIOUS_KEY}, + current="next", + ) + assert (await rotated.purge_session(old_session)).receipt_id == old_receipt.receipt_id + store.get_or_create_session(new_session) + new_receipt = await rotated.purge_session(new_session) + envelopes = { + path.stem: json.loads(path.read_text(encoding="utf-8")) + for path in receipts_dir.glob("del_*.json") + } + assert envelopes[old_receipt.receipt_id]["key_id"] == "previous" + assert envelopes[new_receipt.receipt_id]["key_id"] == "next" + + +@pytest.mark.unit +@pytest.mark.asyncio +async def test_environment_receipt_key_rotation_decodes_current_and_previous_keys(tmp_path, monkeypatch): + store = SessionStore(tmp_path / "sessions.db") + old_session = _session_id("env-old-key") + new_session = _session_id("env-new-key") + store.get_or_create_session(old_session) + receipts_dir = tmp_path / "receipts" + + def encode(value: bytes) -> str: + return base64.urlsafe_b64encode(value).decode("ascii").rstrip("=") + + monkeypatch.setenv("EXCEL_AGENT_LIFECYCLE_RECEIPT_KEY_ID", "previous") + monkeypatch.setenv("EXCEL_AGENT_LIFECYCLE_RECEIPT_HMAC_KEY", encode(PREVIOUS_KEY)) + monkeypatch.delenv("EXCEL_AGENT_LIFECYCLE_RECEIPT_PREVIOUS_KEYS", raising=False) + old = LifecycleCoordinator(store=store, job_manager=_Jobs(), receipts_dir=receipts_dir) + old_receipt = await old.purge_session(old_session) + + monkeypatch.setenv("EXCEL_AGENT_LIFECYCLE_RECEIPT_KEY_ID", "next") + monkeypatch.setenv("EXCEL_AGENT_LIFECYCLE_RECEIPT_HMAC_KEY", encode(NEXT_KEY)) + monkeypatch.setenv( + "EXCEL_AGENT_LIFECYCLE_RECEIPT_PREVIOUS_KEYS", + json.dumps({"previous": encode(PREVIOUS_KEY)}), + ) + rotated = LifecycleCoordinator(store=store, job_manager=_Jobs(), receipts_dir=receipts_dir) + assert (await rotated.purge_session(old_session)).receipt_id == old_receipt.receipt_id + + store.get_or_create_session(new_session) + new_receipt = await rotated.purge_session(new_session) + new_envelope = json.loads((receipts_dir / f"{new_receipt.receipt_id}.json").read_text(encoding="utf-8")) + assert new_envelope["key_id"] == "next" + + +@pytest.mark.unit +@pytest.mark.asyncio +async def test_purging_receipt_recovers_after_database_delete_and_restart(tmp_path): + store = SessionStore(tmp_path / "sessions.db") + session_id = _session_id("post-delete-crash") + store.get_or_create_session(session_id) + receipts_dir = tmp_path / "receipts" + coordinator = _coordinator(store, receipts_dir, keys={"current": CURRENT_KEY}, current="current") + receipt = DeletionReceipt( + receipt_id=f"del_{uuid.uuid4().hex}", + session_id_hash=hashlib.sha256(session_id.encode("utf-8")).hexdigest(), + state=LifecycleState.PURGING, + started_at=time.time(), + ) + coordinator._persist_receipt(receipt) + store.set_session_lifecycle_state( + session_id, + "purging", + receipt_id=receipt.receipt_id, + started_at=receipt.started_at, + ) + assert store.delete_session(session_id) is True + + restarted = _coordinator(store, receipts_dir, keys={"current": CURRENT_KEY}, current="current") + recovered = await restarted.purge_session(session_id) + + assert recovered.receipt_id == receipt.receipt_id + assert recovered.state is LifecycleState.VERIFIED + assert recovered.database_deleted is True + assert store.get_session_lifecycle_state(session_id) == "verified" + assert (await restarted.purge_session(session_id)).to_dict() == recovered.to_dict() + + +@pytest.mark.unit +@pytest.mark.asyncio +async def test_verified_receipt_reconciles_crash_before_database_final_state(tmp_path): + store = SessionStore(tmp_path / "sessions.db") + session_id = _session_id("pre-final-state-crash") + store.get_or_create_session(session_id) + receipts_dir = tmp_path / "receipts" + coordinator = _coordinator(store, receipts_dir, keys={"current": CURRENT_KEY}, current="current") + receipt = DeletionReceipt( + receipt_id=f"del_{uuid.uuid4().hex}", + session_id_hash=hashlib.sha256(session_id.encode("utf-8")).hexdigest(), + state=LifecycleState.VERIFIED, + started_at=time.time(), + completed_at=time.time(), + database_deleted=True, + ) + coordinator._persist_receipt(receipt) + store.set_session_lifecycle_state( + session_id, + "purging", + receipt_id=receipt.receipt_id, + started_at=receipt.started_at, + ) + assert store.delete_session(session_id) is True + + restarted = _coordinator(store, receipts_dir, keys={"current": CURRENT_KEY}, current="current") + recovered = await restarted.purge_session(session_id) + + assert recovered.to_dict() == receipt.to_dict() + assert store.get_session_lifecycle_state(session_id) == "verified" + + +@pytest.mark.unit +@pytest.mark.asyncio +async def test_durable_receipt_id_wins_over_newer_losing_concurrent_receipt(tmp_path): + store = SessionStore(tmp_path / "sessions.db") + session_id = _session_id("concurrent-receipt") + store.get_or_create_session(session_id) + receipts_dir = tmp_path / "receipts" + coordinator = _coordinator(store, receipts_dir, keys={"current": CURRENT_KEY}, current="current") + durable_receipt = DeletionReceipt( + receipt_id=f"del_{uuid.uuid4().hex}", + session_id_hash=hashlib.sha256(session_id.encode("utf-8")).hexdigest(), + state=LifecycleState.PURGING, + started_at=time.time(), + ) + coordinator._persist_receipt(durable_receipt) + store.set_session_lifecycle_state( + session_id, + "purging", + receipt_id=durable_receipt.receipt_id, + started_at=durable_receipt.started_at, + ) + losing_receipt = DeletionReceipt( + receipt_id=f"del_{uuid.uuid4().hex}", + session_id_hash=durable_receipt.session_id_hash, + state=LifecycleState.FAILED, + started_at=durable_receipt.started_at + 1, + completed_at=time.time() + 1, + errors=["lifecycle_state_persist_failed:RuntimeError"], + ) + coordinator._persist_receipt(losing_receipt) + + restarted = _coordinator(store, receipts_dir, keys={"current": CURRENT_KEY}, current="current") + recovered = await restarted.purge_session(session_id) + + assert recovered.receipt_id == durable_receipt.receipt_id + assert recovered.state is LifecycleState.VERIFIED + assert store.get_session_lifecycle_state(session_id) == "verified" + + +@pytest.mark.unit +@pytest.mark.asyncio +async def test_signed_receipt_cannot_replace_a_missing_durable_tombstone(tmp_path): + store = SessionStore(tmp_path / "sessions.db") + session_id = _session_id("missing-tombstone") + store.get_or_create_session(session_id) + receipts_dir = tmp_path / "receipts" + coordinator = _coordinator(store, receipts_dir, keys={"current": CURRENT_KEY}, current="current") + await coordinator.purge_session(session_id) + with store._conn() as connection: + connection.execute("DELETE FROM session_lifecycle WHERE session_id = ?", (session_id,)) + + restarted = _coordinator(store, receipts_dir, keys={"current": CURRENT_KEY}, current="current") + with pytest.raises(RuntimeError, match="no durable lifecycle tombstone"): + await restarted.purge_session(session_id) + + +@pytest.mark.unit +@pytest.mark.asyncio +async def test_concurrent_resume_cannot_downgrade_verified_receipt(tmp_path): + store = SessionStore(tmp_path / "sessions.db") + session_id = _session_id("concurrent-resume") + store.get_or_create_session(session_id) + receipts_dir = tmp_path / "receipts" + seed = _coordinator(store, receipts_dir, keys={"current": CURRENT_KEY}, current="current") + receipt = DeletionReceipt( + receipt_id=f"del_{uuid.uuid4().hex}", + session_id_hash=hashlib.sha256(session_id.encode("utf-8")).hexdigest(), + state=LifecycleState.PURGING, + started_at=time.time(), + ) + seed._persist_receipt(receipt) + store.set_session_lifecycle_state( + session_id, + "purging", + receipt_id=receipt.receipt_id, + started_at=receipt.started_at, + ) + + slow_entered = threading.Event() + release_slow = threading.Event() + + class SlowJobs: + @staticmethod + def purge_session(_session_id: str): + slow_entered.set() + assert release_slow.wait(timeout=5) + return {"jobs_revoked": 0, "threads_pending": 0} + + slow = LifecycleCoordinator( + store=store, + job_manager=SlowJobs(), + receipts_dir=receipts_dir, + receipt_signing_keys={"current": CURRENT_KEY}, + current_receipt_key_id="current", + ) + fast = _coordinator(store, receipts_dir, keys={"current": CURRENT_KEY}, current="current") + slow_task = asyncio.create_task(slow.purge_session(session_id)) + assert await asyncio.to_thread(slow_entered.wait, 2) + + winner = await fast.purge_session(session_id) + release_slow.set() + loser = await asyncio.wait_for(slow_task, timeout=5) + + assert winner.state is LifecycleState.VERIFIED + assert loser.to_dict() == winner.to_dict() + restarted = _coordinator(store, receipts_dir, keys={"current": CURRENT_KEY}, current="current") + assert (await restarted.purge_session(session_id)).to_dict() == winner.to_dict() + + +@pytest.mark.unit +@pytest.mark.asyncio +async def test_idle_replica_refreshes_cached_purging_receipt_after_peer_verifies(tmp_path): + store = SessionStore(tmp_path / "sessions.db") + session_id = _session_id("idle-replica") + store.get_or_create_session(session_id) + receipts_dir = tmp_path / "receipts" + stale = _coordinator(store, receipts_dir, keys={"current": CURRENT_KEY}, current="current") + receipt = DeletionReceipt( + receipt_id=f"del_{uuid.uuid4().hex}", + session_id_hash=hashlib.sha256(session_id.encode("utf-8")).hexdigest(), + state=LifecycleState.PURGING, + started_at=time.time(), + ) + stale._persist_receipt(receipt) + store.set_session_lifecycle_state( + session_id, + "purging", + receipt_id=receipt.receipt_id, + started_at=receipt.started_at, + ) + winner = _coordinator(store, receipts_dir, keys={"current": CURRENT_KEY}, current="current") + + completed = await winner.purge_session(session_id) + retried = await stale.purge_session(session_id) + + assert completed.state is LifecycleState.VERIFIED + assert retried.to_dict() == completed.to_dict() + assert store.get_session_lifecycle_state(session_id) == "verified" + + +@pytest.mark.unit +@pytest.mark.asyncio +async def test_idle_replica_terminal_retry_revokes_its_local_tasks_and_purgers(tmp_path): + store = SessionStore(tmp_path / "sessions.db") + session_id = _session_id("idle-replica-local-state") + store.get_or_create_session(session_id) + receipts_dir = tmp_path / "receipts" + stale = _coordinator(store, receipts_dir, keys={"current": CURRENT_KEY}, current="current") + entered = asyncio.Event() + canceled = asyncio.Event() + volatile = {"secret": "private replica cache"} + purger_called = asyncio.Event() + + async def local_work(): + async with stale.active_task(session_id): + entered.set() + try: + await asyncio.Future() + except asyncio.CancelledError: + canceled.set() + raise + + def purge_volatile(_session_id: str): + volatile.clear() + purger_called.set() + return {"entries_evicted": 1} + + stale.register_purger("volatile", purge_volatile) + local_task = asyncio.create_task(local_work()) + await asyncio.wait_for(entered.wait(), timeout=2) + receipt = DeletionReceipt( + receipt_id=f"del_{uuid.uuid4().hex}", + session_id_hash=hashlib.sha256(session_id.encode("utf-8")).hexdigest(), + state=LifecycleState.PURGING, + started_at=time.time(), + ) + stale._persist_receipt(receipt) + store.set_session_lifecycle_state( + session_id, + "purging", + receipt_id=receipt.receipt_id, + started_at=receipt.started_at, + ) + winner = _coordinator(store, receipts_dir, keys={"current": CURRENT_KEY}, current="current") + + completed = await winner.purge_session(session_id) + retried = await stale.purge_session(session_id) + + assert retried.to_dict() == completed.to_dict() + assert canceled.is_set() + assert purger_called.is_set() + assert volatile == {} + with pytest.raises(asyncio.CancelledError): + await local_task + + +@pytest.mark.unit +def test_receipt_replace_fsyncs_parent_directory(tmp_path, monkeypatch): + store = SessionStore(tmp_path / "sessions.db") + receipts_dir = tmp_path / "receipts" + coordinator = _coordinator(store, receipts_dir, keys={"current": CURRENT_KEY}, current="current") + receipt = DeletionReceipt( + receipt_id=f"del_{uuid.uuid4().hex}", + session_id_hash=hashlib.sha256(b"fsync-parent").hexdigest(), + state=LifecycleState.PURGING, + started_at=time.time(), + ) + synced: list[object] = [] + monkeypatch.setattr(lifecycle_module, "_fsync_directory", lambda path: synced.append(path)) + + coordinator._persist_receipt(receipt) + + assert synced == [receipts_dir] + + +@pytest.mark.unit +def test_receipt_directory_fsync_failure_is_fail_closed(tmp_path, monkeypatch): + receipts_dir = tmp_path / "receipts" + coordinator = _coordinator(None, receipts_dir, keys={"current": CURRENT_KEY}, current="current") + receipt = DeletionReceipt( + receipt_id=f"del_{uuid.uuid4().hex}", + session_id_hash=hashlib.sha256(b"fsync-failure").hexdigest(), + state=LifecycleState.PURGING, + started_at=time.time(), + ) + + def fail_fsync(_path): + raise OSError("simulated directory fsync failure") + + monkeypatch.setattr(lifecycle_module, "_fsync_directory", fail_fsync) + + with pytest.raises(OSError, match="directory fsync failure"): + coordinator._persist_receipt(receipt) + + assert (receipt.session_id_hash, receipt.receipt_id) not in coordinator._receipts_by_identity + + +@pytest.mark.unit +@pytest.mark.asyncio +async def test_verified_replace_fsync_failure_retries_without_durable_downgrade( + tmp_path, + monkeypatch, +): + store = SessionStore(tmp_path / "sessions.db") + session_id = _session_id("verified-fsync-failure") + store.get_or_create_session(session_id) + receipts_dir = tmp_path / "receipts" + coordinator = _coordinator(store, receipts_dir, keys={"current": CURRENT_KEY}, current="current") + real_fsync_directory = lifecycle_module._fsync_directory + calls = 0 + + def fail_verified_fsync(path): + nonlocal calls + calls += 1 + if calls == 2: + raise OSError("simulated verified directory fsync failure") + real_fsync_directory(path) + + monkeypatch.setattr(lifecycle_module, "_fsync_directory", fail_verified_fsync) + + with pytest.raises(RuntimeError, match="final deletion receipt"): + await coordinator.purge_session(session_id) + + envelope = json.loads(next(receipts_dir.glob("del_*.json")).read_text(encoding="utf-8")) + assert envelope["receipt"]["state"] == "verified" + assert store.get_session_lifecycle_state(session_id) == "purging" + + monkeypatch.setattr(lifecycle_module, "_fsync_directory", real_fsync_directory) + restarted = _coordinator(store, receipts_dir, keys={"current": CURRENT_KEY}, current="current") + receipt = await restarted.purge_session(session_id) + + assert receipt.state is LifecycleState.VERIFIED + assert store.get_session_lifecycle_state(session_id) == "verified" + + +@pytest.mark.unit +@pytest.mark.asyncio +async def test_verified_commit_ack_loss_reconciles_without_receipt_downgrade(tmp_path): + delegate = SessionStore(tmp_path / "sessions.db") + session_id = _session_id("verified-ack-loss") + delegate.get_or_create_session(session_id) + receipts_dir = tmp_path / "receipts" + + class CommitThenRaiseStore: + raised = False + + def __getattr__(self, name): + return getattr(delegate, name) + + def set_session_lifecycle_state(self, session_id: str, state: str, **kwargs) -> None: + delegate.set_session_lifecycle_state(session_id, state, **kwargs) + if state == "verified" and not self.raised: + self.raised = True + raise RuntimeError("simulated acknowledgement loss after commit") + + coordinator = _coordinator( + CommitThenRaiseStore(), + receipts_dir, + keys={"current": CURRENT_KEY}, + current="current", + ) + receipt = await coordinator.purge_session(session_id) + + assert receipt.state is LifecycleState.VERIFIED + assert delegate.get_session_lifecycle_state(session_id) == "verified" + envelope = json.loads(next(receipts_dir.glob("del_*.json")).read_text(encoding="utf-8")) + assert envelope["receipt"]["state"] == "verified" + restarted = _coordinator(delegate, receipts_dir, keys={"current": CURRENT_KEY}, current="current") + assert (await restarted.purge_session(session_id)).to_dict() == receipt.to_dict() + + +@pytest.mark.unit +def test_unsigned_legacy_receipt_is_quarantined_and_remains_fail_closed(tmp_path): + session_id = _session_id("legacy-unsigned") + session_hash = hashlib.sha256(session_id.encode("utf-8")).hexdigest() + receipts_dir = tmp_path / "receipts" + receipts_dir.mkdir() + legacy_path = receipts_dir / f"del_{uuid.uuid4().hex}.json" + legacy_path.write_text( + json.dumps( + { + "receipt_id": legacy_path.stem, + "session_id_hash": session_hash, + "state": "verified", + "started_at": time.time(), + "completed_at": time.time(), + "database_deleted": True, + "registered_purgers": {}, + "errors": [], + } + ), + encoding="utf-8", + ) + + first = _coordinator(None, receipts_dir, keys={"current": CURRENT_KEY}, current="current") + quarantine = receipts_dir / "quarantine" / "legacy-unsigned" + + assert first.state_for(session_id) is LifecycleState.FAILED + assert not legacy_path.exists() + assert len(list(quarantine.glob("del_*.json"))) == 1 + assert (quarantine / "RECOVERY.txt").is_file() + + restarted = _coordinator(None, receipts_dir, keys={"current": CURRENT_KEY}, current="current") + assert restarted.state_for(session_id) is LifecycleState.FAILED + assert len(list(quarantine.glob("del_*.json"))) == 1 diff --git a/backend/tests/test_llm_cache_scope.py b/backend/tests/test_llm_cache_scope.py new file mode 100644 index 0000000..770c403 --- /dev/null +++ b/backend/tests/test_llm_cache_scope.py @@ -0,0 +1,77 @@ +from __future__ import annotations + +import asyncio + +from backend.llm.base import LLMMessage, LLMResponse +from backend.llm.router import Complexity, LLMRouter + + +class _Provider: + async def health_check(self) -> bool: + return True + + async def generate( + self, + _messages, + model=None, + _temperature=0.1, + _max_tokens=4096, + _json_mode=False, + ) -> LLMResponse: + return LLMResponse(content="fresh", model=model or "local", provider="lmstudio") + + +class _ForbiddenCache: + def get(self, *_args, **_kwargs): + raise AssertionError("release-disabled cache must not be read") + + def put(self, *_args, **_kwargs): + raise AssertionError("release-disabled cache must not be written") + + +def test_release_disabled_exact_cache_is_never_read_or_written(monkeypatch): + monkeypatch.setattr("backend.llm.router.LLM_EXACT_CACHE_ENABLED", False, raising=False) + monkeypatch.setattr("backend.llm.router.get_llm_cache", lambda: _ForbiddenCache()) + monkeypatch.setenv("AI_ALLOWED_PROVIDERS", "lmstudio") + monkeypatch.setenv("AI_EXTERNAL_SENDING_ALLOWED", "false") + monkeypatch.setattr("backend.api.security.ai_policy._load_enterprise_provider_policy", lambda: {}) + router = LLMRouter() + router.providers = {"lmstudio": _Provider()} + router.routing_table[Complexity.SIMPLE] = [("lmstudio", "local")] + + response = asyncio.run( + router.generate( + [ + LLMMessage(role="system", content="system"), + LLMMessage(role="user", content="private workbook context"), + ], + complexity=Complexity.SIMPLE, + ) + ) + + assert response.content == "fresh" + + +def test_multi_turn_requests_never_use_unscoped_exact_cache(monkeypatch): + monkeypatch.setattr("backend.llm.router.LLM_EXACT_CACHE_ENABLED", True, raising=False) + monkeypatch.setattr("backend.llm.router.get_llm_cache", lambda: _ForbiddenCache()) + monkeypatch.setenv("AI_ALLOWED_PROVIDERS", "lmstudio") + monkeypatch.setenv("AI_EXTERNAL_SENDING_ALLOWED", "false") + monkeypatch.setattr("backend.api.security.ai_policy._load_enterprise_provider_policy", lambda: {}) + router = LLMRouter() + router.providers = {"lmstudio": _Provider()} + router.routing_table[Complexity.SIMPLE] = [("lmstudio", "local")] + + response = asyncio.run( + router.generate( + [ + LLMMessage(role="system", content="system"), + LLMMessage(role="user", content="first turn"), + LLMMessage(role="assistant", content="answer"), + LLMMessage(role="user", content="second turn"), + ], + complexity=Complexity.SIMPLE, + ) + ) + + assert response.content == "fresh" diff --git a/backend/tests/test_local_pairing.py b/backend/tests/test_local_pairing.py new file mode 100644 index 0000000..61dc6d4 --- /dev/null +++ b/backend/tests/test_local_pairing.py @@ -0,0 +1,358 @@ +"""Security regressions for loopback-only desktop client pairing.""" + +from __future__ import annotations + +import asyncio +import base64 +import re +import sys +from concurrent.futures import ThreadPoolExecutor +from types import SimpleNamespace + +import pytest +from fastapi import FastAPI, Request +from fastapi.testclient import TestClient + +from backend.api.middleware.auth import AuthMiddleware +from backend.api.middleware.rate_limiter import RateLimiterMiddleware +from backend.api.security.local_pairing import ( + KeyringSecretLoader, + LOCAL_PAIRING_SCHEMA, + LocalPairingError, + LocalPairingManager, + is_loopback_peer, +) +from backend.api.security.rbac import RBACMiddleware +from backend.config import runtime_config_errors +from backend import main as main_module + + +NONCE = base64.urlsafe_b64encode(b"n" * 32).rstrip(b"=").decode("ascii") + + +class _Clock: + def __init__(self) -> None: + self.now = 1_000.0 + + def __call__(self) -> float: + return self.now + + +def _manager( + clock: _Clock | None = None, + *, + code: str = "PAIR_123456", + max_attempts: int = 3, + code_ttl_seconds: int = 120, + token_ttl_seconds: int = 300, +) -> LocalPairingManager: + return LocalPairingManager( + enabled=True, + production_mode=False, + pairing_code=code, + code_ttl_seconds=code_ttl_seconds, + token_ttl_seconds=token_ttl_seconds, + max_attempts=max_attempts, + lockout_seconds=60, + secret_loader=lambda: b"s" * 32, + clock=clock or _Clock(), + ) + + +@pytest.mark.unit +def test_pairing_code_is_one_time_and_issues_short_lived_nonce_bound_opaque_token(): + clock = _Clock() + manager = _manager(clock) + + grant = manager.exchange_code("PAIR_123456", NONCE) + + assert grant["schema_version"] == LOCAL_PAIRING_SCHEMA + assert grant["token_type"] == "Bearer" + assert grant["expires_in"] == 300 + assert grant["client_nonce"] == NONCE + assert grant["subject"] == "local-owner" + assert re.fullmatch(r"local\.[A-Za-z0-9_-]+\.[A-Za-z0-9_-]+", grant["access_token"]) + assert "PAIR_123456" not in grant["access_token"] + assert NONCE not in grant["access_token"] + actor = manager.authenticate(grant["access_token"]) + assert actor == { + "user_id": "local-owner", + "email": "local-owner@localhost", + "name": "Desktop owner", + "role": "admin", + "auth_source": "local_pairing", + "client_nonce_sha256": actor["client_nonce_sha256"], + } + assert len(actor["client_nonce_sha256"]) == 64 + + with pytest.raises(LocalPairingError) as reused: + manager.exchange_code("PAIR_123456", NONCE) + assert reused.value.code == "PAIRING_REJECTED" + + clock.now += 301 + with pytest.raises(LocalPairingError) as expired: + manager.authenticate(grant["access_token"]) + assert expired.value.code == "LOCAL_TOKEN_INVALID" + + +@pytest.mark.unit +def test_concurrent_pairing_exchange_has_exactly_one_winner_and_restart_revokes_token(): + secret = b"s" * 32 + manager = LocalPairingManager( + enabled=True, + production_mode=False, + pairing_code="PAIR_123456", + secret_loader=lambda: secret, + ) + manager.ensure_ready() + + def exchange(): + try: + return manager.exchange_code("PAIR_123456", NONCE)["access_token"] + except LocalPairingError: + return None + + with ThreadPoolExecutor(max_workers=8) as executor: + results = list(executor.map(lambda _index: exchange(), range(8))) + + issued = [token for token in results if token] + assert len(issued) == 1 + assert manager.authenticate(issued[0])["user_id"] == "local-owner" + + restarted = LocalPairingManager( + enabled=True, + production_mode=False, + pairing_code="PAIR_NEW_123", + secret_loader=lambda: secret, + ) + with pytest.raises(LocalPairingError) as invalid_after_restart: + restarted.authenticate(issued[0]) + assert invalid_after_restart.value.code == "LOCAL_TOKEN_INVALID" + + +@pytest.mark.unit +def test_keyring_secret_loader_persists_and_verifies_os_vault_secret(monkeypatch): + class FakeOSKeyring: + __module__ = "keyring.backends.macOS" + priority = 10 + + values: dict[tuple[str, str], str] = {} + fake_keyring = SimpleNamespace( + get_keyring=FakeOSKeyring, + get_password=lambda service, account: values.get((service, account)), + set_password=lambda service, account, value: values.__setitem__((service, account), value), + ) + monkeypatch.setitem(sys.modules, "keyring", fake_keyring) + loader = KeyringSecretLoader(service="test-service", account="test-account") + + first = loader() + second = loader() + + assert first == second + assert len(first) == 48 + assert values[("test-service", "test-account")] + + fake_keyring.get_keyring = lambda: SimpleNamespace(priority=10) + with pytest.raises(LocalPairingError) as insecure_backend: + loader() + assert insecure_backend.value.code == "PAIRING_KEYCHAIN_UNAVAILABLE" + + +@pytest.mark.unit +def test_pairing_attempts_expiry_and_keychain_failures_are_fail_closed(): + clock = _Clock() + manager = _manager(clock, max_attempts=2) + + with pytest.raises(LocalPairingError) as first: + manager.exchange_code("WRONG_123", NONCE) + assert first.value.status_code == 401 + with pytest.raises(LocalPairingError) as locked: + manager.exchange_code("WRONG_456", NONCE) + assert locked.value.status_code == 429 + assert locked.value.code == "PAIRING_LOCKED" + with pytest.raises(LocalPairingError) as still_locked: + manager.exchange_code("PAIR_123456", NONCE) + assert still_locked.value.code == "PAIRING_LOCKED" + + expiring = _manager(clock, code_ttl_seconds=30) + expiring.ensure_ready() + clock.now += 31 + with pytest.raises(LocalPairingError) as expired: + expiring.exchange_code("PAIR_123456", NONCE) + assert expired.value.code == "PAIRING_REJECTED" + + unavailable = LocalPairingManager( + enabled=True, + production_mode=False, + pairing_code="PAIR_123456", + secret_loader=lambda: (_ for _ in ()).throw(RuntimeError("no keychain")), + ) + with pytest.raises(LocalPairingError) as keychain: + unavailable.ensure_ready() + assert keychain.value.code == "PAIRING_KEYCHAIN_UNAVAILABLE" + + production = LocalPairingManager( + enabled=True, + production_mode=True, + pairing_code="PAIR_123456", + secret_loader=lambda: b"s" * 32, + ) + with pytest.raises(LocalPairingError) as disabled: + production.ensure_ready() + assert disabled.value.code == "LOCAL_PAIRING_DISABLED" + + +@pytest.mark.unit +def test_loopback_peer_validation_ignores_host_headers_and_forwarded_addresses(): + for host in ("127.0.0.1", "127.42.1.9", "::1", "::ffff:127.0.0.1"): + assert is_loopback_peer(host) is True + for host in ("localhost", "testclient", "0.0.0.0", "10.0.0.3", "example.com", ""): + assert is_loopback_peer(host) is False + + +@pytest.mark.unit +def test_local_bearer_is_authoritative_and_hydrates_http_actor(): + manager = _manager() + token = manager.exchange_code("PAIR_123456", NONCE)["access_token"] + + class OIDC: + async def authenticate(self, _token: str): + raise AssertionError("opaque local token must not be sent to OIDC") + + app = FastAPI() + + @app.get("/api/v1/me") + async def me(request: Request): + return request.state.enterprise_user + + app.add_middleware( + AuthMiddleware, + api_key="service-key", + enabled=True, + static_key_admin=True, + oidc_authenticator=OIDC(), + local_pairing_authenticator=manager, + ) + client = TestClient(app) + + response = client.get("/api/v1/me", headers={"Authorization": f"Bearer {token}"}) + assert response.status_code == 200 + assert response.json()["user_id"] == "local-owner" + assert response.json()["auth_source"] == "local_pairing" + + rejected = client.get( + "/api/v1/me", + headers={"Authorization": "Bearer local.invalid.token", "X-API-Key": "service-key"}, + ) + assert rejected.status_code == 401 + assert rejected.json()["error"]["code"] == "AUTH_FAILED" + + +@pytest.mark.unit +def test_pairing_endpoint_is_loopback_only_nonce_echoing_and_no_store(monkeypatch): + manager = _manager() + monkeypatch.setattr(main_module, "local_pairing_manager", manager) + body = { + "schema_version": LOCAL_PAIRING_SCHEMA, + "pairing_code": "PAIR_123456", + "client_nonce": NONCE, + } + + client = TestClient(main_module.app, client=("127.0.0.1", 50100)) + rejected = client.post( + "/auth/local/pair", + json={**body, "pairing_code": "WRONG_123456"}, + headers={"Accept": "application/json"}, + ) + assert rejected.status_code == 401 + assert rejected.headers["cache-control"] == "no-store" + assert rejected.headers["pragma"] == "no-cache" + + response = client.post( + "/auth/local/pair", + json=body, + headers={"Accept": "application/json"}, + ) + + assert response.status_code == 200 + assert response.headers["cache-control"] == "no-store" + assert response.headers["pragma"] == "no-cache" + payload = response.json() + assert payload["schema_version"] == LOCAL_PAIRING_SCHEMA + assert payload["client_nonce"] == NONCE + assert payload["token_type"] == "Bearer" + assert 30 <= payload["expires_in"] <= 3_600 + assert re.fullmatch(r"local\.[A-Za-z0-9._~-]+", payload["access_token"]) + + remote = TestClient(main_module.app, client=("10.0.0.8", 50100)).post( + "/auth/local/pair", + json=body, + ) + assert remote.status_code == 403 + assert remote.headers["cache-control"] == "no-store" + + +@pytest.mark.unit +def test_public_pairing_endpoint_remains_rate_limited_and_bypasses_rbac(): + app = FastAPI() + + @app.post("/auth/local/pair") + async def pair(): + return {"ok": True} + + app.add_middleware(RBACMiddleware, enabled=True) + app.add_middleware( + RateLimiterMiddleware, + default_rpm=60, + endpoint_limits={"/auth/local/pair": 1}, + ) + client = TestClient(app, client=("127.0.0.1", 50100)) + + assert client.post("/auth/local/pair").status_code == 200 + limited = client.post("/auth/local/pair") + assert limited.status_code == 429 + assert limited.headers["cache-control"] == "no-store" + + +@pytest.mark.unit +def test_local_bearer_is_accepted_for_websocket_actor_resolution(monkeypatch): + manager = _manager() + token = manager.exchange_code("PAIR_123456", NONCE)["access_token"] + monkeypatch.setattr(main_module, "local_pairing_manager", manager) + + actor = asyncio.run(main_module._resolve_ws_credential_actor(("bearer", token))) + + assert actor["user_id"] == "local-owner" + assert actor["auth_source"] == "local_pairing" + assert asyncio.run(main_module._resolve_ws_credential_actor(("bearer", "local.invalid.token"))) is None + + +@pytest.mark.unit +def test_production_and_non_loopback_local_pairing_config_is_rejected(): + production_errors = runtime_config_errors( + { + "production_mode": True, + "config_profile": "prod", + "app_env": "prod", + "local_pairing_enabled": True, + } + ) + assert "LOCAL_PAIRING_ENABLED must be false in production profiles" in production_errors + + local_errors = runtime_config_errors( + { + "production_mode": False, + "config_profile": "dev", + "app_env": "dev", + "local_pairing_enabled": True, + "local_pairing_code_configured": True, + "local_pairing_host": "0.0.0.0", + "local_pairing_code_ttl_seconds": 120, + "local_pairing_token_ttl_seconds": 300, + "local_pairing_max_attempts": 3, + "local_pairing_lockout_seconds": 60, + "local_pairing_rate_limit_rpm": 3, + "auth_enabled": True, + } + ) + assert "HOST must be loopback-only when LOCAL_PAIRING_ENABLED is true" in local_errors diff --git a/backend/tests/test_managed_object_store.py b/backend/tests/test_managed_object_store.py new file mode 100644 index 0000000..41dc618 --- /dev/null +++ b/backend/tests/test_managed_object_store.py @@ -0,0 +1,901 @@ +from __future__ import annotations + +import asyncio +import base64 +import hashlib +import io +import json +import os +import shutil +import sqlite3 +import stat +import threading +import time +from concurrent.futures import ThreadPoolExecutor +from pathlib import Path +from typing import Any, cast + +import pytest + +from backend.session_store import SessionStore +from backend.services.lifecycle import LifecycleCoordinator, LifecycleState +from backend.storage import managed_object_store as managed_object_store_module +from backend.storage.managed_object_store import ( + EncryptedManagedObjectStore, + ManagedObjectConflictError, + ManagedObjectIntegrityError, + ManagedObjectKind, + ManagedObjectNotFoundError, + ManagedObjectScope, + ManagedObjectStoreConfig, + ManagedObjectStoreDisabledError, + ManagedObjectStoreLimitError, + SQLiteManagedObjectMetadataRepository, +) + + +def _config(root: Path, **overrides: object) -> ManagedObjectStoreConfig: + values: dict[str, object] = { + "enabled": True, + "root": root, + "current_key_id": "key-2026-08", + "current_key": b"k" * 32, + "previous_keys": {}, + "address_key": b"a" * 32, + "max_object_bytes": 1024 * 1024, + "total_bytes": 4 * 1024 * 1024, + "chunk_bytes": 4096, + } + values.update(overrides) + return ManagedObjectStoreConfig(**cast(Any, values)) + + +def _scope(*, tenant: str = "tenant-a", owner: str = "owner-a", session: str = "session-a") -> ManagedObjectScope: + return ManagedObjectScope(tenant_id=tenant, owner_id=owner, session_id=session) + + +def _store(tmp_path: Path, *, config: ManagedObjectStoreConfig | None = None): + session_store = SessionStore(tmp_path / "sessions.db") + session_store.get_or_create_session("session-a", "owner-a") + repository = SQLiteManagedObjectMetadataRepository(session_store.db_path) + object_store = EncryptedManagedObjectStore( + config or _config(tmp_path / "managed"), + repository, + ) + return object_store, session_store, repository + + +def test_disabled_by_default_and_invalid_key_material_fail_closed(tmp_path: Path) -> None: + repository = SQLiteManagedObjectMetadataRepository(tmp_path / "missing.db") + with pytest.raises(ManagedObjectStoreDisabledError): + EncryptedManagedObjectStore( + _config(tmp_path / "managed", enabled=False), + repository, + ) + with pytest.raises(ValueError, match="32-byte"): + _config(tmp_path / "managed", current_key=b"short") + with pytest.raises(ValueError, match="distinct"): + _config( + tmp_path / "managed", + previous_keys={"key-2026-08": b"p" * 32}, + ) + + +def test_content_address_is_scope_bound_and_round_trips_without_existence_oracle(tmp_path: Path) -> None: + store, session_store, _repository = _store(tmp_path) + payload = (b"enterprise workbook evidence\n" * 1000) + os.urandom(31) + + first = store.put_bytes(_scope(), ManagedObjectKind.WORKBOOK, payload) + replay = store.put_bytes(_scope(), ManagedObjectKind.WORKBOOK, payload) + assert replay.to_dict() == first.to_dict() + assert store.get_bytes(_scope(), first.object_id) == payload + ciphertext = next((tmp_path / "managed" / "objects").rglob("*.mobj")).read_bytes() + for forbidden in (payload[:128], b"tenant-a", b"owner-a", b"session-a"): + assert forbidden not in ciphertext + + foreign_scope = _scope(tenant="tenant-b") + foreign = store.put_bytes(foreign_scope, ManagedObjectKind.WORKBOOK, payload) + assert foreign.object_id != first.object_id + with pytest.raises(ManagedObjectNotFoundError, match="not_found"): + store.get_bytes(foreign_scope, first.object_id) + with pytest.raises(ManagedObjectNotFoundError, match="not_found"): + store.get_bytes(_scope(), foreign.object_id) + + assert session_store.get_session_lifecycle_state("session-a") == "active" + + +def test_ciphertext_tampering_is_detected_before_plaintext_release(tmp_path: Path) -> None: + store, _session_store, _repository = _store(tmp_path) + receipt = store.put_bytes(_scope(), ManagedObjectKind.EVIDENCE, b"never release this") + object_file = next((tmp_path / "managed" / "objects").rglob("*.mobj")) + raw = bytearray(object_file.read_bytes()) + raw[-20] ^= 0x40 + object_file.write_bytes(raw) + + with pytest.raises(ManagedObjectIntegrityError, match="integrity_failed"): + store.get_bytes(_scope(), receipt.object_id) + assert list((tmp_path / "managed" / "staging").iterdir()) == [] + + +def test_size_and_total_budgets_are_transactional(tmp_path: Path) -> None: + config = _config(tmp_path / "managed", max_object_bytes=8, total_bytes=12) + store, _session_store, _repository = _store(tmp_path, config=config) + store.put_bytes(_scope(), ManagedObjectKind.DOCUMENT, b"12345678") + with pytest.raises(ManagedObjectStoreLimitError, match="object_limit"): + store.put_bytes(_scope(), ManagedObjectKind.DOCUMENT, b"123456789") + with pytest.raises(ManagedObjectStoreLimitError, match="store_limit"): + store.put_bytes(_scope(), ManagedObjectKind.EVIDENCE, b"12345") + + +def test_lifecycle_tombstone_revokes_reads_and_writes(tmp_path: Path) -> None: + store, session_store, _repository = _store(tmp_path) + receipt = store.put_bytes(_scope(), ManagedObjectKind.DOCUMENT, b"bound") + session_store.set_session_lifecycle_state( + "session-a", + "purging", + receipt_id="delete-a", + started_at=1.0, + ) + with pytest.raises(ManagedObjectNotFoundError, match="not_found"): + store.get_bytes(_scope(), receipt.object_id) + with pytest.raises(ManagedObjectNotFoundError, match="not_found"): + store.put_bytes(_scope(), ManagedObjectKind.DOCUMENT, b"new") + + purge = store.purge_session("session-a") + assert purge == {"objects_deleted": 1, "quarantined_deleted": 0, "residual_files": 0} + assert list((tmp_path / "managed" / "objects").rglob("*.mobj")) == [] + + +def test_purge_requires_a_present_recognized_lifecycle_tombstone(tmp_path: Path) -> None: + store, session_store, repository = _store(tmp_path) + store.put_bytes(_scope(), ManagedObjectKind.EVIDENCE, b"do not purge without a tombstone") + with sqlite3.connect(session_store.db_path) as conn: + conn.execute("DELETE FROM session_lifecycle WHERE session_id = ?", ("session-a",)) + + with pytest.raises(RuntimeError, match="lifecycle tombstone"): + store.purge_session("session-a") + assert repository.stats()["objects"] == 1 + assert len(list((tmp_path / "managed" / "objects").rglob("*.mobj"))) == 1 + + +def test_opt_in_lifecycle_registration_purges_ciphertext_before_database_rows( + tmp_path: Path, + monkeypatch: pytest.MonkeyPatch, +) -> None: + store, session_store, repository = _store(tmp_path) + store.put_bytes(_scope(), ManagedObjectKind.EVIDENCE, b"lifecycle registered") + + class Jobs: + @staticmethod + def purge_session(_session_id: str): + return {"jobs_revoked": 0, "threads_pending": 0} + + class Workflows: + @staticmethod + def purge_session(_session_id: str): + return 0 + + @staticmethod + def count_session_workflows(_session_id: str): + return 0 + + monkeypatch.setattr("backend.agents.workflows.get_workflow_engine", lambda: Workflows()) + coordinator = LifecycleCoordinator( + store=session_store, + job_manager=Jobs(), + receipts_dir=tmp_path / "receipts", + receipt_signing_keys={"test": b"r" * 32}, + current_receipt_key_id="test", + ) + store.register_lifecycle_purger(coordinator) + receipt = asyncio.run(coordinator.purge_session("session-a")) + assert receipt.state is LifecycleState.VERIFIED + assert receipt.registered_purgers["managed_encrypted_objects"] == { + "objects_deleted": 1, + "quarantined_deleted": 0, + "residual_files": 0, + } + assert not session_store.session_exists("session-a") + assert repository.stats()["objects"] == 0 + + +def test_concurrent_replay_is_single_object_and_delete_is_idempotent(tmp_path: Path) -> None: + store, _session_store, _repository = _store(tmp_path) + payload = os.urandom(96_000) + with ThreadPoolExecutor(max_workers=8) as executor: + receipts = list( + executor.map( + lambda _index: store.put_bytes(_scope(), ManagedObjectKind.WORKBOOK, payload), + range(24), + ) + ) + assert len({receipt.object_id for receipt in receipts}) == 1 + assert len(list((tmp_path / "managed" / "objects").rglob("*.mobj"))) == 1 + assert store.delete(_scope(), receipts[0].object_id).deleted + assert store.delete(_scope(), receipts[0].object_id).deleted + with pytest.raises(ManagedObjectNotFoundError, match="not_found"): + store.get_bytes(_scope(), receipts[0].object_id) + + +def test_put_replay_does_not_report_success_for_missing_ciphertext(tmp_path: Path) -> None: + store, _session_store, _repository = _store(tmp_path) + payload = b"durability must be revalidated" + store.put_bytes(_scope(), ManagedObjectKind.EVIDENCE, payload) + next((tmp_path / "managed" / "objects").rglob("*.mobj")).unlink() + + with pytest.raises(ManagedObjectIntegrityError, match="integrity_failed"): + store.put_bytes(_scope(), ManagedObjectKind.EVIDENCE, payload) + + +def test_put_and_lifecycle_tombstone_are_serialized_without_resurrection( + tmp_path: Path, + monkeypatch: pytest.MonkeyPatch, +) -> None: + store, session_store, _repository = _store(tmp_path) + entered = threading.Event() + release = threading.Event() + original_encrypt = store._encrypt_record + + def slow_encrypt(*args, **kwargs): + entered.set() + assert release.wait(timeout=5) + return original_encrypt(*args, **kwargs) + + monkeypatch.setattr(store, "_encrypt_record", slow_encrypt) + with ThreadPoolExecutor(max_workers=2) as executor: + put = executor.submit( + store.put_bytes, + _scope(), + ManagedObjectKind.EVIDENCE, + b"serialize lifecycle", + ) + assert entered.wait(timeout=5) + tombstone = executor.submit( + session_store.set_session_lifecycle_state, + "session-a", + "purging", + receipt_id="concurrent-purge", + started_at=3.0, + ) + time.sleep(0.1) + assert not tombstone.done() + release.set() + receipt = put.result(timeout=5) + tombstone.result(timeout=5) + + with pytest.raises(ManagedObjectNotFoundError, match="not_found"): + store.get_bytes(_scope(), receipt.object_id) + assert store.purge_session("session-a")["residual_files"] == 0 + + +def test_orphan_recovery_quarantines_only_ciphertext_and_purge_removes_it(tmp_path: Path) -> None: + store, session_store, _repository = _store(tmp_path) + store.put_bytes(_scope(), ManagedObjectKind.EVIDENCE, b"plaintext sentinel") + source = next((tmp_path / "managed" / "objects").rglob("*.mobj")) + orphan = source.with_name("f" * 32 + ".mobj") + orphan.write_bytes(source.read_bytes()) + orphan.chmod(0o600) + + report = store.recover() + assert report["orphans_quarantined"] == 1 + quarantined = list((tmp_path / "managed" / "quarantine").rglob("*.mobj")) + assert len(quarantined) == 1 + assert b"plaintext sentinel" not in quarantined[0].read_bytes() + + session_store.set_session_lifecycle_state("session-a", "purging", receipt_id="delete-orphan", started_at=2.0) + purge = store.purge_session("session-a") + assert purge["quarantined_deleted"] == 1 + assert not list((tmp_path / "managed").rglob("*.mobj")) + + +def test_key_rotation_is_resumable_private_and_previous_key_can_verify(tmp_path: Path) -> None: + store, _session_store, repository = _store(tmp_path) + receipt = store.put_bytes(_scope(), ManagedObjectKind.WORKBOOK, b"rotate me" * 1000) + + rotation = store.rotate( + rotation_id="rotation-2026-08-10", + new_key_id="key-2026-09", + new_key=b"n" * 32, + ) + assert rotation["status"] == "completed" + assert rotation["objects_rotated"] == 1 + serialized = json.dumps(rotation, sort_keys=True) + assert "session-a" not in serialized + assert "tenant-a" not in serialized + assert receipt.object_id not in serialized + assert str(tmp_path) not in serialized + assert store.get_bytes(_scope(), receipt.object_id) == b"rotate me" * 1000 + + restarted = EncryptedManagedObjectStore( + _config( + tmp_path / "managed", + current_key_id="key-2026-09", + current_key=b"n" * 32, + previous_keys={"key-2026-08": b"k" * 32}, + ), + repository, + ) + assert restarted.get_bytes(_scope(), receipt.object_id) == b"rotate me" * 1000 + assert ( + restarted.rotate( + rotation_id="rotation-2026-08-10", + new_key_id="key-2026-09", + new_key=b"n" * 32, + ) + == rotation + ) + + +def test_rotation_recovers_file_replaced_before_database_commit( + tmp_path: Path, + monkeypatch: pytest.MonkeyPatch, +) -> None: + store, _session_store, repository = _store(tmp_path) + receipt = store.put_bytes(_scope(), ManagedObjectKind.EVIDENCE, b"ambiguous rotation") + original_rotate = repository.rotate + + def replace_then_fail(old_key_id, new_key_id, rotator): + record = next(record for record in repository.records() if record.object_id == receipt.object_id) + rotator(record) + raise OSError("simulated commit ambiguity") + + monkeypatch.setattr(repository, "rotate", replace_then_fail) + with pytest.raises(OSError, match="ambiguity"): + store.rotate( + rotation_id="ambiguous-rotation", + new_key_id="key-2026-09", + new_key=b"n" * 32, + ) + monkeypatch.setattr(repository, "rotate", original_rotate) + + resumed = store.rotate( + rotation_id="ambiguous-rotation", + new_key_id="key-2026-09", + new_key=b"n" * 32, + ) + assert resumed["status"] == "completed" + assert resumed["objects_rotated"] == 1 + assert store.get_bytes(_scope(), receipt.object_id) == b"ambiguous rotation" + with pytest.raises(ManagedObjectConflictError, match="rotation_conflict"): + store.rotate( + rotation_id="ambiguous-rotation", + new_key_id="key-2026-09", + new_key=b"x" * 32, + ) + + +def test_source_mutation_between_addressing_and_encryption_cannot_install_object(tmp_path: Path) -> None: + store, _session_store, repository = _store(tmp_path) + + class MutatingStream(io.BytesIO): + def __init__(self) -> None: + super().__init__(b"first payload") + self._rewinds = 0 + + def seek(self, offset: int, whence: int = os.SEEK_SET) -> int: + result = super().seek(offset, whence) + if offset == 0 and whence == os.SEEK_SET: + self._rewinds += 1 + if self._rewinds >= 3: + super().seek(0) + super().write(b"other payload") + super().seek(0) + return result + + with pytest.raises(ManagedObjectIntegrityError, match="integrity_failed"): + store.put_stream(_scope(), ManagedObjectKind.EVIDENCE, MutatingStream()) + assert repository.stats()["objects"] == 0 + assert not list((tmp_path / "managed" / "objects").rglob("*.mobj")) + + +def test_root_path_swap_is_rejected_before_touching_replacement(tmp_path: Path) -> None: + store, _session_store, _repository = _store(tmp_path) + receipt = store.put_bytes(_scope(), ManagedObjectKind.DOCUMENT, b"root identity") + trusted = tmp_path / "trusted-moved" + replacement = tmp_path / "replacement" + replacement.mkdir(mode=0o700) + (tmp_path / "managed").rename(trusted) + (tmp_path / "managed").symlink_to(replacement, target_is_directory=True) + try: + with pytest.raises(ManagedObjectIntegrityError, match="root_identity"): + store.get_bytes(_scope(), receipt.object_id) + assert list(replacement.iterdir()) == [] + finally: + (tmp_path / "managed").unlink() + shutil.rmtree(replacement) + trusted.rename(tmp_path / "managed") + + +def test_root_path_swap_during_operation_is_rejected_before_success( + tmp_path: Path, + monkeypatch: pytest.MonkeyPatch, +) -> None: + store, _session_store, _repository = _store(tmp_path) + store.put_bytes(_scope(), ManagedObjectKind.DOCUMENT, b"root identity during status") + managed_root = tmp_path / "managed" + trusted = tmp_path / "trusted-during-operation" + replacement = tmp_path / "replacement-during-operation" + replacement.mkdir(mode=0o700) + for child in ("objects", "quarantine"): + (replacement / child).mkdir(mode=0o700) + original_scandir = managed_object_store_module.os.scandir + swapped = False + + def swap_before_scan(path): + nonlocal swapped + if not swapped and isinstance(path, Path): + swapped = True + managed_root.rename(trusted) + managed_root.symlink_to(replacement, target_is_directory=True) + return original_scandir(path) + + monkeypatch.setattr(managed_object_store_module.os, "scandir", swap_before_scan) + try: + with pytest.raises(ManagedObjectIntegrityError, match="root_identity"): + store.operator_status() + assert list((replacement / "objects").iterdir()) == [] + finally: + if managed_root.is_symlink(): + managed_root.unlink() + if trusted.exists(): + trusted.rename(managed_root) + shutil.rmtree(replacement) + + +def test_symlink_orphan_is_never_followed_or_counted_as_recovered(tmp_path: Path) -> None: + store, _session_store, _repository = _store(tmp_path) + store.put_bytes(_scope(), ManagedObjectKind.EVIDENCE, b"symlink boundary") + session_dir = next(path for path in (tmp_path / "managed" / "objects").iterdir() if path.is_dir()) + external = tmp_path / "external-ciphertext" + external.write_bytes(b"must remain") + (session_dir / ("e" * 32 + ".mobj")).symlink_to(external) + with pytest.raises(ManagedObjectIntegrityError, match="integrity_failed"): + store.recover() + assert external.read_bytes() == b"must remain" + + +def test_metadata_schema_drift_fails_closed(tmp_path: Path) -> None: + sessions = SessionStore(tmp_path / "sessions.db") + sessions.get_or_create_session("session-a", "owner-a") + with sqlite3.connect(sessions.db_path) as conn: + conn.execute("CREATE TABLE managed_object_store_schema (schema_version TEXT PRIMARY KEY, installed_at REAL NOT NULL)") + conn.execute( + "INSERT INTO managed_object_store_schema VALUES (?, ?)", + ("future-incompatible-schema", 1.0), + ) + with pytest.raises(RuntimeError, match="schema version"): + EncryptedManagedObjectStore( + _config(tmp_path / "managed"), + SQLiteManagedObjectMetadataRepository(sessions.db_path), + ) + + +def test_rotation_restart_seals_committed_database_transition( + tmp_path: Path, + monkeypatch: pytest.MonkeyPatch, +) -> None: + store, _session_store, repository = _store(tmp_path) + receipt = store.put_bytes(_scope(), ManagedObjectKind.EVIDENCE, b"commit before receipt") + + def fail_receipt(*_args, **_kwargs): + raise OSError("simulated receipt fsync ambiguity") + + monkeypatch.setattr(store, "_complete_rotation_receipt", fail_receipt) + with pytest.raises(OSError, match="receipt fsync"): + store.rotate( + rotation_id="commit-before-receipt", + new_key_id="key-2026-09", + new_key=b"n" * 32, + ) + + restarted = EncryptedManagedObjectStore( + _config( + tmp_path / "managed", + current_key_id="key-2026-09", + current_key=b"n" * 32, + previous_keys={"key-2026-08": b"k" * 32}, + ), + repository, + ) + sealed = restarted.rotate( + rotation_id="commit-before-receipt", + new_key_id="key-2026-09", + new_key=b"n" * 32, + ) + assert sealed["status"] == "completed" + assert sealed["objects_rotated"] == 1 + assert restarted.get_bytes(_scope(), receipt.object_id) == b"commit before receipt" + + +def test_rotation_receipt_rejects_same_key_id_with_different_material(tmp_path: Path) -> None: + store, _session_store, repository = _store(tmp_path) + store.put_bytes(_scope(), ManagedObjectKind.EVIDENCE, b"receipt must bind key material") + store.rotate( + rotation_id="material-bound-receipt", + new_key_id="key-2026-09", + new_key=b"n" * 32, + ) + + wrong_restart = EncryptedManagedObjectStore( + _config( + tmp_path / "managed", + current_key_id="key-2026-09", + current_key=b"x" * 32, + previous_keys={"key-2026-08": b"k" * 32}, + ), + repository, + ) + with pytest.raises(ManagedObjectConflictError, match="rotation_conflict"): + wrong_restart.rotate( + rotation_id="material-bound-receipt", + new_key_id="key-2026-09", + new_key=b"x" * 32, + ) + + +def test_rotation_journal_rejects_same_key_id_with_substituted_material( + tmp_path: Path, + monkeypatch: pytest.MonkeyPatch, +) -> None: + store, _session_store, repository = _store(tmp_path) + store.put_bytes(_scope(), ManagedObjectKind.EVIDENCE, b"journal must bind key material") + original_rotate = repository.rotate + monkeypatch.setattr( + repository, + "rotate", + lambda *_args, **_kwargs: (_ for _ in ()).throw(OSError("stop after journal")), + ) + with pytest.raises(OSError, match="after journal"): + store.rotate( + rotation_id="material-bound-journal", + new_key_id="key-2026-09", + new_key=b"n" * 32, + ) + monkeypatch.setattr(repository, "rotate", original_rotate) + + with pytest.raises(ManagedObjectConflictError, match="rotation_conflict"): + store.rotate( + rotation_id="material-bound-journal", + new_key_id="key-2026-09", + new_key=b"x" * 32, + ) + + +def test_unsafe_root_and_non_seekable_stream_fail_closed(tmp_path: Path) -> None: + target = tmp_path / "real" + target.mkdir() + symlink = tmp_path / "managed" + symlink.symlink_to(target, target_is_directory=True) + session_store = SessionStore(tmp_path / "sessions.db") + session_store.get_or_create_session("session-a", "owner-a") + repository = SQLiteManagedObjectMetadataRepository(session_store.db_path) + with pytest.raises(ValueError, match="secure directory"): + EncryptedManagedObjectStore(_config(symlink), repository) + + store = EncryptedManagedObjectStore(_config(tmp_path / "safe"), repository) + + class NonSeekable(io.BytesIO): + def seekable(self) -> bool: + return False + + with pytest.raises(ValueError, match="seekable"): + store.put_stream(_scope(), ManagedObjectKind.DOCUMENT, NonSeekable(b"payload")) + + +def test_storage_modes_and_status_are_privacy_safe(tmp_path: Path) -> None: + store, _session_store, _repository = _store(tmp_path) + store.put_bytes(_scope(), ManagedObjectKind.DOCUMENT, b"mode") + assert stat.S_IMODE((tmp_path / "managed").stat().st_mode) == 0o700 + for path in (tmp_path / "managed").rglob("*"): + expected = 0o700 if path.is_dir() else 0o600 + assert stat.S_IMODE(path.stat().st_mode) == expected + + status = store.operator_status() + encoded = json.dumps(status, sort_keys=True) + assert status["enabled"] is True + assert status["encryption_evidence"] == "local_rehearsal_only" + assert status["objects"] == 1 + assert str(tmp_path) not in encoded + assert "tenant-a" not in encoded + assert "session-a" not in encoded + assert "owner-a" not in encoded + assert "obj_" not in encoded + + +def test_operator_status_fails_when_active_object_key_is_unavailable(tmp_path: Path) -> None: + original, _session_store, repository = _store(tmp_path) + original.put_bytes(_scope(), ManagedObjectKind.EVIDENCE, b"still needs its configured key") + missing_key = EncryptedManagedObjectStore( + _config( + tmp_path / "managed", + current_key_id="key-2026-09", + current_key=b"n" * 32, + ), + repository, + ) + + status = missing_key.operator_status() + assert status["integrity_status"] == "failed" + assert status["unavailable_key_objects"] == 1 + + +def test_live_postgres_metadata_parity_when_explicitly_available(tmp_path: Path) -> None: + dsn = os.getenv("TEST_POSTGRES_DSN", "").strip() + if not dsn: + pytest.skip("TEST_POSTGRES_DSN is not configured") + from backend.storage.managed_object_store import PostgresManagedObjectMetadataRepository + + session_id = f"managed-object-{os.urandom(8).hex()}" + owner_id = "managed-object-owner" + from backend.storage.postgres import PostgresSessionStore + + sessions = PostgresSessionStore(dsn) + sessions.get_or_create_session(session_id, owner_id) + repository = PostgresManagedObjectMetadataRepository(dsn) + store = EncryptedManagedObjectStore(_config(tmp_path / "pg-managed"), repository) + scope = ManagedObjectScope("pg-tenant", owner_id, session_id) + try: + receipt = store.put_bytes(scope, ManagedObjectKind.EVIDENCE, b"postgres parity") + assert store.get_bytes(scope, receipt.object_id) == b"postgres parity" + assert store.delete(scope, receipt.object_id).deleted + finally: + sessions.set_session_lifecycle_state( + session_id, + "purging", + receipt_id=f"cleanup-{session_id}", + started_at=1.0, + ) + store.purge_session(session_id) + sessions.delete_session(session_id) + + +def test_configuration_rejects_role_key_reuse_and_ambiguous_key_encodings(tmp_path: Path) -> None: + with pytest.raises(ValueError, match="distinct"): + _config( + tmp_path / "managed-address-reuse", + previous_keys={"key-2026-07": b"a" * 32}, + ) + with pytest.raises(ValueError, match="distinct"): + _config( + tmp_path / "managed-previous-reuse", + previous_keys={ + "key-2026-06": b"p" * 32, + "key-2026-07": b"p" * 32, + }, + ) + with pytest.raises(ValueError, match="chunk_bytes"): + _config(tmp_path / "managed-bool-budget", chunk_bytes=True) + + encoded_current = base64.urlsafe_b64encode(b"k" * 32).rstrip(b"=").decode() + encoded_address = base64.urlsafe_b64encode(b"a" * 32).rstrip(b"=").decode() + environment = { + "MANAGED_OBJECT_STORE_ENABLED": "true", + "MANAGED_OBJECT_STORE_ROOT": str(tmp_path / "managed-env"), + "MANAGED_OBJECT_STORE_CURRENT_KEY_ID": "key-2026-08", + "MANAGED_OBJECT_STORE_CURRENT_KEY": encoded_current + "!!", + "MANAGED_OBJECT_STORE_ADDRESS_KEY": encoded_address, + } + with pytest.raises(ValueError, match="hex or URL-safe base64"): + ManagedObjectStoreConfig.from_environment(environment) + + +def test_stream_none_is_not_silently_treated_as_end_of_file(tmp_path: Path) -> None: + store, _session_store, repository = _store(tmp_path) + + class NonBlockingStream(io.BytesIO): + def read(self, size: int | None = -1) -> bytes: + del size + return cast(bytes, None) + + with pytest.raises(TypeError, match="return bytes"): + store.put_stream( + _scope(), + ManagedObjectKind.EVIDENCE, + NonBlockingStream(b"must not be silently discarded"), + ) + assert repository.stats()["objects"] == 0 + + +def test_rotation_rejects_role_key_reuse_and_retains_keys_for_live_objects(tmp_path: Path) -> None: + original, _session_store, repository = _store(tmp_path) + receipt = original.put_bytes(_scope(), ManagedObjectKind.EVIDENCE, b"still uses the oldest key") + restarted = EncryptedManagedObjectStore( + _config( + tmp_path / "managed", + current_key_id="key-2026-09", + current_key=b"n" * 32, + previous_keys={"key-2026-08": b"k" * 32}, + ), + repository, + ) + with pytest.raises(ValueError, match="distinct"): + restarted.rotate( + rotation_id="address-key-reuse", + new_key_id="key-address-reuse", + new_key=b"a" * 32, + ) + with pytest.raises(ValueError, match="distinct"): + restarted.rotate( + rotation_id="previous-key-reuse", + new_key_id="key-previous-reuse", + new_key=b"k" * 32, + ) + + rotation = restarted.rotate( + rotation_id="retain-live-previous-key", + new_key_id="key-2026-10", + new_key=b"x" * 32, + ) + assert rotation["status"] == "completed" + assert restarted.get_bytes(_scope(), receipt.object_id) == b"still uses the oldest key" + + +def test_rotation_envelopes_require_canonical_bytes_and_complete_schema(tmp_path: Path) -> None: + store, _session_store, _repository = _store(tmp_path) + journal_id = "noncanonical-journal" + store._write_rotation_journal( + journal_id, + old_key_id="key-2026-08", + old_key=b"k" * 32, + new_key_id="key-2026-09", + new_key=b"n" * 32, + target_tokens=[], + ) + journal_path = store._rotation_path(journal_id, "journals") + envelope = json.loads(journal_path.read_bytes()) + journal_path.write_text(json.dumps(envelope, indent=2), encoding="utf-8") + journal_path.chmod(0o600) + with pytest.raises(ManagedObjectIntegrityError, match="rotation_receipt_invalid"): + store._load_rotation_journal(journal_id) + + receipt_id = "incomplete-receipt" + store._write_signed_json( + store._rotation_path(receipt_id, "receipts"), + { + "schema_version": "2026-08-managed-object-v1", + "rotation_id": receipt_id, + "status": "completed", + "new_key_id_sha256": hashlib.sha256(b"key-2026-08").hexdigest(), + }, + ) + with pytest.raises(ManagedObjectIntegrityError, match="rotation_receipt_invalid"): + store.rotate( + rotation_id=receipt_id, + new_key_id="key-2026-08", + new_key=b"k" * 32, + ) + + +def test_rotation_envelope_writer_enforces_the_reader_size_bound(tmp_path: Path) -> None: + store, _session_store, _repository = _store(tmp_path) + with pytest.raises(ManagedObjectStoreLimitError, match="journal_limit"): + store._write_signed_json( + store._rotation_path("oversized-envelope", "journals"), + {"oversized": "x" * (17 * 1024 * 1024)}, + ) + assert not store._rotation_path("oversized-envelope", "journals").exists() + + +def test_fractional_generation_in_database_fails_closed(tmp_path: Path) -> None: + store, session_store, _repository = _store(tmp_path) + receipt = store.put_bytes(_scope(), ManagedObjectKind.EVIDENCE, b"metadata must remain typed") + with sqlite3.connect(session_store.db_path) as conn: + conn.execute( + "UPDATE managed_encrypted_objects SET generation = ? WHERE object_id = ?", + (1.5, receipt.object_id), + ) + with pytest.raises(ManagedObjectIntegrityError, match="metadata_invalid"): + store.get_bytes(_scope(), receipt.object_id) + + +def test_database_key_id_must_match_authenticated_ciphertext_header(tmp_path: Path) -> None: + config = _config( + tmp_path / "managed", + previous_keys={"key-2026-07": b"p" * 32}, + ) + store, session_store, _repository = _store(tmp_path, config=config) + receipt = store.put_bytes(_scope(), ManagedObjectKind.EVIDENCE, b"key metadata is authenticated") + with sqlite3.connect(session_store.db_path) as conn: + conn.execute( + "UPDATE managed_encrypted_objects SET key_id = ? WHERE object_id = ?", + ("key-2026-07", receipt.object_id), + ) + with pytest.raises(ManagedObjectIntegrityError, match="integrity_failed"): + store.get_bytes(_scope(), receipt.object_id) + + +def test_database_ciphertext_size_must_match_opened_file(tmp_path: Path) -> None: + store, session_store, _repository = _store(tmp_path) + receipt = store.put_bytes(_scope(), ManagedObjectKind.EVIDENCE, b"size metadata is authenticated") + with sqlite3.connect(session_store.db_path) as conn: + conn.execute( + "UPDATE managed_encrypted_objects SET ciphertext_size = ciphertext_size + 1 WHERE object_id = ?", + (receipt.object_id,), + ) + with pytest.raises(ManagedObjectIntegrityError, match="integrity_failed"): + store.get_bytes(_scope(), receipt.object_id) + + +def test_hardlinked_ciphertext_blocks_purge_and_cannot_escape_inventory(tmp_path: Path) -> None: + store, session_store, repository = _store(tmp_path) + store.put_bytes(_scope(), ManagedObjectKind.EVIDENCE, b"must not survive verified purge") + ciphertext = next((tmp_path / "managed" / "objects").rglob("*.mobj")) + escaped = tmp_path / "escaped-ciphertext.mobj" + os.link(ciphertext, escaped) + session_store.set_session_lifecycle_state( + "session-a", + "purging", + receipt_id="hardlink-purge", + started_at=4.0, + ) + + with pytest.raises(ManagedObjectIntegrityError, match="integrity_failed"): + store.purge_session("session-a") + assert escaped.exists() + assert repository.stats()["objects"] == 1 + + +def test_purge_directory_swap_cannot_delete_an_external_file( + tmp_path: Path, + monkeypatch: pytest.MonkeyPatch, +) -> None: + store, session_store, _repository = _store(tmp_path) + store.put_bytes(_scope(), ManagedObjectKind.EVIDENCE, b"known ciphertext") + session_directory = next( + path + for path in (tmp_path / "managed" / "objects").iterdir() + if path.is_dir() + ) + orphan_name = f"{('e' * 32)}.mobj" + orphan = session_directory / orphan_name + orphan.write_bytes(b"orphan ciphertext") + orphan.chmod(0o600) + external_directory = tmp_path / "external" + external_directory.mkdir(mode=0o700) + external_file = external_directory / orphan_name + external_file.write_bytes(b"must not be deleted") + external_file.chmod(0o600) + displaced = tmp_path / "displaced-session-directory" + original_regular_files = managed_object_store_module._regular_files + swapped = False + + def swap_after_inventory(root: Path) -> list[Path]: + nonlocal swapped + result = original_regular_files(root) + if Path(root) == session_directory and not swapped: + swapped = True + session_directory.rename(displaced) + session_directory.symlink_to(external_directory, target_is_directory=True) + return result + + monkeypatch.setattr(managed_object_store_module, "_regular_files", swap_after_inventory) + session_store.set_session_lifecycle_state( + "session-a", + "purging", + receipt_id="directory-swap-purge", + started_at=5.0, + ) + try: + store.purge_session("session-a") + except (ManagedObjectIntegrityError, OSError): + pass + assert swapped is False + assert external_file.read_bytes() == b"must not be deleted" + + +def test_recovery_quarantines_interrupted_ciphertext_temp_files(tmp_path: Path) -> None: + store, _session_store, _repository = _store(tmp_path) + store.put_bytes(_scope(), ManagedObjectKind.EVIDENCE, b"interrupted ciphertext") + ciphertext = next((tmp_path / "managed" / "objects").rglob("*.mobj")) + interrupted = ciphertext.with_name(f".{('e' * 32)}.tmp") + interrupted.write_bytes(ciphertext.read_bytes()) + interrupted.chmod(0o600) + + report = store.recover() + assert report == { + "orphans_quarantined": 1, + "active_files_missing": 0, + "recovery_failures": 0, + } + assert not interrupted.exists() + assert len(list((tmp_path / "managed" / "quarantine").rglob("*.mobj"))) == 1 diff --git a/backend/tests/test_managed_object_store_operator.py b/backend/tests/test_managed_object_store_operator.py new file mode 100644 index 0000000..062e92b --- /dev/null +++ b/backend/tests/test_managed_object_store_operator.py @@ -0,0 +1,146 @@ +from __future__ import annotations + +import base64 +import json +from pathlib import Path + +import pytest + +from backend.operations.managed_object_store_operator import ( + _configured_store, + _decode_key, + build_operator_report, + main, +) +from backend.session_store import SessionStore +from backend.storage.managed_object_store import ( + ManagedObjectKind, + ManagedObjectScope, + ManagedObjectStoreConfig, +) + + +def _encoded(value: bytes) -> str: + return base64.urlsafe_b64encode(value).rstrip(b"=").decode() + + +def test_disabled_status_is_machine_readable_and_does_not_require_secrets() -> None: + report = build_operator_report("status", {"MANAGED_OBJECT_STORE_ENABLED": "false"}) + assert report == { + "schema_version": "2026-08-managed-object-operator-v1", + "command": "status", + "status": "disabled", + "enabled": False, + "production_encryption_claim": False, + "cloud_provider_contacted": False, + } + + +def test_disposable_rehearsal_is_provider_free_and_privacy_safe(tmp_path: Path) -> None: + report = build_operator_report( + "rehearse", + { + "RELEASE_SHA": "a" * 40, + "TMPDIR": str(tmp_path), + }, + ) + encoded = json.dumps(report, sort_keys=True) + assert report["status"] == "passed" + assert report["round_trip_verified"] is True + assert report["tamper_rejected"] is True + assert report["rotation_verified"] is True + assert report["purge_verified"] is True + assert report["provider"] == "disposable_sqlite" + assert report["cloud_provider_contacted"] is False + assert report["production_encryption_claim"] is False + assert report["release_sha"] == "a" * 40 + assert str(tmp_path) not in encoded + assert "rehearsal-tenant" not in encoded + assert "rehearsal-owner" not in encoded + assert "rehearsal-session" not in encoded + assert "obj_" not in encoded + + +def test_enabled_status_never_emits_root_database_or_key_material(tmp_path: Path) -> None: + database = tmp_path / "operator.db" + sessions = SessionStore(database) + sessions.get_or_create_session("session-a", "owner-a") + root = tmp_path / "sensitive-root" + environment = { + "MANAGED_OBJECT_STORE_ENABLED": "true", + "MANAGED_OBJECT_STORE_ROOT": str(root), + "MANAGED_OBJECT_STORE_CURRENT_KEY_ID": "sensitive-key-id", + "MANAGED_OBJECT_STORE_CURRENT_KEY": _encoded(b"k" * 32), + "MANAGED_OBJECT_STORE_ADDRESS_KEY": _encoded(b"a" * 32), + "MANAGED_OBJECT_STORE_PREVIOUS_KEYS": "{}", + "DB_ENGINE": "sqlite", + "DB_PATH": str(database), + "DATABASE_URL": "postgresql://secret:secret@example.invalid/private", + } + report = build_operator_report("status", environment) + encoded = json.dumps(report, sort_keys=True) + assert report["status"] == "ready" + assert report["storage"]["enabled"] is True + for forbidden in ( + str(root), + str(database), + "sensitive-key-id", + "secret:secret", + "example.invalid", + ): + assert forbidden not in encoded + + +def test_cli_sanitizes_unexpected_failures(monkeypatch, capsys) -> None: + monkeypatch.setattr( + "backend.operations.managed_object_store_operator.build_operator_report", + lambda *_args, **_kwargs: (_ for _ in ()).throw(RuntimeError("/secret/path tenant-a")), + ) + assert main(["status"]) == 1 + payload = json.loads(capsys.readouterr().out) + assert payload == { + "schema_version": "2026-08-managed-object-operator-v1", + "status": "failed", + "error_code": "managed_object_operator_failed", + "error_type": "RuntimeError", + } + + +def test_rotation_key_decoder_rejects_noncanonical_base64_aliases() -> None: + encoded = _encoded(b"n" * 32) + for ambiguous in (encoded + "!!", encoded + "===", encoded.replace("-", "+")): + if ambiguous == encoded: + continue + with pytest.raises(ValueError, match="encoding is invalid"): + _decode_key(ambiguous) + + +def test_recover_report_fails_when_active_ciphertext_is_missing(tmp_path: Path) -> None: + database = tmp_path / "operator.db" + sessions = SessionStore(database) + sessions.get_or_create_session("session-a", "owner-a") + environment = { + "MANAGED_OBJECT_STORE_ENABLED": "true", + "MANAGED_OBJECT_STORE_ROOT": str(tmp_path / "managed"), + "MANAGED_OBJECT_STORE_CURRENT_KEY_ID": "key-2026-08", + "MANAGED_OBJECT_STORE_CURRENT_KEY": _encoded(b"k" * 32), + "MANAGED_OBJECT_STORE_ADDRESS_KEY": _encoded(b"a" * 32), + "MANAGED_OBJECT_STORE_PREVIOUS_KEYS": "{}", + "DB_ENGINE": "sqlite", + "DB_PATH": str(database), + } + store = _configured_store( + ManagedObjectStoreConfig.from_environment(environment), + environment, + ) + store.put_bytes( + ManagedObjectScope("tenant-a", "owner-a", "session-a"), + ManagedObjectKind.EVIDENCE, + b"must remain observable", + ) + next((tmp_path / "managed" / "objects").rglob("*.mobj")).unlink() + + report = build_operator_report("recover", environment) + assert report["status"] == "failed" + assert report["recovery"]["active_files_missing"] == 1 + assert report["storage"]["integrity_status"] == "failed" diff --git a/backend/tests/test_multi_user_ownership.py b/backend/tests/test_multi_user_ownership.py new file mode 100644 index 0000000..e6a7dbe --- /dev/null +++ b/backend/tests/test_multi_user_ownership.py @@ -0,0 +1,224 @@ +from __future__ import annotations + +from collections.abc import Generator +from dataclasses import dataclass +from types import SimpleNamespace + +import pytest +from fastapi.testclient import TestClient +from starlette.websockets import WebSocketDisconnect + +from backend.models.operations import AgentRun, BrainRun, OperationPlan, OperationPlanV2 +from backend.api.security.ownership import OwnershipNotFoundError, ownership_actor_scope +from backend.services.workbook_jobs import WorkbookJobManager +from backend.session_store import SessionStore + + +@dataclass +class _MultiUserHarness: + client: TestClient + store: SessionStore + alice_headers: dict[str, str] + bob_headers: dict[str, str] + admin_headers: dict[str, str] + + def upload_for_alice(self, session_id: str = "session-alice") -> dict: + response = self.client.post( + "/api/v1/documents/upload", + headers=self.alice_headers, + data={"session_id": session_id}, + files={"file": ("alice.csv", b"Metric,Value\nRevenue,125\n", "text/csv")}, + ) + assert response.status_code == 200, response.text + return response.json() + + +@pytest.fixture +def multi_user_harness(tmp_path, monkeypatch) -> Generator[_MultiUserHarness, None, None]: + import backend.storage as storage + from backend.config import DB_ENGINE + from backend.api.routes import documents + from backend.main import app + from backend.services import workbook_jobs + + store = SessionStore(tmp_path / "multi-user.db") + keys: dict[str, str] = {} + for user_id, role in (("alice", "analyst"), ("bob", "analyst"), ("root-admin", "admin")): + store.upsert_enterprise_user( + user_id=user_id, + email=f"{user_id}@example.com", + name=user_id.title(), + role=role, + status="api_key_only", + ) + keys[user_id] = store.create_enterprise_api_key(user_id, "ownership-test")["api_key"] + + manager = WorkbookJobManager( + storage_dir=tmp_path / "workbook-jobs", + store=store, + ) + monkeypatch.setattr(storage, "_store", store) + monkeypatch.setattr(storage, "_store_engine", DB_ENGINE.lower()) + monkeypatch.setattr(documents, "DATA_DIR", tmp_path / "data") + monkeypatch.setattr(workbook_jobs, "_manager", manager) + + with TestClient(app) as client: + yield _MultiUserHarness( + client=client, + store=store, + alice_headers={"X-API-Key": keys["alice"]}, + bob_headers={"X-API-Key": keys["bob"]}, + admin_headers={"X-API-Key": keys["root-admin"]}, + ) + + +def _seed_session_artifacts(harness: _MultiUserHarness, session_id: str) -> dict[str, str]: + legacy_plan = OperationPlan(session_id=session_id, prompt="Alice plan") + v2_plan = OperationPlanV2(id=legacy_plan.id, session_id=session_id, prompt="Alice plan") + plan_payload = legacy_plan.model_dump(mode="json") + plan_payload["metadata"]["operation_plan_v2"] = v2_plan.model_dump(mode="json") + harness.store.save_operation_plan(plan_payload) + + snapshot_id = harness.store.record_snapshot( + session_id=session_id, + workbook_state={"state": {"cell_snapshot": {"Sheet1!A1": {"value": "private"}}}}, + ) + agent_run = AgentRun(sessionId=session_id, prompt="Alice agent run") + harness.store.save_agent_run(agent_run.model_dump(mode="json", by_alias=True)) + brain_run = BrainRun(sessionId=session_id, prompt="Alice brain run") + harness.store.save_brain_run(brain_run.model_dump(mode="json", by_alias=True)) + return { + "plan_id": legacy_plan.id, + "snapshot_id": snapshot_id, + "agent_run_id": agent_run.run_id, + "brain_run_id": brain_run.run_id, + } + + +def test_non_admin_sees_only_owned_sessions_and_documents(multi_user_harness: _MultiUserHarness): + alice_document = multi_user_harness.upload_for_alice() + bob_upload = multi_user_harness.client.post( + "/api/v1/documents/upload", + headers=multi_user_harness.bob_headers, + data={"session_id": "session-bob"}, + files={"file": ("bob.csv", b"Metric,Value\nCosts,75\n", "text/csv")}, + ) + assert bob_upload.status_code == 200, bob_upload.text + + cross_tenant = multi_user_harness.client.get( + f"/api/v1/documents/{alice_document['document_id']}", + headers=multi_user_harness.bob_headers, + ) + assert cross_tenant.status_code == 404 + + sessions = multi_user_harness.client.get( + "/api/v1/sessions", + headers=multi_user_harness.bob_headers, + ) + assert sessions.status_code == 200, sessions.text + assert {item["session_id"] for item in sessions.json()["sessions"]} == {"session-bob"} + + +def test_arbitrary_session_bound_artifact_ids_are_not_disclosed(multi_user_harness: _MultiUserHarness): + multi_user_harness.upload_for_alice() + artifacts = _seed_session_artifacts(multi_user_harness, "session-alice") + + checks = [ + ("get", f"/api/v1/tasks/plans/{artifacts['plan_id']}/v2", None), + ("get", f"/api/v1/agent-runs/{artifacts['agent_run_id']}", None), + ("get", f"/api/v1/experiments/brain/runs/{artifacts['brain_run_id']}", None), + ( + "post", + "/api/v1/sessions/session-alice/rollback", + {"snapshot_id": artifacts["snapshot_id"], "current_workbook_state": {}}, + ), + ] + for method, path, payload in checks: + response = multi_user_harness.client.request( + method, + path, + headers=multi_user_harness.bob_headers, + json=payload, + ) + assert response.status_code == 404, (path, response.status_code, response.text) + + +def test_workbook_job_owner_is_actor_bound_and_not_caller_selected(multi_user_harness: _MultiUserHarness): + multi_user_harness.upload_for_alice() + submitted = multi_user_harness.client.post( + "/api/v1/workbook/jobs", + headers=multi_user_harness.alice_headers, + json={ + "command": "fixture-validate", + "payload": {}, + "session_id": "session-alice", + "account_id": "bob-supplied-account", + }, + ) + assert submitted.status_code == 200, submitted.text + run_id = submitted.json()["run_id"] + assert submitted.json()["account_id"] == "alice" + + cross_tenant = multi_user_harness.client.get( + f"/api/v1/workbook/jobs/{run_id}", + headers=multi_user_harness.bob_headers, + ) + assert cross_tenant.status_code == 404 + + filtered_list = multi_user_harness.client.get( + "/api/v1/workbook/jobs", + headers=multi_user_harness.bob_headers, + params={"account_id": "alice"}, + ) + assert filtered_list.status_code == 200, filtered_list.text + assert filtered_list.json()["jobs"] == [] + + +def test_admin_retains_cross_tenant_access(multi_user_harness: _MultiUserHarness): + alice_document = multi_user_harness.upload_for_alice() + + response = multi_user_harness.client.get( + f"/api/v1/documents/{alice_document['document_id']}", + headers=multi_user_harness.admin_headers, + ) + + assert response.status_code == 200, response.text + + +def test_websocket_rejects_a_different_sessions_owner(multi_user_harness: _MultiUserHarness): + multi_user_harness.upload_for_alice() + + with pytest.raises(WebSocketDisconnect) as exc_info: + with multi_user_harness.client.websocket_connect( + "/ws/tasks/session-alice", + headers=multi_user_harness.bob_headers, + ): + pass + + assert exc_info.value.code == 1008 + + +def test_storage_list_surfaces_hide_foreign_session_artifacts(multi_user_harness: _MultiUserHarness): + multi_user_harness.upload_for_alice() + _seed_session_artifacts(multi_user_harness, "session-alice") + + with ownership_actor_scope(SimpleNamespace(user_id="bob", role="analyst")): + assert multi_user_harness.store.list_documents("session-alice") == [] + assert multi_user_harness.store.list_snapshots("session-alice") == [] + assert multi_user_harness.store.list_agent_runs("session-alice") == [] + assert multi_user_harness.store.list_brain_runs("session-alice") == [] + with pytest.raises(OwnershipNotFoundError): + multi_user_harness.store.get_or_create_session("session-alice") + + +def test_auth_disabled_dev_mode_preserves_single_user_storage_behavior(tmp_path): + store = SessionStore(tmp_path / "dev-mode.db") + store.get_or_create_session("dev-session") + document_id = store.save_document( + session_id="dev-session", + filename="dev.csv", + parsed={"rows": [["Metric", "Value"], ["Revenue", 125]]}, + ) + + assert store.get_document(document_id)["session_id"] == "dev-session" + assert [item["document_id"] for item in store.list_documents("dev-session")] == [document_id] diff --git a/backend/tests/test_oidc_auth.py b/backend/tests/test_oidc_auth.py new file mode 100644 index 0000000..d6028ce --- /dev/null +++ b/backend/tests/test_oidc_auth.py @@ -0,0 +1,172 @@ +from __future__ import annotations + +import time + +import jwt +import pytest +from cryptography.hazmat.primitives.asymmetric import rsa + +from backend.api.security.oidc import OIDCAuthenticator, OIDCConfig, OIDCValidationError + + +@pytest.fixture(scope="module") +def signing_keys(): + private_key = rsa.generate_private_key(public_exponent=65537, key_size=2048) + return private_key, private_key.public_key() + + +def _token(private_key, **overrides) -> str: + now = int(time.time()) + claims = { + "iss": "https://identity.example.com/tenant/v2.0", + "aud": "api://excel-agent", + "sub": "oidc-subject-alice", + "email": "alice@example.com", + "tid": "tenant-a", + "iat": now, + "exp": now + 300, + "amr": ["pwd", "mfa"], + } + claims.update(overrides) + return jwt.encode(claims, private_key, algorithm="RS256", headers={"kid": "test-key"}) + + +def _authenticator(public_key, *, user=None, **config_overrides) -> OIDCAuthenticator: + issuer = str(config_overrides.pop("issuer", "https://identity.example.com/tenant/v2.0")) + config = OIDCConfig( + enabled=True, + issuer=issuer, + audience="api://excel-agent", + jwks_url="https://identity.example.com/tenant/discovery/v2.0/keys", + allowed_tenants=frozenset({"tenant-a"}), + algorithms=("RS256",), + require_mfa=True, + **config_overrides, + ) + resolved_user = user or { + "user_id": "user-alice", + "email": "alice@example.com", + "name": "Alice", + "role": "analyst", + "status": "active", + "oidc_issuer": issuer, + "oidc_subject": "oidc-subject-alice", + "metadata": { + "oidc_issuer": issuer, + "oidc_subject": "oidc-subject-alice", + }, + } + return OIDCAuthenticator( + config, + user_loader=lambda _issuer, _subject: resolved_user, + signing_key_resolver=lambda _token: public_key, + ) + + +def test_oidc_maps_verified_identity_to_preprovisioned_role(signing_keys): + private_key, public_key = signing_keys + + actor = _authenticator(public_key).authenticate_sync(_token(private_key)) + + assert actor == { + "user_id": "user-alice", + "email": "alice@example.com", + "name": "Alice", + "role": "analyst", + "auth_source": "oidc", + "oidc_issuer": "https://identity.example.com/tenant/v2.0", + "oidc_subject": "oidc-subject-alice", + "oidc_tenant": "tenant-a", + "mfa_authenticated": True, + } + + +def test_oidc_rejects_server_algorithm_the_office_client_cannot_verify(): + config = OIDCConfig( + enabled=True, + issuer="https://identity.example.com/tenant/v2.0", + audience="api://excel-agent", + jwks_url="https://identity.example.com/tenant/discovery/v2.0/keys", + algorithms=("RS384",), + ) + + with pytest.raises(ValueError, match="Office-client-compatible"): + OIDCAuthenticator(config, user_loader=lambda _issuer, _subject: None) + + +@pytest.mark.parametrize( + ("claim_overrides", "message"), + [ + ({"iss": "https://attacker.example.com"}, "identity token validation failed"), + ({"aud": "api://other"}, "identity token validation failed"), + ({"tid": "tenant-b"}, "tenant is not allowed"), + ({"exp": 1}, "identity token validation failed"), + ({"amr": ["pwd"]}, "multi-factor authentication is required"), + ], +) +def test_oidc_rejects_invalid_security_claims(signing_keys, claim_overrides, message): + private_key, public_key = signing_keys + + with pytest.raises(OIDCValidationError, match=message): + _authenticator(public_key).authenticate_sync(_token(private_key, **claim_overrides)) + + +def test_oidc_rejects_unknown_or_inactive_users(signing_keys): + private_key, public_key = signing_keys + unknown = _authenticator(public_key) + unknown._user_loader = lambda _issuer, _subject: None + inactive = _authenticator( + public_key, + user={ + "user_id": "user-alice", + "email": "alice@example.com", + "name": "Alice", + "role": "admin", + "status": "disabled", + "metadata": {}, + }, + ) + + with pytest.raises(OIDCValidationError, match="identity is not provisioned"): + unknown.authenticate_sync(_token(private_key)) + with pytest.raises(OIDCValidationError, match="identity is not active"): + inactive.authenticate_sync(_token(private_key)) + + +def test_oidc_rejects_subject_or_issuer_mismatch_after_account_binding(signing_keys): + private_key, public_key = signing_keys + bound_user = { + "user_id": "user-alice", + "email": "alice@example.com", + "name": "Alice", + "role": "admin", + "status": "active", + "metadata": { + "oidc_subject": "bound-subject", + "oidc_issuer": "https://identity.example.com/tenant/v2.0", + }, + } + authenticator = _authenticator(public_key, user=bound_user) + + with pytest.raises(OIDCValidationError, match="identity binding does not match"): + authenticator.authenticate_sync(_token(private_key)) + + +def test_oidc_preserves_and_validates_a_trailing_slash_issuer(signing_keys): + private_key, public_key = signing_keys + issuer = "https://identity.example.com/tenant/v2.0/" + user = { + "user_id": "user-alice", + "email": "alice@example.com", + "name": "Alice", + "role": "analyst", + "status": "active", + "oidc_issuer": issuer, + "oidc_subject": "oidc-subject-alice", + "metadata": {"oidc_issuer": issuer, "oidc_subject": "oidc-subject-alice"}, + } + authenticator = _authenticator(public_key, user=user, issuer=issuer) + + actor = authenticator.authenticate_sync(_token(private_key, iss=issuer)) + + assert actor["oidc_issuer"] == issuer diff --git a/backend/tests/test_oidc_client_config.py b/backend/tests/test_oidc_client_config.py new file mode 100644 index 0000000..6d25760 --- /dev/null +++ b/backend/tests/test_oidc_client_config.py @@ -0,0 +1,59 @@ +from __future__ import annotations + +import pytest + +from backend.api.middleware.auth import is_public_path +from backend.api.security.oidc_client import ( + OIDCClientConfigError, + parse_extra_authorize_params, + validate_oidc_client_config, +) + + +def _valid_values() -> dict: + return { + "cors_origins": ["https://excel.internal.example"], + "oidc_issuer": "https://identity.example/tenant", + "oidc_audience": "api://excel-agent", + "oidc_client_id": "excel-agent-office", + "oidc_client_redirect_uri": ("https://excel.internal.example/src/taskpane/taskpane.html?oidc_callback=1"), + "oidc_client_scopes": ("openid", "profile", "email"), + "oidc_discovery_url": "https://identity.example/tenant/.well-known/openid-configuration", + "oidc_authorization_endpoint": "", + "oidc_token_endpoint": "", + "oidc_extra_authorize_params_json": '{"prompt":"select_account"}', + } + + +def test_production_pkce_client_config_is_complete_and_same_origin() -> None: + assert validate_oidc_client_config(_valid_values(), production=True) == [] + assert is_public_path("/auth/oidc/config") is True + + +@pytest.mark.parametrize( + ("field", "value", "expected"), + [ + ("oidc_client_id", "", "OIDC_CLIENT_ID"), + ("oidc_client_redirect_uri", "http://excel.internal.example/callback", "HTTPS"), + ( + "oidc_client_redirect_uri", + "https://other.example/taskpane.html?oidc_callback=1", + "share an origin", + ), + ( + "oidc_client_redirect_uri", + "https://excel.internal.example/taskpane.html", + "oidc_callback=1", + ), + ("oidc_client_scopes", ("profile",), "include openid"), + ], +) +def test_pkce_client_config_rejects_unsafe_values(field: str, value: object, expected: str) -> None: + values = _valid_values() + values[field] = value + assert any(expected in error for error in validate_oidc_client_config(values, production=True)) + + +def test_authorize_extras_cannot_override_pkce_security_parameters() -> None: + with pytest.raises(OIDCClientConfigError, match="reserved"): + parse_extra_authorize_params('{"code_challenge":"attacker"}') diff --git a/backend/tests/test_operation_surface_v2.py b/backend/tests/test_operation_surface_v2.py index 6d6ef1b..9a72a50 100644 --- a/backend/tests/test_operation_surface_v2.py +++ b/backend/tests/test_operation_surface_v2.py @@ -7,17 +7,22 @@ from backend.excel.formula_engine_adapters import get_optional_formula_engine_statuses from backend.excel.formula_analyzer import analyze_formula_payload from backend.excel.workbook_files import WorkbookExportService, WorkbookIngestionService +from backend.excel.workbook_worker import WorkerResult, WorkbookWorkerUnavailable from backend.main import app from backend.models.operations import ( + CreateChartOperationV2, FeatureOperationV2, OperationPlanV2, RangeMutationOperationV2, ReadOperationV2, SheetStructureOperationV2, + WorkbookStructureOperationV2, WriteCellOperationV2, ) +from backend.services.operation_registry import get_operation_registry_v3 from backend.services.plan_intelligence import plan_to_v2_with_intelligence from backend.services.operation_validation import validate_workbook_operations_v2 +from backend.services.workbook_jobs import WorkbookJobManager def test_v2_contract_accepts_read_copy_and_sheet_structure_operations(): @@ -52,8 +57,9 @@ def test_v2_validation_flags_overwriting_formula_and_destructive_delete(): "sheet_protection": {}, }, ) - assert report.status == "warning" + assert report.status == "failed" assert any(issue.code == "overwrite_formula" for issue in report.issues) + assert any(issue.code == "operation_not_apply_ready" for issue in report.issues) destructive = next(issue for issue in report.issues if issue.code == "destructive_operation") assert destructive.schema_family == "row" assert destructive.source_project == "OfficeCLI" @@ -63,12 +69,61 @@ def test_v2_validation_flags_overwriting_formula_and_destructive_delete(): def test_officecli_schema_hint_flags_missing_feature_payload(): op = FeatureOperationV2(type="set_data_validation", sheet="Sheet1", range="B2:B10") report = validate_workbook_operations_v2([op], {"sheet_names": ["Sheet1"], "cell_snapshot": {}}) - assert report.status == "warning" + assert report.status == "failed" + assert any(issue.code == "operation_not_apply_ready" for issue in report.issues) hint = next(issue for issue in report.issues if issue.code == "officecli_schema_hint") assert hint.schema_family == "validation" assert hint.source_project == "OfficeCLI" +def test_incomplete_advertised_operation_contracts_fail_closed(): + operations = [ + FeatureOperationV2( + id="op-cf-criteria", + type="set_conditional_format", + sheet="Sheet1", + range="B2:B10", + conditionalFormat={"type": "CellValue", "criteria": {"operator": "greaterThan", "formula": "0"}}, + ), + FeatureOperationV2( + id="op-filter-criteria", + type="set_autofilter", + sheet="Sheet1", + range="A1:B10", + autofilter={"column": 1, "criteria": {"filterOn": "custom", "criterion1": ">0"}}, + ), + WorkbookStructureOperationV2( + id="op-duplicate-name", + type="duplicate_sheet", + sheet="Sheet1", + newSheetName="Scenario Copy", + ), + CreateChartOperationV2( + id="op-chart-geometry", + sheet="Sheet1", + range="A1:B10", + data_range="A1:B10", + position="D2", + width=640, + height=360, + ), + ] + + report = validate_workbook_operations_v2( + operations, + {"sheet_names": ["Sheet1"], "cell_snapshot": {"Sheet1!A1": {"value": 1}}, "sheet_protection": {}}, + ) + registry = get_operation_registry_v3() + + assert report.status == "failed" + for operation_id in {"op-cf-criteria", "op-filter-criteria", "op-duplicate-name"}: + assert any(issue.operation_id == operation_id and issue.code == "operation_not_apply_ready" for issue in report.issues) + assert registry["set_conditional_format"]["applyReady"] is False + assert registry["set_autofilter"]["applyReady"] is False + assert registry["duplicate_sheet"]["applyReady"] is False + assert any(issue.operation_id == "op-chart-geometry" and issue.code == "unsupported_contract_fields" for issue in report.issues) + + def test_formula_analyzer_reports_dependencies_circular_refs_and_external_links(): result = analyze_formula_payload( operations=[ @@ -129,10 +184,11 @@ def test_formula_analyze_endpoint_prefers_hyperformula_worker(): assert payload["hyperformula_available"] is True -def test_hyperformula_adapter_is_enabled_by_default_after_approval(): +def test_hyperformula_adapter_fails_closed_without_explicit_approval(): statuses = {status.name: status for status in get_optional_formula_engine_statuses()} - assert statuses["hyperformula"].enabled is True - assert statuses["hyperformula"].license_mode == "GPL-3.0/commercial" + assert statuses["hyperformula"].enabled is False + assert statuses["hyperformula"].license_mode == "explicit-approved-license" + assert "commercial HYPERFORMULA_LICENSE_KEY" in statuses["hyperformula"].reason assert statuses["formulas"].enabled is False assert statuses["xlcalculator"].enabled is False @@ -239,7 +295,7 @@ def test_v2_finalize_stores_inspection_and_apply_returns_mutation_only(): assert all(op["type"] not in {"get_cell_ranges", "get_range_as_csv", "search_data", "get_all_objects", "get_workbook_metadata"} for op in approve.json()["operations"]) -def test_apply_result_records_post_apply_verification_summary(): +def test_apply_result_rejects_unrelated_post_apply_snapshot_without_exact_proof(): client = TestClient(app) session_id = "v2-apply-verification-flow" plan_resp = client.post( @@ -276,7 +332,7 @@ def test_apply_result_records_post_apply_verification_summary(): ) assert approve.status_code == 200 token = approve.json()["apply_token"] - op = approve.json()["operations"][0] + operations = approve.json()["operations"] result = client.post( "/api/v1/tasks/apply/result", @@ -287,7 +343,7 @@ def test_apply_result_records_post_apply_verification_summary(): "apply_token": token, "status": "applied", "office_apply_result": { - "applied": 1, + "applied": len(operations), "results": [ { "operationId": op["id"], @@ -295,8 +351,9 @@ def test_apply_result_records_post_apply_verification_summary(): "status": "success", "dirtyRanges": [{"sheet": "Sheet1", "range": op.get("range", "A1")}], } + for op in operations ], - "dirtyRanges": [{"sheet": "Sheet1", "range": op.get("range", "A1")}], + "dirtyRanges": [{"sheet": "Sheet1", "range": op.get("range", "A1")} for op in operations], }, "post_apply_workbook_state": { "sheet_names": ["Sheet1"], @@ -305,11 +362,8 @@ def test_apply_result_records_post_apply_verification_summary(): }, }, ) - assert result.status_code == 200 - verification = result.json()["verification"] - assert verification["status"] in {"verified", "warning"} - assert verification["dirty_ranges"] - assert "formula_analysis" in verification + assert result.status_code == 409 + assert "incomplete" in result.json()["detail"].lower() or "attempt" in result.json()["detail"].lower() def test_workbook_async_job_contract_completes_ingest(): @@ -329,7 +383,7 @@ def test_workbook_async_job_contract_completes_ingest(): assert submit.status_code == 200 run_id = submit.json()["run_id"] final = None - for _ in range(20): + for _ in range(60): poll = client.get(f"/api/v1/workbook/jobs/{run_id}", headers={"X-API-Key": API_KEY}) assert poll.status_code == 200 final = poll.json() @@ -341,6 +395,202 @@ def test_workbook_async_job_contract_completes_ingest(): assert final["result"]["engine"] == "sheetjs" +def test_workbook_job_manager_persists_completed_jobs(tmp_path, monkeypatch): + monkeypatch.setattr( + "backend.services.workbook_jobs._execute_command", + lambda command, payload: {"command": command, "changed": payload["after"]["A1"]}, + ) + manager = WorkbookJobManager(tmp_path) + job = manager.submit("diff", {"before": {"A1": 1}, "after": {"A1": 2}}, account_id="acct-1", session_id="sess-1") + + final = None + for _ in range(20): + final = manager.get(job.run_id) + if final and final.status in {"completed", "failed"}: + break + time.sleep(0.05) + + assert final is not None + assert final.status == "completed" + reloaded = WorkbookJobManager(tmp_path) + loaded = reloaded.get(job.run_id) + assert loaded is not None + assert loaded.status == "completed" + assert loaded.account_id == "acct-1" + assert loaded.session_id == "sess-1" + assert reloaded.list(session_id="sess-1")[0].run_id == job.run_id + + +def test_workbook_job_manager_reloads_from_session_store(tmp_path, monkeypatch): + class MemoryWorkbookJobStore: + def __init__(self): + self.records = {} + + def save_workbook_job(self, job): + self.records[job["run_id"]] = dict(job) + return job["run_id"] + + def get_workbook_job(self, run_id): + record = self.records.get(run_id) + return dict(record) if record else None + + def list_workbook_jobs(self, limit=50, status=None, session_id=None, account_id=None): + rows = list(self.records.values()) + if status: + rows = [row for row in rows if row.get("status") == status] + if session_id: + rows = [row for row in rows if row.get("session_id") == session_id] + if account_id: + rows = [row for row in rows if row.get("account_id") == account_id] + return [dict(row) for row in rows[:limit]] + + monkeypatch.setattr( + "backend.services.workbook_jobs._execute_command", + lambda command, payload: {"command": command, "changed": payload["after"]["A1"]}, + ) + store = MemoryWorkbookJobStore() + manager = WorkbookJobManager(tmp_path / "writer", store=store) + job = manager.submit("diff", {"before": {"A1": 1}, "after": {"A1": 3}}, account_id="acct-store", session_id="sess-store") + + final = None + for _ in range(20): + final = manager.get(job.run_id) + if final and final.status in {"completed", "failed"}: + break + time.sleep(0.05) + + assert final is not None + assert final.status == "completed" + assert store.records[job.run_id]["status"] == "completed" + + reloaded = WorkbookJobManager(tmp_path / "reader", store=store) + loaded = reloaded.get(job.run_id) + assert loaded is not None + assert loaded.status == "completed" + assert loaded.result == {"command": "diff", "changed": 3} + assert reloaded.list(account_id="acct-store")[0].session_id == "sess-store" + + +def test_workbook_job_routes_list_result_and_download_export(): + client = TestClient(app) + submit = client.post( + "/api/v1/workbook/jobs", + headers={"X-API-Key": API_KEY}, + json={ + "command": "export", + "session_id": "job-route-session", + "account_id": "acct-route", + "payload": { + "title": "Route Export", + "sheets": {"Summary": [["Metric", "Value"], ["Revenue", 100]]}, + }, + }, + ) + assert submit.status_code == 200 + run_id = submit.json()["run_id"] + final = None + for _ in range(60): + poll = client.get(f"/api/v1/workbook/jobs/{run_id}", headers={"X-API-Key": API_KEY}) + assert poll.status_code == 200 + final = poll.json() + if final["status"] in {"completed", "failed"}: + break + time.sleep(0.05) + + assert final is not None + assert final["status"] == "completed" + assert final["session_id"] == "job-route-session" + assert final["result_size_bytes"] > 0 + + listed = client.get( + "/api/v1/workbook/jobs?session_id=job-route-session", + headers={"X-API-Key": API_KEY}, + ) + assert listed.status_code == 200 + assert any(job["run_id"] == run_id for job in listed.json()["jobs"]) + + result = client.get(f"/api/v1/workbook/jobs/{run_id}/result", headers={"X-API-Key": API_KEY}) + assert result.status_code == 200 + assert result.json()["result"]["content_base64"] + + download = client.get(f"/api/v1/workbook/jobs/{run_id}/download", headers={"X-API-Key": API_KEY}) + assert download.status_code == 200 + assert download.headers["content-type"] == "application/vnd.openxmlformats-officedocument.spreadsheetml.sheet" + assert download.content.startswith(b"PK") + + +def test_workbook_job_route_preserves_worker_result_error_for_worker_only_command(tmp_path, monkeypatch): + manager = WorkbookJobManager(tmp_path) + + monkeypatch.setattr("backend.services.workbook_jobs.get_workbook_job_manager", lambda: manager) + monkeypatch.setattr( + "backend.excel.workbook_worker.run_worker", + lambda command, payload: WorkerResult(ok=False, error="roundtrip package relationship mismatch"), + ) + + client = TestClient(app) + submit = client.post( + "/api/v1/workbook/jobs", + headers={"X-API-Key": API_KEY}, + json={ + "command": "roundtrip-qa-v2", + "session_id": "worker-result-error", + "payload": {"session": {"workbookId": "wb-worker-error"}}, + }, + ) + assert submit.status_code == 200 + run_id = submit.json()["run_id"] + + final = None + for _ in range(20): + poll = client.get(f"/api/v1/workbook/jobs/{run_id}", headers={"X-API-Key": API_KEY}) + assert poll.status_code == 200 + final = poll.json() + if final["status"] in {"completed", "failed"}: + break + time.sleep(0.05) + + assert final is not None + assert final["status"] == "failed" + assert final["error"] == "roundtrip package relationship mismatch" + + +def test_workbook_job_route_preserves_worker_unavailable_for_worker_only_command(tmp_path, monkeypatch): + manager = WorkbookJobManager(tmp_path) + + def raise_unavailable(command, payload): + raise WorkbookWorkerUnavailable("Workbook worker is disabled or not installed") + + monkeypatch.setattr("backend.services.workbook_jobs.get_workbook_job_manager", lambda: manager) + monkeypatch.setattr("backend.excel.workbook_worker.run_worker", raise_unavailable) + + client = TestClient(app) + submit = client.post( + "/api/v1/workbook/jobs", + headers={"X-API-Key": API_KEY}, + json={ + "command": "open-workbook-v1", + "session_id": "worker-unavailable", + "payload": {"filename": "model.xlsx", "content_base64": "UEsDBAo="}, + }, + ) + assert submit.status_code == 200 + run_id = submit.json()["run_id"] + + final = None + for _ in range(20): + poll = client.get(f"/api/v1/workbook/jobs/{run_id}", headers={"X-API-Key": API_KEY}) + assert poll.status_code == 200 + final = poll.json() + if final["status"] in {"completed", "failed"}: + break + time.sleep(0.05) + + assert final is not None + assert final["status"] == "failed" + assert final["error"] == "Workbook worker is disabled or not installed" + + def test_semantic_map_endpoint_extracts_assumptions_formulas_and_risks(): client = TestClient(app) response = client.post( @@ -436,7 +686,7 @@ def test_v3_plan_lifecycle_inspect_apply_and_verify_records_contract(): assert approve.status_code == 200 approved = approve.json() assert approved["operations"] - op = approved["operations"][0] + operations = approved["operations"] result = client.post( "/api/v1/tasks/apply/result", @@ -447,7 +697,7 @@ def test_v3_plan_lifecycle_inspect_apply_and_verify_records_contract(): "apply_token": approved["apply_token"], "status": "applied", "office_apply_result": { - "applied": 1, + "applied": len(operations), "results": [ { "operationId": op["id"], @@ -455,6 +705,7 @@ def test_v3_plan_lifecycle_inspect_apply_and_verify_records_contract(): "status": "success", "dirtyRanges": [{"sheet": op.get("sheet") or "Sheet1", "range": op.get("range") or "A1"}], } + for op in operations ], }, "post_apply_workbook_state": { @@ -464,11 +715,8 @@ def test_v3_plan_lifecycle_inspect_apply_and_verify_records_contract(): }, }, ) - assert result.status_code == 200 - verification = result.json()["verification"] - assert "v3" in verification - assert verification["v3"]["status"] in {"verified", "warning"} - assert verification["v3"]["changedRanges"] + assert result.status_code == 409 + assert "attempt" in result.json()["detail"].lower() or "incomplete" in result.json()["detail"].lower() def test_operation_registry_contract_endpoint_exposes_source_metadata(): diff --git a/backend/tests/test_operational_readiness.py b/backend/tests/test_operational_readiness.py new file mode 100644 index 0000000..24c9cbc --- /dev/null +++ b/backend/tests/test_operational_readiness.py @@ -0,0 +1,279 @@ +from __future__ import annotations + +from types import SimpleNamespace + +import pytest +from fastapi import FastAPI, Request +from fastapi.testclient import TestClient + +from backend.api.middleware.auth import AuthMiddleware, is_public_path +from backend.api.middleware.error_handler import ErrorHandlerMiddleware +from backend.logging_config import REDACTED, redact_structured_event +from backend.main import app +from backend.observability import route_template_from_scope + + +def test_liveness_is_minimal_and_does_not_probe_optional_providers(monkeypatch) -> None: + async def provider_health_must_not_run() -> dict[str, bool]: + raise AssertionError("liveness must not probe optional providers") + + monkeypatch.setattr("backend.main.llm_router.health", provider_health_must_not_run) + + response = TestClient(app).get("/health") + + assert response.status_code == 200 + assert response.json() == {"status": "alive", "version": "0.5.0"} + assert response.headers["X-Request-ID"] + + +def test_liveness_alias_has_the_same_minimal_contract() -> None: + response = TestClient(app).get("/health/live") + + assert response.status_code == 200 + assert response.json() == {"status": "alive", "version": "0.5.0"} + + +def test_readiness_checks_required_storage_but_not_optional_providers(monkeypatch) -> None: + class ReadyStore: + def list_sessions(self, _limit: int) -> list[dict]: + return [] + + def get_migration_status(self) -> dict: + return {"pending": []} + + async def provider_health_must_not_run() -> dict[str, bool]: + raise AssertionError("readiness must not probe optional providers") + + monkeypatch.setattr("backend.storage.get_session_store", lambda: ReadyStore()) + monkeypatch.setattr("backend.main.llm_router.health", provider_health_must_not_run) + + response = TestClient(app).get("/health/ready") + + assert response.status_code == 200 + assert response.json() == { + "status": "ready", + "version": "0.5.0", + "checks": {"storage": "ok", "migrations": "ok"}, + } + + +def test_readiness_fails_closed_without_leaking_storage_details(monkeypatch) -> None: + def unavailable_store() -> None: + raise RuntimeError("password=super-secret database host /private/internal.db") + + monkeypatch.setattr("backend.storage.get_session_store", unavailable_store) + + response = TestClient(app).get("/health/ready") + + assert response.status_code == 503 + assert response.json() == { + "status": "not_ready", + "version": "0.5.0", + "checks": {"storage": "error", "migrations": "unknown"}, + } + assert "super-secret" not in response.text + assert "/private/internal.db" not in response.text + + +def test_deep_diagnostics_are_not_an_always_public_health_surface() -> None: + assert is_public_path("/health") is True + assert is_public_path("/health/live") is True + assert is_public_path("/health/ready") is True + assert is_public_path("/health/diagnostics") is False + + +def test_deep_diagnostics_require_authentication_when_auth_is_enabled() -> None: + mini_app = FastAPI() + + @mini_app.get("/health/diagnostics") + async def diagnostics() -> dict[str, bool]: + return {"ok": True} + + mini_app.add_middleware( + AuthMiddleware, + api_key="operator-key", + enabled=True, + static_key_admin=True, + ) + client = TestClient(mini_app) + + assert client.get("/health/diagnostics").status_code == 401 + assert client.get("/health/diagnostics", headers={"X-API-Key": "operator-key"}).status_code == 200 + + +def test_deep_diagnostics_keep_provider_and_storage_details(monkeypatch) -> None: + class ReadyStore: + def list_sessions(self, _limit: int) -> list[dict]: + return [] + + def get_migration_status(self) -> dict: + return {"engine": "sqlite", "pending": []} + + async def provider_health() -> dict[str, bool]: + return {"local": True, "optional-cloud": False} + + monkeypatch.setattr("backend.storage.get_session_store", lambda: ReadyStore()) + monkeypatch.setattr("backend.main.llm_router.health", provider_health) + + response = TestClient(app).get("/health/diagnostics") + + assert response.status_code == 200 + payload = response.json() + assert payload["providers"] == {"local": True, "optional-cloud": False} + assert payload["diagnostics"]["database"] == "ok" + assert payload["diagnostics"]["migrations"] == {"engine": "sqlite", "pending": []} + + +def test_route_template_metric_label_never_contains_resource_identifiers() -> None: + scope = { + "route": SimpleNamespace(path="/api/v1/sessions/{session_id}/plans/{plan_id}"), + "path": "/api/v1/sessions/private-session/plans/private-plan", + } + + assert route_template_from_scope(scope) == "/api/v1/sessions/{session_id}/plans/{plan_id}" + assert route_template_from_scope({"path": "/private/unmatched/identifier"}) == "__unmatched__" + + +def test_request_metrics_middleware_records_the_matched_template(monkeypatch) -> None: + observations: list[tuple[str, str, int | str, float]] = [] + + def capture(method: str, endpoint: str, status: int | str, duration: float) -> None: + observations.append((method, endpoint, status, duration)) + + monkeypatch.setattr("backend.observability.record_request_metrics", capture) + + response = TestClient(app).get("/api/v1/templates/private-template-id") + + assert response.status_code == 404 + assert observations[-1][0:3] == ("GET", "/api/v1/templates/{template_id}", 404) + assert "private-template-id" not in repr(observations) + + +def test_request_id_is_propagated_to_state_response_and_sanitized_error() -> None: + mini_app = FastAPI() + + @mini_app.get("/explode") + async def explode(request: Request) -> None: + assert request.state.request_id == "edge-request-123" + raise ValueError("api_key=top-secret workbook=/private/client.xlsx") + + mini_app.add_middleware(ErrorHandlerMiddleware) + + response = TestClient(mini_app, raise_server_exceptions=False).get( + "/explode", + headers={"X-Request-ID": "edge-request-123"}, + ) + + assert response.status_code == 400 + assert response.headers["X-Request-ID"] == "edge-request-123" + assert response.json() == { + "error": { + "code": "INVALID_INPUT", + "message": "Invalid request payload.", + "request_id": "edge-request-123", + } + } + assert "top-secret" not in response.text + assert "client.xlsx" not in response.text + + +def test_invalid_untrusted_request_id_is_replaced() -> None: + mini_app = FastAPI() + + @mini_app.get("/ok") + async def ok() -> dict[str, bool]: + return {"ok": True} + + mini_app.add_middleware(ErrorHandlerMiddleware) + response = TestClient(mini_app).get("/ok", headers={"X-Request-ID": "bad\nforged"}) + + assert response.status_code == 200 + assert response.headers["X-Request-ID"] != "bad\nforged" + assert "\n" not in response.headers["X-Request-ID"] + + +def test_request_id_wraps_middleware_short_circuit_responses() -> None: + mini_app = FastAPI() + + @mini_app.get("/protected") + async def protected() -> dict[str, bool]: + return {"ok": True} + + mini_app.add_middleware(AuthMiddleware, api_key="expected", enabled=True) + mini_app.add_middleware(ErrorHandlerMiddleware) + + response = TestClient(mini_app).get("/protected") + + assert response.status_code == 401 + assert response.headers["X-Request-ID"] + + +@pytest.mark.parametrize( + ("exception", "status_code", "error_code", "message"), + [ + (ValueError("private input"), 400, "INVALID_INPUT", "Invalid request payload."), + (FileNotFoundError("/private/client.xlsx"), 404, "NOT_FOUND", "Resource not found."), + (PermissionError("private policy"), 403, "FORBIDDEN", "Forbidden."), + (TimeoutError("private provider"), 504, "TIMEOUT", "Upstream operation timed out."), + (RuntimeError("private internals"), 500, "INTERNAL_ERROR", "Internal server error."), + ], +) +def test_unhandled_exceptions_have_stable_sanitized_public_codes( + exception: Exception, + status_code: int, + error_code: str, + message: str, +) -> None: + mini_app = FastAPI() + + @mini_app.get("/explode") + async def explode() -> None: + raise exception + + mini_app.add_middleware(ErrorHandlerMiddleware) + response = TestClient(mini_app, raise_server_exceptions=False).get("/explode") + + assert response.status_code == status_code + assert response.json()["error"]["code"] == error_code + assert response.json()["error"]["message"] == message + assert str(exception) not in response.text + + +@pytest.mark.parametrize( + ("key", "value"), + [ + ("authorization", "Bearer secret-token"), + ("api_key", "secret-key"), + ("db", "/private/customer/sessions.db"), + ("session_id", "private-session"), + ("formula", "=SUM(ClientData!A:A)"), + ("filename", "/private/acquisition-model.xlsx"), + ("prompt", "Summarize confidential deal terms"), + ], +) +def test_structured_log_redaction_removes_sensitive_fields(key: str, value: str) -> None: + event = redact_structured_event(None, "info", {"event": "operation", key: value}) + + assert event[key] == REDACTED + assert value not in repr(event) + + +def test_structured_log_redaction_scrubs_nested_and_embedded_secrets() -> None: + event = redact_structured_event( + None, + "error", + { + "event": "provider_failure", + "metadata": { + "headers": {"Authorization": "Bearer nested-token"}, + "error": "request failed: password=hunter2&token=opaque-value", + }, + "request_id": "safe-correlation-id", + }, + ) + + rendered = repr(event) + assert "nested-token" not in rendered + assert "hunter2" not in rendered + assert "opaque-value" not in rendered + assert event["request_id"] == "safe-correlation-id" diff --git a/backend/tests/test_operator_readiness.py b/backend/tests/test_operator_readiness.py new file mode 100644 index 0000000..c11d647 --- /dev/null +++ b/backend/tests/test_operator_readiness.py @@ -0,0 +1,198 @@ +from __future__ import annotations + +import hashlib +import json +import os +import subprocess +import sys +from pathlib import Path +from types import SimpleNamespace + +import pytest + +from backend.operations.operator_cli import build_operator_report +from backend.operations.sqlite_rehearsal import ( + BackupIntegrityError, + backup_sqlite_database, + create_disposable_sqlite_fixture, + restore_and_verify_sqlite_backup, + run_sqlite_backup_restore_rehearsal, + verify_backup_manifest, +) + + +class _ReadyStore: + def list_sessions(self, _limit: int) -> list[dict]: + return [] + + def get_migration_status(self) -> dict: + return {"engine": "sqlite", "pending": [], "applied": ["baseline", "lifecycle"]} + + def get_session_lifecycle_state(self, _session_id: str) -> str: + return "active" + + def get_session_lifecycle_record(self, _session_id: str) -> dict: + return {"state": "active"} + + def session_deletion_verified(self, _session_id: str) -> bool: + return False + + +def _config(**overrides): + values = { + "VERSION": "0.5.0", + "CONFIG_PROFILE": "dev", + "PRODUCTION_MODE": False, + "HOST": "127.0.0.1", + "PORT": 8000, + "AUTH_ENABLED": True, + "RBAC_ENABLED": True, + "OIDC_ENABLED": False, + "STATIC_KEY_ADMIN_ENABLED": False, + "LOCAL_PAIRING_ENABLED": False, + "DB_ENGINE": "sqlite", + "API_KEY": "must-never-appear", + "DATABASE_URL": "postgresql://secret:secret@example.invalid/private", + "DB_PATH": "/private/customer/sessions.db", + } + values.update(overrides) + return SimpleNamespace(**values) + + +def test_operator_report_is_provider_free_redacted_and_lifecycle_aware() -> None: + report = build_operator_report( + config_module=_config(), + store_factory=lambda: _ReadyStore(), + ) + + assert report["status"] == "ready" + assert report["model_providers_contacted"] is False + assert report["storage"] == { + "configured_engine": "sqlite", + "connectivity": "ok", + "migration_status": "ok", + "pending_migration_count": 0, + "applied_migration_count": 2, + } + assert report["lifecycle"] == { + "status": "ready", + "required_capabilities": 3, + "available_capabilities": 3, + } + rendered = json.dumps(report, sort_keys=True) + assert "must-never-appear" not in rendered + assert "secret:secret" not in rendered + assert "customer/sessions.db" not in rendered + + +def test_operator_report_refuses_to_claim_live_postgres_evidence() -> None: + called = False + + def forbidden_store(): + nonlocal called + called = True + return _ReadyStore() + + report = build_operator_report( + config_module=_config(DB_ENGINE="postgres"), + store_factory=forbidden_store, + ) + + assert called is False + assert report["status"] == "not_ready" + assert report["storage"]["connectivity"] == "not_probed" + assert report["storage"]["external_evidence"] is False + + +def test_operator_report_counts_config_errors_without_echoing_them() -> None: + config = _config(PRODUCTION_MODE=True) + config.runtime_config_errors = lambda: ["secret value cli-private-key is invalid"] + + report = build_operator_report( + config_module=config, + store_factory=lambda: _ReadyStore(), + ) + + assert report["status"] == "not_ready" + assert report["configuration"]["validation"] == "error" + assert report["configuration"]["validation_error_count"] == 1 + assert "cli-private-key" not in json.dumps(report) + + +def test_operator_cli_stdout_is_one_machine_readable_document(tmp_path: Path) -> None: + env = { + **os.environ, + "DB_ENGINE": "sqlite", + "DB_PATH": str(tmp_path / "operator.db"), + "API_KEY": "cli-super-secret", + "DATABASE_URL": "postgresql://cli-secret@example.invalid/private", + } + result = subprocess.run( + [sys.executable, "-m", "backend.operations.operator_cli", "report"], + cwd=Path(__file__).parents[2], + env=env, + check=False, + text=True, + stdout=subprocess.PIPE, + stderr=subprocess.PIPE, + ) + + payload = json.loads(result.stdout) + assert result.returncode == 0 + assert payload["status"] == "ready" + assert result.stdout.count("\n") == 1 + assert "cli-super-secret" not in result.stdout + result.stderr + assert "cli-secret" not in result.stdout + result.stderr + assert str(tmp_path) not in result.stdout + result.stderr + + +def test_disposable_sqlite_backup_restore_rehearsal_verifies_manifest_and_integrity() -> None: + report = run_sqlite_backup_restore_rehearsal() + + assert report["status"] == "passed" + assert report["engine"] == "sqlite" + assert report["scope"] == "disposable_fixture_only" + assert report["live_postgres_tested"] is False + assert report["external_evidence"] is False + assert report["checks"] == { + "manifest_digest": "verified", + "backup_digest": "verified", + "sqlite_integrity": "ok", + "logical_restore": "verified", + } + assert len(report["artifact"]["sha256"]) == 64 + assert len(report["artifact"]["manifest_sha256"]) == 64 + + +@pytest.mark.parametrize("tamper_target", ["backup", "manifest"]) +def test_sqlite_backup_manifest_verification_fails_closed_on_tamper( + tmp_path: Path, + tamper_target: str, +) -> None: + source = tmp_path / "source.db" + backup = tmp_path / "backup.db" + manifest_path = tmp_path / "backup.manifest.json" + create_disposable_sqlite_fixture(source) + envelope = backup_sqlite_database(source, backup, manifest_path) + + if tamper_target == "backup": + backup.write_bytes(backup.read_bytes() + b"tampered") + else: + envelope["manifest"]["artifact_size"] += 1 + manifest_path.write_text(json.dumps(envelope), encoding="utf-8") + + with pytest.raises(BackupIntegrityError): + verify_backup_manifest(backup, manifest_path) + + +def test_restore_rechecks_digest_before_opening_backup(tmp_path: Path) -> None: + source = tmp_path / "source.db" + backup = tmp_path / "backup.db" + restored = tmp_path / "restored.db" + manifest_path = tmp_path / "backup.manifest.json" + create_disposable_sqlite_fixture(source) + backup_sqlite_database(source, backup, manifest_path) + backup.write_bytes(backup.read_bytes() + hashlib.sha256(b"tamper").digest()) + + with pytest.raises(BackupIntegrityError): + restore_and_verify_sqlite_backup(backup, manifest_path, restored) diff --git a/backend/tests/test_pdf_worker_windows_limits.py b/backend/tests/test_pdf_worker_windows_limits.py new file mode 100644 index 0000000..5bff8cf --- /dev/null +++ b/backend/tests/test_pdf_worker_windows_limits.py @@ -0,0 +1,294 @@ +from __future__ import annotations + +import ctypes +import json +import sys +from pathlib import Path +from types import SimpleNamespace +from typing import Any + +import pytest + +from backend.documents import pdf_worker + + +pytestmark = [pytest.mark.unit, pytest.mark.security] + + +class _FakeKernel32: + def __init__(self, *, set_result: int = 1, assign_result: int = 1) -> None: + self.set_result = set_result + self.assign_result = assign_result + self.calls: list[tuple[Any, ...]] = [] + self.limit_snapshot: dict[str, int] = {} + + def CreateJobObjectW(self, attributes: Any, name: Any) -> int: + self.calls.append(("CreateJobObjectW", attributes, name)) + return 101 + + def SetInformationJobObject(self, job: int, info_class: int, info_pointer: Any, size: int) -> int: + limits = ctypes.cast( + info_pointer, + ctypes.POINTER(pdf_worker._JobObjectExtendedLimitInformation), + ).contents + self.limit_snapshot = { + "process_cpu": limits.BasicLimitInformation.PerProcessUserTimeLimit, + "job_cpu": limits.BasicLimitInformation.PerJobUserTimeLimit, + "flags": limits.BasicLimitInformation.LimitFlags, + "process_memory": limits.ProcessMemoryLimit, + "job_memory": limits.JobMemoryLimit, + "size": size, + } + self.calls.append(("SetInformationJobObject", job, info_class)) + return self.set_result + + def AssignProcessToJobObject(self, job: int, process: int) -> int: + self.calls.append(("AssignProcessToJobObject", job, process)) + return self.assign_result + + def TerminateJobObject(self, job: int, exit_code: int) -> int: + self.calls.append(("TerminateJobObject", job, exit_code)) + return 1 + + def CloseHandle(self, handle: int) -> int: + self.calls.append(("CloseHandle", handle)) + return 1 + + +class _FakeGate: + def __init__(self, order: list[str]) -> None: + self.order = order + self.released = False + + def set(self) -> None: + self.order.append("gate.set") + self.released = True + + +class _FakeProcess: + def __init__(self, order: list[str], args: tuple[Any, ...], *, complete_after_release: bool) -> None: + self.order = order + self.args = args + self.complete_after_release = complete_after_release + self.pid = 4242 + self.sentinel = 31337 + self.exitcode: int | None = None + self.started = False + self.alive = False + + def start(self) -> None: + self.order.append("process.start") + self.started = True + self.alive = True + + def join(self, timeout: int) -> None: + self.order.append(f"process.join:{timeout}") + gate = self.args[3] + if self.complete_after_release and gate.released: + Path(self.args[1]).write_text( + json.dumps( + { + "ok": True, + "document": { + "filename": self.args[2], + "page_count": 0, + "text": "", + "tables": [], + "metadata": {}, + }, + } + ), + encoding="utf-8", + ) + self.alive = False + self.exitcode = 0 + + def is_alive(self) -> bool: + return self.alive + + def terminate(self) -> None: + self.order.append("process.terminate") + self.alive = False + self.exitcode = -1 + + def kill(self) -> None: + self.order.append("process.kill") + self.alive = False + self.exitcode = -9 + + +class _FakeContext: + def __init__(self, *, complete_after_release: bool) -> None: + self.order: list[str] = [] + self.gate = _FakeGate(self.order) + self.process: _FakeProcess | None = None + self.complete_after_release = complete_after_release + + def Event(self) -> _FakeGate: + return self.gate + + def Process(self, *, target: Any, args: tuple[Any, ...], daemon: bool) -> _FakeProcess: + del target, daemon + self.process = _FakeProcess(self.order, args, complete_after_release=self.complete_after_release) + return self.process + + +class _FakeWindowsJob: + def __init__(self, order: list[str], process: _FakeProcess) -> None: + self.order = order + self.process = process + + def terminate(self) -> None: + self.order.append("job.terminate") + self.process.alive = False + self.process.exitcode = -1 + + def close(self) -> None: + self.order.append("job.close") + + +def test_windows_job_installs_exact_cpu_memory_and_tree_limits(monkeypatch: pytest.MonkeyPatch) -> None: + kernel32 = _FakeKernel32() + monkeypatch.setattr(pdf_worker, "_load_kernel32", lambda: kernel32) + + job = pdf_worker._install_windows_job(31337) + + expected_flags = ( + pdf_worker._WINDOWS_JOB_OBJECT_LIMIT_PROCESS_TIME + | pdf_worker._WINDOWS_JOB_OBJECT_LIMIT_JOB_TIME + | pdf_worker._WINDOWS_JOB_OBJECT_LIMIT_PROCESS_MEMORY + | pdf_worker._WINDOWS_JOB_OBJECT_LIMIT_JOB_MEMORY + | pdf_worker._WINDOWS_JOB_OBJECT_LIMIT_KILL_ON_JOB_CLOSE + ) + assert kernel32.limit_snapshot == { + "process_cpu": pdf_worker.PDF_PARSE_CPU_SECONDS * pdf_worker._WINDOWS_TICKS_PER_SECOND, + "job_cpu": pdf_worker.PDF_PARSE_CPU_SECONDS * pdf_worker._WINDOWS_TICKS_PER_SECOND, + "flags": expected_flags, + "process_memory": pdf_worker.PDF_PARSE_MEMORY_BYTES, + "job_memory": pdf_worker.PDF_PARSE_MEMORY_BYTES, + "size": ctypes.sizeof(pdf_worker._JobObjectExtendedLimitInformation), + } + assert ("AssignProcessToJobObject", 101, 31337) in kernel32.calls + + job.close() + assert kernel32.calls[-1] == ("CloseHandle", 101) + + +def test_windows_job_assignment_failure_closes_job_handle(monkeypatch: pytest.MonkeyPatch) -> None: + kernel32 = _FakeKernel32(assign_result=0) + monkeypatch.setattr(pdf_worker, "_load_kernel32", lambda: kernel32) + + with pytest.raises(OSError, match="AssignProcessToJobObject"): + pdf_worker._install_windows_job(31337) + + assert kernel32.calls[-1] == ("CloseHandle", 101) + + +def test_windows_limit_configuration_failure_closes_job_before_process_is_opened(monkeypatch: pytest.MonkeyPatch) -> None: + kernel32 = _FakeKernel32(set_result=0) + monkeypatch.setattr(pdf_worker, "_load_kernel32", lambda: kernel32) + + with pytest.raises(OSError, match="SetInformationJobObject"): + pdf_worker._install_windows_job(31337) + + assert not any(call[0] == "AssignProcessToJobObject" for call in kernel32.calls) + assert kernel32.calls[-1] == ("CloseHandle", 101) + + +def test_windows_start_gate_releases_only_after_job_assignment(monkeypatch: pytest.MonkeyPatch) -> None: + context = _FakeContext(complete_after_release=True) + monkeypatch.setattr(pdf_worker.multiprocessing, "get_context", lambda method: context) + monkeypatch.setattr(pdf_worker, "_is_windows", lambda: True) + + def install_job(process_handle: int) -> _FakeWindowsJob: + assert process_handle == 31337 + assert context.process is not None and context.process.started + assert context.gate.released is False + context.order.append("job.assign") + return _FakeWindowsJob(context.order, context.process) + + monkeypatch.setattr(pdf_worker, "_install_windows_job", install_job) + + parsed = pdf_worker._run_isolated_parse(b"%PDF", "safe.pdf", 3) + + assert parsed.filename == "safe.pdf" + assert context.order[:4] == ["process.start", "job.assign", "gate.set", "process.join:3"] + assert context.order[-1] == "job.close" + + +def test_windows_limit_install_failure_never_releases_start_gate_and_stops_worker(monkeypatch: pytest.MonkeyPatch) -> None: + context = _FakeContext(complete_after_release=False) + monkeypatch.setattr(pdf_worker.multiprocessing, "get_context", lambda method: context) + monkeypatch.setattr(pdf_worker, "_is_windows", lambda: True) + monkeypatch.setattr( + pdf_worker, + "_install_windows_job", + lambda process_id: (_ for _ in ()).throw(OSError("job unavailable")), + ) + + with pytest.raises(pdf_worker.PDFWorkerError, match="resource limits could not be installed"): + pdf_worker._run_isolated_parse(b"%PDF", "blocked.pdf", 3) + + assert context.gate.released is False + assert context.process is not None and context.process.alive is False + assert "process.terminate" in context.order + + +def test_windows_timeout_terminates_job_tree_before_closing_handle(monkeypatch: pytest.MonkeyPatch) -> None: + context = _FakeContext(complete_after_release=False) + monkeypatch.setattr(pdf_worker.multiprocessing, "get_context", lambda method: context) + monkeypatch.setattr(pdf_worker, "_is_windows", lambda: True) + + def install_job(process_handle: int) -> _FakeWindowsJob: + assert process_handle == 31337 + assert context.process is not None + return _FakeWindowsJob(context.order, context.process) + + monkeypatch.setattr(pdf_worker, "_install_windows_job", install_job) + + with pytest.raises(pdf_worker.PDFWorkerTimeout, match="timed out"): + pdf_worker._run_isolated_parse(b"%PDF", "slow.pdf", 1) + + assert context.order.index("job.terminate") < context.order.index("job.close") + assert "process.terminate" not in context.order + + +def test_posix_worker_installs_cpu_file_and_address_space_limits(monkeypatch: pytest.MonkeyPatch) -> None: + calls: list[tuple[str, tuple[int, int]]] = [] + resource = SimpleNamespace( + RLIMIT_CPU="cpu", + RLIMIT_FSIZE="file", + RLIMIT_AS="memory", + setrlimit=lambda limit, values: calls.append((limit, values)), + ) + monkeypatch.setattr(pdf_worker, "_is_windows", lambda: False) + monkeypatch.setitem(sys.modules, "resource", resource) + + pdf_worker._apply_process_limits() + + assert calls == [ + ("cpu", (pdf_worker.PDF_PARSE_CPU_SECONDS, pdf_worker.PDF_PARSE_CPU_SECONDS)), + ("file", (pdf_worker.PDF_PARSE_MAX_OUTPUT_BYTES, pdf_worker.PDF_PARSE_MAX_OUTPUT_BYTES)), + ("memory", (pdf_worker.PDF_PARSE_MEMORY_BYTES, pdf_worker.PDF_PARSE_MEMORY_BYTES)), + ] + + +def test_worker_does_not_parse_when_start_gate_is_not_released(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None: + class ClosedGate: + def wait(self, timeout: int) -> bool: + assert timeout == pdf_worker._WORKER_START_GATE_TIMEOUT_SECONDS + return False + + monkeypatch.setattr( + pdf_worker, + "_apply_process_limits", + lambda: pytest.fail("resource setup must not run before the start gate"), + ) + output_path = tmp_path / "result.json" + + pdf_worker._parse_worker(str(tmp_path / "input.pdf"), str(output_path), "blocked.pdf", ClosedGate()) + + payload = json.loads(output_path.read_text(encoding="utf-8")) + assert payload["ok"] is False + assert payload["error_type"] == "PDFWorkerError" + assert "start gate" in payload["error"] diff --git a/backend/tests/test_phase2.py b/backend/tests/test_phase2.py index 90e021e..aa86f02 100644 --- a/backend/tests/test_phase2.py +++ b/backend/tests/test_phase2.py @@ -1,7 +1,5 @@ """Unit tests for Phase 2 components: formula validator, formatting engine, context extractor, diff engine.""" -import asyncio -import json import sys import os @@ -100,7 +98,6 @@ def test_formatting_engine(): """Test the IB/HF formatting engine.""" from backend.excel.formatting_engine import ( get_format_dict, infer_number_format, post_process_operations, - PROFILES, NUMBER_FORMATS, ) print("=" * 60) diff --git a/backend/tests/test_phase3.py b/backend/tests/test_phase3.py index cd322b2..9408ece 100644 --- a/backend/tests/test_phase3.py +++ b/backend/tests/test_phase3.py @@ -1,10 +1,8 @@ """Phase 3 Tests — Auth, Rules, Cache, Validators, Document Pipeline.""" -import json import time -from pathlib import Path from types import SimpleNamespace -from unittest.mock import MagicMock, AsyncMock, patch +from unittest.mock import MagicMock import pytest @@ -175,7 +173,7 @@ def test_invalidate(self): # AI RULES # ═══════════════════════════════════════════════════════════════════════ -from backend.agents.rules import RulesEngine, Rule +from backend.agents.rules import RulesEngine class TestRulesEngine: diff --git a/backend/tests/test_plan_state_machine.py b/backend/tests/test_plan_state_machine.py new file mode 100644 index 0000000..fbc83f3 --- /dev/null +++ b/backend/tests/test_plan_state_machine.py @@ -0,0 +1,91 @@ +"""Plan state machine and supporting safety helpers.""" + +from __future__ import annotations + +import pytest + +from backend.services.cell_value_safety import ( + is_potential_formula, + sanitize_literal, +) +from backend.services.plan_state_machine import ( + InvalidPlanTransition, + can_transition, + ensure_transition, +) + + +@pytest.mark.unit +class TestPlanStateMachine: + def test_plan_status_valid_path(self) -> None: + assert can_transition("plan", "draft", "approved") + assert can_transition("plan", "approved", "applied") + assert can_transition("plan", "applied", "rollback_pending") + assert can_transition("plan", "rollback_pending", "rolled_back") + assert not can_transition("plan", "applied", "rolled_back") + + def test_plan_status_terminal_rejects(self) -> None: + assert not can_transition("plan", "failed", "approved") + assert not can_transition("plan", "rolled_back", "applied") + assert not can_transition("plan", "abandoned", "approved") + + def test_plan_status_cannot_skip_approval(self) -> None: + assert not can_transition("plan", "draft", "applied") + + def test_plan_status_unknown_state_rejects(self) -> None: + assert not can_transition("plan", "fictional_state", "approved") + + def test_v3_lifecycle_valid_path(self) -> None: + assert can_transition("v3", "draft", "ready") + assert can_transition("v3", "ready", "approved") + assert can_transition("v3", "approved", "applied") + assert can_transition("v3", "applied", "verified") + + def test_v3_lifecycle_terminal(self) -> None: + assert not can_transition("v3", "verified", "applied") + assert not can_transition("v3", "failed", "approved") + + def test_ensure_transition_raises_on_illegal(self) -> None: + with pytest.raises(InvalidPlanTransition) as exc: + ensure_transition("plan", "failed", "approved", plan_id="plan_42") + assert exc.value.plan_id == "plan_42" + assert exc.value.kind == "plan" + + def test_ensure_transition_idempotent_on_same_state(self) -> None: + # Re-applying a state is allowed by default (the table lists each + # state as a permitted target of itself). + ensure_transition("plan", "applied", "applied") + + +@pytest.mark.unit +class TestCellValueSafety: + def test_formula_trigger_detected(self) -> None: + assert is_potential_formula("=SUM(A1:A10)") + assert is_potential_formula("@MyFunc()") + assert is_potential_formula("\tHIDDEN_FORMULA") + + def test_negative_number_literal_not_flagged(self) -> None: + assert not is_potential_formula("-5") + assert not is_potential_formula("-1.25") + + def test_negative_function_call_flagged(self) -> None: + # "-SUM(A1)" should be flagged: starts with - and contains a function call. + assert is_potential_formula("-SUM(A1)") + + def test_plain_text_not_flagged(self) -> None: + assert not is_potential_formula("Revenue") + assert not is_potential_formula("") + assert not is_potential_formula(None) + assert not is_potential_formula(123) + + def test_sanitize_literal_prepends_quote(self) -> None: + assert sanitize_literal("=DANGER()") == "'=DANGER()" + + def test_sanitize_literal_is_idempotent(self) -> None: + # Already-quoted strings should not be double-quoted. + assert sanitize_literal("'=DANGER()") == "'=DANGER()" + + def test_sanitize_literal_passthrough_for_safe(self) -> None: + assert sanitize_literal("Revenue") == "Revenue" + assert sanitize_literal(42) == "42" + assert sanitize_literal(None) == "" diff --git a/backend/tests/test_postgres_backfill_sql_contract.py b/backend/tests/test_postgres_backfill_sql_contract.py new file mode 100644 index 0000000..484e1a6 --- /dev/null +++ b/backend/tests/test_postgres_backfill_sql_contract.py @@ -0,0 +1,215 @@ +import inspect +import json +import re +import subprocess +import sys +from dataclasses import dataclass +from pathlib import Path + +import pytest + +from backend.storage.postgres import ( + POSTGRES_TYPED_PROJECTION_BACKFILLS, + POSTGRES_TYPED_PROJECTION_MIGRATIONS, + POSTGRES_TYPED_PROJECTION_RECONCILIATION, + POSTGRES_TYPED_PROJECTION_TABLES, + PostgresSessionStore, +) + + +ROOT = Path(__file__).resolve().parents[2] + + +@dataclass(frozen=True) +class _BackfillSqlContract: + target_table: str + source_table: str + insert_columns: list[str] + select_expressions: list[str] + conflict_columns: list[str] + + +def _split_top_level_csv(sql_fragment: str) -> list[str]: + items: list[str] = [] + start = 0 + depth = 0 + quote: str | None = None + index = 0 + while index < len(sql_fragment): + char = sql_fragment[index] + if quote: + if char == quote: + if quote == "'" and index + 1 < len(sql_fragment) and sql_fragment[index + 1] == "'": + index += 2 + continue + quote = None + index += 1 + continue + if char in {"'", '"'}: + quote = char + elif char == "(": + depth += 1 + elif char == ")": + depth -= 1 + elif char == "," and depth == 0: + items.append(sql_fragment[start:index].strip()) + start = index + 1 + index += 1 + tail = sql_fragment[start:].strip() + if tail: + items.append(tail) + return items + + +def _parse_backfill_sql(sql: str) -> _BackfillSqlContract: + insert_match = re.search( + r""" + \bINSERT\s+INTO\s+(?P[a-z_][a-z0-9_]*) + \s*\((?P.*?)\) + \s*SELECT\s+(?P.*?) + \s+FROM\s+(?P[a-z_][a-z0-9_]*)\b + """, + sql, + flags=re.IGNORECASE | re.DOTALL | re.VERBOSE, + ) + assert insert_match, f"Could not parse INSERT ... SELECT backfill SQL:\n{sql}" + conflict_match = re.search( + r"\bON\s+CONFLICT\s*\((?P.*?)\)\s+DO\s+NOTHING\b", + sql, + flags=re.IGNORECASE | re.DOTALL, + ) + assert conflict_match, f"Backfill SQL is missing ON CONFLICT(... ) DO NOTHING:\n{sql}" + return _BackfillSqlContract( + target_table=insert_match.group("target"), + source_table=insert_match.group("source"), + insert_columns=[ + column.strip().strip('"') + for column in _split_top_level_csv(insert_match.group("insert_columns")) + ], + select_expressions=_split_top_level_csv(insert_match.group("select_expressions")), + conflict_columns=[ + column.strip().strip('"') + for column in _split_top_level_csv(conflict_match.group("conflict_columns")) + ], + ) + + +def _typed_table_schema_contract(table: str) -> tuple[list[str], str]: + schema_source = inspect.getsource(PostgresSessionStore._init_schema) + table_match = re.search( + rf"\bCREATE\s+TABLE\s+IF\s+NOT\s+EXISTS\s+{re.escape(table)}\s*\((?P.*?)\n\s*\);", + schema_source, + flags=re.IGNORECASE | re.DOTALL, + ) + assert table_match, f"Could not find CREATE TABLE contract for {table}" + columns: list[str] = [] + primary_key = "" + for definition in _split_top_level_csv(table_match.group("body")): + tokens = definition.strip().split() + if not tokens or tokens[0].upper() in {"CONSTRAINT", "PRIMARY", "FOREIGN", "UNIQUE", "CHECK"}: + continue + column = tokens[0].strip('"') + columns.append(column) + if "PRIMARY KEY" in definition.upper(): + primary_key = column + if not primary_key: + primary_key_match = re.search( + r"\bPRIMARY\s+KEY\s*\((?P.*?)\)", + table_match.group("body"), + flags=re.IGNORECASE | re.DOTALL, + ) + assert primary_key_match, f"Could not find primary key contract for {table}" + primary_key = _split_top_level_csv(primary_key_match.group("columns"))[0].strip().strip('"') + return columns, primary_key + + +def _projection_ledger_details() -> dict[str, dict]: + schema_source = inspect.getsource(PostgresSessionStore._init_schema) + ledger_rows = dict( + re.findall( + r"\('([^']+)',\s*EXTRACT\(EPOCH FROM NOW\(\)\),\s*'([^']+)'::jsonb\)", + schema_source, + ) + ) + return { + migration_id: json.loads(ledger_rows[migration_id]) + for migration_id in POSTGRES_TYPED_PROJECTION_MIGRATIONS + if migration_id in ledger_rows + } + + +def test_postgres_typed_projection_backfill_registry_matches_migration_mapping_and_status_report(): + typed_tables = set(POSTGRES_TYPED_PROJECTION_TABLES) + migration_tables = [ + table + for tables in POSTGRES_TYPED_PROJECTION_MIGRATIONS.values() + for table in tables + ] + status = PostgresSessionStore("postgresql://offline").get_migration_status() + + assert len(POSTGRES_TYPED_PROJECTION_TABLES) == len(typed_tables) + assert set(POSTGRES_TYPED_PROJECTION_BACKFILLS) == typed_tables + assert len(migration_tables) == len(set(migration_tables)) + assert set(migration_tables) == typed_tables + assert status["typed_projections"] == POSTGRES_TYPED_PROJECTION_TABLES + assert status["projection_migrations"] == POSTGRES_TYPED_PROJECTION_MIGRATIONS + assert status["typed_projection_backfills"] == { + table: len(statements) + for table, statements in POSTGRES_TYPED_PROJECTION_BACKFILLS.items() + } + + +@pytest.mark.parametrize( + ("table", "statements"), + sorted(POSTGRES_TYPED_PROJECTION_BACKFILLS.items()), +) +def test_postgres_typed_projection_backfill_insert_select_contract_matches_schema(table, statements): + schema_columns, primary_key = _typed_table_schema_contract(table) + + assert len(statements) == 1 + for statement in statements: + parsed = _parse_backfill_sql(statement) + + assert parsed.target_table == table + assert parsed.source_table == "flagship_artifacts" + assert parsed.insert_columns == schema_columns + assert parsed.insert_columns[0] == primary_key + assert parsed.conflict_columns == [primary_key] + assert len(parsed.insert_columns) == len(parsed.select_expressions) + + +def test_postgres_projection_migration_ledger_details_match_mapping(): + ledger_details = _projection_ledger_details() + + assert set(ledger_details) == set(POSTGRES_TYPED_PROJECTION_MIGRATIONS) + for migration_id, tables in POSTGRES_TYPED_PROJECTION_MIGRATIONS.items(): + assert ledger_details[migration_id]["backfill"] is True + assert ledger_details[migration_id]["schema"].split("+") == tables + + +def test_agent_run_artifact_backfill_covers_every_dual_write_generic_type(): + sql = "\n".join(POSTGRES_TYPED_PROJECTION_BACKFILLS["agent_run_artifacts"]) + + assert "artifact_type LIKE 'agent_run:%'" in sql + assert "artifact_type = 'agent_run_step_artifact_v3'" in sql + assert "artifact_type = 'brain_run'" in sql + assert "artifact_type = 'brain_artifact_bundle'" in sql + + +def test_projection_reconciliation_registry_covers_every_typed_backfill(): + assert set(POSTGRES_TYPED_PROJECTION_RECONCILIATION) == set( + POSTGRES_TYPED_PROJECTION_BACKFILLS + ) + + +def test_postgres_validation_script_direct_entrypoint_can_load_project(): + result = subprocess.run( + [sys.executable, "scripts/validate_postgres_migration_backfill.py", "--help"], + cwd=ROOT, + text=True, + stdout=subprocess.PIPE, + stderr=subprocess.PIPE, + check=False, + ) + + assert result.returncode == 0, result.stdout + result.stderr diff --git a/backend/tests/test_postgres_lifecycle_fencing.py b/backend/tests/test_postgres_lifecycle_fencing.py new file mode 100644 index 0000000..ad75510 --- /dev/null +++ b/backend/tests/test_postgres_lifecycle_fencing.py @@ -0,0 +1,697 @@ +"""PostgreSQL cross-process lifecycle fencing regressions.""" + +from __future__ import annotations + +import asyncio +import os +import time +import uuid +from contextlib import asynccontextmanager + +import pytest + +from backend.agents import workflows as workflows_module +from backend.services import lifecycle as lifecycle_module +from backend.services.lifecycle import LifecycleCoordinator, LifecycleState +from backend.api.security.ownership import OwnershipNotFoundError, ownership_actor_scope +from backend.storage.postgres import PostgresSessionStore + + +@pytest.fixture(autouse=True) +def _lifecycle_receipt_signing_key(monkeypatch): + monkeypatch.setenv("EXCEL_AGENT_LIFECYCLE_RECEIPT_KEY_ID", "test-current") + monkeypatch.setenv("EXCEL_AGENT_LIFECYCLE_RECEIPT_HMAC_KEY", "cnJy" * 16) + + +class _Acquire: + def __init__(self, connection): + self.connection = connection + + async def __aenter__(self): + return self.connection + + async def __aexit__(self, exc_type, exc, traceback): + return False + + +class _Pool: + def __init__(self, connection): + self.connection = connection + + def acquire(self): + return _Acquire(self.connection) + + +class _SchemaConnection: + def __init__(self) -> None: + self.executed: list[tuple[str, tuple[object, ...]]] = [] + + @asynccontextmanager + async def transaction(self): + yield + + async def execute(self, query: str, *params): + self.executed.append((query, params)) + return "INSERT 0 1" + + async def fetch(self, query: str, *params): + self.executed.append((query, params)) + return [] + + async def fetchrow(self, query: str, *params): + self.executed.append((query, params)) + if "JOIN session_lifecycle" in query: + return {"owner_user_id": "", "lifecycle_state": "active"} + return None + + async def fetchval(self, query: str, *params): + self.executed.append((query, params)) + return None + + +@pytest.mark.unit +@pytest.mark.asyncio +async def test_postgres_schema_installs_durable_session_lifecycle_fences(): + connection = _SchemaConnection() + store = PostgresSessionStore("postgresql://example") + store._pool = _Pool(connection) + + await store._init_schema() + + schema_sql = "\n".join(query for query, _params in connection.executed) + assert "CREATE TABLE IF NOT EXISTS session_lifecycle" in schema_sql + assert "enforce_session_lifecycle_write" in schema_sql + assert "pg_advisory_xact_lock" in schema_sql + for table in ( + "sessions", + "conversations", + "task_history", + "flagship_artifacts", + "local_replay_artifacts", + "local_replay_baselines", + "agent_run_artifacts", + "verification_reports_v2", + "operation_audits", + "workbook_maps", + "source_evidence_graphs", + "operation_simulations", + "workbook_invariant_runs", + "agent_runs", + "workbook_truth_models", + "formula_graph_artifacts", + "operation_simulations_v2", + "v7_workbook_reviews", + ): + assert table in schema_sql + assert "postgres_session_lifecycle_fencing_v1" in schema_sql + + +@pytest.mark.unit +@pytest.mark.asyncio +async def test_lifecycle_coordinator_persists_database_states_around_cleanup( + tmp_path, + monkeypatch, +): + events: list[str] = [] + + class Store: + present = True + + @classmethod + def set_session_lifecycle_state( + cls, + session_id: str, + state: str, + *, + receipt_id: str, + started_at: float, + completed_at: float = 0.0, + errors: list[str] | None = None, + ) -> None: + assert session_id == "session-1" + assert receipt_id.startswith("del_") + assert started_at > 0 + if state == LifecycleState.PURGING.value: + assert completed_at == 0 + assert not errors + else: + assert completed_at >= started_at + events.append(f"state:{state}") + + @classmethod + def session_exists(cls, _session_id: str) -> bool: + events.append("session_exists") + return cls.present + + @staticmethod + def list_document_file_paths(_session_id: str) -> list[str]: + events.append("document_inventory") + return [] + + @classmethod + def delete_session(cls, _session_id: str) -> bool: + events.append("database_delete") + cls.present = False + return True + + class Jobs: + @staticmethod + def purge_session(_session_id: str): + events.append("jobs") + return {"jobs_revoked": 0, "threads_pending": 0} + + class Workflows: + @staticmethod + def purge_session(_session_id: str): + events.append("workflows") + return 0 + + @staticmethod + def count_session_workflows(_session_id: str): + events.append("workflow_inventory") + return 0 + + monkeypatch.setattr(workflows_module, "get_workflow_engine", lambda: Workflows()) + coordinator = LifecycleCoordinator( + store=Store(), + job_manager=Jobs(), + receipts_dir=tmp_path / "receipts", + ) + + receipt = await coordinator.purge_session("session-1") + + assert receipt.state is LifecycleState.VERIFIED + assert events[0] == "state:purging" + assert events.index("state:purging") < events.index("document_inventory") + assert events.index("database_delete") < events.index("state:verified") + assert events[-1] == "state:verified" + + +@pytest.mark.unit +@pytest.mark.asyncio +async def test_postgres_conversation_and_task_writes_do_not_create_sessions_implicitly(): + connection = _SchemaConnection() + store = PostgresSessionStore("postgresql://example") + store._pool = _Pool(connection) + + async def implicit_creation_forbidden(*_args, **_kwargs): + pytest.fail("low-level session write attempted to create a session") + + store._async_get_or_create_session = implicit_creation_forbidden + + await store._async_add_message("session-1", "user", "hello") + await store._async_record_task( + "session-1", + "task-1", + "hello", + "action", + [], + "done", + [], + "local", + "model", + {}, + ) + + sql = "\n".join(query for query, _params in connection.executed) + assert "INSERT INTO conversations" in sql + assert "INSERT INTO task_history" in sql + + +@pytest.mark.integration +@pytest.mark.asyncio +async def test_live_postgres_lifecycle_race_and_restart_proof(tmp_path, monkeypatch): + dsn = os.environ.get("EXCEL_AGENT_TEST_POSTGRES_DSN", "").strip() + if not dsn: + pytest.skip("EXCEL_AGENT_TEST_POSTGRES_DSN is not configured") + + session_id = f"lifecycle-live-{uuid.uuid4().hex}" + other_session_id = f"lifecycle-live-other-{uuid.uuid4().hex}" + receipt_id = f"del_{uuid.uuid4().hex}" + collision_id = f"collision-{uuid.uuid4().hex}" + composite_map_id = f"map-composite-{uuid.uuid4().hex}" + global_suffix = uuid.uuid4().hex + job_id = f"job-before-{global_suffix}" + run_id = f"run-before-{global_suffix}" + run_artifact_id = f"artifact-before-{global_suffix}" + started_at = time.time() + store_a = PostgresSessionStore(dsn) + store_b = PostgresSessionStore(dsn) + store_c = PostgresSessionStore(dsn) + writer_connection = None + writer_transaction = None + try: + monkeypatch.setattr( + lifecycle_module, + "_coordinator", + LifecycleCoordinator(receipts_dir=tmp_path / "receipts"), + ) + await store_a._get_pool() + await store_b._get_pool() + await store_a._async_get_or_create_session(session_id) + await store_a._async_get_or_create_session(other_session_id) + await store_a._async_add_message(session_id, "user", "before purge") + await store_a._async_record_task( + session_id, + "task-before", + "before purge", + "action", + [], + "recorded", + [], + "local", + "model", + {}, + ) + await store_a._async_save_artifact( + job_id, + "workbook_job", + session_id, + job_id, + "", + {"run_id": job_id, "session_id": session_id}, + ) + await store_a._async_save_agent_run_artifact( + run_artifact_id, + run_id, + session_id, + "step:inspect", + "", + {"artifactType": "inspect"}, + generic_artifact_type="agent_run_step_artifact_v3", + ) + await store_a._async_save_artifact( + collision_id, + "operation_plan", + session_id, + "", + collision_id, + {"plan_id": collision_id, "session_id": session_id}, + ) + with pytest.raises(Exception, match="session ownership is immutable"): + await store_a._async_save_artifact( + collision_id, + "operation_plan", + other_session_id, + "", + collision_id, + {"plan_id": collision_id, "session_id": other_session_id}, + ) + await store_a._async_save_artifact( + composite_map_id, + "workbook_map", + session_id, + "", + "", + {"mapId": composite_map_id, "sessionId": session_id}, + ) + + global_artifacts = { + "enterprise_user": f"enterprise_user:{global_suffix}", + "enterprise_api_key": f"enterprise_api_key:{global_suffix}", + "enterprise_config": f"enterprise_config:{global_suffix}", + } + for artifact_type, artifact_id in global_artifacts.items(): + payload = {"artifact_id": artifact_id, "kind": artifact_type} + if artifact_type == "enterprise_user": + payload["status"] = "api_key_only" + await store_a._async_save_artifact( + artifact_id, + artifact_type, + "", + "", + "", + payload, + ) + assert (await store_a._async_get_artifact(artifact_id, artifact_type))["kind"] == artifact_type + + attacker = {"user_id": "attacker", "role": "analyst"} + admin = {"user_id": "real-admin", "role": "admin"} + with ownership_actor_scope(attacker): + await store_a._async_get_or_create_session("enterprise-admin") + with ownership_actor_scope(admin): + canonical_admin_audit_id = await store_a._async_record_enterprise_admin_audit( + "enterprise_admin.user_updated", + {"actor": {"email": "real-admin@example.com"}, "secret": "rotated"}, + ) + admin_audits = await store_a._async_list_enterprise_admin_audits() + assert admin_audits[0]["audit_id"] == canonical_admin_audit_id + with ownership_actor_scope(attacker): + with pytest.raises(OwnershipNotFoundError): + await store_a._async_list_enterprise_admin_audits() + attacker_rows = await store_a._async_list_artifacts_for_session("enterprise-admin") + assert all(row["artifact_id"] != canonical_admin_audit_id for row in attacker_rows) + with pytest.raises(OwnershipNotFoundError): + await store_a._async_save_artifact( + f"audit-lookalike-{global_suffix}", + "audit:enterprise_admin.user_updated", + "enterprise-admin-lookalike", + "", + "", + {}, + ) + active_workbook_job_count = await store_a._async_count_artifacts_by_prefix("workbook_job") + ordinary_audit_id = f"audit-non-admin-{global_suffix}" + with ownership_actor_scope(attacker): + await store_a._async_save_artifact( + ordinary_audit_id, + "audit:user_updated", + "enterprise-admin", + "", + "", + {}, + ) + attacker_rows = await store_a._async_list_artifacts_for_session("enterprise-admin") + assert ordinary_audit_id in {row["artifact_id"] for row in attacker_rows} + assert canonical_admin_audit_id not in {row["artifact_id"] for row in attacker_rows} + + pool_a = await store_a._get_pool() + writer_connection = await pool_a.acquire() + writer_transaction = writer_connection.transaction() + await writer_transaction.start() + await writer_connection.execute( + "SELECT pg_advisory_xact_lock(hashtextextended($1, 0))", + session_id, + ) + await writer_connection.execute( + "INSERT INTO conversations (session_id, role, content, timestamp, metadata) " + "VALUES ($1, 'assistant', 'committed before purge', $2, '{}'::jsonb)", + session_id, + time.time(), + ) + + begin_purge = asyncio.create_task( + store_b._async_set_session_lifecycle_state( + session_id, + "purging", + receipt_id=receipt_id, + started_at=started_at, + ) + ) + await asyncio.sleep(0.05) + assert not begin_purge.done(), "purge did not wait for the in-flight writer" + await writer_transaction.commit() + writer_transaction = None + await pool_a.release(writer_connection) + writer_connection = None + await asyncio.wait_for(begin_purge, timeout=5) + + assert await store_b._async_get_session_lifecycle_state(session_id) == "purging" + assert await store_b._async_get_conversation(session_id) == [] + assert await store_b._async_get_task_history(session_id) == [] + assert await store_b._async_get_artifact(job_id, "workbook_job") is None + assert await store_b._async_list_workbook_jobs(session_id=session_id) == [] + assert await store_b._async_list_agent_run_artifacts(run_id) == [] + assert await store_b._async_list_run_artifacts(run_id) == [] + assert await store_b._async_count_artifacts_by_prefix("workbook_job") == active_workbook_job_count - 1 + assert session_id not in { + str(session.get("session_id") or "") + for session in await store_b._async_list_sessions(limit=1000) + } + with pytest.raises(OwnershipNotFoundError): + await store_a._async_get_or_create_session(session_id) + with pytest.raises(OwnershipNotFoundError): + await store_a._async_add_message(session_id, "assistant", "late") + with pytest.raises(OwnershipNotFoundError): + await store_a._async_record_task( + session_id, + "task-late", + "late", + "action", + [], + "late", + [], + "local", + "model", + {}, + ) + with pytest.raises(OwnershipNotFoundError): + await store_a._async_save_artifact( + "job-late", + "workbook_job", + session_id, + "job-late", + "", + {"run_id": "job-late", "session_id": session_id}, + ) + + pool_b = await store_b._get_pool() + async with pool_b.acquire() as connection: + with pytest.raises(Exception, match="session lifecycle rejected durable write"): + await connection.execute( + "INSERT INTO workbook_maps " + "(map_id, session_id, created_at, updated_at) VALUES ($1, $2, $3, $3)", + composite_map_id, + session_id, + time.time(), + ) + + # Unrelated global metadata and the isolated administrator audit stream + # remain available while this session is purging. + for artifact_type, artifact_id in global_artifacts.items(): + assert (await store_b._async_get_artifact(artifact_id, artifact_type))["kind"] == artifact_type + with ownership_actor_scope(admin): + admin_audits = await store_b._async_list_enterprise_admin_audits() + assert admin_audits[0]["audit_id"] == canonical_admin_audit_id + + assert await store_b._async_delete_session(session_id) is True + await store_b._async_set_session_lifecycle_state( + session_id, + "verified", + receipt_id=receipt_id, + started_at=started_at, + completed_at=time.time(), + ) + + await store_c._get_pool() + assert await store_c._async_get_session_lifecycle_state(session_id) == "verified" + with pytest.raises(OwnershipNotFoundError): + await store_c._async_get_or_create_session(session_id) + async with (await store_c._get_pool()).acquire() as connection: + assert not await store_c._async_session_has_durable_rows(connection, session_id) + with pytest.raises(Exception, match="session lifecycle rejected durable write"): + await connection.execute( + "INSERT INTO sessions (session_id, created_at, updated_at) VALUES ($1, $2, $2)", + session_id, + time.time(), + ) + for artifact_type, artifact_id in global_artifacts.items(): + assert (await store_c._async_get_artifact(artifact_id, artifact_type))["kind"] == artifact_type + finally: + if writer_transaction is not None: + await writer_transaction.rollback() + if writer_connection is not None: + await (await store_a._get_pool()).release(writer_connection) + await store_a.close() + await store_b.close() + await store_c.close() + + +@pytest.mark.integration +@pytest.mark.asyncio +async def test_live_postgres_sensitive_read_and_purge_have_one_linearization_boundary( + monkeypatch, +): + dsn = os.environ.get("EXCEL_AGENT_TEST_POSTGRES_DSN", "").strip() + if not dsn: + pytest.skip("EXCEL_AGENT_TEST_POSTGRES_DSN is not configured") + + session_id = f"lifecycle-live-read-{uuid.uuid4().hex}" + receipt_id = f"del_{uuid.uuid4().hex}" + reader_store = PostgresSessionStore(dsn) + purger_store = PostgresSessionStore(dsn) + visibility_checked = asyncio.Event() + release_reader = asyncio.Event() + original_visible = reader_store._async_session_visible_in_transaction + + async def paused_visibility(conn, checked_session_id: str) -> bool: + visible = await original_visible(conn, checked_session_id) + visibility_checked.set() + await asyncio.wait_for(release_reader.wait(), timeout=5) + return visible + + monkeypatch.setattr( + reader_store, + "_async_session_visible_in_transaction", + paused_visibility, + ) + try: + await reader_store._async_get_or_create_session(session_id) + await reader_store._async_add_message(session_id, "user", "secret") + reader = asyncio.create_task(reader_store._async_get_conversation(session_id)) + await asyncio.wait_for(visibility_checked.wait(), timeout=5) + purge = asyncio.create_task( + purger_store._async_set_session_lifecycle_state( + session_id, + "purging", + receipt_id=receipt_id, + started_at=time.time(), + ) + ) + await asyncio.sleep(0.1) + assert not purge.done(), "PURGING committed while an authorized read held its session lock" + release_reader.set() + messages = await asyncio.wait_for(reader, timeout=5) + await asyncio.wait_for(purge, timeout=5) + + assert [message.content for message in messages] == ["secret"] + assert await purger_store._async_get_conversation(session_id) == [] + finally: + await reader_store.close() + await purger_store.close() + + +@pytest.mark.integration +@pytest.mark.asyncio +async def test_live_postgres_concurrent_profile_patch_preserves_disabled_security_state(): + dsn = os.environ.get("EXCEL_AGENT_TEST_POSTGRES_DSN", "").strip() + if not dsn: + pytest.skip("EXCEL_AGENT_TEST_POSTGRES_DSN is not configured") + + user_id = f"metadata-live-{uuid.uuid4().hex}" + store_a = PostgresSessionStore(dsn) + store_b = PostgresSessionStore(dsn) + start = asyncio.Event() + try: + await store_a._async_upsert_enterprise_user( + user_id=user_id, + email=f"{user_id}@example.com", + name="Original", + role="analyst", + status="active", + mfa_enabled=True, + metadata={"base": "preserve"}, + oidc_issuer="https://issuer.example", + oidc_subject=f"subject-{user_id}", + ) + + async def disable_user(): + await start.wait() + return await store_a._async_upsert_enterprise_user( + user_id=user_id, + email=f"{user_id}@example.com", + name="Original", + role="analyst", + status="disabled", + mfa_enabled=True, + metadata={"security_disabled_by": "admin"}, + oidc_issuer=None, + oidc_subject=None, + ) + + async def refresh_profile(): + await start.wait() + return await store_b._async_upsert_enterprise_user( + user_id=user_id, + email=f"{user_id}@example.com", + name="Updated profile", + role="analyst", + status=None, + mfa_enabled=None, + metadata={"profile_refresh": True}, + oidc_issuer=None, + oidc_subject=None, + ) + + disabled = asyncio.create_task(disable_user()) + refreshed = asyncio.create_task(refresh_profile()) + start.set() + await asyncio.gather(disabled, refreshed) + + final = await store_a._async_get_artifact( + store_a._enterprise_user_artifact_id(user_id), + "enterprise_user", + ) + assert final is not None + assert final["status"] == "disabled" + assert final["mfa_enabled"] is True + assert final["metadata"]["base"] == "preserve" + assert final["metadata"]["security_disabled_by"] == "admin" + assert final["metadata"]["profile_refresh"] is True + assert final["oidc_issuer"] == "https://issuer.example" + assert final["oidc_subject"] == f"subject-{user_id}" + finally: + await store_a.close() + await store_b.close() + + +@pytest.mark.integration +@pytest.mark.asyncio +async def test_live_postgres_restart_repairs_pre_atomic_typed_projection_drift(): + dsn = os.environ.get("EXCEL_AGENT_TEST_POSTGRES_DSN", "").strip() + if not dsn: + pytest.skip("EXCEL_AGENT_TEST_POSTGRES_DSN is not configured") + + session_id = f"typed-repair-session-{uuid.uuid4().hex}" + map_id = f"typed-repair-map-{uuid.uuid4().hex}" + writer = PostgresSessionStore(dsn) + restarted = PostgresSessionStore(dsn) + try: + await writer._async_get_or_create_session(session_id) + await writer._async_save_workbook_map( + map_id, + { + "sessionId": session_id, + "workbookFingerprint": "fp-v1", + "version": 1, + }, + ) + + # Reproduce the historical crash boundary: the flagship commit lands, + # then the process dies before the projection commit. + await writer._async_save_artifact( + map_id, + "workbook_map", + session_id, + "", + "", + { + "mapId": map_id, + "sessionId": session_id, + "workbookFingerprint": "fp-v2", + "version": 2, + }, + ) + stale = await writer._async_get_typed_json( + "workbook_maps", + "map_json", + "map_id", + map_id, + ) + assert stale is not None and stale["version"] == 1 + await writer.close() + + # Pool initialization runs an atomic, single-flight reconciliation + # before the restarted store serves requests. + await restarted._get_pool() + repaired = await restarted._async_get_typed_json( + "workbook_maps", + "map_json", + "map_id", + map_id, + ) + flagship = await restarted._async_get_artifact(map_id, "workbook_map") + assert repaired is not None and flagship is not None + assert repaired["version"] == flagship["version"] == 2 + assert repaired["workbookFingerprint"] == flagship["workbookFingerprint"] == "fp-v2" + + await restarted._async_save_workbook_map( + map_id, + { + "sessionId": session_id, + "workbookFingerprint": "fp-v3", + "version": 3, + }, + ) + projected_v3 = await restarted._async_get_typed_json( + "workbook_maps", + "map_json", + "map_id", + map_id, + ) + flagship_v3 = await restarted._async_get_artifact(map_id, "workbook_map") + assert projected_v3 is not None and flagship_v3 is not None + assert projected_v3["version"] == flagship_v3["version"] == 3 + finally: + await writer.close() + await restarted.close() diff --git a/backend/tests/test_postgres_protocol_parity.py b/backend/tests/test_postgres_protocol_parity.py new file mode 100644 index 0000000..1284cbe --- /dev/null +++ b/backend/tests/test_postgres_protocol_parity.py @@ -0,0 +1,1304 @@ +import asyncio +import inspect +import time +from contextlib import asynccontextmanager + +from backend.storage import postgres_protocol_gap_report, postgres_protocol_gaps +from backend.storage.enterprise_identity import prepare_enterprise_identity_binding +from backend.storage.postgres import ( + POSTGRES_TYPED_PROJECTION_BACKFILLS, + POSTGRES_TYPED_PROJECTION_MIGRATIONS, + POSTGRES_TYPED_PROJECTION_TABLES, + PostgresSessionStore, +) +from backend.storage.protocol import SessionStoreProtocol + + +class _FakeAcquire: + def __init__(self, conn): + self.conn = conn + + async def __aenter__(self): + return self.conn + + async def __aexit__(self, exc_type, exc, tb): + return False + + +class _FakePool: + def __init__(self, conn): + self.conn = conn + + def acquire(self): + return _FakeAcquire(self.conn) + + +class _RecordingConn: + def __init__(self, fetch_results=None): + self.executed = [] + self.fetch_results = list(fetch_results or []) + + async def execute(self, query, *params): + self.executed.append((query, params)) + return "INSERT 0 1" + + @asynccontextmanager + async def transaction(self): + yield + + async def fetch(self, query, *params): + self.executed.append((query, params)) + return self.fetch_results.pop(0) if self.fetch_results else [] + + async def fetchrow(self, query, *params): + self.executed.append((query, params)) + if self.fetch_results: + rows = self.fetch_results.pop(0) + return rows[0] if rows else None + if "JOIN session_lifecycle" in query: + return {"owner_user_id": "", "lifecycle_state": "active"} + return None + + async def fetchval(self, query, *params): + self.executed.append((query, params)) + if self.fetch_results: + rows = self.fetch_results.pop(0) + if not rows: + return None + row = rows[0] + if isinstance(row, dict): + return next(iter(row.values())) + return row + return False if "SELECT EXISTS" in query else None + + +def _executed_entry(connection: _RecordingConn, sql_fragment: str): + return next( + (query, params) + for query, params in connection.executed + if sql_fragment in query + ) + + +def test_postgres_parity_report_groups_protocol_methods_by_production_surface(): + report = postgres_protocol_gap_report(PostgresSessionStore) + gaps = postgres_protocol_gaps(PostgresSessionStore) + categories = {category["id"]: category for category in report["categories"]} + + assert report["status"] == "ready" + assert report["ready"] is True + assert report["missing_methods"] == gaps + assert report["missing_count"] == len(gaps) + assert "enterprise_dashboard" in report["required_methods"] + assert "list_benchmark_runs" in report["required_methods"] + assert "save_workbook_job" in report["required_methods"] + assert report["blocking_count"] == report["missing_count"] + report["signature_mismatch_count"] + assert report["missing_methods"] == [] + assert report["signature_mismatches"] == [] + assert report["blocking_methods"] == [] + assert "uncategorized" not in categories + + assert categories["operation_apply_audit"]["blocking_count"] == 0 + assert categories["documents_citations_benchmarks"]["blocking_count"] == 0 + assert categories["workbook_job_runs"]["blocking_count"] == 0 + assert categories["enterprise_admin"]["blocking_count"] == 0 + assert categories["workbook_accuracy_artifacts"]["blocking_count"] == 0 + assert report["blocking_categories"] == [] + + +def test_postgres_parity_report_has_no_signature_mismatches(): + report = postgres_protocol_gap_report(PostgresSessionStore) + issues = {issue["method"]: issue for issue in report["signature_mismatches"]} + apply_category = { + category["id"]: category + for category in report["categories"] + }["operation_apply_audit"] + + assert "record_apply_attempt" not in report["missing_methods"] + assert "record_apply_attempt" not in report["blocking_methods"] + assert "record_apply_attempt" not in issues + assert apply_category["signature_mismatches"] == [] + + +def test_postgres_parity_report_marks_complete_store_ready(): + def _stub(self, *args, **kwargs): + return None + + methods = { + name: _stub + for name, value in inspect.getmembers(SessionStoreProtocol) + if not name.startswith("_") and callable(value) + } + complete_store = type("CompleteStore", (), methods) + + report = postgres_protocol_gap_report(complete_store) + + assert report["status"] == "ready" + assert report["ready"] is True + assert report["missing_methods"] == [] + assert report["signature_mismatches"] == [] + assert report["blocking_methods"] == [] + assert all(category["blocking_count"] == 0 for category in report["categories"]) + + +def test_postgres_sync_wrapper_runs_inside_existing_event_loop(): + store = PostgresSessionStore("postgresql://example") + + async def fake_get_or_create_session(session_id: str) -> dict: + return {"session_id": session_id, "source": "background-loop"} + + store._async_get_or_create_session = fake_get_or_create_session + + async def invoke_from_running_loop(): + return store.get_or_create_session("loop-session") + + try: + result = asyncio.run(invoke_from_running_loop()) + finally: + store._stop_sync_loop() + + assert result == {"session_id": "loop-session", "source": "background-loop"} + + +def test_postgres_operation_plan_compare_and_swap_is_row_locked_and_revision_bound(): + plan_id = "plan-rollback-cas" + session_id = "session-rollback-cas" + expected = { + "id": plan_id, + "session_id": session_id, + "status": "rollback_pending", + "metadata": {"rollback_capability_used": False}, + } + updated = { + **expected, + "metadata": { + "rollback_capability_id": "rollback_cap_one", + "rollback_capability_used": False, + }, + } + conn = _RecordingConn( + fetch_results=[ + [{"owner_user_id": "", "lifecycle_state": "active"}], + [{"session_id": session_id, "payload": expected}], + [{"artifact_id": plan_id}], + ] + ) + store = PostgresSessionStore("postgresql://example") + store._pool = _FakePool(conn) + + swapped = asyncio.run( + store._async_compare_and_swap_operation_plan(plan_id, expected, updated) + ) + + assert swapped is True + sql = "\n".join(query for query, _ in conn.executed) + assert "pg_advisory_xact_lock" in sql + assert "artifact_type = 'operation_plan' FOR UPDATE" in sql + assert "payload = $5::jsonb RETURNING artifact_id" in sql + + +def test_postgres_operation_plan_compare_and_swap_rejects_stale_revision_without_update(): + plan_id = "plan-rollback-stale" + session_id = "session-rollback-stale" + expected = {"id": plan_id, "session_id": session_id, "status": "rollback_pending"} + durable = {**expected, "metadata": {"rollback_capability_id": "already-issued"}} + updated = {**expected, "status": "rolled_back"} + conn = _RecordingConn( + fetch_results=[ + [{"owner_user_id": "", "lifecycle_state": "active"}], + [{"session_id": session_id, "payload": durable}], + ] + ) + store = PostgresSessionStore("postgresql://example") + store._pool = _FakePool(conn) + + swapped = asyncio.run( + store._async_compare_and_swap_operation_plan(plan_id, expected, updated) + ) + + assert swapped is False + assert not any(query.lstrip().startswith("UPDATE flagship_artifacts") for query, _ in conn.executed) + + +def test_postgres_artifact_schema_uses_type_scoped_identity(): + conn = _RecordingConn() + store = PostgresSessionStore("postgresql://example") + store._pool = _FakePool(conn) + + asyncio.run(store._init_schema()) + + schema_sql = "\n".join(query for query, _params in conn.executed) + assert "CREATE TABLE IF NOT EXISTS postgres_schema_migrations" in schema_sql + assert "PRIMARY KEY (artifact_id, artifact_type)" in schema_sql + assert "CREATE TABLE IF NOT EXISTS v7_default_readiness_reports" in schema_sql + assert "idx_v7_default_readiness_status" in schema_sql + assert "CREATE TABLE IF NOT EXISTS local_replay_artifacts" in schema_sql + assert "idx_local_replay_artifacts_gate" in schema_sql + assert "CREATE TABLE IF NOT EXISTS local_replay_baselines" in schema_sql + assert "idx_local_replay_baselines_production" in schema_sql + assert "CREATE TABLE IF NOT EXISTS agent_run_artifacts" in schema_sql + assert "idx_agent_run_artifacts_run" in schema_sql + assert "CREATE TABLE IF NOT EXISTS verification_reports_v2" in schema_sql + assert "idx_verification_reports_v2_run" in schema_sql + assert "CREATE TABLE IF NOT EXISTS workbook_benchmark_fixture_results" in schema_sql + assert "idx_workbook_fixture_results_fixture" in schema_sql + assert "CREATE TABLE IF NOT EXISTS operation_audits" in schema_sql + assert "idx_operation_audits_plan" in schema_sql + assert "CREATE TABLE IF NOT EXISTS workbook_maps" in schema_sql + assert "idx_workbook_maps_fingerprint" in schema_sql + assert "CREATE TABLE IF NOT EXISTS source_evidence_graphs" in schema_sql + assert "CREATE TABLE IF NOT EXISTS operation_simulations" in schema_sql + assert "CREATE TABLE IF NOT EXISTS workbook_invariant_runs" in schema_sql + assert "CREATE TABLE IF NOT EXISTS agent_runs" in schema_sql + assert "v7_gate_status TEXT" in schema_sql + assert "CREATE TABLE IF NOT EXISTS workbook_truth_models" in schema_sql + assert "semantic_target_decision_count" in schema_sql + assert "CREATE TABLE IF NOT EXISTS formula_graph_artifacts" in schema_sql + assert "parser_confidence DOUBLE PRECISION" in schema_sql + assert "CREATE TABLE IF NOT EXISTS operation_simulations_v2" in schema_sql + assert "CREATE TABLE IF NOT EXISTS v7_workbook_reviews" in schema_sql + assert "default_readiness_status TEXT" in schema_sql + assert "DROP CONSTRAINT" in schema_sql + assert "postgres_artifact_identity_v2" in schema_sql + assert "idx_flagship_enterprise_oidc_identity" in schema_sql + assert "enforce_enterprise_oidc_identity" in schema_sql + assert "postgres_enterprise_oidc_identity_v1" in schema_sql + assert "postgres_v7_readiness_projection_v1" in schema_sql + assert "postgres_local_replay_projection_v1" in schema_sql + assert "postgres_run_evidence_projection_v1" in schema_sql + assert "postgres_workbook_intelligence_projection_v1" in schema_sql + assert schema_sql.index("INSERT INTO v7_default_readiness_reports") < schema_sql.index( + "postgres_v7_readiness_projection_v1" + ) + for projection_table in POSTGRES_TYPED_PROJECTION_TABLES: + assert f"INSERT INTO {projection_table}" in schema_sql + + +def test_postgres_oidc_identity_lookup_uses_issuer_subject_and_active_status(): + user = { + "user_id": "analyst-1", + "email": "analyst@example.com", + "status": "active", + "oidc_issuer": "https://identity.example.com/tenant/v2.0", + "oidc_subject": "subject-1", + } + conn = _RecordingConn(fetch_results=[[{"payload": user}]]) + store = PostgresSessionStore("postgresql://example") + store._pool = _FakePool(conn) + + resolved = asyncio.run( + store._async_get_enterprise_user_by_oidc_identity( + "https://identity.example.com/tenant/v2.0", + "subject-1", + ) + ) + + assert resolved == user + query, params = conn.executed[0] + assert "payload->>'oidc_issuer' = $1" in query + assert "payload->>'oidc_subject' = $2" in query + assert "LOWER(COALESCE(payload->>'status', '')) = 'active'" in query + assert params == ("https://identity.example.com/tenant/v2.0", "subject-1") + + +def test_postgres_artifact_upsert_is_type_scoped(): + conn = _RecordingConn() + store = PostgresSessionStore("postgresql://example") + store._pool = _FakePool(conn) + + asyncio.run( + store._async_save_artifact( + "shared-id", + "workbook_truth_model", + "session-1", + "run-1", + "plan-1", + {"truthModelId": "shared-id"}, + ) + ) + + upsert_sql, _params = _executed_entry(conn, "INSERT INTO flagship_artifacts") + assert "ON CONFLICT(artifact_id, artifact_type)" in upsert_sql + assert "artifact_type = EXCLUDED.artifact_type" not in upsert_sql + + +def test_postgres_migration_status_reports_artifact_identity_revision(): + store = PostgresSessionStore("postgresql://example") + + status = store.get_migration_status() + + assert status["ledger"] == "postgres_schema_migrations" + assert status["identity"] == ["artifact_id", "artifact_type"] + assert "postgres_artifact_identity_v2" in status["applied"] + assert "postgres_v7_readiness_projection_v1" in status["applied"] + assert "postgres_local_replay_projection_v1" in status["applied"] + assert "postgres_run_evidence_projection_v1" in status["applied"] + assert "postgres_workbook_intelligence_projection_v1" in status["applied"] + assert status["typed_projections"] == POSTGRES_TYPED_PROJECTION_TABLES + assert status["projection_migrations"] == POSTGRES_TYPED_PROJECTION_MIGRATIONS + assert set(status["typed_projection_backfills"]) == set(POSTGRES_TYPED_PROJECTION_TABLES) + assert all(count > 0 for count in status["typed_projection_backfills"].values()) + + +def test_postgres_typed_projection_backfills_cover_status_contract(): + status = PostgresSessionStore("postgresql://example").get_migration_status() + migration_tables = { + table + for tables in status["projection_migrations"].values() + for table in tables + } + + assert migration_tables == set(status["typed_projections"]) + assert migration_tables == set(POSTGRES_TYPED_PROJECTION_BACKFILLS) + for table, statements in POSTGRES_TYPED_PROJECTION_BACKFILLS.items(): + joined_sql = "\n".join(statements) + assert f"INSERT INTO {table}" in joined_sql + assert "FROM flagship_artifacts" in joined_sql + assert "ON CONFLICT" in joined_sql + + +def test_postgres_v7_readiness_reports_dual_write_typed_projection(): + conn = _RecordingConn() + store = PostgresSessionStore("postgresql://example") + store._pool = _FakePool(conn) + + readiness_id = asyncio.run( + store._async_save_v7_default_readiness_report( + {"readinessId": "v7ready-test", "status": "ready", "defaultAllowed": True, "score": 0.97} + ) + ) + + executed_sql = "\n".join(query for query, _params in conn.executed) + assert readiness_id == "v7ready-test" + assert "INSERT INTO flagship_artifacts" in executed_sql + assert "v7_default_readiness_report" in str(conn.executed[0][1]) + assert "INSERT INTO v7_default_readiness_reports" in executed_sql + typed_params = conn.executed[-1][1] + assert typed_params[0] == "v7ready-test" + assert typed_params[1] == "ready" + assert typed_params[2] is True + assert typed_params[3] == 0.97 + + +def test_postgres_v7_readiness_reports_read_from_typed_projection(): + conn = _RecordingConn(fetch_results=[[{"report_json": {"readinessId": "v7ready-new", "status": "warning"}}]]) + store = PostgresSessionStore("postgresql://example") + store._pool = _FakePool(conn) + + reports = asyncio.run(store._async_list_v7_default_readiness_reports(5)) + + assert reports == [{"readinessId": "v7ready-new", "status": "warning"}] + assert "FROM v7_default_readiness_reports" in conn.executed[0][0] + assert conn.executed[0][1] == (5,) + + +def test_postgres_v7_readiness_public_readers_fall_back_to_artifacts(): + store = PostgresSessionStore("postgresql://example") + + async def list_typed(_limit=20): + return [] + + async def list_artifacts_by_prefix(_prefix, _limit=100): + return [{"payload": {"readinessId": "v7ready-legacy", "status": "ready"}}] + + store._async_list_v7_default_readiness_reports = list_typed + store._async_list_artifacts_by_prefix = list_artifacts_by_prefix + + try: + assert store.get_latest_v7_default_readiness_report()["readinessId"] == "v7ready-legacy" + assert store.list_v7_default_readiness_reports()[0]["status"] == "ready" + finally: + store._stop_sync_loop() + + +def test_postgres_local_replay_artifacts_dual_write_typed_projection(): + conn = _RecordingConn() + store = PostgresSessionStore("postgresql://example") + store._pool = _FakePool(conn) + + replay_id = asyncio.run( + store._async_save_local_replay_artifact( + "replay-test", + "session-1", + "v7_plan", + { + "replayId": "replay-test", + "schemaVersion": "2026-05-local-replay-v1", + "v7GateStatus": "checkpoint_required", + "accuracyScore": 0.91, + "confidenceGapCount": 2, + "checkpointReasonCount": 1, + "formulaFailureCount": 0, + "truthSchema": "truth-v5", + "simulationSchema": "simulation-v6", + }, + ) + ) + + executed_sql = "\n".join(query for query, _params in conn.executed) + assert replay_id == "replay-test" + assert "INSERT INTO flagship_artifacts" in executed_sql + assert "local_replay:v7_plan" in str(_executed_entry(conn, "INSERT INTO flagship_artifacts")[1]) + assert "INSERT INTO local_replay_artifacts" in executed_sql + typed_params = conn.executed[-1][1] + assert typed_params[0] == "replay-test" + assert typed_params[1] == "session-1" + assert typed_params[2] == "v7_plan" + assert typed_params[3] == "2026-05-local-replay-v1" + assert typed_params[4] == "checkpoint_required" + assert typed_params[5] == 0.91 + assert typed_params[6] == 2 + assert typed_params[7] == 1 + assert typed_params[8] == 0 + assert typed_params[9] == "truth-v5" + assert typed_params[10] == "simulation-v6" + + +def test_postgres_local_replay_artifacts_read_from_typed_projection(): + conn = _RecordingConn( + fetch_results=[ + [ + { + "replay_id": "replay-new", + "session_id": "session-1", + "artifact_kind": "v7_workbook_review", + "replay_json": {"replayId": "replay-new", "artifactType": "v7_workbook_review"}, + "created_at": 123.0, + } + ] + ] + ) + store = PostgresSessionStore("postgresql://example") + store._pool = _FakePool(conn) + + replays = asyncio.run(store._async_list_local_replay_artifacts(5)) + + assert replays == [ + { + "artifact_id": "replay-new", + "artifact_type": "local_replay:v7_workbook_review", + "session_id": "session-1", + "payload": {"replayId": "replay-new", "artifactType": "v7_workbook_review"}, + "created_at": 123.0, + } + ] + assert "FROM local_replay_artifacts" in conn.executed[0][0] + assert conn.executed[0][1] == (5,) + + +def test_postgres_local_replay_artifacts_public_reader_falls_back_to_artifacts(): + store = PostgresSessionStore("postgresql://example") + + async def list_typed(_limit=100): + return [] + + async def list_artifacts_by_prefix(_prefix, _limit=100): + return [{"payload": {"replayId": "replay-legacy", "artifactType": "v7_plan"}}] + + store._async_list_local_replay_artifacts = list_typed + store._async_list_artifacts_by_prefix = list_artifacts_by_prefix + + try: + assert store.list_local_replay_artifacts()[0]["payload"]["replayId"] == "replay-legacy" + finally: + store._stop_sync_loop() + + +def test_postgres_local_replay_baselines_dual_write_typed_projection(): + conn = _RecordingConn() + store = PostgresSessionStore("postgresql://example") + store._pool = _FakePool(conn) + + baseline_id = asyncio.run( + store._async_save_local_replay_baseline( + { + "baselineId": "baseline-test", + "sessionId": "local-replay-baseline", + "status": "passed", + "productionEvidence": True, + "evidenceClass": "anonymized_production", + "replayCount": 12, + "baselineComparison": {"deterministicDriftClass": "none"}, + } + ) + ) + + executed_sql = "\n".join(query for query, _params in conn.executed) + assert baseline_id == "baseline-test" + assert "INSERT INTO flagship_artifacts" in executed_sql + assert "local_replay_baseline" in str(_executed_entry(conn, "INSERT INTO flagship_artifacts")[1]) + assert "INSERT INTO local_replay_baselines" in executed_sql + typed_params = conn.executed[-1][1] + assert typed_params[0] == "baseline-test" + assert typed_params[1] == "local-replay-baseline" + assert typed_params[2] == "passed" + assert typed_params[3] is True + assert typed_params[4] == "anonymized_production" + assert typed_params[5] == 12 + assert typed_params[6] == "none" + + +def test_postgres_local_replay_baselines_read_from_typed_projection(): + conn = _RecordingConn( + fetch_results=[ + [ + { + "session_id": "session-replay", + "baseline_json": { + "baselineId": "baseline-new", + "status": "passed", + "productionEvidence": True, + } + } + ] + ] + ) + store = PostgresSessionStore("postgresql://example") + store._pool = _FakePool(conn) + + async def session_is_visible(_session_id: str) -> bool: + return True + + store._async_session_is_visible = session_is_visible + + baselines = asyncio.run(store._async_list_local_replay_baselines(5)) + + assert baselines == [{"baselineId": "baseline-new", "status": "passed", "productionEvidence": True}] + assert "FROM local_replay_baselines" in conn.executed[0][0] + assert conn.executed[0][1] == (5,) + + +def test_postgres_local_replay_baselines_public_reader_falls_back_to_artifacts(): + store = PostgresSessionStore("postgresql://example") + + async def list_typed(_limit=20): + return [] + + async def list_artifacts_by_prefix(_prefix, _limit=20): + return [{"payload": {"baselineId": "baseline-legacy", "status": "passed"}}] + + store._async_list_local_replay_baselines = list_typed + store._async_list_artifacts_by_prefix = list_artifacts_by_prefix + + try: + assert store.list_local_replay_baselines()[0]["baselineId"] == "baseline-legacy" + finally: + store._stop_sync_loop() + + +def test_postgres_agent_run_artifacts_dual_write_typed_projection(): + conn = _RecordingConn() + store = PostgresSessionStore("postgresql://example") + store._pool = _FakePool(conn) + + artifact_id = asyncio.run( + store._async_save_agent_run_artifact( + "artifact-test", + "run-1", + "session-1", + "step:simulate", + "plan-1", + {"artifactType": "simulate", "status": "passed"}, + generic_artifact_type="agent_run_step_artifact_v3", + ) + ) + + executed_sql = "\n".join(query for query, _params in conn.executed) + assert artifact_id == "artifact-test" + assert "INSERT INTO flagship_artifacts" in executed_sql + assert "agent_run_step_artifact_v3" in str(_executed_entry(conn, "INSERT INTO flagship_artifacts")[1]) + assert "INSERT INTO agent_run_artifacts" in executed_sql + typed_params = conn.executed[-1][1] + assert typed_params[0] == "artifact-test" + assert typed_params[1] == "run-1" + assert typed_params[2] == "session-1" + assert typed_params[3] == "step:simulate" + assert typed_params[4] == "plan-1" + + +def test_postgres_agent_run_step_artifact_public_writer_uses_step_type_projection(): + store = PostgresSessionStore("postgresql://example") + captured = {} + + async def save_agent_run_artifact( + artifact_id, + run_id, + session_id, + artifact_type, + plan_id, + artifact, + *, + generic_artifact_type, + ): + captured.update( + { + "artifact_id": artifact_id, + "run_id": run_id, + "session_id": session_id, + "artifact_type": artifact_type, + "plan_id": plan_id, + "artifact": artifact, + "generic_artifact_type": generic_artifact_type, + } + ) + return artifact_id + + store._async_save_agent_run_artifact = save_agent_run_artifact + + try: + artifact_id = store.save_agent_run_step_artifact_v3( + "run-1", + "session-1", + "step-1", + {"artifactId": "stepartifact-test", "artifactType": "formula_trace", "planId": "plan-1"}, + ) + finally: + store._stop_sync_loop() + + assert artifact_id == "stepartifact-test" + assert captured["artifact_type"] == "step:formula_trace" + assert captured["generic_artifact_type"] == "agent_run_step_artifact_v3" + assert captured["artifact"]["runId"] == "run-1" + assert captured["artifact"]["stepId"] == "step-1" + + +def test_postgres_agent_run_artifacts_read_from_typed_projection(): + conn = _RecordingConn( + fetch_results=[ + [ + { + "artifact_id": "artifact-new", + "artifact_type": "step:simulate", + "artifact_json": {"artifactId": "artifact-new", "artifactType": "simulate"}, + "created_at": 123.0, + } + ] + ] + ) + store = PostgresSessionStore("postgresql://example") + store._pool = _FakePool(conn) + + artifacts = asyncio.run(store._async_list_agent_run_artifacts("run-1", 5)) + + assert artifacts == [ + { + "artifact_id": "artifact-new", + "artifact_type": "step:simulate", + "artifact": {"artifactId": "artifact-new", "artifactType": "simulate"}, + "created_at": 123.0, + } + ] + assert "FROM agent_run_artifacts" in conn.executed[0][0] + assert conn.executed[0][1] == ("run-1", 5) + + +def test_postgres_agent_run_artifacts_legacy_reader_normalizes_step_prefix(): + conn = _RecordingConn( + fetch_results=[ + [ + { + "artifact_id": "artifact-legacy", + "artifact_type": "agent_run:step:simulate", + "session_id": "session-run", + "payload": {"artifactId": "artifact-legacy", "artifactType": "simulate"}, + "created_at": 123.0, + }, + { + "artifact_id": "artifact-legacy-v3", + "artifact_type": "agent_run_step_artifact_v3", + "session_id": "session-run", + "payload": {"artifactId": "artifact-legacy-v3", "artifactType": "audit"}, + "created_at": 124.0, + }, + ] + ] + ) + store = PostgresSessionStore("postgresql://example") + store._pool = _FakePool(conn) + + async def session_is_visible(_session_id: str) -> bool: + return True + + store._async_session_is_visible = session_is_visible + + artifacts = asyncio.run(store._async_list_run_artifacts("run-1", 5)) + + assert [item["artifact_type"] for item in artifacts] == ["step:simulate", "step:audit"] + assert "FROM flagship_artifacts" in conn.executed[0][0] + + +def test_postgres_verification_report_v2_dual_writes_typed_projection(): + conn = _RecordingConn() + store = PostgresSessionStore("postgresql://example") + store._pool = _FakePool(conn) + + verification_id = asyncio.run( + store._async_save_verification_report_v2( + "session-1", + "verify-test", + {"runId": "run-1", "planId": "plan-1", "status": "verified"}, + ) + ) + + executed_sql = "\n".join(query for query, _params in conn.executed) + assert verification_id == "verify-test" + assert "INSERT INTO flagship_artifacts" in executed_sql + assert "verification_report" in str(_executed_entry(conn, "INSERT INTO flagship_artifacts")[1]) + assert "INSERT INTO verification_reports_v2" in executed_sql + typed_params = conn.executed[-1][1] + assert typed_params[0] == "verify-test" + assert typed_params[1] == "session-1" + assert typed_params[2] == "run-1" + assert typed_params[3] == "plan-1" + assert typed_params[4] == "verified" + + +def test_postgres_workbook_benchmark_fixture_result_dual_writes_typed_projection(): + conn = _RecordingConn() + store = PostgresSessionStore("postgresql://example") + store._pool = _FakePool(conn) + + fixture_result_id = asyncio.run( + store._async_save_workbook_benchmark_fixture_result( + "bench-1", + "fixture-result-test", + {"fixtureId": "dcf", "status": "passed", "score": 1.0}, + ) + ) + + executed_sql = "\n".join(query for query, _params in conn.executed) + assert fixture_result_id == "fixture-result-test" + assert "INSERT INTO flagship_artifacts" in executed_sql + assert "workbook_benchmark_fixture" in str(conn.executed[0][1]) + assert "INSERT INTO workbook_benchmark_fixture_results" in executed_sql + typed_params = conn.executed[-1][1] + assert typed_params[0] == "fixture-result-test" + assert typed_params[1] == "bench-1" + assert typed_params[2] == "dcf" + assert typed_params[3] == "passed" + + +def test_postgres_operation_audits_dual_write_typed_projection(): + conn = _RecordingConn() + store = PostgresSessionStore("postgresql://example") + store._pool = _FakePool(conn) + + count = asyncio.run( + store._async_save_operation_audits( + "session-1", + "plan-1", + [ + { + "auditId": "opaudit-test", + "operationId": "op-1", + "type": "write_cell", + "dirtyRanges": ["Sheet1!A1"], + "warnings": ["manual checkpoint"], + } + ], + ) + ) + + executed_sql = "\n".join(query for query, _params in conn.executed) + assert count == 1 + assert "INSERT INTO flagship_artifacts" in executed_sql + assert "operation_audit" in str(_executed_entry(conn, "INSERT INTO flagship_artifacts")[1]) + assert "INSERT INTO operation_audits" in executed_sql + typed_params = conn.executed[-1][1] + assert typed_params[0] == "opaudit-test" + assert typed_params[1] == "session-1" + assert typed_params[2] == "plan-1" + assert typed_params[3] == "op-1" + assert typed_params[4] == "write_cell" + assert typed_params[5] == 1 + assert typed_params[6] == 1 + + +def test_postgres_audit_csv_exports_actor_columns(): + store = PostgresSessionStore("postgresql://example") + + async def list_artifacts_for_session(_session_id, limit=1000): + return [ + { + "artifact_id": "audit-1", + "artifact_type": "audit:plan_applied", + "payload": { + "audit_id": "audit-1", + "event_type": "plan_applied", + "plan_id": "", + "snapshot_id": "", + "created_at": 123.0, + "payload": { + "actor": { + "user_id": "admin", + "email": "admin@localhost", + "role": "admin", + "auth_source": "static_api_key", + } + }, + }, + } + ][:limit] + + store._async_list_artifacts_for_session = list_artifacts_for_session + + try: + csv_export = store.export_audit_csv("enterprise-admin") + finally: + store._stop_sync_loop() + + assert "actor_user_id" in csv_export + assert "actor_auth_source" in csv_export + assert "admin@localhost" in csv_export + assert "static_api_key" in csv_export + + +def test_postgres_workbook_intelligence_artifacts_dual_write_typed_projections(): + cases = [ + ( + "_async_save_workbook_map", + ("map-test", { + "sessionId": "session-1", + "workbookFingerprint": "fp-1", + "workbookName": "Model.xlsx", + "schemaVersion": "map-v2", + "entities": [{}, {}], + "formulaFacts": [{}], + "confidenceGaps": ["gap"], + }), + "workbook_map", + "INSERT INTO workbook_maps", + ("map-test", "session-1", "fp-1", "Model.xlsx", "map-v2", 2, 1, 1), + ), + ( + "_async_save_source_evidence_graph", + ("graph-test", {"sessionId": "session-1", "documentIds": ["doc-1"]}), + "source_evidence_graph", + "INSERT INTO source_evidence_graphs", + ("graph-test", "session-1"), + ), + ( + "_async_save_operation_simulation", + ("sim-test", "session-1", {"status": "passed"},), + "operation_simulation", + "INSERT INTO operation_simulations", + ("sim-test", "session-1", "plan-1", "run-1", "passed"), + {"plan_id": "plan-1", "run_id": "run-1"}, + ), + ( + "_async_save_workbook_invariant_run", + ("inv-test", "session-1", {"status": "warning"},), + "workbook_invariant", + "INSERT INTO workbook_invariant_runs", + ("inv-test", "session-1", "run-1", "plan-1", "warning"), + {"run_id": "run-1", "plan_id": "plan-1"}, + ), + ( + "_async_save_agent_run", + ("run-test", { + "sessionId": "session-1", + "status": "completed", + "schemaVersion": "agent-v4", + "planId": "plan-1", + "semanticMapId": "map-test", + "simulationId": "sim-test", + "verificationId": "verify-test", + "v7QualityGate": {"status": "passed"}, + "executionGroups": [{}, {}], + "checkpointReasons": ["inspect"], + "prompt": "build model", + }), + "agent_run", + "INSERT INTO agent_runs", + ("run-test", "session-1", "completed", "agent-v4", "plan-1", "map-test", "sim-test", "verify-test", "passed", 2, 1, "build model"), + ), + ( + "_async_save_workbook_truth_model", + ("truth-test", { + "sessionId": "session-1", + "workbookFingerprint": "fp-1", + "schemaVersion": "truth-v5", + "semanticMapId": "map-test", + "formulaGraphId": "graph-test", + "accuracyScore": 0.96, + "modelTaxonomy": "dcf", + "confidenceGaps": ["gap"], + "semanticTargetDecisions": [{}, {}], + }), + "workbook_truth_model", + "INSERT INTO workbook_truth_models", + ("truth-test", "session-1", "fp-1", "truth-v5", "map-test", "graph-test", 0.96, "dcf", 1, 2), + ), + ( + "_async_save_formula_graph_artifact", + ("fgraph-test", { + "sessionId": "session-1", + "workbookFingerprint": "fp-1", + "schemaVersion": "formula-v5", + "engine": "hyperformula", + "formulas": [{}, {}], + "materialFormulaFailures": [{}], + "preApplySimulationReady": True, + "parserConfidence": 0.88, + "quality": {"status": "passed"}, + }), + "formula_graph", + "INSERT INTO formula_graph_artifacts", + ("fgraph-test", "session-1", "fp-1", "formula-v5", "hyperformula", 2, 1, True, 0.88), + ), + ( + "_async_save_operation_simulation_v2", + ("simv2-test", "session-1", {"truthModelId": "truth-test", "simulationGate": "checkpoint_required"},), + "operation_simulation_v2", + "INSERT INTO operation_simulations_v2", + ("simv2-test", "session-1", "truth-test", "plan-1", "run-1", "checkpoint_required"), + {"plan_id": "plan-1", "run_id": "run-1"}, + ), + ( + "_async_save_v7_workbook_review", + ("review-test", { + "sessionId": "session-1", + "planId": "plan-1", + "runId": "run-1", + "workbookFingerprint": "fp-1", + "accuracyScore": 0.94, + "confidenceGaps": ["gap"], + "checkpointReasons": ["inspect"], + "defaultReadiness": {"status": "ready"}, + }), + "v7_workbook_review", + "INSERT INTO v7_workbook_reviews", + ("review-test", "session-1", "plan-1", "run-1", "fp-1", 0.94, 1, 1, "ready"), + ), + ] + + for case in cases: + method_name, args, generic_type, projection_sql, expected_prefix, *rest = case + kwargs = rest[0] if rest else {} + conn = _RecordingConn() + store = PostgresSessionStore("postgresql://example") + store._pool = _FakePool(conn) + + artifact_id = asyncio.run(getattr(store, method_name)(*args, **kwargs)) + + executed_sql = "\n".join(query for query, _params in conn.executed) + assert artifact_id == expected_prefix[0] + assert "INSERT INTO flagship_artifacts" in executed_sql + assert generic_type in str(_executed_entry(conn, "INSERT INTO flagship_artifacts")[1]) + assert projection_sql in executed_sql + assert conn.executed[-1][1][: len(expected_prefix)] == expected_prefix + + +def test_postgres_workbook_intelligence_getters_prefer_typed_projection(): + cases = [ + ("get_workbook_map", "workbook_maps", "map_json", {"mapId": "map-new"}, "map-new"), + ("get_agent_run", "agent_runs", "run_json", {"runId": "run-new"}, "run-new"), + ("get_workbook_truth_model", "workbook_truth_models", "truth_json", {"truthModelId": "truth-new"}, "truth-new"), + ("get_formula_graph_artifact", "formula_graph_artifacts", "graph_json", {"graphId": "graph-new"}, "graph-new"), + ("get_v7_workbook_review", "v7_workbook_reviews", "review_json", {"reviewId": "review-new"}, "review-new"), + ] + + for public_method, table_name, json_column, payload, artifact_id in cases: + store = PostgresSessionStore("postgresql://example") + + async def get_typed( + table, + column, + _id_column, + _artifact_id, + expected_table=table_name, + expected_column=json_column, + expected_artifact_id=artifact_id, + expected_payload=payload, + ): + assert table == expected_table + assert column == expected_column + assert _artifact_id == expected_artifact_id + return expected_payload + + async def get_artifact(_artifact_id, _artifact_type): + raise AssertionError("legacy fallback should not run when typed projection has a row") + + store._async_get_typed_json = get_typed + store._async_get_artifact = get_artifact + try: + assert getattr(store, public_method)(artifact_id) == payload + finally: + store._stop_sync_loop() + + +def test_postgres_workbook_intelligence_getters_fall_back_to_legacy_artifacts(): + cases = [ + ("get_workbook_map", "workbook_map", {"mapId": "map-legacy"}, "map-legacy"), + ("get_agent_run", "agent_run", {"runId": "run-legacy"}, "run-legacy"), + ("get_workbook_truth_model", "workbook_truth_model", {"truthModelId": "truth-legacy"}, "truth-legacy"), + ("get_formula_graph_artifact", "formula_graph", {"graphId": "graph-legacy"}, "graph-legacy"), + ("get_v7_workbook_review", "v7_workbook_review", {"reviewId": "review-legacy"}, "review-legacy"), + ] + + for public_method, artifact_type, payload, artifact_id in cases: + store = PostgresSessionStore("postgresql://example") + + async def get_typed(_table, _column, _id_column, _artifact_id): + return None + + async def get_artifact( + _artifact_id, + _artifact_type, + expected_artifact_id=artifact_id, + expected_artifact_type=artifact_type, + expected_payload=payload, + ): + assert _artifact_id == expected_artifact_id + assert _artifact_type == expected_artifact_type + return expected_payload + + store._async_get_typed_json = get_typed + store._async_get_artifact = get_artifact + try: + assert getattr(store, public_method)(artifact_id) == payload + finally: + store._stop_sync_loop() + + +def test_postgres_delete_session_cleans_typed_workbook_intelligence_tables(): + conn = _RecordingConn( + fetch_results=[ + [{"state": "purging"}], + [{"session_id": "session-1"}], + ] + ) + store = PostgresSessionStore("postgresql://example") + store._pool = _FakePool(conn) + + asyncio.run(store._async_delete_session("session-1")) + + executed_sql = "\n".join(query for query, _params in conn.executed) + for table in ( + "local_replay_baselines", + "workbook_maps", + "source_evidence_graphs", + "operation_simulations", + "workbook_invariant_runs", + "agent_runs", + "workbook_truth_models", + "formula_graph_artifacts", + "operation_simulations_v2", + "v7_workbook_reviews", + ): + assert f"DELETE FROM {table}" in executed_sql + + +def test_postgres_lists_only_documents_for_the_requested_session(): + conn = _RecordingConn( + fetch_results=[ + [{"owner_user_id": "", "lifecycle_state": "active"}], + [ + { + "payload": { + "document_id": "doc-1", + "session_id": "session-1", + "filename": "model.xlsx", + } + } + ] + ] + ) + store = PostgresSessionStore("postgresql://example") + store._pool = _FakePool(conn) + + documents = asyncio.run(store._async_list_documents("session-1", 20)) + + assert documents == [ + { + "document_id": "doc-1", + "session_id": "session-1", + "filename": "model.xlsx", + } + ] + query, params = _executed_entry(conn, "artifact_type = 'document'") + assert "artifact_type = 'document'" in query + assert params == ("session-1", 20) + + +def test_postgres_enterprise_and_benchmark_routes_have_artifact_backing(): + store = PostgresSessionStore("postgresql://example") + artifacts = {} + + async def save_artifact(artifact_id, artifact_type, session_id, run_id, plan_id, payload): + artifacts[(artifact_id, artifact_type)] = { + "artifact_id": artifact_id, + "artifact_type": artifact_type, + "session_id": session_id, + "run_id": run_id, + "plan_id": plan_id, + "payload": payload, + } + return artifact_id + + async def get_artifact(artifact_id, artifact_type): + row = artifacts.get((artifact_id, artifact_type)) + return row["payload"] if row else None + + async def list_artifacts_by_prefix(artifact_type_prefix, limit=100): + rows = [ + row + for (_artifact_id, artifact_type), row in artifacts.items() + if artifact_type.startswith(artifact_type_prefix) + ] + return rows[:limit] + + async def count_artifacts_by_prefix(artifact_type_prefix): + return sum( + 1 + for (_artifact_id, artifact_type) in artifacts + if artifact_type.startswith(artifact_type_prefix) + ) + + async def purge_sessions_before(_cutoff_timestamp): + return 0 + + async def upsert_enterprise_user(**kwargs): + user_id = kwargs["user_id"] + artifact_id = store._enterprise_user_artifact_id(user_id) + row = artifacts.get((artifact_id, "enterprise_user")) + existing = row["payload"] if row else {} + binding = prepare_enterprise_identity_binding( + status=kwargs.get("status") or existing.get("status") or "active", + metadata=kwargs.get("metadata"), + oidc_issuer=kwargs.get("oidc_issuer"), + oidc_subject=kwargs.get("oidc_subject"), + existing=existing, + ) + now = time.time() + requested_mfa = kwargs.get("mfa_enabled") + payload = { + **existing, + "user_id": user_id, + "email": kwargs["email"], + "name": kwargs["name"], + "role": kwargs["role"], + "status": binding.status, + "mfa_enabled": ( + bool(existing.get("mfa_enabled")) + if requested_mfa is None + else bool(requested_mfa) + ), + "metadata": binding.metadata, + "oidc_issuer": binding.issuer, + "oidc_subject": binding.subject, + "created_at": existing.get("created_at", now), + "updated_at": now, + } + await save_artifact(artifact_id, "enterprise_user", "", "", "", payload) + return payload + + store._async_save_artifact = save_artifact + store._async_get_artifact = get_artifact + store._async_list_artifacts_by_prefix = list_artifacts_by_prefix + store._async_count_artifacts_by_prefix = count_artifacts_by_prefix + store._async_purge_sessions_before = purge_sessions_before + store._async_upsert_enterprise_user = upsert_enterprise_user + + try: + user = store.upsert_enterprise_user( + user_id="analyst-1", + email="analyst1@example.com", + name="Analyst One", + role="analyst", + mfa_enabled=True, + oidc_issuer="https://identity.example.com/tenant/v2.0", + oidc_subject="subject-analyst-1", + ) + key = store.create_enterprise_api_key("analyst-1", "test") + team = store.create_enterprise_team("Banking") + + assert user["mfa_enabled"] is True + migration_markers = { + "legacy_oidc_issuer": "https://legacy.example.com", + "legacy_oidc_subject": "legacy-subject", + "oidc_binding_state": "partial", + } + store.upsert_enterprise_user( + user_id="legacy-review", + email="legacy-review@example.com", + name="Legacy Review", + role="analyst", + status="binding_required", + metadata=migration_markers, + ) + assert store.upsert_enterprise_user( + user_id="legacy-review", + email="legacy-review@example.com", + name="Legacy Review", + role="analyst", + metadata=None, + )["metadata"] == migration_markers + assert store.upsert_enterprise_user( + user_id="legacy-review", + email="legacy-review@example.com", + name="Legacy Review", + role="analyst", + metadata={}, + )["metadata"] == migration_markers + assert key["api_key"].startswith("fai_") + assert store.get_enterprise_user_by_api_key(key["api_key"])["user_id"] == "analyst-1" + assert store.revoke_enterprise_api_key(key["key_id"]) is True + assert store.get_enterprise_user_by_api_key(key["api_key"]) is None + key = store.create_enterprise_api_key("analyst-1", "status-test") + assert store.get_enterprise_user_by_api_key(key["api_key"])["user_id"] == "analyst-1" + store.set_enterprise_config("sso", {"require_mfa": True}) + store.upsert_enterprise_user( + user_id="analyst-1", + email="analyst1@example.com", + name="Analyst One", + role="analyst", + mfa_enabled=False, + ) + assert store.get_enterprise_user_by_api_key(key["api_key"]) is None + store.upsert_enterprise_user( + user_id="analyst-1", + email="analyst1@example.com", + name="Analyst One", + role="analyst", + mfa_enabled=True, + ) + assert store.get_enterprise_user_by_api_key(key["api_key"])["user_id"] == "analyst-1" + store.set_enterprise_config("sso", {"require_mfa": False}) + store.set_enterprise_config("provider_policy", {"requireMfa": True}) + store.upsert_enterprise_user( + user_id="analyst-1", + email="analyst1@example.com", + name="Analyst One", + role="analyst", + mfa_enabled=False, + ) + assert store.get_enterprise_user_by_api_key(key["api_key"]) is None + store.upsert_enterprise_user( + user_id="analyst-1", + email="analyst1@example.com", + name="Analyst One", + role="analyst", + mfa_enabled=True, + ) + assert store.get_enterprise_user_by_api_key(key["api_key"])["user_id"] == "analyst-1" + store.upsert_enterprise_user( + user_id="analyst-1", + email="analyst1@example.com", + name="Analyst One", + role="analyst", + status="suspended", + mfa_enabled=True, + ) + assert store.get_enterprise_user_by_api_key(key["api_key"]) is None + assert store.add_enterprise_team_member(team["team_id"], "analyst-1", "member") is True + assert store.get_enterprise_team(team["team_id"])["members"][0]["email"] == "analyst1@example.com" + listed_keys = store.list_enterprise_api_keys("analyst-1") + assert len(listed_keys) == 2 + assert {item["active"] for item in listed_keys} == {False, True} + assert store.enterprise_dashboard()["users"] == 2 + + store.save_benchmark_run({"run_id": "bench-1", "status": "completed"}) + assert store.list_benchmark_runs()[0]["run_id"] == "bench-1" + finally: + store._stop_sync_loop() diff --git a/backend/tests/test_postgres_release_rehearsal.py b/backend/tests/test_postgres_release_rehearsal.py new file mode 100644 index 0000000..b6fdec9 --- /dev/null +++ b/backend/tests/test_postgres_release_rehearsal.py @@ -0,0 +1,603 @@ +from __future__ import annotations + +import hashlib +import json +import subprocess +from dataclasses import replace +from pathlib import Path +from typing import Any, cast + +import pytest + +from backend.operations import operator_cli, postgres_release_rehearsal +from backend.operations.postgres_release_rehearsal import ( + DatabaseProbe, + DatabaseSnapshot, + LibpqToolRunner, + PostgresRehearsalError, + ToolVersions, + build_target_authorization, + create_backup_manifest, + run_postgres_release_rehearsal, + verify_backup_manifest, +) + + +RELEASE_SHA = "a" * 40 +SOURCE_DSN = "postgresql://source-user:source-secret@staging-db.internal:5432/excel_agent" +RESTORE_DSN = ( + "postgresql://restore-user:restore-secret@localhost:5432/" + "excel_agent_rehearsal_wave3c" +) + + +def _probe(name: str) -> DatabaseProbe: + host = "staging-db.internal" if name == "excel_agent" else "localhost" + fingerprint = hashlib.sha256(f"{host}:5432/{name}".encode()).hexdigest() + return DatabaseProbe( + database_name=name, + database_fingerprint=fingerprint, + server_major=16, + server_version_sha256="3" * 64, + transaction_probe=True, + live_postgres=True, + ) + + +def _snapshot(*, count: int = 7, suffix: str = "post") -> DatabaseSnapshot: + return DatabaseSnapshot( + table_counts={"sessions": count, "session_lifecycle": count, "operation_audits": 2}, + counts_sha256=("4" if suffix == "pre" else "5") * 64, + schema_sha256=("6" if suffix == "pre" else "7") * 64, + logical_sha256=("8" if suffix == "pre" else "9") * 64, + sequence_sha256="b" * 64, + lifecycle_tombstone_sha256="c" * 64, + lifecycle_state_counts={"active": count - 1, "verified": 1}, + ) + + +class _FakeDatabase: + def __init__(self) -> None: + self.source_probe = _probe("excel_agent") + self.restore_probe = _probe("excel_agent_rehearsal_wave3c") + self.pre = _snapshot(suffix="pre") + self.post = _snapshot() + self.source_snapshot_calls = 0 + self.restore_snapshot_calls = 0 + self.migrations: list[str] = [] + self.validations: list[str] = [] + self.target_preflighted = False + self.target_prepared = False + self.cleanup_calls = 0 + self.fail_validation_code = "" + + def probe(self, dsn: str, *, timeout_seconds: int) -> DatabaseProbe: + del timeout_seconds + return self.source_probe if dsn == SOURCE_DSN else self.restore_probe + + def assert_disposable_target_empty( + self, + dsn: str, + *, + expected_database: str, + timeout_seconds: int, + ) -> None: + del timeout_seconds + assert dsn == RESTORE_DSN + assert expected_database == "excel_agent_rehearsal_wave3c" + self.target_preflighted = True + + def prepare_disposable_target( + self, + dsn: str, + *, + expected_database: str, + timeout_seconds: int, + ) -> None: + del timeout_seconds + assert dsn == RESTORE_DSN + assert expected_database == "excel_agent_rehearsal_wave3c" + self.target_prepared = True + + def run_current_migrations(self, dsn: str, *, timeout_seconds: int) -> None: + del timeout_seconds + self.migrations.append(dsn) + + def capture_snapshot(self, dsn: str, *, timeout_seconds: int) -> DatabaseSnapshot: + del timeout_seconds + if dsn == SOURCE_DSN: + self.source_snapshot_calls += 1 + return self.pre if self.source_snapshot_calls == 1 else self.post + self.restore_snapshot_calls += 1 + return self.post + + def validate_release_contracts(self, dsn: str, *, timeout_seconds: int) -> dict[str, object]: + del timeout_seconds + self.validations.append(dsn) + if self.fail_validation_code: + raise PostgresRehearsalError(self.fail_validation_code) + return { + "migrationLedger": "verified", + "typedProjectionBackfills": "verified", + "lifecycleFencing": "verified", + "lifecycleTombstones": "verified", + "ownership": "verified", + "auditPosture": "encrypted_or_redacted", + } + + def cleanup_disposable_target( + self, + dsn: str, + *, + expected_database: str, + timeout_seconds: int, + ) -> None: + del dsn, expected_database, timeout_seconds + self.cleanup_calls += 1 + + +class _FakeTools: + def __init__(self) -> None: + self.tool_versions = ToolVersions( + pg_dump="16.4", + pg_restore="16.4", + pg_dump_sha256="d" * 64, + pg_restore_sha256="e" * 64, + ) + self.dump_calls = 0 + self.restore_calls = 0 + self.fail_code = "" + + def versions(self, *, timeout_seconds: int) -> ToolVersions: + del timeout_seconds + return self.tool_versions + + def dump( + self, + dsn: str, + artifact_path: Path, + *, + timeout_seconds: int, + workspace: Path, + ) -> None: + del timeout_seconds, workspace + assert dsn == SOURCE_DSN + self.dump_calls += 1 + if self.fail_code: + raise PostgresRehearsalError(self.fail_code) + artifact_path.write_bytes(b"PGDMP\x01\x10disposable-test") + + def restore( + self, + dsn: str, + artifact_path: Path, + *, + expected_sha256: str, + timeout_seconds: int, + workspace: Path, + ) -> None: + del timeout_seconds, workspace + assert dsn == RESTORE_DSN + assert artifact_path.is_file() + assert len(expected_sha256) == 64 + self.restore_calls += 1 + if self.fail_code: + raise PostgresRehearsalError(self.fail_code) + + +def _run(database: _FakeDatabase | None = None, tools: _FakeTools | None = None): + return run_postgres_release_rehearsal( + source_dsn=SOURCE_DSN, + restore_dsn=RESTORE_DSN, + target_marker="wave3c", + release_sha=RELEASE_SHA, + database=database or _FakeDatabase(), + tools=tools or _FakeTools(), + generated_at="2026-08-10T12:00:00+00:00", + ) + + +def test_release_rehearsal_runs_complete_provider_free_restore_and_cleanup() -> None: + database = _FakeDatabase() + tools = _FakeTools() + + report = _run(database, tools) + + assert report["status"] == "passed" + assert report["schemaVersion"] == "2026-08-postgres-release-rehearsal-v1" + assert report["releaseSha"] == RELEASE_SHA + assert report["liveDatabase"] is True + assert report["modelProvidersContacted"] is False + assert report["source"]["databaseFingerprint"] == _probe("excel_agent").database_fingerprint + assert report["restore"]["databaseFingerprint"] == _probe( + "excel_agent_rehearsal_wave3c" + ).database_fingerprint + assert report["checks"] == { + "applicationRollbackCompatibility": "verified_without_schema_downgrade", + "auditPosture": "encrypted_or_redacted", + "backupDigest": "verified", + "lifecycleFencing": "verified", + "lifecycleTombstones": "verified", + "logicalEquality": "verified", + "manifestDigest": "verified", + "migrationLedger": "verified", + "ownership": "verified", + "restoreCleanup": "verified", + "schemaEquality": "verified", + "transactionProbes": "verified", + "typedProjectionBackfills": "verified", + } + assert report["canonicalCounts"]["preMigration"] == database.pre.table_counts + assert report["canonicalCounts"]["postMigration"] == database.post.table_counts + assert report["canonicalCounts"]["restored"] == database.post.table_counts + assert database.migrations == [SOURCE_DSN, SOURCE_DSN, RESTORE_DSN] + assert database.validations == [SOURCE_DSN, RESTORE_DSN] + assert database.target_preflighted is True + assert database.target_prepared is True + assert database.cleanup_calls == 1 + assert tools.dump_calls == 1 + assert tools.restore_calls == 1 + + candidate = report["releaseEvidenceCandidate"] + assert candidate["id"] == "postgres_backup_restore" + assert candidate["releaseSha"] == RELEASE_SHA + assert candidate["status"] == "passed" + assert candidate["details"]["successful"] is True + assert candidate["details"]["liveDatabase"] is True + assert candidate["details"]["restoreVerified"] is True + + +def test_report_never_contains_dsns_paths_or_row_data(tmp_path: Path) -> None: + report = _run() + rendered = json.dumps(report, sort_keys=True) + + for forbidden in ( + SOURCE_DSN, + RESTORE_DSN, + "source-secret", + "restore-secret", + str(tmp_path), + "disposable-test", + ): + assert forbidden not in rendered + + +@pytest.mark.parametrize( + ("overrides", "code"), + [ + ({"source_dsn": ""}, "source_dsn_required"), + ({"restore_dsn": ""}, "restore_dsn_required"), + ({"target_marker": ""}, "target_marker_invalid"), + ({"release_sha": "abc"}, "release_sha_invalid"), + ({"restore_dsn": SOURCE_DSN}, "source_restore_not_separate"), + ( + {"restore_dsn": "postgresql://u:p@localhost/ordinary_database"}, + "restore_database_marker_mismatch", + ), + ], +) +def test_request_validation_fails_before_any_probe(overrides: dict[str, str], code: str) -> None: + database = _FakeDatabase() + kwargs: dict[str, Any] = { + "source_dsn": SOURCE_DSN, + "restore_dsn": RESTORE_DSN, + "target_marker": "wave3c", + "release_sha": RELEASE_SHA, + "database": database, + "tools": _FakeTools(), + } + kwargs.update(overrides) + + with pytest.raises(PostgresRehearsalError) as caught: + run_postgres_release_rehearsal(**kwargs) + + assert caught.value.code == code + assert database.migrations == [] + + +def test_production_looking_target_requires_two_step_authorization_digest() -> None: + production_restore = ( + "postgresql://restore-user:restore-secret@prod-primary.internal:5432/" + "excel_agent_rehearsal_wave3c" + ) + + with pytest.raises(PostgresRehearsalError) as caught: + run_postgres_release_rehearsal( + source_dsn=SOURCE_DSN, + restore_dsn=production_restore, + target_marker="wave3c", + release_sha=RELEASE_SHA, + database=_FakeDatabase(), + tools=_FakeTools(), + ) + + assert caught.value.code == "production_target_authorization_required" + authorization = build_target_authorization( + restore_dsn=production_restore, + target_marker="wave3c", + release_sha=RELEASE_SHA, + ) + assert set(authorization) == { + "authorizationDigest", + "releaseSha", + "targetFingerprint", + "targetMarker", + } + assert production_restore not in json.dumps(authorization) + + class ProbeReached: + @staticmethod + def probe(_dsn: str, *, timeout_seconds: int): + del timeout_seconds + raise PostgresRehearsalError("probe_reached") + + with pytest.raises(PostgresRehearsalError) as reached: + run_postgres_release_rehearsal( + source_dsn=SOURCE_DSN, + restore_dsn=production_restore, + target_marker="wave3c", + release_sha=RELEASE_SHA, + target_authorization_digest=authorization["authorizationDigest"], + database=ProbeReached(), # type: ignore[arg-type] + tools=_FakeTools(), + ) + assert reached.value.code == "probe_reached" + + +@pytest.mark.parametrize( + ("mutator", "code"), + [ + (lambda db, tools: setattr(db, "source_probe", replace(db.source_probe, server_major=15)), "postgres_version_unsupported"), + (lambda db, tools: setattr(db, "restore_probe", replace(db.restore_probe, transaction_probe=False)), "postgres_transaction_probe_failed"), + (lambda db, tools: setattr(tools, "tool_versions", replace(tools.tool_versions, pg_dump="15.9")), "postgres_tool_version_mismatch"), + (lambda db, tools: setattr(db, "fail_validation_code", "ownership_validation_failed"), "ownership_validation_failed"), + (lambda db, tools: setattr(tools, "fail_code", "postgres_tool_timeout"), "postgres_tool_timeout"), + ], +) +def test_probe_tool_contract_and_timeout_faults_fail_closed_and_cleanup(mutator, code: str) -> None: + database = _FakeDatabase() + tools = _FakeTools() + mutator(database, tools) + + with pytest.raises(PostgresRehearsalError) as caught: + _run(database, tools) + + assert caught.value.code == code + if database.target_preflighted: + assert database.cleanup_calls == 1 + + +def test_restore_logical_drift_fails_closed_and_cleans_target() -> None: + database = _FakeDatabase() + + def drifted_snapshot(dsn: str, *, timeout_seconds: int) -> DatabaseSnapshot: + del timeout_seconds + if dsn == SOURCE_DSN: + database.source_snapshot_calls += 1 + return database.pre if database.source_snapshot_calls == 1 else database.post + return replace(database.post, logical_sha256="f" * 64) + + database.capture_snapshot = drifted_snapshot # type: ignore[method-assign] + + with pytest.raises(PostgresRehearsalError) as caught: + _run(database) + + assert caught.value.code == "restore_logical_drift" + assert database.cleanup_calls == 1 + + +def test_backup_manifest_digest_and_artifact_tamper_fail_closed(tmp_path: Path) -> None: + artifact = tmp_path / "backup.dump" + manifest_path = tmp_path / "backup.manifest.json" + artifact.write_bytes(b"PGDMP\x01\x10manifest-test") + source = _snapshot() + versions = _FakeTools().tool_versions + create_backup_manifest( + artifact_path=artifact, + manifest_path=manifest_path, + release_sha=RELEASE_SHA, + source_snapshot=source, + source_probe=_probe("excel_agent"), + tool_versions=versions, + created_at="2026-08-10T12:00:00+00:00", + ) + verify_backup_manifest(artifact_path=artifact, manifest_path=manifest_path) + + artifact.write_bytes(artifact.read_bytes() + b"tampered") + with pytest.raises(PostgresRehearsalError) as caught: + verify_backup_manifest(artifact_path=artifact, manifest_path=manifest_path) + assert caught.value.code == "backup_artifact_digest_mismatch" + + artifact.write_bytes(b"PGDMP\x01\x10manifest-test") + envelope = json.loads(manifest_path.read_text(encoding="utf-8")) + envelope["manifest"]["releaseSha"] = "b" * 40 + manifest_path.write_text(json.dumps(envelope), encoding="utf-8") + with pytest.raises(PostgresRehearsalError) as caught: + verify_backup_manifest(artifact_path=artifact, manifest_path=manifest_path) + assert caught.value.code == "backup_manifest_digest_mismatch" + + +def test_operator_cli_generates_production_target_authorization_without_connecting(capsys) -> None: + production_restore = ( + "postgresql://restore-user:restore-secret@prod-primary.internal:5432/" + "excel_agent_rehearsal_wave3c" + ) + + result = operator_cli.main( + [ + "postgres-release-rehearsal", + "--authorize-target-only", + "--restore-dsn", + production_restore, + "--target-marker", + "wave3c", + "--release-sha", + RELEASE_SHA, + ] + ) + + output = capsys.readouterr() + payload = json.loads(output.out) + assert result == 0 + assert output.err == "" + assert payload["status"] == "authorization_generated" + assert payload["liveDatabase"] is False + assert len(payload["authorization"]["authorizationDigest"]) == 64 + assert production_restore not in output.out + assert "restore-secret" not in output.out + + +def test_operator_cli_failure_is_one_redacted_document(monkeypatch, capsys) -> None: + def fail(**_kwargs): + raise PostgresRehearsalError("pg_dump_failed") + + monkeypatch.setattr(operator_cli, "run_postgres_release_rehearsal", fail) + result = operator_cli.main( + [ + "postgres-release-rehearsal", + "--source-dsn", + SOURCE_DSN, + "--restore-dsn", + RESTORE_DSN, + "--target-marker", + "wave3c", + "--release-sha", + RELEASE_SHA, + ] + ) + + output = capsys.readouterr() + payload = json.loads(output.out) + assert result == 1 + assert output.out.count("\n") == 1 + assert output.err == "" + assert payload == { + "errorCode": "pg_dump_failed", + "liveDatabase": False, + "modelProvidersContacted": False, + "releaseSha": RELEASE_SHA, + "schemaVersion": "2026-08-postgres-release-rehearsal-v1", + "status": "failed", + } + assert SOURCE_DSN not in output.out + assert RESTORE_DSN not in output.out + + +def test_libpq_runner_uses_service_file_without_dsn_in_subprocess_args( + tmp_path: Path, + monkeypatch, +) -> None: + tool = tmp_path / "pg_dump" + tool.write_bytes(b"disposable-binary") + runner = object.__new__(LibpqToolRunner) + runner._pg_dump = tool # type: ignore[attr-defined] + captured: dict[str, object] = {} + + def fake_run_tool( + args: list[str], + *, + service_file: Path | None, + timeout_seconds: int, + failure_code: str, + ) -> bytes: + captured.update( + args=args, + service_file=service_file, + timeout_seconds=timeout_seconds, + failure_code=failure_code, + ) + output_index = args.index("--file") + 1 + Path(args[output_index]).write_bytes(b"PGDMP\x01\x10fixture") + return b"" + + monkeypatch.setattr(LibpqToolRunner, "_run_tool", staticmethod(fake_run_tool)) + artifact = tmp_path / "source.dump" + runner.dump(SOURCE_DSN, artifact, timeout_seconds=30, workspace=tmp_path) + + rendered_args = json.dumps(captured["args"]) + assert SOURCE_DSN not in rendered_args + assert "source-secret" not in rendered_args + captured_args = cast(list[str], captured["args"]) + assert "--dbname=service=source" in captured_args + service_file = captured["service_file"] + assert isinstance(service_file, Path) + assert service_file.stat().st_mode & 0o777 == 0o600 + + +@pytest.mark.parametrize("interrupt", ["timeout", "cancel"]) +def test_libpq_subprocess_timeout_and_cancellation_terminate_process_group( + monkeypatch, + interrupt: str, +) -> None: + captured: dict[str, object] = {} + + class FakeProcess: + pid = 4312 + returncode = None + + def communicate(self, *, timeout: int): + captured["communicate_timeout"] = timeout + if interrupt == "timeout": + raise subprocess.TimeoutExpired(cmd="pg_dump", timeout=timeout) + raise KeyboardInterrupt + + def fake_popen(args, **kwargs): + captured["args"] = args + captured["kwargs"] = kwargs + return FakeProcess() + + terminated: list[int] = [] + monkeypatch.setattr(postgres_release_rehearsal.subprocess, "Popen", fake_popen) + monkeypatch.setattr( + postgres_release_rehearsal, + "_terminate_process_group", + lambda process: terminated.append(process.pid), + ) + + if interrupt == "timeout": + with pytest.raises(PostgresRehearsalError) as caught: + LibpqToolRunner._run_tool( + ["/trusted/pg_dump", "--version"], + service_file=None, + timeout_seconds=7, + failure_code="postgres_tool_failed", + ) + assert caught.value.code == "postgres_tool_timeout" + else: + with pytest.raises(KeyboardInterrupt): + LibpqToolRunner._run_tool( + ["/trusted/pg_dump", "--version"], + service_file=None, + timeout_seconds=7, + failure_code="postgres_tool_failed", + ) + assert terminated == [4312] + captured_kwargs = cast(dict[str, object], captured["kwargs"]) + assert captured_kwargs["shell"] is False + assert captured_kwargs["start_new_session"] is True + + +def test_libpq_failure_discards_stderr_that_may_contain_credentials(monkeypatch) -> None: + class FakeProcess: + pid = 4313 + returncode = 1 + + @staticmethod + def communicate(*, timeout: int): + del timeout + return b"", SOURCE_DSN.encode() + + monkeypatch.setattr( + postgres_release_rehearsal.subprocess, + "Popen", + lambda *_args, **_kwargs: FakeProcess(), + ) + with pytest.raises(PostgresRehearsalError) as caught: + LibpqToolRunner._run_tool( + ["/trusted/pg_dump"], + service_file=None, + timeout_seconds=7, + failure_code="pg_dump_failed", + ) + assert caught.value.code == "pg_dump_failed" + assert SOURCE_DSN not in str(caught.value) diff --git a/backend/tests/test_prod_compose_config.py b/backend/tests/test_prod_compose_config.py index 64e3319..f27a5e7 100644 --- a/backend/tests/test_prod_compose_config.py +++ b/backend/tests/test_prod_compose_config.py @@ -1,3 +1,4 @@ +import re from pathlib import Path @@ -12,6 +13,13 @@ def test_production_compose_is_postgres_first(): assert "postgres:" in compose assert "condition: service_healthy" in compose assert 'profiles: ["postgres"]' not in compose + assert r'pg_isready -U \"$$POSTGRES_USER\" -d \"$$POSTGRES_DB\"' in compose + assert "HYPERFORMULA_ENABLED=${HYPERFORMULA_ENABLED:-false}" in compose + assert "HYPERFORMULA_DISTRIBUTED=${HYPERFORMULA_DISTRIBUTED:-false}" in compose + assert "FORMULA_ENGINE=${FORMULA_ENGINE:-univer}" in compose + assert "EXCEL_AGENT_LIFECYCLE_RECEIPT_KEY_ID=${EXCEL_AGENT_LIFECYCLE_RECEIPT_KEY_ID:?" in compose + assert "EXCEL_AGENT_LIFECYCLE_RECEIPT_HMAC_KEY=${EXCEL_AGENT_LIFECYCLE_RECEIPT_HMAC_KEY:?" in compose + assert "EXCEL_AGENT_LIFECYCLE_RECEIPT_PREVIOUS_KEYS=${EXCEL_AGENT_LIFECYCLE_RECEIPT_PREVIOUS_KEYS:-{}}" in compose def test_nginx_redacts_query_strings_and_enforces_upload_limit(): @@ -21,3 +29,79 @@ def test_nginx_redacts_query_strings_and_enforces_upload_limit(): assert '"uri":"$uri"' in nginx assert '"request":"$request"' not in nginx assert "proxy_set_header X-Request-ID $request_id;" in nginx + assert "connect-src 'self' ${OIDC_BROWSER_CONNECT_ORIGINS};" in nginx + assert "connect-src 'self' https: wss:" not in nginx + assert "OIDC_BROWSER_CONNECT_ORIGINS=${OIDC_BROWSER_CONNECT_ORIGINS:?" in ( + ROOT / "docker-compose.prod.yml" + ).read_text(encoding="utf-8") + + +def test_production_edge_forwards_user_identity_without_shared_admin_key(): + nginx = (ROOT / "infra" / "nginx" / "nginx.conf").read_text(encoding="utf-8") + compose = (ROOT / "docker-compose.prod.yml").read_text(encoding="utf-8") + smoke = (ROOT / "infra" / "nginx" / "smoke-production.sh").read_text(encoding="utf-8") + readme = (ROOT / "README.md").read_text(encoding="utf-8") + runbook = (ROOT / "docs" / "PRODUCTION_RUNBOOK.md").read_text(encoding="utf-8") + + for release_surface in (nginx, compose, readme, runbook): + assert "EDGE_BACKEND_API_KEY" not in release_surface + assert "proxy_set_header Authorization $http_authorization;" in nginx + assert 'proxy_set_header X-API-Key "";' in nginx + assert "proxy_pass http://backend" not in nginx + assert "proxy_ssl_verify on;" in nginx + assert "--ssl-cert-reqs" in compose + assert "OIDC_ENABLED=true" in compose + assert "OIDC_ALLOWED_TENANTS" in compose + assert "SMOKE_OIDC_BEARER_TOKEN" in smoke + assert "SMOKE_OIDC_CONNECT_ORIGINS" in smoke + assert 'unauthenticated_status" != "401"' in smoke + assert "authorization-code" in readme + assert "mTLS" in readme and "mTLS" in runbook + + +def test_production_edge_renders_only_exact_oidc_csp_origins() -> None: + render = (ROOT / "infra" / "nginx" / "40-render-production-manifest.sh").read_text(encoding="utf-8") + workflow = (ROOT / ".github" / "workflows" / "v7-readiness.yml").read_text(encoding="utf-8") + + assert "OIDC_BROWSER_CONNECT_ORIGINS entries must be exact HTTPS origins" in render + assert "^https://[A-Za-z0-9.-]+(:[0-9]{1,5})?$" in render + assert "envsubst '${OIDC_BROWSER_CONNECT_ORIGINS}'" in render + assert "OIDC_BROWSER_CONNECT_ORIGINS=https://identity.example.test" in workflow + assert "connect-src[^;]*[[:space:]](https:|wss:" in workflow + + +def test_development_compose_uses_same_origin_vite_proxy_and_loopback_ports(): + compose = (ROOT / "docker-compose.yml").read_text(encoding="utf-8") + vite = (ROOT / "plugin" / "vite.config.js").read_text(encoding="utf-8") + + for port in ("3443", "8000", "11434"): + assert f'"127.0.0.1:{port}:{port}"' in compose + assert "VITE_DEV_BACKEND_TARGET=http://backend:8000" in compose + assert "VITE_API_BASE=" not in compose + assert '"/api"' in vite and '"/ws"' in vite + assert "ws: true" in vite + + +def test_shipped_container_bases_are_immutable_digest_pins(): + backend_dockerfile = (ROOT / "backend" / "Dockerfile").read_text(encoding="utf-8") + edge_dockerfile = (ROOT / "infra" / "nginx" / "Dockerfile").read_text(encoding="utf-8") + production_compose = (ROOT / "docker-compose.prod.yml").read_text(encoding="utf-8") + development_compose = (ROOT / "docker-compose.yml").read_text(encoding="utf-8") + digest_reference = re.compile(r"@sha256:[0-9a-f]{64}(?:\s|$)") + + from_lines = [ + line.strip() + for document in (backend_dockerfile, edge_dockerfile) + for line in document.splitlines() + if line.strip().startswith("FROM ") + ] + assert len(from_lines) == 4 + assert all(digest_reference.search(line) for line in from_lines), from_lines + assert re.search(r"image:\s+postgres:[^\s]+@sha256:[0-9a-f]{64}", production_compose) + assert re.search(r"image:\s+node:[^\s]+@sha256:[0-9a-f]{64}", development_compose) + assert "USER nginx" in edge_dockerfile + assert "EXPOSE 8080 8443" in edge_dockerfile + assert "listen 8080;" in (ROOT / "infra" / "nginx" / "nginx.conf").read_text(encoding="utf-8") + assert "listen 8443 ssl;" in (ROOT / "infra" / "nginx" / "nginx.conf").read_text(encoding="utf-8") + assert '"443:8443"' in production_compose + assert '"80:8080"' in production_compose diff --git a/backend/tests/test_production_capability_surface.py b/backend/tests/test_production_capability_surface.py new file mode 100644 index 0000000..01fc3e6 --- /dev/null +++ b/backend/tests/test_production_capability_surface.py @@ -0,0 +1,38 @@ +from __future__ import annotations + +import os +import subprocess +import sys +from pathlib import Path + + +ROOT = Path(__file__).resolve().parents[2] + + +def test_production_openapi_omits_experimental_and_unscoped_rag_routes(): + env = { + **os.environ, + "APP_ENV": "prod", + "CONFIG_PROFILE": "prod", + "EXPERIMENTAL_ROUTES_ENABLED": "false", + "RAG_ENABLED": "false", + } + script = """ +from backend.main import app +paths = set(app.openapi()["paths"]) +assert not any(path.startswith("/api/v1/experiments") for path in paths), paths +assert "/api/v1/rag/search" not in paths, paths +assert "/api/v1/intelligence/stats" in paths, paths +""" + + result = subprocess.run( + [sys.executable, "-c", script], + cwd=ROOT, + env=env, + capture_output=True, + text=True, + timeout=30, + check=False, + ) + + assert result.returncode == 0, result.stdout + result.stderr diff --git a/backend/tests/test_rebuild_contracts.py b/backend/tests/test_rebuild_contracts.py index 97cd36c..e0c6d47 100644 --- a/backend/tests/test_rebuild_contracts.py +++ b/backend/tests/test_rebuild_contracts.py @@ -1,12 +1,14 @@ """Trust-first rebuild contract tests.""" +import uuid + from fastapi.testclient import TestClient from backend.config import API_KEY from backend.main import app from backend.models.operations import OperationPlan, OperationPlanV2, WriteCellOperationV2, legacy_to_operation, plan_to_v2 from backend.services.operation_validation import validate_workbook_operations -from backend.session_store import SessionStore +from backend.session_store import SessionStore, get_session_store def test_legacy_operation_normalizes_to_typed_write_cell(): @@ -39,6 +41,7 @@ def test_session_store_persists_operation_plan(tmp_path): operations=[legacy_to_operation({"cell": "A1", "value": "Test"}, index=0)], ) payload = plan.model_dump(mode="json", by_alias=True) + store.get_or_create_session("session-a") store.save_operation_plan(payload) loaded = store.get_operation_plan(plan.id) @@ -81,13 +84,15 @@ def test_existing_plan_can_be_viewed_as_v2_contract(): def test_plan_apply_timeline_flow_uses_durable_contract(): client = TestClient(app) headers = {"X-API-Key": API_KEY} + session_id = f"contract-flow-{uuid.uuid4().hex[:10]}" + get_session_store().get_or_create_session(session_id) plan_resp = client.post( "/api/v1/tasks/plan", headers=headers, json={ "prompt": "Build a DCF model", - "session_id": "contract-flow", + "session_id": session_id, "context": {"sheet_names": ["Sheet1"], "active_sheet": "Sheet1", "cell_snapshot": {}}, }, ) @@ -102,7 +107,7 @@ def test_plan_apply_timeline_flow_uses_durable_contract(): headers=headers, json={ "plan_id": plan["id"], - "session_id": "contract-flow", + "session_id": session_id, "workbook_fingerprint": "Sheet1|A1|0|0|0", "workbook_state": {"cell_snapshot": {}}, }, @@ -112,7 +117,7 @@ def test_plan_apply_timeline_flow_uses_durable_contract(): assert applied["rollback_snapshot_id"].startswith("snap_") assert applied["audit_event_id"].startswith("audit_") - timeline_resp = client.get("/api/v1/sessions/contract-flow/timeline", headers=headers) + timeline_resp = client.get(f"/api/v1/sessions/{session_id}/timeline", headers=headers) assert timeline_resp.status_code == 200 timeline = timeline_resp.json() assert timeline["plans"] @@ -123,7 +128,8 @@ def test_plan_apply_timeline_flow_uses_durable_contract(): def test_apply_result_rejects_wrong_apply_token(): client = TestClient(app) headers = {"X-API-Key": API_KEY} - session_id = "contract-token-flow" + session_id = f"contract-token-flow-{uuid.uuid4().hex[:10]}" + get_session_store().get_or_create_session(session_id) plan_resp = client.post( "/api/v1/tasks/plan", @@ -149,6 +155,7 @@ def test_apply_result_rejects_wrong_apply_token(): }, ) assert approve_resp.status_code == 200 + approved = approve_resp.json() bad_result = client.post( "/api/v1/tasks/apply/result", @@ -157,6 +164,7 @@ def test_apply_result_rejects_wrong_apply_token(): "plan_id": plan["id"], "session_id": session_id, "apply_token": "wrong-token", + "apply_attempt_id": approved["apply_attempt_id"], "status": "applied", "office_apply_result": {"applied": 1}, }, @@ -167,7 +175,8 @@ def test_apply_result_rejects_wrong_apply_token(): def test_contract_v2_endpoint_exposes_existing_plan(): client = TestClient(app) headers = {"X-API-Key": API_KEY} - session_id = "contract-v2-endpoint" + session_id = f"contract-v2-endpoint-{uuid.uuid4().hex[:10]}" + get_session_store().get_or_create_session(session_id) plan_resp = client.post( "/api/v1/tasks/plan", headers=headers, diff --git a/backend/tests/test_release_evidence_prereqs.py b/backend/tests/test_release_evidence_prereqs.py new file mode 100644 index 0000000..a94bd09 --- /dev/null +++ b/backend/tests/test_release_evidence_prereqs.py @@ -0,0 +1,345 @@ +from __future__ import annotations + +import json +from typing import Any + +from scripts import verify_release_evidence_prereqs as prereqs + + +def test_release_evidence_prereq_report_blocks_external_gaps(monkeypatch) -> None: + monkeypatch.setattr( + prereqs.trust, + "windows_excel_smoke_snapshot", + lambda _path=None: {"status": "missing_evidence", "requiredActions": ["Complete Windows Excel smoke."]}, + ) + monkeypatch.setattr( + prereqs.trust, + "smoke_artifact_validation_snapshot", + lambda _path=None: {"status": "skipped", "ready": False, "requiredActions": []}, + ) + monkeypatch.setattr( + prereqs, + "private_golden_fixture_policy_snapshot", + lambda *, require_private_fixtures: { + "status": "missing", + "ready": False, + "required": require_private_fixtures, + "fixtureCount": 0, + "requiredActions": ["Populate private golden fixtures."], + }, + ) + monkeypatch.setattr( + prereqs.trust, + "fixture_inventory_audit_snapshot", + lambda: { + "status": "missing", + "fixtureCount": 0, + "checkedFixtureCount": 0, + "failedFixtureCount": 0, + "uncheckedFixtureCount": 0, + "missingFeatures": [], + "requiredActions": ["Run fixture inventory audit."], + }, + ) + monkeypatch.setattr( + prereqs.trust, + "parser_comparison_corpus_snapshot", + lambda: { + "status": "missing", + "comparisonCorpusReady": False, + "fixtureCount": 0, + "materialMismatchCount": 0, + "failed": 0, + "requiredActions": ["Run parser comparison corpus."], + }, + ) + monkeypatch.setattr( + prereqs, + "replay_baseline_snapshot", + lambda _baseline_id=None: { + "status": "non_production", + "ready": False, + "baselineId": "replaybase_v7_local_seed_v1", + "requiredActions": ["Capture production replay evidence."], + }, + ) + monkeypatch.setattr( + prereqs, + "dependency_risk_snapshot", + lambda: { + "hyperformulaLicenseStatus": { + "status": "blocked_unapproved", + "releaseBlocker": True, + "unresolved": [ + "hyperformula_license_approval_missing", + "hyperformula_license_key_missing", + ], + } + }, + ) + + report = prereqs.build_prereq_report(replay_baseline_id="replaybase_v7_local_seed_v1") + + assert report["status"] == "blocked" + assert report["ready"] is False + assert report["blockerCount"] == 7 + assert [item["key"] for item in report["prerequisites"]] == [ + "windows_excel_manual_smoke", + "windows_excel_artifacts", + "private_golden_fixtures", + "fixture_inventory_audit", + "parser_comparison_corpus", + "production_replay_baseline", + "hyperformula_license", + ] + assert ( + "Run scripts/validate_windows_excel_smoke.py --require-existing-artifacts --json and pass the JSON path." + in report["requiredActions"] + ) + assert any("HYPERFORMULA_LICENSE_APPROVED" in action for action in report["requiredActions"]) + assert any("HYPERFORMULA_LICENSE_KEY" in action for action in report["requiredActions"]) + + +def test_release_evidence_prereq_report_ready_when_all_external_evidence_is_present(monkeypatch) -> None: + monkeypatch.setattr(prereqs.trust, "windows_excel_smoke_snapshot", lambda _path=None: {"status": "present", "requiredActions": []}) + monkeypatch.setattr( + prereqs.trust, + "smoke_artifact_validation_snapshot", + lambda _path=None: {"status": "present", "ready": True, "requiredActions": []}, + ) + monkeypatch.setattr( + prereqs, + "private_golden_fixture_policy_snapshot", + lambda *, require_private_fixtures: { + "status": "ready", + "ready": True, + "required": require_private_fixtures, + "fixtureCount": 4, + "requiredActions": [], + }, + ) + monkeypatch.setattr( + prereqs.trust, + "fixture_inventory_audit_snapshot", + lambda: { + "status": "passed", + "fixtureCount": 11, + "checkedFixtureCount": 11, + "failedFixtureCount": 0, + "uncheckedFixtureCount": 0, + "missingFeatures": [], + "requiredActions": [], + }, + ) + monkeypatch.setattr( + prereqs.trust, + "parser_comparison_corpus_snapshot", + lambda: { + "status": "matched", + "comparisonCorpusReady": True, + "fixtureCount": 11, + "materialMismatchCount": 0, + "failed": 0, + "requiredActions": [], + }, + ) + monkeypatch.setattr( + prereqs, + "replay_baseline_snapshot", + lambda _baseline_id=None: { + "status": "ready", + "ready": True, + "baselineId": "replaybase_v7_prod", + "requiredActions": [], + }, + ) + monkeypatch.setattr( + prereqs, + "dependency_risk_snapshot", + lambda: { + "hyperformulaLicenseStatus": { + "status": "approved", + "releaseBlocker": False, + "unresolved": [], + } + }, + ) + + report = prereqs.build_prereq_report() + + assert report["status"] == "ready" + assert report["ready"] is True + assert report["blockerCount"] == 0 + assert report["requiredActions"] == [] + assert "| HyperFormula license posture | `approved` | `yes` |" in prereqs.markdown_summary(report) + + +def test_replay_baseline_snapshot_requires_production_evidence(monkeypatch) -> None: + class Store: + def list_local_replay_baselines(self, _limit: int) -> list[dict[str, Any]]: + return [ + { + "baselineId": "replaybase_v7_local_seed_v1", + "replayCount": 2, + "metadata": { + "productionEvidence": False, + "evidenceClass": "deterministic_plumbing_seed", + }, + } + ] + + monkeypatch.setattr(prereqs, "get_session_store", lambda: Store()) + + snapshot = prereqs.replay_baseline_snapshot("replaybase_v7_local_seed_v1") + + assert snapshot["status"] == "non_production" + assert snapshot["ready"] is False + assert snapshot["evidenceClass"] == "deterministic_plumbing_seed" + assert any("production-like anonymized replay corpus" in action for action in snapshot["requiredActions"]) + + +def test_replay_baseline_snapshot_prefers_production_baseline(monkeypatch) -> None: + class Store: + def list_local_replay_baselines(self, _limit: int) -> list[dict[str, Any]]: + return [ + { + "baselineId": "replaybase_v7_local_seed_v1", + "replayCount": 2, + "metadata": {"productionEvidence": False, "evidenceClass": "deterministic_plumbing_seed"}, + }, + { + "baselineId": "replaybase_v7_prod", + "replayCount": 25, + "metadata": {"productionEvidence": True, "evidenceClass": "production_anonymized_corpus"}, + }, + ] + + monkeypatch.setattr(prereqs, "get_session_store", lambda: Store()) + + snapshot = prereqs.replay_baseline_snapshot() + + assert snapshot["status"] == "ready" + assert snapshot["ready"] is True + assert snapshot["baselineId"] == "replaybase_v7_prod" + assert snapshot["replayCount"] == 25 + + +def test_prereq_cli_strict_returns_nonzero_and_json(monkeypatch, capsys) -> None: + monkeypatch.setattr( + prereqs, + "build_prereq_report", + lambda **_kwargs: { + "schemaVersion": prereqs.SCHEMA_VERSION, + "status": "blocked", + "ready": False, + "blockerCount": 1, + "prerequisites": [], + "requiredActions": ["Attach evidence."], + "evidence": {}, + }, + ) + + result = prereqs.main(["--strict", "--json"]) + payload = json.loads(capsys.readouterr().out) + + assert result == 1 + assert payload["status"] == "blocked" + assert payload["requiredActions"] == ["Attach evidence."] + + +def test_prereq_cli_suppresses_incidental_diagnostics(monkeypatch, capsys) -> None: + def report(**_kwargs): + print("storage diagnostic db='/Users/example/private/sessions.db'") + return { + "schemaVersion": prereqs.SCHEMA_VERSION, + "status": "blocked", + "ready": False, + "blockerCount": 1, + "prerequisites": [], + "requiredActions": [], + "evidence": {}, + } + + monkeypatch.setattr(prereqs, "build_prereq_report", report) + + assert prereqs.main(["--strict", "--json"]) == 1 + captured = capsys.readouterr() + assert json.loads(captured.out)["status"] == "blocked" + assert captured.err == "" + assert "/Users/example" not in captured.out + + +def test_release_evidence_index_snapshot_fails_closed_when_missing_or_stale(monkeypatch, tmp_path) -> None: + missing = prereqs.evidence_index_snapshot( + evidence_index_path=None, + evidence_signature_path=None, + evidence_root=None, + release_sha=None, + trusted_public_key_path=None, + trusted_signing_key_id=None, + ) + assert missing["status"] == "missing" + assert missing["ready"] is False + assert "--evidence-index" in missing["requiredActions"][0] + + def reject_stale(**_kwargs): + raise prereqs.evidence_index.EvidenceIndexError("evidence item targets a different SHA") + + monkeypatch.setattr(prereqs.evidence_index, "validate_release_evidence_index", reject_stale) + stale = prereqs.evidence_index_snapshot( + evidence_index_path=tmp_path / "release-evidence-index.json", + evidence_signature_path=tmp_path / "release-evidence-index.ed25519", + evidence_root=tmp_path, + release_sha="a" * 40, + trusted_public_key_path=tmp_path / "trusted-public-key.pem", + trusted_signing_key_id="release-key-1", + ) + assert stale["status"] == "blocked" + assert stale["ready"] is False + assert "different SHA" in stale["requiredActions"][0] + + +def test_strict_prereq_cli_requires_exact_sha_evidence_index(monkeypatch, capsys) -> None: + captured: dict[str, Any] = {} + + def report(**kwargs): + captured.update(kwargs) + return { + "schemaVersion": prereqs.SCHEMA_VERSION, + "status": "blocked", + "ready": False, + "blockerCount": 1, + "prerequisites": [], + "requiredActions": ["Supply exact-SHA evidence."], + "evidence": {}, + } + + monkeypatch.setattr(prereqs, "build_prereq_report", report) + result = prereqs.main( + [ + "--strict", + "--json", + "--evidence-index", + "artifact/release-evidence-index.json", + "--evidence-signature", + "artifact/release-evidence-index.ed25519", + "--evidence-root", + "artifact", + "--release-sha", + "a" * 40, + "--trusted-public-key", + "runner-temp/trusted-release-key.pem", + "--trusted-signing-key-id", + "release-key-1", + ] + ) + capsys.readouterr() + + assert result == 1 + assert captured["require_evidence_index"] is True + assert captured["evidence_index_path"].as_posix() == "artifact/release-evidence-index.json" + assert captured["evidence_signature_path"].as_posix() == "artifact/release-evidence-index.ed25519" + assert captured["evidence_root"].as_posix() == "artifact" + assert captured["release_sha"] == "a" * 40 + assert captured["trusted_public_key_path"].as_posix() == "runner-temp/trusted-release-key.pem" + assert captured["trusted_signing_key_id"] == "release-key-1" diff --git a/backend/tests/test_release_trust_bundle.py b/backend/tests/test_release_trust_bundle.py new file mode 100644 index 0000000..da0fa86 --- /dev/null +++ b/backend/tests/test_release_trust_bundle.py @@ -0,0 +1,1027 @@ +"""Tests for release trust bundle evidence metadata.""" + +from __future__ import annotations + +import json +import subprocess +import sys +from pathlib import Path +from typing import Any + +from scripts import generate_release_trust_bundle as trust +from scripts import scaffold_windows_excel_smoke as smoke_scaffold +from scripts import validate_windows_excel_smoke as smoke_validator + + +def test_windows_excel_smoke_snapshot_reports_actionable_template_gaps(tmp_path: Path, monkeypatch) -> None: + smoke_doc = tmp_path / "WINDOWS_EXCEL_MANUAL_SMOKE.md" + smoke_doc.write_text( + """# Windows Excel Manual Smoke Checklist + +## Run Metadata + +| Field | Value | +| --- | --- | +| Date | | +| Tester | | +| Overall result | Pass / Pass with Warning / Blocked / Fail | + +## Automated Preflight + +| Gate | Command | Result | Evidence / Notes | +| --- | --- | --- | --- | +| Full stack | `cmd` | | | +""", + encoding="utf-8", + ) + monkeypatch.setattr(trust, "SMOKE_DOC", smoke_doc) + + snapshot = trust.windows_excel_smoke_snapshot() + + assert snapshot["status"] == "missing_evidence" + assert "Evidence bundle path" in snapshot["runMetadata"]["missingRequiredFields"] + assert snapshot["checklistCoverage"]["missingStatusRows"] == 1 + assert snapshot["checklistCoverage"]["missingSections"] + assert snapshot["requiredActions"] + + +def test_windows_excel_smoke_snapshot_accepts_complete_evidence(tmp_path: Path, monkeypatch) -> None: + smoke_doc = tmp_path / "WINDOWS_EXCEL_MANUAL_SMOKE.md" + smoke_doc.write_text(complete_smoke_doc(), encoding="utf-8") + monkeypatch.setattr(trust, "SMOKE_DOC", smoke_doc) + + snapshot = trust.windows_excel_smoke_snapshot() + + assert snapshot["status"] == "present" + assert snapshot["runMetadata"]["semanticGaps"] == [] + assert snapshot["checklistCoverage"]["coveredRows"] == snapshot["checklistCoverage"]["totalRows"] + assert snapshot["checklistCoverage"]["missingEvidenceRows"] == 0 + assert snapshot["blockers"]["count"] == 0 + assert snapshot["requiredActions"] == [] + + +def test_windows_excel_smoke_snapshot_accepts_explicit_smoke_doc_path(tmp_path: Path, monkeypatch) -> None: + default_doc = tmp_path / "default.md" + explicit_doc = tmp_path / "explicit.md" + default_doc.write_text("# Missing\n", encoding="utf-8") + explicit_doc.write_text(complete_smoke_doc(), encoding="utf-8") + monkeypatch.setattr(trust, "SMOKE_DOC", default_doc) + + snapshot = trust.windows_excel_smoke_snapshot(explicit_doc) + + assert snapshot["status"] == "present" + assert snapshot["path"].endswith("explicit.md") + + +def test_windows_excel_smoke_snapshot_rejects_vague_semantic_metadata(tmp_path: Path, monkeypatch) -> None: + smoke_doc = tmp_path / "WINDOWS_EXCEL_MANUAL_SMOKE.md" + smoke_doc.write_text( + complete_smoke_doc() + .replace("| Git commit | a1b2c3d |", "| Git commit | captured |") + .replace("| Excel version/build | Microsoft Excel 2405 Build 17628 |", "| Excel version/build | captured |") + .replace("| Backend command | python -m uvicorn backend.main:app |", "| Backend command | captured |") + .replace("| Preflight trust bundle ID | windows-excel-preflight-20260615T190539Z |", "| Preflight trust bundle ID | captured |"), + encoding="utf-8", + ) + monkeypatch.setattr(trust, "SMOKE_DOC", smoke_doc) + + snapshot = trust.windows_excel_smoke_snapshot() + + assert snapshot["status"] == "incomplete" + gap_fields = {gap["field"] for gap in snapshot["runMetadata"]["semanticGaps"]} + assert {"Git commit", "Excel version/build", "Backend command", "Preflight trust bundle ID"}.issubset(gap_fields) + assert any("concrete evidence" in action for action in snapshot["requiredActions"]) + + +def test_windows_excel_smoke_snapshot_does_not_require_final_bundle_id(tmp_path: Path, monkeypatch) -> None: + smoke_doc = tmp_path / "WINDOWS_EXCEL_MANUAL_SMOKE.md" + smoke_doc.write_text( + complete_smoke_doc().replace( + "| Preflight trust bundle ID | captured |", + "| Preflight trust bundle ID | preflight-123 |\n| Release trust bundle ID | |", + ), + encoding="utf-8", + ) + monkeypatch.setattr(trust, "SMOKE_DOC", smoke_doc) + + snapshot = trust.windows_excel_smoke_snapshot() + + assert snapshot["status"] == "present" + assert "Release trust bundle ID" not in snapshot["runMetadata"]["missingRequiredFields"] + + +def test_full_mode_resolves_windows_npm_cmd_for_gates_and_environment(monkeypatch) -> None: + npm_cmd = r"C:\Program Files\nodejs\npm.CMD" + command_text_calls: list[list[str]] = [] + + def fake_which(name: str) -> str | None: + return npm_cmd if name == "npm.CMD" else None + + def fake_command_text(command: list[str]) -> str: + command_text_calls.append(command) + return "ok" + + monkeypatch.setattr(trust.shutil, "which", fake_which) + monkeypatch.setattr(trust, "command_text", fake_command_text) + monkeypatch.setattr(trust, "command_exists", lambda _name: False) + + npm_gates = [gate for gate in trust.selected_gates("full") if gate.name.startswith("plugin_") or gate.name == "workbook_worker_tests"] + environment = trust.environment_snapshot() + + assert npm_gates + assert all(gate.command[0] == npm_cmd for gate in npm_gates) + assert environment["npmExecutable"] == npm_cmd + assert [npm_cmd, "--version"] in command_text_calls + + +def test_quick_mode_runs_parser_comparison_corpus_gate() -> None: + gates = trust.selected_gates("quick") + + assert any( + gate.name == "sbom_drift" + and gate.command[-2:] == ["scripts/generate_sbom.py", "--check"] + for gate in gates + ) + assert any( + gate.name == "parser_comparison_corpus" + and gate.command[-2:] == ["scripts/run_parser_comparison_corpus.py", "--fail-on-mismatch"] + for gate in gates + ) + + +def test_summarize_promotes_manual_smoke_failures_to_release_failures() -> None: + bundle: dict[str, Any] = { + "gates": [{"name": "metadata_only", "status": "skipped"}], + "windowsExcelSmoke": {"status": "failed", "requiredActions": ["Fix Excel smoke failure."]}, + "fixtureInventoryAudit": passed_fixture_inventory(), + "parserComparisonCorpus": passed_parser_corpus(), + "summary": {}, + } + + trust.summarize(bundle, fail_on_warning=False) + + assert bundle["summary"]["status"] == "failed" + assert bundle["summary"]["failed"] == 1 + assert bundle["summary"]["windowsExcelSmokeReady"] is False + assert bundle["summary"]["requiredActions"] == ["Fix Excel smoke failure."] + + +def test_summarize_promotes_missing_parser_corpus_to_release_failure() -> None: + bundle: dict[str, Any] = { + "gates": [{"name": "metadata_only", "status": "skipped"}], + "windowsExcelSmoke": {"status": "present", "requiredActions": []}, + "fixtureInventoryAudit": passed_fixture_inventory(), + "parserComparisonCorpus": { + "status": "missing", + "requiredActions": ["Run python scripts/run_parser_comparison_corpus.py --fail-on-mismatch."], + }, + "summary": {}, + } + + trust.summarize(bundle, fail_on_warning=False) + + assert bundle["summary"]["status"] == "failed" + assert bundle["summary"]["failed"] == 1 + assert bundle["summary"]["parserComparisonCorpusReady"] is False + assert "Run python scripts/run_parser_comparison_corpus.py --fail-on-mismatch." in bundle["summary"]["requiredActions"] + + +def test_summarize_promotes_required_private_fixture_gap_to_release_failure() -> None: + bundle: dict[str, Any] = { + "gates": [{"name": "metadata_only", "status": "skipped"}], + "windowsExcelSmoke": {"status": "present", "requiredActions": []}, + "smokeArtifactValidation": {"status": "present", "requiredActions": []}, + "fixtureInventoryAudit": passed_fixture_inventory(), + "parserComparisonCorpus": passed_parser_corpus(), + "privateGoldenFixtures": { + "status": "missing", + "ready": False, + "required": True, + "fixtureCount": 0, + "issueCodes": [], + "requiredActions": ["Run private golden fixture validation until status is ready."], + }, + "summary": {}, + } + + trust.summarize(bundle, fail_on_warning=False) + + assert bundle["summary"]["status"] == "failed" + assert bundle["summary"]["failed"] == 1 + assert bundle["summary"]["privateGoldenFixturesStatus"] == "missing" + assert bundle["summary"]["privateGoldenFixturesReady"] is False + assert bundle["summary"]["privateGoldenFixturesRequired"] is True + assert "Run private golden fixture validation until status is ready." in bundle["summary"]["requiredActions"] + + +def test_summarize_promotes_fixture_inventory_gaps_to_release_warning() -> None: + bundle: dict[str, Any] = { + "gates": [{"name": "metadata_only", "status": "skipped"}], + "windowsExcelSmoke": {"status": "present", "requiredActions": []}, + "fixtureInventoryAudit": { + "status": "warning", + "requiredActions": ["Add or document real workbook fixture coverage for: charts, macros, drawings."], + }, + "parserComparisonCorpus": passed_parser_corpus(), + "summary": {}, + } + + trust.summarize(bundle, fail_on_warning=True) + + assert bundle["summary"]["status"] == "warning" + assert bundle["summary"]["fixtureInventoryAuditReady"] is False + assert "charts, macros, drawings" in bundle["summary"]["requiredActions"][0] + + +def test_summarize_does_not_mark_skipped_smoke_artifact_validation_ready() -> None: + bundle: dict[str, Any] = { + "gates": [{"name": "metadata_only", "status": "skipped"}], + "windowsExcelSmoke": {"status": "present", "requiredActions": []}, + "smokeArtifactValidation": {"status": "skipped", "requiredActions": []}, + "fixtureInventoryAudit": passed_fixture_inventory(), + "parserComparisonCorpus": passed_parser_corpus(), + "summary": {}, + } + + trust.summarize(bundle, fail_on_warning=False) + + assert bundle["summary"]["status"] == "passed" + assert bundle["summary"]["warning"] == 1 + assert bundle["summary"]["smokeArtifactValidationReady"] is False + + +def test_summarize_fails_warning_mode_when_smoke_artifact_validation_is_skipped() -> None: + bundle: dict[str, Any] = { + "gates": [{"name": "metadata_only", "status": "skipped"}], + "windowsExcelSmoke": {"status": "present", "requiredActions": []}, + "smokeArtifactValidation": {"status": "skipped", "requiredActions": ["Run strict validation."]}, + "fixtureInventoryAudit": passed_fixture_inventory(), + "parserComparisonCorpus": passed_parser_corpus(), + "summary": {}, + } + + trust.summarize(bundle, fail_on_warning=True) + + assert bundle["summary"]["status"] == "warning" + assert bundle["summary"]["smokeArtifactValidationReady"] is False + assert "Run strict validation." in bundle["summary"]["requiredActions"] + + +def test_parser_comparison_corpus_snapshot_surfaces_worker_readiness(tmp_path: Path, monkeypatch) -> None: + fixture_dir = parser_fixture_dir(tmp_path, 31) + fixture_set = trust.parser_fixture_set_snapshot(fixture_dir) + corpus = tmp_path / "parser-comparison-corpus-latest.json" + corpus.write_text( + json.dumps( + { + "releaseEvidence": { + "generatedAt": "2026-06-15T00:00:00Z", + "fixtureCount": 31, + "fixtureSetDigest": fixture_set["fixtureSetDigest"], + "minimumReleaseFixtures": 30, + }, + "result": { + "status": "matched", + "fixtureCount": 31, + "summary": {"matched": 31, "mismatched": 0, "failed": 0, "materialMismatchCount": 0}, + "readiness": { + "status": "comparison_corpus_matched", + "comparisonCorpusReady": True, + "parityEvidenceReady": True, + "replacementReady": False, + "replacementCandidateCount": 0, + "blockers": [{"id": "comparison-adapter-not-replacement"}], + "nextEvidenceNeeded": [ + "Promote a non-comparison-only parser adapter before enabling parser replacement." + ], + }, + }, + }, + indent=2, + ), + encoding="utf-8", + ) + monkeypatch.setattr(trust, "PARSER_COMPARISON_CORPUS", corpus) + monkeypatch.setattr(trust, "PARSER_COMPARISON_FIXTURE_DIR", fixture_dir) + + snapshot = trust.parser_comparison_corpus_snapshot() + + assert snapshot["status"] == "matched" + assert snapshot["readinessStatus"] == "comparison_corpus_matched" + assert snapshot["comparisonCorpusReady"] is True + assert snapshot["parityEvidenceReady"] is True + assert snapshot["currentFixtureCount"] == 31 + assert snapshot["failureReasons"] == [] + assert snapshot["replacementReady"] is False + assert snapshot["replacementCandidateCount"] == 0 + assert snapshot["blockers"][0]["id"] == "comparison-adapter-not-replacement" + assert "non-comparison-only" in snapshot["nextEvidenceNeeded"][0] + + +def test_parser_comparison_corpus_snapshot_rejects_stale_fixture_set(tmp_path: Path, monkeypatch) -> None: + fixture_dir = parser_fixture_dir(tmp_path, 31) + corpus = tmp_path / "parser-comparison-corpus-latest.json" + corpus.write_text( + """{ + "releaseEvidence": { + "generatedAt": "2026-06-15T00:00:00Z", + "fixtureCount": 31, + "fixtureSetDigest": "old-digest", + "minimumReleaseFixtures": 30 + }, + "result": { + "status": "matched", + "fixtureCount": 31, + "summary": {"matched": 31, "mismatched": 0, "failed": 0, "materialMismatchCount": 0}, + "readiness": { + "status": "comparison_corpus_matched", + "comparisonCorpusReady": true, + "parityEvidenceReady": true, + "replacementReady": false, + "replacementCandidateCount": 0, + "blockers": [{"id": "comparison-adapter-not-replacement"}], + "nextEvidenceNeeded": ["Promote a non-comparison-only parser adapter before enabling parser replacement."] + } + } +} +""", + encoding="utf-8", + ) + monkeypatch.setattr(trust, "PARSER_COMPARISON_CORPUS", corpus) + monkeypatch.setattr(trust, "PARSER_COMPARISON_FIXTURE_DIR", fixture_dir) + + snapshot = trust.parser_comparison_corpus_snapshot() + + assert snapshot["status"] == "stale" + assert snapshot["comparisonCorpusReady"] is False + assert "fixture_set_changed_since_corpus" in snapshot["failureReasons"] + assert any("current generated fixture set" in action for action in snapshot["requiredActions"]) + + +def test_release_bundle_records_strict_smoke_artifact_validation(tmp_path: Path) -> None: + validation_json = tmp_path / "windows-excel-smoke-validation.json" + transcript = tmp_path / "verify-full-stack.log" + evidence = tmp_path / "taskpane.png" + evidence.write_bytes(b"captured taskpane pixels\n") + validation_json.write_text( + json.dumps( + { + "status": "present", + "ready": True, + "requireExistingArtifacts": True, + "artifactValidation": { + "missing": [], + "inventory": [ + { + "field": "Screenshot/video directory", + "kind": "run_metadata", + "reference": str(evidence), + "path": str(evidence), + "sizeBytes": evidence.stat().st_size, + "sha256": "30b366d0b4d6c2c5b4f12a5f4a1a3a28f97f0dc8f093ec38cb6d8d3759db3af8", + "insideEvidenceBundle": True, + } + ], + "inventoryCount": 1, + "localDigestCount": 1, + }, + "requiredActions": [], + }, + indent=2, + ), + encoding="utf-8", + ) + transcript.write_text("release evidence transcript\n", encoding="utf-8") + + snapshot = trust.smoke_artifact_validation_snapshot(validation_json, transcript_path=transcript) + bundle = { + "git": {"branch": "test", "head": "a1b2c3d"}, + "gates": [{"name": "metadata_only", "status": "skipped", "durationSeconds": 0}], + "windowsExcelSmoke": { + "status": "present", + "requiredActions": [], + "runMetadata": {"overallResult": "pass", "completeRequiredFields": len(trust.SMOKE_REQUIRED_METADATA_FIELDS)}, + "checklistCoverage": { + "coveredRows": 1, + "totalRows": 1, + "missingStatusRows": 0, + "missingEvidenceRows": 0, + "failedRows": 0, + "notCoveredRows": 0, + }, + "blockers": {"count": 0}, + }, + "smokeArtifactValidation": snapshot, + "fixtureInventoryAudit": passed_fixture_inventory(), + "parserComparisonCorpus": passed_parser_corpus(fixture_count=1), + "privateGoldenFixtures": passed_private_fixtures(), + "summary": {}, + "bundleId": "bundle", + "generatedAt": "2026-06-15T00:00:00Z", + } + + trust.summarize(bundle, fail_on_warning=True) + markdown = trust.markdown_summary(bundle) + + assert snapshot["status"] == "present" + assert snapshot["ready"] is True + assert snapshot["transcriptPath"] + assert snapshot["artifactInventoryCount"] == 1 + assert snapshot["localArtifactDigestCount"] == 1 + assert bundle["summary"]["status"] == "passed" + assert bundle["summary"]["smokeArtifactValidationReady"] is True + assert bundle["summary"]["fixtureInventoryAuditReady"] is True + assert bundle["summary"]["parserParityEvidenceReady"] is True + assert bundle["summary"]["parserReplacementReady"] is False + assert bundle["summary"]["privateGoldenFixturesReady"] is True + assert "## Smoke Artifact Validation" in markdown + assert "## Fixture Inventory Audit" in markdown + assert "## Private Golden Fixtures" in markdown + assert "| Parity evidence ready | `True` |" in markdown + assert "| Replacement ready | `False` |" in markdown + assert "| Fixture count | 2 |" in markdown + assert "windows-excel-smoke-validation.json" in markdown + assert "verify-full-stack.log" in markdown + assert "| Artifact inventory count | 1 |" in markdown + assert "| Local artifact digests | 1 |" in markdown + + +def test_release_bundle_records_private_golden_fixture_policy_snapshot_without_leaking_details( + tmp_path: Path, + monkeypatch, +) -> None: + calls: list[bool] = [] + private_snapshot = { + "schemaVersion": "2026-08-private-golden-fixtures-v2", + "status": "missing", + "ready": False, + "required": True, + "fixtureCount": 0, + "issueCodes": [], + "requiredActions": ["Run private golden fixture validation until status is ready."], + } + + def fake_private_snapshot(*, require_private_fixtures: bool) -> dict[str, object]: + calls.append(require_private_fixtures) + return private_snapshot + + monkeypatch.setattr(trust, "environment_snapshot", lambda: {"platform": "test"}) + monkeypatch.setattr(trust, "git_snapshot", lambda: {"branch": "test", "head": "a1b2c3d", "statusShort": ""}) + monkeypatch.setattr( + trust, + "windows_excel_smoke_snapshot", + lambda _path=None: { + "status": "present", + "requiredActions": [], + "runMetadata": {"overallResult": "pass", "completeRequiredFields": len(trust.SMOKE_REQUIRED_METADATA_FIELDS)}, + "checklistCoverage": { + "coveredRows": 1, + "totalRows": 1, + "missingStatusRows": 0, + "missingEvidenceRows": 0, + "failedRows": 0, + "notCoveredRows": 0, + }, + "blockers": {"count": 0}, + }, + ) + monkeypatch.setattr(trust, "smoke_artifact_validation_snapshot", lambda _path, transcript_path=None: {"status": "present", "ready": True, "requiredActions": []}) + monkeypatch.setattr(trust, "fixture_inventory_audit_snapshot", passed_fixture_inventory) + monkeypatch.setattr(trust, "parser_comparison_corpus_snapshot", passed_parser_corpus) + monkeypatch.setattr(trust, "private_golden_fixture_policy_snapshot", fake_private_snapshot) + monkeypatch.setattr( + sys, + "argv", + [ + "generate_release_trust_bundle.py", + "--mode", + "metadata-only", + "--output-dir", + str(tmp_path), + "--label", + "private-fixture-test", + "--require-private-fixtures", + ], + ) + + result = trust.main() + + bundle_path = next(tmp_path.rglob("release-trust-bundle.json")) + markdown_path = next(tmp_path.rglob("release-trust-bundle.md")) + bundle_text = bundle_path.read_text(encoding="utf-8") + markdown = markdown_path.read_text(encoding="utf-8") + bundle = json.loads(bundle_text) + + assert result == 1 + assert calls == [True] + assert bundle["privateGoldenFixtures"] == private_snapshot + assert bundle["summary"]["privateGoldenFixturesRequired"] is True + assert bundle["summary"]["status"] == "failed" + assert "## Private Golden Fixtures" in markdown + assert "| Required | `True` |" in markdown + assert "workbookPath" not in bundle_text + assert "manifest.json" not in bundle_text + assert "Secret Customers" not in bundle_text + assert "raw formula" not in bundle_text + + +def test_release_bundle_rejects_status_only_smoke_artifact_validation(tmp_path: Path) -> None: + validation_json = tmp_path / "windows-excel-smoke-validation.json" + validation_json.write_text( + """{ + "status": "present", + "ready": true, + "requireExistingArtifacts": true, + "artifactValidation": {"missing": []}, + "requiredActions": [] +} +""", + encoding="utf-8", + ) + + snapshot = trust.smoke_artifact_validation_snapshot(validation_json) + + assert snapshot["status"] == "invalid" + assert snapshot["ready"] is False + assert snapshot["artifactInventoryCount"] == 0 + assert snapshot["localArtifactDigestCount"] == 0 + assert any("non-empty local artifact digest inventory" in action for action in snapshot["requiredActions"]) + + +def test_release_bundle_rejects_outside_bundle_smoke_artifacts(tmp_path: Path) -> None: + validation_json = tmp_path / "windows-excel-smoke-validation.json" + outside_artifact = tmp_path / "outside-taskpane.png" + outside_artifact.write_bytes(b"captured pixels\n") + validation_json.write_text( + json.dumps( + { + "status": "present", + "ready": True, + "requireExistingArtifacts": True, + "artifactValidation": { + "missing": [], + "inventory": [ + { + "field": "Sideload And Link: Open the taskpane", + "kind": "checklist_evidence", + "reference": str(outside_artifact), + "path": str(outside_artifact), + "sizeBytes": outside_artifact.stat().st_size, + "sha256": "fe75f9db1c0ee7d81a337de5611c07f64d7adad956d75c2bbfbac3f7662248f5", + "insideEvidenceBundle": False, + } + ], + "inventoryCount": 1, + "localDigestCount": 1, + }, + "requiredActions": [], + }, + indent=2, + ), + encoding="utf-8", + ) + + snapshot = trust.smoke_artifact_validation_snapshot(validation_json) + + assert snapshot["status"] == "invalid" + assert snapshot["ready"] is False + assert snapshot["artifactInventoryCount"] == 1 + assert snapshot["localArtifactDigestCount"] == 1 + assert snapshot["artifactOutsideBundleCount"] == 1 + assert any("under Evidence bundle path" in action for action in snapshot["requiredActions"]) + + +def test_v7_readiness_snapshot_surfaces_release_blockers(tmp_path: Path) -> None: + readiness_json = tmp_path / "v7-readiness.json" + readiness_json.write_text( + json.dumps( + { + "status": "blocked", + "defaultAllowed": False, + "optInAllowed": False, + "score": 0.72, + "blockers": ["Production replay evidence is required for release/default-switch readiness."], + "warnings": ["No anonymized local replay artifacts have been captured yet."], + "baselineStatus": { + "baselineId": "replaybase_v7_local_seed_v1", + "productionEvidence": False, + "evidenceClass": "deterministic_plumbing_seed", + "evidenceTier": "deterministic_seed", + }, + "dependencyRisk": { + "hyperformulaLicenseStatus": { + "status": "blocked_unapproved", + "releaseBlocker": True, + "approvalEvidence": False, + "keyEvidence": False, + "unresolved": [ + "hyperformula_license_approval_missing", + "hyperformula_license_key_missing", + ], + } + }, + "metadata": { + "defaultSwitchBlockers": [ + "Replay baseline is deterministic plumbing evidence; capture production-like anonymized local replays before switching V7 on by default." + ] + }, + "gates": {"localReplay": {"status": "passed"}}, + }, + indent=2, + ), + encoding="utf-8", + ) + + snapshot = trust.v7_readiness_snapshot(readiness_json) + + assert snapshot["status"] == "blocked" + assert snapshot["ready"] is False + assert snapshot["defaultAllowed"] is False + assert snapshot["productionReplayEvidence"] is False + assert snapshot["replayEvidenceClass"] == "deterministic_plumbing_seed" + assert snapshot["hyperformulaLicenseStatus"]["status"] == "blocked_unapproved" + assert snapshot["hyperformulaLicenseStatus"]["releaseBlocker"] is True + assert "HYPERFORMULA_LICENSE_APPROVED" in " ".join(snapshot["requiredActions"]) + assert "HYPERFORMULA_LICENSE_KEY" in " ".join(snapshot["requiredActions"]) + + +def test_summarize_promotes_blocked_v7_readiness_to_release_failure(tmp_path: Path) -> None: + v7_snapshot = { + "status": "blocked", + "ready": False, + "path": "data/release-trust/v7-readiness.json", + "defaultAllowed": False, + "optInAllowed": False, + "score": 0.72, + "blockerCount": 1, + "warningCount": 0, + "blockers": ["Production replay evidence is required for release/default-switch readiness."], + "warnings": [], + "defaultSwitchBlockers": ["Replay baseline is deterministic plumbing evidence."], + "productionReplayEvidence": False, + "replayEvidenceClass": "deterministic_plumbing_seed", + "replayEvidenceTier": "deterministic_seed", + "replayBaselineId": "replaybase_v7_local_seed_v1", + "hyperformulaLicenseStatus": {"status": "approved", "releaseBlocker": False}, + "gates": {"localReplay": {"status": "passed"}}, + "requiredActions": ["Capture production-like anonymized replay evidence before release/default-switch readiness."], + } + bundle: dict[str, Any] = { + "bundleId": "bundle", + "generatedAt": "2026-06-15T00:00:00Z", + "git": {"branch": "test", "head": "a1b2c3d", "statusShort": ""}, + "gates": [{"name": "metadata_only", "status": "skipped", "durationSeconds": 0}], + "windowsExcelSmoke": { + "status": "present", + "requiredActions": [], + "runMetadata": {"overallResult": "pass", "completeRequiredFields": len(trust.SMOKE_REQUIRED_METADATA_FIELDS)}, + "checklistCoverage": { + "coveredRows": 1, + "totalRows": 1, + "missingStatusRows": 0, + "missingEvidenceRows": 0, + "failedRows": 0, + "notCoveredRows": 0, + }, + "blockers": {"count": 0}, + }, + "smokeArtifactValidation": {"status": "present", "ready": True, "requiredActions": []}, + "v7DefaultReadiness": v7_snapshot, + "fixtureInventoryAudit": passed_fixture_inventory(), + "parserComparisonCorpus": passed_parser_corpus(fixture_count=1), + "privateGoldenFixtures": passed_private_fixtures(), + "summary": {}, + } + + trust.summarize(bundle, fail_on_warning=True) + markdown = trust.markdown_summary(bundle) + + assert bundle["summary"]["status"] == "failed" + assert bundle["summary"]["v7DefaultReadinessStatus"] == "blocked" + assert bundle["summary"]["v7DefaultAllowed"] is False + assert bundle["summary"]["v7ProductionReplayEvidence"] is False + assert "production-like anonymized replay evidence" in " ".join(bundle["summary"]["requiredActions"]) + assert "## V7 Default Readiness" in markdown + assert "| Replay evidence class | `deterministic_plumbing_seed` |" in markdown + assert "| HyperFormula license | `approved` |" in markdown + + +def test_windows_excel_smoke_validator_reports_incomplete_template(tmp_path: Path) -> None: + smoke_doc = tmp_path / "WINDOWS_EXCEL_MANUAL_SMOKE.md" + smoke_doc.write_text( + """# Windows Excel Manual Smoke Checklist + +## Run Metadata + +| Field | Value | +| --- | --- | +| Date | | +| Tester | | +| Overall result | Pass / Pass with Warning / Blocked / Fail | + +## Automated Preflight + +| Gate | Command | Result | Evidence / Notes | +| --- | --- | --- | --- | +| Full stack | `cmd` | | | +""", + encoding="utf-8", + ) + + report = smoke_validator.validate_smoke_doc(smoke_doc) + + assert report["ready"] is False + assert report["status"] == "missing_evidence" + assert report["requiredActions"] + + +def test_windows_excel_smoke_validator_checks_core_artifact_paths(tmp_path: Path) -> None: + bundle_dir = tmp_path / "evidence-bundle" + screenshots_dir = bundle_dir / "screens" + excel_log = bundle_dir / "excel.log" + smoke_doc = tmp_path / "WINDOWS_EXCEL_MANUAL_SMOKE.md" + smoke_doc.write_text( + complete_smoke_doc() + .replace("evidence/", "evidence-bundle/evidence/") + .replace("| Evidence bundle path | captured |", f"| Evidence bundle path | {bundle_dir} |") + .replace("| Screenshot/video directory | captured |", f"| Screenshot/video directory | {screenshots_dir} |") + .replace("| Excel prompt/log evidence | captured |", f"| Excel prompt/log evidence | {excel_log} |"), + encoding="utf-8", + ) + + missing_report = smoke_validator.validate_smoke_doc(smoke_doc, require_existing_artifacts=True) + + assert missing_report["ready"] is False + assert missing_report["status"] == "incomplete" + assert { + "Evidence bundle path", + "Screenshot/video directory", + "Excel prompt/log evidence", + }.issubset({gap["field"] for gap in missing_report["artifactValidation"]["missing"]}) + assert any(gap["kind"] == "checklist_evidence" for gap in missing_report["artifactValidation"]["missing"]) + + screenshots_dir.mkdir(parents=True) + (screenshots_dir / "taskpane.png").write_text("captured screenshot\n", encoding="utf-8") + excel_log.write_text("captured Excel prompt/log evidence\n", encoding="utf-8") + write_complete_smoke_artifacts(bundle_dir) + + ready_report = smoke_validator.validate_smoke_doc(smoke_doc, require_existing_artifacts=True) + + assert ready_report["ready"] is True + assert ready_report["status"] == "present" + assert ready_report["artifactValidation"]["missing"] == [] + assert ready_report["artifactValidation"]["outsideEvidenceBundle"] == [] + assert ready_report["artifactValidation"]["inventoryCount"] > 0 + assert ready_report["artifactValidation"]["localDigestCount"] > 0 + assert any(item.get("sha256") and item.get("sizeBytes", 0) > 0 for item in ready_report["artifactValidation"]["inventory"]) + + +def test_windows_excel_smoke_validator_rejects_local_artifacts_outside_evidence_bundle(tmp_path: Path) -> None: + bundle_dir = tmp_path / "evidence-bundle" + screenshots_dir = bundle_dir / "screens" + excel_log = bundle_dir / "excel.log" + smoke_doc = tmp_path / "WINDOWS_EXCEL_MANUAL_SMOKE.md" + smoke_doc.write_text( + complete_smoke_doc() + .replace("| Evidence bundle path | captured |", f"| Evidence bundle path | {bundle_dir} |") + .replace("| Screenshot/video directory | captured |", f"| Screenshot/video directory | {screenshots_dir} |") + .replace("| Excel prompt/log evidence | captured |", f"| Excel prompt/log evidence | {excel_log} |"), + encoding="utf-8", + ) + screenshots_dir.mkdir(parents=True) + (screenshots_dir / "taskpane.png").write_text("captured screenshot\n", encoding="utf-8") + excel_log.write_text("captured Excel prompt/log evidence\n", encoding="utf-8") + write_complete_smoke_artifacts(tmp_path) + + report = smoke_validator.validate_smoke_doc(smoke_doc, require_existing_artifacts=True) + + assert report["ready"] is False + assert report["status"] == "incomplete" + assert report["artifactValidation"]["missing"] == [] + assert report["artifactValidation"]["outsideEvidenceBundleCount"] > 0 + assert any("under Evidence bundle path" in action for action in report["requiredActions"]) + + +def test_windows_excel_smoke_validator_cli_runs_from_script_path(tmp_path: Path) -> None: + smoke_doc = tmp_path / "WINDOWS_EXCEL_MANUAL_SMOKE.md" + smoke_doc.write_text(complete_smoke_doc(), encoding="utf-8") + + proc = subprocess.run( + [sys.executable, "scripts/validate_windows_excel_smoke.py", "--smoke-doc", str(smoke_doc)], + cwd=trust.ROOT, + text=True, + stdout=subprocess.PIPE, + stderr=subprocess.PIPE, + check=False, + ) + + assert proc.returncode == 0 + assert "Ready for release evidence: yes" in proc.stdout + assert proc.stderr == "" + + +def test_windows_excel_smoke_scaffold_creates_run_pack_without_faking_manual_pass(tmp_path: Path) -> None: + report = smoke_scaffold.scaffold_smoke_run( + output_dir=tmp_path, + label="smoke", + template=trust.SMOKE_DOC, + tester="Analyst", + excel_version="Microsoft Excel 2405 Build 17628", + ) + smoke_doc = tmp_path / Path(report["runPack"]).name / "WINDOWS_EXCEL_MANUAL_SMOKE.md" + manifest = tmp_path / Path(report["runPack"]).name / "evidence-slot-manifest.json" + slot_index = tmp_path / Path(report["runPack"]).name / "evidence-slot-index.md" + + assert smoke_doc.exists() + assert manifest.exists() + assert slot_index.exists() + assert report["validation"]["ready"] is False + assert report["validation"]["status"] in {"incomplete", "missing_evidence"} + + validation = smoke_validator.validate_smoke_doc(smoke_doc, require_existing_artifacts=True) + assert validation["ready"] is False + assert validation["snapshot"]["runMetadata"]["fields"]["Tester"] == "Analyst" + assert validation["snapshot"]["runMetadata"]["fields"]["Excel version/build"] == "Microsoft Excel 2405 Build 17628" + assert validation["snapshot"]["checklistCoverage"]["missingStatusRows"] > 0 + assert "Overall result" in validation["snapshot"]["runMetadata"]["missingRequiredFields"] + metadata_gaps = { + gap["field"] + for gap in validation["artifactValidation"]["missing"] + if gap["kind"] == "run_metadata" + } + assert "Evidence bundle path" not in metadata_gaps + assert "Screenshot/video directory" not in metadata_gaps + assert "Excel prompt/log evidence" in metadata_gaps + + manifest_payload = json.loads(manifest.read_text(encoding="utf-8")) + assert manifest_payload["slotCount"] == validation["snapshot"]["checklistCoverage"]["totalRows"] + assert manifest_payload["manualRequirement"].startswith("Attach real Windows Excel Desktop evidence") + first_slot = manifest_payload["slots"][0] + first_slot_dir = tmp_path / Path(report["runPack"]).name / first_slot["expectedAttachmentSlot"] + assert first_slot_dir.is_dir() + assert (first_slot_dir / "README.md").exists() + assert manifest_payload["slotIndex"] == report["evidenceSlotIndex"] + assert first_slot["slotId"] in slot_index.read_text(encoding="utf-8") + + +def write_complete_smoke_artifacts(base_dir: Path) -> None: + for relative in [ + "evidence/preflight.log", + "evidence/env.png", + "evidence/simple.mp4", + "evidence/sideload.png", + "evidence/sync.mp4", + "evidence/drift.mp4", + "evidence/protected.png", + "evidence/export.xlsx", + ]: + path = base_dir / relative + path.parent.mkdir(parents=True, exist_ok=True) + path.write_text("captured evidence\n", encoding="utf-8") + + +def passed_fixture_inventory() -> dict[str, object]: + return { + "status": "passed", + "fixtureCount": 31, + "checkedFixtureCount": 31, + "inventoryStatus": "passed", + "coveredFeatureCount": 11, + "requiredFeatureCount": 11, + "coveredFeatures": [ + "charts", + "tables", + "pivots", + "comments", + "macros", + "validations", + "conditionalFormats", + "protections", + "freezePanes", + "definedNames", + "drawings", + ], + "missingFeatures": [], + "requiredActions": [], + } + + +def passed_parser_corpus(*, fixture_count: int = 31) -> dict[str, object]: + return { + "status": "matched", + "fixtureCount": fixture_count, + "matched": fixture_count, + "mismatched": 0, + "failed": 0, + "materialMismatchCount": 0, + "readinessStatus": "comparison_corpus_matched", + "comparisonCorpusReady": True, + "parityEvidenceReady": True, + "replacementReady": False, + "replacementCandidateCount": 0, + "blockers": [{"id": "comparison-adapter-not-replacement"}], + "nextEvidenceNeeded": ["Promote a non-comparison-only parser adapter before enabling parser replacement."], + } + + +def passed_private_fixtures() -> dict[str, object]: + return { + "status": "ready", + "ready": True, + "required": False, + "fixtureCount": 2, + "issueCodes": [], + "requiredActions": [], + } + + +def parser_fixture_dir(tmp_path: Path, count: int) -> Path: + fixture_dir = tmp_path / "fixtures" + fixture_dir.mkdir() + for index in range(count): + (fixture_dir / f"fixture-{index:02d}.xlsx").write_bytes(f"fixture-{index}".encode("utf-8")) + return fixture_dir + + +def complete_smoke_doc() -> str: + metadata_values = { + "Date": "2026-06-15", + "Tester": "Analyst", + "Git commit": "a1b2c3d", + "Windows version": "Windows 11 23H2", + "Excel version/build": "Microsoft Excel 2405 Build 17628", + "Add-in manifest": "plugin/manifest.xml", + "Backend command": "python -m uvicorn backend.main:app", + "Plugin command": "npm run dev-http -- --host 127.0.0.1", + "Worker command": "npm --prefix workbook-worker test", + "Test fixture set": "generated workbook fixtures", + "Evidence bundle path": "captured", + "Screenshot/video directory": "captured", + "Excel prompt/log evidence": "captured", + "Preflight trust bundle ID": "windows-excel-preflight-20260615T190539Z", + "Release owner signoff": "Analyst", + "Overall result": "Pass", + } + metadata_rows = "\n".join(f"| {field} | {metadata_values[field]} |" for field in trust.SMOKE_REQUIRED_METADATA_FIELDS) + return f"""# Windows Excel Manual Smoke Checklist + +## Run Metadata + +| Field | Value | +| --- | --- | +{metadata_rows} + +## Automated Preflight + +| Gate | Command | Result | Evidence / Notes | +| --- | --- | --- | --- | +| Full stack | `cmd` | Pass | evidence/preflight.log | + +## Required Evidence Coverage + +| ID | Evidence Area | Required Capture | Status | Evidence / Notes | +| --- | --- | --- | --- | --- | +| EXCEL-ENV | Excel Desktop environment | Metadata captured | Pass | evidence/env.png | + +## Fixture Matrix + +| Fixture | Contains | Open | Link | Edit | Sync | Export | Reopen | Save/Reopen | Result | Evidence | +| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | +| Simple `.xlsx` | Values/formulas/styles | Pass | Pass | Pass | Pass | Pass | Pass | Pass | Pass | evidence/simple.mp4 | + +## Sideload And Link + +| Step | Expected Result | Status | Evidence / Notes | +| --- | --- | --- | --- | +| Run `npm run sideload` from `plugin` | Add-in sideloads | Pass | evidence/sideload.png | + +## Browser Edit To Excel Sync + +| Patch Type | Action | Expected Result | Status | Evidence / Notes | +| --- | --- | --- | --- | --- | +| Value | Edit one value | Excel receives value | Pass | evidence/sync.mp4 | + +## Host Drift And Reconcile + +| Step | Expected Result | Status | Evidence / Notes | +| --- | --- | --- | --- | +| Make an external Excel edit | Browser detects host drift | Pass | evidence/drift.mp4 | + +## Failure And Recovery + +| Step | Expected Result | Status | Evidence / Notes | +| --- | --- | --- | --- | +| Attempt write to protected sheet | Sync pauses | Blocked | evidence/protected.png | + +## Export Fidelity + +| Step | Expected Result | Status | Evidence / Notes | +| --- | --- | --- | --- | +| Export preserved workbook | Primary export uses preserved OOXML | Pass | evidence/export.xlsx | + +## Blockers And Follow-Up + +| ID | Severity | Area | Description | Owner | Target Fix | +| --- | --- | --- | --- | --- | --- | +| | | | | | | +""" diff --git a/backend/tests/test_rollback_apply_boundary.py b/backend/tests/test_rollback_apply_boundary.py new file mode 100644 index 0000000..5b02695 --- /dev/null +++ b/backend/tests/test_rollback_apply_boundary.py @@ -0,0 +1,1289 @@ +from __future__ import annotations + +import threading +import uuid +from concurrent.futures import ThreadPoolExecutor + +import pytest +from fastapi.testclient import TestClient + +from backend.api.routes.tasks import _verify_apply_result +from backend.api.routes import sessions, tasks +from backend.config import API_KEY +from backend.main import app +from backend.models.operations import OperationPlan, OperationPlanV2 +from backend.services.operation_validation import validate_workbook_operations_v2 +from backend.session_store import SessionStore, get_session_store + + +def _headers() -> dict[str, str]: + return {"X-API-Key": API_KEY} + + +def _write_plan() -> OperationPlanV2: + return OperationPlanV2.model_validate( + { + "session_id": "apply-result-boundary", + "prompt": "Write a value", + "operations": [ + { + "id": "op-write", + "type": "write_cell", + "sheet": "Sheet1", + "range": "A1", + "value": 42, + } + ], + } + ) + + +def _rollback_context(value: object, *, sheets: list[str] | None = None) -> dict: + sheet_names = sheets or ["Sheet1"] + return { + "active_sheet": sheet_names[0] if sheet_names else "Sheet1", + "sheet_names": sheet_names, + "sheet_metadata": [{"name": name, "usedRange": f"{name}!A1"} for name in sheet_names], + "cell_snapshot": { + "Sheet1!A1": { + "sheet": "Sheet1", + "address": "A1", + "value": value, + "formula": None, + "format": "General", + "blank": value is None or value == "", + } + }, + "objects": {"charts": [], "tables": [], "pivots": [], "comments": []}, + "rollback_snapshot": {"complete": True, "errors": []}, + } + + +def _host_proof(fingerprint: str) -> dict: + return { + "schemaVersion": "2026-08-host-snapshot-v2", + "complete": True, + "truncated": False, + "errors": [], + "fingerprint": fingerprint, + } + + +def _authorize_rollback( + client: TestClient, + session_id: str, + rollback: dict, + *, + operations: list[dict] | None = None, + registry_hash: str | None = None, +): + return client.post( + f"/api/v1/sessions/{session_id}/rollback/authorize", + headers=_headers(), + json={ + "snapshot_id": rollback["snapshot_id"], + "rollback_token": rollback["rollback_token"], + "rollback_attempt_id": rollback["rollback_attempt_id"], + "rollback_operation_registry_hash": registry_hash + or rollback["rollback_operation_registry_hash"], + "operations": operations if operations is not None else rollback["operations"], + }, + ) + + +def _rollback_capability_fields(authorization: dict) -> dict: + return { + "rollback_capability_id": authorization["rollback_capability_id"], + "rollback_capability_nonce": authorization["rollback_capability_nonce"], + "rollback_operation_registry_hash": authorization["rollback_operation_registry_hash"], + } + + +def _run_two_requests_after_same_plan_read( + monkeypatch: pytest.MonkeyPatch, + store: SessionStore, + plan_id: str, + request_call, +): + """Force two route calls to validate the same durable plan revision.""" + original_get_operation_plan = store.get_operation_plan + read_barrier = threading.Barrier(2) + counter_lock = threading.Lock() + synchronized_reads = 0 + + def synchronized_get_operation_plan(requested_plan_id: str): + nonlocal synchronized_reads + plan = original_get_operation_plan(requested_plan_id) + should_wait = False + if requested_plan_id == plan_id: + with counter_lock: + if synchronized_reads < 2: + synchronized_reads += 1 + should_wait = True + if should_wait: + read_barrier.wait(timeout=5) + return plan + + monkeypatch.setattr(store, "get_operation_plan", synchronized_get_operation_plan) + with ThreadPoolExecutor(max_workers=2) as executor: + futures = [executor.submit(request_call) for _ in range(2)] + return [future.result(timeout=10) for future in futures] + + +def _start_cell_rollback(client: TestClient) -> tuple[SessionStore, str, str, str, dict]: + store = get_session_store() + suffix = uuid.uuid4().hex[:10] + session_id = f"rollback-capability-{suffix}" + plan = OperationPlan( + id=f"plan-rollback-capability-{suffix}", + session_id=session_id, + prompt="Fill A1", + status="applied", + operations=[{"id": "op-fill", "type": "write_cell", "sheet": "Sheet1", "range": "A1", "value": 42}], + ) + store.get_or_create_session(session_id) + store.save_operation_plan(plan.model_dump(mode="json", by_alias=True)) + snapshot_id = store.record_snapshot( + session_id=session_id, + plan_id=plan.id, + label="Before fill", + workbook_state={"state": _rollback_context("")}, + ) + requested = client.post( + f"/api/v1/sessions/{session_id}/rollback", + headers=_headers(), + json={"snapshot_id": snapshot_id, "current_workbook_state": _rollback_context(42)}, + ) + assert requested.status_code == 200, requested.text + return store, session_id, plan.id, snapshot_id, requested.json() + + +@pytest.mark.parametrize("office_status", ["warning", "skipped"]) +def test_apply_verification_rejects_non_successful_mutation_results(office_status: str) -> None: + report = _verify_apply_result( + _write_plan(), + { + "applied": 1, + "results": [ + { + "operationId": "op-write", + "type": "write_cell", + "status": office_status, + "dirtyRanges": [{"sheet": "Sheet1", "range": "A1"}], + "rollbackSnapshot": { + "proofClass": "values_formats", + "restoreOperations": [ + { + "type": "rollback_restore", + "sheet": "Sheet1", + "range": "A1", + "values": [[""]], + "numberFormats": [["General"]], + } + ], + }, + } + ], + }, + {}, + "applied", + None, + ) + + assert report["status"] == "failed" + assert any(office_status in warning for warning in report["verification_warnings"]) + + +def test_apply_verification_rejects_manual_rollback_proof_before_success() -> None: + report = _verify_apply_result( + _write_plan(), + { + "applied": 1, + "results": [ + { + "operationId": "op-write", + "type": "write_cell", + "status": "success", + "dirtyRanges": [{"sheet": "Sheet1", "range": "A1"}], + "rollbackSnapshot": { + "proofClass": "manual", + "unsupportedRollbackWarnings": ["Manual restore required"], + }, + } + ], + }, + {}, + "applied", + None, + ) + + assert report["status"] == "failed" + assert any("rollback" in warning.lower() for warning in report["verification_warnings"]) + + +def test_apply_route_requires_complete_exact_host_targets_before_terminal_success() -> None: + client = TestClient(app) + store = get_session_store() + suffix = uuid.uuid4().hex[:10] + session_id = f"apply-proof-{suffix}" + plan = OperationPlan( + id=f"plan-apply-proof-{suffix}", + session_id=session_id, + prompt="Write A1", + operations=[{"id": "op-write", "type": "write_cell", "sheet": "Sheet1", "range": "A1", "value": 42}], + ) + store.get_or_create_session(session_id) + store.save_operation_plan(plan.model_dump(mode="json", by_alias=True)) + pre_context = { + "active_sheet": "Sheet1", + "sheet_names": ["Sheet1"], + "sheet_metadata": [{"name": "Sheet1", "usedRange": "Sheet1!A1"}], + "cell_snapshot": { + "Sheet1!A1": {"sheet": "Sheet1", "address": "A1", "value": "", "formula": None, "blank": True} + }, + "objects": {"charts": [], "tables": [], "pivots": [], "comments": []}, + "rollback_snapshot": {"complete": True, "errors": []}, + } + approval = client.post( + "/api/v1/tasks/apply", + headers=_headers(), + json={ + "plan_id": plan.id, + "session_id": session_id, + "workbook_fingerprint": "pre-fingerprint", + "workbook_state": pre_context, + }, + ) + assert approval.status_code == 200, approval.text + approved = approval.json() + office_result = { + "applied": 1, + "results": [{ + "operationId": "op-write", + "type": "write_cell", + "status": "success", + "dirtyRanges": [{"sheet": "Sheet1", "range": "A1"}], + "rollbackSnapshot": { + "proofClass": "values_formats", + "restoreOperations": [{ + "type": "rollback_restore", "sheet": "Sheet1", "range": "A1", "values": [[""]], "numberFormats": [["General"]], + }], + }, + }], + } + common = { + "plan_id": plan.id, + "session_id": session_id, + "apply_token": approved["apply_token"], + "apply_attempt_id": approved["apply_attempt_id"], + "status": "applied", + "office_apply_result": office_result, + "pre_apply_workbook_state": pre_context, + "pre_apply_workbook_fingerprint": "pre-fingerprint", + "pre_host_snapshot": {"complete": True, "truncated": False, "errors": [], "fingerprint": "pre-host"}, + "post_apply_workbook_fingerprint": "post-fingerprint", + "post_host_snapshot": {"complete": True, "truncated": False, "errors": [], "fingerprint": "post-host"}, + } + unrelated = client.post( + "/api/v1/tasks/apply/result", + headers=_headers(), + json={ + **common, + "post_apply_workbook_state": { + "active_sheet": "Sheet1", + "sheet_names": ["Sheet1"], + "cell_snapshot": {"Sheet1!XFD1048576": {"sheet": "Sheet1", "address": "XFD1048576", "value": 42}}, + "rollback_snapshot": {"complete": True, "errors": []}, + }, + }, + ) + assert unrelated.status_code == 409 + + matching = client.post( + "/api/v1/tasks/apply/result", + headers=_headers(), + json={ + **common, + "post_apply_workbook_state": { + "active_sheet": "Sheet1", + "sheet_names": ["Sheet1"], + "cell_snapshot": { + "Sheet1!A1": {"sheet": "Sheet1", "address": "A1", "value": 42, "formula": None, "blank": False} + }, + "rollback_snapshot": {"complete": True, "errors": []}, + }, + }, + ) + assert matching.status_code == 200, matching.text + verification = matching.json()["verification"] + assert verification["status"] == "verified" + assert verification["proof"]["attempt_id"] == approved["apply_attempt_id"] + assert verification["proof"]["plan_id"] == plan.id + assert verification["proof"]["pre_target_hashes"] + assert verification["proof"]["post_target_hashes"] + + +def test_legacy_final_status_is_rejected_even_when_compatibility_flag_is_enabled(monkeypatch: pytest.MonkeyPatch) -> None: + client = TestClient(app) + store = get_session_store() + suffix = uuid.uuid4().hex[:10] + session_id = f"legacy-final-{suffix}" + plan = OperationPlan( + id=f"plan-legacy-final-{suffix}", + session_id=session_id, + prompt="Write A1", + operations=[{"id": "op-write", "type": "write_cell", "sheet": "Sheet1", "range": "A1", "value": 42}], + ) + store.get_or_create_session(session_id) + store.save_operation_plan(plan.model_dump(mode="json", by_alias=True)) + monkeypatch.setattr(tasks, "ALLOW_LEGACY_APPLY_FINAL_STATUS", True) + + response = client.post( + "/api/v1/tasks/apply", + headers=_headers(), + json={ + "plan_id": plan.id, + "session_id": session_id, + "final_status": "applied", + "office_apply_result": {"applied": 1}, + }, + ) + + assert response.status_code == 400 + assert "cannot prove an Office mutation" in response.json()["detail"] + assert store.get_latest_apply_attempt(plan.id) is None + assert (store.get_operation_plan(plan.id) or {}).get("status") != "applied" + + +@pytest.mark.parametrize( + ("operation", "expected_code"), + [ + ( + { + "id": "op-pivot", + "type": "create_pivot_table", + "sheet": "Sheet1", + "range": "A1:D10", + "name": "UnsafePivot", + }, + "operation_safe_gated", + ), + ( + { + "id": "op-sort", + "type": "sort_range", + "sheet": "Sheet1", + "range": "A1:D10", + }, + "operation_not_apply_ready", + ), + ], +) +def test_backend_validation_enforces_registry_apply_boundary(operation: dict, expected_code: str) -> None: + plan = OperationPlanV2.model_validate( + { + "session_id": "registry-boundary", + "prompt": "Unsafe operation", + "operations": [operation], + } + ) + + report = validate_workbook_operations_v2( + plan.operations, + {"sheet_names": ["Sheet1"], "cell_snapshot": {"Sheet1!A1": {"value": "Header"}}}, + ) + + assert report.status == "failed" + assert any(issue.code == expected_code for issue in report.issues) + + +def test_rollback_stays_pending_until_successful_office_ack_and_restores_explicit_blank() -> None: + client = TestClient(app) + store = get_session_store() + suffix = uuid.uuid4().hex[:10] + session_id = f"rollback-boundary-{suffix}" + plan = OperationPlan( + id=f"plan-rollback-{suffix}", + session_id=session_id, + prompt="Fill A1", + status="applied", + operations=[ + { + "id": "op-fill", + "type": "write_cell", + "sheet": "Sheet1", + "range": "A1", + "value": 42, + } + ], + ) + store.get_or_create_session(session_id) + store.save_operation_plan(plan.model_dump(mode="json", by_alias=True)) + snapshot_id = store.record_snapshot( + session_id=session_id, + plan_id=plan.id, + label="Before fill", + workbook_state={ + "state": { + "context_version": "2026-08-workbook-context-v3", + "cell_snapshot": { + "Sheet1!A1": { + "sheet": "Sheet1", + "address": "A1", + "value": "", + "formula": None, + "format": "General", + "blank": True, + } + }, + "objects": {"charts": [], "tables": [], "pivots": [], "comments": []}, + } + }, + ) + + requested = client.post( + f"/api/v1/sessions/{session_id}/rollback", + headers=_headers(), + json={ + "snapshot_id": snapshot_id, + "current_workbook_state": { + "cell_snapshot": { + "Sheet1!A1": {"value": 42, "formula": None, "format": "General"} + } + }, + }, + ) + + assert requested.status_code == 200 + rollback = requested.json() + assert rollback["status"] == "rollback_pending" + assert rollback["rollback_token"] + assert len(rollback["rollback_operation_registry_hash"]) == 64 + pending_plan = store.get_operation_plan(plan.id) + assert pending_plan is not None + assert pending_plan["status"] == "rollback_pending" + assert rollback["operations"] == [ + { + "id": "rollback_Sheet1_A1", + "type": "rollback_restore", + "sheet": "Sheet1", + "range": "A1", + "values": [[""]], + "formulas": [[""]], + "numberFormats": [["General"]], + "intent": "rollback restore cell", + } + ] + + rejected_ack = client.post( + f"/api/v1/sessions/{session_id}/rollback/result", + headers=_headers(), + json={ + "snapshot_id": snapshot_id, + "rollback_token": rollback["rollback_token"], + "rollback_attempt_id": rollback["rollback_attempt_id"], + "status": "rolled_back", + "office_apply_result": { + "applied": 0, + "results": [ + { + "operationId": "rollback_Sheet1_A1", + "type": "rollback_restore", + "status": "warning", + "dirtyRanges": [], + } + ], + }, + }, + ) + assert rejected_ack.status_code == 409 + rejected_plan = store.get_operation_plan(plan.id) + assert rejected_plan is not None + assert rejected_plan["status"] == "rollback_pending" + + authorized_response = _authorize_rollback(client, session_id, rollback) + assert authorized_response.status_code == 200, authorized_response.text + authorization = authorized_response.json() + assert authorization["status"] == "authorized" + assert authorization["rollback_operation_registry_hash"] == rollback["rollback_operation_registry_hash"] + + accepted_ack = client.post( + f"/api/v1/sessions/{session_id}/rollback/result", + headers=_headers(), + json={ + "snapshot_id": snapshot_id, + "rollback_token": rollback["rollback_token"], + "rollback_attempt_id": rollback["rollback_attempt_id"], + **_rollback_capability_fields(authorization), + "status": "rolled_back", + "office_apply_result": { + "applied": 1, + "results": [ + { + "operationId": "rollback_Sheet1_A1", + "type": "rollback_restore", + "status": "success", + "dirtyRanges": [{"sheet": "Sheet1", "range": "A1"}], + } + ], + }, + "pre_apply_workbook_state": _rollback_context(42), + "pre_apply_workbook_fingerprint": "rollback-pre", + "pre_host_snapshot": _host_proof("rollback-host-pre"), + "post_apply_workbook_state": _rollback_context(""), + "post_apply_workbook_fingerprint": "rollback-post", + "post_host_snapshot": _host_proof("rollback-host-post"), + }, + ) + assert accepted_ack.status_code == 200 + assert accepted_ack.json()["status"] == "rolled_back" + rolled_back_plan = store.get_operation_plan(plan.id) + assert rolled_back_plan is not None + assert rolled_back_plan["status"] == "rolled_back" + + +def test_rollback_rejects_office_success_without_matching_post_target_proof() -> None: + client = TestClient(app) + store = get_session_store() + suffix = uuid.uuid4().hex[:10] + session_id = f"rollback-proof-{suffix}" + plan = OperationPlan( + id=f"plan-rollback-proof-{suffix}", + session_id=session_id, + prompt="Fill A1", + status="applied", + operations=[{"id": "op-fill", "type": "write_cell", "sheet": "Sheet1", "range": "A1", "value": 42}], + ) + store.get_or_create_session(session_id) + store.save_operation_plan(plan.model_dump(mode="json", by_alias=True)) + snapshot_id = store.record_snapshot( + session_id=session_id, + plan_id=plan.id, + label="Before fill", + workbook_state={"state": _rollback_context("")}, + ) + requested = client.post( + f"/api/v1/sessions/{session_id}/rollback", + headers=_headers(), + json={"snapshot_id": snapshot_id, "current_workbook_state": _rollback_context(42)}, + ) + assert requested.status_code == 200, requested.text + rollback = requested.json() + authorized_response = _authorize_rollback(client, session_id, rollback) + assert authorized_response.status_code == 200, authorized_response.text + authorization = authorized_response.json() + + response = client.post( + f"/api/v1/sessions/{session_id}/rollback/result", + headers=_headers(), + json={ + "snapshot_id": snapshot_id, + "rollback_token": rollback["rollback_token"], + "rollback_attempt_id": rollback["rollback_attempt_id"], + **_rollback_capability_fields(authorization), + "status": "rolled_back", + "office_apply_result": { + "applied": 1, + "results": [{ + "operationId": "rollback_Sheet1_A1", + "type": "rollback_restore", + "status": "success", + "dirtyRanges": [{"sheet": "Sheet1", "range": "A1"}], + }], + }, + "pre_apply_workbook_state": _rollback_context(42), + "pre_apply_workbook_fingerprint": "rollback-pre", + "pre_host_snapshot": _host_proof("rollback-host-pre"), + "post_apply_workbook_state": _rollback_context(99), + "post_apply_workbook_fingerprint": "rollback-post", + "post_host_snapshot": _host_proof("rollback-host-post"), + }, + ) + + assert response.status_code == 200, response.text + assert response.json()["status"] == "rollback_failed" + assert "post-apply" in str(response.json()).lower() + failed_plan = store.get_operation_plan(plan.id) or {} + assert failed_plan["status"] == "applied" + assert failed_plan["metadata"]["rollback_capability_used"] is True + + +def test_rollback_proof_rejects_formula_cell_number_format_mismatch() -> None: + operation = { + "id": "rollback_Sheet1_A1", + "type": "rollback_restore", + "sheet": "Sheet1", + "range": "A1", + "values": [[2]], + "formulas": [["=1+1"]], + "numberFormats": [["$#,##0.00"]], + } + post_state = _rollback_context(2) + post_cell = post_state["cell_snapshot"]["Sheet1!A1"] + post_cell["formula"] = "=1+1" + post_cell["format"] = "General" + + verification = sessions._verify_rollback_result( + [operation], + { + "applied": 1, + "results": [{ + "operationId": operation["id"], + "type": operation["type"], + "status": "success", + "dirtyRanges": [{"sheet": "Sheet1", "range": "A1"}], + }], + }, + _rollback_context(42), + "format-proof-pre", + _host_proof("format-proof-host-pre"), + post_state, + "format-proof-post", + _host_proof("format-proof-host-post"), + expected_registry_hash=sessions._rollback_operation_registry_hash([operation]), + rollback_attempt_id="rollback_format_proof", + ) + + assert verification["status"] == "failed" + assert any("number format" in failure for failure in verification["failures"]) + + +def test_rollback_proof_does_not_treat_boolean_as_numeric_value() -> None: + operation = { + "id": "rollback_Sheet1_A1", + "type": "rollback_restore", + "sheet": "Sheet1", + "range": "A1", + "values": [[1]], + "formulas": [[""]], + "numberFormats": [["General"]], + } + + verification = sessions._verify_rollback_result( + [operation], + { + "applied": 1, + "results": [{ + "operationId": operation["id"], + "type": operation["type"], + "status": "success", + "dirtyRanges": [{"sheet": "Sheet1", "range": "A1"}], + }], + }, + _rollback_context(42), + "typed-proof-pre", + _host_proof("typed-proof-host-pre"), + _rollback_context(True), + "typed-proof-post", + _host_proof("typed-proof-host-post"), + expected_registry_hash=sessions._rollback_operation_registry_hash([operation]), + rollback_attempt_id="rollback_typed_proof", + ) + + assert verification["status"] == "failed" + assert any("value matrix" in failure for failure in verification["failures"]) + + +@pytest.mark.parametrize( + ("result_rows", "applied", "failure_fragment"), + [ + ( + [ + { + "operationId": "rollback_Sheet1_A1", + "type": "rollback_restore", + "status": "success", + "dirtyRanges": [{"sheet": "Sheet1", "range": "A1"}], + }, + { + "operationId": "", + "type": "rollback_restore", + "status": "success", + "dirtyRanges": [{"sheet": "Sheet1", "range": "Z99"}], + }, + ], + 1, + "missing operation id", + ), + ( + [ + { + "operationId": "rollback_Sheet1_A1", + "type": "rollback_restore", + "status": "success", + "dirtyRanges": [{"sheet": "Sheet1", "range": "A1"}], + }, + { + "operationId": "rollback_Sheet1_A1", + "type": "rollback_restore", + "status": "success", + "dirtyRanges": [{"sheet": "Sheet1", "range": "A1"}], + }, + ], + 1, + "duplicate operation id", + ), + ([], 0, "returned 0 results"), + ( + [{ + "operationId": "rollback_Sheet1_A1", + "type": "delete_sheet", + "status": "success", + "dirtyRanges": [{"sheet": "Sheet1", "range": "A1"}], + }], + 1, + "result type", + ), + ], +) +def test_rollback_proof_requires_exact_office_result_registry( + result_rows: list[dict], + applied: int, + failure_fragment: str, +) -> None: + operation = { + "id": "rollback_Sheet1_A1", + "type": "rollback_restore", + "sheet": "Sheet1", + "range": "A1", + "values": [[""]], + "formulas": [[""]], + "numberFormats": [["General"]], + } + + verification = sessions._verify_rollback_result( + [operation], + {"applied": applied, "results": result_rows}, + _rollback_context(42), + "registry-proof-pre", + _host_proof("registry-proof-host-pre"), + _rollback_context(""), + "registry-proof-post", + _host_proof("registry-proof-host-post"), + expected_registry_hash=sessions._rollback_operation_registry_hash([operation]), + rollback_attempt_id="rollback_registry_proof", + ) + + assert verification["status"] == "failed" + assert any( + failure_fragment in failure.lower() + for failure in verification["failures"] + ) + + +def test_rollback_proof_rejects_number_format_matrix_with_wrong_shape() -> None: + operation = { + "id": "rollback_Sheet1_A1_B2", + "type": "rollback_restore", + "sheet": "Sheet1", + "range": "A1:B2", + "values": [[1, 2], [3, 4]], + "formulas": [["", ""], ["", ""]], + "numberFormats": [["General", "General", "General", "General"]], + } + context = _rollback_context(1) + context["cell_snapshot"] = { + f"Sheet1!{address}": { + "sheet": "Sheet1", + "address": address, + "value": value, + "formula": None, + "format": "General", + "blank": False, + } + for address, value in (("A1", 1), ("B1", 2), ("A2", 3), ("B2", 4)) + } + + binding = tasks._workbook_target_binding(context, [operation], require_post_match=True) + + assert binding["complete"] is False + assert any("number format matrix dimensions" in error for error in binding["errors"]) + + +def test_create_sheet_inverse_is_persisted_as_bound_rollback_only_delete() -> None: + accepted, errors = tasks._validated_result_restore_operations( + {"id": "op-create-sheet", "type": "create_sheet", "sheet_name": "Scenario"}, + { + "rollbackSnapshot": { + "proofClass": "created_object_pointer", + "restoreOperations": [{"type": "delete_sheet", "sheet": "Scenario", "range": "A1"}], + } + }, + ) + + assert errors == [] + assert accepted == [{ + "type": "delete_sheet", + "sheet": "Scenario", + "range": "A1", + "rollbackOnly": True, + "rollbackStrategy": "delete_created_sheet", + "sourceOperationId": "op-create-sheet", + }] + + +def test_bound_created_sheet_rollback_requires_absence_in_post_reread() -> None: + client = TestClient(app) + store = get_session_store() + suffix = uuid.uuid4().hex[:10] + session_id = f"rollback-created-sheet-{suffix}" + plan = OperationPlan( + id=f"plan-created-sheet-{suffix}", + session_id=session_id, + prompt="Create scenario sheet", + status="applied", + operations=[{"id": "op-create-sheet", "type": "create_sheet", "sheet_name": "Scenario", "range": "A1"}], + metadata={ + "durable_rollback_operations": [{ + "type": "delete_sheet", + "sheet": "Scenario", + "range": "A1", + "rollbackOnly": True, + "rollbackStrategy": "delete_created_sheet", + "sourceOperationId": "op-create-sheet", + }] + }, + ) + store.get_or_create_session(session_id) + store.save_operation_plan(plan.model_dump(mode="json", by_alias=True)) + snapshot_id = store.record_snapshot( + session_id=session_id, + plan_id=plan.id, + label="Before scenario sheet", + workbook_state={"state": _rollback_context("", sheets=["Sheet1"])}, + ) + requested = client.post( + f"/api/v1/sessions/{session_id}/rollback", + headers=_headers(), + json={"snapshot_id": snapshot_id, "current_workbook_state": _rollback_context("", sheets=["Sheet1", "Scenario"])}, + ) + assert requested.status_code == 200, requested.text + rollback = requested.json() + operation = rollback["operations"][0] + assert operation["rollbackOnly"] is True + assert operation["rollbackStrategy"] == "delete_created_sheet" + assert operation["rollbackAttemptId"] == rollback["rollback_attempt_id"] + authorized_response = _authorize_rollback(client, session_id, rollback) + assert authorized_response.status_code == 200, authorized_response.text + authorization = authorized_response.json() + + office_result = { + "applied": 1, + "results": [{ + "operationId": operation["id"], + "type": "delete_sheet", + "status": "success", + "dirtyRanges": [{"sheet": "Scenario", "range": "A1"}], + }], + } + rejected = client.post( + f"/api/v1/sessions/{session_id}/rollback/result", + headers=_headers(), + json={ + "snapshot_id": snapshot_id, + "rollback_token": rollback["rollback_token"], + "rollback_attempt_id": rollback["rollback_attempt_id"], + **_rollback_capability_fields(authorization), + "status": "rolled_back", + "office_apply_result": office_result, + "pre_apply_workbook_state": _rollback_context("", sheets=["Sheet1", "Scenario"]), + "pre_apply_workbook_fingerprint": "rollback-sheet-pre", + "pre_host_snapshot": _host_proof("rollback-sheet-host-pre"), + "post_apply_workbook_state": _rollback_context("", sheets=["Sheet1", "Scenario"]), + "post_apply_workbook_fingerprint": "rollback-sheet-post-stale", + "post_host_snapshot": _host_proof("rollback-sheet-host-post-stale"), + }, + ) + assert rejected.status_code == 200, rejected.text + assert rejected.json()["status"] == "rollback_failed" + assert "deleted sheet remains present" in str(rejected.json()).lower() + failed_plan = store.get_operation_plan(plan.id) or {} + assert failed_plan["status"] == "applied" + assert failed_plan["metadata"]["rollback_capability_used"] is True + + +def test_rollback_authorization_rejects_wrong_attempt_registry_and_operations() -> None: + client = TestClient(app) + store, session_id, plan_id, _, rollback = _start_cell_rollback(client) + authorize_url = f"/api/v1/sessions/{session_id}/rollback/authorize" + request_payload = { + "snapshot_id": rollback["snapshot_id"], + "rollback_token": rollback["rollback_token"], + "rollback_attempt_id": rollback["rollback_attempt_id"], + "rollback_operation_registry_hash": rollback["rollback_operation_registry_hash"], + "operations": rollback["operations"], + } + + wrong_attempt = client.post( + authorize_url, + headers=_headers(), + json={**request_payload, "rollback_attempt_id": "rollback_forged"}, + ) + assert wrong_attempt.status_code == 409 + + wrong_registry = client.post( + authorize_url, + headers=_headers(), + json={**request_payload, "rollback_operation_registry_hash": "0" * 64}, + ) + assert wrong_registry.status_code == 409 + + wrong_operations = [dict(operation) for operation in rollback["operations"]] + wrong_operations[0]["range"] = "B1" + wrong_operation_response = client.post( + authorize_url, + headers=_headers(), + json={**request_payload, "operations": wrong_operations}, + ) + assert wrong_operation_response.status_code == 409 + + authorized = client.post(authorize_url, headers=_headers(), json=request_payload) + assert authorized.status_code == 200, authorized.text + duplicate = client.post(authorize_url, headers=_headers(), json=request_payload) + assert duplicate.status_code == 409 + pending = store.get_operation_plan(plan_id) or {} + assert pending["status"] == "rollback_pending" + assert pending["metadata"]["rollback_capability_used"] is False + + +def test_concurrent_rollback_authorizations_issue_exactly_one_capability( + monkeypatch: pytest.MonkeyPatch, +) -> None: + client = TestClient(app) + store, session_id, plan_id, _, rollback = _start_cell_rollback(client) + + responses = _run_two_requests_after_same_plan_read( + monkeypatch, + store, + plan_id, + lambda: _authorize_rollback(client, session_id, rollback), + ) + + assert sorted(response.status_code for response in responses) == [200, 409] + authorized = [response.json() for response in responses if response.status_code == 200] + assert len(authorized) == 1 + durable = store.get_operation_plan(plan_id) or {} + assert durable["metadata"]["rollback_capability_id"] == authorized[0]["rollback_capability_id"] + assert durable["metadata"]["rollback_capability_used"] is False + + +def test_stale_concurrent_rollback_cannot_overwrite_authorized_attempt( + monkeypatch: pytest.MonkeyPatch, +) -> None: + client = TestClient(app) + store = get_session_store() + suffix = uuid.uuid4().hex[:10] + session_id = f"rollback-request-cas-{suffix}" + plan = OperationPlan( + id=f"plan-rollback-request-cas-{suffix}", + session_id=session_id, + prompt="Fill A1", + status="applied", + operations=[{ + "id": "op-fill", + "type": "write_cell", + "sheet": "Sheet1", + "range": "A1", + "value": 42, + }], + ) + store.get_or_create_session(session_id) + store.save_operation_plan(plan.model_dump(mode="json", by_alias=True)) + snapshot_id = store.record_snapshot( + session_id=session_id, + plan_id=plan.id, + label="Before fill", + workbook_state={"state": _rollback_context("")}, + ) + rollback_url = f"/api/v1/sessions/{session_id}/rollback" + rollback_payload = { + "snapshot_id": snapshot_id, + "current_workbook_state": _rollback_context(42), + } + + original_get_operation_plan = store.get_operation_plan + delayed_read_complete = threading.Event() + release_delayed_request = threading.Event() + delay_lock = threading.Lock() + delayed_once = False + + def staged_get_operation_plan(requested_plan_id: str): + nonlocal delayed_once + durable_plan = original_get_operation_plan(requested_plan_id) + with delay_lock: + should_delay = requested_plan_id == plan.id and not delayed_once + if should_delay: + delayed_once = True + if should_delay: + delayed_read_complete.set() + if not release_delayed_request.wait(timeout=5): + raise TimeoutError("Timed out waiting to resume stale rollback request") + return durable_plan + + def delayed_rollback_request(): + return client.post( + rollback_url, + headers=_headers(), + json=rollback_payload, + ) + + monkeypatch.setattr(store, "get_operation_plan", staged_get_operation_plan) + with ThreadPoolExecutor(max_workers=1) as executor: + delayed_future = executor.submit(delayed_rollback_request) + try: + assert delayed_read_complete.wait(timeout=5) + winner = client.post(rollback_url, headers=_headers(), json=rollback_payload) + assert winner.status_code == 200, winner.text + winner_payload = winner.json() + authorization = _authorize_rollback(client, session_id, winner_payload) + assert authorization.status_code == 200, authorization.text + finally: + release_delayed_request.set() + stale = delayed_future.result(timeout=10) + + assert stale.status_code == 409 + durable = store.get_operation_plan(plan.id) or {} + assert durable["metadata"]["rollback_attempt_id"] == winner_payload["rollback_attempt_id"] + assert durable["metadata"]["rollback_capability_id"] == authorization.json()["rollback_capability_id"] + assert durable["metadata"]["rollback_capability_used"] is False + + +def test_rollback_authorization_rejects_stale_attempt_and_allows_late_exact_ack() -> None: + client = TestClient(app) + store, session_id, plan_id, _, rollback = _start_cell_rollback(client) + stale_plan = store.get_operation_plan(plan_id) or {} + stale_plan["metadata"]["rollback_token_expires_at"] = 0 + store.save_operation_plan(stale_plan) + + stale_authorization = _authorize_rollback(client, session_id, rollback) + assert stale_authorization.status_code == 403 + assert "expired" in str(stale_authorization.json()).lower() + + store, session_id, plan_id, snapshot_id, rollback = _start_cell_rollback(client) + authorization_response = _authorize_rollback(client, session_id, rollback) + assert authorization_response.status_code == 200, authorization_response.text + authorization = authorization_response.json() + expired_plan = store.get_operation_plan(plan_id) or {} + expired_plan["metadata"]["rollback_capability_expires_at"] = 1 + expired_plan["metadata"]["rollback_token_expires_at"] = 1 + store.save_operation_plan(expired_plan) + late_ack = client.post( + f"/api/v1/sessions/{session_id}/rollback/result", + headers=_headers(), + json={ + "snapshot_id": snapshot_id, + "rollback_token": rollback["rollback_token"], + "rollback_attempt_id": rollback["rollback_attempt_id"], + **_rollback_capability_fields(authorization), + "status": "rolled_back", + "office_apply_result": { + "applied": 1, + "results": [{ + "operationId": "rollback_Sheet1_A1", + "type": "rollback_restore", + "status": "success", + "dirtyRanges": [{"sheet": "Sheet1", "range": "A1"}], + }], + }, + "pre_apply_workbook_state": _rollback_context(42), + "pre_apply_workbook_fingerprint": "late-ack-pre", + "pre_host_snapshot": _host_proof("late-ack-host-pre"), + "post_apply_workbook_state": _rollback_context(""), + "post_apply_workbook_fingerprint": "late-ack-post", + "post_host_snapshot": _host_proof("late-ack-host-post"), + }, + ) + assert late_ack.status_code == 200, late_ack.text + assert late_ack.json()["status"] == "rolled_back" + assert (store.get_operation_plan(plan_id) or {})["status"] == "rolled_back" + + +def test_rollback_capability_rejects_fabrication_mismatch_and_replay() -> None: + client = TestClient(app) + store, session_id, plan_id, snapshot_id, rollback = _start_cell_rollback(client) + fabricated_before_authorize = client.post( + f"/api/v1/sessions/{session_id}/rollback/result", + headers=_headers(), + json={ + "snapshot_id": snapshot_id, + "rollback_token": rollback["rollback_token"], + "rollback_attempt_id": rollback["rollback_attempt_id"], + "rollback_capability_id": "rollback_cap_fabricated", + "rollback_capability_nonce": "fabricated", + "rollback_operation_registry_hash": rollback["rollback_operation_registry_hash"], + "status": "failed", + }, + ) + assert fabricated_before_authorize.status_code == 409 + + authorization_response = _authorize_rollback(client, session_id, rollback) + assert authorization_response.status_code == 200, authorization_response.text + authorization = authorization_response.json() + result_url = f"/api/v1/sessions/{session_id}/rollback/result" + base_result = { + "snapshot_id": snapshot_id, + "rollback_token": rollback["rollback_token"], + "rollback_attempt_id": rollback["rollback_attempt_id"], + **_rollback_capability_fields(authorization), + "status": "failed", + "error_detail": "Office rejected the rollback", + } + + wrong_id = client.post( + result_url, + headers=_headers(), + json={**base_result, "rollback_capability_id": "rollback_cap_forged"}, + ) + assert wrong_id.status_code == 409 + wrong_nonce = client.post( + result_url, + headers=_headers(), + json={**base_result, "rollback_capability_nonce": "forged"}, + ) + assert wrong_nonce.status_code == 403 + wrong_registry = client.post( + result_url, + headers=_headers(), + json={**base_result, "rollback_operation_registry_hash": "f" * 64}, + ) + assert wrong_registry.status_code == 409 + pending = store.get_operation_plan(plan_id) or {} + assert pending["status"] == "rollback_pending" + assert pending["metadata"]["rollback_capability_used"] is False + + consumed = client.post(result_url, headers=_headers(), json=base_result) + assert consumed.status_code == 200, consumed.text + assert consumed.json()["status"] == "rollback_failed" + replay = client.post(result_url, headers=_headers(), json=base_result) + assert replay.status_code == 409 + terminal = store.get_operation_plan(plan_id) or {} + assert terminal["status"] == "applied" + assert terminal["metadata"]["rollback_capability_used"] is True + + +def test_concurrent_rollback_results_consume_capability_exactly_once( + monkeypatch: pytest.MonkeyPatch, +) -> None: + client = TestClient(app) + store, session_id, plan_id, snapshot_id, rollback = _start_cell_rollback(client) + authorization_response = _authorize_rollback(client, session_id, rollback) + assert authorization_response.status_code == 200, authorization_response.text + authorization = authorization_response.json() + result_payload = { + "snapshot_id": snapshot_id, + "rollback_token": rollback["rollback_token"], + "rollback_attempt_id": rollback["rollback_attempt_id"], + **_rollback_capability_fields(authorization), + "status": "failed", + "error_detail": "Office rejected the rollback", + } + + responses = _run_two_requests_after_same_plan_read( + monkeypatch, + store, + plan_id, + lambda: client.post( + f"/api/v1/sessions/{session_id}/rollback/result", + headers=_headers(), + json=result_payload, + ), + ) + + assert sorted(response.status_code for response in responses) == [200, 409] + durable = store.get_operation_plan(plan_id) or {} + assert durable["status"] == "applied" + assert durable["metadata"]["rollback_capability_used"] is True + + +def test_rollback_authorization_revalidates_delete_sheet_source_binding() -> None: + client = TestClient(app) + store = get_session_store() + suffix = uuid.uuid4().hex[:10] + session_id = f"rollback-source-binding-{suffix}" + plan = OperationPlan( + id=f"plan-rollback-source-binding-{suffix}", + session_id=session_id, + prompt="Create scenario sheet", + status="applied", + operations=[{"id": "op-create", "type": "create_sheet", "sheet_name": "Scenario", "range": "A1"}], + metadata={ + "durable_rollback_operations": [{ + "type": "delete_sheet", + "sheet": "Scenario", + "range": "A1", + "rollbackOnly": True, + "rollbackStrategy": "delete_created_sheet", + "sourceOperationId": "op-create", + }] + }, + ) + store.get_or_create_session(session_id) + store.save_operation_plan(plan.model_dump(mode="json", by_alias=True)) + snapshot_id = store.record_snapshot( + session_id=session_id, + plan_id=plan.id, + label="Before scenario sheet", + workbook_state={"state": _rollback_context("", sheets=["Sheet1"])}, + ) + requested = client.post( + f"/api/v1/sessions/{session_id}/rollback", + headers=_headers(), + json={ + "snapshot_id": snapshot_id, + "current_workbook_state": _rollback_context("", sheets=["Sheet1", "Scenario"]), + }, + ) + assert requested.status_code == 200, requested.text + rollback = requested.json() + + corrupted_plan = store.get_operation_plan(plan.id) or {} + corrupted_plan["operations"][0]["sheet_name"] = "OtherSheet" + store.save_operation_plan(corrupted_plan) + authorization = _authorize_rollback(client, session_id, rollback) + assert authorization.status_code == 409 + assert "create-sheet source" in str(authorization.json()).lower() + assert (store.get_operation_plan(plan.id) or {})["status"] == "rollback_pending" + + +def test_failed_rollback_can_close_before_capability_issuance() -> None: + client = TestClient(app) + store, session_id, plan_id, snapshot_id, rollback = _start_cell_rollback(client) + failed = client.post( + f"/api/v1/sessions/{session_id}/rollback/result", + headers=_headers(), + json={ + "snapshot_id": snapshot_id, + "rollback_token": rollback["rollback_token"], + "rollback_attempt_id": rollback["rollback_attempt_id"], + "status": "failed", + "error_detail": "Preflight host read failed before authorization", + }, + ) + assert failed.status_code == 200, failed.text + assert failed.json()["status"] == "rollback_failed" + terminal = store.get_operation_plan(plan_id) or {} + assert terminal["status"] == "applied" + assert not terminal["metadata"].get("rollback_capability_id") + + +def test_rollback_registry_hash_is_stable_across_json_number_spellings() -> None: + python_numbers = [{ + "id": "rollback_numeric", + "values": [[1.0, 1e21, 1e-7, -0.0]], + "enabled": True, + }] + browser_numbers = [{ + "id": "rollback_numeric", + "values": [[1, 1_000_000_000_000_000_000_000, 0.0000001, 0]], + "enabled": True, + }] + + assert sessions._rollback_operation_registry_hash(python_numbers) == ( + sessions._rollback_operation_registry_hash(browser_numbers) + ) + assert sessions._rollback_operation_registry_hash([{"value": True}]) != ( + sessions._rollback_operation_registry_hash([{"value": 1}]) + ) + assert sessions._rollback_operation_registry_hash([{"value": 9_007_199_254_740_993}]) != ( + sessions._rollback_operation_registry_hash([{"value": 9_007_199_254_740_992}]) + ) diff --git a/backend/tests/test_runtime_config.py b/backend/tests/test_runtime_config.py index fdec24c..b267144 100644 --- a/backend/tests/test_runtime_config.py +++ b/backend/tests/test_runtime_config.py @@ -1,12 +1,20 @@ +import base64 +import json from pathlib import Path -from backend.config import runtime_config_errors +from backend.config import ANTHROPIC_RETIRED_MODELS, runtime_config_errors def test_dev_profile_allows_local_defaults(): assert runtime_config_errors({"config_profile": "dev", "app_env": "dev", "production_mode": False}) == [] +def test_retired_anthropic_model_map_points_at_current_replacements(): + assert ANTHROPIC_RETIRED_MODELS["claude-3-5-haiku-20241022"] == "claude-haiku-4-5-20251001" + assert ANTHROPIC_RETIRED_MODELS["claude-sonnet-4-20250514"] == "claude-sonnet-4-6" + assert ANTHROPIC_RETIRED_MODELS["claude-opus-4-20250514"] == "claude-opus-4-8" + + def test_prod_profile_fails_closed_on_dev_security_defaults(): errors = runtime_config_errors( { @@ -14,6 +22,7 @@ def test_prod_profile_fails_closed_on_dev_security_defaults(): "app_env": "prod", "production_mode": True, "auth_enabled": False, + "rbac_enabled": False, "api_key": "dev-key-change-in-production", "cors_origins": ["https://localhost:3000"], "db_engine": "sqlite", @@ -23,12 +32,240 @@ def test_prod_profile_fails_closed_on_dev_security_defaults(): ) assert "AUTH_ENABLED must be true in production profiles" in errors + assert "RBAC_ENABLED must be true in production profiles" in errors + assert "OIDC_ENABLED must be true in production profiles" in errors assert "API_KEY must be a strong non-placeholder secret in production profiles" in errors + assert "API_KEY_HMAC_PEPPER must be a strong secret of at least 32 characters in production profiles" in errors + assert any("EXCEL_AGENT_LIFECYCLE_RECEIPT_KEY_ID" in error for error in errors) + assert any("EXCEL_AGENT_LIFECYCLE_RECEIPT_HMAC_KEY" in error for error in errors) + assert "AI_EXTERNAL_SENDING_ALLOWED must be explicitly set in production profiles" in errors + assert "AI_EXTERNAL_DOCUMENTS_ALLOWED must be explicitly set in production profiles" in errors assert "CORS_ORIGINS must be explicit HTTPS production origins" in errors assert "DB_ENGINE must be postgres in production profiles" in errors assert any(error.startswith("WORKBOOK_WORKER_DIR must exist") for error in errors) +def test_prod_profile_requires_complete_hardened_oidc_configuration(): + errors = runtime_config_errors( + { + "config_profile": "prod", + "app_env": "prod", + "production_mode": True, + "auth_enabled": True, + "rbac_enabled": True, + "api_key": "prod-api-key-with-enough-entropy", + "api_key_hmac_pepper": "p" * 32, + "cors_origins": ["https://excel.example.com"], + "db_engine": "postgres", + "database_url": "postgresql://financeai:strong-secret@example.com:5432/financeai", + "workbook_worker_dir": Path("workbook-worker"), + "ai_external_sending_allowed": "false", + "ai_external_documents_allowed": "false", + "formula_engine": "univer", + "hyperformula_enabled": False, + "oidc_enabled": True, + "oidc_issuer": "http://identity.example.com/tenant", + "oidc_audience": "", + "oidc_jwks_url": "http://identity.example.com/keys", + "oidc_allowed_tenants": frozenset(), + "oidc_allowed_algorithms": ("HS256",), + "oidc_require_mfa": False, + } + ) + + assert "OIDC_ISSUER must be an HTTPS URL in production profiles" in errors + assert "OIDC_AUDIENCE must be set in production profiles" in errors + assert "OIDC_JWKS_URL must be an HTTPS URL in production profiles" in errors + assert "OIDC_ALLOWED_TENANTS must explicitly allow at least one tenant in production profiles" in errors + assert ( + "OIDC_ALLOWED_ALGORITHMS must contain only Office-client-compatible algorithms: " + "RS256, PS256, or ES256" + ) in errors + assert "OIDC_REQUIRE_MFA must be true in production profiles" in errors + + +def test_prod_profile_requires_valid_deletion_receipt_signing_material(): + current_key = base64.urlsafe_b64encode(b"c" * 32).decode("ascii").rstrip("=") + previous_key = base64.urlsafe_b64encode(b"p" * 32).decode("ascii").rstrip("=") + base = { + "config_profile": "prod", + "app_env": "prod", + "production_mode": True, + "lifecycle_receipt_key_id": "receipt-2026-08", + "lifecycle_receipt_hmac_key": current_key, + "lifecycle_receipt_previous_keys": json.dumps({"receipt-2026-07": previous_key}), + } + + errors = runtime_config_errors(base) + assert not any("EXCEL_AGENT_LIFECYCLE_RECEIPT" in error for error in errors) + + malformed = runtime_config_errors( + { + **base, + "lifecycle_receipt_key_id": "bad key id", + "lifecycle_receipt_hmac_key": "not-base64!", + } + ) + assert any("EXCEL_AGENT_LIFECYCLE_RECEIPT_KEY_ID" in error for error in malformed) + assert any("EXCEL_AGENT_LIFECYCLE_RECEIPT_HMAC_KEY" in error for error in malformed) + + redefined = runtime_config_errors( + { + **base, + "lifecycle_receipt_previous_keys": json.dumps({"receipt-2026-08": current_key}), + } + ) + assert "EXCEL_AGENT_LIFECYCLE_RECEIPT_PREVIOUS_KEYS must not redefine the current key ID" in redefined + + +def test_prod_profile_rejects_incomplete_experimental_capabilities(): + errors = runtime_config_errors( + { + "config_profile": "prod", + "app_env": "prod", + "production_mode": True, + "experimental_routes_enabled": True, + "rag_enabled": True, + } + ) + + assert "EXPERIMENTAL_ROUTES_ENABLED must be false in production profiles" in errors + assert ( + "RAG_ENABLED must be false in production profiles until owner-scoped retrieval is available" + in errors + ) + + +def test_prod_profile_rejects_oidc_algorithm_not_supported_by_office_client(): + errors = runtime_config_errors( + { + "config_profile": "prod", + "app_env": "prod", + "production_mode": True, + "oidc_allowed_algorithms": ("RS384",), + } + ) + + assert ( + "OIDC_ALLOWED_ALGORITHMS must contain only Office-client-compatible algorithms: " + "RS256, PS256, or ES256" + ) in errors + +def test_prod_profile_requires_explicit_hyperformula_license_evidence(): + errors = runtime_config_errors( + { + "config_profile": "prod", + "app_env": "prod", + "production_mode": True, + "auth_enabled": True, + "rbac_enabled": True, + "api_key": "prod-api-key-with-enough-entropy", + "cors_origins": ["https://excel.example.com"], + "db_engine": "postgres", + "database_url": "postgresql://financeai:strong-secret@example.com:5432/financeai", + "workbook_worker_dir": Path("workbook-worker"), + "ai_external_sending_allowed": "false", + "ai_external_documents_allowed": "false", + "formula_engine": "hyperformula", + "hyperformula_enabled": True, + "hyperformula_distributed": False, + "hyperformula_license_approved": False, + "hyperformula_license_key": "", + } + ) + + assert any("HYPERFORMULA_DISTRIBUTED" in error for error in errors) + assert "HYPERFORMULA_LICENSE_APPROVED must be true before production HyperFormula use" in errors + assert "HYPERFORMULA_LICENSE_KEY must be an approved commercial key before production HyperFormula use" in errors + + sentinel_errors = runtime_config_errors( + { + "config_profile": "prod", + "app_env": "prod", + "production_mode": True, + "formula_engine": "hyperformula", + "hyperformula_enabled": True, + "hyperformula_distributed": True, + "hyperformula_license_approved": True, + "hyperformula_license_key": "gpl-v3", + } + ) + assert "HYPERFORMULA_LICENSE_KEY must be an approved commercial key before production HyperFormula use" in sentinel_errors + + +def test_prod_profile_rejects_retired_anthropic_models(): + errors = runtime_config_errors( + { + "config_profile": "prod", + "app_env": "prod", + "production_mode": True, + "auth_enabled": True, + "rbac_enabled": True, + "api_key": "prod-api-key-with-enough-entropy", + "cors_origins": ["https://excel.example.com"], + "db_engine": "postgres", + "database_url": "postgresql://financeai:strong-secret@example.com:5432/financeai", + "workbook_worker_dir": Path("workbook-worker"), + "ai_external_sending_allowed": "false", + "ai_external_documents_allowed": "false", + "formula_engine": "hyperformula", + "hyperformula_enabled": True, + "hyperformula_distributed": True, + "hyperformula_license_approved": True, + "hyperformula_license_key": "hf-commercial-license-key", + "anthropic_model_medium": "claude-3-5-haiku-20241022", + "anthropic_model_complex": "claude-sonnet-4-20250514", + } + ) + + assert "ANTHROPIC_MODEL_MEDIUM uses retired Anthropic model claude-3-5-haiku-20241022; migrate to claude-haiku-4-5-20251001" in errors + assert "ANTHROPIC_MODEL_COMPLEX uses retired Anthropic model claude-sonnet-4-20250514; migrate to claude-sonnet-4-6" in errors + + +def test_prod_profile_requires_ai_egress_policy_to_be_explicit(): + errors = runtime_config_errors( + { + "config_profile": "prod", + "app_env": "prod", + "production_mode": True, + "auth_enabled": True, + "rbac_enabled": True, + "api_key": "prod-api-key-with-enough-entropy", + "cors_origins": ["https://excel.example.com"], + "db_engine": "postgres", + "database_url": "postgresql://financeai:strong-secret@example.com:5432/financeai", + "workbook_worker_dir": Path("workbook-worker"), + } + ) + + assert "AI_EXTERNAL_SENDING_ALLOWED must be explicitly set in production profiles" in errors + assert "AI_EXTERNAL_DOCUMENTS_ALLOWED must be explicitly set in production profiles" in errors + + +def test_prod_profile_requires_byok_and_zdr_for_external_ai_documents(): + errors = runtime_config_errors( + { + "config_profile": "prod", + "app_env": "prod", + "production_mode": True, + "auth_enabled": True, + "rbac_enabled": True, + "api_key": "prod-api-key-with-enough-entropy", + "cors_origins": ["https://excel.example.com"], + "db_engine": "postgres", + "database_url": "postgresql://financeai:strong-secret@example.com:5432/financeai", + "workbook_worker_dir": Path("workbook-worker"), + "ai_external_sending_allowed": True, + "ai_external_documents_allowed": True, + "byok_enabled": False, + "zero_data_retention": False, + } + ) + + assert "AI_EXTERNAL_SENDING_ALLOWED=true requires BYOK_ENABLED=true in production profiles" in errors + assert "AI_EXTERNAL_DOCUMENTS_ALLOWED=true requires BYOK_ENABLED=true and ZERO_DATA_RETENTION=true in production profiles" in errors + + def test_enterprise_profile_requires_operating_security_controls(): errors = runtime_config_errors( { diff --git a/backend/tests/test_security_hardening.py b/backend/tests/test_security_hardening.py index 8365b4e..1d0ba17 100644 --- a/backend/tests/test_security_hardening.py +++ b/backend/tests/test_security_hardening.py @@ -1,13 +1,63 @@ -from fastapi import FastAPI +import asyncio +import time + +from fastapi import FastAPI, Request, WebSocketDisconnect from fastapi.testclient import TestClient -from backend.api.middleware.auth import AuthMiddleware +from backend.api.middleware.auth import AuthMiddleware, validate_ws_api_key +from backend.api.middleware.rate_limiter import RateLimiterMiddleware from backend.api.routes.documents import _validate_upload_signature -from backend.api.security.rbac import RBACMiddleware +from backend.api.security.oidc import OIDCValidationError +from backend.api.security.rbac import RBACMiddleware, ZeroDataRetention, _normalize_path, _permission_for +from backend.api.security.soc2 import get_soc2_auditor +from backend.main import _extract_ws_api_key, _extract_ws_credential, _resolve_ws_actor, _resolve_ws_credential_actor +from backend.session_store import SessionStore from backend.storage import postgres_protocol_gaps from backend.storage.postgres import PostgresSessionStore +class _FakeWebSocket: + def __init__(self, headers: dict[str, str] | None = None, query_params: dict[str, str] | None = None): + self.headers = {key.lower(): value for key, value in (headers or {}).items()} + self.query_params = query_params or {} + + +class _OneMessageWebSocket: + def __init__(self, message: dict): + self.message = message + self.accepted = False + self.received = False + self.sent: list[dict] = [] + + async def accept(self): + self.accepted = True + + async def receive_json(self): + if self.received: + raise WebSocketDisconnect() + self.received = True + return self.message + + async def send_json(self, payload: dict): + self.sent.append(payload) + + +def _diagnostic_client(diagnostic_paths_public: bool) -> TestClient: + app = FastAPI() + + @app.get("/metrics") + async def metrics(): + return {"ok": True} + + app.add_middleware( + AuthMiddleware, + api_key="secret-key", + enabled=True, + diagnostic_paths_public=diagnostic_paths_public, + ) + return TestClient(app) + + def test_http_auth_rejects_query_string_api_keys(): app = FastAPI() @@ -22,6 +72,263 @@ async def rules(): assert "X-API-Key" in response.json()["error"]["message"] +def test_http_oidc_bearer_hydrates_user_bound_actor(): + class Authenticator: + async def authenticate(self, token: str): + assert token == "signed-user-token" + return { + "user_id": "alice", + "email": "alice@example.com", + "role": "analyst", + "auth_source": "oidc", + } + + app = FastAPI() + + @app.get("/api/v1/me") + async def me(request: Request): + return request.state.enterprise_user + + app.add_middleware( + AuthMiddleware, + api_key="service-key", + enabled=True, + oidc_authenticator=Authenticator(), + ) + response = TestClient(app).get( + "/api/v1/me", + headers={"Authorization": "Bearer signed-user-token"}, + ) + + assert response.status_code == 200 + assert response.json()["user_id"] == "alice" + assert response.json()["auth_source"] == "oidc" + + +def test_http_invalid_bearer_never_falls_through_to_valid_api_key(): + class Authenticator: + async def authenticate(self, _token: str): + raise OIDCValidationError("invalid") + + app = FastAPI() + + @app.get("/api/v1/protected") + async def protected(): + return {"ok": True} + + app.add_middleware( + AuthMiddleware, + api_key="service-key", + enabled=True, + static_key_admin=True, + oidc_authenticator=Authenticator(), + ) + response = TestClient(app).get( + "/api/v1/protected", + headers={ + "Authorization": "Bearer invalid-token", + "X-API-Key": "service-key", + }, + ) + + assert response.status_code == 401 + assert response.json()["error"]["code"] == "AUTH_FAILED" + + +def test_diagnostic_paths_require_auth_when_not_dev_safe(): + client = _diagnostic_client(diagnostic_paths_public=False) + + for path in ("/docs", "/redoc", "/openapi.json", "/metrics"): + response = client.get(path) + assert response.status_code == 401 + + authorized = client.get(path, headers={"X-API-Key": "secret-key"}) + assert authorized.status_code == 200 + + +def test_diagnostic_paths_remain_public_when_dev_safe(): + client = _diagnostic_client(diagnostic_paths_public=True) + + for path in ("/docs", "/redoc", "/openapi.json", "/metrics"): + response = client.get(path) + assert response.status_code == 200 + + +def test_rate_limiter_does_not_bypass_protected_metrics_surface(): + app = FastAPI() + + @app.get("/metrics") + async def metrics(): + return {"ok": True} + + app.add_middleware(RateLimiterMiddleware, default_rpm=1, diagnostic_paths_public=False) + client = TestClient(app) + + assert client.get("/metrics").status_code == 200 + response = client.get("/metrics") + + assert response.status_code == 429 + assert response.json()["error"]["code"] == "RATE_LIMITED" + + +def test_websocket_auth_rejects_query_string_api_key_fallback(): + websocket = _FakeWebSocket(query_params={"api_key": "secret-key"}) + + extracted = _extract_ws_api_key(websocket) + + assert extracted is None + assert validate_ws_api_key(extracted, "secret-key") is False + + +def test_websocket_auth_accepts_header_and_subprotocol_keys(): + assert _extract_ws_credential(_FakeWebSocket(headers={"Authorization": "Bearer signed-token"})) == ( + "bearer", + "signed-token", + ) + assert _extract_ws_api_key(_FakeWebSocket(headers={"Authorization": "Bearer signed-token"})) is None + assert _extract_ws_api_key(_FakeWebSocket(headers={"X-API-Key": "secret-key"})) == "secret-key" + assert _extract_ws_api_key(_FakeWebSocket(headers={"Sec-WebSocket-Protocol": "chat, api-key.secret-key"})) == "secret-key" + assert _extract_ws_credential( + _FakeWebSocket(headers={"Sec-WebSocket-Protocol": "chat, bearer.signed.token"}) + ) == ("bearer", "signed.token") + + +def test_websocket_invalid_bearer_does_not_fall_through_to_api_key(): + websocket = _FakeWebSocket( + headers={"Authorization": "Basic bad", "X-API-Key": "secret-key"} + ) + + assert _extract_ws_credential(websocket) == ("invalid", "") + + +def test_websocket_oidc_bearer_hydrates_actor(monkeypatch): + class Authenticator: + async def authenticate(self, token: str): + assert token == "signed-token" + return { + "user_id": "alice", + "email": "alice@example.com", + "role": "analyst", + "auth_source": "oidc", + } + + monkeypatch.setattr("backend.main.oidc_authenticator", Authenticator()) + + actor = asyncio.run(_resolve_ws_credential_actor(("bearer", "signed-token"))) + + assert actor["user_id"] == "alice" + assert actor["auth_source"] == "oidc" + + +def test_websocket_static_key_follows_static_admin_gate(monkeypatch): + monkeypatch.setattr("backend.main.API_KEY", "secret-key") + monkeypatch.setattr("backend.main.STATIC_KEY_ADMIN_ENABLED", False) + + assert _resolve_ws_actor("secret-key") is None + + +def test_websocket_enterprise_key_hydrates_actor(monkeypatch): + class Store: + def get_enterprise_user_by_api_key(self, api_key: str): + assert api_key == "enterprise-key" + return { + "user_id": "analyst-1", + "email": "analyst@example.com", + "name": "Analyst", + "role": "analyst", + } + + monkeypatch.setattr("backend.main.API_KEY", "secret-key") + monkeypatch.setattr("backend.storage.get_session_store", lambda: Store()) + + actor = _resolve_ws_actor("enterprise-key") + + assert actor["user_id"] == "analyst-1" + assert actor["auth_source"] == "enterprise_api_key" + + +def test_websocket_handler_enforces_role_policy_before_pipeline(monkeypatch): + async def unexpected_pipeline_call(**_kwargs): + raise AssertionError("viewer action websocket message should not reach the pipeline") + + from backend.api.routes import tasks + + monkeypatch.setattr(tasks, "run_pipeline", unexpected_pipeline_call) + websocket = _OneMessageWebSocket( + { + "prompt": "Apply a formatting change", + "mode": "action", + "provider": "lmstudio", + } + ) + + asyncio.run( + tasks.websocket_task_handler( + websocket, + "session-ws-policy", + actor={"user_id": "viewer-1", "email": "viewer@example.com", "role": "viewer"}, + ) + ) + + assert websocket.accepted is True + assert websocket.sent[0]["type"] == "error" + assert websocket.sent[0]["detail"]["code"] == "ai_role_not_permitted" + + +def test_enterprise_provider_policy_mfa_gate_blocks_non_mfa_key(tmp_path): + store = SessionStore(db_path=tmp_path / "sessions.db") + store.upsert_enterprise_user( + user_id="analyst-provider-policy-mfa", + email="analyst-provider-policy-mfa@example.com", + name="Provider Policy MFA", + role="analyst", + status="api_key_only", + mfa_enabled=False, + ) + api_key = store.create_enterprise_api_key("analyst-provider-policy-mfa", "provider-policy-test")["api_key"] + + assert store.get_enterprise_user_by_api_key(api_key)["user_id"] == "analyst-provider-policy-mfa" + + store.set_enterprise_config("provider_policy", {"requireMfa": True}) + assert store.get_enterprise_user_by_api_key(api_key) is None + + store.upsert_enterprise_user( + user_id="analyst-provider-policy-mfa", + email="analyst-provider-policy-mfa@example.com", + name="Provider Policy MFA", + role="analyst", + mfa_enabled=True, + ) + assert store.get_enterprise_user_by_api_key(api_key)["user_id"] == "analyst-provider-policy-mfa" + + +def test_enterprise_security_policy_mfa_gate_blocks_non_mfa_key(tmp_path): + store = SessionStore(db_path=tmp_path / "sessions.db") + store.upsert_enterprise_user( + user_id="analyst-security-policy-mfa", + email="analyst-security-policy-mfa@example.com", + name="Security Policy MFA", + role="analyst", + status="api_key_only", + mfa_enabled=False, + ) + api_key = store.create_enterprise_api_key("analyst-security-policy-mfa", "security-policy-test")["api_key"] + + assert store.get_enterprise_user_by_api_key(api_key)["user_id"] == "analyst-security-policy-mfa" + + store.set_enterprise_config("security_policy", {"require_mfa": True}) + assert store.get_enterprise_user_by_api_key(api_key) is None + + store.upsert_enterprise_user( + user_id="analyst-security-policy-mfa", + email="analyst-security-policy-mfa@example.com", + name="Security Policy MFA", + role="analyst", + mfa_enabled=True, + ) + assert store.get_enterprise_user_by_api_key(api_key)["user_id"] == "analyst-security-policy-mfa" + + def test_rbac_denies_unmapped_api_routes_by_default(): app = FastAPI() @@ -39,25 +346,79 @@ async def unmapped(): assert "RBAC route mapping is missing" in response.json()["error"]["message"] +def test_rbac_maps_spreadsheet_routes_to_task_execution(): + assert _permission_for("POST", "/api/v1/spreadsheet/open") == "tasks.execute" + assert _permission_for("POST", "/api/v1/spreadsheet/export") == "tasks.execute" + assert _permission_for("POST", "/api/v1/spreadsheet/roundtrip-qa") == "tasks.execute" + + +def test_rbac_normalization_preserves_short_hex_route_segments(): + assert _normalize_path("/api/v1/tasks/plans/abcdef12") == "/api/v1/tasks/plans/abcdef12" + assert _normalize_path("/api/v1/tasks/plans/abcdef12/inspect") == "/api/v1/tasks/plans/abcdef12/inspect" + + +def test_rbac_segment_prefixes_do_not_confuse_plan_and_plans_routes(): + assert _permission_for("POST", "/api/v1/tasks/plans/abcdef12/inspect") == "tasks.plan" + assert _permission_for("POST", "/api/v1/tasks/plans/abcdef12/verify") == "tasks.plan" + assert _permission_for("POST", "/api/v1/tasks/plankton") == "tasks.execute" + + +def test_rbac_session_mutation_routes_require_write_permissions(): + assert _permission_for("DELETE", "/api/v1/sessions/abcdef12") == "sessions.delete" + assert _permission_for("POST", "/api/v1/sessions/abcdef12/undo") == "tasks.execute" + assert _permission_for("POST", "/api/v1/sessions/abcdef12/redo") == "tasks.execute" + assert _permission_for("POST", "/api/v1/sessions/abcdef12/rollback") == "tasks.execute" + assert _permission_for("GET", "/api/v1/sessions/abcdef12/audit/export") == "audit.export" + + +def test_zdr_purge_expired_deletes_old_sessions(monkeypatch): + class Receipt: + database_deleted = True + errors = [] + + class Coordinator: + cutoff = 0.0 + + async def purge_before(self, cutoff_timestamp: float): + self.cutoff = cutoff_timestamp + return [Receipt(), Receipt(), Receipt()] + + coordinator = Coordinator() + monkeypatch.setenv("ZERO_DATA_RETENTION", "true") + monkeypatch.setenv("DATA_RETENTION_HOURS", "2") + monkeypatch.setattr( + "backend.services.lifecycle.get_lifecycle_coordinator", + lambda: coordinator, + ) + + before = time.time() - (2 * 3600) + deleted = asyncio.run(ZeroDataRetention().purge_expired()) + after = time.time() - (2 * 3600) + + assert deleted == 3 + assert before <= coordinator.cutoff <= after + + +def test_soc2_auth_check_reports_required_mfa_gate(): + result = get_soc2_auditor().check_auth() + + assert result["passed"] is True + assert result["auth_middleware"] is True + assert result["rbac_module"] is True + assert result["required_mfa_policy_gate"] is True + assert result["enterprise_admin_audit_gate"] is True + assert all(result["enterprise_admin_audit_actions"].values()) + + def test_postgres_store_parity_gaps_are_explicit(): - """The factory's gap detector must still report missing methods. - - The list of gaps is allowed to shrink as Postgres parity work lands; what - matters is that the detector reports SOMETHING when something is missing - (so the factory keeps refusing to start under partial parity) AND that - the methods that have already been implemented are not falsely reported - as gaps. - """ + """The factory's gap detector must not report false Postgres gaps.""" gaps = postgres_protocol_gaps(PostgresSessionStore) - # Implemented in the May 2026 audit pass — must NOT appear as gaps. assert "save_operation_plan" not in gaps assert "get_operation_plan" not in gaps assert "record_snapshot" not in gaps assert "record_audit_event" not in gaps - - # Any remaining gap is sufficient to keep the factory from booting Postgres. - assert isinstance(gaps, list) + assert gaps == [] def test_upload_signature_validation_rejects_disguised_office_files(): diff --git a/backend/tests/test_security_lifecycle_closure.py b/backend/tests/test_security_lifecycle_closure.py new file mode 100644 index 0000000..772f78b --- /dev/null +++ b/backend/tests/test_security_lifecycle_closure.py @@ -0,0 +1,838 @@ +"""Focused regressions for the closure-release lifecycle security boundaries.""" + +from __future__ import annotations + +import asyncio +import io +import json +import threading +import time +from types import SimpleNamespace + +import pytest +from fastapi import FastAPI +from fastapi.testclient import TestClient +from openpyxl import Workbook + +from backend.agents import workflows as workflows_module +from backend.agents.workflows import WorkflowEngine +from backend.api import teams as teams_module +from backend.api.routes import documents as document_routes +from backend.api.routes import sessions as session_routes +from backend.api.security.ownership import ( + OwnershipNotFoundError, + ownership_actor_scope, + require_session_access, +) +from backend.api.security.rbac import ZeroDataRetention +from backend.data.connectors import XLSXConnector +from backend.documents.pdf_parser import PDFBudgetExceeded, PDFParseLimits, parse_pdf +from backend.documents.pdf_worker import parse_pdf_sandboxed +from backend.excel.workbook_files import WorkbookIngestionService +from backend.excel import diff_engine as diff_engine_module +from backend.services import lifecycle as lifecycle_module +from backend.services.lifecycle import LifecycleCoordinator, LifecycleState +from backend.services.workbook_jobs import WorkbookJobManager +from backend.session_store import SessionStore + + +@pytest.fixture(autouse=True) +def _lifecycle_receipt_signing_key(monkeypatch): + monkeypatch.setenv("EXCEL_AGENT_LIFECYCLE_RECEIPT_KEY_ID", "test-current") + monkeypatch.setenv("EXCEL_AGENT_LIFECYCLE_RECEIPT_HMAC_KEY", "cnJy" * 16) + + +class _MemoryJobStore: + def __init__(self) -> None: + self.records: dict[str, dict] = {} + self.sessions: set[str] = {"session-1"} + + def save_workbook_job(self, job: dict) -> str: + self.records[job["run_id"]] = dict(job) + return str(job["run_id"]) + + def get_workbook_job(self, run_id: str) -> dict | None: + record = self.records.get(run_id) + return dict(record) if record else None + + def list_workbook_jobs(self, limit=50, status=None, session_id=None, account_id=None): + rows = list(self.records.values()) + if status: + rows = [row for row in rows if row.get("status") == status] + if session_id: + rows = [row for row in rows if row.get("session_id") == session_id] + if account_id: + rows = [row for row in rows if row.get("account_id") == account_id] + return [dict(row) for row in rows[:limit]] + + def session_exists(self, session_id: str) -> bool: + return session_id in self.sessions + + def get_or_create_session(self, session_id: str): + self.sessions.add(session_id) + return {"session_id": session_id} + + +@pytest.mark.unit +def test_session_store_mode_is_the_only_workbook_job_authority(tmp_path, monkeypatch): + monkeypatch.setattr( + "backend.services.workbook_jobs._execute_command", + lambda command, payload: {"command": command}, + ) + store = _MemoryJobStore() + manager = WorkbookJobManager(tmp_path, store=store) + + job = manager.submit("diff", {"before": {}, "after": {}}, session_id="session-1") + for _ in range(50): + if manager.get(job.run_id).status == "completed": + break + import time + + time.sleep(0.01) + + assert store.records[job.run_id]["status"] == "completed" + assert not list(tmp_path.glob("run_*.json")) + + store.sessions.remove("session-1") + assert manager.get(job.run_id) is None + assert manager.list(session_id="session-1") == [] + + +@pytest.mark.unit +def test_configured_session_store_mode_never_falls_back_when_store_is_unavailable(tmp_path, monkeypatch): + legacy = tmp_path / "run_legacy.json" + legacy.write_text( + json.dumps( + { + "run_id": "run_legacy", + "command": "diff", + "status": "completed", + "result": {"private": "result"}, + } + ), + encoding="utf-8", + ) + + def unavailable_store(): + raise RuntimeError("database unavailable") + + monkeypatch.setattr("backend.storage.get_session_store", unavailable_store) + manager = WorkbookJobManager(tmp_path, use_session_store=True) + + assert manager.get("run_legacy") is None + assert manager.list() == [] + assert (tmp_path / "quarantine" / legacy.name).exists() + with pytest.raises(RuntimeError, match="store is unavailable"): + manager.submit("diff", {"before": {}, "after": {}}) + assert list(tmp_path.glob("run_*.json")) == [] + + +@pytest.mark.unit +def test_authenticated_admin_cannot_select_a_rate_limit_account(tmp_path, monkeypatch): + monkeypatch.setattr( + "backend.services.workbook_jobs._execute_command", + lambda command, payload: {"command": command}, + ) + store = _MemoryJobStore() + manager = WorkbookJobManager(tmp_path, store=store) + + with ownership_actor_scope(SimpleNamespace(user_id="root-admin", role="admin")): + job = manager.submit( + "diff", + {"before": {}, "after": {}}, + session_id="session-1", + account_id="evade-limit", + ) + + assert job.account_id == "root-admin" + assert job.owner_user_id == "root-admin" + + +@pytest.mark.unit +def test_orphan_job_json_is_quarantined_and_never_loaded_as_authority(tmp_path): + orphan = tmp_path / "run_orphan.json" + orphan.write_text( + json.dumps( + { + "run_id": "run_orphan", + "command": "diff", + "status": "completed", + "session_id": "session-1", + "result": {"private": "result"}, + } + ), + encoding="utf-8", + ) + manager = WorkbookJobManager(tmp_path, store=_MemoryJobStore()) + + assert manager.get("run_orphan") is None + quarantined = tmp_path / "quarantine" / orphan.name + assert quarantined.exists() + assert not orphan.exists() + + purge = manager.purge_session("session-1") + assert purge["files_removed"] == 1 + assert not quarantined.exists() + + +@pytest.mark.unit +def test_workbook_job_purge_keeps_pending_thread_visible_until_joined(tmp_path, monkeypatch): + started = threading.Event() + release = threading.Event() + + def blocked_command(_command, _payload): + started.set() + release.wait(timeout=5) + return {"private": "result"} + + monkeypatch.setattr("backend.services.workbook_jobs._execute_command", blocked_command) + store = _MemoryJobStore() + manager = WorkbookJobManager(tmp_path, store=store) + manager.submit("diff", {"before": {}, "after": {}}, session_id="session-1") + assert started.wait(timeout=2) + + first = manager.purge_session("session-1", join_timeout=0) + second = manager.purge_session("session-1", join_timeout=0) + + assert first["threads_pending"] == 1 + assert second["threads_pending"] == 1 + + release.set() + final = manager.purge_session("session-1", join_timeout=2) + assert final["threads_pending"] == 0 + + +@pytest.mark.unit +def test_upload_parse_failure_removes_raw_file(tmp_path, monkeypatch): + async def fail_parse(*_args, **_kwargs): + raise RuntimeError("parser rejected payload") + + monkeypatch.setattr(document_routes, "DATA_DIR", tmp_path) + monkeypatch.setattr(document_routes, "_parse_uploaded_content", fail_parse) + app = FastAPI() + app.include_router(document_routes.router) + + response = TestClient(app).post( + "/api/v1/documents/upload", + data={"session_id": "cleanup-test"}, + files={"file": ("source.pdf", b"%PDF-1.7\ninvalid", "application/pdf")}, + ) + + assert response.status_code == 500 + assert list((tmp_path / "documents").glob("*")) == [] + + +@pytest.mark.unit +def test_upload_persistence_failure_removes_raw_file(tmp_path, monkeypatch): + async def parsed(*_args, **_kwargs): + return {"text": "bounded"}, {"kind": "pdf"} + + class FailingStore: + def save_document(self, **_kwargs): + raise RuntimeError("database unavailable") + + monkeypatch.setattr(document_routes, "DATA_DIR", tmp_path) + monkeypatch.setattr(document_routes, "_parse_uploaded_content", parsed) + monkeypatch.setattr(document_routes, "get_session_store", lambda: FailingStore()) + app = FastAPI() + app.include_router(document_routes.router) + + response = TestClient(app).post( + "/api/v1/documents/upload", + data={"session_id": "cleanup-test"}, + files={"file": ("source.pdf", b"%PDF-1.7\ninvalid", "application/pdf")}, + ) + + assert response.status_code == 500 + assert list((tmp_path / "documents").glob("*")) == [] + + +@pytest.mark.unit +def test_workbook_ingest_fails_closed_when_worker_rejects(monkeypatch): + def worker_rejected(_command, _payload): + raise RuntimeError("archive budget rejected") + + monkeypatch.setattr("backend.excel.workbook_worker.run_worker", worker_rejected) + monkeypatch.setattr( + WorkbookIngestionService, + "_summarize_python", + lambda *_args, **_kwargs: pytest.fail("unsafe in-process fallback executed"), + ) + + workbook = Workbook() + output = io.BytesIO() + workbook.save(output) + with pytest.raises(RuntimeError, match="archive budget rejected"): + WorkbookIngestionService().summarize(output.getvalue(), "model.xlsx") + + +@pytest.mark.unit +def test_workbook_ingest_preflights_ooxml_before_worker(monkeypatch): + called = False + + def worker_should_not_run(_command, _payload): + nonlocal called + called = True + + monkeypatch.setattr("backend.excel.workbook_worker.run_worker", worker_should_not_run) + + with pytest.raises(ValueError, match="not a valid ZIP"): + WorkbookIngestionService().summarize(b"PK\x03\x04invalid", "model.xlsx") + assert called is False + + +@pytest.mark.unit +@pytest.mark.asyncio +async def test_xlsx_connector_requires_owned_managed_document_id(tmp_path, monkeypatch): + documents_dir = tmp_path / "documents" + documents_dir.mkdir() + workbook_path = documents_dir / "doc_owned_model.xlsx" + workbook = Workbook() + workbook.active.append(["Metric", "Value"]) + workbook.active.append(["Revenue", 100]) + workbook.save(workbook_path) + + class Store: + def get_document(self, document_id: str): + if document_id != "doc_owned": + return None + return { + "document_id": document_id, + "session_id": "session-1", + "filename": "model.xlsx", + "file_path": str(workbook_path), + } + + monkeypatch.setattr("backend.storage.get_session_store", lambda: Store()) + monkeypatch.setattr("backend.data.connectors.DATA_DIR", tmp_path, raising=False) + connector = XLSXConnector() + + with pytest.raises(ValueError, match="owned document"): + await connector.fetch(str(workbook_path)) + + result = await connector.fetch("doc_owned") + assert result.headers == ["Metric", "Value"] + assert result.data == [{"Metric": "Revenue", "Value": 100}] + + +@pytest.mark.unit +@pytest.mark.asyncio +async def test_saved_workflows_are_owner_scoped_and_never_echo_prompts(tmp_path, monkeypatch): + monkeypatch.setattr(workflows_module, "WORKFLOWS_DIR", tmp_path) + engine = WorkflowEngine() + + with ownership_actor_scope(SimpleNamespace(user_id="alice", role="analyst")): + workflow = engine.create_workflow( + name="Alice only", + description="Private prompt", + steps=[{"step_type": "prompt", "config": {"prompt": "Secret {ticker}"}}], + ) + result = await engine.execute_workflow(workflow.id, {"ticker": "ACME"}) + assert "Secret ACME" not in json.dumps(result) + assert result["results"][0]["reason"] == "pipeline_unavailable" + + with ownership_actor_scope(SimpleNamespace(user_id="bob", role="analyst")): + assert workflow.id not in {item["id"] for item in engine.list_workflows()} + assert engine.get_workflow(workflow.id) is None + with pytest.raises(ValueError, match="not found"): + await engine.execute_workflow(workflow.id) + + with ownership_actor_scope(SimpleNamespace(user_id="alice", role="analyst")): + assert engine.get_workflow(workflow.id) is not None + reloaded = WorkflowEngine() + assert reloaded.get_workflow(workflow.id) is not None + + +@pytest.mark.unit +@pytest.mark.asyncio +async def test_lifecycle_coordinator_revokes_jobs_files_workflows_and_session(tmp_path, monkeypatch): + data_dir = tmp_path / "data" + documents_dir = data_dir / "documents" + documents_dir.mkdir(parents=True) + document_path = documents_dir / "doc_source.pdf" + document_path.write_bytes(b"%PDF-1.7\nsource") + store = SessionStore(tmp_path / "sessions.db") + store.get_or_create_session("session-1") + store.save_document( + "session-1", + "source.pdf", + file_path=str(document_path), + document_id="doc_source", + ) + manager = WorkbookJobManager(tmp_path / "jobs", store=store) + monkeypatch.setattr( + "backend.services.workbook_jobs._execute_command", + lambda _command, _payload: {"ok": True}, + ) + job = manager.submit("diff", {"before": {}, "after": {}}, session_id="session-1") + for _ in range(50): + if manager.get(job.run_id).status == "completed": + break + await asyncio.sleep(0.01) + + workflows_dir = tmp_path / "workflows" + monkeypatch.setattr(workflows_module, "WORKFLOWS_DIR", workflows_dir) + workflow_engine = WorkflowEngine(store=store) + workflow = workflow_engine.create_workflow( + name="Session workflow", + description="Removed with session", + steps=[], + session_id="session-1", + ) + monkeypatch.setattr(workflows_module, "get_workflow_engine", lambda: workflow_engine) + monkeypatch.setattr(lifecycle_module, "DATA_DIR", data_dir) + receipts_dir = tmp_path / "receipts" + coordinator = LifecycleCoordinator( + store=store, + job_manager=manager, + receipts_dir=receipts_dir, + ) + + receipt = await coordinator.purge_session("session-1") + + assert receipt.state is LifecycleState.VERIFIED + assert receipt.database_deleted is True + assert receipt.document_files_deleted == 1 + assert receipt.workbook_jobs_revoked == 1 + assert receipt.workflows_deleted == 1 + assert not document_path.exists() + assert not store.session_exists("session-1") + assert manager.get(job.run_id) is None + assert workflow_engine.get_workflow(workflow.id) is None + receipt_payload = next(receipts_dir.glob("*.json")).read_text(encoding="utf-8") + assert "session-1" not in receipt_payload + + +@pytest.mark.unit +@pytest.mark.asyncio +async def test_failed_file_cleanup_keeps_database_inventory_and_can_be_retried(tmp_path, monkeypatch): + data_dir = tmp_path / "data" + documents_dir = data_dir / "documents" + documents_dir.mkdir(parents=True) + document_path = documents_dir / "doc_retry.pdf" + document_path.write_bytes(b"%PDF-1.7\nprivate") + store = SessionStore(tmp_path / "sessions.db") + store.get_or_create_session("session-retry") + store.save_document( + "session-retry", + "private.pdf", + file_path=str(document_path), + document_id="doc-retry", + ) + + class Jobs: + @staticmethod + def purge_session(_session_id: str): + return {"jobs_revoked": 0, "threads_pending": 0} + + class Workflows: + @staticmethod + def purge_session(_session_id: str): + return 0 + + @staticmethod + def count_session_workflows(_session_id: str): + return 0 + + original_delete = lifecycle_module._delete_managed_document + attempts = 0 + + def flaky_delete(raw_path: str) -> bool: + nonlocal attempts + attempts += 1 + if attempts == 1: + raise OSError("transient unlink failure") + return original_delete(raw_path) + + monkeypatch.setattr(lifecycle_module, "DATA_DIR", data_dir) + monkeypatch.setattr(lifecycle_module, "_delete_managed_document", flaky_delete) + monkeypatch.setattr(workflows_module, "get_workflow_engine", lambda: Workflows()) + coordinator = LifecycleCoordinator( + store=store, + job_manager=Jobs(), + receipts_dir=tmp_path / "receipts", + ) + + first = await coordinator.purge_session("session-retry") + + assert first.state is LifecycleState.FAILED + assert first.database_deleted is False + assert store.session_exists("session-retry") is True + assert store.list_document_file_paths("session-retry") == [str(document_path)] + assert document_path.exists() + assert coordinator.reads_revoked("session-retry") is True + + second = await coordinator.purge_session("session-retry") + + assert second.state is LifecycleState.VERIFIED + assert second.database_deleted is True + assert not document_path.exists() + assert not store.session_exists("session-retry") + + +@pytest.mark.unit +@pytest.mark.asyncio +async def test_sealed_purging_marker_revokes_reads_after_process_restart(tmp_path): + class SimulatedCrash(BaseException): + pass + + class Store: + @staticmethod + def list_document_file_paths(_session_id: str): + return [] + + class CrashingJobs: + @staticmethod + def purge_session(_session_id: str): + raise SimulatedCrash() + + receipts_dir = tmp_path / "receipts" + coordinator = LifecycleCoordinator( + store=Store(), + job_manager=CrashingJobs(), + receipts_dir=receipts_dir, + ) + + with pytest.raises(SimulatedCrash): + await coordinator.purge_session("session-crash") + + sealed_payload = next(receipts_dir.glob("del_*.json")).read_text(encoding="utf-8") + assert "session-crash" not in sealed_payload + restarted = LifecycleCoordinator( + store=Store(), + job_manager=CrashingJobs(), + receipts_dir=receipts_dir, + ) + assert restarted.state_for("session-crash") is LifecycleState.PURGING + assert restarted.reads_revoked("session-crash") is True + + +@pytest.mark.unit +def test_session_delete_route_requires_verified_lifecycle_receipt(monkeypatch): + class Store: + @staticmethod + def session_exists(session_id: str) -> bool: + return session_id == "session-1" + + verified = SimpleNamespace( + state=LifecycleState.VERIFIED, + to_dict=lambda: {"state": "verified", "receipt_id": "receipt-1"}, + ) + failed = SimpleNamespace( + state=LifecycleState.FAILED, + to_dict=lambda: {"state": "failed", "receipt_id": "receipt-2"}, + ) + + class Coordinator: + receipt = verified + + @staticmethod + def reads_revoked(_session_id: str) -> bool: + return True + + async def purge_session(self, session_id: str): + assert session_id == "session-1" + return self.receipt + + coordinator = Coordinator() + monkeypatch.setattr(session_routes, "get_session_store", lambda: Store()) + monkeypatch.setattr(lifecycle_module, "get_lifecycle_coordinator", lambda: coordinator) + app = FastAPI() + app.include_router(session_routes.router) + + response = TestClient(app).delete("/api/v1/sessions/session-1") + + assert response.status_code == 200 + assert response.json() == {"state": "verified", "receipt_id": "receipt-1"} + + coordinator.receipt = failed + response = TestClient(app).delete("/api/v1/sessions/session-1") + + assert response.status_code == 500 + assert response.json()["detail"]["code"] == "session_deletion_unverified" + + +@pytest.mark.unit +def test_sqlite_session_deletion_covers_all_typed_session_projections(tmp_path): + store = SessionStore(tmp_path / "sessions.db") + store.get_or_create_session("session-1") + typed_tables = [ + "agent_run_steps_v3", + "agent_checkpoints_v3", + "operation_diff_previews_v2", + "source_cell_provenance_v2", + "local_replay_baselines", + ] + with store._conn() as connection: + now = 1.0 + connection.execute( + "INSERT INTO agent_run_steps_v3 " + "(step_record_id, run_id, session_id, step_id, created_at, updated_at) " + "VALUES (?, ?, ?, ?, ?, ?)", + ("step-1", "run-1", "session-1", "inspect", now, now), + ) + connection.execute( + "INSERT INTO agent_checkpoints_v3 " + "(checkpoint_record_id, run_id, session_id, checkpoint_id, created_at, updated_at) " + "VALUES (?, ?, ?, ?, ?, ?)", + ("checkpoint-1", "run-1", "session-1", "approval", now, now), + ) + connection.execute( + "INSERT INTO operation_diff_previews_v2 " + "(diff_preview_id, session_id, created_at) VALUES (?, ?, ?)", + ("diff-1", "session-1", now), + ) + connection.execute( + "INSERT INTO source_cell_provenance_v2 " + "(provenance_id, session_id, created_at, updated_at) VALUES (?, ?, ?, ?)", + ("provenance-1", "session-1", now, now), + ) + connection.execute( + "INSERT INTO local_replay_baselines " + "(baseline_id, session_id, created_at) VALUES (?, ?, ?)", + ("baseline-1", "session-1", now), + ) + + store.set_session_lifecycle_state( + "session-1", + "purging", + receipt_id="del_test_typed_inventory", + started_at=time.time(), + ) + assert store.delete_session("session-1") is True + + with store._conn() as connection: + for table in typed_tables: + remaining = connection.execute( + f"SELECT COUNT(*) FROM {table} WHERE session_id = ?", # noqa: S608 -- static test table names + ("session-1",), + ).fetchone()[0] + assert remaining == 0, table + + +@pytest.mark.unit +def test_purging_state_immediately_revokes_session_document_and_workflow_reads(tmp_path, monkeypatch): + session_id = "lifecycle-revocation-test" + store = SessionStore(tmp_path / "sessions.db") + store.get_or_create_session(session_id) + store.save_document(session_id, "source.csv", document_id="doc-1") + workflows_dir = tmp_path / "workflows" + monkeypatch.setattr(workflows_module, "WORKFLOWS_DIR", workflows_dir) + engine = WorkflowEngine(store=store) + workflow = engine.create_workflow( + name="Session workflow", + description="Revoked before physical deletion finishes", + steps=[], + session_id=session_id, + ) + coordinator = LifecycleCoordinator(store=store, receipts_dir=tmp_path / "receipts") + coordinator._states[session_id] = LifecycleState.PURGING + monkeypatch.setattr(lifecycle_module, "_coordinator", coordinator) + + with pytest.raises(OwnershipNotFoundError, match="not found"): + require_session_access(store, session_id) + assert store.get_document("doc-1") is None + assert engine.get_workflow(workflow.id) is None + assert store.session_exists(session_id) is True + + +@pytest.mark.unit +def test_verified_purge_fence_prevents_sqlite_session_resurrection(tmp_path, monkeypatch): + session_id = "lifecycle-late-write" + store = SessionStore(tmp_path / "sessions.db") + store.get_or_create_session(session_id) + store.add_message(session_id, "user", "private before deletion") + coordinator = LifecycleCoordinator(store=store, receipts_dir=tmp_path / "receipts") + receipt_id = "del_test_verified_fence" + started_at = time.time() + store.set_session_lifecycle_state( + session_id, + "purging", + receipt_id=receipt_id, + started_at=started_at, + ) + coordinator._states[session_id] = LifecycleState.VERIFIED + monkeypatch.setattr(lifecycle_module, "_coordinator", coordinator) + assert store.delete_session(session_id) is True + store.set_session_lifecycle_state( + session_id, + "verified", + receipt_id=receipt_id, + started_at=started_at, + completed_at=time.time(), + ) + + with pytest.raises(OwnershipNotFoundError, match="not found"): + store.add_message(session_id, "assistant", "late private write") + with pytest.raises(OwnershipNotFoundError, match="not found"): + store.save_document(session_id, "late.xlsx", document_id="doc-late") + + assert store.session_exists(session_id) is False + with store._conn() as connection: + assert connection.execute( + "SELECT COUNT(*) FROM conversations WHERE session_id = ?", + (session_id,), + ).fetchone()[0] == 0 + assert connection.execute( + "SELECT COUNT(*) FROM documents WHERE session_id = ?", + (session_id,), + ).fetchone()[0] == 0 + + +@pytest.mark.unit +@pytest.mark.asyncio +async def test_revoked_lifecycle_fails_before_postgres_session_recreation(tmp_path, monkeypatch): + from backend.storage.postgres import PostgresSessionStore + + session_id = "postgres-late-write" + coordinator = LifecycleCoordinator(receipts_dir=tmp_path / "receipts") + coordinator._states[session_id] = LifecycleState.FAILED + monkeypatch.setattr(lifecycle_module, "_coordinator", coordinator) + store = object.__new__(PostgresSessionStore) + + async def database_must_not_be_touched(): + pytest.fail("revoked write reached the PostgreSQL pool") + + store._get_pool = database_must_not_be_touched + + with pytest.raises(OwnershipNotFoundError, match="not found"): + await store._async_get_or_create_session(session_id) + + +@pytest.mark.unit +def test_pdf_parser_rejects_page_budget_before_extraction(monkeypatch): + class FakeDocument: + closed = False + + def __len__(self): + return 2 + + def close(self): + self.closed = True + + document = FakeDocument() + monkeypatch.setitem(__import__("sys").modules, "pymupdf", SimpleNamespace(open=lambda *args, **kwargs: document)) + + with pytest.raises(PDFBudgetExceeded, match="page count"): + parse_pdf( + io.BytesIO(b"%PDF"), + limits=PDFParseLimits( + max_pages=1, + max_text_chars=1_000, + max_tables=1, + max_table_rows=1, + max_table_cells=1, + ), + ) + assert document.closed is True + + +@pytest.mark.unit +def test_pdf_worker_parses_real_document_in_isolated_process(): + pymupdf = pytest.importorskip("pymupdf") + document = pymupdf.open() + page = document.new_page() + page.insert_text((72, 72), "Revenue 100") + content = document.tobytes() + document.close() + + parsed = parse_pdf_sandboxed(content, "financial.pdf", timeout_seconds=10) + + assert parsed.filename == "financial.pdf" + assert parsed.page_count == 1 + assert "Revenue 100" in parsed.text + + +@pytest.mark.unit +@pytest.mark.asyncio +async def test_zero_hour_retention_runs_immediate_lifecycle_purge(monkeypatch): + class Coordinator: + cutoff = 0.0 + + async def purge_before(self, cutoff_timestamp: float): + self.cutoff = cutoff_timestamp + return [] + + coordinator = Coordinator() + monkeypatch.setenv("ZERO_DATA_RETENTION", "true") + monkeypatch.setenv("DATA_RETENTION_HOURS", "0") + monkeypatch.setattr(lifecycle_module, "get_lifecycle_coordinator", lambda: coordinator) + before = __import__("time").time() + + assert await ZeroDataRetention().purge_expired() == 0 + + after = __import__("time").time() + assert before <= coordinator.cutoff <= after + + +@pytest.mark.unit +@pytest.mark.asyncio +async def test_default_lifecycle_coordinator_purges_session_caches_and_team_artifact( + tmp_path, + monkeypatch, +): + session_id = "session-cache-purge" + + class Store: + present = True + + @staticmethod + def list_document_file_paths(_session_id: str): + return [] + + @classmethod + def delete_session(cls, _session_id: str): + cls.present = False + return True + + @classmethod + def session_exists(cls, _session_id: str): + return cls.present + + class Jobs: + @staticmethod + def purge_session(_session_id: str): + return {"jobs_revoked": 0, "threads_pending": 0} + + class Workflows: + @staticmethod + def purge_session(_session_id: str): + return 0 + + @staticmethod + def count_session_workflows(_session_id: str): + return 0 + + monkeypatch.setattr(lifecycle_module, "DATA_DIR", tmp_path) + monkeypatch.setattr("backend.storage.get_session_store", lambda: Store()) + monkeypatch.setattr("backend.services.workbook_jobs.get_workbook_job_manager", lambda: Jobs()) + monkeypatch.setattr(workflows_module, "get_workflow_engine", lambda: Workflows()) + monkeypatch.setattr(teams_module, "TEAMS_DIR", tmp_path / "teams") + team_manager = teams_module.TeamSessionManager() + shared_session = teams_module.SharedSession( + session_id=session_id, + team_id="team-1", + owner_id="alice", + title="Private model", + created_at="2026-08-09T00:00:00Z", + updated_at="2026-08-09T00:00:00Z", + ) + team_manager._sessions[session_id] = shared_session + team_manager._save_session(shared_session) + monkeypatch.setattr(teams_module, "_manager", team_manager) + monkeypatch.setattr(lifecycle_module, "_coordinator", None) + + diff_engine = diff_engine_module.get_diff_engine(session_id) + diff_engine.push(diff_engine.create_changeset([], prompt="private prompt")) + + receipt = await lifecycle_module.get_lifecycle_coordinator().purge_session(session_id) + + assert receipt.state is LifecycleState.VERIFIED + assert receipt.registered_purgers["diff_engine"] == {"engines_evicted": 1} + assert receipt.registered_purgers["team_session"] == { + "memory_sessions_evicted": 1, + "files_deleted": 1, + } + assert diff_engine_module.purge_diff_engine(session_id) == {"engines_evicted": 0} + assert session_id not in team_manager._sessions + assert not (teams_module.TEAMS_DIR / f"session_{session_id}.json").exists() diff --git a/backend/tests/test_security_remediation_lane_a.py b/backend/tests/test_security_remediation_lane_a.py new file mode 100644 index 0000000..ed73e95 --- /dev/null +++ b/backend/tests/test_security_remediation_lane_a.py @@ -0,0 +1,345 @@ +from __future__ import annotations + +import base64 +import json +import sqlite3 +import time + +import jwt +import pytest +from cryptography.hazmat.primitives.asymmetric import rsa + +from backend.api.security.oidc import OIDCAuthenticator, OIDCConfig, OIDCValidationError +from backend.excel import formula_analyzer, workbook_worker +from backend.excel import formula_engine_adapters +from backend.excel.fingerprint import generate_fingerprint +from backend.excel.formula_analyzer import analyze_formula_payload, analyze_formula_payload_best +from backend.excel.formula_parser import expand_range +from backend.excel.workbook_files import WorkbookExportService +from backend.session_store import SessionStore +from backend.storage.migrations import _enterprise_oidc_identity_bindings + + +def _oidc_token(private_key, *, subject: str, email: str = "alice@example.com") -> str: + now = int(time.time()) + return jwt.encode( + { + "iss": "https://identity.example.com/tenant/v2.0", + "aud": "api://excel-agent", + "sub": subject, + "email": email, + "tid": "tenant-a", + "iat": now, + "exp": now + 300, + "amr": ["pwd", "mfa"], + }, + private_key, + algorithm="RS256", + headers={"kid": "test-key"}, + ) + + +def _oidc_authenticator(public_key, loader) -> OIDCAuthenticator: + return OIDCAuthenticator( + OIDCConfig( + enabled=True, + issuer="https://identity.example.com/tenant/v2.0", + audience="api://excel-agent", + jwks_url="https://identity.example.com/tenant/discovery/v2.0/keys", + allowed_tenants=frozenset({"tenant-a"}), + algorithms=("RS256",), + require_mfa=True, + ), + user_loader=loader, + signing_key_resolver=lambda _token: public_key, + ) + + +def test_oidc_uses_immutable_identity_and_ignores_mutable_email_claim(): + private_key = rsa.generate_private_key(public_exponent=65537, key_size=2048) + issuer = "https://identity.example.com/tenant/v2.0" + user = { + "user_id": "user-alice", + "email": "alice-contact@example.com", + "name": "Alice", + "role": "admin", + "status": "active", + "oidc_issuer": issuer, + "oidc_subject": "immutable-alice", + "metadata": {"oidc_issuer": issuer, "oidc_subject": "immutable-alice"}, + } + lookups: list[tuple[str, str]] = [] + + def loader(candidate_issuer: str, candidate_subject: str): + lookups.append((candidate_issuer, candidate_subject)) + if (candidate_issuer, candidate_subject) == (issuer, "immutable-alice"): + return user + return None + + authenticator = _oidc_authenticator(private_key.public_key(), loader) + + actor = authenticator.authenticate_sync( + _oidc_token(private_key, subject="immutable-alice", email="changed-at-idp@example.com") + ) + + assert actor["user_id"] == "user-alice" + assert actor["email"] == "alice-contact@example.com" + assert lookups == [(issuer, "immutable-alice")] + + with pytest.raises(OIDCValidationError, match="identity is not provisioned"): + authenticator.authenticate_sync( + _oidc_token(private_key, subject="different-subject", email="alice-contact@example.com") + ) + + +def test_oidc_rejects_an_unbound_legacy_row_even_if_a_loader_returns_it(): + private_key = rsa.generate_private_key(public_exponent=65537, key_size=2048) + authenticator = _oidc_authenticator( + private_key.public_key(), + lambda _issuer, _subject: { + "user_id": "legacy-admin", + "email": "legacy@example.com", + "name": "Legacy", + "role": "admin", + "status": "active", + "metadata": {}, + }, + ) + + with pytest.raises(OIDCValidationError, match="identity binding is incomplete"): + authenticator.authenticate_sync(_oidc_token(private_key, subject="replacement-subject")) + + +def test_sqlite_enterprise_user_binding_is_required_unique_and_immutable(tmp_path): + store = SessionStore(tmp_path / "sessions.db") + issuer = "https://identity.example.com/tenant/v2.0" + + created = store.upsert_enterprise_user( + user_id="alice", + email="alice@example.com", + name="Alice", + role="admin", + oidc_issuer=issuer, + oidc_subject="subject-alice", + ) + assert created["oidc_issuer"] == issuer + assert created["oidc_subject"] == "subject-alice" + assert store.get_enterprise_user_by_oidc_identity(issuer, "subject-alice")["user_id"] == "alice" + + with pytest.raises(ValueError, match="requires an immutable OIDC issuer and subject"): + store.upsert_enterprise_user( + user_id="unbound", + email="unbound@example.com", + name="Unbound", + role="analyst", + ) + + with pytest.raises(ValueError, match="already bound"): + store.upsert_enterprise_user( + user_id="mallory", + email="mallory@example.com", + name="Mallory", + role="analyst", + oidc_issuer=issuer, + oidc_subject="subject-alice", + ) + + with pytest.raises(ValueError, match="immutable"): + store.upsert_enterprise_user( + user_id="alice", + email="alice@example.com", + name="Alice", + role="admin", + oidc_issuer=issuer, + oidc_subject="replacement-subject", + ) + + updated = store.upsert_enterprise_user( + user_id="alice", + email="new-contact@example.com", + name="Alice", + role="admin", + ) + assert updated["email"] == "new-contact@example.com" + assert updated["oidc_subject"] == "subject-alice" + + migration_markers = { + "legacy_oidc_issuer": "https://legacy.example.com", + "legacy_oidc_subject": "legacy-subject", + "oidc_binding_state": "partial", + } + store.upsert_enterprise_user( + user_id="legacy-review", + email="legacy-review@example.com", + name="Legacy Review", + role="analyst", + status="binding_required", + metadata=migration_markers, + ) + preserved_none = store.upsert_enterprise_user( + user_id="legacy-review", + email="legacy-review@example.com", + name="Legacy Review", + role="analyst", + metadata=None, + ) + preserved_empty = store.upsert_enterprise_user( + user_id="legacy-review", + email="legacy-review@example.com", + name="Legacy Review", + role="analyst", + metadata={}, + ) + assert preserved_none["metadata"] == migration_markers + assert preserved_empty["metadata"] == migration_markers + + +def test_sqlite_legacy_unbound_and_ambiguous_bindings_migrate_explicitly(): + conn = sqlite3.connect(":memory:") + conn.execute( + """ + CREATE TABLE enterprise_users ( + user_id TEXT PRIMARY KEY, + email TEXT NOT NULL UNIQUE, + name TEXT NOT NULL, + role TEXT NOT NULL, + status TEXT NOT NULL DEFAULT 'active', + mfa_enabled INTEGER NOT NULL DEFAULT 0, + metadata_json TEXT NOT NULL DEFAULT '{}', + created_at REAL NOT NULL, + updated_at REAL NOT NULL + ) + """ + ) + rows = [ + ("unbound", "unbound@example.com", {}), + ("duplicate-a", "duplicate-a@example.com", {"oidc_issuer": "issuer", "oidc_subject": "subject"}), + ("duplicate-b", "duplicate-b@example.com", {"oidc_issuer": "issuer", "oidc_subject": "subject"}), + ("partial", "partial@example.com", {"oidc_issuer": "issuer"}), + ] + for user_id, email, metadata in rows: + conn.execute( + "INSERT INTO enterprise_users " + "(user_id, email, name, role, status, metadata_json, created_at, updated_at) " + "VALUES (?, ?, ?, 'analyst', 'active', ?, 1, 1)", + (user_id, email, user_id, json.dumps(metadata)), + ) + + _enterprise_oidc_identity_bindings(conn) + + migrated = { + row[0]: (row[1], json.loads(row[2]), row[3], row[4]) + for row in conn.execute( + "SELECT user_id, status, metadata_json, oidc_issuer, oidc_subject FROM enterprise_users" + ) + } + assert migrated["unbound"][0] == "api_key_only" + assert migrated["unbound"][1]["oidc_binding_state"] == "legacy_unbound" + for user_id in ("duplicate-a", "duplicate-b", "partial"): + assert migrated[user_id][0] == "binding_required" + assert migrated[user_id][2:] == ("", "") + + with pytest.raises(sqlite3.IntegrityError, match="requires an immutable OIDC"): + conn.execute( + "INSERT INTO enterprise_users " + "(user_id, email, name, role, status, metadata_json, oidc_issuer, oidc_subject, created_at, updated_at) " + "VALUES ('new-unbound', 'new@example.com', 'New', 'analyst', 'active', '{}', '', '', 1, 1)" + ) + conn.close() + + +def test_formula_worker_rejection_is_terminal(monkeypatch): + monkeypatch.setattr(formula_analyzer, "FORMULA_ENGINE", "hyperformula") + monkeypatch.setattr( + workbook_worker, + "run_worker", + lambda *_args, **_kwargs: workbook_worker.WorkerResult( + ok=False, + error="FORMULA_REFERENCE_BUDGET_EXCEEDED", + ), + ) + + with pytest.raises(RuntimeError, match="FORMULA_REFERENCE_BUDGET_EXCEEDED"): + analyze_formula_payload_best(formulas=[{"address": "A1", "formula": "=SUM(A1:Z500)"}]) + + +def test_python_formula_profile_enforces_excel_and_reference_budgets(monkeypatch): + monkeypatch.setattr(formula_analyzer, "FORMULA_ENGINE", "python_fallback") + + with pytest.raises(ValueError, match="reference budget"): + analyze_formula_payload_best(formulas=[{"address": "A1", "formula": "=SUM(A1:Z500)"}]) + with pytest.raises(ValueError, match="outside Excel worksheet bounds"): + expand_range("A1", "XFE1") + + result = analyze_formula_payload(formulas=[{"address": "B1", "formula": "=SUM(A1:A10)"}]) + assert set(result.formulas[0].precedents) == {f"A{row}" for row in range(1, 11)} + + +def test_hyperformula_adapter_requires_approved_injected_license(monkeypatch): + monkeypatch.setattr(formula_engine_adapters, "FORMULA_ENGINE", "hyperformula") + monkeypatch.setattr(formula_engine_adapters, "HYPERFORMULA_DISTRIBUTED", True) + monkeypatch.setattr(formula_engine_adapters, "HYPERFORMULA_ENABLED", True) + monkeypatch.setattr(formula_engine_adapters, "HYPERFORMULA_LICENSE_APPROVED", True) + monkeypatch.setattr(formula_engine_adapters, "HYPERFORMULA_LICENSE_KEY", "") + disabled = formula_engine_adapters.HyperFormulaAdapter().status() + assert disabled.enabled is False + assert "commercial HYPERFORMULA_LICENSE_KEY" in disabled.reason + + monkeypatch.setattr(formula_engine_adapters, "HYPERFORMULA_LICENSE_KEY", "gpl-v3") + noncommercial = formula_engine_adapters.HyperFormulaAdapter().status() + assert noncommercial.enabled is False + + monkeypatch.setattr(formula_engine_adapters, "HYPERFORMULA_LICENSE_KEY", "injected-commercial-key") + enabled = formula_engine_adapters.HyperFormulaAdapter().status() + assert enabled.enabled is True + assert enabled.license_mode == "explicit-approved-license" + + +def test_export_worker_rejection_never_calls_openpyxl_fallback(monkeypatch): + service = WorkbookExportService() + monkeypatch.setattr( + workbook_worker, + "run_worker", + lambda *_args, **_kwargs: workbook_worker.WorkerResult(ok=False, error="PAYLOAD_LIMIT_EXCEEDED"), + ) + monkeypatch.setattr( + service, + "_export_python", + lambda *_args, **_kwargs: pytest.fail("openpyxl fallback must not run"), + ) + + with pytest.raises(RuntimeError, match="PAYLOAD_LIMIT_EXCEEDED"): + service.export({"Summary": [["Metric", "Value"], ["Revenue", 100]]}) + + +def test_export_worker_success_remains_supported(monkeypatch): + monkeypatch.setattr( + workbook_worker, + "run_worker", + lambda *_args, **_kwargs: workbook_worker.WorkerResult( + ok=True, + result={"content_base64": base64.b64encode(b"bounded-xlsx").decode("ascii")}, + ), + ) + + assert WorkbookExportService().export({"Summary": [["ok"]]}) == b"bounded-xlsx" + + +def test_fingerprint_rejects_oversized_snapshot_before_block_detection(): + snapshot = { + f"A{row}": {"value": "Assumptions", "formula": None, "type": "text"} + for row in range(1, 3001) + } + + with pytest.raises(ValueError, match="Fingerprint cell budget"): + generate_fingerprint("Sheet1", snapshot) + + valid = generate_fingerprint( + "Sheet1", + { + "A1": {"value": "Assumptions", "formula": None, "type": "text"}, + "B2": {"value": 10, "formula": None, "type": "number"}, + "B3": {"value": 20, "formula": "=B2*2", "type": "number"}, + }, + ) + assert valid.total_cells == 3 diff --git a/backend/tests/test_shortcut_parity.py b/backend/tests/test_shortcut_parity.py index e2932d1..408127e 100644 --- a/backend/tests/test_shortcut_parity.py +++ b/backend/tests/test_shortcut_parity.py @@ -1,11 +1,14 @@ """Shortcut-parity rebuild tests.""" +from uuid import uuid4 + from fastapi.testclient import TestClient from backend.config import API_KEY from backend.main import app -from backend.models.operations import WorkbookOperation +from backend.models.operations import OperationPlan, WorkbookOperation from backend.services.operation_validation import validate_workbook_operations +from backend.storage import get_session_store def _headers(): @@ -14,7 +17,7 @@ def _headers(): def test_document_upload_to_typed_cited_plan_flow(): client = TestClient(app) - session_id = "shortcut-doc-flow" + session_id = f"shortcut-doc-flow-{uuid4().hex}" upload = client.post( "/api/v1/documents/upload", headers=_headers(), @@ -48,7 +51,7 @@ def test_document_upload_to_typed_cited_plan_flow(): def test_document_upload_to_v2_plan_flow_is_mainline_ready(): client = TestClient(app) - session_id = "shortcut-doc-v2-flow" + session_id = f"shortcut-doc-v2-flow-{uuid4().hex}" upload = client.post( "/api/v1/documents/upload", headers=_headers(), @@ -76,45 +79,191 @@ def test_document_upload_to_v2_plan_flow_is_mainline_ready(): assert plan["metadata"]["inspection_required"] is True -def test_apply_result_marks_plan_applied_and_audited(): +def test_source_refresh_reuses_uploaded_session_documents(): client = TestClient(app) - session_id = "shortcut-apply-flow" - plan_resp = client.post( - "/api/v1/tasks/plan", + session_id = f"shortcut-source-refresh-{uuid4().hex}" + upload = client.post( + "/api/v1/documents/upload", headers=_headers(), - json={"prompt": "Build a DCF model", "session_id": session_id, "template_id": "dcf_model"}, + data={"session_id": session_id}, + files={ + "file": ( + "financials.csv", + b"Metric,Value\nRevenue,100000000\nEBITDA,30000000\n", + "text/csv", + ) + }, ) - assert plan_resp.status_code == 200 - plan = plan_resp.json() + assert upload.status_code == 200 + document_id = upload.json()["document_id"] + workbook_context = { + "active_sheet": "Sheet1", + "cell_snapshot": {"A1": {"value": "Revenue"}}, + } + + refresh = client.post( + "/api/v1/source-evidence/v2", + headers=_headers(), + json={ + "session_id": session_id, + "workbook_context": workbook_context, + "downstream_outputs": ["Sheet1!B1"], + }, + ) + + assert refresh.status_code == 200 + graph = refresh.json() + assert graph["documentIds"] == [document_id] + assert graph["metadata"]["documentCount"] == 1 + assert graph["metadata"]["documentSource"] == "session_store" + assert graph["evidence"] + assert graph["cellToSource"]["Sheet1!B1"] == [document_id] + assert graph["citationCoverage"]["evidence_count"] > 0 + + v3_refresh = client.post( + "/api/v1/source-evidence/v3", + headers=_headers(), + json={ + "session_id": session_id, + "document_ids": [document_id], + "workbook_context": workbook_context, + "downstream_outputs": ["Sheet1!B1"], + }, + ) + + assert v3_refresh.status_code == 200 + v3_graph = v3_refresh.json() + assert v3_graph["metadata"]["documentCount"] == 1 + assert v3_graph["metadata"]["documentSource"] == "request" + assert v3_graph["sourceBackedAssumptionImpact"]["downstreamOutputs"] == ["Sheet1!B1"] + assert v3_graph["refreshPlan"] + + +def test_apply_result_marks_plan_applied_and_audited(): + client = TestClient(app) + session_id = f"shortcut-apply-{uuid4().hex}" + store = get_session_store() + store.get_or_create_session(session_id) + plan = OperationPlan( + id=f"plan-{session_id}", + session_id=session_id, + prompt="Write a proven test value", + operations=[ + { + "id": "op-shortcut-write", + "type": "write_cell", + "sheet": "Sheet1", + "range": "A1", + "value": 42, + } + ], + ) + store.save_operation_plan(plan.model_dump(mode="json", by_alias=True)) + pre_context = { + "active_sheet": "Sheet1", + "sheet_names": ["Sheet1"], + "sheet_metadata": [{"name": "Sheet1", "usedRange": "Sheet1!A1"}], + "cell_snapshot": { + "Sheet1!A1": { + "sheet": "Sheet1", + "address": "A1", + "value": "", + "formula": None, + "blank": True, + } + }, + "objects": {"charts": [], "tables": [], "pivots": [], "comments": []}, + "rollback_snapshot": {"complete": True, "errors": []}, + } approve = client.post( "/api/v1/tasks/apply", headers=_headers(), json={ - "plan_id": plan["id"], + "plan_id": plan.id, "session_id": session_id, - "workbook_fingerprint": "Sheet1|A1|0|0|0", - "workbook_state": {"cell_snapshot": {}}, + "workbook_fingerprint": "pre-fingerprint", + "workbook_state": pre_context, }, ) assert approve.status_code == 200 - token = approve.json()["apply_token"] + approved = approve.json() + token = approved["apply_token"] + operations = approved["operations"] result = client.post( "/api/v1/tasks/apply/result", headers=_headers(), json={ - "plan_id": plan["id"], + "plan_id": plan.id, "session_id": session_id, "apply_token": token, + "apply_attempt_id": approved["apply_attempt_id"], "status": "applied", - "office_apply_result": {"applied": len(plan["operations"])}, + "pre_apply_workbook_state": pre_context, + "pre_apply_workbook_fingerprint": "pre-fingerprint", + "pre_host_snapshot": { + "complete": True, + "truncated": False, + "errors": [], + "fingerprint": "pre-host", + }, + "post_apply_workbook_state": { + "active_sheet": "Sheet1", + "sheet_names": ["Sheet1"], + "cell_snapshot": { + "Sheet1!A1": { + "sheet": "Sheet1", + "address": "A1", + "value": 42, + "formula": None, + "blank": False, + } + }, + "rollback_snapshot": {"complete": True, "errors": []}, + }, + "post_apply_workbook_fingerprint": "post-fingerprint", + "post_host_snapshot": { + "complete": True, + "truncated": False, + "errors": [], + "fingerprint": "post-host", + }, + "office_apply_result": { + "applied": len(operations), + "results": [ + { + "operationId": op["id"], + "type": op["type"], + "status": "success", + "dirtyRanges": [ + { + "sheet": op.get("sheet") or "Sheet1", + "range": op.get("range") or "A1", + } + ], + "rollbackSnapshot": { + "proofClass": "values_formats", + "restoreOperations": [ + { + "type": "rollback_restore", + "sheet": "Sheet1", + "range": "A1", + "values": [[""]], + "numberFormats": [["General"]], + } + ], + }, + } + for op in operations + ], + }, }, ) assert result.status_code == 200 timeline = client.get(f"/api/v1/sessions/{session_id}/timeline", headers=_headers()).json() - applied_plan = next(item for item in timeline["plans"] if item["plan_id"] == plan["id"]) + applied_plan = next(item for item in timeline["plans"] if item["plan_id"] == plan.id) assert applied_plan["status"] == "applied" assert any(event["event_type"] == "plan_applied" for event in timeline["audit_events"]) @@ -145,6 +294,7 @@ def test_enterprise_users_teams_and_api_keys_are_sqlite_backed(): "email": "analyst1@example.com", "name": "Analyst One", "role": "analyst", + "status": "api_key_only", "mfa_enabled": True, }, ) @@ -158,6 +308,19 @@ def test_enterprise_users_teams_and_api_keys_are_sqlite_backed(): ) assert key.status_code == 200 assert key.json()["api_key"].startswith("fai_") + enterprise_key = key.json()["api_key"] + assert get_session_store().get_enterprise_user_by_api_key(enterprise_key)["user_id"] == "analyst-1" + + revoke = client.delete(f"/api/v1/enterprise/api-keys/{key.json()['key_id']}", headers=_headers()) + assert revoke.status_code == 200 + assert get_session_store().get_enterprise_user_by_api_key(enterprise_key) is None + + key = client.post( + "/api/v1/enterprise/users/analyst-1/api-keys", + headers=_headers(), + json={"label": "status-test"}, + ) + assert key.status_code == 200 team = client.post( "/api/v1/enterprise/teams", @@ -178,6 +341,299 @@ def test_enterprise_users_teams_and_api_keys_are_sqlite_backed(): assert dashboard.status_code == 200 assert dashboard.json()["summary"]["users"] >= 1 + enterprise_key = key.json()["api_key"] + assert get_session_store().get_enterprise_user_by_api_key(enterprise_key)["user_id"] == "analyst-1" + suspended = client.post( + "/api/v1/enterprise/users", + headers=_headers(), + json={ + "user_id": "analyst-1", + "email": "analyst1@example.com", + "name": "Analyst One", + "role": "analyst", + "status": "suspended", + }, + ) + assert suspended.status_code == 200 + assert get_session_store().get_enterprise_user_by_api_key(enterprise_key) is None + + +def test_enterprise_api_keys_enforce_required_mfa_policy(): + client = TestClient(app) + user_id = f"mfa-required-{uuid4().hex}" + user = client.post( + "/api/v1/enterprise/users", + headers=_headers(), + json={ + "user_id": user_id, + "email": f"{user_id}@example.com", + "name": "MFA Required", + "role": "analyst", + "status": "api_key_only", + "mfa_enabled": False, + }, + ) + assert user.status_code == 200 + key = client.post( + f"/api/v1/enterprise/users/{user_id}/api-keys", + headers=_headers(), + json={"label": "mfa-test"}, + ) + assert key.status_code == 200 + enterprise_key = key.json()["api_key"] + + try: + policy = client.put( + "/api/v1/enterprise/sso/config", + headers=_headers(), + json={"config": {"require_mfa": True}}, + ) + assert policy.status_code == 200 + assert get_session_store().get_enterprise_user_by_api_key(enterprise_key) is None + + enabled = client.post( + "/api/v1/enterprise/users", + headers=_headers(), + json={ + "user_id": user_id, + "email": f"{user_id}@example.com", + "name": "MFA Required", + "role": "analyst", + "mfa_enabled": True, + }, + ) + assert enabled.status_code == 200 + assert get_session_store().get_enterprise_user_by_api_key(enterprise_key)["user_id"] == user_id + finally: + client.put( + "/api/v1/enterprise/sso/config", + headers=_headers(), + json={"config": {"require_mfa": False}}, + ) + + +def test_enterprise_sso_and_security_policy_are_effective_across_surfaces(): + client = TestClient(app) + user_id = f"effective-policy-{uuid4().hex}" + client.post( + "/api/v1/enterprise/users", + headers=_headers(), + json={ + "user_id": user_id, + "email": f"{user_id}@example.com", + "name": "Effective Policy", + "role": "analyst", + "status": "api_key_only", + "mfa_enabled": False, + }, + ) + key = client.post( + f"/api/v1/enterprise/users/{user_id}/api-keys", + headers=_headers(), + json={"label": "effective-policy"}, + ) + assert key.status_code == 200 + enterprise_key = key.json()["api_key"] + + try: + client.put( + "/api/v1/enterprise/security-policy", + headers=_headers(), + json={"config": {}}, + ) + sso = client.put( + "/api/v1/enterprise/sso/config", + headers=_headers(), + json={ + "config": { + "enabled": True, + "provider": "okta", + "idp_metadata_url": "https://idp.example.com/metadata.xml", + "require_mfa": True, + } + }, + ) + assert sso.status_code == 200 + sso_get = client.get("/api/v1/enterprise/sso/config", headers=_headers()) + assert sso_get.status_code == 200 + sso_body = sso_get.json() + assert sso_body["policySource"] == "enterprise_config" + assert sso_body["config"]["provider"] == "okta" + assert sso_body["config"]["require_mfa"] is True + assert sso_body["config"]["idp_configured"] is True + assert get_session_store().get_enterprise_user_by_api_key(enterprise_key) is None + + security_policy = client.get("/api/v1/enterprise/security-policy", headers=_headers()) + assert security_policy.status_code == 200 + policy_body = security_policy.json() + assert policy_body["policySource"] == "derived" + assert policy_body["config"]["require_mfa"] is True + assert policy_body["config"]["api_key_mfa_gate"] is True + assert policy_body["derivedFrom"]["sso"] is True + + dashboard = client.get("/api/v1/enterprise/dashboard", headers=_headers()).json() + assert dashboard["sso"]["config"]["provider"] == "okta" + assert dashboard["security_policy"]["config"]["requireMfa"] is True + + general_security = client.get("/api/v1/security/config", headers=_headers()).json() + assert general_security["sso"]["policySource"] == "enterprise_config" + assert general_security["security_policy"]["config"]["require_mfa"] is True + + client.put( + "/api/v1/enterprise/sso/config", + headers=_headers(), + json={"config": {"require_mfa": False}}, + ) + direct_policy = client.put( + "/api/v1/enterprise/security-policy", + headers=_headers(), + json={"config": {"require_mfa": True, "allowed_auth_sources": ["enterprise_api_key"]}}, + ) + assert direct_policy.status_code == 200 + direct_body = direct_policy.json()["effective"] + assert direct_body["policySource"] == "enterprise_config" + assert direct_body["derivedFrom"]["security_policy"] is True + assert direct_body["config"]["allowed_auth_sources"] == ["enterprise_api_key"] + assert get_session_store().get_enterprise_user_by_api_key(enterprise_key) is None + finally: + client.put( + "/api/v1/enterprise/sso/config", + headers=_headers(), + json={"config": {"require_mfa": False}}, + ) + client.put( + "/api/v1/enterprise/security-policy", + headers=_headers(), + json={"config": {}}, + ) + + +def test_enterprise_admin_mutations_are_actor_audited(): + client = TestClient(app) + user_id = f"audited-admin-{uuid4().hex}" + audit_session_id = f"enterprise-admin-audit-{uuid4().hex}" + trace_id = f"trace-{user_id}" + headers = { + **_headers(), + "X-Session-Id": audit_session_id, + "X-Trace-Id": trace_id, + } + + user = client.post( + "/api/v1/enterprise/users", + headers=headers, + json={ + "user_id": user_id, + "email": f"{user_id}@example.com", + "name": "Audited Admin", + "role": "admin", + "status": "api_key_only", + "mfa_enabled": True, + }, + ) + assert user.status_code == 200 + + key = client.post( + f"/api/v1/enterprise/users/{user_id}/api-keys", + headers=headers, + json={"label": "audited-key"}, + ) + assert key.status_code == 200 + revoke = client.delete(f"/api/v1/enterprise/api-keys/{key.json()['key_id']}", headers=headers) + assert revoke.status_code == 200 + + team = client.post( + "/api/v1/enterprise/teams", + headers=headers, + json={"name": "Audited Banking"}, + ) + assert team.status_code == 200 + member = client.post( + f"/api/v1/enterprise/teams/{team.json()['team']['team_id']}/members", + headers=headers, + json={"user_id": user_id, "role": "owner"}, + ) + assert member.status_code == 200 + + assert client.put( + "/api/v1/enterprise/sso/config", + headers=headers, + json={"config": {"require_mfa": False, "provider": "okta"}}, + ).status_code == 200 + assert client.put( + "/api/v1/enterprise/provider-policy", + headers=headers, + json={"config": {"allow_cloud": False, "allow_local": True}}, + ).status_code == 200 + assert client.put( + "/api/v1/enterprise/security-policy", + headers=headers, + json={"config": {"require_mfa": False, "admin_audit_required": True}}, + ).status_code == 200 + assert client.put( + "/api/v1/enterprise/retention-policy", + headers=headers, + json={"config": {"enabled": True, "retention_hours": 24}}, + ).status_code == 200 + assert client.post( + "/api/v1/enterprise/retention/purge", + headers=headers, + json={"retention_hours": 1}, + ).status_code == 200 + + events = [ + event + for event in get_session_store().list_enterprise_admin_audits(limit=100) + if event["payload"]["request"]["trace_id"] == trace_id + ] + actions = {event["payload"]["action"]: event for event in events} + assert { + "upsert_user", + "create_api_key", + "revoke_api_key", + "create_team", + "add_team_member", + "set_sso_config", + "set_provider_policy", + "set_security_policy", + "set_retention_policy", + "purge_expired_sessions", + }.issubset(actions) + for event in actions.values(): + payload = event["payload"] + assert event["event_type"].startswith("enterprise_admin.") + assert payload["actor"]["user_id"] == "admin" + assert payload["actor"]["auth_source"] == "static_api_key" + assert payload["request"]["trace_id"] == trace_id + assert payload["status"] == "success" + assert actions["create_api_key"]["payload"]["metadata"]["label"] == "audited-key" + assert actions["add_team_member"]["payload"]["metadata"]["user_id"] == user_id + + session_export = get_session_store().export_audit_csv(audit_session_id) + assert "admin@localhost" not in session_export + assert "static_api_key" not in session_export + + client.put( + "/api/v1/enterprise/sso/config", + headers=headers, + json={"config": {"require_mfa": False}}, + ) + client.put( + "/api/v1/enterprise/provider-policy", + headers=headers, + json={"config": {}}, + ) + client.put( + "/api/v1/enterprise/security-policy", + headers=headers, + json={"config": {}}, + ) + client.put( + "/api/v1/enterprise/retention-policy", + headers=headers, + json={"config": {}}, + ) + def test_validation_blocks_protected_sheet_conflict(): op = WorkbookOperation(type="write_cell", sheet="Locked", range="B2", value=123) diff --git a/backend/tests/test_snapshot_integrity.py b/backend/tests/test_snapshot_integrity.py new file mode 100644 index 0000000..17337ec --- /dev/null +++ b/backend/tests/test_snapshot_integrity.py @@ -0,0 +1,68 @@ +"""Snapshot integrity hashes — verify record/get cycle preserves state hash.""" + +from __future__ import annotations + +import pytest + +from backend.session_store import SessionStore + + +@pytest.mark.integration +class TestSnapshotIntegrity: + @pytest.fixture + def store(self, tmp_path) -> SessionStore: + return SessionStore(db_path=tmp_path / "snap.db") + + def test_record_and_get_returns_state_hash(self, store: SessionStore) -> None: + store.get_or_create_session("sess_alpha") + snap_id = store.record_snapshot( + session_id="sess_alpha", + workbook_state={"sheet": "Model", "value": 42}, + ) + retrieved = store.get_snapshot(snap_id) + assert retrieved is not None + assert retrieved["state_verified"] is True + assert retrieved["state_hash"] + assert retrieved["workbook_state"] == {"sheet": "Model", "value": 42} + + def test_hash_is_canonical_across_key_order(self, store: SessionStore) -> None: + # Two snapshots with same contents in different key insertion order + # should produce the same hash (canonical sort_keys serialization). + store.get_or_create_session("sess_x") + snap_a = store.record_snapshot( + session_id="sess_x", + workbook_state={"a": 1, "b": 2, "c": 3}, + ) + snap_b = store.record_snapshot( + session_id="sess_x", + workbook_state={"c": 3, "b": 2, "a": 1}, + ) + retrieved_a = store.get_snapshot(snap_a) + retrieved_b = store.get_snapshot(snap_b) + assert retrieved_a is not None and retrieved_b is not None + assert retrieved_a["state_hash"] == retrieved_b["state_hash"] + + def test_get_missing_snapshot_returns_none(self, store: SessionStore) -> None: + assert store.get_snapshot("does_not_exist") is None + + def test_tampering_detected(self, store: SessionStore) -> None: + # Round-trip a snapshot, then tamper with the DB row and re-read; the + # state_verified field must flip to False so callers can refuse the + # rollback (or alarm the operator). + import sqlite3 + + store.get_or_create_session("sess_tamper") + snap_id = store.record_snapshot( + session_id="sess_tamper", + workbook_state={"key": "original"}, + ) + with sqlite3.connect(store.db_path) as conn: + import json as _json + tampered = _json.dumps({"key": "modified"}, sort_keys=True, separators=(",", ":")) + conn.execute( + "UPDATE workbook_snapshots SET workbook_state = ? WHERE snapshot_id = ?", + (tampered, snap_id), + ) + retrieved = store.get_snapshot(snap_id) + assert retrieved is not None + assert retrieved["state_verified"] is False diff --git a/backend/tests/test_spreadsheet_routes.py b/backend/tests/test_spreadsheet_routes.py new file mode 100644 index 0000000..4281bf1 --- /dev/null +++ b/backend/tests/test_spreadsheet_routes.py @@ -0,0 +1,623 @@ +"""Route-level tests for the local-first browser spreadsheet APIs.""" + +from __future__ import annotations + +import asyncio + +from fastapi.testclient import TestClient + +from backend.config import API_KEY +from backend.excel.workbook_worker import WorkerResult, WorkbookWorkerUnavailable +from backend.main import app + + +def _headers() -> dict[str, str]: + return {"X-API-Key": API_KEY} + + +def test_spreadsheet_worker_routes_delegate_to_thread_boundary(monkeypatch): + handoffs: list[tuple[str, str, dict]] = [] + + results = { + "open-workbook-v1": WorkerResult( + ok=True, + result={"schemaVersion": "2026-05-workbook-session-v1", "workbookId": "wb_open"}, + ), + "export-workbook-v1": WorkerResult(ok=True, result={"content_base64": "VGVzdA=="}), + "roundtrip-qa-v2": WorkerResult( + ok=True, + result={"schemaVersion": "2026-05-spreadsheet-roundtrip-qa-v2", "status": "passed"}, + ), + } + + def fake_run_worker(command: str, payload: dict) -> WorkerResult: + return results[command] + + async def fake_to_thread(fn, command: str, payload: dict): + handoffs.append((fn.__name__, command, payload)) + return fn(command, payload) + + monkeypatch.setattr("backend.excel.workbook_worker.run_worker", fake_run_worker) + monkeypatch.setattr("backend.api.routes.tasks.asyncio.to_thread", fake_to_thread) + + client = TestClient(app) + assert client.post( + "/api/v1/spreadsheet/open", + headers=_headers(), + json={"filename": "open.xlsx", "contentBase64": "UEsDBAo="}, + ).status_code == 200 + assert client.post( + "/api/v1/spreadsheet/export", + headers=_headers(), + json={"session": {"workbookId": "wb_export"}}, + ).status_code == 200 + assert client.post( + "/api/v1/spreadsheet/roundtrip-qa", + headers=_headers(), + json={"session": {"workbookId": "wb_qa"}}, + ).status_code == 200 + + assert [command for _, command, _ in handoffs] == [ + "open-workbook-v1", + "export-workbook-v1", + "roundtrip-qa-v2", + ] + + +def test_spreadsheet_worker_route_timeout_returns_504(monkeypatch): + async def fake_to_thread(fn, command: str, payload: dict): + await asyncio.sleep(0.01) + return WorkerResult( + ok=True, + result={"schemaVersion": "2026-05-workbook-session-v1", "workbookId": "wb_late"}, + ) + + monkeypatch.setattr("backend.config.WORKBOOK_WORKER_TIMEOUT_SECONDS", 0.001) + monkeypatch.setattr("backend.api.routes.tasks.asyncio.to_thread", fake_to_thread) + + response = TestClient(app).post( + "/api/v1/spreadsheet/open", + headers=_headers(), + json={"filename": "slow.xlsx", "contentBase64": "UEsDBAo="}, + ) + + assert response.status_code == 504 + assert response.json()["detail"] == "Spreadsheet open timed out" + + +def test_spreadsheet_worker_unavailability_returns_503(monkeypatch): + def fake_run_worker(command: str, payload: dict) -> WorkerResult: + raise WorkbookWorkerUnavailable("Workbook worker is disabled or not installed") + + monkeypatch.setattr("backend.excel.workbook_worker.run_worker", fake_run_worker) + + response = TestClient(app).post( + "/api/v1/spreadsheet/open", + headers=_headers(), + json={"filename": "unavailable.xlsx", "contentBase64": "UEsDBAo="}, + ) + + assert response.status_code == 503 + assert "worker unavailable" in response.json()["detail"] + assert "disabled or not installed" in response.json()["detail"] + + +def test_spreadsheet_routes_reject_malformed_payloads_before_worker(monkeypatch): + calls: list[tuple[str, dict]] = [] + + def fake_run_worker(command: str, payload: dict) -> WorkerResult: + calls.append((command, payload)) + return WorkerResult(ok=True, result={}) + + monkeypatch.setattr("backend.excel.workbook_worker.run_worker", fake_run_worker) + client = TestClient(app) + + open_response = client.post( + "/api/v1/spreadsheet/open", + headers=_headers(), + json={"filename": "missing-content.xlsx"}, + ) + export_response = client.post( + "/api/v1/spreadsheet/export", + headers=_headers(), + json={"session": ["not", "a", "mapping"]}, + ) + qa_response = client.post( + "/api/v1/spreadsheet/roundtrip-qa", + headers=_headers(), + json={"session": "not-a-mapping"}, + ) + + assert open_response.status_code == 422 + assert export_response.status_code == 422 + assert qa_response.status_code == 422 + assert calls == [] + + +def test_spreadsheet_open_forwards_alias_fields_to_worker(monkeypatch): + calls: list[tuple[str, dict]] = [] + + def fake_run_worker(command: str, payload: dict) -> WorkerResult: + calls.append((command, payload)) + return WorkerResult(ok=True, result={"schemaVersion": "2026-05-workbook-session-v1", "workbookId": "wb_test"}) + + monkeypatch.setattr("backend.excel.workbook_worker.run_worker", fake_run_worker) + + response = TestClient(app).post( + "/api/v1/spreadsheet/open", + headers=_headers(), + json={"filename": "model.xlsx", "contentBase64": "UEsDBAo=", "title": "Model", "workbookId": "wb_in"}, + ) + + assert response.status_code == 200 + assert response.json()["workbookId"] == "wb_test" + assert calls == [ + ( + "open-workbook-v1", + { + "filename": "model.xlsx", + "content_base64": "UEsDBAo=", + "title": "Model", + "workbook_id": "wb_in", + }, + ) + ] + + +def test_spreadsheet_routes_forward_additive_request_fields(monkeypatch): + calls: list[tuple[str, dict]] = [] + results = { + "open-workbook-v1": WorkerResult( + ok=True, + result={"schemaVersion": "2026-05-workbook-session-v1", "workbookId": "wb_open"}, + ), + "export-workbook-v1": WorkerResult(ok=True, result={"content_base64": "VGVzdA=="}), + "roundtrip-qa-v2": WorkerResult( + ok=True, + result={"schemaVersion": "2026-05-spreadsheet-roundtrip-qa-v2", "status": "passed"}, + ), + } + + def fake_run_worker(command: str, payload: dict) -> WorkerResult: + calls.append((command, payload)) + return results[command] + + monkeypatch.setattr("backend.excel.workbook_worker.run_worker", fake_run_worker) + client = TestClient(app) + + assert client.post( + "/api/v1/spreadsheet/open", + headers=_headers(), + json={ + "filename": "model.xlsx", + "contentBase64": "UEsDBAo=", + "clientRequestId": "req-open", + "traceContext": {"spanId": "span-open"}, + }, + ).status_code == 200 + assert client.post( + "/api/v1/spreadsheet/export", + headers=_headers(), + json={ + "session": {"workbookId": "wb_export"}, + "clientRequestId": "req-export", + "roundtripPolicy": {"strict": True}, + }, + ).status_code == 200 + assert client.post( + "/api/v1/spreadsheet/roundtrip-qa", + headers=_headers(), + json={ + "session": {"workbookId": "wb_qa"}, + "clientRequestId": "req-qa", + "qaPolicy": {"compareFormulas": True}, + }, + ).status_code == 200 + + assert calls[0] == ( + "open-workbook-v1", + { + "clientRequestId": "req-open", + "traceContext": {"spanId": "span-open"}, + "filename": "model.xlsx", + "content_base64": "UEsDBAo=", + "title": None, + "workbook_id": None, + }, + ) + assert calls[1][0] == "export-workbook-v1" + assert calls[1][1]["clientRequestId"] == "req-export" + assert calls[1][1]["roundtripPolicy"] == {"strict": True} + assert calls[1][1]["session"] == {"workbookId": "wb_export"} + assert calls[2][0] == "roundtrip-qa-v2" + assert calls[2][1]["clientRequestId"] == "req-qa" + assert calls[2][1]["qaPolicy"] == {"compareFormulas": True} + assert calls[2][1]["session"] == {"workbookId": "wb_qa"} + + +def test_spreadsheet_open_worker_error_returns_400(monkeypatch): + monkeypatch.setattr( + "backend.excel.workbook_worker.run_worker", + lambda command, payload: WorkerResult(ok=False, error="bad workbook"), + ) + + response = TestClient(app).post( + "/api/v1/spreadsheet/open", + headers=_headers(), + json={"filename": "bad.xlsx", "content_base64": "UEsDBAo="}, + ) + + assert response.status_code == 400 + assert "bad workbook" in response.json()["detail"] + + +def test_spreadsheet_open_worker_exception_returns_400(monkeypatch): + def fake_run_worker(command: str, payload: dict) -> WorkerResult: + raise RuntimeError("worker disabled") + + monkeypatch.setattr("backend.excel.workbook_worker.run_worker", fake_run_worker) + + response = TestClient(app).post( + "/api/v1/spreadsheet/open", + headers=_headers(), + json={"filename": "bad.xlsx", "content_base64": "UEsDBAo="}, + ) + + assert response.status_code == 400 + assert "worker disabled" in response.json()["detail"] + + +def test_spreadsheet_export_returns_xlsx_response_headers_and_forwards_policy(monkeypatch): + calls: list[tuple[str, dict]] = [] + monkeypatch.setenv("AI_EXPORT_MACRO_REGENERATED_FALLBACK_ALLOWED", "true") + + def fake_run_worker(command: str, payload: dict) -> WorkerResult: + calls.append((command, payload)) + return WorkerResult( + ok=True, + result={ + "engine": "spreadsheet-session-export-v1", + "content_base64": "VGVzdCB3b3JrYm9vaw==", + "fidelity": { + "mode": "preserve_ooxml", + "unsupportedObjectsPreserved": True, + "report": {"status": "passed"}, + }, + }, + ) + + monkeypatch.setattr("backend.excel.workbook_worker.run_worker", fake_run_worker) + + response = TestClient(app).post( + "/api/v1/spreadsheet/export", + headers=_headers(), + json={ + "session": {"workbookId": "wb_test"}, + "filename": "exported.xlsx", + "fidelity_mode": "preserve_ooxml", + "allow_regenerated_fallback": True, + "allow_macro_regenerated_fallback": True, + "prefer_original": False, + }, + ) + + assert response.status_code == 200 + assert response.content == b"Test workbook" + assert response.headers["content-type"].startswith("application/vnd.openxmlformats-officedocument.spreadsheetml.sheet") + assert response.headers["content-disposition"] == 'attachment; filename="exported.xlsx"' + assert response.headers["x-spreadsheet-engine"] == "spreadsheet-session-export-v1" + assert response.headers["x-spreadsheet-fidelity-mode"] == "preserve_ooxml" + assert response.headers["x-spreadsheet-fidelity-status"] == "passed" + assert response.headers["x-spreadsheet-unsupported-preserved"] == "true" + assert calls[0][0] == "export-workbook-v1" + assert calls[0][1]["fidelityMode"] == "preserve_ooxml" + assert calls[0][1]["allowRegeneratedFallback"] is True + assert calls[0][1]["allowMacroRegeneratedFallback"] is True + assert calls[0][1]["preferOriginal"] is False + + +def test_spreadsheet_export_invalid_worker_base64_returns_500(monkeypatch): + monkeypatch.setattr( + "backend.excel.workbook_worker.run_worker", + lambda command, payload: WorkerResult(ok=True, result={"content_base64": "not-base64!"}), + ) + + response = TestClient(app).post( + "/api/v1/spreadsheet/export", + headers=_headers(), + json={"session": {"workbookId": "wb_test"}}, + ) + + assert response.status_code == 500 + assert "invalid workbook bytes" in response.json()["detail"] + + +def test_spreadsheet_export_defaults_fidelity_headers_when_worker_omits_report(monkeypatch): + monkeypatch.setattr( + "backend.excel.workbook_worker.run_worker", + lambda command, payload: WorkerResult(ok=True, result={"content_base64": "VGVzdA=="}), + ) + + response = TestClient(app).post( + "/api/v1/spreadsheet/export", + headers=_headers(), + json={"session": {"workbookId": "wb_test"}}, + ) + + assert response.status_code == 200 + assert response.headers["x-spreadsheet-engine"] == "spreadsheet-session-export-v1" + assert response.headers["x-spreadsheet-fidelity-mode"] == "" + assert response.headers["x-spreadsheet-fidelity-status"] == "" + assert response.headers["x-spreadsheet-unsupported-preserved"] == "false" + + +def test_spreadsheet_export_accepts_camel_case_policy_aliases(monkeypatch): + calls: list[tuple[str, dict]] = [] + monkeypatch.setenv("AI_EXPORT_MACRO_REGENERATED_FALLBACK_ALLOWED", "true") + + def fake_run_worker(command: str, payload: dict) -> WorkerResult: + calls.append((command, payload)) + return WorkerResult(ok=True, result={"content_base64": "VGVzdA==", "fidelity": {"report": {"status": "passed"}}}) + + monkeypatch.setattr("backend.excel.workbook_worker.run_worker", fake_run_worker) + + response = TestClient(app).post( + "/api/v1/spreadsheet/export", + headers=_headers(), + json={ + "session": {"workbookId": "wb_test"}, + "fidelityMode": "preserve_ooxml", + "allowRegeneratedFallback": True, + "allowMacroRegeneratedFallback": True, + "preferOriginal": True, + }, + ) + + assert response.status_code == 200 + assert calls[0][1]["fidelityMode"] == "preserve_ooxml" + assert calls[0][1]["allowRegeneratedFallback"] is True + assert calls[0][1]["allowMacroRegeneratedFallback"] is True + assert calls[0][1]["preferOriginal"] is True + + +def test_spreadsheet_export_defaults_to_preserve_policy_for_edited_original_workbook(monkeypatch): + calls: list[tuple[str, dict]] = [] + + def fake_run_worker(command: str, payload: dict) -> WorkerResult: + calls.append((command, payload)) + return WorkerResult( + ok=True, + result={ + "content_base64": "VGVzdA==", + "fidelity": { + "mode": "preserve_ooxml", + "unsupportedObjectsPreserved": True, + "report": {"status": "passed"}, + }, + }, + ) + + monkeypatch.setattr("backend.excel.workbook_worker.run_worker", fake_run_worker) + + response = TestClient(app).post( + "/api/v1/spreadsheet/export", + headers=_headers(), + json={"session": {"workbookId": "wb_test", "originalContentBase64": "UEsDBAo=", "revision": 1}}, + ) + + assert response.status_code == 200 + assert response.headers["x-spreadsheet-fidelity-mode"] == "preserve_ooxml" + assert response.headers["x-spreadsheet-unsupported-preserved"] == "true" + payload = calls[0][1] + assert payload["session"] == {"workbookId": "wb_test", "originalContentBase64": "UEsDBAo=", "revision": 1} + assert payload["fidelityMode"] is None + assert payload["preferOriginal"] is None + assert payload["allowRegeneratedFallback"] is False + assert payload["allowMacroRegeneratedFallback"] is False + + +def test_spreadsheet_export_surfaces_explicit_regenerated_fallback(monkeypatch): + calls: list[tuple[str, dict]] = [] + monkeypatch.setenv("AI_EXPORT_REGENERATED_FALLBACK_ALLOWED", "true") + + def fake_run_worker(command: str, payload: dict) -> WorkerResult: + calls.append((command, payload)) + return WorkerResult( + ok=True, + result={ + "content_base64": "VGVzdA==", + "fidelity": { + "mode": "regenerated_supported_xlsx", + "unsupportedObjectsPreserved": False, + "fallbackTriggered": {"from": "preserve_ooxml", "to": "regenerated_supported_xlsx"}, + "report": {"status": "warning"}, + }, + }, + ) + + monkeypatch.setattr("backend.excel.workbook_worker.run_worker", fake_run_worker) + + response = TestClient(app).post( + "/api/v1/spreadsheet/export", + headers=_headers(), + json={"session": {"workbookId": "wb_test"}, "allowRegeneratedFallback": True}, + ) + + assert response.status_code == 200 + assert response.headers["x-spreadsheet-fidelity-mode"] == "regenerated_supported_xlsx" + assert response.headers["x-spreadsheet-fidelity-status"] == "warning" + assert response.headers["x-spreadsheet-unsupported-preserved"] == "false" + assert calls[0][1]["allowRegeneratedFallback"] is True + assert calls[0][1]["allowMacroRegeneratedFallback"] is False + + +def test_spreadsheet_export_blocks_disabled_regenerated_fallback_before_worker(monkeypatch): + calls: list[tuple[str, dict]] = [] + monkeypatch.setenv("AI_EXPORT_REGENERATED_FALLBACK_ALLOWED", "false") + monkeypatch.setattr( + "backend.excel.workbook_worker.run_worker", + lambda command, payload: calls.append((command, payload)) or WorkerResult(ok=True, result={"content_base64": "VGVzdA=="}), + ) + + response = TestClient(app).post( + "/api/v1/spreadsheet/export", + headers=_headers(), + json={"session": {"workbookId": "wb_test"}, "allowRegeneratedFallback": True}, + ) + + assert response.status_code == 403 + assert response.json()["detail"]["code"] == "export_regenerated_fallback_disabled" + assert calls == [] + + +def test_spreadsheet_export_blocks_disabled_macro_fallback_before_worker(monkeypatch): + calls: list[tuple[str, dict]] = [] + monkeypatch.setenv("AI_EXPORT_MACRO_REGENERATED_FALLBACK_ALLOWED", "false") + monkeypatch.setattr( + "backend.excel.workbook_worker.run_worker", + lambda command, payload: calls.append((command, payload)) or WorkerResult(ok=True, result={"content_base64": "VGVzdA=="}), + ) + + response = TestClient(app).post( + "/api/v1/spreadsheet/export", + headers=_headers(), + json={"session": {"workbookId": "wb_test"}, "allowMacroRegeneratedFallback": True}, + ) + + assert response.status_code == 403 + assert response.json()["detail"]["code"] == "export_macro_regenerated_fallback_disabled" + assert calls == [] + + +def test_spreadsheet_export_worker_error_returns_400(monkeypatch): + monkeypatch.setattr( + "backend.excel.workbook_worker.run_worker", + lambda command, payload: WorkerResult(ok=False, error="export blocked"), + ) + + response = TestClient(app).post( + "/api/v1/spreadsheet/export", + headers=_headers(), + json={"session": {"workbookId": "wb_test"}}, + ) + + assert response.status_code == 400 + assert "export blocked" in response.json()["detail"] + + +def test_spreadsheet_roundtrip_qa_returns_worker_result_and_accepts_aliases(monkeypatch): + calls: list[tuple[str, dict]] = [] + + def fake_run_worker(command: str, payload: dict) -> WorkerResult: + calls.append((command, payload)) + return WorkerResult(ok=True, result={"schemaVersion": "2026-05-spreadsheet-roundtrip-qa-v2", "status": "passed"}) + + monkeypatch.setattr("backend.excel.workbook_worker.run_worker", fake_run_worker) + + response = TestClient(app).post( + "/api/v1/spreadsheet/roundtrip-qa", + headers=_headers(), + json={ + "filename": "roundtrip.xlsx", + "contentBase64": "UEsDBAo=", + "fidelity_mode": "preserve_ooxml", + "allow_regenerated_fallback": False, + }, + ) + + assert response.status_code == 200 + assert response.json()["status"] == "passed" + assert calls == [ + ( + "roundtrip-qa-v2", + { + "session": {}, + "filename": "roundtrip.xlsx", + "title": None, + "content_base64": "UEsDBAo=", + "fidelityMode": "preserve_ooxml", + "allowRegeneratedFallback": False, + }, + ) + ] + + +def test_spreadsheet_roundtrip_qa_defaults_regenerated_fallback_to_false(monkeypatch): + calls: list[tuple[str, dict]] = [] + + def fake_run_worker(command: str, payload: dict) -> WorkerResult: + calls.append((command, payload)) + return WorkerResult(ok=True, result={"schemaVersion": "2026-05-spreadsheet-roundtrip-qa-v2", "status": "passed"}) + + monkeypatch.setattr("backend.excel.workbook_worker.run_worker", fake_run_worker) + + response = TestClient(app).post( + "/api/v1/spreadsheet/roundtrip-qa", + headers=_headers(), + json={"session": {"workbookId": "wb_test", "originalContentBase64": "UEsDBAo=", "revision": 1}}, + ) + + assert response.status_code == 200 + assert calls == [ + ( + "roundtrip-qa-v2", + { + "session": {"workbookId": "wb_test", "originalContentBase64": "UEsDBAo=", "revision": 1}, + "filename": "workbook.xlsx", + "title": None, + "content_base64": None, + "fidelityMode": None, + "allowRegeneratedFallback": False, + }, + ) + ] + + +def test_spreadsheet_roundtrip_qa_worker_failure_paths(monkeypatch): + monkeypatch.setattr( + "backend.excel.workbook_worker.run_worker", + lambda command, payload: WorkerResult(ok=False, error="qa blocked"), + ) + response = TestClient(app).post( + "/api/v1/spreadsheet/roundtrip-qa", + headers=_headers(), + json={"session": {"workbookId": "wb_test"}}, + ) + assert response.status_code == 400 + assert "qa blocked" in response.json()["detail"] + + def fake_raise(command: str, payload: dict) -> WorkerResult: + raise RuntimeError("worker unavailable") + + monkeypatch.setattr("backend.excel.workbook_worker.run_worker", fake_raise) + response = TestClient(app).post( + "/api/v1/spreadsheet/roundtrip-qa", + headers=_headers(), + json={"session": {"workbookId": "wb_test"}}, + ) + assert response.status_code == 400 + assert "worker unavailable" in response.json()["detail"] + + +def test_spreadsheet_openapi_documents_response_contracts_and_export_headers(): + schema = TestClient(app).get("/openapi.json").json() + paths = schema["paths"] + + open_200 = paths["/api/v1/spreadsheet/open"]["post"]["responses"]["200"]["content"]["application/json"]["schema"] + qa_200 = paths["/api/v1/spreadsheet/roundtrip-qa"]["post"]["responses"]["200"]["content"]["application/json"]["schema"] + export_200 = paths["/api/v1/spreadsheet/export"]["post"]["responses"]["200"] + export_request_ref = paths["/api/v1/spreadsheet/export"]["post"]["requestBody"]["content"]["application/json"]["schema"]["$ref"] + export_request_name = export_request_ref.rsplit("/", 1)[-1] + export_props = schema["components"]["schemas"][export_request_name]["properties"] + + assert open_200 != {} + assert qa_200 != {} + assert open_200["$ref"].endswith("/SpreadsheetOpenResponse") + assert qa_200["$ref"].endswith("/SpreadsheetRoundtripQaResponse") + assert set(export_200["headers"]) >= { + "Content-Disposition", + "X-Spreadsheet-Fidelity-Mode", + "X-Spreadsheet-Fidelity-Status", + "X-Spreadsheet-Unsupported-Preserved", + } + assert {"fidelityMode", "allowRegeneratedFallback", "allowMacroRegeneratedFallback", "preferOriginal"} <= set(export_props) diff --git a/backend/tests/test_sprint4.py b/backend/tests/test_sprint4.py index 4e389d1..f2f464f 100644 --- a/backend/tests/test_sprint4.py +++ b/backend/tests/test_sprint4.py @@ -1,6 +1,5 @@ """Tests for Sprint 4 upgrades: template router, structured output, streaming.""" -import re import pytest # ── Template Router Tests ──────────────────────────────────────────────── @@ -10,6 +9,17 @@ extract_template_params, try_template_route, ) +from backend.templates.registry import get_template, list_templates + + +GUIDED_FINANCE_TEMPLATE_IDS = { + "dcf_model", + "lbo_model", + "three_statement", + "comps_analysis", + "merger_model", + "sensitivity", +} class TestTemplateIntentClassifier: @@ -36,6 +46,12 @@ def test_three_statement_full(self): def test_sensitivity(self): assert classify_template_intent("Build a sensitivity analysis table") == "sensitivity" + def test_merger_model(self): + assert classify_template_intent("Create a merger accretion dilution model") == "merger_model" + + def test_comps_analysis(self): + assert classify_template_intent("Build a trading comps analysis") == "comps_analysis" + def test_amortization(self): assert classify_template_intent("Create an amortization schedule") == "amortization" @@ -132,6 +148,41 @@ def test_params_appear_in_summary(self): if result["metadata"].get("params_extracted"): assert "wacc" in result["metadata"]["params_extracted"] + def test_template_result_has_guided_workflow_metadata(self): + result = try_template_route("Build a DCF model") + assert result is not None + workflow = result["metadata"].get("guided_workflow") + assert result["metadata"]["workflow_id"] == "guided_dcf_valuation" + assert workflow["workflow_id"] == "guided_dcf_valuation" + assert workflow["required_inputs"] + assert workflow["setup_steps"] + assert workflow["key_outputs"] + + +class TestGuidedFinanceTemplateMetadata: + """Tests for Shortcut-style guided workflow metadata on finance templates.""" + + def test_registry_exposes_guided_workflows_for_target_finance_templates(self): + for template_id in GUIDED_FINANCE_TEMPLATE_IDS: + meta = get_template(template_id) + assert meta is not None + workflow = meta.get("guided_workflow") + assert meta["workflow_id"] == workflow["workflow_id"] + assert workflow["workflow_id"].startswith("guided_") + assert workflow["title"] + assert workflow["automation_prompt"] + assert len(workflow["required_inputs"]) >= 3 + assert len(workflow["setup_steps"]) >= 3 + assert len(workflow["key_outputs"]) >= 3 + assert len(workflow["quality_checks"]) >= 2 + + def test_list_templates_includes_guided_workflow_summary(self): + templates = {template["id"]: template for template in list_templates()} + for template_id in GUIDED_FINANCE_TEMPLATE_IDS: + assert template_id in templates + assert templates[template_id]["workflow_id"].startswith("guided_") + assert templates[template_id]["guided_workflow"]["use_case"] + # ── JSON Mode Tests ────────────────────────────────────────────────────── diff --git a/backend/tests/test_sqlite_lifecycle_fencing.py b/backend/tests/test_sqlite_lifecycle_fencing.py new file mode 100644 index 0000000..7e52182 --- /dev/null +++ b/backend/tests/test_sqlite_lifecycle_fencing.py @@ -0,0 +1,512 @@ +"""SQLite lifecycle fencing and cross-instance race regressions.""" + +from __future__ import annotations + +import json +import sqlite3 +import threading +import time +import uuid + +import pytest + +from backend.api.security.ownership import OwnershipNotFoundError, require_session_access +from backend.session_store import SessionStore +from backend.storage.migrations import ( + SQLITE_SESSION_LIFECYCLE_TABLES, + SQLITE_SESSION_PURGE_ORDER, +) + + +@pytest.mark.unit +def test_sqlite_restart_repairs_legacy_background_job_orphan(tmp_path): + db_path = tmp_path / "sessions.db" + store = SessionStore(db_path) + orphan_session_id = "legacy-background-job-orphan" + with store._conn() as conn: + conn.execute("DROP TRIGGER trg_background_jobs_session_lifecycle_insert") + conn.execute("DROP TRIGGER trg_background_jobs_session_lifecycle_update") + conn.execute( + """ + INSERT INTO background_jobs + (job_id, job_type, status, session_id, payload_json, result_json, + created_at, updated_at) + VALUES (?, 'workbook_job', 'queued', ?, '{}', '{}', 1, 1) + """, + ("legacy-orphan-job", orphan_session_id), + ) + + restarted = SessionStore(db_path) + assert restarted.session_exists(orphan_session_id) + assert restarted.get_session_lifecycle_state(orphan_session_id) == "active" + with restarted._conn() as conn: + session = conn.execute( + "SELECT owner_user_id, metadata FROM sessions WHERE session_id = ?", + (orphan_session_id,), + ).fetchone() + assert session["owner_user_id"] is None + metadata = json.loads(session["metadata"]) + assert metadata["quarantined_legacy_orphan"] is True + assert metadata["source_tables"] == ["background_jobs"] + + receipt_id = "del_legacy_orphan" + restarted.set_session_lifecycle_state( + orphan_session_id, + "purging", + receipt_id=receipt_id, + started_at=2, + ) + assert restarted.delete_session(orphan_session_id) is True + restarted.set_session_lifecycle_state( + orphan_session_id, + "verified", + receipt_id=receipt_id, + started_at=2, + completed_at=3, + ) + assert restarted.session_deletion_verified(orphan_session_id) + + +@pytest.mark.unit +def test_sqlite_restart_destroys_verified_session_orphan_without_raw_quarantine(tmp_path): + db_path = tmp_path / "sessions.db" + store = SessionStore(db_path) + session_id = _session_id("verified-orphan") + store.get_or_create_session(session_id) + receipt_id = f"del_{uuid.uuid4().hex}" + store.set_session_lifecycle_state( + session_id, + "purging", + receipt_id=receipt_id, + started_at=time.time(), + ) + assert store.delete_session(session_id) is True + store.set_session_lifecycle_state( + session_id, + "verified", + receipt_id=receipt_id, + started_at=1, + completed_at=2, + ) + + legacy = sqlite3.connect(db_path) + try: + legacy.execute("PRAGMA foreign_keys = OFF") + legacy.execute("DROP TRIGGER trg_conversations_session_lifecycle_insert") + legacy.execute("DROP TRIGGER trg_conversations_session_lifecycle_update") + legacy.execute( + "INSERT INTO conversations (session_id, role, content, timestamp, metadata) " + "VALUES (?, 'user', 'TOP SECRET AFTER VERIFIED', 3, '{}')", + (session_id,), + ) + legacy.commit() + finally: + legacy.close() + + restarted = SessionStore(db_path) + with restarted._conn() as conn: + assert conn.execute( + "SELECT 1 FROM conversations WHERE session_id = ?", + (session_id,), + ).fetchone() is None + quarantine = conn.execute( + "SELECT 1 FROM sqlite_master " + "WHERE type = 'table' AND name = 'session_lifecycle_orphan_quarantine'" + ).fetchone() + + assert quarantine is None + assert restarted.get_session_lifecycle_state(session_id) == "verified" + assert restarted.session_deletion_verified(session_id) is True + + +@pytest.mark.unit +def test_sqlite_verified_orphan_purge_is_child_first_for_plan_snapshots(tmp_path): + db_path = tmp_path / "sessions.db" + store = SessionStore(db_path) + session_id = _session_id("verified-plan-snapshot") + plan_id = f"plan_{uuid.uuid4().hex}" + snapshot_id = f"snapshot_{uuid.uuid4().hex}" + store.get_or_create_session(session_id) + receipt_id = f"del_{uuid.uuid4().hex}" + store.set_session_lifecycle_state( + session_id, + "purging", + receipt_id=receipt_id, + started_at=1, + ) + assert store.delete_session(session_id) is True + store.set_session_lifecycle_state( + session_id, + "verified", + receipt_id=receipt_id, + started_at=1, + completed_at=2, + ) + + legacy = sqlite3.connect(db_path) + try: + legacy.execute("PRAGMA foreign_keys = OFF") + legacy.execute("DROP TRIGGER trg_operation_plans_session_lifecycle_insert") + legacy.execute("DROP TRIGGER trg_workbook_snapshots_session_lifecycle_insert") + legacy.execute( + "DELETE FROM schema_migrations " + "WHERE migration_id = '2026_08_10_0019_session_lifecycle_orphan_repair'" + ) + legacy.execute( + "INSERT INTO operation_plans " + "(plan_id, session_id, prompt, status, plan_json, created_at, updated_at) " + "VALUES (?, ?, 'TOP SECRET PLAN', 'draft', '{}', 3, 3)", + (plan_id, session_id), + ) + legacy.execute( + "INSERT INTO workbook_snapshots " + "(snapshot_id, session_id, plan_id, label, workbook_state, created_at) " + "VALUES (?, ?, ?, 'secret', '{\"A1\": \"TOP SECRET\"}', 3)", + (snapshot_id, session_id, plan_id), + ) + legacy.commit() + finally: + legacy.close() + + restarted = SessionStore(db_path) + with restarted._conn() as conn: + assert conn.execute( + "SELECT 1 FROM operation_plans WHERE plan_id = ?", + (plan_id,), + ).fetchone() is None + assert conn.execute( + "SELECT 1 FROM workbook_snapshots WHERE snapshot_id = ?", + (snapshot_id,), + ).fetchone() is None + assert conn.execute("PRAGMA foreign_key_check").fetchall() == [] + + assert restarted.session_deletion_verified(session_id) is True + + +@pytest.mark.unit +@pytest.mark.parametrize("lifecycle_state", ["purging", "failed"]) +def test_sqlite_nonterminal_orphan_preserves_managed_document_inventory( + tmp_path, + lifecycle_state, +): + db_path = tmp_path / f"sessions-{lifecycle_state}.db" + store = SessionStore(db_path) + session_id = _session_id(f"{lifecycle_state}-document") + file_path = str(tmp_path / f"{lifecycle_state}-private.xlsx") + store.get_or_create_session(session_id) + document_id = store.save_document( + session_id, + "private.xlsx", + file_path=file_path, + parsed={"text": "TOP SECRET MANAGED DOCUMENT"}, + ) + receipt_id = f"del_{uuid.uuid4().hex}" + store.set_session_lifecycle_state( + session_id, + "purging", + receipt_id=receipt_id, + started_at=1, + ) + if lifecycle_state == "failed": + store.set_session_lifecycle_state( + session_id, + "failed", + receipt_id=receipt_id, + started_at=1, + completed_at=2, + errors=["retry required"], + ) + + legacy = sqlite3.connect(db_path) + try: + legacy.execute("PRAGMA foreign_keys = OFF") + legacy.execute("DELETE FROM sessions WHERE session_id = ?", (session_id,)) + legacy.commit() + finally: + legacy.close() + + restarted = SessionStore(db_path) + + assert restarted.session_exists(session_id) is True + assert restarted.get_session_lifecycle_state(session_id) == lifecycle_state + assert restarted.list_document_file_paths(session_id) == [file_path] + assert restarted.get_document(document_id) is None + with restarted._conn() as conn: + recovery = conn.execute( + "SELECT owner_user_id, metadata FROM sessions WHERE session_id = ?", + (session_id,), + ).fetchone() + assert recovery["owner_user_id"] is None + assert json.loads(recovery["metadata"])["lifecycle_recovery_state"] == lifecycle_state + assert conn.execute("PRAGMA foreign_key_check").fetchall() == [] + + +@pytest.mark.unit +def test_sqlite_restart_destroys_legacy_raw_quarantine_despite_applied_migration(tmp_path): + db_path = tmp_path / "sessions.db" + store = SessionStore(db_path) + with store._conn() as conn: + assert conn.execute( + "SELECT 1 FROM schema_migrations WHERE migration_id = " + "'2026_08_10_0020_destroy_deleted_session_orphan_quarantine'" + ).fetchone() + conn.execute( + "CREATE TABLE session_lifecycle_orphan_quarantine " + "(quarantine_id TEXT PRIMARY KEY, payload_json TEXT NOT NULL)" + ) + conn.execute( + "INSERT INTO session_lifecycle_orphan_quarantine " + "(quarantine_id, payload_json) VALUES ('legacy', 'TOP SECRET RESTORED PAYLOAD')" + ) + + restarted = SessionStore(db_path) + with restarted._conn() as conn: + quarantine = conn.execute( + "SELECT 1 FROM sqlite_master " + "WHERE type = 'table' AND name = 'session_lifecycle_orphan_quarantine'" + ).fetchone() + + assert quarantine is None + + +def _session_id(prefix: str) -> str: + return f"{prefix}-{uuid.uuid4().hex}" + + +@pytest.mark.unit +def test_sqlite_lifecycle_migration_fences_every_session_projection(tmp_path): + store = SessionStore(tmp_path / "sessions.db") + with store._conn() as conn: + tables = { + str(row["name"]) + for row in conn.execute("SELECT name FROM sqlite_master WHERE type = 'table'") + } + triggers = { + str(row["name"]) + for row in conn.execute("SELECT name FROM sqlite_master WHERE type = 'trigger'") + } + + assert "session_lifecycle" in tables + assert len(SQLITE_SESSION_PURGE_ORDER) == len(SQLITE_SESSION_LIFECYCLE_TABLES) + assert set(SQLITE_SESSION_PURGE_ORDER) == set(SQLITE_SESSION_LIFECYCLE_TABLES) + assert SQLITE_SESSION_PURGE_ORDER[-1] == "sessions" + assert SQLITE_SESSION_PURGE_ORDER.index("apply_attempts") < SQLITE_SESSION_PURGE_ORDER.index( + "workbook_snapshots" + ) + assert SQLITE_SESSION_PURGE_ORDER.index( + "workbook_snapshots" + ) < SQLITE_SESSION_PURGE_ORDER.index("operation_plans") + assert SQLITE_SESSION_PURGE_ORDER.index("document_chunks") < SQLITE_SESSION_PURGE_ORDER.index( + "documents" + ) + for table_name in SQLITE_SESSION_LIFECYCLE_TABLES: + assert f"trg_{table_name}_session_lifecycle_insert" in triggers + assert f"trg_{table_name}_session_lifecycle_update" in triggers + + +@pytest.mark.unit +def test_unauth_boundary_creates_explicitly_but_low_level_writes_require_existing(tmp_path): + store = SessionStore(tmp_path / "sessions.db") + session_id = _session_id("local-boundary") + + with pytest.raises(OwnershipNotFoundError, match="not found"): + store.add_message(session_id, "user", "must not create implicitly") + + created = require_session_access(store, session_id, create=True) + assert created and created["session_id"] == session_id + store.add_message(session_id, "user", "created at boundary") + assert [message.content for message in store.get_conversation(session_id)] == ["created at boundary"] + + +@pytest.mark.unit +def test_sqlite_writer_and_purge_are_serialized_across_store_instances(tmp_path): + db_path = tmp_path / "sessions.db" + writer_store = SessionStore(db_path) + purger_store = SessionStore(db_path) + session_id = _session_id("sqlite-race") + writer_store.get_or_create_session(session_id) + writer_started = threading.Event() + release_writer = threading.Event() + purge_completed = threading.Event() + failures: list[BaseException] = [] + + def writer() -> None: + try: + with writer_store._session_write_conn(session_id) as conn: + conn.execute( + "INSERT INTO conversations (session_id, role, content, timestamp, metadata) " + "VALUES (?, 'user', 'racing write', ?, '{}')", + (session_id, time.time()), + ) + writer_started.set() + assert release_writer.wait(timeout=5) + except BaseException as exc: # pragma: no cover - surfaced below + failures.append(exc) + + def purger() -> None: + try: + purger_store.set_session_lifecycle_state( + session_id, + "purging", + receipt_id="del_sqlite_race", + started_at=time.time(), + ) + purge_completed.set() + except BaseException as exc: # pragma: no cover - surfaced below + failures.append(exc) + + writer_thread = threading.Thread(target=writer) + purge_thread = threading.Thread(target=purger) + writer_thread.start() + assert writer_started.wait(timeout=5) + purge_thread.start() + assert not purge_completed.wait(timeout=0.2) + release_writer.set() + writer_thread.join(timeout=5) + purge_thread.join(timeout=5) + + assert not failures + assert purge_completed.is_set() + assert purger_store.get_session_lifecycle_state(session_id) == "purging" + assert purger_store.get_conversation(session_id) == [] + assert purger_store.delete_session(session_id) is True + purger_store.set_session_lifecycle_state( + session_id, + "verified", + receipt_id="del_sqlite_race", + started_at=1.0, + completed_at=time.time(), + ) + + restarted = SessionStore(db_path) + assert restarted.get_session_lifecycle_state(session_id) == "verified" + assert restarted.session_deletion_verified(session_id) is True + with pytest.raises(OwnershipNotFoundError, match="not found"): + restarted.get_or_create_session(session_id) + with pytest.raises(OwnershipNotFoundError, match="not found"): + restarted.record_task(session_id, "late", "late", "action", [], "late") + + +@pytest.mark.unit +def test_sqlite_sensitive_read_and_purge_have_one_linearization_boundary(tmp_path, monkeypatch): + db_path = tmp_path / "sessions.db" + reader_store = SessionStore(db_path) + purger_store = SessionStore(db_path) + session_id = _session_id("sqlite-read-race") + reader_store.get_or_create_session(session_id) + reader_store.add_message(session_id, "user", "secret") + visibility_checked = threading.Event() + release_reader = threading.Event() + purge_completed = threading.Event() + result: list[list[str]] = [] + failures: list[BaseException] = [] + original_visible = reader_store._session_visible_in_conn + + def paused_visibility(conn, checked_session_id: str) -> bool: + visible = original_visible(conn, checked_session_id) + visibility_checked.set() + assert release_reader.wait(timeout=5) + return visible + + monkeypatch.setattr(reader_store, "_session_visible_in_conn", paused_visibility) + + def reader() -> None: + try: + result.append([message.content for message in reader_store.get_conversation(session_id)]) + except BaseException as exc: # pragma: no cover - surfaced below + failures.append(exc) + + def purger() -> None: + try: + purger_store.set_session_lifecycle_state( + session_id, + "purging", + receipt_id="del_sqlite_read_race", + started_at=time.time(), + ) + purge_completed.set() + except BaseException as exc: # pragma: no cover - surfaced below + failures.append(exc) + + reader_thread = threading.Thread(target=reader) + purge_thread = threading.Thread(target=purger) + reader_thread.start() + assert visibility_checked.wait(timeout=5) + purge_thread.start() + assert not purge_completed.wait(timeout=0.2) + release_reader.set() + reader_thread.join(timeout=5) + purge_thread.join(timeout=5) + + assert not failures + assert result == [["secret"]] + assert purge_completed.is_set() + assert purger_store.get_conversation(session_id) == [] + + +@pytest.mark.unit +def test_sqlite_session_id_rebinding_and_post_purge_projection_commit_are_rejected(tmp_path): + store = SessionStore(tmp_path / "sessions.db") + first = _session_id("owner-a") + second = _session_id("owner-b") + store.get_or_create_session(first) + store.get_or_create_session(second) + store.save_workbook_map({"mapId": "map-1", "sessionId": first}) + + with pytest.raises(sqlite3.IntegrityError, match="session_id is immutable"): + with store._conn() as conn: + conn.execute( + "INSERT INTO workbook_maps " + "(map_id, session_id, map_json, created_at, updated_at) VALUES (?, ?, '{}', 1, 1) " + "ON CONFLICT(map_id) DO UPDATE SET session_id = excluded.session_id", + ("map-1", second), + ) + + store.set_session_lifecycle_state( + first, + "purging", + receipt_id="del_projection_fence", + started_at=time.time(), + ) + with pytest.raises(sqlite3.IntegrityError, match="session lifecycle is not active"): + with store._conn() as conn: + conn.execute( + "INSERT INTO workbook_maps " + "(map_id, session_id, map_json, created_at, updated_at) VALUES (?, ?, '{}', 1, 1)", + ("map-after-purge", first), + ) + + +@pytest.mark.unit +def test_sqlite_purging_revokes_read_and_list_matrix_before_physical_delete(tmp_path): + store = SessionStore(tmp_path / "sessions.db") + session_id = _session_id("read-matrix") + store.get_or_create_session(session_id) + store.add_message(session_id, "user", "secret") + store.record_task(session_id, "task-1", "secret", "action", [], "secret") + store.save_workbook_map({"mapId": "map-secret", "sessionId": session_id}) + store.save_local_replay_artifact(session_id, "v7_plan", {"secret": True}) + store.save_local_replay_baseline( + {"baselineId": "baseline-secret", "sessionId": session_id, "name": "secret"} + ) + store.save_workbook_job( + {"run_id": "job-secret", "session_id": session_id, "status": "completed", "result": {"secret": True}} + ) + + store.set_session_lifecycle_state( + session_id, + "purging", + receipt_id="del_read_matrix", + started_at=time.time(), + ) + + assert store.get_conversation(session_id) == [] + assert store.get_task_history(session_id) == [] + assert store.get_total_tokens(session_id)["total_tokens"] == 0 + assert store.get_workbook_map("map-secret") is None + assert store.get_latest_workbook_map(session_id) is None + assert store.get_workbook_job("job-secret") is None + assert store.list_workbook_jobs(session_id=session_id) == [] + assert store.list_local_replay_artifacts() == [] + assert store.list_local_replay_baselines() == [] + assert all(row["session_id"] != session_id for row in store.list_sessions()) diff --git a/backend/tests/test_storage_encryption_evidence.py b/backend/tests/test_storage_encryption_evidence.py new file mode 100644 index 0000000..2065783 --- /dev/null +++ b/backend/tests/test_storage_encryption_evidence.py @@ -0,0 +1,1027 @@ +from __future__ import annotations + +import hashlib +import json +import os +import threading +import time +from pathlib import Path +from typing import Any + +import pytest +from cryptography.hazmat.primitives import serialization +from cryptography.hazmat.primitives.asymmetric.ed25519 import ( + Ed25519PrivateKey, + Ed25519PublicKey, +) + +from backend.operations import operator_cli +from backend.operations.sqlcipher_rehearsal import run_disposable_sqlcipher_rehearsal +from backend.operations.storage_encryption_attestation import ( + ATTESTATION_SCHEMA_VERSION, + StorageEncryptionAttestationError, + runtime_external_storage_encryption_status, + validate_storage_encryption_attestation, +) +from backend.storage.encrypted_store import ( + EncryptedSessionStore, + RotationError, + canonical_rotation_receipt_bytes, +) + + +RELEASE_SHA = "a" * 40 +OLD_KEY = "1" * 64 +NEW_KEY = "2" * 64 +_RUNTIME_ATTESTATION_ENV = ( + "STORAGE_ENCRYPTION_ATTESTATION_PATH", + "STORAGE_ENCRYPTION_ATTESTATION_SIGNATURE_PATH", + "STORAGE_ENCRYPTION_ATTESTATION_PUBLIC_KEY_PATH", + "STORAGE_ENCRYPTION_ATTESTATION_KEY_ID", + "STORAGE_ENCRYPTION_ATTESTATION_RELEASE_SHA", + "STORAGE_ENCRYPTION_ATTESTATION_DEPLOYMENT_SHA256", + "STORAGE_ENCRYPTION_ATTESTATION_RUNNER_SHA256", + "STORAGE_ENCRYPTION_ATTESTATION_KEY_IDENTITY_SHA256", + "STORAGE_ENCRYPTION_ATTESTATION_PUBLIC_KEY_SHA256", + "STORAGE_ENCRYPTION_ATTESTATION_MAX_AGE_SECONDS", +) + + +def _sha(value: str) -> str: + return hashlib.sha256(value.encode()).hexdigest() + + +def _rotation_provider( + tmp_path: Path, + monkeypatch: pytest.MonkeyPatch, + *, + fail_new_verification: bool = False, +) -> tuple[EncryptedSessionStore, list[tuple[str, str]]]: + database = tmp_path / "encrypted.db" + database.write_bytes(b"encrypted-before") + provider = EncryptedSessionStore( + db_path=str(database), + encrypt_key=OLD_KEY, + key_id="old-key-2026", + encrypt_enabled=True, + ) + provider._sqlcipher_available = True + provider._sqlcipher_driver = "test-sqlcipher" + state = {"key": OLD_KEY, "failed_new": False} + transitions: list[tuple[str, str]] = [] + + def verify(key: str) -> dict[str, object]: + if key != state["key"]: + raise RuntimeError("/private/customer/key mismatch secret") + if key == NEW_KEY and fail_new_verification and not state["failed_new"]: + state["failed_new"] = True + raise RuntimeError("/private/customer/post rotate failure secret") + return { + "cipherVersionSha256": "b" * 64, + "cipherIntegrity": "verified", + "databaseReadable": True, + "sqliteIntegrity": "verified", + } + + def apply(old_key: str, new_key: str) -> None: + assert state["key"] == old_key + transitions.append((old_key, new_key)) + state["key"] = new_key + database.write_bytes(b"encrypted-" + new_key[:4].encode()) + + monkeypatch.setattr(provider, "_verify_key_and_integrity", verify) + monkeypatch.setattr(provider, "_apply_rekey", apply) + return provider, transitions + + +def _rotate(provider: EncryptedSessionStore, **overrides: Any) -> dict[str, Any]: + values = { + "rotation_id": "rotation-2026-08-a", + "old_key_id": "old-key-2026", + "new_key_id": "new-key-2026", + "new_key": NEW_KEY, + "release_sha": RELEASE_SHA, + "lock_timeout_seconds": 2, + } + values.update(overrides) + return provider.rotate_key( + rotation_id=str(values["rotation_id"]), + old_key_id=str(values["old_key_id"]), + new_key_id=str(values["new_key_id"]), + new_key=str(values["new_key"]), + release_sha=str(values["release_sha"]), + lock_timeout_seconds=float(str(values["lock_timeout_seconds"])), + ) + + +def test_sqlcipher_rotation_receipt_is_exact_sha_external_signable_and_private( + tmp_path: Path, + monkeypatch: pytest.MonkeyPatch, +) -> None: + provider, transitions = _rotation_provider(tmp_path, monkeypatch) + + result = _rotate(provider) + + assert result["outcome"] == "rotated" + envelope = result["receipt"] + receipt = envelope["receipt"] + assert receipt["schemaVersion"] == "2026-08-sqlcipher-key-rotation-v1" + assert receipt["releaseSha"] == RELEASE_SHA + assert receipt["status"] == "passed" + assert receipt["oldKeyIdSha256"] == _sha("old-key-2026") + assert receipt["newKeyIdSha256"] == _sha("new-key-2026") + assert receipt["preVerification"]["cipherIntegrity"] == "verified" + assert receipt["postVerification"]["cipherIntegrity"] == "verified" + assert receipt["rollback"]["attempted"] is False + assert receipt["externallySignable"] is True + canonical = canonical_rotation_receipt_bytes(receipt) + assert hashlib.sha256(canonical).hexdigest() == envelope["receiptSha256"] + signing_key = Ed25519PrivateKey.generate() + signing_key.public_key().verify(signing_key.sign(canonical), canonical) + rendered = json.dumps(result, sort_keys=True) + assert OLD_KEY not in rendered + assert NEW_KEY not in rendered + assert "old-key-2026" not in rendered + assert "new-key-2026" not in rendered + assert "rotation-2026-08-a" not in rendered + assert str(tmp_path) not in rendered + assert transitions == [(OLD_KEY, NEW_KEY)] + assert provider.encrypt_key == NEW_KEY + assert provider.key_id == "new-key-2026" + + +def test_sqlcipher_rotation_replays_exact_receipt_and_fences_conflict( + tmp_path: Path, + monkeypatch: pytest.MonkeyPatch, +) -> None: + provider, transitions = _rotation_provider(tmp_path, monkeypatch) + first = _rotate(provider) + + second = _rotate(provider) + + assert second["outcome"] == "replayed" + assert second["receipt"] == first["receipt"] + assert transitions == [(OLD_KEY, NEW_KEY)] + with pytest.raises(RotationError) as caught: + _rotate( + provider, + old_key_id="different-old-id", + new_key_id="different-new-id", + new_key="3" * 64, + ) + assert caught.value.code == "rotation_id_conflict" + assert OLD_KEY not in str(caught.value) + assert NEW_KEY not in str(caught.value) + + +def test_sqlcipher_rotation_rolls_back_and_revalidates_old_key_on_post_failure( + tmp_path: Path, + monkeypatch: pytest.MonkeyPatch, +) -> None: + provider, transitions = _rotation_provider( + tmp_path, + monkeypatch, + fail_new_verification=True, + ) + + with pytest.raises(RotationError) as caught: + _rotate(provider) + + assert caught.value.code == "post_rotate_verification_failed_recovered" + assert transitions == [(OLD_KEY, NEW_KEY), (NEW_KEY, OLD_KEY)] + assert provider.encrypt_key == OLD_KEY + assert provider.key_id == "old-key-2026" + encoded = json.dumps(caught.value.receipt, sort_keys=True) + assert caught.value.receipt["receipt"]["status"] == "failed" + assert caught.value.receipt["receipt"]["rollback"] == { + "attempted": True, + "oldKeyRevalidated": True, + "status": "recovered", + } + assert "private/customer" not in encoded + assert "secret" not in encoded + + +def test_sqlcipher_rotation_recovers_ambiguous_driver_error_after_durable_rekey( + tmp_path: Path, + monkeypatch: pytest.MonkeyPatch, +) -> None: + provider, transitions = _rotation_provider(tmp_path, monkeypatch) + apply_rekey = provider._apply_rekey + + def ambiguous_apply(old_key: str, new_key: str) -> None: + apply_rekey(old_key, new_key) + raise RuntimeError("/private/customer/driver-close-error secret") + + monkeypatch.setattr(provider, "_apply_rekey", ambiguous_apply) + + result = _rotate(provider) + + assert result["outcome"] == "rotated" + assert result["receipt"]["receipt"]["recoveredInterruptedRotation"] is True + assert transitions == [(OLD_KEY, NEW_KEY)] + rendered = json.dumps(result, sort_keys=True) + assert "private/customer" not in rendered + assert "secret" not in rendered + + +def test_sqlcipher_rotation_serializes_concurrent_replay( + tmp_path: Path, + monkeypatch: pytest.MonkeyPatch, +) -> None: + provider, transitions = _rotation_provider(tmp_path, monkeypatch) + barrier = threading.Barrier(3) + results: list[dict] = [] + + def worker() -> None: + barrier.wait() + results.append(_rotate(provider)) + + threads = [threading.Thread(target=worker) for _ in range(2)] + for thread in threads: + thread.start() + barrier.wait() + for thread in threads: + thread.join(timeout=5) + + assert sorted(result["outcome"] for result in results) == ["replayed", "rotated"] + assert transitions == [(OLD_KEY, NEW_KEY)] + assert results[0]["receipt"] == results[1]["receipt"] + + +def test_sqlcipher_rotation_drains_existing_connections_before_rekey( + tmp_path: Path, + monkeypatch: pytest.MonkeyPatch, +) -> None: + provider, transitions = _rotation_provider(tmp_path, monkeypatch) + started = threading.Event() + results: list[dict[str, Any]] = [] + + def worker() -> None: + started.set() + results.append(_rotate(provider)) + + with provider._connection_admission(): + thread = threading.Thread(target=worker) + thread.start() + assert started.wait(timeout=1) + time.sleep(0.05) + assert thread.is_alive() + assert transitions == [] + thread.join(timeout=3) + + assert not thread.is_alive() + assert results[0]["outcome"] == "rotated" + assert transitions == [(OLD_KEY, NEW_KEY)] + + +def test_sqlcipher_rotation_recovers_durable_in_progress_transition( + tmp_path: Path, + monkeypatch: pytest.MonkeyPatch, +) -> None: + provider, transitions = _rotation_provider(tmp_path, monkeypatch) + database_identity = provider._database_identity_sha256() + request = provider._rotation_request( + rotation_id="rotation-2026-08-a", + old_key_id="old-key-2026", + new_key_id="new-key-2026", + new_key=NEW_KEY, + release_sha=RELEASE_SHA, + database_identity_sha256=database_identity, + ) + pre_verification = provider._verify_key_and_integrity(OLD_KEY) + record: dict[str, Any] = { + "status": "in_progress", + "requestSha256": request["requestSha256"], + "request": request, + "preVerification": pre_verification, + "preDatabaseSha256": provider._database_sha256(), + } + journal: dict[str, Any] = { + "schemaVersion": "2026-08-sqlcipher-key-rotation-journal-v1", + "rotations": {"rotation-2026-08-a": record}, + } + provider._save_rotation_record( + journal, + "rotation-2026-08-a", + record, + ) + # Rehearse a crash after SQLCipher committed but before the passed receipt. + provider._apply_rekey(OLD_KEY, NEW_KEY) + transitions.clear() + + result = _rotate(provider) + + assert result["outcome"] == "rotated" + assert result["receipt"]["receipt"]["recoveredInterruptedRotation"] is True + assert transitions == [] + + +def test_sqlcipher_rotation_replay_forces_new_key_validation( + tmp_path: Path, + monkeypatch: pytest.MonkeyPatch, +) -> None: + provider, _transitions = _rotation_provider(tmp_path, monkeypatch) + _rotate(provider) + provider.encrypt_key = OLD_KEY + provider.key_id = "old-key-2026" + + def only_old_key_verifies(key: str) -> dict[str, object]: + if key != OLD_KEY: + raise RuntimeError("/private/customer/forged replay") + return { + "cipherVersionSha256": "b" * 64, + "cipherIntegrity": "verified", + "databaseReadable": True, + "sqliteIntegrity": "verified", + } + + monkeypatch.setattr(provider, "_verify_key_and_integrity", only_old_key_verifies) + + with pytest.raises(RotationError) as caught: + _rotate(provider) + + assert caught.value.code == "rotation_replay_verification_failed" + assert provider.encrypt_key == OLD_KEY + assert provider.key_id == "old-key-2026" + + +def test_sqlcipher_rotation_rejects_tampered_passed_receipt_without_leaking_it( + tmp_path: Path, + monkeypatch: pytest.MonkeyPatch, +) -> None: + provider, _transitions = _rotation_provider(tmp_path, monkeypatch) + _rotate(provider) + journal = json.loads(provider._rotation_journal_path.read_text(encoding="utf-8")) + record = journal["rotations"]["rotation-2026-08-a"] + record["receiptEnvelope"]["receipt"]["attackerControlled"] = str(tmp_path) + provider._rotation_journal_path.write_text( + json.dumps(journal, sort_keys=True, separators=(",", ":")) + "\n", + encoding="utf-8", + ) + + with pytest.raises(RotationError) as caught: + _rotate(provider) + + assert caught.value.code == "invalid_rotation_journal" + assert str(tmp_path) not in str(caught.value) + assert caught.value.receipt is None + + +def test_sqlcipher_rotation_rejects_untrusted_in_progress_verification_fields( + tmp_path: Path, + monkeypatch: pytest.MonkeyPatch, +) -> None: + provider, _transitions = _rotation_provider(tmp_path, monkeypatch) + request = provider._rotation_request( + rotation_id="rotation-2026-08-a", + old_key_id="old-key-2026", + new_key_id="new-key-2026", + new_key=NEW_KEY, + release_sha=RELEASE_SHA, + database_identity_sha256=provider._database_identity_sha256(), + ) + record: dict[str, Any] = { + "status": "in_progress", + "requestSha256": request["requestSha256"], + "request": request, + "preVerification": { + "cipherIntegrity": str(tmp_path), + "databaseReadable": True, + }, + "preDatabaseSha256": provider._database_sha256(), + } + provider._save_rotation_record( + { + "schemaVersion": "2026-08-sqlcipher-key-rotation-journal-v1", + "rotations": {"rotation-2026-08-a": record}, + }, + "rotation-2026-08-a", + record, + ) + provider._apply_rekey(OLD_KEY, NEW_KEY) + + with pytest.raises(RotationError) as caught: + _rotate(provider) + + assert caught.value.code == "invalid_rotation_journal" + assert str(tmp_path) not in str(caught.value) + + +@pytest.mark.skipif(os.name == "nt", reason="POSIX special-file and mode semantics") +@pytest.mark.parametrize("unsafe_kind", ["fifo", "world_writable"]) +def test_sqlcipher_rotation_rejects_unsafe_lock_files( + tmp_path: Path, + monkeypatch: pytest.MonkeyPatch, + unsafe_kind: str, +) -> None: + provider, _transitions = _rotation_provider(tmp_path, monkeypatch) + lock_path = provider._rotation_lock_path + if unsafe_kind == "fifo": + os.mkfifo(lock_path) + else: + lock_path.write_bytes(b"lock") + lock_path.chmod(0o666) + + with pytest.raises(RotationError) as caught: + with provider._rotation_admission(0.2): + pass + + assert caught.value.code == "unsafe_rotation_lock" + + +@pytest.mark.skipif(os.name == "nt", reason="POSIX ownership and mode semantics") +def test_sqlcipher_rotation_rejects_world_writable_journal( + tmp_path: Path, + monkeypatch: pytest.MonkeyPatch, +) -> None: + provider, _transitions = _rotation_provider(tmp_path, monkeypatch) + _rotate(provider) + provider._rotation_journal_path.chmod(0o666) + + with pytest.raises(RotationError) as caught: + _rotate(provider) + + assert caught.value.code == "unsafe_rotation_journal" + + +@pytest.mark.skipif(os.name == "nt", reason="POSIX ownership and mode semantics") +def test_sqlcipher_rotation_rejects_world_writable_storage_directory( + tmp_path: Path, + monkeypatch: pytest.MonkeyPatch, +) -> None: + provider, transitions = _rotation_provider(tmp_path, monkeypatch) + original_mode = tmp_path.stat().st_mode & 0o777 + tmp_path.chmod(0o777) + try: + with pytest.raises(RotationError) as caught: + _rotate(provider) + finally: + tmp_path.chmod(original_mode) + + assert caught.value.code == "unsafe_storage_directory" + assert transitions == [] + + +def test_sqlcipher_rotation_fences_database_identity_swap_before_rekey( + tmp_path: Path, + monkeypatch: pytest.MonkeyPatch, +) -> None: + provider, transitions = _rotation_provider(tmp_path, monkeypatch) + database_identity = provider._database_identity_sha256() + calls = 0 + + def changing_identity() -> str: + nonlocal calls + calls += 1 + return database_identity if calls == 1 else "9" * 64 + + monkeypatch.setattr(provider, "_database_identity_sha256", changing_identity) + + with pytest.raises(RotationError) as caught: + _rotate(provider) + + assert caught.value.code == "database_identity_changed" + assert transitions == [] + + +@pytest.mark.parametrize( + ("overrides", "code"), + [ + ({"old_key_id": "same", "new_key_id": "same"}, "rotation_key_id_conflict"), + ({"new_key": OLD_KEY}, "rotation_key_material_not_distinct"), + ({"release_sha": "not-a-sha"}, "rotation_release_sha_invalid"), + ({"rotation_id": "../customer/path"}, "rotation_id_invalid"), + ], +) +def test_sqlcipher_rotation_request_validation_is_fail_closed( + tmp_path: Path, + monkeypatch: pytest.MonkeyPatch, + overrides: dict[str, str], + code: str, +) -> None: + provider, transitions = _rotation_provider(tmp_path, monkeypatch) + + with pytest.raises(RotationError) as caught: + _rotate(provider, **overrides) + + assert caught.value.code == code + assert transitions == [] + + +class _UnavailableSqlCipherStore: + @staticmethod + def get_encryption_status() -> dict[str, object]: + return {"driverAvailable": False, "active": False} + + +def test_disposable_sqlcipher_rehearsal_reports_not_available_without_claiming_evidence() -> None: + report = run_disposable_sqlcipher_rehearsal( + release_sha=RELEASE_SHA, + store_factory=lambda _path, _key, _key_id: _UnavailableSqlCipherStore(), + ) + + assert report == { + "schemaVersion": "2026-08-disposable-sqlcipher-rehearsal-v1", + "status": "not_available", + "reasonCode": "sqlcipher_driver_unavailable", + "releaseSha": RELEASE_SHA, + "liveSqlcipherTested": False, + "externalEvidence": False, + "releaseEvidenceEligible": False, + "modelProvidersContacted": False, + } + + +def _attestation_payload(**changes) -> dict: + issued_at = int(time.time()) + payload = { + "schemaVersion": ATTESTATION_SCHEMA_VERSION, + "releaseSha": RELEASE_SHA, + "signingKeyId": "storage-attestation-key-2026", + "issuedAtUnixSeconds": issued_at, + "expiresAtUnixSeconds": issued_at + 3600, + "deployment": { + "deploymentSha256": "b" * 64, + "runnerSha256": "c" * 64, + "databaseEngine": "postgresql", + "platform": "aws-rds-postgresql", + }, + "encryption": { + "atRestVerified": True, + "algorithm": "AES-256-GCM", + "provider": "aws-kms", + "keyIdentitySha256": "d" * 64, + "hardwareBacked": True, + }, + "rotation": { + "rehearsalVerified": True, + "oldKeyIdentitySha256": "e" * 64, + "newKeyIdentitySha256": "d" * 64, + "rotationReceiptSha256": "1" * 64, + "completedAtUnixSeconds": issued_at - 60, + "currentKeyVerified": True, + "previousKeyDisabled": True, + }, + "backupRestore": { + "encryptedBackupVerified": True, + "restoreVerified": True, + "restoredAtRestVerified": True, + "backupArtifactSha256": "2" * 64, + "restoredLogicalSha256": "3" * 64, + "completedAtUnixSeconds": issued_at - 60, + }, + "privacy": { + "containsDsn": False, + "containsFilesystemPaths": False, + "containsCustomerData": False, + }, + } + payload.update(changes) + return payload + + +def _write_attestation( + tmp_path: Path, + payload: dict | None = None, +) -> tuple[Path, Path, Path]: + signing_key = Ed25519PrivateKey.generate() + receipt_path = tmp_path / "storage-encryption-attestation.json" + signature_path = tmp_path / "storage-encryption-attestation.ed25519" + public_key_path = tmp_path / "trusted-storage-attestation.pem" + raw = json.dumps( + payload or _attestation_payload(), + sort_keys=True, + separators=(",", ":"), + ).encode() + b"\n" + receipt_path.write_bytes(raw) + signature_path.write_bytes(signing_key.sign(raw)) + public_key_path.write_bytes( + signing_key.public_key().public_bytes( + encoding=serialization.Encoding.PEM, + format=serialization.PublicFormat.SubjectPublicKeyInfo, + ) + ) + return receipt_path, signature_path, public_key_path + + +def _public_key_sha256(public_key_path: Path) -> str: + loaded = serialization.load_pem_public_key(public_key_path.read_bytes()) + assert isinstance(loaded, Ed25519PublicKey) + raw = loaded.public_bytes( + encoding=serialization.Encoding.Raw, + format=serialization.PublicFormat.Raw, + ) + return hashlib.sha256(raw).hexdigest() + + +def _configure_runtime_attestation( + monkeypatch: pytest.MonkeyPatch, + receipt: Path, + signature: Path, + public_key: Path, +) -> None: + monkeypatch.setenv("STORAGE_ENCRYPTION_ATTESTATION_PATH", str(receipt)) + monkeypatch.setenv("STORAGE_ENCRYPTION_ATTESTATION_SIGNATURE_PATH", str(signature)) + monkeypatch.setenv("STORAGE_ENCRYPTION_ATTESTATION_PUBLIC_KEY_PATH", str(public_key)) + monkeypatch.setenv("STORAGE_ENCRYPTION_ATTESTATION_KEY_ID", "storage-attestation-key-2026") + monkeypatch.setenv("STORAGE_ENCRYPTION_ATTESTATION_RELEASE_SHA", RELEASE_SHA) + monkeypatch.setenv("STORAGE_ENCRYPTION_ATTESTATION_DEPLOYMENT_SHA256", "b" * 64) + monkeypatch.setenv("STORAGE_ENCRYPTION_ATTESTATION_RUNNER_SHA256", "c" * 64) + monkeypatch.setenv("STORAGE_ENCRYPTION_ATTESTATION_KEY_IDENTITY_SHA256", "d" * 64) + monkeypatch.setenv( + "STORAGE_ENCRYPTION_ATTESTATION_PUBLIC_KEY_SHA256", + _public_key_sha256(public_key), + ) + + +def _validate_attestation(tmp_path: Path, payload: dict | None = None): + receipt, signature, public_key = _write_attestation(tmp_path, payload) + repository = tmp_path / "repository" + repository.mkdir() + return validate_storage_encryption_attestation( + receipt_path=receipt, + signature_path=signature, + trusted_public_key_path=public_key, + trusted_signing_key_id="storage-attestation-key-2026", + expected_release_sha=RELEASE_SHA, + expected_deployment_sha256="b" * 64, + expected_runner_sha256="c" * 64, + expected_key_identity_sha256="d" * 64, + expected_public_key_sha256=_public_key_sha256(public_key), + repository_root=repository, + ) + + +def test_external_storage_encryption_attestation_is_authenticated_typed_and_private( + tmp_path: Path, +) -> None: + report = _validate_attestation(tmp_path) + + assert report["status"] == "passed" + assert report["releaseSha"] == RELEASE_SHA + assert report["authenticatedExternalAttestation"] is True + assert report["primaryStoreProtected"] is True + assert report["databaseEngine"] == "postgresql" + assert report["algorithm"] == "AES-256-GCM" + assert report["provider"] == "aws-kms" + assert report["keyIdentitySha256"] == "d" * 64 + assert report["signatureVerified"] is True + assert report["releaseEvidenceCandidate"]["details"] == { + "algorithm": "AES-256-GCM", + "authenticatedExternalAttestation": True, + "backupArtifactSha256": "2" * 64, + "backupRestoreVerified": True, + "currentKeyVerified": True, + "deploymentSha256": "b" * 64, + "keyIdentitySha256": "d" * 64, + "primaryStoreProtected": True, + "previousKeyDisabled": True, + "provider": "aws-kms", + "rotationRehearsalVerified": True, + "rotationReceiptSha256": "1" * 64, + "runnerSha256": "c" * 64, + "restoredLogicalSha256": "3" * 64, + "successful": True, + } + rendered = json.dumps(report, sort_keys=True) + assert str(tmp_path) not in rendered + assert "postgresql://" not in rendered + assert "customer" not in rendered.lower() + + +@pytest.mark.parametrize( + ("mutator", "message"), + [ + ( + lambda payload: payload["encryption"].update(provider="unreviewed-provider"), + "provider", + ), + ( + lambda payload: payload["encryption"].update(algorithm="AES-128-CBC"), + "algorithm", + ), + ( + lambda payload: payload["encryption"].update(atRestVerified=False), + "at-rest", + ), + ( + lambda payload: payload["rotation"].update( + newKeyIdentitySha256=payload["rotation"]["oldKeyIdentitySha256"] + ), + "distinct", + ), + ( + lambda payload: payload["encryption"].update(keyIdentitySha256="9" * 64), + "active key", + ), + ( + lambda payload: payload["rotation"].update(currentKeyVerified=False), + "active key", + ), + ( + lambda payload: payload["rotation"].update(previousKeyDisabled=False), + "previous key", + ), + ( + lambda payload: payload["rotation"].update(completedAtUnixSeconds=1), + "stale", + ), + ( + lambda payload: payload["backupRestore"].update(restoreVerified=False), + "backup", + ), + ( + lambda payload: payload["backupRestore"].update(completedAtUnixSeconds=1), + "stale", + ), + ( + lambda payload: payload["privacy"].update(containsDsn=True), + "privacy", + ), + ], +) +def test_external_attestation_fails_closed_on_false_or_unreviewed_claims( + tmp_path: Path, + mutator, + message: str, +) -> None: + payload = _attestation_payload() + mutator(payload) + + with pytest.raises(StorageEncryptionAttestationError, match=message): + _validate_attestation(tmp_path, payload) + + +def test_external_attestation_rejects_signature_tamper_and_in_repository_key( + tmp_path: Path, +) -> None: + receipt, signature, public_key = _write_attestation(tmp_path) + signature.write_bytes(b"x" * 64) + repository = tmp_path / "repository" + repository.mkdir() + with pytest.raises(StorageEncryptionAttestationError, match="signature"): + validate_storage_encryption_attestation( + receipt_path=receipt, + signature_path=signature, + trusted_public_key_path=public_key, + trusted_signing_key_id="storage-attestation-key-2026", + expected_release_sha=RELEASE_SHA, + repository_root=repository, + ) + + receipt, signature, public_key = _write_attestation(repository) + with pytest.raises(StorageEncryptionAttestationError, match="outside"): + validate_storage_encryption_attestation( + receipt_path=receipt, + signature_path=signature, + trusted_public_key_path=public_key, + trusted_signing_key_id="storage-attestation-key-2026", + expected_release_sha=RELEASE_SHA, + repository_root=repository, + ) + + +def test_external_attestation_rejects_stale_signed_evidence( + tmp_path: Path, +) -> None: + payload = _attestation_payload( + issuedAtUnixSeconds=1, + expiresAtUnixSeconds=2, + ) + + with pytest.raises(StorageEncryptionAttestationError, match="expired"): + _validate_attestation(tmp_path, payload) + + +@pytest.mark.skipif(os.name == "nt", reason="POSIX ownership and mode semantics") +def test_external_attestation_rejects_writable_trust_key( + tmp_path: Path, +) -> None: + receipt, signature, public_key = _write_attestation(tmp_path) + public_key.chmod(0o666) + repository = tmp_path / "repository" + repository.mkdir() + + with pytest.raises(StorageEncryptionAttestationError, match="permissions"): + validate_storage_encryption_attestation( + receipt_path=receipt, + signature_path=signature, + trusted_public_key_path=public_key, + trusted_signing_key_id="storage-attestation-key-2026", + expected_release_sha=RELEASE_SHA, + repository_root=repository, + ) + + +def test_external_attestation_binds_expected_deployment_runner_key_and_trust_root( + tmp_path: Path, +) -> None: + receipt, signature, public_key = _write_attestation(tmp_path) + repository = tmp_path / "repository" + repository.mkdir() + initial = validate_storage_encryption_attestation( + receipt_path=receipt, + signature_path=signature, + trusted_public_key_path=public_key, + trusted_signing_key_id="storage-attestation-key-2026", + expected_release_sha=RELEASE_SHA, + repository_root=repository, + ) + + expected = { + "expected_deployment_sha256": "b" * 64, + "expected_runner_sha256": "c" * 64, + "expected_key_identity_sha256": "d" * 64, + "expected_public_key_sha256": initial["publicKeySha256"], + } + for field, message in ( + ("expected_deployment_sha256", "deployment"), + ("expected_runner_sha256", "runner"), + ("expected_key_identity_sha256", "active key"), + ("expected_public_key_sha256", "trust root"), + ): + mismatched = {**expected, field: "9" * 64} + with pytest.raises(StorageEncryptionAttestationError, match=message): + validate_storage_encryption_attestation( + receipt_path=receipt, + signature_path=signature, + trusted_public_key_path=public_key, + trusted_signing_key_id="storage-attestation-key-2026", + expected_release_sha=RELEASE_SHA, + repository_root=repository, + **mismatched, + ) + + +def test_operator_attestation_unexpected_failure_is_sanitized_json( + monkeypatch: pytest.MonkeyPatch, + capsys: pytest.CaptureFixture[str], +) -> None: + monkeypatch.setattr( + operator_cli, + "validate_storage_encryption_attestation", + lambda **_kwargs: (_ for _ in ()).throw( + RuntimeError("/private/customer/attestation.pem secret") + ), + ) + + exit_code = operator_cli.main( + [ + "storage-encryption-attestation", + "--attestation", + "/private/customer/receipt.json", + "--attestation-signature", + "/private/customer/receipt.sig", + "--attestation-public-key", + "/private/customer/trust.pem", + "--attestation-key-id", + "storage-attestation-key-2026", + "--release-sha", + RELEASE_SHA, + ] + ) + rendered = capsys.readouterr().out + + assert exit_code == 1 + assert json.loads(rendered)["status"] == "blocked" + assert "private/customer" not in rendered + assert "secret" not in rendered + + +def test_runtime_external_attestation_requires_complete_out_of_band_configuration( + tmp_path: Path, + monkeypatch: pytest.MonkeyPatch, +) -> None: + for env_name in _RUNTIME_ATTESTATION_ENV: + monkeypatch.delenv(env_name, raising=False) + assert runtime_external_storage_encryption_status()["status"] == "not_configured" + + receipt, signature, public_key = _write_attestation(tmp_path) + _configure_runtime_attestation(monkeypatch, receipt, signature, public_key) + monkeypatch.setenv("STORAGE_ENCRYPTION_ATTESTATION_REPOSITORY_ROOT", str(tmp_path / "repo")) + (tmp_path / "repo").mkdir() + + status = runtime_external_storage_encryption_status() + + assert status["status"] == "authenticated" + assert status["primaryStoreProtected"] is True + assert status["authenticatedExternalAttestation"] is True + assert str(tmp_path) not in json.dumps(status) + + +def test_primary_postgres_and_soc2_require_authenticated_external_attestation( + tmp_path: Path, + monkeypatch: pytest.MonkeyPatch, +) -> None: + import backend.config as config + from backend.api.security.soc2 import SOC2Auditor + from backend.storage.encrypted_store import get_primary_storage_encryption_status + + receipt, signature, public_key = _write_attestation(tmp_path) + monkeypatch.setattr(config, "DB_ENGINE", "postgres") + _configure_runtime_attestation(monkeypatch, receipt, signature, public_key) + + posture = get_primary_storage_encryption_status() + control = SOC2Auditor().check_encryption() + + assert posture["status"] == "active" + assert posture["primaryStoreProtected"] is True + assert posture["authenticatedExternalAttestation"] is True + assert control["passed"] is True + assert str(tmp_path) not in json.dumps(posture) + + +def test_sqlite_posture_never_passes_on_configuration_alone( + tmp_path: Path, + monkeypatch: pytest.MonkeyPatch, +) -> None: + database = tmp_path / "configured-but-unverified.db" + database.write_bytes(b"not-sqlcipher") + provider = EncryptedSessionStore( + db_path=str(database), + encrypt_key=OLD_KEY, + key_id="old-key-2026", + encrypt_enabled=True, + ) + provider._sqlcipher_available = True + provider._sqlcipher_driver = "test-sqlcipher" + monkeypatch.setattr( + provider, + "_verify_key_and_integrity", + lambda _key: (_ for _ in ()).throw(RuntimeError("/private/customer/key")), + ) + + posture = provider.get_encryption_status() + + assert posture["status"] == "blocked" + assert posture["active"] is False + assert posture["primaryStoreProtected"] is False + assert posture["reasonCode"] == "runtime_verification_failed" + assert str(tmp_path) not in json.dumps(posture) + + +def test_operator_sqlcipher_rehearsal_preserves_not_available_truth( + monkeypatch: pytest.MonkeyPatch, + capsys: pytest.CaptureFixture[str], +) -> None: + monkeypatch.setattr( + operator_cli, + "run_disposable_sqlcipher_rehearsal", + lambda **_kwargs: { + "schemaVersion": "2026-08-disposable-sqlcipher-rehearsal-v1", + "status": "not_available", + "reasonCode": "sqlcipher_driver_unavailable", + "releaseSha": RELEASE_SHA, + "liveSqlcipherTested": False, + "externalEvidence": False, + "releaseEvidenceEligible": False, + "modelProvidersContacted": False, + }, + ) + + exit_code = operator_cli.main(["sqlcipher-rehearsal", "--release-sha", RELEASE_SHA]) + output = json.loads(capsys.readouterr().out) + + assert exit_code == 1 + assert output["status"] == "not_available" + assert output["releaseEvidenceEligible"] is False + assert output["modelProvidersContacted"] is False + + +def test_operator_external_attestation_emits_typed_candidate_without_paths( + tmp_path: Path, + capsys: pytest.CaptureFixture[str], +) -> None: + receipt, signature, public_key = _write_attestation(tmp_path) + + exit_code = operator_cli.main( + [ + "storage-encryption-attestation", + "--attestation", + str(receipt), + "--attestation-signature", + str(signature), + "--attestation-public-key", + str(public_key), + "--attestation-key-id", + "storage-attestation-key-2026", + "--release-sha", + RELEASE_SHA, + "--attestation-deployment-sha256", + "b" * 64, + "--attestation-runner-sha256", + "c" * 64, + "--attestation-key-identity-sha256", + "d" * 64, + "--attestation-public-key-sha256", + _public_key_sha256(public_key), + ] + ) + rendered = capsys.readouterr().out + output = json.loads(rendered) + + assert exit_code == 0 + assert output["status"] == "passed" + assert output["releaseEvidenceCandidate"]["id"] == "storage_encryption_attestation" + assert str(tmp_path) not in rendered diff --git a/backend/tests/test_template_composer.py b/backend/tests/test_template_composer.py new file mode 100644 index 0000000..71f6e53 --- /dev/null +++ b/backend/tests/test_template_composer.py @@ -0,0 +1,12 @@ +from backend.templates.composer import TemplateComposer + + +def test_composer_uses_live_function_registry() -> None: + operations = TemplateComposer().compose("full_valuation") + + assert operations + assert {operation.get("sheet") for operation in operations} >= { + "Financials", + "DCF", + "Sensitivity", + } diff --git a/backend/tests/test_terminal_retention_hook.py b/backend/tests/test_terminal_retention_hook.py new file mode 100644 index 0000000..38b1384 --- /dev/null +++ b/backend/tests/test_terminal_retention_hook.py @@ -0,0 +1,154 @@ +import asyncio +from types import SimpleNamespace + +import pytest +from starlette.requests import Request + +from backend.api.routes import tasks +from backend.services.retention import ImmediateRetentionError, purge_terminal_session_if_immediate + + +class _AskStore: + def __init__(self) -> None: + self.messages: list[tuple[str, str, str]] = [] + self.audits: list[tuple[str, str]] = [] + + def add_message(self, session_id: str, role: str, content: str, **_kwargs) -> None: + self.messages.append((session_id, role, content)) + + def record_audit_event(self, session_id: str, event_type: str, **_kwargs) -> str: + self.audits.append((session_id, event_type)) + return "audit-ask" + + +def _request() -> Request: + return Request({"type": "http", "method": "POST", "path": "/api/v1/tasks/ask", "headers": []}) + + +def _prepare_ask(monkeypatch: pytest.MonkeyPatch, store: _AskStore) -> None: + async def answer(*_args, **_kwargs) -> str: + return "answer" + + monkeypatch.setattr(tasks, "get_session_store", lambda: store) + monkeypatch.setattr(tasks, "_handle_ask_mode", answer) + monkeypatch.setattr(tasks, "_claim_request_session", lambda *_args, **_kwargs: None) + monkeypatch.setattr(tasks, "_enforce_ai_policy", lambda *_args, **_kwargs: None) + + +def test_zero_hour_retention_awaits_terminal_session_purge(monkeypatch) -> None: + purged: list[str] = [] + + async def purge_session(session_id: str): + purged.append(session_id) + return {"state": "verified", "receipt_id": "receipt-1"} + + monkeypatch.setattr( + "backend.api.security.rbac.get_zdr", + lambda: SimpleNamespace(enabled=True, retention_hours=0, purge_session=purge_session), + ) + + receipt = asyncio.run(purge_terminal_session_if_immediate("session-1", "applied")) + + assert receipt == {"state": "verified", "receipt_id": "receipt-1"} + assert purged == ["session-1"] + + +def test_zero_hour_retention_does_not_purge_nonterminal_state(monkeypatch) -> None: + purged: list[str] = [] + + async def purge_session(session_id: str): + purged.append(session_id) + return {"state": "verified"} + + monkeypatch.setattr( + "backend.api.security.rbac.get_zdr", + lambda: SimpleNamespace(enabled=True, retention_hours=0, purge_session=purge_session), + ) + + receipt = asyncio.run(purge_terminal_session_if_immediate("session-1", "awaiting_approval")) + + assert receipt is None + assert purged == [] + + +def test_nonzero_retention_window_defers_terminal_purge(monkeypatch) -> None: + purged: list[str] = [] + + async def purge_session(session_id: str): + purged.append(session_id) + return {"state": "verified"} + + monkeypatch.setattr( + "backend.api.security.rbac.get_zdr", + lambda: SimpleNamespace(enabled=True, retention_hours=24, purge_session=purge_session), + ) + + receipt = asyncio.run(purge_terminal_session_if_immediate("session-1", "failed")) + + assert receipt is None + assert purged == [] + + +def test_zero_hour_terminal_result_fails_closed_without_verified_receipt(monkeypatch) -> None: + async def purge_session(_session_id: str): + return {"state": "failed", "errors": ["cache_purge_failed"]} + + monkeypatch.setattr( + "backend.api.security.rbac.get_zdr", + lambda: SimpleNamespace(enabled=True, retention_hours=0, purge_session=purge_session), + ) + + with pytest.raises(ImmediateRetentionError, match="deletion was not verified"): + asyncio.run(purge_terminal_session_if_immediate("session-1", "completed")) + + +def test_zero_hour_ask_persists_response_then_awaits_verified_deletion(monkeypatch: pytest.MonkeyPatch) -> None: + store = _AskStore() + _prepare_ask(monkeypatch, store) + events: list[str] = [] + + async def purge(session_id: str, terminal_state: str): + assert (session_id, terminal_state) == ("ask-session", "completed") + assert [item[1] for item in store.messages] == ["user", "assistant"] + assert store.audits == [("ask-session", "ask_answered")] + events.append("purged") + store.messages.clear() + store.audits.clear() + return {"state": "verified"} + + monkeypatch.setattr("backend.services.retention.purge_terminal_session_if_immediate", purge) + response = asyncio.run(tasks.ask_task(tasks.AskRequest(prompt="question", session_id="ask-session"), _request())) + + assert response.answer == "answer" + assert events == ["purged"] + assert store.messages == [] + assert store.audits == [] + + +def test_zero_hour_ask_does_not_return_success_when_verified_deletion_fails(monkeypatch: pytest.MonkeyPatch) -> None: + store = _AskStore() + _prepare_ask(monkeypatch, store) + + async def purge(_session_id: str, _terminal_state: str): + raise ImmediateRetentionError("verified deletion failed") + + monkeypatch.setattr("backend.services.retention.purge_terminal_session_if_immediate", purge) + with pytest.raises(ImmediateRetentionError, match="verified deletion failed"): + asyncio.run(tasks.ask_task(tasks.AskRequest(prompt="question", session_id="ask-session"), _request())) + + assert [item[1] for item in store.messages] == ["user", "assistant"] + assert store.audits == [("ask-session", "ask_answered")] + + +def test_deferred_retention_keeps_ask_response_and_persistence_semantics(monkeypatch: pytest.MonkeyPatch) -> None: + store = _AskStore() + _prepare_ask(monkeypatch, store) + + async def defer(_session_id: str, _terminal_state: str): + return None + + monkeypatch.setattr("backend.services.retention.purge_terminal_session_if_immediate", defer) + response = asyncio.run(tasks.ask_task(tasks.AskRequest(prompt="question", session_id="ask-session"), _request())) + + assert response.model_dump() == {"session_id": "ask-session", "answer": "answer", "citations": []} + assert [item[1] for item in store.messages] == ["user", "assistant"] diff --git a/backend/tests/test_v05_import_consumption.py b/backend/tests/test_v05_import_consumption.py new file mode 100644 index 0000000..88acbcb --- /dev/null +++ b/backend/tests/test_v05_import_consumption.py @@ -0,0 +1,708 @@ +from __future__ import annotations + +import asyncio +from concurrent.futures import ThreadPoolExecutor +from contextlib import asynccontextmanager +import hashlib +import json +import os +from pathlib import Path +from types import SimpleNamespace +import uuid + +import pytest + +import backend.operations.v05_import_consumer as v05_consumer +from backend.api.security.ownership import ownership_actor_scope +from backend.migration.v05_bundle import export_v05_bundle +from backend.operations.v05_import_consumer import ( + consume_validated_v05_stage, + main as v05_import_main, +) +from backend.session_store import SessionStore +from backend.storage.postgres import ( + POSTGRES_SESSION_LIFECYCLE_TABLES, + PostgresSessionStore, +) +from backend.storage.v05_import_store import ( + NeutralV05ImportError, + neutral_import_id, + neutral_session_id, +) +from vnext.migration.v05_import import validate_and_stage_bundle + + +class _PortableStore: + def __init__(self, *, owner: str, session_id: str, document_path: Path) -> None: + self._owner = owner + self._session_id = session_id + self._document = { + "document_id": f"document-{session_id}", + "session_id": session_id, + "filename": "source.xlsx", + "content_type": "application/vnd.openxmlformats-officedocument.spreadsheetml.sheet", + "size_bytes": document_path.stat().st_size, + "file_path": str(document_path), + "summary": {"legacy": "must-not-cross"}, + "parsed": {"legacy": "must-not-cross"}, + } + + def session_exists(self, session_id: str) -> bool: + return session_id == self._session_id + + def get_session_owner(self, session_id: str) -> str | None: + return self._owner if self.session_exists(session_id) else None + + def list_documents(self, session_id: str, limit: int = 20) -> list[dict]: + assert limit > 1 + return [dict(self._document)] if session_id == self._session_id else [] + + def get_document(self, document_id: str) -> dict | None: + return dict(self._document) if document_id == self._document["document_id"] else None + + +class _PortableWorkflows: + def __init__(self, *, owner: str, session_id: str) -> None: + self._workflow = SimpleNamespace( + id=f"workflow-{session_id}", + name="Review imported recipe", + description="Neutral recipe only", + steps=[ + SimpleNamespace( + step_type="prompt", + description="Prepare a draft", + config={"prompt": "Prepare a reviewed draft", "temperature": 0}, + ) + ], + variables={"ticker": "Public symbol"}, + tags=["migration"], + created_at="2026-01-01T00:00:00Z", + updated_at="2026-01-02T00:00:00Z", + execution_count=0, + last_executed=None, + owner_user_id=owner, + session_id=session_id, + built_in=False, + ) + + def list_workflows(self) -> list[dict]: + return [{"id": self._workflow.id}] + + def get_workflow(self, workflow_id: str): + return self._workflow if workflow_id == self._workflow.id else None + + +def _stage_bundle( + tmp_path: Path, + *, + owner: str = "owner-alice", + nonce: str = "one", +): + fixture_root = tmp_path / f"fixture-{nonce}" + fixture_root.mkdir() + managed_root = fixture_root / "managed" + managed_root.mkdir() + document_path = managed_root / "source.xlsx" + document_path.write_bytes(b"PK\x03\x04portable-workbook-" + nonce.encode()) + session_id = f"legacy-session-{nonce}" + exported = export_v05_bundle( + store=_PortableStore( + owner=owner, + session_id=session_id, + document_path=document_path, + ), + workflow_engine=_PortableWorkflows(owner=owner, session_id=session_id), + owner_id=owner, + session_ids=[session_id], + managed_documents_root=managed_root, + output_path=fixture_root / "migration.zip", + ) + staged = validate_and_stage_bundle( + bundle_path=exported.archive_path, + expected_archive_sha256=exported.archive_sha256, + target_owner_id=owner, + inbox_root=fixture_root / "inbox", + ) + return exported, staged, session_id + + +def _consume(store, exported, staged, *, owner: str = "owner-alice") -> dict: + return consume_validated_v05_stage( + store=store, + staging_path=staged.staging_path, + expected_archive_sha256=exported.archive_sha256, + expected_content_sha256=exported.content_sha256, + target_owner_id=owner, + ) + + +class _FakeAcquire: + def __init__(self, connection) -> None: + self._connection = connection + + async def __aenter__(self): + return self._connection + + async def __aexit__(self, exc_type, exc, tb) -> bool: + return False + + +class _FakePool: + def __init__(self, connection) -> None: + self._connection = connection + + def acquire(self) -> _FakeAcquire: + return _FakeAcquire(self._connection) + + +class _StatefulPostgresConnection: + def __init__(self) -> None: + self.lifecycle: dict | None = None + self.session: dict | None = None + self.import_row: dict | None = None + self.payloads: list[dict] = [] + self.queries: list[str] = [] + self.transaction_options: list[dict] = [] + + @asynccontextmanager + async def transaction(self, **options): + self.transaction_options.append(dict(options)) + yield + + async def execute(self, query: str, *params): + self.queries.append(query) + if "INSERT INTO session_lifecycle" in query: + self.lifecycle = {"state": "active"} + elif "INSERT INTO sessions" in query: + self.session = {"owner_user_id": params[1]} + elif "INSERT INTO migration_v05_imports" in query: + self.import_row = { + "import_id": params[0], + "session_id": params[1], + "schema_version": params[2], + "state": "validated_pending_import", + "version": 0, + "attempt_count": 0, + "binding_sha256": params[3], + "archive_sha256": params[4], + "content_sha256": params[5], + "owner_scope_sha256": params[6], + "manifest_sha256": params[7], + "stage_receipt_sha256": params[8], + "counts_json": json.loads(params[9]), + "payload_count": 0, + "receipt_json": {}, + "failure_code": "", + } + elif "SET state = 'importing'" in query: + assert self.import_row is not None + self.import_row["state"] = "importing" + self.import_row["version"] += 1 + self.import_row["attempt_count"] += 1 + self.import_row["failure_code"] = "" + elif "INSERT INTO migration_v05_payloads" in query: + self.payloads.append( + { + "session_id": params[1], + "relative_path": params[2], + "payload_kind": params[3], + "payload_sha256": params[4], + "size_bytes": params[5], + "payload_bytes": params[6], + } + ) + elif "SET state = 'imported'" in query: + assert self.import_row is not None + self.import_row["state"] = "imported" + self.import_row["version"] += 1 + self.import_row["payload_count"] = params[0] + self.import_row["receipt_json"] = json.loads(params[1]) + self.import_row["failure_code"] = "" + elif "SET state = 'failed_retryable'" in query: + assert self.import_row is not None + self.import_row["state"] = "failed_retryable" + self.import_row["version"] += 1 + self.import_row["failure_code"] = params[0] + return "UPDATE 1" if query.lstrip().startswith("UPDATE") else "INSERT 0 1" + + async def fetchrow(self, query: str, *params): + self.queries.append(query) + if "AS payload_count" in query and "migration_v05_payloads" in query: + sizes = [int(item["size_bytes"]) for item in self.payloads] + actual_sizes = [len(item["payload_bytes"]) for item in self.payloads] + return { + "payload_count": len(self.payloads), + "declared_bytes": sum(sizes), + "max_declared_bytes": max(sizes, default=0), + "actual_bytes": sum(actual_sizes), + "max_actual_bytes": max(actual_sizes, default=0), + } + if "SELECT state FROM session_lifecycle" in query: + return dict(self.lifecycle) if self.lifecycle else None + if "SELECT owner_user_id FROM sessions" in query: + return dict(self.session) if self.session else None + if "SELECT session.owner_user_id, lifecycle.state" in query: + if not self.session or not self.lifecycle: + return None + return { + "owner_user_id": self.session["owner_user_id"], + "state": self.lifecycle["state"], + } + if "lifecycle.state AS lifecycle_state" in query: + if not self.session or not self.lifecycle: + return None + return { + "owner_user_id": self.session["owner_user_id"], + "lifecycle_state": self.lifecycle["state"], + } + if "SELECT * FROM migration_v05_imports" in query: + return dict(self.import_row) if self.import_row else None + if "SELECT session_id FROM migration_v05_imports" in query: + if not self.import_row: + return None + return {"session_id": self.import_row["session_id"]} + raise AssertionError(f"Unexpected fetchrow query: {query}") + + async def fetch(self, query: str, *params): + self.queries.append(query) + if "FROM migration_v05_payloads" in query: + return [dict(row) for row in sorted(self.payloads, key=lambda item: item["relative_path"])] + raise AssertionError(f"Unexpected fetch query: {query}") + + async def fetchval(self, query: str, *params): + self.queries.append(query) + if "COUNT(*) FROM migration_v05_payloads" in query: + return len(self.payloads) + raise AssertionError(f"Unexpected fetchval query: {query}") + + +def test_sqlite_consumes_stage_into_neutral_lifecycle_storage_only(tmp_path: Path) -> None: + exported, staged, legacy_session_id = _stage_bundle(tmp_path) + store = SessionStore(tmp_path / "runtime.db") + + result = _consume(store, exported, staged) + + assert result["state"] == "imported" + assert result["importId"] == neutral_import_id(exported.content_sha256) + receipt = result["receipt"]["receipt"] + assert receipt["archiveSha256"] == exported.archive_sha256 + assert receipt["contentSha256"] == exported.content_sha256 + assert receipt["neutralStorageOnly"] is True + assert receipt["runtimeWorkbookCreated"] is False + assert receipt["legacyStorageMutated"] is False + assert receipt["reverseConversionAvailable"] is False + serialized = json.dumps(result, sort_keys=True) + assert "owner-alice" not in serialized + assert str(staged.staging_path) not in serialized + + target_session_id = neutral_session_id(exported.content_sha256) + with store._conn() as conn: + assert conn.execute("SELECT COUNT(*) FROM documents").fetchone()[0] == 0 + assert conn.execute( + "SELECT COUNT(*) FROM sessions WHERE session_id = ?", + (legacy_session_id,), + ).fetchone()[0] == 0 + session = conn.execute( + "SELECT owner_user_id FROM sessions WHERE session_id = ?", + (target_session_id,), + ).fetchone() + assert session["owner_user_id"] == "owner-alice" + assert conn.execute( + "SELECT state FROM session_lifecycle WHERE session_id = ?", + (target_session_id,), + ).fetchone()["state"] == "active" + assert conn.execute( + "SELECT COUNT(*) FROM migration_v05_payloads WHERE import_id = ?", + (result["importId"],), + ).fetchone()[0] == len(json.loads((staged.staging_path / "manifest.json").read_text())["payloads"]) + + +def test_sqlite_replay_and_concurrent_consumers_return_exact_receipt(tmp_path: Path) -> None: + exported, staged, _legacy_session_id = _stage_bundle(tmp_path) + db_path = tmp_path / "runtime.db" + store_a = SessionStore(db_path) + store_b = SessionStore(db_path) + + with ThreadPoolExecutor(max_workers=2) as pool: + results = list( + pool.map( + lambda store: _consume(store, exported, staged), + (store_a, store_b), + ) + ) + + replay = _consume(store_a, exported, staged) + assert results[0] == results[1] == replay + with store_a._conn() as conn: + row = conn.execute( + "SELECT attempt_count, state FROM migration_v05_imports WHERE import_id = ?", + (neutral_import_id(exported.content_sha256),), + ).fetchone() + assert dict(row) == {"attempt_count": 1, "state": "imported"} + assert conn.execute( + "SELECT COUNT(*) FROM migration_v05_imports" + ).fetchone()[0] == 1 + + +def test_sqlite_crash_after_import_claim_resumes_exactly(tmp_path: Path, monkeypatch) -> None: + exported, staged, _legacy_session_id = _stage_bundle(tmp_path) + db_path = tmp_path / "runtime.db" + interrupted = SessionStore(db_path) + + def crash_after_claim(*_args, **_kwargs): + raise KeyboardInterrupt + + monkeypatch.setattr(interrupted, "_complete_v05_import", crash_after_claim) + with pytest.raises(KeyboardInterrupt): + _consume(interrupted, exported, staged) + + with interrupted._conn() as conn: + row = conn.execute( + "SELECT state, attempt_count FROM migration_v05_imports WHERE import_id = ?", + (neutral_import_id(exported.content_sha256),), + ).fetchone() + assert dict(row) == {"state": "importing", "attempt_count": 1} + assert conn.execute("SELECT COUNT(*) FROM migration_v05_payloads").fetchone()[0] == 0 + + resumed = _consume(SessionStore(db_path), exported, staged) + assert resumed["state"] == "imported" + assert resumed["attemptCount"] == 1 + + +def test_sqlite_storage_failure_is_retryable_and_resume_increments_attempt( + tmp_path: Path, + monkeypatch, +) -> None: + exported, staged, _legacy_session_id = _stage_bundle(tmp_path) + store = SessionStore(tmp_path / "runtime.db") + original_complete = store._complete_v05_import + + def fail_storage_write(*_args, **_kwargs): + raise RuntimeError("/customer/private/storage.db") + + monkeypatch.setattr(store, "_complete_v05_import", fail_storage_write) + with pytest.raises(NeutralV05ImportError, match="storage_write_failed") as failure: + _consume(store, exported, staged) + assert "/customer/private" not in str(failure.value) + with store._conn() as conn: + row = conn.execute( + "SELECT state, version, attempt_count, failure_code " + "FROM migration_v05_imports WHERE import_id = ?", + (neutral_import_id(exported.content_sha256),), + ).fetchone() + assert dict(row) == { + "state": "failed_retryable", + "version": 2, + "attempt_count": 1, + "failure_code": "storage_write_failed", + } + + monkeypatch.setattr(store, "_complete_v05_import", original_complete) + resumed = _consume(store, exported, staged) + assert resumed["state"] == "imported" + assert resumed["attemptCount"] == 2 + assert resumed["version"] == 4 + + +def test_sqlite_replay_detects_payload_and_receipt_tampering(tmp_path: Path) -> None: + exported, staged, _legacy_session_id = _stage_bundle(tmp_path) + store = SessionStore(tmp_path / "runtime.db") + result = _consume(store, exported, staged) + with store._conn() as conn: + conn.execute( + "UPDATE migration_v05_payloads SET payload_bytes = ? " + "WHERE import_id = ? AND relative_path = " + "(SELECT MIN(relative_path) FROM migration_v05_payloads WHERE import_id = ?)", + (b"tampered", result["importId"], result["importId"]), + ) + with pytest.raises(NeutralV05ImportError, match="corrupt_import_payloads"): + _consume(store, exported, staged) + + with store._conn() as conn: + payload = next( + item + for item in json.loads((staged.staging_path / "manifest.json").read_text())["payloads"] + if item["path"] + == conn.execute( + "SELECT MIN(relative_path) FROM migration_v05_payloads WHERE import_id = ?", + (result["importId"],), + ).fetchone()[0] + ) + original = (staged.staging_path / payload["path"]).read_bytes() + conn.execute( + "UPDATE migration_v05_payloads SET payload_bytes = ? WHERE import_id = ? AND relative_path = ?", + (original, result["importId"], payload["path"]), + ) + conn.execute( + "UPDATE migration_v05_imports SET receipt_json = ? WHERE import_id = ?", + ('{"receipt":{},"receiptSha256":"' + "0" * 64 + '"}', result["importId"]), + ) + with pytest.raises(NeutralV05ImportError, match="corrupt_import_receipt"): + _consume(store, exported, staged) + + +def test_stage_reader_detects_inode_swap_during_descriptor_read( + tmp_path: Path, + monkeypatch, +) -> None: + exported, staged, _legacy_session_id = _stage_bundle(tmp_path) + manifest_path = staged.staging_path / "manifest.json" + original_manifest = manifest_path.read_bytes() + original_read = v05_consumer.os.read + swapped = False + + def swap_path_after_open(file_descriptor: int, size: int) -> bytes: + nonlocal swapped + if not swapped: + swapped = True + displaced = staged.staging_path / ".manifest.displaced" + os.rename(manifest_path, displaced) + manifest_path.write_bytes(original_manifest) + os.chmod(manifest_path, 0o600) + return original_read(file_descriptor, size) + + monkeypatch.setattr(v05_consumer.os, "read", swap_path_after_open) + with pytest.raises(NeutralV05ImportError, match="stage_changed_during_read"): + _consume(SessionStore(tmp_path / "runtime.db"), exported, staged) + + +def test_import_fences_owner_conflict_and_lifecycle_deletion(tmp_path: Path) -> None: + exported, staged, _legacy_session_id = _stage_bundle(tmp_path) + store = SessionStore(tmp_path / "runtime.db") + + with ownership_actor_scope({"user_id": "mallory", "role": "user"}): + with pytest.raises(NeutralV05ImportError, match="owner_scope_forbidden") as forbidden: + _consume(store, exported, staged) + assert forbidden.value.code == "owner_scope_forbidden" + assert "owner-alice" not in str(forbidden.value) + assert str(staged.staging_path) not in str(forbidden.value) + + result = _consume(store, exported, staged) + target_session_id = neutral_session_id(exported.content_sha256) + store.set_session_lifecycle_state( + target_session_id, + "purging", + receipt_id="purge-v05-neutral", + started_at=1.0, + ) + assert store.delete_session(target_session_id) is True + store.set_session_lifecycle_state( + target_session_id, + "verified", + receipt_id="purge-v05-neutral", + started_at=1.0, + completed_at=2.0, + ) + assert store.get_v05_import(result["importId"]) is None + with pytest.raises(NeutralV05ImportError, match="session_fenced"): + _consume(store, exported, staged) + + +def test_import_rejects_binding_conflict_and_unsafe_stage(tmp_path: Path) -> None: + exported, staged, _legacy_session_id = _stage_bundle(tmp_path) + store = SessionStore(tmp_path / "runtime.db") + _consume(store, exported, staged) + + receipt_path = staged.staging_path / "receipt.json" + receipt = json.loads(receipt_path.read_text()) + conflicting_archive = "f" * 64 if exported.archive_sha256 != "f" * 64 else "e" * 64 + receipt["archive_sha256"] = conflicting_archive + receipt_path.write_text(json.dumps(receipt, sort_keys=True, separators=(",", ":")) + "\n") + with pytest.raises(NeutralV05ImportError, match="import_binding_conflict"): + consume_validated_v05_stage( + store=store, + staging_path=staged.staging_path, + expected_archive_sha256=conflicting_archive, + expected_content_sha256=exported.content_sha256, + target_owner_id="owner-alice", + ) + + os.chmod(staged.staging_path, 0o777) + with pytest.raises(NeutralV05ImportError, match="unsafe_stage"): + consume_validated_v05_stage( + store=SessionStore(tmp_path / "other.db"), + staging_path=staged.staging_path, + expected_archive_sha256=conflicting_archive, + expected_content_sha256=exported.content_sha256, + target_owner_id="owner-alice", + ) + + +def test_import_cli_emits_only_categorical_failure(capsys, tmp_path: Path) -> None: + secret_path = tmp_path / "customer-secret-stage" + secret_owner = "customer-secret-owner" + + exit_code = v05_import_main( + [ + "--staging-path", + str(secret_path), + "--expected-archive-sha256", + "a" * 64, + "--expected-content-sha256", + "b" * 64, + "--target-owner-id", + secret_owner, + ] + ) + + output = capsys.readouterr() + assert exit_code == 2 + assert json.loads(output.out) == {"errorCode": "stage_unavailable", "status": "blocked"} + assert str(secret_path) not in output.out + output.err + assert secret_owner not in output.out + output.err + + +def test_import_cli_success_is_one_redacted_json_document( + capsys, + monkeypatch, + tmp_path: Path, +) -> None: + import backend.storage as storage_module + + exported, staged, _legacy_session_id = _stage_bundle(tmp_path) + store = SessionStore(tmp_path / "runtime.db") + monkeypatch.setattr(storage_module, "get_session_store", lambda: store) + capsys.readouterr() + + exit_code = v05_import_main( + [ + "--staging-path", + str(staged.staging_path), + "--expected-archive-sha256", + exported.archive_sha256, + "--expected-content-sha256", + exported.content_sha256, + "--target-owner-id", + "owner-alice", + ] + ) + + output = capsys.readouterr() + parsed = json.loads(output.out) + assert exit_code == 0 + assert output.err == "" + assert parsed["state"] == "imported" + assert parsed["receipt"]["receipt"]["neutralStorageOnly"] is True + assert "owner-alice" not in output.out + assert str(staged.staging_path) not in output.out + + +def test_postgres_schema_and_protocol_cover_neutral_import_lifecycle() -> None: + import inspect + + schema_source = inspect.getsource(PostgresSessionStore._init_schema) + assert "CREATE TABLE IF NOT EXISTS migration_v05_imports" in schema_source + assert "CREATE TABLE IF NOT EXISTS migration_v05_payloads" in schema_source + assert "payload_bytes BYTEA" in schema_source + assert "validated_pending_import" in schema_source + assert "failed_retryable" in schema_source + assert "postgres_v05_neutral_import_v1" in schema_source + assert "migration_v05_imports" in POSTGRES_SESSION_LIFECYCLE_TABLES + assert "migration_v05_payloads" in POSTGRES_SESSION_LIFECYCLE_TABLES + assert callable(PostgresSessionStore.consume_v05_import) + assert callable(PostgresSessionStore.get_v05_import) + + +def test_postgres_neutral_import_uses_transactional_cas_and_exact_replay(tmp_path: Path) -> None: + exported, staged, _legacy_session_id = _stage_bundle(tmp_path) + connection = _StatefulPostgresConnection() + store = PostgresSessionStore("postgresql://not-contacted.invalid/example") + store.__dict__["_pool"] = _FakePool(connection) + try: + first = _consume(store, exported, staged) + replay = _consume(store, exported, staged) + finally: + store._stop_sync_loop() + + assert first == replay + assert first["state"] == "imported" + assert first["attemptCount"] == 1 + assert connection.import_row is not None + assert connection.import_row["version"] == 2 + assert len(connection.payloads) == first["receipt"]["receipt"]["storedPayloads"] + sql = "\n".join(connection.queries) + assert "pg_advisory_xact_lock" in sql + assert "FOR UPDATE" in sql + assert "AND version = $4 AND state = $5" in sql + assert "AND version = $6 AND state = 'importing'" in sql + + +def test_postgres_bundle_read_uses_owner_fenced_repeatable_snapshot(tmp_path: Path) -> None: + exported, staged, _legacy_session_id = _stage_bundle(tmp_path) + connection = _StatefulPostgresConnection() + store = PostgresSessionStore("postgresql://not-contacted.invalid/example") + store.__dict__["_pool"] = _FakePool(connection) + try: + imported = _consume(store, exported, staged) + assert connection.import_row is not None + query_start = len(connection.queries) + bundle = store.read_v05_import_bundle( + imported["importId"], + expected_binding_sha256=connection.import_row["binding_sha256"], + expected_content_sha256=exported.content_sha256, + expected_owner_id="owner-alice", + expected_owner_scope_sha256=connection.import_row["owner_scope_sha256"], + expected_target_session_id=neutral_session_id(exported.content_sha256), + ) + finally: + store._stop_sync_loop() + + assert bundle is not None + assert bundle["request"]["ownerId"] == "owner-alice" + assert tuple(item["path"] for item in bundle["payloads"]) == tuple( + sorted(item["relative_path"] for item in connection.payloads) + ) + assert {"isolation": "repeatable_read"} in connection.transaction_options + read_queries = connection.queries[query_start:] + advisory_index = next( + index for index, query in enumerate(read_queries) if "pg_advisory_xact_lock" in query + ) + import_index = next( + index for index, query in enumerate(read_queries) if "migration_v05_imports" in query + ) + assert advisory_index < import_index + + +@pytest.mark.skipif( + not os.getenv("EXCEL_AGENT_TEST_POSTGRES_DSN", "").strip(), + reason="EXCEL_AGENT_TEST_POSTGRES_DSN is required for live disposable parity validation", +) +def test_live_postgres_neutral_import_replay_when_available(tmp_path: Path) -> None: + dsn = os.environ["EXCEL_AGENT_TEST_POSTGRES_DSN"] + nonce = uuid.uuid4().hex + owner = f"v05-live-owner-{nonce}" + exported, staged, _legacy_session_id = _stage_bundle(tmp_path, owner=owner, nonce=nonce) + store = PostgresSessionStore(dsn) + target_session_id = neutral_session_id(exported.content_sha256) + import_id = neutral_import_id(exported.content_sha256) + try: + first = _consume(store, exported, staged, owner=owner) + replay = _consume(store, exported, staged, owner=owner) + assert first == replay + assert first["state"] == "imported" + finally: + async def cleanup() -> None: + pool = await store._get_pool() + async with pool.acquire() as conn: + async with conn.transaction(): + await conn.execute("DELETE FROM migration_v05_payloads WHERE import_id = $1", import_id) + await conn.execute("DELETE FROM migration_v05_imports WHERE import_id = $1", import_id) + await conn.execute("DELETE FROM sessions WHERE session_id = $1", target_session_id) + await conn.execute("DELETE FROM session_lifecycle WHERE session_id = $1", target_session_id) + + try: + store._run(cleanup()) + finally: + asyncio.run(store.close()) + + +def test_receipt_digest_is_exact_and_not_path_or_owner_derived(tmp_path: Path) -> None: + exported, staged, _legacy_session_id = _stage_bundle(tmp_path) + result = _consume(SessionStore(tmp_path / "runtime.db"), exported, staged) + body = result["receipt"]["receipt"] + expected = hashlib.sha256( + json.dumps(body, sort_keys=True, separators=(",", ":"), ensure_ascii=False).encode() + ).hexdigest() + assert result["receipt"]["receiptSha256"] == expected + assert "owner-alice" not in json.dumps(result) + assert str(staged.staging_path) not in json.dumps(result) diff --git a/backend/tests/test_v05_managed_materializer.py b/backend/tests/test_v05_managed_materializer.py new file mode 100644 index 0000000..1afbcb2 --- /dev/null +++ b/backend/tests/test_v05_managed_materializer.py @@ -0,0 +1,608 @@ +from __future__ import annotations + +import asyncio +from concurrent.futures import ThreadPoolExecutor +import json +import os +from pathlib import Path +from typing import Any +import uuid + +import pytest + +from backend.operations.v05_managed_materializer import ( + MATERIALIZATION_MANIFEST_SCHEMA_VERSION, + V05ManagedMaterializationError, + materialize_v05_import, +) +from backend.session_store import SessionStore +from backend.agents import workflows as workflows_module +from backend.services.lifecycle import LifecycleCoordinator, LifecycleState +from backend.storage.managed_object_store import ( + EncryptedManagedObjectStore, + ManagedObjectKind, + ManagedObjectNotFoundError, + ManagedObjectScope, + ManagedObjectStoreConfig, + PostgresManagedObjectMetadataRepository, + SQLiteManagedObjectMetadataRepository, +) +from backend.storage.postgres import PostgresSessionStore +from backend.storage.v05_import_store import ( + STAGE_SCHEMA_VERSION, + binding_sha256, + build_import_receipt, + canonical_json, + neutral_import_id, + neutral_session_id, + sha256_bytes, + sha256_text, +) + + +def _neutral_bundle( + payload_specs: list[tuple[str, str, bytes]], + *, + owner_id: str = "owner-alice", +) -> tuple[dict[str, Any], list[dict[str, Any]]]: + payloads = [ + { + "content": content, + "kind": kind, + "path": path, + "sha256": sha256_bytes(content), + "sizeBytes": len(content), + } + for path, kind, content in sorted(payload_specs) + ] + descriptors = [ + { + "kind": item["kind"], + "path": item["path"], + "sha256": item["sha256"], + "size_bytes": item["sizeBytes"], + } + for item in payloads + ] + content_sha256 = sha256_bytes(canonical_json(descriptors, newline=True)) + counts = { + "documents": sum(item["kind"] == "document" for item in payloads), + "sessions": sum(item["kind"] == "session" for item in payloads), + "workflows": sum(item["kind"] == "workflow_recipe" for item in payloads), + } + request: dict[str, Any] = { + "archiveSha256": "a" * 64, + "contentSha256": content_sha256, + "counts": counts, + "importId": neutral_import_id(content_sha256), + "manifestSha256": "b" * 64, + "ownerId": owner_id, + "ownerScopeSha256": sha256_text(owner_id), + "schemaVersion": STAGE_SCHEMA_VERSION, + "stageReceiptSha256": "c" * 64, + "targetSessionId": neutral_session_id(content_sha256), + } + request["bindingSha256"] = binding_sha256(request) + return request, payloads + + +def _source_store( + tmp_path: Path, + payload_specs: list[tuple[str, str, bytes]] | None = None, + *, + owner_id: str = "owner-alice", +) -> tuple[SessionStore, dict[str, Any], list[dict[str, Any]]]: + request, payloads = _neutral_bundle( + payload_specs + or [ + ("blobs/sha256/one", "blob", b"portable-workbook-one"), + ("records/documents/two.json", "document", b'{"kind":"document"}'), + ("records/sessions/three.json", "session", b'{"kind":"session"}'), + ], + owner_id=owner_id, + ) + store = SessionStore(tmp_path / "sessions.db") + result = store.consume_v05_import(request, payloads) + assert result["state"] == "imported" + return store, request, payloads + + +def _managed_store( + tmp_path: Path, + source: SessionStore, + *, + max_object_bytes: int = 1024 * 1024, + total_bytes: int = 8 * 1024 * 1024, +) -> EncryptedManagedObjectStore: + return EncryptedManagedObjectStore( + ManagedObjectStoreConfig( + enabled=True, + root=tmp_path / "managed", + current_key_id="managed-current", + current_key=b"m" * 32, + previous_keys={}, + address_key=b"a" * 32, + max_object_bytes=max_object_bytes, + total_bytes=total_bytes, + chunk_bytes=4096, + ), + SQLiteManagedObjectMetadataRepository(source.db_path), + ) + + +def _materialize( + source: SessionStore, + managed: EncryptedManagedObjectStore, + request: dict[str, Any], + *, + tenant_id: str = "tenant-one", + owner_id: str = "owner-alice", + coordinator: LifecycleCoordinator | None = None, +) -> dict[str, Any]: + coordinator = coordinator or source.__dict__.get("_v05_materialization_test_coordinator") + if coordinator is None: + coordinator = LifecycleCoordinator( + store=source, + receipts_dir=source.db_path.parent / "lifecycle-receipts", + ) + source.__dict__["_v05_materialization_test_coordinator"] = coordinator + return materialize_v05_import( + source_store=source, + managed_store=managed, + import_id=request["importId"], + expected_binding_sha256=request["bindingSha256"], + tenant_id=tenant_id, + owner_id=owner_id, + lifecycle_coordinator=coordinator, + ) + + +def test_materializes_exact_objects_then_deterministic_manifest_and_replays( + tmp_path: Path, +) -> None: + source, request, payloads = _source_store(tmp_path) + managed = _managed_store(tmp_path, source) + + first = _materialize(source, managed, request) + replay = _materialize(source, managed, request) + + assert first == replay + assert first["status"] == "materialized" + assert first["object_count"] == len(payloads) + assert first["reread_verified"] is True + assert first["materialization_max_payloads"] == 2048 + assert first["materialization_max_entry_bytes"] == 16 * 1024 * 1024 + assert first["materialization_max_payload_bytes"] == 64 * 1024 * 1024 + assert first["production_encryption_claim"] is False + assert first["cloud_provider_contacted"] is False + serialized = json.dumps(first, sort_keys=True) + assert "owner-alice" not in serialized + assert "tenant-one" not in serialized + + scope = ManagedObjectScope("tenant-one", "owner-alice", request["targetSessionId"]) + manifest_bytes = managed.get_bytes(scope, first["manifest_object_id"]) + assert sha256_bytes(manifest_bytes) == first["manifest_sha256"] + manifest = json.loads(manifest_bytes) + assert manifest["schemaVersion"] == MATERIALIZATION_MANIFEST_SCHEMA_VERSION + assert [entry["path"] for entry in manifest["objects"]] == sorted( + item["path"] for item in payloads + ) + by_path = {item["path"]: item for item in payloads} + for entry in manifest["objects"]: + content = managed.get_bytes(scope, entry["objectId"]) + assert content == by_path[entry["path"]]["content"] + assert sha256_bytes(content) == entry["sha256"] + for wrong_scope in ( + ManagedObjectScope("tenant-two", "owner-alice", request["targetSessionId"]), + ManagedObjectScope("tenant-one", "owner-mallory", request["targetSessionId"]), + ManagedObjectScope("tenant-one", "owner-alice", "other-session"), + ): + with pytest.raises(ManagedObjectNotFoundError): + managed.get_bytes(wrong_scope, first["manifest_object_id"]) + with source._conn() as conn: + assert conn.execute("SELECT COUNT(*) FROM documents").fetchone()[0] == 0 + assert conn.execute("SELECT COUNT(*) FROM conversations").fetchone()[0] == 0 + assert conn.execute("SELECT COUNT(*) FROM task_history").fetchone()[0] == 0 + + +def test_same_content_at_multiple_paths_converges_to_one_payload_object( + tmp_path: Path, +) -> None: + duplicate = b"same portable bytes" + source, request, _payloads = _source_store( + tmp_path, + [ + ("blobs/sha256/first", "blob", duplicate), + ("blobs/sha256/second", "blob", duplicate), + ], + ) + managed = _managed_store(tmp_path, source) + + result = _materialize(source, managed, request) + scope = ManagedObjectScope("tenant-one", "owner-alice", request["targetSessionId"]) + manifest = json.loads(managed.get_bytes(scope, result["manifest_object_id"])) + + assert [item["path"] for item in manifest["objects"]] == [ + "blobs/sha256/first", + "blobs/sha256/second", + ] + assert len({item["objectId"] for item in manifest["objects"]}) == 1 + assert result["object_count"] == 2 + assert result["unique_payload_objects"] == 1 + assert managed.operator_status()["objects"] == 2 + + +def test_partial_crash_replays_content_addressed_objects_before_manifest( + tmp_path: Path, + monkeypatch: pytest.MonkeyPatch, +) -> None: + source, request, _payloads = _source_store(tmp_path) + managed = _managed_store(tmp_path, source) + original_put = managed.put_bytes + calls = 0 + + def interrupted_put(scope, kind, payload): + nonlocal calls + calls += 1 + if calls == 2: + raise OSError("/private/operator/path") + return original_put(scope, kind, payload) + + monkeypatch.setattr(managed, "put_bytes", interrupted_put) + with pytest.raises(V05ManagedMaterializationError, match="managed_object_write_failed") as failure: + _materialize(source, managed, request) + assert "/private/operator/path" not in str(failure.value) + assert managed.operator_status()["objects"] == 1 + + monkeypatch.setattr(managed, "put_bytes", original_put) + resumed = _materialize(source, managed, request) + assert resumed["status"] == "materialized" + assert managed.operator_status()["objects"] == resumed["unique_payload_objects"] + 1 + + +def test_exact_payload_and_manifest_reread_proof_is_mandatory( + tmp_path: Path, + monkeypatch: pytest.MonkeyPatch, +) -> None: + source, request, _payloads = _source_store( + tmp_path, + [("blobs/sha256/one", "blob", b"proof-required")], + ) + managed = _managed_store(tmp_path, source) + original_get = managed.get_bytes + reads = 0 + + def corrupt_manifest_reread(scope, object_id): + nonlocal reads + reads += 1 + value = original_get(scope, object_id) + return b"corrupt" if reads == 2 else value + + monkeypatch.setattr(managed, "get_bytes", corrupt_manifest_reread) + with pytest.raises( + V05ManagedMaterializationError, + match="managed_object_reread_failed", + ): + _materialize(source, managed, request) + + +def test_concurrent_materializers_converge_on_one_exact_receipt(tmp_path: Path) -> None: + source, request, _payloads = _source_store(tmp_path) + managed_a = _managed_store(tmp_path, source) + managed_b = _managed_store(tmp_path, source) + + with ThreadPoolExecutor(max_workers=2) as pool: + results = list( + pool.map( + lambda store: _materialize(source, store, request), + (managed_a, managed_b), + ) + ) + + assert results[0] == results[1] + assert managed_a.operator_status()["objects"] == results[0]["unique_payload_objects"] + 1 + + +def test_initial_read_does_not_authorize_writes_after_purge_boundary( + tmp_path: Path, + monkeypatch: pytest.MonkeyPatch, +) -> None: + source, request, _payloads = _source_store( + tmp_path, + [("blobs/sha256/one", "blob", b"one")], + ) + managed = _managed_store(tmp_path, source) + original_read = source.read_v05_import_bundle + + def revoke_after_read(*args, **kwargs): + bundle = original_read(*args, **kwargs) + source.set_session_lifecycle_state( + request["targetSessionId"], + "purging", + receipt_id="purge-after-read", + started_at=1.0, + ) + return bundle + + monkeypatch.setattr(source, "read_v05_import_bundle", revoke_after_read) + with pytest.raises(V05ManagedMaterializationError, match="managed_object_write_failed"): + _materialize(source, managed, request) + assert managed.operator_status()["objects"] == 0 + + +def test_full_source_is_revalidated_before_manifest_publication( + tmp_path: Path, + monkeypatch: pytest.MonkeyPatch, +) -> None: + source, request, _payloads = _source_store( + tmp_path, + [ + ("blobs/sha256/one", "blob", b"one"), + ("blobs/sha256/two", "blob", b"two"), + ], + ) + managed = _managed_store(tmp_path, source) + original_read = source.read_v05_import_bundle + read_count = 0 + written_kinds: list[str] = [] + original_put = managed.put_bytes + + def purge_before_second_read(*args, **kwargs): + nonlocal read_count + read_count += 1 + if read_count == 2: + source.set_session_lifecycle_state( + request["targetSessionId"], + "purging", + receipt_id="purge-before-manifest", + started_at=1.0, + ) + return original_read(*args, **kwargs) + + def record_put(scope, kind, payload): + written_kinds.append(kind.value if isinstance(kind, ManagedObjectKind) else str(kind)) + return original_put(scope, kind, payload) + + monkeypatch.setattr(source, "read_v05_import_bundle", purge_before_second_read) + monkeypatch.setattr(managed, "put_bytes", record_put) + with pytest.raises(V05ManagedMaterializationError, match="neutral_import_unavailable"): + _materialize(source, managed, request) + assert written_kinds == ["document", "document"] + + +def test_tamper_role_mismatch_stale_binding_and_budget_exhaustion_fail_closed( + tmp_path: Path, +) -> None: + source, request, _payloads = _source_store(tmp_path / "tamper") + managed = _managed_store(tmp_path / "tamper", source) + with source._conn() as conn: + conn.execute( + "UPDATE migration_v05_payloads SET payload_bytes = ? " + "WHERE import_id = ? AND relative_path = " + "(SELECT MIN(relative_path) FROM migration_v05_payloads WHERE import_id = ?)", + (b"tampered", request["importId"], request["importId"]), + ) + with pytest.raises(V05ManagedMaterializationError, match="neutral_import_invalid"): + _materialize(source, managed, request) + + clean, clean_request, _ = _source_store(tmp_path / "roles") + clean_managed = _managed_store(tmp_path / "roles", clean) + with pytest.raises(V05ManagedMaterializationError, match="neutral_import_unavailable"): + _materialize(clean, clean_managed, clean_request, owner_id="owner-mallory") + stale_request = dict(clean_request) + stale_request["bindingSha256"] = "f" * 64 + with pytest.raises(V05ManagedMaterializationError, match="neutral_import_unavailable"): + _materialize(clean, clean_managed, stale_request) + + large, large_request, _ = _source_store( + tmp_path / "budget", + [ + ("blobs/sha256/one", "blob", b"1" * 1200), + ("blobs/sha256/two", "blob", b"2" * 1200), + ], + ) + constrained = _managed_store( + tmp_path / "budget", + large, + max_object_bytes=2048, + total_bytes=2200, + ) + with pytest.raises(V05ManagedMaterializationError, match="managed_object_budget_exhausted"): + _materialize(large, constrained, large_request) + + +def test_materialization_specific_source_cap_blocks_large_in_memory_bundle( + tmp_path: Path, + monkeypatch: pytest.MonkeyPatch, +) -> None: + import backend.session_store as session_store_module + + source, request, _payloads = _source_store( + tmp_path, + [("blobs/sha256/one", "blob", b"bounded-source")], + ) + managed = _managed_store(tmp_path, source) + monkeypatch.setattr(session_store_module, "MATERIALIZATION_MAX_ENTRY_BYTES", 1) + + with pytest.raises( + V05ManagedMaterializationError, + match="neutral_import_budget_exhausted", + ): + _materialize(source, managed, request) + assert managed.operator_status()["objects"] == 0 + + +def test_malformed_empty_inventory_and_owner_row_drift_are_rejected_in_snapshot( + tmp_path: Path, +) -> None: + source, request, _payloads = _source_store(tmp_path) + with source._conn() as conn: + conn.execute( + "DELETE FROM migration_v05_payloads WHERE import_id = ?", + (request["importId"],), + ) + empty_receipt = build_import_receipt(request, stored_payloads=0) + conn.execute( + "UPDATE migration_v05_imports SET payload_count = 0, receipt_json = ? " + "WHERE import_id = ?", + ( + canonical_json(empty_receipt).decode("utf-8"), + request["importId"], + ), + ) + with pytest.raises(Exception, match="corrupt_import_payloads"): + source.read_v05_import_bundle( + request["importId"], + expected_binding_sha256=request["bindingSha256"], + expected_content_sha256=request["contentSha256"], + expected_owner_id=request["ownerId"], + expected_owner_scope_sha256=request["ownerScopeSha256"], + expected_target_session_id=request["targetSessionId"], + ) + + owner_source, owner_request, _ = _source_store(tmp_path / "owner") + with owner_source._conn() as conn: + conn.execute( + "UPDATE sessions SET owner_user_id = ? WHERE session_id = ?", + ("owner-mallory", owner_request["targetSessionId"]), + ) + assert ( + owner_source.read_v05_import_bundle( + owner_request["importId"], + expected_binding_sha256=owner_request["bindingSha256"], + expected_content_sha256=owner_request["contentSha256"], + expected_owner_id=owner_request["ownerId"], + expected_owner_scope_sha256=owner_request["ownerScopeSha256"], + expected_target_session_id=owner_request["targetSessionId"], + ) + is None + ) + + +def test_lifecycle_purge_removes_neutral_and_managed_bytes( + tmp_path: Path, + monkeypatch: pytest.MonkeyPatch, +) -> None: + class Jobs: + @staticmethod + def purge_session(_session_id: str) -> dict[str, int]: + return {"jobs_revoked": 0, "threads_pending": 0} + + class Workflows: + @staticmethod + def purge_session(_session_id: str) -> int: + return 0 + + @staticmethod + def count_session_workflows(_session_id: str) -> int: + return 0 + + source, request, _payloads = _source_store(tmp_path) + managed = _managed_store(tmp_path, source) + monkeypatch.setattr(workflows_module, "get_workflow_engine", lambda: Workflows()) + coordinator = LifecycleCoordinator( + store=source, + job_manager=Jobs(), + receipts_dir=tmp_path / "signed-lifecycle-receipts", + receipt_signing_keys={"current": b"v05-materialization-purge-key" * 2}, + current_receipt_key_id="current", + ) + result = _materialize(source, managed, request, coordinator=coordinator) + + deletion = asyncio.run(coordinator.purge_session(request["targetSessionId"])) + + assert deletion.state is LifecycleState.VERIFIED + assert deletion.registered_purgers["managed_encrypted_objects"]["residual_files"] == 0 + assert source.get_v05_import(request["importId"]) is None + assert managed.operator_status()["objects"] == 0 + with pytest.raises(ManagedObjectNotFoundError): + managed.get_bytes( + ManagedObjectScope("tenant-one", "owner-alice", request["targetSessionId"]), + result["manifest_object_id"], + ) + + +@pytest.mark.skipif( + not os.getenv("EXCEL_AGENT_TEST_POSTGRES_DSN", "").strip(), + reason="EXCEL_AGENT_TEST_POSTGRES_DSN is required for live disposable parity validation", +) +def test_live_postgres_reader_and_managed_materializer_parity(tmp_path: Path) -> None: + dsn = os.environ["EXCEL_AGENT_TEST_POSTGRES_DSN"] + nonce = uuid.uuid4().hex + owner = f"v05-materializer-owner-{nonce}" + request, payloads = _neutral_bundle( + [ + (f"blobs/sha256/{nonce}", "blob", f"portable-{nonce}".encode()), + ( + f"records/sessions/{nonce}.json", + "session", + canonical_json({"kind": "session", "nonce": nonce}), + ), + ], + owner_id=owner, + ) + source = PostgresSessionStore(dsn) + managed: EncryptedManagedObjectStore | None = None + try: + assert source.consume_v05_import(request, payloads)["state"] == "imported" + managed = EncryptedManagedObjectStore( + ManagedObjectStoreConfig( + enabled=True, + root=tmp_path / "managed-pg", + current_key_id="managed-pg-current", + current_key=b"p" * 32, + previous_keys={}, + address_key=b"q" * 32, + max_object_bytes=1024 * 1024, + total_bytes=8 * 1024 * 1024, + chunk_bytes=4096, + ), + PostgresManagedObjectMetadataRepository(dsn), + ) + coordinator = LifecycleCoordinator( + store=source, + receipts_dir=tmp_path / "pg-lifecycle-receipts", + ) + result = materialize_v05_import( + source_store=source, + managed_store=managed, + import_id=request["importId"], + expected_binding_sha256=request["bindingSha256"], + tenant_id=f"tenant-{nonce}", + owner_id=owner, + lifecycle_coordinator=coordinator, + ) + scope = ManagedObjectScope(f"tenant-{nonce}", owner, request["targetSessionId"]) + manifest = managed.get_bytes(scope, result["manifest_object_id"]) + assert sha256_bytes(manifest) == result["manifest_sha256"] + assert json.loads(manifest)["objectCount"] == len(payloads) + finally: + async def cleanup() -> None: + pool = await source._get_pool() + async with pool.acquire() as conn: + async with conn.transaction(): + await conn.execute( + "DELETE FROM managed_encrypted_objects WHERE session_id = $1", + request["targetSessionId"], + ) + await conn.execute( + "DELETE FROM migration_v05_payloads WHERE import_id = $1", + request["importId"], + ) + await conn.execute( + "DELETE FROM migration_v05_imports WHERE import_id = $1", + request["importId"], + ) + await conn.execute( + "DELETE FROM sessions WHERE session_id = $1", + request["targetSessionId"], + ) + await conn.execute( + "DELETE FROM session_lifecycle WHERE session_id = $1", + request["targetSessionId"], + ) + + try: + source._run(cleanup()) + finally: + asyncio.run(source.close()) diff --git a/backend/tests/test_v05_migration_bundle.py b/backend/tests/test_v05_migration_bundle.py new file mode 100644 index 0000000..5654943 --- /dev/null +++ b/backend/tests/test_v05_migration_bundle.py @@ -0,0 +1,514 @@ +from __future__ import annotations + +import hashlib +import json +import os +from pathlib import Path +from types import SimpleNamespace +import zipfile + +import pytest + +from backend.migration.v05_bundle import ( + ExportLimits, + MigrationExportError, + export_v05_bundle, +) +from vnext.migration.v05_import import ( + ImportLimits, + MigrationImportError, + validate_and_stage_bundle, +) +from vnext.migration.portable_secret_policy import redact_secret_text, secret_paths + + +# These runtime strings deliberately exercise migration secret scrubbing. Keep +# their source fragments split so repository secret scanners and push protection +# do not mistake synthetic controls for live credentials. +SYNTHETIC_OPENAI_TOKEN = "sk" + "-proj-" + "abcdefghijklmnopqrstuvwxyz" +SYNTHETIC_GITHUB_TOKEN = "gh" + "p_" + "ABCDEFGHIJKLMNOPQRSTUVWXYZ1234567890" +SYNTHETIC_GOOGLE_TOKEN = "ya" + "29.a0AfH6SMExampleLongCredentialValue" +SYNTHETIC_WEBHOOK_SECRET = "wh" + "sec_" + "abcdefghijklmnopqrstuvwxyz0123456789" +SYNTHETIC_DATABASE_URL = "postgresql://finance_user:" + "S3cretValue" + "@db.internal/model" + + +class _Store: + def __init__(self, *, owner: str = "alice", document_path: Path) -> None: + self.owner = owner + self.document_path = document_path + self.document = { + "document_id": "doc-1", + "session_id": "session-1", + "filename": "source.xlsx", + "content_type": "application/vnd.openxmlformats-officedocument.spreadsheetml.sheet", + "size_bytes": document_path.stat().st_size, + "file_path": str(document_path), + "summary": {"legacy": "must not cross the clean-break boundary"}, + "parsed": {"v7Model": {"must": "not be exported"}}, + } + + def session_exists(self, session_id: str) -> bool: + return session_id == "session-1" + + def get_session_owner(self, session_id: str) -> str | None: + return self.owner if self.session_exists(session_id) else None + + def list_documents(self, session_id: str, limit: int = 20) -> list[dict]: + assert limit > 1 + return [dict(self.document)] if session_id == "session-1" else [] + + def get_document(self, document_id: str) -> dict | None: + return dict(self.document) if document_id == "doc-1" else None + + +class _WorkflowEngine: + def __init__(self, *, owner: str = "alice") -> None: + self.workflow = SimpleNamespace( + id="workflow-1", + name="Refresh model", + description="Refresh a reviewed workbook", + steps=[ + SimpleNamespace( + step_type="prompt", + description="Prepare a draft", + config={ + "prompt": f"Use {SYNTHETIC_OPENAI_TOKEN} for this run", + "api_key": "top-secret-value", + "temperature": 0, + }, + ) + ], + variables={"ticker": "Public symbol", "password": "do-not-export"}, + tags=["finance"], + created_at="2026-01-01T00:00:00Z", + updated_at="2026-01-02T00:00:00Z", + execution_count=9, + last_executed="2026-01-03T00:00:00Z", + owner_user_id=owner, + session_id="session-1", + built_in=False, + ) + + def list_workflows(self) -> list[dict]: + return [{"id": self.workflow.id}] + + def get_workflow(self, workflow_id: str): + return self.workflow if workflow_id == self.workflow.id else None + + +def _fixture(tmp_path: Path, *, owner: str = "alice"): + managed_root = tmp_path / "managed-documents" + managed_root.mkdir(parents=True) + document_path = managed_root / "doc-1_source.xlsx" + document_path.write_bytes(b"PK\x03\x04portable-workbook") + return _Store(owner=owner, document_path=document_path), _WorkflowEngine(owner=owner), managed_root + + +def _export(tmp_path: Path, *, owner: str = "alice"): + store, workflows, managed_root = _fixture(tmp_path, owner=owner) + return export_v05_bundle( + store=store, + workflow_engine=workflows, + owner_id=owner, + session_ids=["session-1"], + managed_documents_root=managed_root, + output_path=tmp_path / "migration.zip", + ) + + +def _rewrite_zip(source: Path, target: Path, transform) -> str: + with zipfile.ZipFile(source, "r") as archive: + entries = [(item.filename, archive.read(item)) for item in archive.infolist()] + transformed = transform(entries) + with zipfile.ZipFile(target, "w", compression=zipfile.ZIP_STORED) as archive: + for name, payload in transformed: + info = zipfile.ZipInfo(name, date_time=(1980, 1, 1, 0, 0, 0)) + info.external_attr = 0o100600 << 16 + archive.writestr(info, payload) + return hashlib.sha256(target.read_bytes()).hexdigest() + + +def _canonical_json(value) -> bytes: + return json.dumps(value, sort_keys=True, separators=(",", ":"), ensure_ascii=False).encode("utf-8") + b"\n" + + +def _rewrite_valid_workflow(source: Path, target: Path, transform) -> str: + with zipfile.ZipFile(source, "r") as archive: + entries = {item.filename: archive.read(item) for item in archive.infolist()} + manifest = json.loads(entries["manifest.json"]) + descriptor = next(item for item in manifest["payloads"] if item["kind"] == "workflow_recipe") + record = json.loads(entries[descriptor["path"]]) + content = _canonical_json(transform(record)) + entries[descriptor["path"]] = content + descriptor["sha256"] = hashlib.sha256(content).hexdigest() + descriptor["size_bytes"] = len(content) + manifest["content_sha256"] = hashlib.sha256(_canonical_json(manifest["payloads"])).hexdigest() + entries["manifest.json"] = _canonical_json(manifest) + with zipfile.ZipFile(target, "w", compression=zipfile.ZIP_STORED) as archive: + for name, payload in entries.items(): + info = zipfile.ZipInfo(name, date_time=(1980, 1, 1, 0, 0, 0)) + info.external_attr = 0o100600 << 16 + archive.writestr(info, payload) + return hashlib.sha256(target.read_bytes()).hexdigest() + + +def test_export_is_deterministic_content_addressed_and_excludes_legacy_models_and_secrets(tmp_path: Path) -> None: + store, workflows, managed_root = _fixture(tmp_path) + first = export_v05_bundle( + store=store, + workflow_engine=workflows, + owner_id="alice", + session_ids=["session-1"], + managed_documents_root=managed_root, + output_path=tmp_path / "first.zip", + ) + second = export_v05_bundle( + store=store, + workflow_engine=workflows, + owner_id="alice", + session_ids=["session-1"], + managed_documents_root=managed_root, + output_path=tmp_path / "second.zip", + ) + + assert first.archive_sha256 == second.archive_sha256 + assert first.content_sha256 == second.content_sha256 + assert first.archive_path.read_bytes() == second.archive_path.read_bytes() + assert first.sha256_path.read_text(encoding="ascii") == f"{first.archive_sha256} first.zip\n" + + raw = first.archive_path.read_bytes() + assert b"top-secret-value" not in raw + assert SYNTHETIC_OPENAI_TOKEN.encode() not in raw + assert b"do-not-export" not in raw + assert b"v7Model" not in raw + assert b"must not cross the clean-break boundary" not in raw + + with zipfile.ZipFile(first.archive_path) as archive: + manifest = json.loads(archive.read("manifest.json")) + names = archive.namelist() + assert manifest["schema_version"] == "com.excelagent.migration.v05.v1" + assert manifest["migration_mode"] == "one_way_clean_break" + assert manifest["counts"] == {"documents": 1, "sessions": 1, "workflows": 1} + assert any(name.startswith("blobs/sha256/") for name in names) + assert all("session-1" not in name and "doc-1" not in name for name in names) + for payload in manifest["payloads"]: + content = archive.read(payload["path"]) + assert len(content) == payload["size_bytes"] + assert hashlib.sha256(content).hexdigest() == payload["sha256"] + + +def test_export_drops_unknown_parameters_and_alternate_credential_shapes(tmp_path: Path) -> None: + store, workflows, managed_root = _fixture(tmp_path) + # Build this synthetic control at runtime so repository push protection does not + # mistake a deliberately fake Slack credential for a live secret. + slack_token = "-".join(("xoxb", "123456789012", "123456789012", "abcdefghijklmnopqrstuvwx")) + workflows.workflow.steps[0].config = { + "prompt": "Use Bearer KNOWNBEARERTOKEN1234567890 only as a synthetic control", + "integrations": { + "github": {"access_token": SYNTHETIC_GITHUB_TOKEN}, + "slack": {"token": slack_token}, + "google": {"oauth": SYNTHETIC_GOOGLE_TOKEN}, + "database": {"url": SYNTHETIC_DATABASE_URL}, + "webhook": {"signing_key": SYNTHETIC_WEBHOOK_SECRET}, + }, + "opaque_value": "MIGRATION_DEMO_OPAQUE_TOKEN_abcdefghijklmnopqrstuvwxyz", + "callback": "https://example.invalid/hook?signature=abcdefghijklmnopqrstuvwxyz0123456789", + "temperature": 0, + } + + exported = export_v05_bundle( + store=store, + workflow_engine=workflows, + owner_id="alice", + session_ids=["session-1"], + managed_documents_root=managed_root, + output_path=tmp_path / "alternate-secrets.zip", + ) + + raw = exported.archive_path.read_bytes() + for secret in ( + SYNTHETIC_GITHUB_TOKEN.encode(), + slack_token.encode(), + SYNTHETIC_GOOGLE_TOKEN.encode(), + SYNTHETIC_DATABASE_URL.encode(), + SYNTHETIC_WEBHOOK_SECRET.encode(), + b"MIGRATION_DEMO_OPAQUE_TOKEN_abcdefghijklmnopqrstuvwxyz", + b"abcdefghijklmnopqrstuvwxyz0123456789", + b"KNOWNBEARERTOKEN1234567890", + ): + assert secret not in raw + with zipfile.ZipFile(exported.archive_path) as archive: + manifest = json.loads(archive.read("manifest.json")) + descriptor = next(item for item in manifest["payloads"] if item["kind"] == "workflow_recipe") + workflow = json.loads(archive.read(descriptor["path"])) + assert workflow["steps"][0]["parameters"] == {"prompt": "Use [REDACTED_SECRET] only as a synthetic control"} + assert workflow["variables"] == {"ticker": "Public symbol"} + assert "/steps/0/parameters/integrations" in workflow["redactions"] + assert "/steps/0/parameters/opaque_value" in workflow["redactions"] + + +def test_portable_secret_policy_redacts_signed_urls_and_lowercase_opaque_tokens() -> None: + signed_url, url_changed = redact_secret_text("https://example.invalid/hook?signature=abcdefghijklmnopqrstuvwxyz0123456789") + opaque, opaque_changed = redact_secret_text("abcdefghijklmnopqrstuvwxzyabcdef") + + assert url_changed is True + assert "abcdefghijklmnopqrstuvwxyz0123456789" not in signed_url + assert opaque_changed is True + assert opaque == "[REDACTED_SECRET]" + + +@pytest.mark.parametrize( + "credential_text", + [ + "credential=hunter2", + "cookie: sessionid=abc123", + "dsn=finance-prod", + "private key: short-secret", + "session token: short-secret", + "authorization: abc123", + ], +) +def test_portable_secret_policy_redacts_short_labeled_credentials(credential_text: str) -> None: + sanitized, changed = redact_secret_text(credential_text) + + assert changed is True + assert sanitized == "[REDACTED_SECRET]" + assert secret_paths({"prompt": credential_text}) == ["/prompt"] + + +def test_export_rejects_foreign_sessions_unmanaged_paths_and_oversize_blobs(tmp_path: Path) -> None: + store, workflows, managed_root = _fixture(tmp_path, owner="bob") + with pytest.raises(MigrationExportError, match="owned by the requested owner"): + export_v05_bundle( + store=store, + workflow_engine=workflows, + owner_id="alice", + session_ids=["session-1"], + managed_documents_root=managed_root, + output_path=tmp_path / "foreign.zip", + ) + + store, workflows, managed_root = _fixture(tmp_path / "unmanaged") + outside = tmp_path / "outside.xlsx" + outside.write_bytes(b"outside") + store.document_path = outside + store.document["file_path"] = str(outside) + store.document["size_bytes"] = outside.stat().st_size + with pytest.raises(MigrationExportError, match="managed document root"): + export_v05_bundle( + store=store, + workflow_engine=workflows, + owner_id="alice", + session_ids=["session-1"], + managed_documents_root=managed_root, + output_path=tmp_path / "unmanaged.zip", + ) + + store, workflows, managed_root = _fixture(tmp_path / "oversize") + with pytest.raises(MigrationExportError, match="blob size limit"): + export_v05_bundle( + store=store, + workflow_engine=workflows, + owner_id="alice", + session_ids=["session-1"], + managed_documents_root=managed_root, + output_path=tmp_path / "oversize.zip", + limits=ExportLimits(max_blob_bytes=4), + ) + + +def test_import_validates_and_stages_neutral_records_without_live_storage(tmp_path: Path) -> None: + exported = _export(tmp_path) + result = validate_and_stage_bundle( + bundle_path=exported.archive_path, + expected_archive_sha256=exported.archive_sha256, + target_owner_id="alice", + inbox_root=tmp_path / "vnext-inbox", + ) + + assert result.content_sha256 == exported.content_sha256 + assert result.status == "validated_pending_import" + assert result.staging_path.name == exported.content_sha256 + receipt = json.loads((result.staging_path / "receipt.json").read_text(encoding="utf-8")) + assert receipt["status"] == "validated_pending_import" + assert receipt["runtime_storage_mutated"] is False + assert receipt["compatibility_layer_installed"] is False + assert not list(tmp_path.rglob("*.db")) + + # Idempotent restaging verifies the existing receipt instead of overwriting it. + repeated = validate_and_stage_bundle( + bundle_path=exported.archive_path, + expected_archive_sha256=exported.archive_sha256, + target_owner_id="alice", + inbox_root=tmp_path / "vnext-inbox", + ) + assert repeated == result + + +def test_import_uses_one_open_file_when_selected_path_is_replaced(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None: + exported = _export(tmp_path) + selected = tmp_path / "selected.zip" + selected.write_bytes(exported.archive_path.read_bytes()) + replacement = tmp_path / "replacement.zip" + + def rename_workflow(record): + record["name"] = "Replacement workflow" + return record + + _rewrite_valid_workflow(exported.archive_path, replacement, rename_workflow) + original_zip_file = zipfile.ZipFile + swapped = False + opened_with_descriptor = False + + def swap_path_then_open(file, mode="r", *args, **kwargs): + nonlocal opened_with_descriptor, swapped + if not swapped: + opened_with_descriptor = hasattr(file, "fileno") + os.replace(replacement, selected) + swapped = True + return original_zip_file(file, mode, *args, **kwargs) + + monkeypatch.setattr("vnext.migration.v05_import.zipfile.ZipFile", swap_path_then_open) + result = validate_and_stage_bundle( + bundle_path=selected, + expected_archive_sha256=exported.archive_sha256, + target_owner_id="alice", + inbox_root=tmp_path / "vnext-inbox", + ) + + assert swapped is True + assert opened_with_descriptor is True + assert hashlib.sha256(selected.read_bytes()).hexdigest() != exported.archive_sha256 + manifest = json.loads((result.staging_path / "manifest.json").read_text(encoding="utf-8")) + descriptor = next(item for item in manifest["payloads"] if item["kind"] == "workflow_recipe") + workflow = json.loads((result.staging_path / descriptor["path"]).read_text(encoding="utf-8")) + assert result.archive_sha256 == exported.archive_sha256 + assert workflow["name"] == "Refresh model" + + +def test_import_rejects_archive_hash_and_payload_tampering(tmp_path: Path) -> None: + exported = _export(tmp_path) + tampered = tmp_path / "tampered.zip" + tampered.write_bytes(exported.archive_path.read_bytes() + b"tamper") + with pytest.raises(MigrationImportError, match="archive SHA-256"): + validate_and_stage_bundle( + bundle_path=tampered, + expected_archive_sha256=exported.archive_sha256, + target_owner_id="alice", + inbox_root=tmp_path / "inbox-archive-tamper", + ) + + def alter_record(entries): + result = [] + changed = False + for name, payload in entries: + if not changed and name.startswith("records/"): + payload = bytes([payload[0] ^ 1]) + payload[1:] + changed = True + result.append((name, payload)) + return result + + payload_tamper = tmp_path / "payload-tamper.zip" + recalculated_archive_hash = _rewrite_zip(exported.archive_path, payload_tamper, alter_record) + with pytest.raises(MigrationImportError, match="Payload SHA-256"): + validate_and_stage_bundle( + bundle_path=payload_tamper, + expected_archive_sha256=recalculated_archive_hash, + target_owner_id="alice", + inbox_root=tmp_path / "inbox-payload-tamper", + ) + + +def test_import_semantically_rejects_residual_credentials_in_a_rehashed_bundle(tmp_path: Path) -> None: + exported = _export(tmp_path) + + def inject_secret(record): + record["steps"][0]["parameters"]["prompt"] = f"Use {SYNTHETIC_GITHUB_TOKEN} for this run" + return record + + tampered = tmp_path / "credential-tamper.zip" + recalculated_archive_hash = _rewrite_valid_workflow( + exported.archive_path, + tampered, + inject_secret, + ) + + with pytest.raises(MigrationImportError, match="residual credential material"): + validate_and_stage_bundle( + bundle_path=tampered, + expected_archive_sha256=recalculated_archive_hash, + target_owner_id="alice", + inbox_root=tmp_path / "inbox-credential-tamper", + ) + + +def test_import_rejects_traversal_duplicate_entries_unsupported_versions_and_wrong_owner(tmp_path: Path) -> None: + exported = _export(tmp_path) + + traversal = tmp_path / "traversal.zip" + traversal_hash = _rewrite_zip( + exported.archive_path, + traversal, + lambda entries: [*entries, ("../escape", b"bad")], + ) + with pytest.raises(MigrationImportError, match="canonical relative POSIX path"): + validate_and_stage_bundle( + bundle_path=traversal, + expected_archive_sha256=traversal_hash, + target_owner_id="alice", + inbox_root=tmp_path / "inbox-traversal", + ) + + duplicate = tmp_path / "duplicate.zip" + duplicate_hash = _rewrite_zip( + exported.archive_path, + duplicate, + lambda entries: [*entries, next(entry for entry in entries if entry[0] == "manifest.json")], + ) + with pytest.raises(MigrationImportError, match="duplicate archive entry"): + validate_and_stage_bundle( + bundle_path=duplicate, + expected_archive_sha256=duplicate_hash, + target_owner_id="alice", + inbox_root=tmp_path / "inbox-duplicate", + ) + + def unsupported_version(entries): + result = [] + for name, payload in entries: + if name == "manifest.json": + manifest = json.loads(payload) + manifest["schema_version"] = "com.excelagent.migration.v05.v999" + payload = json.dumps(manifest, sort_keys=True, separators=(",", ":"), ensure_ascii=False).encode("utf-8") + b"\n" + result.append((name, payload)) + return result + + unsupported = tmp_path / "unsupported.zip" + unsupported_hash = _rewrite_zip(exported.archive_path, unsupported, unsupported_version) + with pytest.raises(MigrationImportError, match="Unsupported migration schema"): + validate_and_stage_bundle( + bundle_path=unsupported, + expected_archive_sha256=unsupported_hash, + target_owner_id="alice", + inbox_root=tmp_path / "inbox-version", + ) + + with pytest.raises(MigrationImportError, match="owner scope"): + validate_and_stage_bundle( + bundle_path=exported.archive_path, + expected_archive_sha256=exported.archive_sha256, + target_owner_id="mallory", + inbox_root=tmp_path / "inbox-owner", + ) + + +def test_import_rejects_oversize_archives_before_extraction(tmp_path: Path) -> None: + exported = _export(tmp_path) + with pytest.raises(MigrationImportError, match="uncompressed size limit"): + validate_and_stage_bundle( + bundle_path=exported.archive_path, + expected_archive_sha256=exported.archive_sha256, + target_owner_id="alice", + inbox_root=tmp_path / "inbox", + limits=ImportLimits(max_total_uncompressed_bytes=32), + ) diff --git a/backend/tests/test_work_admission.py b/backend/tests/test_work_admission.py new file mode 100644 index 0000000..30c82e0 --- /dev/null +++ b/backend/tests/test_work_admission.py @@ -0,0 +1,567 @@ +from __future__ import annotations + +import asyncio +import json +import subprocess +import sys +import threading +import time +from types import SimpleNamespace + +import pytest + +from backend.operations.work_admission import ( + WorkAdmissionCancelled, + WorkAdmissionController, + WorkAdmissionDraining, + WorkExecutionTimeout, + WorkQueueFull, + WorkQueueTimeout, + work_context, +) + + +def _controller(*, concurrency: int = 1, queue: int = 4, timeout: float = 1.0) -> WorkAdmissionController: + return WorkAdmissionController( + max_concurrency=concurrency, + max_queue=queue, + max_queue_per_session=queue, + queue_timeout_seconds=timeout, + pool_concurrency={"workbook": concurrency, "pdf": concurrency}, + pool_queue={"workbook": queue, "pdf": queue}, + ) + + +def _wait_for_queue(controller: WorkAdmissionController, expected: int) -> None: + deadline = time.monotonic() + 2 + while time.monotonic() < deadline: + if controller.snapshot().queued == expected: + return + time.sleep(0.005) + raise AssertionError(f"queue never reached {expected}: {controller.redacted_status()}") + + +def test_queue_budget_rejects_saturation_without_starting_work() -> None: + controller = _controller(queue=1) + queued_started = threading.Event() + + def queued_work() -> None: + with controller.admit("workbook", session_id="session-a"): + queued_started.set() + + with controller.admit("workbook", session_id="holder"): + thread = threading.Thread(target=queued_work) + thread.start() + _wait_for_queue(controller, 1) + with pytest.raises(WorkQueueFull) as exc_info: + controller.admit("workbook", session_id="session-b") + assert exc_info.value.code == "WORK_QUEUE_FULL" + assert queued_started.is_set() is False + + thread.join(timeout=2) + assert queued_started.is_set() + + +def test_queue_timeout_is_distinct_from_execution_timeout() -> None: + controller = _controller(timeout=0.02) + with controller.admit("workbook", session_id="holder"): + with pytest.raises(WorkQueueTimeout) as exc_info: + controller.admit("workbook", session_id="waiting") + assert exc_info.value.code == "WORK_QUEUE_TIMEOUT" + assert not isinstance(exc_info.value, WorkExecutionTimeout) + + +def test_session_round_robin_prevents_one_session_from_monopolizing_queue() -> None: + controller = _controller(queue=4) + order: list[str] = [] + + def work(name: str, session_id: str) -> None: + with controller.admit("workbook", session_id=session_id): + order.append(name) + + with controller.admit("workbook", session_id="holder"): + threads = [ + threading.Thread(target=work, args=("a1", "session-a")), + threading.Thread(target=work, args=("a2", "session-a")), + threading.Thread(target=work, args=("b1", "session-b")), + ] + for index, thread in enumerate(threads, start=1): + thread.start() + _wait_for_queue(controller, index) + + for thread in threads: + thread.join(timeout=2) + assert not thread.is_alive() + assert order == ["a1", "b1", "a2"] + + +def test_per_session_queue_budget_reserves_admission_for_other_sessions() -> None: + controller = WorkAdmissionController( + max_concurrency=1, + max_queue=3, + max_queue_per_session=1, + queue_timeout_seconds=1, + pool_concurrency={"workbook": 1, "pdf": 1}, + pool_queue={"workbook": 3, "pdf": 3}, + ) + releases = {"a": threading.Event(), "b": threading.Event()} + + def work(name: str, session_id: str) -> None: + with controller.admit("workbook", session_id=session_id): + releases[name].wait(timeout=2) + + with controller.admit("workbook", session_id="holder"): + session_a = threading.Thread(target=work, args=("a", "session-a")) + session_a.start() + _wait_for_queue(controller, 1) + with pytest.raises(WorkQueueFull): + controller.admit("workbook", session_id="session-a") + session_b = threading.Thread(target=work, args=("b", "session-b")) + session_b.start() + _wait_for_queue(controller, 2) + + releases["a"].set() + releases["b"].set() + session_a.join(timeout=2) + session_b.join(timeout=2) + assert not session_a.is_alive() and not session_b.is_alive() + + +def test_queued_cancellation_wakes_promptly_and_never_acquires() -> None: + controller = _controller() + cancelled = threading.Event() + result: list[BaseException] = [] + + def waiting_work() -> None: + try: + controller.admit("pdf", session_id="session-a", cancellation_event=cancelled) + except BaseException as exc: + result.append(exc) + + with controller.admit("pdf", session_id="holder"): + thread = threading.Thread(target=waiting_work) + thread.start() + _wait_for_queue(controller, 1) + cancelled.set() + controller.wake() + thread.join(timeout=1) + assert not thread.is_alive() + assert controller.snapshot().active == 1 + + assert len(result) == 1 + assert isinstance(result[0], WorkAdmissionCancelled) + + +def test_shutdown_rejects_queue_then_propagates_cancel_to_running_lease() -> None: + controller = _controller() + running = threading.Event() + observed_cancel = threading.Event() + + def active_work() -> None: + with controller.admit("workbook", session_id="session-active") as lease: + running.set() + if lease.cancellation.wait(timeout=2): + observed_cancel.set() + + thread = threading.Thread(target=active_work) + thread.start() + assert running.wait(timeout=1) + + assert controller.begin_drain() is True + with pytest.raises(WorkAdmissionDraining) as exc_info: + controller.admit("pdf", session_id="session-new") + assert exc_info.value.code == "WORK_ADMISSION_DRAINING" + controller.cancel_running() + + thread.join(timeout=2) + assert not thread.is_alive() + assert observed_cancel.is_set() + assert controller.wait_for_idle_blocking(0.2) + + +def test_operator_status_is_bounded_and_never_contains_session_identifiers() -> None: + controller = _controller(queue=3) + with controller.admit("managed_xlsx", session_id="private-session-token"): + encoded = json.dumps(controller.redacted_status(), sort_keys=True) + + assert "private-session-token" not in encoded + assert set(controller.redacted_status()) == { + "status", + "draining", + "active", + "queued", + "budgets", + "active_by_kind", + "queued_by_kind", + } + + +def test_job_cancellation_reaches_the_worker_context(tmp_path, monkeypatch: pytest.MonkeyPatch) -> None: + from backend.services.workbook_jobs import WorkbookJobManager + + worker_started = threading.Event() + worker_cancelled = threading.Event() + + def blocking_worker(_command: str, _payload: dict): + from backend.operations.work_admission import current_work_context + + context = current_work_context() + worker_started.set() + assert context.cancellation_event is not None + assert context.cancellation_event.wait(timeout=2) # type: ignore[attr-defined] + worker_cancelled.set() + raise WorkAdmissionCancelled("workbook", "request") + + monkeypatch.setattr("backend.excel.workbook_worker.run_worker", blocking_worker) + manager = WorkbookJobManager(tmp_path) + job = manager.submit("diff", {"before": {}, "after": {}}, session_id="session-cancel") + assert worker_started.wait(timeout=1) + + cancelled = manager.cancel(job.run_id) + assert cancelled is not None and cancelled.status == "cancelled" + assert worker_cancelled.wait(timeout=1) + + +def test_admission_failure_never_invokes_in_process_workbook_fallback(monkeypatch: pytest.MonkeyPatch) -> None: + from backend.services import workbook_jobs + + monkeypatch.setattr( + "backend.excel.workbook_worker.run_worker", + lambda *_args, **_kwargs: (_ for _ in ()).throw(WorkQueueTimeout("workbook", 0.1)), + ) + monkeypatch.setattr( + workbook_jobs, + "_execute_python_command", + lambda *_args, **_kwargs: pytest.fail("admission failure must remain fail-closed"), + ) + + with pytest.raises(WorkQueueTimeout): + workbook_jobs._execute_command("diff", {"before": {}, "after": {}}) + + +def test_worker_rejection_never_invokes_in_process_job_fallback(monkeypatch: pytest.MonkeyPatch) -> None: + from backend.excel.workbook_worker import WorkerResult + from backend.services import workbook_jobs + + monkeypatch.setattr( + "backend.excel.workbook_worker.run_worker", + lambda *_args, **_kwargs: WorkerResult(ok=False, error="bounded worker rejected payload"), + ) + monkeypatch.setattr( + workbook_jobs, + "_execute_python_command", + lambda *_args, **_kwargs: pytest.fail("worker rejection must remain fail-closed"), + ) + + with pytest.raises(RuntimeError, match="bounded worker rejected payload"): + workbook_jobs._execute_command("ingest", {}) + + +def test_workbook_process_cancellation_terminates_before_success(tmp_path) -> None: + from backend.excel import workbook_worker + + cancellation = threading.Event() + started = time.monotonic() + + def cancel_soon() -> None: + time.sleep(0.1) + cancellation.set() + + thread = threading.Thread(target=cancel_soon) + thread.start() + with pytest.raises(WorkAdmissionCancelled): + workbook_worker._run_bounded_process( + [sys.executable, "-c", "import time; time.sleep(30)"], + input_text="{}", + cwd=str(tmp_path), + env={}, + timeout_seconds=10, + max_output_bytes=1024, + cancellation_event=cancellation, + ) + thread.join(timeout=1) + assert time.monotonic() - started < 2 + + +def test_managed_workbook_gateway_rejects_before_spawning_when_pool_is_saturated( + monkeypatch: pytest.MonkeyPatch, +) -> None: + from backend.excel import workbook_worker + + controller = WorkAdmissionController( + max_concurrency=1, + max_queue=0, + queue_timeout_seconds=1, + pool_concurrency={"workbook": 1, "pdf": 1}, + pool_queue={"workbook": 0, "pdf": 0}, + ) + monkeypatch.setattr(workbook_worker, "work_admission_controller", controller) + monkeypatch.setattr(workbook_worker, "worker_enabled", lambda: True) + monkeypatch.setattr(workbook_worker, "_validate_payload", lambda *_args: None) + monkeypatch.setattr( + workbook_worker, + "_invoke_worker_process", + lambda *_args, **_kwargs: pytest.fail("saturated work must never spawn"), + ) + + with controller.admit("managed_xlsx", session_id="holder"): + with work_context(session_id="waiting", kind="managed_xlsx"): + with pytest.raises(WorkQueueFull) as exc_info: + workbook_worker.run_worker("ingest", {}) + + assert exc_info.value.kind == "managed_xlsx" + + +def test_pdf_execution_timeout_uses_shared_execution_taxonomy() -> None: + from backend.documents.pdf_worker import PDFWorkerTimeout + + assert isinstance(PDFWorkerTimeout("slow"), WorkExecutionTimeout) + + +def test_context_overrides_are_nested_and_restore_prior_values() -> None: + from backend.operations.work_admission import current_work_context + + assert current_work_context().session_id == "" + with work_context(session_id="outer", kind="managed_xlsx"): + assert current_work_context().session_id == "outer" + assert current_work_context().kind == "managed_xlsx" + with work_context(session_id="inner"): + assert current_work_context().session_id == "inner" + assert current_work_context().kind == "managed_xlsx" + assert current_work_context().session_id == "outer" + assert current_work_context().session_id == "" + + +def test_workbook_timeout_remains_subprocess_compatible() -> None: + from backend.excel.workbook_worker import WorkbookWorkerTimeout + + timeout = WorkbookWorkerTimeout(["node"], 1) + assert isinstance(timeout, subprocess.TimeoutExpired) + assert isinstance(timeout, WorkExecutionTimeout) + + +def test_operator_report_includes_only_redacted_work_admission_state(monkeypatch: pytest.MonkeyPatch) -> None: + from backend.operations import operator_cli + + controller = _controller() + config = SimpleNamespace( + DB_ENGINE="postgres", + CONFIG_PROFILE="production", + VERSION="test", + HOST="127.0.0.1", + PORT=8000, + PRODUCTION_MODE=True, + AUTH_ENABLED=True, + RBAC_ENABLED=True, + OIDC_ENABLED=False, + STATIC_KEY_ADMIN_ENABLED=False, + LOCAL_PAIRING_ENABLED=False, + runtime_config_errors=lambda: [], + ) + monkeypatch.setattr(operator_cli, "work_admission_controller", controller) + + with work_context(session_id="operator-secret"): + report = operator_cli.build_operator_report(config_module=config) + + encoded = json.dumps(report, sort_keys=True) + assert "operator-secret" not in encoded + assert report["work_admission"]["budgets"]["max_concurrency"] == 1 + + +@pytest.mark.asyncio +async def test_async_caller_cancellation_is_forwarded_into_worker_thread() -> None: + from backend.operations.work_admission import current_work_context, run_cancellable_in_thread + + started = threading.Event() + stopped = threading.Event() + + def blocking_work() -> None: + cancellation = current_work_context().cancellation_event + assert cancellation is not None + started.set() + assert cancellation.wait(timeout=2) # type: ignore[attr-defined] + stopped.set() + + task = asyncio.create_task(run_cancellable_in_thread(blocking_work, session_id="cancelled-session")) + assert await asyncio.to_thread(started.wait, 1) + task.cancel() + with pytest.raises(asyncio.CancelledError): + await task + assert stopped.wait(timeout=1) + + +@pytest.mark.asyncio +async def test_async_timeout_cancels_worker_instead_of_leaving_orphaned_work() -> None: + from backend.operations.work_admission import current_work_context, run_cancellable_in_thread + + stopped = threading.Event() + + def blocking_work() -> None: + cancellation = current_work_context().cancellation_event + assert cancellation is not None + assert cancellation.wait(timeout=2) # type: ignore[attr-defined] + stopped.set() + + with pytest.raises(TimeoutError): + await run_cancellable_in_thread(blocking_work, timeout_seconds=0.02) + assert stopped.wait(timeout=1) + + +def test_managed_xlsx_and_workbook_share_one_subprocess_pool() -> None: + controller = WorkAdmissionController( + max_concurrency=2, + max_queue=2, + queue_timeout_seconds=1, + pool_concurrency={"workbook": 1, "pdf": 1}, + pool_queue={"workbook": 2, "pdf": 2}, + ) + workbook_started = threading.Event() + + def workbook_work() -> None: + with controller.admit("workbook", session_id="workbook-session"): + workbook_started.set() + + with controller.admit("managed_xlsx", session_id="managed-session"): + with controller.admit("pdf", session_id="pdf-session"): + thread = threading.Thread(target=workbook_work) + thread.start() + _wait_for_queue(controller, 1) + assert workbook_started.is_set() is False + assert workbook_started.is_set() is False + + thread.join(timeout=2) + assert workbook_started.is_set() + + +def test_pdf_queue_saturation_uses_shared_queue_timeout(monkeypatch: pytest.MonkeyPatch) -> None: + from backend.documents import pdf_worker + + controller = _controller(timeout=0.02) + monkeypatch.setattr(pdf_worker, "work_admission_controller", controller) + monkeypatch.setattr(pdf_worker, "PDF_PARSE_QUEUE_TIMEOUT_SECONDS", 0.02) + + with controller.admit("pdf", session_id="holder"): + with pytest.raises(WorkQueueTimeout): + pdf_worker.parse_pdf_sandboxed(b"%PDF", "queued.pdf") + + +def test_pdf_cancellation_terminates_isolated_process(monkeypatch: pytest.MonkeyPatch) -> None: + from backend.documents import pdf_worker + + class Gate: + def set(self) -> None: + pass + + class Process: + sentinel = 42 + exitcode = None + + def __init__(self, **_kwargs) -> None: + self.alive = True + self.terminated = False + + def start(self) -> None: + pass + + def is_alive(self) -> bool: + return self.alive + + def join(self, timeout=None) -> None: + pass + + def terminate(self) -> None: + self.terminated = True + self.alive = False + + def kill(self) -> None: + self.alive = False + + process = Process() + context = SimpleNamespace(Event=Gate, Process=lambda **_kwargs: process) + cancellation = threading.Event() + cancellation.set() + monkeypatch.setattr(pdf_worker.multiprocessing, "get_context", lambda _method: context) + monkeypatch.setattr(pdf_worker, "_is_windows", lambda: False) + + with pytest.raises(WorkAdmissionCancelled): + pdf_worker._run_isolated_parse( + b"%PDF", + "cancelled.pdf", + 30, + cancellation_event=cancellation, + ) + assert process.terminated is True + + +def test_global_error_boundary_emits_stable_capacity_code_without_identifier() -> None: + from fastapi import FastAPI + from fastapi.testclient import TestClient + + from backend.api.middleware.error_handler import ErrorHandlerMiddleware + + mini_app = FastAPI() + + @mini_app.get("/work/{private_id}") + async def fail_capacity(private_id: str) -> None: + raise WorkQueueFull("workbook") + + mini_app.add_middleware(ErrorHandlerMiddleware) + response = TestClient(mini_app).get("/work/private-session-id") + + assert response.status_code == 503 + assert response.json()["error"]["code"] == "WORK_QUEUE_FULL" + assert "private-session-id" not in json.dumps(response.json()) + + +def test_workbook_route_maps_queue_timeout_to_retryable_stable_error(monkeypatch: pytest.MonkeyPatch) -> None: + from fastapi import FastAPI + from fastapi.testclient import TestClient + + from backend.api.routes import tasks as task_routes + + monkeypatch.setattr( + "backend.excel.workbook_files.WorkbookIngestionService.summarize", + lambda *_args, **_kwargs: (_ for _ in ()).throw(WorkQueueTimeout("workbook", 0.1)), + ) + mini_app = FastAPI() + mini_app.include_router(task_routes.router) + + response = TestClient(mini_app).post( + "/api/v1/workbook/ingest", + json={"filename": "queued.xlsx", "content_base64": "eA=="}, + ) + + assert response.status_code == 503 + assert response.headers["Retry-After"] == "1" + assert response.json()["detail"]["code"] == "WORK_QUEUE_TIMEOUT" + + +def test_pdf_route_maps_queue_full_to_retryable_stable_error(monkeypatch: pytest.MonkeyPatch) -> None: + from fastapi import FastAPI + from fastapi.testclient import TestClient + + from backend.api.routes import documents as document_routes + + monkeypatch.setattr( + document_routes, + "parse_pdf_sandboxed", + lambda *_args, **_kwargs: (_ for _ in ()).throw(WorkQueueFull("pdf")), + ) + mini_app = FastAPI() + mini_app.include_router(document_routes.router) + + response = TestClient(mini_app).post( + "/api/v1/documents/parse", + files={"file": ("queued.pdf", b"%PDF-1.7\n", "application/pdf")}, + ) + + assert response.status_code == 503 + assert response.headers["Retry-After"] == "1" + assert response.json()["detail"]["code"] == "WORK_QUEUE_FULL" + + +def test_metric_label_normalizers_collapse_unrecognized_values() -> None: + from backend.observability import _bounded_work_kind, _bounded_work_outcome + + assert _bounded_work_kind("private-session-id") == "unknown" + assert _bounded_work_outcome("private-error-message") == "failed" diff --git a/backend/tests/test_workbook_benchmarks.py b/backend/tests/test_workbook_benchmarks.py index 87a0ddd..ab22dd5 100644 --- a/backend/tests/test_workbook_benchmarks.py +++ b/backend/tests/test_workbook_benchmarks.py @@ -10,6 +10,7 @@ private_workbook_fixtures_enabled, workbook_fixture_directory, ) +from backend.benchmarks.v7_golden_cases import list_v7_golden_cases from backend.config import API_KEY from backend.main import app from fastapi.testclient import TestClient @@ -27,6 +28,40 @@ def test_workbook_golden_fixtures_cover_core_v5_risks(): assert (fixture_dir / fixture.workbook_filename).exists() +def test_workbook_golden_fixture_manifest_declares_shortcut_grade_local_realism(): + fixtures = {fixture.fixture_id: fixture for fixture in WORKBOOK_GOLDEN_FIXTURES} + + formula = fixtures["formula_heavy_model"] + assert formula.expected_operation_count == 2 + assert [operation["type"] for operation in formula.operations] == ["write_cell", "write_cell"] + assert set(formula.expected_dirty_ranges) == {"Model!B2", "Model!B3"} + assert formula.expected_simulation_gates == ["checkpoint_required"] + assert formula.manifest_context["simulationAssertions"]["exactDirtyRanges"] is True + assert formula.manifest_context["realismMetadata"]["evidenceClass"] == "generated_local" + + cross_sheet = fixtures["multi_sheet_dependency"] + assert cross_sheet.expected_operation_count == 2 + assert {operation["sheet"] for operation in cross_sheet.operations} == {"Model", "Assumptions"} + assert set(cross_sheet.expected_dirty_ranges) == {"Model!B2", "Assumptions!B2"} + assert cross_sheet.expected_simulation_gates == ["checkpoint_required"] + + audit = fixtures["audit_export_flow"] + assert audit.expected_operation_count == 2 + assert any(operation["type"] == "clear_range" and operation["range"] == "B3:B4" for operation in audit.operations) + assert set(audit.expected_dirty_ranges) == {"Model!B2", "Model!B3:B4"} + assert audit.expected_simulation_gates == ["blocked"] + assert audit.manifest_context["simulationAssertions"]["blockedBy"] == "destructive_multi_cell_clear" + + +def test_v7_golden_case_list_surfaces_manifest_realism_expectations(): + cases = {case.case_id: case for case in list_v7_golden_cases()} + + assert cases["audit_export_flow"].expected_simulation_gates == ["blocked"] + assert cases["audit_export_flow"].metadata["simulationAssertions"]["blockedBy"] == "destructive_multi_cell_clear" + assert cases["chart_source_update"].metadata["operationCount"] == 3 + assert cases["formula_heavy_model"].metadata["fixtureRealism"]["scenario"] == "multi-cell formula driver update" + + def test_private_workbook_fixture_hooks_are_gitignored_and_opt_in(monkeypatch): monkeypatch.delenv("INCLUDE_PRIVATE_WORKBOOK_FIXTURES", raising=False) assert private_workbook_fixture_directory().name == "private" @@ -55,6 +90,36 @@ def test_workbook_fixture_runner_produces_truth_graph_simulation_contracts(): assert objects.metadata["sheetjs_ooxml"]["table_files"] +@pytest.mark.slow +def test_workbook_fixture_runner_enforces_manifest_realism_gates_for_selected_cases(): + selected = { + "formula_heavy_model", + "chart_source_update", + "multi_sheet_dependency", + "audit_export_flow", + "protected_formula_overwrite", + "object_fidelity_matrix", + } + fixtures = {fixture.fixture_id: fixture for fixture in WORKBOOK_GOLDEN_FIXTURES if fixture.fixture_id in selected} + runner = get_workbook_fixture_benchmark_runner() + results = {} + + for fixture_id in selected: + fixture = fixtures[fixture_id] + result = runner.run_fixture(fixture) + results[fixture_id] = result + dirty = {f"{item.sheet}!{item.range}".strip("!") for item in result.expected_dirty_ranges} + + assert result.status == "passed", result.failures + assert dirty == set(fixture.expected_dirty_ranges) + assert result.metadata["fixture_operation_count"] == fixture.expected_operation_count + assert result.metadata["simulation_gate"] in fixture.expected_simulation_gates + assert result.metadata["simulation_gate_report_gate"] in fixture.expected_simulation_gates + assert result.metadata["simulation_assertions"]["exactDirtyRanges"] is True + + assert results["audit_export_flow"].verification_status == "blocked" + + @pytest.mark.slow def test_workbook_aware_benchmark_api_mode_runs_and_persists_shape(): client = TestClient(app) diff --git a/backend/tests/test_workbook_boundary_security.py b/backend/tests/test_workbook_boundary_security.py new file mode 100644 index 0000000..147937c --- /dev/null +++ b/backend/tests/test_workbook_boundary_security.py @@ -0,0 +1,332 @@ +"""Security regressions for untrusted workbook and upload boundaries.""" + +from __future__ import annotations + +import io +import subprocess +from pathlib import Path + +import pytest +from fastapi import FastAPI +from fastapi.testclient import TestClient + +from backend.api.routes import documents as document_routes +from backend.api.routes import tasks as task_routes +from backend.data.connectors import XLSXConnector +from backend.excel.workbook_files import WorkbookDiffService, WorkbookIngestionService +from backend.excel.workbook_worker import WorkbookWorkerUnavailable, WorkerResult + + +def _valid_worker_diff() -> dict[str, object]: + return { + "engine": "workbook-worker", + "added_sheets": [], + "removed_sheets": [], + "changed_dimensions": [], + "added_formulas": [], + "removed_formulas": [], + "changed_formulas": [], + "added_cells": [], + "removed_cells": [], + "changed_cells": [], + "added_named_ranges": [], + "removed_named_ranges": [], + "changed_named_ranges": [], + "macro_flag_changed": False, + "changed_merges": [], + "changed_formula_counts": [], + "changed_comment_counts": [], + "changed_table_counts": [], + "changed_validation_counts": [], + "changed_conditional_format_counts": [], + "changed_freeze_panes": [], + "changed_protection": [], + } + + +@pytest.mark.unit +def test_workbook_diff_worker_rejection_never_falls_back(monkeypatch: pytest.MonkeyPatch) -> None: + monkeypatch.setattr( + "backend.excel.workbook_worker.run_worker", + lambda _command, _payload: WorkerResult(ok=False, error="diff budget rejected"), + ) + + with pytest.raises(RuntimeError, match="diff budget rejected"): + WorkbookDiffService().diff({"sheets": []}, {"sheets": []}) + + +@pytest.mark.unit +def test_workbook_diff_rejects_invalid_worker_result(monkeypatch: pytest.MonkeyPatch) -> None: + monkeypatch.setattr( + "backend.excel.workbook_worker.run_worker", + lambda _command, _payload: WorkerResult(ok=True, result=["not", "a", "diff"]), + ) + + with pytest.raises(RuntimeError, match="invalid diff result"): + WorkbookDiffService().diff({}, {}) + + monkeypatch.setattr( + "backend.excel.workbook_worker.run_worker", + lambda _command, _payload: WorkerResult(ok=True, result={}), + ) + with pytest.raises(RuntimeError, match="invalid diff result"): + WorkbookDiffService().diff({}, {}) + + +@pytest.mark.unit +def test_workbook_diff_preserves_valid_worker_result(monkeypatch: pytest.MonkeyPatch) -> None: + expected = _valid_worker_diff() + observed: dict[str, object] = {} + + def worker(command: str, payload: dict[str, object]) -> WorkerResult: + observed.update(command=command, payload=payload) + return WorkerResult(ok=True, result=expected) + + monkeypatch.setattr("backend.excel.workbook_worker.run_worker", worker) + + result = WorkbookDiffService().diff({"sheets": []}, {"sheets": [{"name": "Output"}]}) + + assert result is expected + assert observed == { + "command": "diff", + "payload": {"before": {"sheets": []}, "after": {"sheets": [{"name": "Output"}]}}, + } + + +@pytest.mark.unit +def test_workbook_diff_payload_budget_rejection_never_falls_back(monkeypatch: pytest.MonkeyPatch) -> None: + from backend.excel import workbook_worker + + monkeypatch.setattr(workbook_worker, "WORKBOOK_WORKER_MAX_PAYLOAD_BYTES", 64) + monkeypatch.setattr(workbook_worker, "worker_enabled", lambda: True) + + with pytest.raises(WorkbookWorkerUnavailable, match="payload exceeds configured safety limit"): + WorkbookDiffService().diff({"sheets": [{"name": "A" * 256}]}, {}) + + +@pytest.mark.unit +def test_public_workbook_diff_maps_worker_rejection_to_bounded_error(monkeypatch: pytest.MonkeyPatch) -> None: + monkeypatch.setattr( + WorkbookDiffService, + "diff", + lambda _self, _before, _after: (_ for _ in ()).throw(RuntimeError("worker rejected payload")), + ) + app = FastAPI() + app.include_router(task_routes.router) + + response = TestClient(app, raise_server_exceptions=False).post( + "/api/v1/workbook/diff", + json={"before": {}, "after": {}}, + ) + + assert response.status_code == 400 + assert response.json()["detail"] == "Workbook diff failed: worker rejected payload" + + +@pytest.mark.unit +def test_public_workbook_diff_maps_worker_timeout_to_gateway_timeout(monkeypatch: pytest.MonkeyPatch) -> None: + monkeypatch.setattr( + WorkbookDiffService, + "diff", + lambda _self, _before, _after: (_ for _ in ()).throw(subprocess.TimeoutExpired(["node", "cli.mjs", "diff"], 1)), + ) + app = FastAPI() + app.include_router(task_routes.router) + + response = TestClient(app, raise_server_exceptions=False).post( + "/api/v1/workbook/diff", + json={"before": {}, "after": {}}, + ) + + assert response.status_code == 504 + assert response.json()["detail"] == "Workbook diff timed out" + + +@pytest.mark.unit +@pytest.mark.asyncio +async def test_xlsx_connector_uses_bounded_ingestion_for_owned_bytes( + tmp_path: Path, + monkeypatch: pytest.MonkeyPatch, +) -> None: + documents_dir = tmp_path / "documents" + documents_dir.mkdir() + workbook_path = documents_dir / "doc_owned_model.xlsx" + workbook_path.write_bytes(b"owned-workbook-bytes") + + class Store: + def get_document(self, document_id: str): + if document_id != "doc_owned": + return None + return { + "document_id": document_id, + "session_id": "session-1", + "filename": "model.xlsx", + "file_path": str(workbook_path), + } + + captured: dict[str, object] = {} + + def bounded_summary(_self, content: bytes, filename: str): + captured.update(content=content, filename=filename) + return { + "engine": "workbook-worker", + "sheet_names": ["Model"], + "active_sheet": "Model", + "sheets": [ + { + "name": "Model", + "max_row": 2, + "max_column": 2, + "preview": [["Metric", "Value"], ["Revenue", 100]], + } + ], + } + + monkeypatch.setattr("backend.storage.get_session_store", lambda: Store()) + monkeypatch.setattr("backend.data.connectors.DATA_DIR", tmp_path, raising=False) + monkeypatch.setattr(WorkbookIngestionService, "summarize", bounded_summary) + + result = await XLSXConnector().fetch("doc_owned") + + assert captured == {"content": b"owned-workbook-bytes", "filename": "model.xlsx"} + assert result.headers == ["Metric", "Value"] + assert result.data == [{"Metric": "Revenue", "Value": 100}] + assert result.metadata["engine"] == "workbook-worker" + + +@pytest.mark.unit +@pytest.mark.asyncio +async def test_xlsx_connector_propagates_worker_rejection_instead_of_empty_success( + tmp_path: Path, + monkeypatch: pytest.MonkeyPatch, +) -> None: + documents_dir = tmp_path / "documents" + documents_dir.mkdir() + workbook_path = documents_dir / "doc_owned_model.xlsx" + workbook_path.write_bytes(b"owned-workbook-bytes") + + class Store: + def get_document(self, _document_id: str): + return {"filename": "model.xlsx", "file_path": str(workbook_path)} + + monkeypatch.setattr("backend.storage.get_session_store", lambda: Store()) + monkeypatch.setattr("backend.data.connectors.DATA_DIR", tmp_path, raising=False) + monkeypatch.setattr( + WorkbookIngestionService, + "summarize", + lambda *_args, **_kwargs: (_ for _ in ()).throw(RuntimeError("archive rejected")), + ) + + with pytest.raises(RuntimeError, match="archive rejected"): + await XLSXConnector().fetch("doc_owned") + + +@pytest.mark.unit +def test_upload_partial_write_failure_leaves_no_unregistered_bytes( + tmp_path: Path, + monkeypatch: pytest.MonkeyPatch, +) -> None: + from backend.services import document_cleanup + + def fail_after_partial_write(stream: io.BufferedWriter, content: bytes) -> None: + stream.write(content[:8]) + stream.flush() + raise OSError("disk write failed") + + monkeypatch.setattr(document_routes, "DATA_DIR", tmp_path) + monkeypatch.setattr(document_cleanup, "_write_upload_stream", fail_after_partial_write) + app = FastAPI() + app.include_router(document_routes.router) + + response = TestClient(app).post( + "/api/v1/documents/upload", + data={"session_id": "write-failure"}, + files={"file": ("source.pdf", b"%PDF-1.7\nprivate", "application/pdf")}, + ) + + assert response.status_code == 500 + assert list((tmp_path / "documents").glob("**/*")) == [] + assert list((tmp_path / "document-cleanup").glob("*.json")) == [] + + +@pytest.mark.unit +def test_upload_unlink_failure_is_quarantined_registered_and_recovered( + tmp_path: Path, + monkeypatch: pytest.MonkeyPatch, +) -> None: + from backend.services import document_cleanup + + async def fail_parse(*_args, **_kwargs): + raise RuntimeError("parser rejected payload") + + original_unlink = document_cleanup._unlink_file + + def fail_unlink(_path: Path) -> None: + raise OSError("unlink denied") + + monkeypatch.setattr(document_routes, "DATA_DIR", tmp_path) + monkeypatch.setattr(document_routes, "_parse_uploaded_content", fail_parse) + monkeypatch.setattr(document_cleanup, "_unlink_file", fail_unlink) + app = FastAPI() + app.include_router(document_routes.router) + + response = TestClient(app).post( + "/api/v1/documents/upload", + data={"session_id": "cleanup-recovery"}, + files={"file": ("source.pdf", b"%PDF-1.7\nprivate", "application/pdf")}, + ) + + assert response.status_code == 500 + assert list((tmp_path / "documents").glob("**/*")) == [] + assert len(list((tmp_path / "document-quarantine").glob("*.quarantine"))) == 1 + assert len(list((tmp_path / "document-cleanup").glob("cleanup_*.json"))) == 1 + + monkeypatch.setattr(document_cleanup, "_unlink_file", original_unlink) + assert document_cleanup.recover_pending_upload_cleanups(tmp_path) == 1 + assert list((tmp_path / "document-quarantine").glob("*")) == [] + assert list((tmp_path / "document-cleanup").glob("*")) == [] + + +@pytest.mark.unit +def test_upload_quarantine_is_recoverable_even_when_receipt_storage_fails( + tmp_path: Path, + monkeypatch: pytest.MonkeyPatch, +) -> None: + from backend.services import document_cleanup + + async def fail_parse(*_args, **_kwargs): + raise RuntimeError("parser rejected payload") + + original_unlink = document_cleanup._unlink_file + original_atomic_write = document_cleanup._atomic_json_write + monkeypatch.setattr(document_routes, "DATA_DIR", tmp_path) + monkeypatch.setattr(document_routes, "_parse_uploaded_content", fail_parse) + monkeypatch.setattr( + document_cleanup, + "_unlink_file", + lambda _path: (_ for _ in ()).throw(OSError("unlink denied")), + ) + monkeypatch.setattr( + document_cleanup, + "_atomic_json_write", + lambda _path, _payload: (_ for _ in ()).throw(OSError("receipt disk full")), + ) + app = FastAPI() + app.include_router(document_routes.router) + + response = TestClient(app).post( + "/api/v1/documents/upload", + data={"session_id": "quarantine-recovery"}, + files={"file": ("source.pdf", b"%PDF-1.7\nprivate", "application/pdf")}, + ) + + assert response.status_code == 500 + assert list((tmp_path / "documents").glob("**/*")) == [] + assert len(list((tmp_path / "document-quarantine").glob("*.quarantine"))) == 1 + assert list((tmp_path / "document-cleanup").glob("cleanup_*.json")) == [] + (tmp_path / "document-cleanup").rmdir() + + monkeypatch.setattr(document_cleanup, "_unlink_file", original_unlink) + monkeypatch.setattr(document_cleanup, "_atomic_json_write", original_atomic_write) + assert document_cleanup.recover_pending_upload_cleanups(tmp_path) == 1 + assert list((tmp_path / "document-quarantine").glob("*")) == [] diff --git a/backend/tests/test_workbook_worker_bridge.py b/backend/tests/test_workbook_worker_bridge.py new file mode 100644 index 0000000..0a06111 --- /dev/null +++ b/backend/tests/test_workbook_worker_bridge.py @@ -0,0 +1,167 @@ +from __future__ import annotations + +import json +import os +import subprocess +import sys +import time + +import pytest + +from backend.excel import workbook_worker +from backend.excel.workbook_worker import WorkbookWorkerUnavailable + + +def test_run_worker_maps_result_engine_and_structured_fallback(monkeypatch): + captured: dict = {} + monkeypatch.setattr(workbook_worker, "worker_enabled", lambda: True) + monkeypatch.setattr(workbook_worker, "_validate_payload", lambda command, payload: None) + + def fake_run(*args, **kwargs): + captured["env"] = kwargs["env"] + return subprocess.CompletedProcess( + args[0], + 0, + stdout=json.dumps( + { + "ok": True, + "result": {"metadata": {"warnings": []}}, + "metadata": { + "resultEngine": "hyperformula", + "fallback": {"used": False, "allowed": True}, + "warningCount": 0, + }, + } + ), + stderr="", + ) + + recorded: dict = {} + monkeypatch.setattr(workbook_worker, "_run_bounded_process", fake_run) + monkeypatch.setattr( + "backend.observability.record_worker_execution", + lambda command, duration, **fields: recorded.update({"command": command, **fields}), + ) + + result = workbook_worker.run_worker("formula-analyze", {"formulas": []}) + + assert result.ok is True + assert recorded["engine"] == "hyperformula" + assert recorded["fallback"] is False + assert "--max-old-space-size=" in captured["env"]["NODE_OPTIONS"] + + +def test_run_worker_rejects_oversized_output_before_json_decode(monkeypatch): + monkeypatch.setattr(workbook_worker, "worker_enabled", lambda: True) + monkeypatch.setattr(workbook_worker, "_validate_payload", lambda command, payload: None) + monkeypatch.setattr(workbook_worker, "WORKBOOK_WORKER_MAX_OUTPUT_BYTES", 32) + + def fake_overflow(*args, **kwargs): + assert kwargs["max_output_bytes"] == 32 + raise WorkbookWorkerUnavailable("Workbook worker output exceeds configured 32 byte safety limit") + + monkeypatch.setattr(workbook_worker, "_run_bounded_process", fake_overflow) + + with pytest.raises(WorkbookWorkerUnavailable, match="output exceeds"): + workbook_worker.run_worker("ingest", {}) + + +def test_bounded_process_preserves_stdin_stdout_and_stderr(tmp_path): + process = workbook_worker._run_bounded_process( + [ + sys.executable, + "-c", + ( + "import json,sys; " + "payload=json.load(sys.stdin); " + "sys.stderr.write('worker warning'); " + "json.dump({'ok': True, 'result': payload}, sys.stdout)" + ), + ], + input_text=json.dumps({"value": 42}), + cwd=str(tmp_path), + env=os.environ.copy(), + timeout_seconds=2, + max_output_bytes=1024, + ) + + assert process.returncode == 0 + assert json.loads(process.stdout) == {"ok": True, "result": {"value": 42}} + assert process.stderr == "worker warning" + + +def test_bounded_process_preserves_timeout_partial_output_bytes(tmp_path): + with pytest.raises(subprocess.TimeoutExpired) as exc_info: + workbook_worker._run_bounded_process( + [ + sys.executable, + "-c", + ( + "import sys,time; " + "sys.stdout.write('partial stdout'); sys.stdout.flush(); " + "sys.stderr.write('partial stderr'); sys.stderr.flush(); " + "time.sleep(5)" + ), + ], + input_text="{}", + cwd=str(tmp_path), + env=os.environ.copy(), + timeout_seconds=0.2, + max_output_bytes=1024, + ) + + assert exc_info.value.output == b"partial stdout" + assert exc_info.value.stderr == b"partial stderr" + + +@pytest.mark.parametrize("stream_name", ["stdout", "stderr"]) +def test_bounded_process_terminates_on_stream_overflow(tmp_path, stream_name): + started = time.monotonic() + + with pytest.raises(WorkbookWorkerUnavailable, match="output exceeds"): + workbook_worker._run_bounded_process( + [ + sys.executable, + "-c", + (f"import sys,time; stream=sys.{stream_name}; stream.write('x' * 131072); stream.flush(); time.sleep(5)"), + ], + input_text="{}", + cwd=str(tmp_path), + env=os.environ.copy(), + timeout_seconds=3, + max_output_bytes=1024, + ) + + assert time.monotonic() - started < 2 + + +@pytest.mark.parametrize("trigger", ["timeout", "overflow"]) +def test_bounded_process_terminates_descendants(tmp_path, trigger): + launch_marker = tmp_path / f"descendant-{trigger}-launched.txt" + descendant_marker = tmp_path / f"descendant-{trigger}.txt" + launch_literal = json.dumps(str(launch_marker)) + marker_literal = json.dumps(str(descendant_marker)) + child_code = f"import pathlib,time; time.sleep(0.6); pathlib.Path({marker_literal}).write_text('survived', encoding='utf-8')" + parent_code = [ + "import pathlib,subprocess,sys,time", + f"child=subprocess.Popen([sys.executable, '-c', {child_code!r}])", + f"pathlib.Path({launch_literal}).write_text(str(child.pid), encoding='utf-8')", + ] + if trigger == "overflow": + parent_code.append("sys.stdout.write('x' * 131072); sys.stdout.flush()") + parent_code.append("time.sleep(5)") + expected_error = subprocess.TimeoutExpired if trigger == "timeout" else WorkbookWorkerUnavailable + + with pytest.raises(expected_error): + workbook_worker._run_bounded_process( + [sys.executable, "-c", "; ".join(parent_code)], + input_text="{}", + cwd=str(tmp_path), + env=os.environ.copy(), + timeout_seconds=0.3 if trigger == "timeout" else 3, + max_output_bytes=1024, + ) + + time.sleep(0.8) + assert launch_marker.exists() + assert not descendant_marker.exists() diff --git a/backend/tests/test_workflow_lifecycle_fencing.py b/backend/tests/test_workflow_lifecycle_fencing.py new file mode 100644 index 0000000..5ac7e80 --- /dev/null +++ b/backend/tests/test_workflow_lifecycle_fencing.py @@ -0,0 +1,501 @@ +from __future__ import annotations + +import asyncio +import base64 +import multiprocessing +import threading +from pathlib import Path +from types import SimpleNamespace + +import pytest + +from backend.agents import workflows as workflows_module +from backend.agents.workflows import WorkflowEngine +from backend.api.security.ownership import OwnershipNotFoundError, ownership_actor_scope +from backend.services import lifecycle as lifecycle_module +from backend.services.lifecycle import LifecycleCoordinator, LifecycleState +from backend.session_store import SessionStore + + +class _NoopJobManager: + def purge_session(self, _session_id: str) -> dict[str, int]: + return {"jobs_revoked": 0, "threads_pending": 0} + + +def _hold_workflow_process_lock( + workflows_dir: str, + session_id: str, + ready, + release, +) -> None: + from backend.agents import workflows as child_workflows + + child_workflows.WORKFLOWS_DIR = Path(workflows_dir) + with child_workflows._workflow_storage_lock(session_id): + ready.set() + release.wait(timeout=5) + + +def _attempt_late_workflow_create( + workflows_dir: str, + database_path: str, + session_id: str, + result_queue, +) -> None: + from backend.agents import workflows as child_workflows + from backend.agents.workflows import WorkflowEngine as ChildWorkflowEngine + from backend.session_store import SessionStore as ChildSessionStore + + child_workflows.WORKFLOWS_DIR = Path(workflows_dir) + engine = ChildWorkflowEngine(store=ChildSessionStore(database_path)) + try: + engine.create_workflow( + name="Late process write", + description="Must be fenced by the durable tombstone", + steps=[], + session_id=session_id, + ) + except Exception as exc: + result_queue.put(exc.__class__.__name__) + else: # pragma: no cover - security regression branch + result_queue.put("created") + + +@pytest.fixture(autouse=True) +def _receipt_signing_key(monkeypatch: pytest.MonkeyPatch) -> None: + monkeypatch.setenv("EXCEL_AGENT_LIFECYCLE_RECEIPT_KEY_ID", "workflow-test-key") + monkeypatch.setenv( + "EXCEL_AGENT_LIFECYCLE_RECEIPT_HMAC_KEY", + base64.urlsafe_b64encode(b"workflow-lifecycle-test-key-material-32-bytes").decode(), + ) + + +@pytest.mark.unit +@pytest.mark.asyncio +async def test_late_workflow_create_cannot_survive_session_purge( + tmp_path, + monkeypatch: pytest.MonkeyPatch, +) -> None: + session_id = "workflow-late-write" + store = SessionStore(tmp_path / "sessions.db") + store.get_or_create_session(session_id) + workflows_dir = tmp_path / "workflows" + monkeypatch.setattr(workflows_module, "WORKFLOWS_DIR", workflows_dir) + + engine = WorkflowEngine(store=store) + original = engine.create_workflow( + name="Before purge", + description="Must be deleted", + steps=[], + session_id=session_id, + ) + monkeypatch.setattr(workflows_module, "get_workflow_engine", lambda: engine) + + pause_reached = asyncio.Event() + allow_purge_to_finish = asyncio.Event() + + async def pause_after_workflow_purge(_session_id: str) -> None: + pause_reached.set() + await allow_purge_to_finish.wait() + + coordinator = LifecycleCoordinator( + store=store, + job_manager=_NoopJobManager(), + receipts_dir=tmp_path / "receipts", + ) + coordinator.register_purger("pause_after_workflow_purge", pause_after_workflow_purge) + monkeypatch.setattr(lifecycle_module, "_coordinator", coordinator) + + purge_task = asyncio.create_task(coordinator.purge_session(session_id)) + await asyncio.wait_for(pause_reached.wait(), timeout=2) + + try: + with pytest.raises(OwnershipNotFoundError, match="Session not found"): + engine.create_workflow( + name="Late write", + description="Must be rejected", + steps=[], + session_id=session_id, + ) + finally: + allow_purge_to_finish.set() + receipt = await asyncio.wait_for(purge_task, timeout=2) + + assert receipt.state is LifecycleState.VERIFIED + assert receipt.workflows_deleted == 1 + assert engine.count_session_workflows(session_id) == 0 + assert WorkflowEngine(store=store).get_workflow(original.id) is None + + +@pytest.mark.unit +def test_session_bound_workflow_writes_require_an_existing_active_session( + tmp_path, + monkeypatch: pytest.MonkeyPatch, +) -> None: + store = SessionStore(tmp_path / "sessions.db") + monkeypatch.setattr(workflows_module, "WORKFLOWS_DIR", tmp_path / "workflows") + engine = WorkflowEngine(store=store) + + with pytest.raises(OwnershipNotFoundError, match="Session not found"): + engine.create_workflow( + name="Unknown session", + description="Must not create a session as a side effect", + steps=[], + session_id="missing-session", + ) + + assert store.get_session_lifecycle_state("missing-session") is None + assert store.session_exists("missing-session") is False + assert not list((tmp_path / "workflows").glob("*.json")) + + +@pytest.mark.unit +def test_workflow_session_binding_cannot_be_changed_to_bypass_a_purge_lock( + tmp_path, + monkeypatch: pytest.MonkeyPatch, +) -> None: + store = SessionStore(tmp_path / "sessions.db") + store.get_or_create_session("target-session") + monkeypatch.setattr(workflows_module, "WORKFLOWS_DIR", tmp_path / "workflows") + engine = WorkflowEngine(store=store) + workflow = engine.create_workflow( + name="Unbound workflow", + description="Binding is immutable once persisted", + steps=[], + ) + + workflow.session_id = "target-session" + with pytest.raises(ValueError, match="not found"): + engine.save_workflow(workflow) + + persisted = workflows_module._read_workflow_record( + workflows_module._workflow_path(workflow.id) + ) + assert persisted is not None + assert persisted["session_id"] == "" + assert workflow.session_id == "" + + +@pytest.mark.unit +@pytest.mark.asyncio +async def test_create_update_delete_and_execute_persistence_fail_after_purging( + tmp_path, + monkeypatch: pytest.MonkeyPatch, +) -> None: + session_id = "workflow-write-fence" + store = SessionStore(tmp_path / "sessions.db") + store.get_or_create_session(session_id) + monkeypatch.setattr(workflows_module, "WORKFLOWS_DIR", tmp_path / "workflows") + engine = WorkflowEngine(store=store) + workflow = engine.create_workflow( + name="Active workflow", + description="Initially writable", + steps=[], + session_id=session_id, + ) + store.set_session_lifecycle_state( + session_id, + "purging", + receipt_id="del_workflow_write_fence", + started_at=1.0, + ) + + workflow.name = "Rejected update" + with pytest.raises(OwnershipNotFoundError, match="Session not found"): + engine.save_workflow(workflow) + with pytest.raises(OwnershipNotFoundError, match="Session not found"): + engine.create_workflow( + name="Rejected create", + description="Too late", + steps=[], + session_id=session_id, + ) + with pytest.raises(OwnershipNotFoundError, match="Session not found"): + engine.delete_workflow(workflow.id) + with pytest.raises(ValueError, match="not found"): + await engine.execute_workflow(workflow.id) + + assert engine.count_session_workflows(session_id) == 1 + assert engine.purge_session(session_id) == 1 + assert engine.count_session_workflows(session_id) == 0 + + +@pytest.mark.unit +def test_purge_inventory_includes_crash_orphaned_atomic_write_files( + tmp_path, + monkeypatch: pytest.MonkeyPatch, +) -> None: + session_id = "workflow-orphaned-write" + store = SessionStore(tmp_path / "sessions.db") + store.get_or_create_session(session_id) + workflows_dir = tmp_path / "workflows" + monkeypatch.setattr(workflows_module, "WORKFLOWS_DIR", workflows_dir) + engine = WorkflowEngine(store=store) + workflow = engine.create_workflow( + name="Crash candidate", + description="Atomic temp payload must be lifecycle managed", + steps=[], + session_id=session_id, + ) + canonical = workflows_module._workflow_path(workflow.id) + orphan = workflows_dir / f".{canonical.name}.crash.tmp" + orphan.write_bytes(canonical.read_bytes()) + store.set_session_lifecycle_state( + session_id, + "purging", + receipt_id="del_workflow_orphaned_write", + started_at=1.0, + ) + + assert engine.count_session_workflows(session_id) == 2 + assert engine.purge_session(session_id) == 1 + assert engine.count_session_workflows(session_id) == 0 + assert orphan.exists() is False + + +@pytest.mark.unit +def test_workflow_deletion_inventory_fails_closed_on_unreadable_records( + tmp_path, + monkeypatch: pytest.MonkeyPatch, +) -> None: + workflows_dir = tmp_path / "workflows" + workflows_dir.mkdir() + (workflows_dir / "corrupt.json").write_text("{", encoding="utf-8") + monkeypatch.setattr(workflows_module, "WORKFLOWS_DIR", workflows_dir) + engine = WorkflowEngine(store=SessionStore(tmp_path / "sessions.db")) + + with pytest.raises(RuntimeError, match="Could not read durable workflow"): + engine.count_session_workflows("any-session") + + +@pytest.mark.unit +@pytest.mark.asyncio +async def test_execution_stat_write_fails_if_purge_starts_during_pipeline( + tmp_path, + monkeypatch: pytest.MonkeyPatch, +) -> None: + session_id = "workflow-execute-fence" + store = SessionStore(tmp_path / "sessions.db") + store.get_or_create_session(session_id) + monkeypatch.setattr(workflows_module, "WORKFLOWS_DIR", tmp_path / "workflows") + engine = WorkflowEngine(store=store) + workflow = engine.create_workflow( + name="Long execution", + description="Purge begins while the model step is running", + steps=[{"step_type": "prompt", "config": {"prompt": "Run"}}], + session_id=session_id, + ) + pipeline_entered = asyncio.Event() + pipeline_release = asyncio.Event() + + async def paused_pipeline(_prompt: str) -> dict[str, list]: + pipeline_entered.set() + await pipeline_release.wait() + return {"operations": []} + + execution = asyncio.create_task( + engine.execute_workflow(workflow.id, pipeline_fn=paused_pipeline) + ) + await asyncio.wait_for(pipeline_entered.wait(), timeout=2) + store.set_session_lifecycle_state( + session_id, + "purging", + receipt_id="del_workflow_execute_fence", + started_at=1.0, + ) + pipeline_release.set() + + with pytest.raises(OwnershipNotFoundError, match="Session not found"): + await asyncio.wait_for(execution, timeout=2) + persisted = workflows_module._read_workflow_record( + workflows_module._workflow_path(workflow.id) + ) + assert persisted is not None + assert persisted["execution_count"] == 0 + assert engine.purge_session(session_id) == 1 + + +@pytest.mark.unit +def test_inflight_save_is_deleted_before_overlapping_purge_returns( + tmp_path, + monkeypatch: pytest.MonkeyPatch, +) -> None: + session_id = "workflow-overlap-fence" + store = SessionStore(tmp_path / "sessions.db") + store.get_or_create_session(session_id) + monkeypatch.setattr(workflows_module, "WORKFLOWS_DIR", tmp_path / "workflows") + engine = WorkflowEngine(store=store) + workflow = engine.create_workflow( + name="Original", + description="Original", + steps=[], + session_id=session_id, + ) + original_atomic_write = workflows_module._atomic_write_workflow + save_entered = threading.Event() + allow_save = threading.Event() + + def paused_atomic_write(path: Path, payload: dict) -> None: + if payload.get("name") == "Racing update": + save_entered.set() + assert allow_save.wait(timeout=5) + original_atomic_write(path, payload) + + monkeypatch.setattr(workflows_module, "_atomic_write_workflow", paused_atomic_write) + workflow.name = "Racing update" + writer_errors: list[BaseException] = [] + + def save() -> None: + try: + engine.save_workflow(workflow) + except BaseException as exc: # pragma: no cover - asserted empty below + writer_errors.append(exc) + + writer = threading.Thread(target=save) + writer.start() + assert save_entered.wait(timeout=2) + store.set_session_lifecycle_state( + session_id, + "purging", + receipt_id="del_workflow_overlap_fence", + started_at=1.0, + ) + purge_result: list[int] = [] + purge = threading.Thread(target=lambda: purge_result.append(engine.purge_session(session_id))) + purge.start() + assert purge.is_alive() + + allow_save.set() + writer.join(timeout=5) + purge.join(timeout=5) + + assert not writer.is_alive() + assert not purge.is_alive() + assert writer_errors == [] + assert purge_result == [1] + assert engine.count_session_workflows(session_id) == 0 + + +@pytest.mark.unit +def test_workflow_lock_excludes_another_process( + tmp_path, + monkeypatch: pytest.MonkeyPatch, +) -> None: + workflows_dir = tmp_path / "workflows" + session_id = "workflow-process-lock" + monkeypatch.setattr(workflows_module, "WORKFLOWS_DIR", workflows_dir) + context = multiprocessing.get_context("spawn") + child_ready = context.Event() + release_child = context.Event() + child = context.Process( + target=_hold_workflow_process_lock, + args=(str(workflows_dir), session_id, child_ready, release_child), + ) + child.start() + assert child_ready.wait(timeout=5) + parent_acquired = threading.Event() + + def acquire_in_parent() -> None: + with workflows_module._workflow_storage_lock(session_id): + parent_acquired.set() + + parent = threading.Thread(target=acquire_in_parent) + parent.start() + assert parent_acquired.wait(timeout=0.2) is False + release_child.set() + assert parent_acquired.wait(timeout=5) + parent.join(timeout=5) + child.join(timeout=5) + + assert child.exitcode == 0 + assert not parent.is_alive() + + +@pytest.mark.unit +def test_durable_purge_tombstone_fences_a_different_process( + tmp_path, + monkeypatch: pytest.MonkeyPatch, +) -> None: + workflows_dir = tmp_path / "workflows" + database_path = tmp_path / "sessions.db" + session_id = "workflow-process-tombstone" + monkeypatch.setattr(workflows_module, "WORKFLOWS_DIR", workflows_dir) + store = SessionStore(database_path) + store.get_or_create_session(session_id) + store.set_session_lifecycle_state( + session_id, + "purging", + receipt_id="del_workflow_process_tombstone", + started_at=1.0, + ) + context = multiprocessing.get_context("spawn") + result_queue = context.Queue() + child = context.Process( + target=_attempt_late_workflow_create, + args=(str(workflows_dir), str(database_path), session_id, result_queue), + ) + child.start() + child.join(timeout=10) + + assert child.exitcode == 0 + assert result_queue.get(timeout=2) == "OwnershipNotFoundError" + assert not list(workflows_dir.glob("*.json")) + + +@pytest.mark.unit +@pytest.mark.asyncio +async def test_active_session_workflow_behavior_and_owner_scope_are_preserved( + tmp_path, + monkeypatch: pytest.MonkeyPatch, +) -> None: + session_id = "workflow-happy-path" + store = SessionStore(tmp_path / "sessions.db") + store.get_or_create_session(session_id) + monkeypatch.setattr(workflows_module, "WORKFLOWS_DIR", tmp_path / "workflows") + engine = WorkflowEngine(store=store) + workflow = engine.create_workflow( + name="Happy path", + description="Still supported", + steps=[], + session_id=session_id, + ) + workflow.description = "Updated" + engine.save_workflow(workflow) + + result = await engine.execute_workflow(workflow.id) + + assert result["workflow_id"] == workflow.id + assert engine.get_workflow(workflow.id) is workflow + assert workflow.execution_count == 1 + assert engine.count_session_workflows(session_id) == 1 + assert engine.delete_workflow(workflow.id) is True + assert engine.count_session_workflows(session_id) == 0 + + +@pytest.mark.unit +def test_session_workflow_owner_is_enforced_by_the_durable_session_store( + tmp_path, + monkeypatch: pytest.MonkeyPatch, +) -> None: + session_id = "workflow-owner-scope" + store = SessionStore(tmp_path / "sessions.db") + monkeypatch.setattr(workflows_module, "WORKFLOWS_DIR", tmp_path / "workflows") + engine = WorkflowEngine(store=store) + with ownership_actor_scope(SimpleNamespace(user_id="alice", role="analyst")): + store.get_or_create_session(session_id) + workflow = engine.create_workflow( + name="Alice workflow", + description="Private", + steps=[], + session_id=session_id, + ) + + with ownership_actor_scope(SimpleNamespace(user_id="bob", role="analyst")): + assert engine.get_workflow(workflow.id) is None + assert workflow.id not in {item["id"] for item in engine.list_workflows()} + with pytest.raises(OwnershipNotFoundError, match="Session not found"): + engine.save_workflow(workflow) + + with ownership_actor_scope(SimpleNamespace(user_id="alice", role="analyst")): + assert engine.get_workflow(workflow.id) is workflow + assert engine.delete_workflow(workflow.id) is True diff --git a/docker-compose.prod.yml b/docker-compose.prod.yml index 0dcc50a..53023fc 100644 --- a/docker-compose.prod.yml +++ b/docker-compose.prod.yml @@ -1,5 +1,3 @@ -version: "3.8" - # ── Production Docker Compose ──────────────────────────────────────────── # Includes: HTTPS termination, health checks, volumes, resource limits, # log aggregation, and optional PostgreSQL. @@ -7,22 +5,38 @@ version: "3.8" services: # ── Nginx Reverse Proxy (HTTPS termination) ───────────── nginx: - image: nginx:1.27-alpine + image: finance-ai-edge:${IMAGE_TAG:-local} + build: + context: . + dockerfile: infra/nginx/Dockerfile ports: - - "443:443" - - "80:80" + - "443:8443" + - "80:8080" volumes: - - ./infra/nginx/nginx.conf:/etc/nginx/nginx.conf:ro - ./infra/nginx/certs:/etc/nginx/certs:ro + - ./infra/mtls/ca.crt:/etc/excel-agent/mtls/ca.crt:ro + - ./infra/mtls/edge-client.crt:/etc/excel-agent/mtls/edge-client.crt:ro + - ./infra/mtls/edge-client.key:/etc/excel-agent/mtls/edge-client.key:ro + environment: + - DEPLOY_ORIGIN=${DEPLOY_ORIGIN:?Set DEPLOY_ORIGIN to the public HTTPS add-in origin} + - OIDC_BROWSER_CONNECT_ORIGINS=${OIDC_BROWSER_CONNECT_ORIGINS:?Set exact space-separated HTTPS origins for OIDC discovery, token exchange, and JWKS} depends_on: backend: condition: service_healthy restart: unless-stopped healthcheck: - test: ["CMD", "wget", "--spider", "-q", "http://localhost:80/health"] + test: + - CMD-SHELL + - >- + wget --spider -q http://127.0.0.1:8080/edge-health + && test -s /usr/share/nginx/html/src/taskpane/taskpane.html + && test -s /usr/share/nginx/html/src/spreadsheet/spreadsheet.html + && grep -Fq "$$DEPLOY_ORIGIN" /usr/share/nginx/html/manifest.xml + && ! grep -Eiq 'localhost|\$\{' /usr/share/nginx/html/manifest.xml interval: 30s timeout: 5s retries: 3 + start_period: 10s # ── Backend API ────────────────────────────────────────── backend: @@ -31,17 +45,72 @@ services: dockerfile: backend/Dockerfile expose: - "8000" + command: + - python + - -m + - uvicorn + - backend.main:app + - --host + - 0.0.0.0 + - --port + - "8000" + - --ssl-keyfile + - /etc/excel-agent/mtls/backend-server.key + - --ssl-certfile + - /etc/excel-agent/mtls/backend-server.crt + - --ssl-ca-certs + - /etc/excel-agent/mtls/ca.crt + - --ssl-cert-reqs + - "2" volumes: - backend_data:/app/data + - ./infra/mtls/ca.crt:/etc/excel-agent/mtls/ca.crt:ro + - ./infra/mtls/backend-server.crt:/etc/excel-agent/mtls/backend-server.crt:ro + - ./infra/mtls/backend-server.key:/etc/excel-agent/mtls/backend-server.key:ro + - ./infra/mtls/health-client.crt:/etc/excel-agent/mtls/health-client.crt:ro + - ./infra/mtls/health-client.key:/etc/excel-agent/mtls/health-client.key:ro env_file: - - .env + - path: .env + required: false environment: - APP_ENV=prod + - CONFIG_PROFILE=prod - AUTH_ENABLED=true + - API_KEY=${API_KEY:?Set a strong reserved backend API key} + - EXCEL_AGENT_LIFECYCLE_RECEIPT_KEY_ID=${EXCEL_AGENT_LIFECYCLE_RECEIPT_KEY_ID:?Set the deletion-receipt signing key ID} + - EXCEL_AGENT_LIFECYCLE_RECEIPT_HMAC_KEY=${EXCEL_AGENT_LIFECYCLE_RECEIPT_HMAC_KEY:?Set a URL-safe base64 deletion-receipt key of at least 32 bytes} + - EXCEL_AGENT_LIFECYCLE_RECEIPT_PREVIOUS_KEYS=${EXCEL_AGENT_LIFECYCLE_RECEIPT_PREVIOUS_KEYS:-{}} + - STATIC_KEY_ADMIN_ENABLED=false + - RBAC_ENABLED=true + - OIDC_ENABLED=true + - OIDC_ISSUER=${OIDC_ISSUER:?Set the exact HTTPS OIDC issuer} + - OIDC_AUDIENCE=${OIDC_AUDIENCE:?Set the Excel Agent API audience} + - OIDC_JWKS_URL=${OIDC_JWKS_URL:?Set the HTTPS OIDC JWKS URL} + - OIDC_ALLOWED_TENANTS=${OIDC_ALLOWED_TENANTS:?Set an explicit comma-separated OIDC tenant allowlist} + - OIDC_ALLOWED_ALGORITHMS=${OIDC_ALLOWED_ALGORITHMS:-RS256} + - OIDC_REQUIRE_MFA=${OIDC_REQUIRE_MFA:-true} + - OIDC_CLIENT_ID=${OIDC_CLIENT_ID:?Set the public OIDC authorization-code PKCE client ID} + - OIDC_CLIENT_REDIRECT_URI=${DEPLOY_ORIGIN:?Set DEPLOY_ORIGIN}/src/taskpane/taskpane.html?oidc_callback=1 + - OIDC_CLIENT_SCOPES=${OIDC_CLIENT_SCOPES:-openid,profile,email} + - OIDC_DISCOVERY_URL=${OIDC_DISCOVERY_URL:-} + - OIDC_AUTHORIZATION_ENDPOINT=${OIDC_AUTHORIZATION_ENDPOINT:-} + - OIDC_TOKEN_ENDPOINT=${OIDC_TOKEN_ENDPOINT:-} + - OIDC_EXTRA_AUTHORIZE_PARAMS_JSON=${OIDC_EXTRA_AUTHORIZE_PARAMS_JSON:-{}} + - CORS_ORIGINS=${DEPLOY_ORIGIN:?Set DEPLOY_ORIGIN to the public HTTPS add-in origin} + - TRUSTED_PROXY_HEADERS=true - OLLAMA_BASE_URL=http://ollama:11434 - DB_ENGINE=${DB_ENGINE:-postgres} - DATABASE_URL=${DATABASE_URL:?Set DATABASE_URL for production Postgres} - WORKBOOK_WORKER_DIR=/app/workbook-worker + - FORMULA_ENGINE=${FORMULA_ENGINE:-univer} + - HYPERFORMULA_ENABLED=${HYPERFORMULA_ENABLED:-false} + - HYPERFORMULA_DISTRIBUTED=${HYPERFORMULA_DISTRIBUTED:-false} + - HYPERFORMULA_LICENSE_APPROVED=${HYPERFORMULA_LICENSE_APPROVED:-false} + - HYPERFORMULA_LICENSE_KEY=${HYPERFORMULA_LICENSE_KEY:-} + - BYOK_ENABLED=${BYOK_ENABLED:-false} + - ZERO_DATA_RETENTION=${ZERO_DATA_RETENTION:-false} + - AI_EXTERNAL_SENDING_ALLOWED=${AI_EXTERNAL_SENDING_ALLOWED:?Set the production AI egress policy to true or false} + - AI_EXTERNAL_DOCUMENTS_ALLOWED=${AI_EXTERNAL_DOCUMENTS_ALLOWED:?Set the production document egress policy to true or false} depends_on: ollama: condition: service_started @@ -49,7 +118,17 @@ services: condition: service_healthy restart: unless-stopped healthcheck: - test: ["CMD", "python", "-c", "import httpx; httpx.get('http://localhost:8000/health').raise_for_status()"] + test: + - CMD + - python + - -c + - >- + import httpx; + httpx.get( + 'https://backend:8000/health', + verify='/etc/excel-agent/mtls/ca.crt', + cert=('/etc/excel-agent/mtls/health-client.crt','/etc/excel-agent/mtls/health-client.key'), + timeout=5).raise_for_status() interval: 30s timeout: 10s retries: 3 @@ -96,7 +175,7 @@ services: # ── PostgreSQL (required for production runtime storage) ─ postgres: - image: postgres:16-alpine + image: postgres:16.14-alpine3.24@sha256:57c72fd2a128e416c7fcc499958864df5301e940bca0a56f58fddf30ffc07777 environment: POSTGRES_DB: financeai POSTGRES_USER: ${POSTGRES_USER:-financeai} @@ -107,7 +186,7 @@ services: - "5432" restart: unless-stopped healthcheck: - test: ["CMD-SHELL", "pg_isready -U financeai"] + test: ["CMD-SHELL", "pg_isready -U \"$$POSTGRES_USER\" -d \"$$POSTGRES_DB\""] interval: 10s timeout: 5s retries: 5 diff --git a/docker-compose.yml b/docker-compose.yml index 3fb1594..1717492 100644 --- a/docker-compose.yml +++ b/docker-compose.yml @@ -1,5 +1,3 @@ -version: "3.8" - services: # ── Backend API ────────────────────────────────────────── backend: @@ -7,7 +5,7 @@ services: context: . dockerfile: backend/Dockerfile ports: - - "8000:8000" + - "127.0.0.1:8000:8000" volumes: - ./backend:/app/backend env_file: @@ -18,11 +16,33 @@ services: - ollama restart: unless-stopped + # ── Excel add-in development server ────────────────────── + plugin: + image: node:22.23.2-alpine3.24@sha256:c610fcdfb1d5b4740dd70c284ed3cb16bb857e0f7166196e36a5501df7a3aa32 + working_dir: /app/plugin + command: sh -c "npm ci && npm run dev -- --host 0.0.0.0" + ports: + - "127.0.0.1:3443:3443" + volumes: + - ./plugin:/app/plugin + - plugin_node_modules:/app/plugin/node_modules + environment: + - VITE_DEV_BACKEND_TARGET=http://backend:8000 + depends_on: + - backend + restart: unless-stopped + healthcheck: + test: ["CMD-SHELL", "wget --no-check-certificate --spider -q https://localhost:3443/src/taskpane/taskpane.html"] + interval: 15s + timeout: 5s + retries: 10 + start_period: 30s + # ── Ollama (Local LLM) ────────────────────────────────── ollama: image: ollama/ollama:latest ports: - - "11434:11434" + - "127.0.0.1:11434:11434" volumes: - ollama_data:/root/.ollama deploy: @@ -36,3 +56,4 @@ services: volumes: ollama_data: + plugin_node_modules: diff --git a/docs/API_COOKBOOK.md b/docs/API_COOKBOOK.md new file mode 100644 index 0000000..fa30279 --- /dev/null +++ b/docs/API_COOKBOOK.md @@ -0,0 +1,99 @@ +# Generated OpenAPI cookbook + +This cookbook uses the FastAPI application as the contract source. It does not +paste request/response schemas into prose. Inspect the current operation or +component first, and keep the generated taskpane types checked: + +```sh docs-check=syntax +.venv/bin/python examples/openapi_inspect.py --list +.venv/bin/python examples/openapi_inspect.py --operation POST /api/v1/tasks/plan +.venv/bin/python scripts/generate_openapi_types.py --check +``` + +`examples/openapi_inspect.py` calls `backend.main.app.openapi()` directly. +`plugin/src/taskpane/api-types.ts` is generated by +`scripts/generate_openapi_types.py`; neither file should be hand-synchronized +from examples in this document. + +## Discover supported workbook operations + + + + +With the local server from [getting started](GETTING_STARTED.md): + +```sh docs-check=syntax +API_ORIGIN=http://127.0.0.1:8000 +API_KEY=replace-with-your-local-key +curl --fail --silent --show-error \ + -H "X-API-Key: $API_KEY" \ + "$API_ORIGIN/api/v1/contracts/operation-registry" +curl --fail --silent --show-error \ + -H "X-API-Key: $API_KEY" \ + "$API_ORIGIN/api/v1/capabilities/office" +``` + +The registry describes backend validation support; the capability response is +the host-aware boundary. Neither grants a model permission to write Excel. + +## Validate a V2 plan without touching Excel + + + +Inspect both the operation and every referenced component before constructing +the request: + +```sh docs-check=syntax +.venv/bin/python examples/openapi_inspect.py \ + --operation POST /api/v1/contracts/operation-plan-v2/validate +.venv/bin/python examples/openapi_inspect.py --schema OperationPlanV2 +``` + +The endpoint validates and echoes a plan; it does not mutate a workbook. Keep +payloads in local scratch files outside Git when they contain workbook data. + +## Plan, approve, apply, and acknowledge + + + + + +The supported authority sequence is: + +1. `tasks/plan` proposes typed operations. +2. The user reviews the preview. +3. `tasks/apply` records approval and issues the bound apply handoff. +4. Office.js performs the live Excel mutation. +5. `tasks/apply/result` submits the exact client result and post-write proof. + +Do not mark a plan applied after step 3. Durable success follows only from the +verified result acknowledgement in step 5. Generate each operation schema at +the candidate SHA before integration; do not infer it from this sequence. + +## Timeline, rollback, and rollback proof + + + + + +Read the timeline before requesting rollback. The rollback endpoint returns a +bound restore handoff; Office.js remains the live mutation authority, and the +result endpoint verifies the post-restore acknowledgement. An HTTP success from +the rollback request alone is not proof that Excel was restored. + +## Delete a session + + + +Deletion is a lifecycle operation, not a blind database row delete. The +coordinator creates a durable purge fence, removes registered session-owned +state, verifies the inventories, and returns a deletion receipt. See the +[deletion authority diagram](ARCHITECTURE.md#deletion-and-retention-authority) +and [production recovery procedures](PRODUCTION_RUNBOOK.md). + +## Experimental API surfaces + +Routes under `/api/v1/experiments/*` remain explicit opt-in behavior. Their +presence in OpenAPI does not promote V7 to the default planner and does not +make deterministic replay synthetic production evidence. The definitive +boundary is the [implementation status ledger](IMPLEMENTATION_STATUS.md). diff --git a/docs/ARCHITECTURE.md b/docs/ARCHITECTURE.md new file mode 100644 index 0000000..874ec57 --- /dev/null +++ b/docs/ARCHITECTURE.md @@ -0,0 +1,99 @@ +# Architecture and authority + +These diagrams describe the implemented `0.5.0` private-beta candidate. Solid +paths are implemented repository behavior. vNext services remain **Designed** +unless the [status ledger](IMPLEMENTATION_STATUS.md) says otherwise. + +## Runtime architecture + +```mermaid +flowchart LR + User["User"] --> Client["Excel taskpane or browser spreadsheet"] + Client -->|"short-lived user bearer"| Edge["HTTPS edge"] + Edge -->|"same bearer + edge mTLS"| API["FastAPI policy and orchestration"] + API --> Store["SQLite (optional SQLCipher) or PostgreSQL state"] + API --> Worker["Bounded workbook worker"] + API --> LocalModel["Approved local model profile"] + API -. "explicit policy-gated egress only" .-> CloudModel["Optional cloud model"] + Client -->|"approved operations only"| Host["Office.js / browser workbook authority"] +``` + +The edge transport identity never replaces the user's identity. The backend +proposes and verifies operations; it does not directly mutate a live Excel +workbook. When SQLite encryption is enabled, the primary store requires a real +SQLCipher driver and an exact 256-bit raw key and refuses plaintext fallback. +PostgreSQL encryption at rest is deployment infrastructure and remains an +external-attestation requirement. + +## Workbook mutation authority + +```mermaid +sequenceDiagram + participant U as User + participant C as Taskpane + participant A as FastAPI + participant X as Excel via Office.js + U->>C: Request and review + C->>A: Create typed plan + A-->>C: Preview plus risks + U->>C: Explicit approval + C->>A: Approve plan + A-->>C: Bound apply token and operations + C->>X: Atomic host mutation + X-->>C: Result plus post-write reread + C->>A: Apply-result acknowledgement + A->>A: Verify token, targets, and proof + A-->>C: Applied or fail-closed state +``` + +The same split applies to rollback: the backend authorizes bound restore +operations, Office.js executes them, and durable rollback completes only after +verified client proof. Browser spreadsheet writes use its explicit local +transaction authority, not a hidden second backend writer. + +## Deletion and retention authority + +```mermaid +flowchart TD + Request["Owner/admin deletion request"] --> Fence["Durable purging tombstone"] + Fence --> Block["Fence new reads, writes, workflows, and resurrection"] + Block --> Inventories["Enumerate registered session-owned stores"] + Inventories --> Purge["Purge DB rows, artifacts, caches, workflows, citations, diffs"] + Purge --> Verify{"All registered inventories empty?"} + Verify -->|"No"| Pending["Remain purging / fail closed"] + Verify -->|"Yes"| Receipt["Canonical verified deletion receipt"] + Receipt --> Terminal["Terminal deleted state"] +``` + +A receipt cannot replace a missing durable tombstone, and a background task +cannot recreate state across the purge fence. Retention uses the same lifecycle +coordinator rather than a separate best-effort delete. + +## Verification and promotion authority + +```mermaid +flowchart LR + Local["Local deterministic gates"] --> SelfHosted["Ephemeral self-hosted runner"] + SelfHosted --> Receipt["SHA-bound verification receipt"] + External["Native host, workbook/collaboration corpus, production, legal, security evidence"] --> Evidence["Signed release-evidence index"] + Receipt --> Gate{"All exact-SHA gates genuine?"} + Evidence --> Gate + Gate -->|"No"| Blocked["Promotion blocked"] + Gate -->|"Yes"| Promotion["Dual release/security-signed promotion bundle"] + Promotion --> Distribution["Separately signed offline distribution"] + Distribution --> Rehearsal["Offline verify and transactional rehearsal"] +``` + +Repository tests cannot manufacture the `External` branch. The offline bundle +contains the Office manifest and container archives; it is not an MSI, macOS +pkg, native desktop app, updater, or substitute for platform signing and +notarization evidence. The signed 100-schedule collaboration execution receipt +proves only the declared private schedule run; it does not turn the +experimental vNext collaboration contracts and core into a deployed service. + +Operational detail: + +- [Self-hosted verification](SELF_HOSTED_VERIFICATION.md) +- [Release promotion](RELEASE_PROMOTION.md) +- [Offline distribution](OFFLINE_DISTRIBUTION.md) +- [Release blockers](RELEASE_CHECKLIST.md) diff --git a/docs/EXTERNAL_CODEBASE_RESEARCH.md b/docs/EXTERNAL_CODEBASE_RESEARCH.md index eb9f5b3..3d86b16 100644 --- a/docs/EXTERNAL_CODEBASE_RESEARCH.md +++ b/docs/EXTERNAL_CODEBASE_RESEARCH.md @@ -1,5 +1,10 @@ # External Codebase Utilization Notes +> **Historical status:** Retained research and licensing context, not a claim +> that a referenced project is shipped, authoritative, or approved for +> production. Use the [implementation status ledger](IMPLEMENTATION_STATUS.md) +> and [roadmap index](ROADMAP_INDEX.md) for current boundaries. + This project keeps the current React + Office.js taskpane and FastAPI backend. The researched codebases are used as reference implementations and selective service-boundary inspiration, not wholesale replacements. @@ -22,8 +27,8 @@ service-boundary inspiration, not wholesale replacements. ## License Guardrails -- HyperFormula is GPL/commercial and is now approved for direct worker - integration with explicit license configuration. +- HyperFormula is GPL/commercial. The proprietary release keeps it disabled + unless a separately approved commercial key is injected explicitly. - formulas is EUPL and remains research-only until legal review. - xlcalculator is useful for tests/research, but its tokenizer/parser license caveat must be reviewed before embedding. diff --git a/docs/FLAGSHIP_V5_IMPLEMENTATION_PLAN.md b/docs/FLAGSHIP_V5_IMPLEMENTATION_PLAN.md index 2f62287..9ca64bd 100644 --- a/docs/FLAGSHIP_V5_IMPLEMENTATION_PLAN.md +++ b/docs/FLAGSHIP_V5_IMPLEMENTATION_PLAN.md @@ -1,5 +1,11 @@ # Flagship V5 Implementation Plan +> **Historical status:** Archived planning context. Baseline counts, +> implementation gaps, and formula-engine assumptions in this document are not +> current status or release evidence. Use the +> [implementation status ledger](IMPLEMENTATION_STATUS.md) and +> [roadmap index](ROADMAP_INDEX.md). + This plan turns the current verified V4 foundation into a workbook-native, accuracy-first finance copilot. It preserves V2/V3 compatibility, keeps V4/V5 behind flags until benchmark-proven, and prioritizes functionality, accuracy, diff --git a/docs/GETTING_STARTED.md b/docs/GETTING_STARTED.md new file mode 100644 index 0000000..3e2be2c --- /dev/null +++ b/docs/GETTING_STARTED.md @@ -0,0 +1,68 @@ +# Getting started locally + +This path gives a fresh contributor a useful, no-cloud result in under 15 +minutes on a host that satisfies `toolchain.lock.json` (or has restored the +verified dependency cache). It exercises the current supported `0.5.0` API +with disposable SQLite state. It does not start vNext services, call a cloud +model, sideload Excel, or claim native-host release evidence. + +## Minute 0–2: inspect the locked toolchain + +From a new clone, run the bootstrap wrapper. It delegates version policy to +`toolchain.lock.json` and the toolchain doctor rather than duplicating version +numbers in this guide. + +```sh docs-check=syntax +./scripts/bootstrap-contributor.sh --profile fast +``` + +If the host is air-gapped, verify and restore the dependency cache described in +[zero-hosted-minutes operations](ZERO_HOSTED_MINUTES_OPERATIONS.md) first. The +bootstrap accepts `--dependency-cache /approved/cache` to verify that cache; it +does not silently install or fetch tools. + +## Minute 2–8: create the backend environment + +```sh docs-check=syntax +python3 -m venv .venv +. .venv/bin/activate +python -m pip install --require-hashes -r backend/requirements.lock.txt +``` + +Use the checked-in lock exactly. Do not replace it with an unpinned install. +The taskpane and workbook worker have separate `npm ci` steps; they are not +needed for this API-only first result. + +## Minute 8–10: run the disposable no-cloud example + +```sh docs-check=syntax +EXCEL_AGENT_PYTHON=.venv/bin/python ./examples/no-cloud-local-quickstart.sh +``` + +The example creates a temporary data directory, binds Uvicorn to loopback, +enables a disposable API key, explicitly selects local Ollama routing with no +cloud credentials, calls liveness and the supported operation-registry API, +checks that operation against generated OpenAPI, then stops the server and +removes its database. Ollama does not need to be installed for these +non-inference calls. + +## Minute 10–15: get the first repository signal + +```sh docs-check=syntax +./scripts/verify-local.sh --profile fast +``` + +This command runs locally or on a self-hosted runner; it never dispatches a +GitHub-hosted job. The duration target assumes locked dependencies are already +present or restored from the verified cache. If it fails, keep the first +failure intact and follow the reported prerequisite instead of skipping gates. + +Next steps: + +- [API cookbook](API_COOKBOOK.md) for current protocol discovery and + plan/apply/rollback sequencing. +- [Architecture](ARCHITECTURE.md) before changing workbook authority or + deletion behavior. +- [Contributing](../CONTRIBUTING.md) before opening a change. +- [Operations index](OPERATIONS_INDEX.md) for admin, recovery, and air-gap + procedures. diff --git a/docs/IMPLEMENTATION_STATUS.md b/docs/IMPLEMENTATION_STATUS.md new file mode 100644 index 0000000..8d1d4e6 --- /dev/null +++ b/docs/IMPLEMENTATION_STATUS.md @@ -0,0 +1,96 @@ +# Implementation status + +Current repository product version: `0.5.0`. + +This ledger is the concise source of truth for what Excel Agent currently is. +It deliberately distinguishes repository implementation from external release +evidence. A green unit test, workflow, or container build does not substitute +for a signed native-host, private-corpus, production, legal, or security +attestation. + +Status vocabulary: + +- **Implemented** — production code and repository-side verification exist. +- **Experimental** — callable only through an explicit opt-in and not part of + the supported private-beta default. +- **Designed** — contracts or architecture exist, but the end-to-end runtime is + intentionally unavailable. +- **Blocked** — promotion is fail-closed pending genuine external evidence. + +## Supported private-beta candidate (`0.5.x`) + +| Surface | Status | Authority and boundary | +|---|---|---| +| FastAPI orchestration, typed plans, apply/rollback proof | **Implemented** | Backend policy plus exact Office pre/post proof; Office is the workbook mutation authority. | +| Excel taskpane and browser workbook workspace | **Implemented** | Explicit review/apply, owner-scoped sessions, generated API contracts, and revision/fingerprint-bound undo/redo recovery committed atomically with the browser workbook and sync journal; no direct model mutation. | +| Workbook worker and bounded document parsing | **Implemented** | Process/resource boundaries; public parser rejection fails closed. | +| OIDC PKCE, mTLS edge, API-key break glass, local pairing | **Implemented** | Per-user bearer identity is preserved end to end; local pairing is loopback-only and restart-revoked. | +| PostgreSQL/SQLite lifecycle, signed deletion receipts, workflow cleanup | **Implemented** | Durable tombstones fence reads/writes; deletion is verified only after all registered inventories are empty. | +| Primary SQLite SQLCipher provider | **Implemented** | `DB_ENCRYPT=true` requires a verified SQLCipher driver and exactly 32 raw key bytes encoded as 64 lowercase hex characters; invalid/unavailable encryption never falls back to plaintext. PostgreSQL at-rest encryption remains operator/external evidence. | +| Enterprise encrypted managed-object substrate | **Experimental** | A disabled-by-default, explicitly constructed local encrypted object store has scoped content addressing, SQLite/PostgreSQL metadata parity, lifecycle purge hooks, crash recovery, and resumable key rotation. It is not wired into the private-beta workbook path and makes no S3/KMS/cloud or live-PostgreSQL evidence claim. | +| Self-hosted/local verification plane | **Implemented** | No GitHub-hosted runner labels; exact command graph and signed evidence hooks remain SHA-bound. | +| Private collaboration schedule/receipt validation | **Implemented** | Release-scale inventory requires 100 schedules; detached Ed25519 execution receipts bind exact state vectors, conflicts, acknowledgements, authority modes, fences, runner hashes, and deterministic replay. This is validation tooling, not a production collaboration service. | +| Production add-in/container distribution | **Implemented** | CSP, same-origin API/WS, non-root images, physical HyperFormula omission in the standard build. | +| Authenticated offline distribution tooling | **Implemented** | Deterministic Office/container/SBOM/provenance bundle, separate external packaging key, offline verification and transactional rehearsal; no MSI/pkg or signing/notarization claim. | +| Native-host certification tooling | **Implemented** | Exact Windows 54-row, Mac 44-row, and Web 35-row pending scaffolds plus closed Ed25519-authenticated pack verification exist; no genuine host pack is checked in or implied. | +| `V7` accuracy workbench | **Experimental** | Read-mostly opt-in analysis/replay surfaces; the shipped planner default remains V2. | +| `0.5.0` tag/release | **Blocked** | Every exact-SHA row in `docs/RELEASE_CHECKLIST.md` must be independently supplied and signed. | + +## vNext clean-break program + +| Surface | Status | Current limit | +|---|---|---| +| Protobuf v2 contracts and generated Rust/TypeScript/Python bindings | **Implemented** | Breaking checks are pinned to the frozen baseline; this does not imply server availability. | +| One-way v0.5 migration intake and neutral-object materialization | **Implemented** | Validated stages can be consumed idempotently into lifecycle-owned neutral bytes/receipts with SQLite/PostgreSQL parity, then copied by an operator into bounded, encrypted, tenant/owner/session-scoped managed objects with a manifest-last publication fence. This remains inert neutral data: domain workbook/document/CRDT/workflow registration is unavailable, and no reverse conversion or hidden dual write exists. | +| Rust workbook transaction and identity core | **Experimental** | Sparse atomic transactions now include stable row/column/cell identities, bounded structural edits, deterministic snapshots, restart-safe idempotency, and explicit capability/export gates. It is a library slice, not a deployed workbook service. | +| Bounded native/WASM OPC package core | **Experimental** | ZIP, part-identity, relationship, content-type, XML, and archive-inventory validation plus exact no-edit passthrough and owned-part preservation proofs exist. It is a package foundation, not Excel certification. | +| Bounded native/WASM XLSX semantic codec | **Experimental** | Reviewed SpreadsheetML semantics, exact no-edit bytes, constrained preconditioned edits to existing cells, opaque-part equality proofs, signed-VBA/package-signature and calculation-chain fences, and deterministic restart-safe idempotency exist. Export authority is exact-package/exact-operation scoped; no native Excel or production certification is claimed. | +| Transport-neutral CRDT collaboration core | **Experimental** | Bounded Yrs updates, actor-scoped undo, conflict journaling, replay-equivalent compaction, snapshot recovery, authority epochs, and fenced bridge leases exist as a native/WASM library. There is no network service or Excel writer. | +| Transport-neutral fenced Excel bridge core | **Experimental** | Lease/fencing, checkpointed authority epochs, read/hash/preflight/write/reread/ACK proofs, exact echo suppression, crash recovery, and validator-shaped deterministic schedule receipts exist as a native/WASM library. There is no Office API adapter, host certification, or deployed bridge service. | +| Transport-neutral AI run core | **Experimental** | Tenant/workbook/actor-scoped append-only events, CAS transitions, snapshots, bounded cursors, typed tool/apply events, terminal fences, and digest-bound one-use egress approval exist as a native/WASM library. There is no model provider, network server, or Office host implementation. | +| TypeScript and Python vNext SDK adapters | **Experimental** | Transport-neutral clients cover every current Workbook, Run, Collaboration, and Formula RPC with absolute deadlines, cancellation, bounded safe/idempotent retries, sanitized ambiguity, metadata limits, and non-replayed streams. Callers must supply authentication, serialization, endpoint, and wire transports; the adapters are not deployed services. | +| Native local vNext service host | **Experimental** | A native-only local POSIX persistence boundary provides five core recovery adapters, exact-generation compare-and-set transactions, authenticated canonical snapshots, and restart checks. It has no network listener, provider, Office adapter, remote-filesystem certification, external monotonic anchor, or production deployment claim. | +| Workbook/Run/Collaboration/Formula network services | **Designed** | Additive contracts and selected transport-neutral cores exist; production service implementations and transport exposure remain unavailable. | +| Transport-neutral formula core | **Experimental** | Lossless syntax/reference handling, stable-ID transforms, bounded dependency invalidation, canonical recovery, and a tiny reviewed evaluator exist as native/WASM library code. Authority receipts are exact-formula and exact-binding scoped; no Rust function family is production calculation authority yet. | +| Production OOXML export certification and Rust calculation-authority promotion | **Blocked** | The experimental codec permits only exact proved exports; commit-matched native Excel/corpus evidence and production certification remain unavailable. Calculation promotion is function-family and exact-binding specific, never wholesale. | + +## External promotion gates + +The strict verifier must remain red until the exact candidate SHA has all of +the following authentic evidence: + +1. Windows Excel manual matrix and artifact validation, plus the declared + Mac/Web host matrix where applicable. +2. Privacy-approved 250-workbook execution and 100-schedule private + collaboration execution, each with its exact detached signed receipt, plus + anonymized production-like replay. Collaboration receipt validation does + not claim the collaboration service is shipped. +3. Live PostgreSQL migration/backfill/backup/restore/rollback and at-rest + encryption evidence. Repository-proven SQLCipher posture applies only to + primary SQLite when explicitly enabled. +4. Target OIDC, mTLS, DNS/TLS, and on-prem topology smoke evidence. +5. Image vulnerability scan, artifact signatures, provenance, SBOM, and + licensing/legal approvals. +6. Independent security disposition, release owner approval, and exact-SHA + signed evidence index. + +Missing evidence is a release blocker, not a reason to substitute synthetic +fixtures, local mock-host results, or a successful pull-request merge. + +## Explicit non-goals + +- No GA, Marketplace, native MSI/pkg installer, desktop updater, or + completed-vNext claim before the corresponding signed host/distribution + evidence exists. The offline bundle's load/rehearsal tooling is not a native + application installer. +- No model-direct workbook mutation, implicit cloud egress, or silent fallback + from a bounded parser/worker to an unbounded implementation. +- No hidden legacy compatibility inside vNext and no dual workbook authority. +- No private workbook, production replay, credential, signing key, or customer + data in Git. +- No dependence on GitHub-hosted runner minutes. + +The [documentation map](README.md) routes current guidance. Detailed sequence +and acceptance criteria live in the +[program completion roadmap](PROGRAM_COMPLETION_ROADMAP.md); current release +blockers live in the [release checklist](RELEASE_CHECKLIST.md). diff --git a/docs/NATIVE_HOST_CERTIFICATION.md b/docs/NATIVE_HOST_CERTIFICATION.md new file mode 100644 index 0000000..3932f18 --- /dev/null +++ b/docs/NATIVE_HOST_CERTIFICATION.md @@ -0,0 +1,189 @@ +# Native-host certification + +**Implemented:** the repository contains a cross-platform, offline scaffold, +builder, and fail-closed verifier for native Excel host evidence. + +**Blocked:** no Windows Excel Desktop, Mac Excel Desktop, or Excel Web +certification pack is checked in or claimed. A scaffold contains pending slots +only. Release eligibility remains blocked until genuine observations and +portable artifacts from each declared host are independently reviewed, signed, +and retained outside the repository. + +This procedure never logs in to Microsoft 365, drives Excel, obtains +credentials, publishes a status, or uses GitHub-hosted minutes. Operators run +the tests on approved hosts and supply the evidence. The tooling validates and +authenticates what they collected; it cannot make an invented observation true. + +## Coverage contract + +The versioned profile source is +[`templates/native-host-certification/host-profiles.json`](../templates/native-host-certification/host-profiles.json). +Unknown profiles, fields, rows, assertions, or artifact kinds fail closed. + +| Host profile | Row contract | Rows | Current evidence state | +|---|---|---:|---| +| `windows-excel-desktop` | `windows-excel-smoke-54-v1` | 54 | **Blocked — pending external host run** | +| `mac-excel-desktop` | `mac-excel-desktop-44-v1` | 44 | **Blocked — pending external host run** | +| `excel-web` | `excel-web-35-v1` | 35 | **Blocked — pending external host run** | + +The Windows row titles are compared at runtime with the existing 54-row +[Windows Excel smoke contract](WINDOWS_EXCEL_MANUAL_SMOKE.md); the new harness +cannot silently weaken or replace that checklist. Every completed collection +must bind: + +- the exact 40-character release SHA; +- host, Office, and browser/WebView build identifiers; +- the production manifest digest, detached-signature digest, and prior + signature-verification result; +- the add-in digest plus exact backend and edge OCI image digests; +- the runner and executor SHA-256 hashes; +- distinct reviewer and witness role hashes with an independent-witness + assertion; +- every profile row, its required assertions, and symmetric artifact links; +- screenshot, log, and output-workbook artifacts with privacy review; and +- global no-repair-dialog, recalculation, save, coauthoring, and recovery + assertions. + +## Trust and artifact boundary + +Keep collections, raw artifacts, private signing keys, trusted public keys, and +finished packs outside the source checkout. The builder requires a clean +checkout at the exact candidate SHA. It rejects pre-existing or overlapping +destinations, symlinks in any artifact path, traversal, case collisions, +unreferenced or missing artifacts, secret/private-path patterns, malformed +screenshots, unsafe workbook packages, and noncanonical collection JSON. + +The builder copies stable file-descriptor reads into a new staging directory, +rehashes every artifact, writes a canonical manifest, signs its exact bytes +with an external Ed25519 key, verifies the completed pack, and atomically +publishes it. Any failure removes the partial output. A finished directory is a +closed inventory containing only: + +```text +certification.json +certification.ed25519 +artifacts/. +``` + +The verifier authenticates the detached signature before parsing the manifest, +requires an out-of-band key ID and public key, checks the exact SHA/profile and +trusted template digest, scans the closed portable inventory, and rechecks +every artifact digest, size, type, row link, and privacy assertion. It reads the +tree a second time to detect changes during verification. Its JSON report is a +verification receipt, not a deployment, release, Marketplace, or GA claim. + +Binary screenshots and workbooks still require human privacy review. Do not +collect customer identifiers, credentials, private paths, access tokens, or +unredacted tenant data. Use disposable synthetic workbooks or an approved, +sanitized corpus; keep the raw private source behind the evidence boundary. + +## 1. Create pending slots + +Start from a clean exact-SHA checkout. Create the scaffold outside the +repository; it is deliberately unsigned, has no passing row, and reports +`releaseEvidenceEligible: false`. + +```sh docs-check=syntax +EXACT_RELEASE_SHA=$(git rev-parse HEAD) +scripts/native-host-certification.sh scaffold \ + --output "/approved/pending/windows-$EXACT_RELEASE_SHA" \ + --release-sha "$EXACT_RELEASE_SHA" \ + --host windows-excel-desktop \ + --json +``` + +Windows PowerShell operators use the same Python contract: + +```powershell +$ExactReleaseSha = git rev-parse HEAD +.\scripts\native-host-certification.ps1 scaffold ` + --output "D:\approved\pending\windows-$ExactReleaseSha" ` + --release-sha $ExactReleaseSha ` + --host windows-excel-desktop ` + --json +``` + +Repeat with `mac-excel-desktop` and `excel-web`. Do not rename +`pending-collection.json` to look like a completed collection. It is a work +list, not an accepted evidence schema. + +## 2. Collect and independently review + +On the declared host, execute every row in order against the signed production +manifest and exact candidate artifacts. Record the host/build identities, +assertions, and reviewer/witness role hashes in a canonical completed +collection using schema `excel-agent-native-host-collection-v1`. Store its +artifacts under one external root with portable relative paths. Each row must +reference at least its profile-required artifact kinds, and every artifact must +link back to exactly the rows it proves. + +The collection status `passed` is an operator assertion. It is admissible only +after the actual host action succeeded and the referenced artifacts were +privacy reviewed. The harness supplies no command that turns pending rows into +passing rows. + +## 3. Build the authenticated pack + +Use a private Ed25519 key stored outside both the repository and evidence +roots. The default build path refuses a dirty checkout or a HEAD different from +`--expected-sha`. + +```sh docs-check=syntax +scripts/native-host-certification.sh build \ + --collection "/approved/collections/windows-$EXACT_RELEASE_SHA.json" \ + --artifact-root "/approved/raw/windows-$EXACT_RELEASE_SHA" \ + --output "/approved/packs/windows-$EXACT_RELEASE_SHA" \ + --expected-sha "$EXACT_RELEASE_SHA" \ + --host windows-excel-desktop \ + --signing-private-key "/secure/keys/native-host-ed25519.pem" \ + --signing-key-id native-host-2026-08 \ + --json +``` + +Build one separately signed pack per host. Never reuse a Windows observation as +Mac or Web evidence, and never put the private key into a pack. + +## 4. Verify after air-gapped transfer + +Transfer the closed pack and template through the approved artifact boundary. +Supply the public key and expected key ID out of band, then verify on a separate +offline machine: + +```sh docs-check=syntax +scripts/native-host-certification.sh verify \ + --certification "/transferred/packs/windows-$EXACT_RELEASE_SHA" \ + --expected-sha "$EXACT_RELEASE_SHA" \ + --host windows-excel-desktop \ + --trusted-public-key "/secure/trust/native-host-ed25519-public.pem" \ + --trusted-key-id native-host-2026-08 \ + --json +``` + +On Windows: + +```powershell +.\scripts\native-host-certification.ps1 verify ` + --certification "D:\transferred\packs\windows-$ExactReleaseSha" ` + --expected-sha $ExactReleaseSha ` + --host windows-excel-desktop ` + --trusted-public-key "D:\secure\trust\native-host-ed25519-public.pem" ` + --trusted-key-id native-host-2026-08 ` + --json +``` + +A nonzero exit, missing host pack, pending row, privacy failure, changed byte, +unexpected path, key mismatch, stale SHA, or incomplete assertion keeps release +promotion blocked. Do not repair, regenerate, or relabel a failed pack in place; +repeat collection on the real host into a new destination and preserve the +failed material for review according to retention policy. + +## Relationship to release evidence + +Certification packs are host-specific inputs to the signed +[release evidence index](RELEASE_CHECKLIST.md#external-evidence-from-this-sha). +They do not replace the private-corpus execution receipt, production replay, +PostgreSQL rehearsal, self-hosted verification receipt, penetration test, +owner sign-offs, promotion bundle, or offline distribution verification. +Follow the [production runbook](PRODUCTION_RUNBOOK.md) for recovery and release +sequencing and [self-hosted verification](SELF_HOSTED_VERIFICATION.md) for the +zero-hosted-minutes command graph. diff --git a/docs/OFFLINE_DISTRIBUTION.md b/docs/OFFLINE_DISTRIBUTION.md new file mode 100644 index 0000000..2606eb0 --- /dev/null +++ b/docs/OFFLINE_DISTRIBUTION.md @@ -0,0 +1,204 @@ +# Authenticated offline distribution and installation + +The offline distribution is a deterministic packaging layer over an already +approved release-promotion bundle. It does not replace release/security +approval, self-hosted verification, release evidence, image provenance, or the +SBOM. It independently re-verifies those inputs, derives a closed install plan, +and signs the exact distribution inventory with a separate Ed25519 packaging +key. + +No GitHub-hosted runner minutes or online installer service are involved. + +## Trust model + +The distribution has three distinct trust roles: + +1. the release and security approval keys authenticate the canonical promotion + manifest; +2. the verification and release-evidence keys authenticate the promotion's + retained inner evidence; and +3. the distribution key authenticates the derived install plan, copied + promotion bytes, tooling, and checksum inventory. + +All five public keys are supplied out of band. No trust key is embedded in the +distribution, and the distribution private key must be outside the repository, +promotion bundle, and output. Key IDs for release, security, and distribution +must be distinct; the distribution public key must also differ from both +approval keys. + +`packaging/offline-distribution-contract.json` is the machine-readable format +and policy summary. + +## Assemble on the exact release checkout + +The production CLI requires the tooling checkout to be clean and at the same +40-character release SHA authenticated by the promotion. The promotion and +output must be disjoint directories. Image tags and digests come from reviewed +build/provenance evidence, not from a mutable local tag. + +```sh +python scripts/build_offline_distribution.py \ + --promotion-bundle "/approved/promotion/excel-agent-$COMMIT_SHA" \ + --output "/approved/distribution/excel-agent-$COMMIT_SHA" \ + --expected-sha "$COMMIT_SHA" \ + --release-public-key /trusted/release-public.pem \ + --release-key-id release-owner-k1 \ + --security-public-key /trusted/security-public.pem \ + --security-key-id security-owner-k1 \ + --verification-public-key /trusted/verification-public.pem \ + --verification-key-id self-hosted-verification-k1 \ + --release-evidence-public-key /trusted/release-evidence-public.pem \ + --release-evidence-key-id release-evidence-k1 \ + --distribution-private-key /offline-keys/distribution-private.pem \ + --distribution-key-id distribution-owner-k1 \ + --backend-image-tag "excel-agent-backend:$COMMIT_SHA" \ + --backend-image-digest "sha256:$BACKEND_IMAGE_DIGEST" \ + --edge-image-tag "excel-agent-edge:$COMMIT_SHA" \ + --edge-image-digest "sha256:$EDGE_IMAGE_DIGEST" +``` + +The assembler verifies the input promotion before copying, copies through +non-following descriptors, verifies the staged promotion again, validates the +Office XML and its HTTPS runtime locations, and inspects each image archive. +An image archive must contain exactly one tag and image. Docker archives bind +the exact image-config digest; OCI archives bind the exact index manifest +digest and reference annotation. Links, special entries, traversal, case +collisions, extra images, and metadata drift fail closed. + +The output contains: + +- the complete original promotion bundle and its two approval signatures; +- the production Office manifest; +- backend and edge container-image archives; +- the CycloneDX SBOM and authenticated provenance evidence; +- canonical `install-plan.json`, `SHA256SUMS`, and a complete signed byte + inventory; +- offline verifier/installer modules and thin shell/PowerShell wrappers; and +- `distribution-manifest.json` plus its raw Ed25519 signature. + +There are no timestamps or host paths in the signed distribution manifest or +plan. Reassembling identical inputs with the same Ed25519 key produces +identical manifest, signature, plan, and checksum bytes. + +## Verify on the isolated installation host + +Run the shipped verifier or the repository verifier before any rehearsal or +install: + +```sh +python scripts/verify_offline_distribution.py \ + --distribution "/transferred/excel-agent-$COMMIT_SHA" \ + --expected-sha "$COMMIT_SHA" \ + --release-public-key /trusted/release-public.pem \ + --release-key-id release-owner-k1 \ + --security-public-key /trusted/security-public.pem \ + --security-key-id security-owner-k1 \ + --verification-public-key /trusted/verification-public.pem \ + --verification-key-id self-hosted-verification-k1 \ + --release-evidence-public-key /trusted/release-evidence-public.pem \ + --release-evidence-key-id release-evidence-k1 \ + --distribution-public-key /trusted/distribution-public.pem \ + --distribution-key-id distribution-owner-k1 \ + --json +``` + +The verifier authenticates the distribution signature before parsing the +manifest, rejects extra/missing/link/special/traversal paths, hashes every +file, compares `SHA256SUMS`, validates the exact closed install plan, re-runs +the complete dual-signed promotion verification, and re-inspects both image +archives. + +The host needs Python 3.12 and the locked `cryptography` dependency. This is a +toolchain prerequisite, not an invitation to fetch packages during install. +Use the separately verified dependency cache when provisioning an air-gapped +host. + +## Rehearse without mutation + +The install rehearsal verifies and stages every install byte, then discards the +stage. It does not contact or require a container engine and does not create the +destination: + +```sh +./install.sh \ + --install-root /srv/excel-agent \ + --receipt /approved/receipts/install-rehearsal.json \ + --rehearse \ + --expected-sha "$COMMIT_SHA" \ + --release-public-key /trusted/release-public.pem \ + --release-key-id release-owner-k1 \ + --security-public-key /trusted/security-public.pem \ + --security-key-id security-owner-k1 \ + --verification-public-key /trusted/verification-public.pem \ + --release-evidence-public-key /trusted/release-evidence-public.pem \ + --distribution-public-key /trusted/distribution-public.pem \ + --distribution-key-id distribution-owner-k1 +``` + +Use `Install-Windows.ps1` with the same long-form arguments on Windows. Both +wrappers reject administrator/root execution. The install root must already be +an explicit, non-symlink directory and cannot be `/`, the user's home, +distribution directory, or an ancestor/descendant of the distribution. + +## Install and load images + +Omit `--rehearse` and select `--container-engine docker` or +`--container-engine podman`. Docker and OCI archives are loaded only from the +verified local files; there is no pull or update command. OCI-manifest digest +installation requires Podman because the post-load manifest digest must be +observable. Remote Docker/Podman endpoints are rejected. + +The installer: + +1. verifies the complete distribution; +2. copies each planned byte into a sibling stage while rechecking its digest; +3. atomically swaps the destination, retaining any explicit `--replace` + destination as a rollback backup; +4. checks for a pre-existing exact image tag/digest; +5. rehashes each staged archive through one open descriptor, rewinds that same + descriptor, and passes it directly to the local engine; +6. verifies the exact loaded tag/digest; and +7. writes canonical internal and external machine-readable receipts. + +If either load or any later receipt step fails, newly loaded images are removed +when their digest still matches, the new destination is removed, and the prior +destination is restored. A load that mutates engine state and then reports an +error is also detected and rolled back. The failure receipt records whether +rollback completed. + +## Uninstall + +Uninstall uses only the exact release destination and its validated install +receipt: + +```sh +./uninstall.sh \ + --install-root /srv/excel-agent \ + --release-sha "$COMMIT_SHA" \ + --container-engine podman \ + --receipt /approved/receipts/uninstall.json +``` + +It never removes a pre-existing image, refuses to remove a tag whose digest has +changed, and reloads already removed images from their retained verified +archives if a later removal step fails. Repeating uninstall after the +destination is absent succeeds with a signed-scope, machine-readable no-op +receipt. `--rehearse` validates removal eligibility without changing files or +images. + +## Why there is no MSI or pkg + +The retained product has no desktop executable or background desktop service. +Its client release artifact is an Office add-in manifest, and its server +runtime is the backend/edge container pair. The offline installer stages that +manifest for an operator-controlled Microsoft 365 catalog or documented local +sideload and loads the server images; it does not install a native application. + +Creating an MSI or macOS pkg would therefore wrap no retained desktop product, +would introduce a second updater/privilege surface, and could falsely imply +Authenticode signing, Apple Developer ID signing, or notarization evidence. +Those packages are intentionally not release artifacts. This plane does not +claim, synthesize, or substitute code-signing/notarization evidence. If a +native desktop component is introduced later, its packaging, signing, +notarization, update, rollback, and independent verification contracts require +a separate approved release design. diff --git a/docs/OPERATIONS_INDEX.md b/docs/OPERATIONS_INDEX.md new file mode 100644 index 0000000..9fc6116 --- /dev/null +++ b/docs/OPERATIONS_INDEX.md @@ -0,0 +1,23 @@ +# Operations index + +Use this page to choose an operator procedure. Repository-side tooling is +**Implemented**; live environment evidence and release approval remain +**Blocked** until the exact candidate SHA has the required signed artifacts. + +| Need | Procedure | Boundary | +|---|---|---| +| Bootstrap first administrator | [Production runbook](PRODUCTION_RUNBOOK.md#3-start-storage-and-bootstrap-the-first-administrator) | Break-glass key stays offline; interactive users use OIDC PKCE. | +| Deploy or roll back on-prem | [Production runbook](PRODUCTION_RUNBOOK.md) | Requires real TLS, mTLS, OIDC, database, backup, and smoke evidence. | +| Recover or delete state | [Production runbook](PRODUCTION_RUNBOOK.md) and [authority diagram](ARCHITECTURE.md#deletion-and-retention-authority) | Tombstones and verified receipts fence resurrection. | +| Validate without hosted minutes | [Self-hosted verification](SELF_HOSTED_VERIFICATION.md) | Local/self-hosted evidence is SHA-bound; it is not native-host evidence. | +| Bootstrap ephemeral runners or caches | [Zero-hosted-minutes operations](ZERO_HOSTED_MINUTES_OPERATIONS.md) | Clean workspace and verified cache contracts apply. | +| Run Windows Excel manual smoke | [Windows Excel smoke](WINDOWS_EXCEL_MANUAL_SMOKE.md) | Must run on the declared real host against the exact manifest. | +| Validate private execution evidence | [Release checklist](RELEASE_CHECKLIST.md) and [production runbook](PRODUCTION_RUNBOOK.md) | Workbook and 100-schedule collaboration receipts use external keys and exact-SHA inventories; validator success is not a shipped collaboration service. | +| Assemble the approved release bytes | [Release promotion](RELEASE_PROMOTION.md) | Requires distinct release/security approval keys outside the bundle. | +| Transfer and rehearse air-gapped | [Offline distribution](OFFLINE_DISTRIBUTION.md) | Separate external distribution key; no network, pull, updater, MSI, or pkg. | +| Decide whether release may advance | [Release checklist](RELEASE_CHECKLIST.md) | Missing external evidence keeps promotion red. | + +For contributor setup, use [getting started](GETTING_STARTED.md), not the +production runbook. For current feature truth, use the +[implementation status ledger](IMPLEMENTATION_STATUS.md), not a historical +roadmap. diff --git a/docs/PRODUCTION_RUNBOOK.md b/docs/PRODUCTION_RUNBOOK.md new file mode 100644 index 0000000..1e2e8ee --- /dev/null +++ b/docs/PRODUCTION_RUNBOOK.md @@ -0,0 +1,416 @@ +# Production Operations Runbook + +This runbook covers the private on-prem distribution. Every interactive client +authenticates as a real user through OIDC authorization-code plus PKCE; the edge +forwards that short-lived bearer token and reaches the backend through a +mutually authenticated TLS channel. The reserved backend API key is offline +break-glass/bootstrap material, never a browser or add-in credential. External +release evidence and the formula-engine licensing decision remain independent +release gates. + +## 1. Required inputs + +Provision these outside Git before building: + +- a public HTTPS origin in `DEPLOY_ORIGIN`, with no path or trailing slash; +- `infra/nginx/certs/server.crt` and `server.key` for that origin; keep the + private key non-world-readable but readable by the edge container's + unprivileged UID/GID 101 (for example, owner `root:101` and mode `0640` on a + Linux deployment host); +- the edge/backend mTLS CA, edge-client certificate/key, and backend + certificate/key described under `infra/mtls/`; +- OIDC issuer, audience, client ID, JWKS/discovery endpoints, an explicit + tenant allowlist, and a same-origin PKCE callback URI; +- exact space-separated HTTPS origins for every browser-fetched discovery, + token, and JWKS endpoint in `OIDC_BROWSER_CONNECT_ORIGINS`; wildcards, + `https:`, and unrelated IdP origins are prohibited; +- strong, distinct offline `API_KEY`, `API_KEY_HMAC_PEPPER`, and Postgres + credentials; +- a stable `EXCEL_AGENT_LIFECYCLE_RECEIPT_KEY_ID` and a distinct random + 32-byte-or-longer deletion-receipt HMAC key encoded as URL-safe base64 in + `EXCEL_AGENT_LIFECYCLE_RECEIPT_HMAC_KEY`; follow the staged rotation procedure + below rather than swapping the signing key on all replicas at once; +- a pinned Ollama tag or digest in `OLLAMA_IMAGE`; +- explicit `AI_EXTERNAL_SENDING_ALLOWED` and + `AI_EXTERNAL_DOCUMENTS_ALLOWED` decisions; +- provider credentials only for providers allowed by policy; +- `HYPERFORMULA_ENABLED=false` and `HYPERFORMULA_DISTRIBUTED=false` for the + standard proprietary build, which omits the runtime; a custom build may set + both true only after adding the production dependency and recording approval + for an explicitly injected commercial `HYPERFORMULA_LICENSE_KEY`; +- a tested Postgres backup and restore destination. + +Keep `.env` mode `0600` on a trusted deployment host. Never put an API key, +OIDC token, or private key in `VITE_*`, the manifest, static files, image +layers, or source control. + +## 2. Verify the exact source tree + +From the repository root, use the checked-in lockfiles: + +```bash +.venv/bin/python scripts/check_package_integrity.py +.venv/bin/python scripts/check_dependency_risks.py +.venv/bin/python scripts/generate_openapi_types.py --check +cd plugin && npm ci && npm run verify && cd .. +cd workbook-worker && npm ci && npm test && cd .. +docker compose -f docker-compose.prod.yml config +``` + +The release workflow is the authoritative repeatable gate. A successful local +run does not replace the required Windows Excel, private-fixture, anonymized +production replay, or live Postgres evidence. + +This build was also validated against a disposable local Postgres 16 instance, +including all typed projection backfills, first-admin bootstrap, key rotation, +and cross-user ownership isolation. That is regression evidence only; archive a +fresh run against the target staging/production topology for release evidence. + +## 3. Start storage and bootstrap the first administrator + +Start Postgres, the configured local model service, and the backend without the +edge. Backend startup creates or updates its schema and fails closed on unsafe +production configuration. + +```bash +docker compose -f docker-compose.prod.yml up -d postgres ollama backend +docker compose -f docker-compose.prod.yml ps +``` + +Create the first administrator from an offline container command. Mount an +absolute directory outside the repository; the credential file is created once +with mode `0600` and is never overwritten. + +```bash +docker compose -f docker-compose.prod.yml run --rm --no-deps \ + -v /absolute/secure/bootstrap-output:/run/bootstrap \ + backend python -m backend.admin_bootstrap \ + --user-id initial-admin \ + --email admin@example.com \ + --name "Initial Administrator" \ + --output /run/bootstrap/initial-admin.json +``` + +Move the plaintext key into the deployment secret manager as an offline +break-glass credential, then securely remove the one-time file. Do not inject +it at the edge or distribute it to interactive clients. `--rotate` mints a +replacement for the same user and revokes that user's existing active keys; +preserve and test the break-glass procedure before rotating it. + +Databases upgraded from an older build may contain intentionally unowned legacy +sessions. They fail closed to non-admin callers. Define and audit an explicit +administrator reassignment/import policy before those records are exposed to +tenant users. + +### Deletion-receipt key rotation + +Rotate receipt keys across the fleet in three ordered deployments. Let `K1` be +the current key and `K2` the replacement; every replica must finish each phase +and pass a restart/readiness check before the next phase begins. + +1. Pre-distribute `K2` as verification-only: keep `K1` in + `EXCEL_AGENT_LIFECYCLE_RECEIPT_KEY_ID` and + `EXCEL_AGENT_LIFECYCLE_RECEIPT_HMAC_KEY`, and add `K2` to + `EXCEL_AGENT_LIFECYCLE_RECEIPT_PREVIOUS_KEYS`. Despite the variable's legacy + name, this map is the non-signing verification keyring. Confirm every replica + still signs new receipts with `K1` and can verify receipts for both key IDs. +2. Promote `K2`: set it as the current key and retain `K1` in the verification + keyring. Restart every replica and prove that pre-rotation `K1` receipts and + new `K2` receipts both survive restart and remain idempotent. +3. Retire `K1` only after the longest receipt/backup/legal-retention horizon has + elapsed and restored backups no longer contain a `K1` receipt that must be + trusted. Remove it from the verification keyring, restart the complete fleet, + and archive evidence that all retained receipts verify. An unknown key fails + closed; re-add `K1` from the secret manager if a retained receipt is found. + +Never deploy a direct `K1`-to-`K2` swap, and never remove `K1` while a replica +can still sign with it. Preserve both key materials outside the application data +volume for the full overlap and retention windows. + +### Enterprise managed object-store rehearsal (opt-in only) + +The repository includes a disabled-by-default encrypted managed-object +substrate for enterprise integration work. It is not imported by the default +backend, does not replace the local-first workbook path, and must not be called +production storage merely because its local rehearsal succeeds. An enterprise +deployment must explicitly construct the store and register its lifecycle +purger; missing lifecycle/session storage fails closed. + +Provision the `MANAGED_OBJECT_STORE_*` keys and root from an external secret +manager and a dedicated `0700` directory. Keep the address key distinct from +every encryption key. Inspect bounded, identifier-free posture before any +rehearsal: + +```bash +python -m backend.operations.managed_object_store_operator status +python -m backend.operations.managed_object_store_operator rehearse +python -m backend.operations.managed_object_store_operator recover +``` + +Rotation additionally requires a unique `MANAGED_OBJECT_STORE_ROTATION_ID`, +`MANAGED_OBJECT_STORE_NEW_KEY_ID`, and `MANAGED_OBJECT_STORE_NEW_KEY`: + +```bash +python -m backend.operations.managed_object_store_operator rotate +``` + +Retain the previous key until recovery and receipt verification prove that no +live metadata references it. Operator output is privacy-safe local rehearsal +evidence only. A target deployment still requires its own live PostgreSQL, +backup/restore, filesystem durability, key-custody, and lifecycle-deletion +evidence; no cloud object store or KMS is contacted or certified here. + +## 4. Validate Postgres migrations and backfills + +Run the live validation against staging before production. It creates an +isolated validation schema, verifies typed projection backfills and +idempotency, and removes that schema unless `--keep-schema` is supplied. + +```bash +POSTGRES_VALIDATION_DATABASE_URL='postgresql://...' \ + .venv/bin/python scripts/validate_postgres_migration_backfill.py +``` + +Archive the JSON output with the release evidence. Before every production +upgrade, take a database backup and perform a restore drill against a separate +database. Do not attempt to reverse a schema migration in place; restore the +matching database backup when application rollback requires old schema state. + +For the release rehearsal, create a separate empty PostgreSQL 16 database whose +exact name is `excel_agent_rehearsal_`. Then run the provider-free +operator command with two explicit DSNs and the exact lowercase 40-character +release commit. The command applies current migrations twice to prove +idempotency, validates lifecycle and typed-projection backfills, performs a +custom-format `pg_dump`/`pg_restore`, compares canonical counts and logical and +schema digests, checks tombstones, ownership, and encrypted-or-redacted audit posture, +rehearses application boot compatibility without downgrading the schema, and +cleans the disposable target. + +```bash +.venv/bin/python -m backend.operations.operator_cli postgres-release-rehearsal \ + --source-dsn "$POSTGRES_REHEARSAL_SOURCE_DSN" \ + --restore-dsn "$POSTGRES_REHEARSAL_RESTORE_DSN" \ + --target-marker "release_2026_08" \ + --release-sha "$EXACT_RELEASE_SHA" +``` + +The JSON is a release-evidence candidate only when `liveDatabase` is `true` and +every check is `verified`; a configured DSN alone never produces that claim. +The output contains no DSN, filesystem path, row value, tenant, owner, or +session identifier. PostgreSQL and both libpq tools must be major version 16, +and tool, transaction, manifest, drift, timeout, cancellation, and cleanup +failures stop the rehearsal. + +If the target host or database name looks production-like, first generate a +privacy-safe confirmation digest, review the target fingerprint, and then pass +that digest in a separate invocation: + +```bash +.venv/bin/python -m backend.operations.operator_cli postgres-release-rehearsal \ + --authorize-target-only \ + --restore-dsn "$POSTGRES_REHEARSAL_RESTORE_DSN" \ + --target-marker "release_2026_08" \ + --release-sha "$EXACT_RELEASE_SHA" + +.venv/bin/python -m backend.operations.operator_cli postgres-release-rehearsal \ + --source-dsn "$POSTGRES_REHEARSAL_SOURCE_DSN" \ + --restore-dsn "$POSTGRES_REHEARSAL_RESTORE_DSN" \ + --target-marker "release_2026_08" \ + --release-sha "$EXACT_RELEASE_SHA" \ + --target-authorization-digest "$REVIEWED_TARGET_AUTHORIZATION_DIGEST" +``` + +The disposable PG16 integration gate is local and opt-in. It uses only an +already-present `postgres:16-alpine` image, never pulls one, and removes its +container when complete: + +```bash +RUN_POSTGRES_RELEASE_REHEARSAL_DOCKER=1 \ + .venv/bin/pytest -q \ + backend/tests/integration/test_postgres_release_rehearsal_pg16.py +``` + +## 5. Start and verify the edge + +```bash +docker compose -f docker-compose.prod.yml up -d nginx +DEPLOY_ORIGIN=https://excel.example.com \ +SMOKE_OIDC_BEARER_TOKEN="$SHORT_LIVED_TEST_TOKEN" \ +SMOKE_OIDC_CONNECT_ORIGINS="$OIDC_BROWSER_CONNECT_ORIGINS" \ +sh infra/nginx/smoke-production.sh +``` + +Verify all of the following before distributing the manifest: + +- `$DEPLOY_ORIGIN/manifest.xml` contains only the immutable public origin; +- the taskpane, browser spreadsheet, icons, API health route, and WebSocket + upgrade succeed; +- CSP permits Office framing, contains every exact OIDC fetch origin, contains + no broad outbound source, and no `X-Frame-Options` header blocks the add-in; +- static assets and source maps contain no credential; +- an authenticated request is attributed to the intended enterprise identity; +- `/health` reports no pending storage migration. + +Distribute the manifest from the same immutable build that passed these checks. + +## 6. Runtime monitoring and incident response + +Monitor container health, backend request/error latency, worker timeouts and +memory exits, Postgres capacity, AI-provider selection, apply/rollback failure +rates, and Nginx 4xx/5xx rates. Alert on any cloud provider use when external +sending is disabled. + +If credentials may be exposed, rotate the affected enterprise key and provider +keys, restart the edge so its generated configuration uses the replacement, +and audit session/apply logs. If workbook mutation correctness is in doubt, +disable apply access, retain audit and recovery artifacts, and keep the service +read-only until the affected workbook is verified in desktop Excel. + +## 7. Application rollback + +Keep the previously verified edge/backend image tags and the pre-upgrade +database backup. To roll back: + +1. stop new apply traffic; +2. capture logs and the current database state for diagnosis; +3. restore the matching pre-upgrade database into a new database instance when + schema compatibility is uncertain; +4. point the previous backend image at that restored database; +5. deploy the previous edge image and rerun the production smoke check; +6. re-enable mutation only after a desktop Excel verification pass. + +## 8. Release evidence + +The strict gate intentionally fails until genuine evidence is supplied. The +downloaded artifact must have exactly one index and its detached signature at +the artifact root: `release-evidence-index.json` and +`release-evidence-index.ed25519`. The signature must be the raw 64-byte +Ed25519 signature over the exact index bytes. Its trusted PEM public key and +case-sensitive key ID are provisioned to CI out of band; neither may come from +the downloaded artifact or repository. The workflow fails closed when the +`RELEASE_EVIDENCE_ED25519_PUBLIC_KEY_B64` secret or +`RELEASE_EVIDENCE_SIGNING_KEY_ID` variable is absent. + +The index schema is `2026-08-closure-release-evidence-index-v2` and has these +top-level fields: `schemaVersion`, `releaseSha`, `signingKeyId`, and `items` +(with optional `generatedAt` and `producer`). Every item has exactly `id`, +`releaseSha`, `status`, `path`, +`sha256`, `size`, and `details`. The index and every item must use the same exact +lowercase 40-character release SHA; every required item must have +`status: "passed"` and `details.successful: true`. + +The required IDs are: + +- `signed_production_manifest`, `production_manifest_windows`, + `production_manifest_macos`, and `production_manifest_excel_web`; +- `private_corpus_execution`, `private_corpus_inventory`, + `production_replay`, and `parser_corpus`; +- `postgres_migration`, `postgres_backfill`, `postgres_rollback`, and + `postgres_backup_restore`; +- `hyperformula_posture`, `sbom`, `trust_report`, `dependency_review`, and + `container_scan`; +- `container_signature`, `provenance_verification`, and + `independent_pentest`; +- `release_signoff`, with one verified signature each from the release, + security, workbook, client, and legal owners. + +The typed details prove the important disposition rather than treating a file +attachment as success: host reports identify Windows, macOS, or Excel Web and +verify the signed manifest; private-corpus execution records native Excel and +a verified detached execution receipt binding at least 250 workbook digests, +the exact private manifest, Excel build, and release SHA with zero failures; +inventory has zero uninventoried workbooks; replay is marked production evidence; +parser mismatches are zero; Postgres evidence is from a live database and the +restore is verified; container scan High/Critical counts are zero; signatures +and provenance are verified against the release SHA; and the independent test +has zero open Critical/High or undispositioned Medium findings. HyperFormula +must record either the fully disabled/omitted standard posture or an approved, +injected commercial posture, and must attest that the GPL sentinel is absent. + +Only these restore destinations are accepted: +`docs/WINDOWS_EXCEL_MANUAL_SMOKE.md`, `data/sessions.db`, files below +`data/release-trust/`, and files below +`backend/benchmarks/fixtures/private/`. Additional private fixtures and +attachments use unique `attachment.*` IDs. Every regular file in the artifact +must be indexed; absolute paths, traversal, links, case-colliding paths, +unindexed files, digest/size mismatches, stale SHAs, and workflow-output path +collisions block before any file is restored. Restore preflights every +destination, prepares replacements and rollback backups, and commits as one +transaction. A later replacement failure rolls back every earlier replacement +and new file; any rollback failure is surfaced with the retained recovery +backup path. + +Validate an artifact independently before dispatching the strict workflow: + +```bash +.venv/bin/python scripts/validate_release_evidence_index.py \ + --index /absolute/evidence/release-evidence-index.json \ + --signature /absolute/evidence/release-evidence-index.ed25519 \ + --artifact-root /absolute/evidence \ + --release-sha "$(git rev-parse HEAD)" \ + --trusted-public-key /secure/out-of-band/release-evidence-public.pem \ + --trusted-signing-key-id release-evidence-key-2026-08 \ + --json +``` + +Then run the local prerequisite summary against the same immutable artifact: + +```bash +.venv/bin/python scripts/verify_release_evidence_prereqs.py \ + --strict --json \ + --evidence-index /absolute/evidence/release-evidence-index.json \ + --evidence-signature /absolute/evidence/release-evidence-index.ed25519 \ + --evidence-root /absolute/evidence \ + --release-sha "$(git rev-parse HEAD)" \ + --trusted-public-key /secure/out-of-band/release-evidence-public.pem \ + --trusted-signing-key-id release-evidence-key-2026-08 +``` + +Before the private-corpus execution report is admitted to that index, validate +the native host receipt against the local private inventory and an out-of-band +Ed25519 public key: + +```bash +python scripts/validate_private_corpus_execution_receipt.py \ + --receipt /secure/evidence/private-corpus-execution.json \ + --signature /secure/evidence/private-corpus-execution.ed25519 \ + --public-key /secure/keys/excel-corpus-public.pem \ + --key-id excel-corpus-windows-2026-08 \ + --release-sha "$(git rev-parse HEAD)" \ + --private-root backend/benchmarks/fixtures/private +``` + +The public key must not be stored in the repository, corpus, or receipt +artifact root. A bounded worker preflight is useful local validation but always +reports `releaseEvidenceEligible=false`; it is never a native-host substitute. + +Collect and authenticate each host-matrix result using the +[native-host certification procedure](NATIVE_HOST_CERTIFICATION.md). Its +scaffold creates pending slots only. The build command requires a clean +exact-SHA checkout, external artifacts, and an out-of-repository Ed25519 key; +the offline verifier requires the out-of-band public key/key ID and a closed +inventory. Windows, Mac, and Web packs are separate evidence inputs and none +may substitute for another host or for the private-corpus execution receipt. + +A releasable bundle requires a commit-matched Windows Excel smoke artifact, +executed private fixtures, generated fixture inventory and parser-corpus +evidence, a production-like anonymized replay baseline, live Postgres migration +and restore evidence, the HyperFormula posture, supply-chain verification, host +matrix proof, an independent penetration-test disposition, and all five owner +sign-offs. The authenticated index turns its typed details and artifact digests +into signed attestations; it does not make synthetic or fabricated content valid +evidence. + +After every prerequisite above is green, follow +[`docs/RELEASE_PROMOTION.md`](RELEASE_PROMOTION.md) to copy and bind the exact +distributable bytes, the authenticated evidence tree, the signed self-hosted +verification transcript, SBOM, notices, and toolchain manifest into one +offline bundle. Release and security owners sign the same canonical manifest +with distinct keys; neither private key nor any trusted public key is placed in +the bundle. + +Record the resulting canary, rollback, blocker, and promotion disposition with +the offline [promotion ledger and evidence portal](RELEASE_EVIDENCE_PORTAL.md). +Verify its two-person hash chain and the separately signed portal from the +exact-SHA checkout before local review. The portal contains only redacted typed +summaries and is not evidence, a deployment action, or a publication claim. diff --git a/docs/PROGRAM_COMPLETION_ROADMAP.md b/docs/PROGRAM_COMPLETION_ROADMAP.md new file mode 100644 index 0000000..1ab4123 --- /dev/null +++ b/docs/PROGRAM_COMPLETION_ROADMAP.md @@ -0,0 +1,1098 @@ +# Program Completion Roadmap + +> **Roadmap status:** Current `0.5.x` planning context. The +> [implementation status ledger](IMPLEMENTATION_STATUS.md) and +> [release checklist](RELEASE_CHECKLIST.md) override milestone prose when +> implementation or evidence has changed. See the +> [roadmap index](ROADMAP_INDEX.md) for the clean-break vNext plan and archived +> predecessors. + +Last updated: 2026-08-09 + +This roadmap defines the path from the current local-first Excel AI workbook +platform to a completion-grade product. Completion here does not mean another +demo milestone. It means the program can be used for serious workbook work with +trustworthy open, edit, AI plan, sync, export, audit, rollback, security, and +release operations. + +The program has two product surfaces: + +- A browser spreadsheet platform: Shortcut-like, full-page, grid-dominant, + local-first, AI-assisted, and able to open, edit, export, and QA Excel files. +- An Excel-native add-in surface: Office.js-driven, host-aware, workbook-safe, + and able to apply approved AI/manual operations inside real Excel. + +The product should eventually feel like one coherent platform, not a loose +collection of backend planning APIs, taskpane controls, worker commands, and a +separate spreadsheet page. + +## Current Release Posture + +The engineering result is a supported private-beta release candidate. The +automated backend, add-in, browser, worker, dependency, container, migration, +and release-workflow gates are implemented and locally exercised. GA remains +deliberately blocked on evidence or decisions that cannot be manufactured in a +source tree: a commit-matched Windows Excel Desktop run, executed private and +Excel-authored fixtures, an anonymized production replay baseline, archived +staging/production Postgres migration evidence, a HyperFormula disabled-or- +commercially-licensed decision, live OIDC/PKCE and host-matrix evidence, and +signed release artifacts. + +## Research Inputs + +### Local Implementation Snapshot + +The current repo already contains a wide V2.0 foundation: + +- `docs/STACK_CAPABILITY_MATRIX.md` documents the V1 through V2 spreadsheet + platform, V7 accuracy surfaces, worker commands, readiness gates, and known + gaps. +- `plugin/src/spreadsheet/` contains the Shortcut-like spreadsheet page, + `WorkbookSessionV1`, `WorkbookPatchV1`, IndexedDB storage, Excel sync queue, + formula parity, object inspector, trust report, and Univer host adapter. +- `workbook-worker/src/cli.mjs` contains workbook commands including + `open-workbook-v1`, `export-workbook-v1`, and `roundtrip-qa-v2`. +- `backend/api/routes/tasks.py` exposes stateless spreadsheet open, export, and + roundtrip QA routes under `/api/v1/spreadsheet/*`. +- `plugin/test/spreadsheet-model.test.ts` covers patch application, sync + mapping, queue idempotency, formula trace, trust reports, and migration. +- `workbook-worker/test/cli.test.mjs` covers worker open/export/reopen, + object patching, opaque OOXML part preservation, macro blob preservation, and + fixture roundtrip checks. +- `scripts/verify-full-stack.ps1`, `scripts/verify-local-accuracy.ps1`, and + `scripts/verify-v7-default-prep.ps1` provide the start of a release gate. + +### External Reference Constraints + +The roadmap is shaped by current primary sources: + +- Microsoft Excel JavaScript API events are session-bound, can be triggered by + user actions, add-in code, or VBA, and can be temporarily disabled during + batch operations: + https://learn.microsoft.com/en-us/office/dev/add-ins/excel/excel-add-ins-events +- Excel JavaScript API support must be checked through requirement sets at + runtime or through manifest requirements: + https://learn.microsoft.com/javascript/api/requirement-sets/excel/excel-api-requirement-sets +- Shared runtime can coordinate taskpane, ribbon, and custom functions, but it + has client support limits and supports only one task pane: + https://learn.microsoft.com/en-us/office/dev/add-ins/develop/configure-your-add-in-to-use-a-shared-runtime +- Microsoft macro security guidance reinforces that VBA execution must remain + trusted-host delegated, never browser-side: + https://learn.microsoft.com/microsoft-365-apps/security/internet-macros-blocked +- Univer Sheets core provides spreadsheet UI, formula, rendering, ribbon, sheet + bar, and status bar building blocks, but it is not the export fidelity source + of truth: + https://docs.univer.ai/guides/sheets/features/core +- HyperFormula requires an explicit GPLv3 or proprietary license posture before + commercial release: + https://hyperformula.handsontable.com/docs/guide/licensing.html +- Open XML spreadsheet work requires package, part, relationship, and content + type discipline: + https://learn.microsoft.com/office/open-xml/spreadsheet/overview +- Shortcut's public product positioning emphasizes Excel parity, formula-driven + outputs, auditability, restore/undo, native Excel integration, large files, + macros, keyboard shortcuts, enterprise controls, and speed: + https://shortcut.ai/ +- Shortcut's current changelog shows an active bar around team administration, + Excel add-in recovery, calculation modes, macro warnings, checkpoints, + review panels, formula tracing, pivot support, comments, large files, APIs, + file attachments, data connections, and workspace controls: + https://shortcut.ai/changelog +- Microsoft Marketplace describes Shortcut as an Excel add-in for natural + language workflows, PDF import, financial models, dataset cleaning, + dashboards, and Excel-compatible edits: + https://marketplace.microsoft.com/en-us/product/office/WA200009152 + +## Completion Definition + +The program is complete when a serious analyst can: + +1. Open normal and complex `.xlsx` or `.xlsm` files without silent loss. +2. Edit cells, formulas, styles, workbook objects, sheets, comments, tables, + validations, named ranges, charts metadata, and protections through the web + editor or Excel add-in. +3. Ask AI to inspect, plan, build, repair, transform, audit, and explain + workbook work using the same operation rails as manual edits. +4. Preview every material AI/manual change before it mutates the workbook. +5. Recalculate in-browser with clear formula parity warnings, and use Excel as + a host parity oracle when linked. +6. Sync local browser edits into Excel through Office.js with durable queueing, + idempotency, drift detection, and reconciliation. +7. Export a workbook that reopens in Excel with supported edits preserved and + unsupported parts explicitly preserved or blocked. +8. See trust reports that combine formula parity, OOXML fidelity, source + evidence, sync state, unsupported features, and rollback availability. +9. Undo or restore from checkpoints after destructive or AI-generated changes. +10. Work with large workbooks without the UI freezing or pretending it has + analyzed more than it has. +11. Operate inside clear privacy, security, licensing, and enterprise controls. +12. Pass deterministic, fixture, Office mock, browser visual, dependency, + manual Windows Excel, and release packaging gates. + +## Non-Negotiable Guardrails + +- Office.js remains the only live Excel mutation path. +- Browser standalone mode remains local-first; no durable backend workbook + document storage is introduced unless a future enterprise mode explicitly + adds it with a separate data-governance design. +- `WorkbookPatchV1` remains the canonical edit contract for manual edits, + AI edits, sync replay, undo/redo, and export QA. +- Existing public endpoints remain backward compatible: + `/api/v1/tasks/plan/v2`, `/api/v1/tasks/plan/v3`, + `/api/v1/tasks/apply`, `/api/v1/tasks/apply/result`, + `/api/v1/workbooks/map`, `/api/v1/formulas/analyze`, + `/api/v1/spreadsheet/open`, `/api/v1/spreadsheet/export`, and + `/api/v1/spreadsheet/roundtrip-qa`. +- The standard proprietary `0.5.0` build omits HyperFormula and treats Excel as + the calculation authority. A separately licensed custom calculator build is + gated by package, approval, commercial-key, and parity evidence. +- VBA is preserved and warned. Browser JavaScript never executes VBA. +- Unsupported workbook parts are preserved or explicitly blocked. Silent loss is + a release-blocking bug. +- V7/accuracy behavior remains opt-in until readiness gates prove it can become + the default in a separate small change. +- Security, dependency advisories, licensing, privacy posture, and manual smoke + evidence are release blockers, not polish items. + +## Current Truth, Not Marketing Truth + +### What Is Real Today + +- Browser session, patch, sync, trust, formula parity, and storage types exist. +- The spreadsheet page is grid-dominant and has a real right-side AI/trust pane. +- AI preview can become browser patches before apply. +- Excel sync queue and Office operation mapping exist for common patch types. +- IndexedDB stores sessions, sync queues, sync journals, recovery checkpoints, + and bounded revision/fingerprint-bound undo/redo stacks in one CAS recovery + transaction. +- Worker open/export/roundtrip commands exist. +- OOXML preserve mode patches many worksheet-level and object-level constructs. +- Generated workbook fixtures exist and are wired into worker roundtrip tests. +- V7 backend artifacts, readiness reports, replay evaluation, deterministic + seed evidence, fail-closed anonymized replay corpus import, golden cases, + operation registry, and taskpane workbench hooks exist behind flags. +- Apply and rollback use two-phase, token-bound acknowledgements. Rollback is + not recorded complete until Office.js reports the restore result, and + incomplete restores retain recovery state. +- Targeted Office context reads are dimension-first, sheet-qualified, bounded, + and fail loudly; per-workbook session identity persists through Office + document settings without storing workbook or user content. +- The worker enforces archive, entry, expansion, path, formula-cell/reference, + output, memory, and process-tree limits. Formula graphs are sheet-qualified, + edited-formula precedence is deterministic, and supported OOXML deletions + remove owned relationships/parts while forcing Excel recalculation. +- Production packaging builds and serves both add-in surfaces and a generated + HTTPS manifest from an unprivileged edge container. The shared edge identity + is suitable only for a controlled internal deployment. +- AI-provider policy and durable resource ownership now fail closed across the + planner, documents, agent runs, jobs, sessions, and WebSockets. A one-shot + offline administrator bootstrap and key rotation path is available. + +### Where The Program Can Fool Us + +- Worker roundtrip QA mostly proves the same worker can reopen the export. It + does not yet prove Excel Desktop opens, recalculates, renders, refreshes, and + saves the workbook cleanly. +- Some generated fixture names imply charts, pivots, images, or macro coverage + that is not yet equivalent to real Excel-authored OOXML. +- Preserve mode still has conditional paths and regenerated fallback behavior + that can hide package-level fidelity gaps. +- SheetJS remains the isolated active parser, now pinned to the official + patched 0.20.3 distribution with a clean dependency audit and independent + ExcelJS/JSZip corpus comparison evidence. Replacement remains a GA decision + if future advisories or material corpus mismatches appear. +- Excel live sync has durable queue/replay, range-scoped drift detection, + bounded context snapshots, and failure recovery in automated host mocks, but + still needs the commit-matched Windows Excel Desktop evidence before GA. +- V7 replay evidence has deterministic seed and sanitized corpus import paths, + but still needs a real production-like anonymized corpus before any default + switch can be justified. +- Formula trace and parity are useful but still heuristic for structured + references, dynamic arrays, external links, volatile functions, and modern + Excel behavior. +- The UI now has tested object-editor patch flows for tables, charts, images, + named ranges, and sheet protection, but richer production editing for pivots, + drawings, slicers, and complex object lifecycles remains incomplete. +- Backend route tests for `/spreadsheet/*` now catch worker unavailability, + malformed payloads, fidelity headers, and additive request fields; remaining + API hardening should follow future worker commands and client contract growth. +- Durable tenant ownership, RBAC, admin bootstrap, key rotation, and AI egress + controls are implemented. Per-user SSO/OIDC, MFA/compliance workflows, and a + migration policy for legacy unowned sessions remain product-operating gaps. + +## Target Architecture + +### Client Platform + +- React/Vite taskpane and standalone spreadsheet surfaces. +- Univer as the spreadsheet UI and interaction layer. +- `WorkbookSessionV1` plus additive V2/V3 platform state as local workbook + state. +- `WorkbookPatchV1` as the single edit operation format. +- IndexedDB for sessions, queue, journal, QA artifacts, checkpoints, and local + restore points. +- Persistent Web Worker calculation protocol; the standard `0.5.0` build + returns a typed unavailable result for formula workbooks, while a separately + licensed calculator remains an optional custom-build path. +- AI pane, formula trace, object inspector, sync journal, trust panel, export + panel, source evidence, rollback, and audit surfaces. + +### Excel Host Layer + +- Existing `plugin/src/taskpane/excel-adapter.ts` as the only live mutation + path. +- Runtime host capability checks using Excel requirement sets. +- Event subscriptions for workbook, worksheet, selection, changed, calculated, + table, chart, comment, and sheet events where supported. +- Shared runtime evaluation remains a future option for ribbon/taskpane + coordination. Custom functions are intentionally not part of the shipped + contract: the old unregistered stubs required a browser credential and were + removed rather than presenting a nonfunctional or insecure Excel surface. +- Ordered sync queue with idempotency, retry, pause, reconcile, and rollback + metadata. + +### Backend + +- FastAPI for planning, validation, operation registry, source evidence, + benchmarks, readiness, audit, local agent runs, and stateless workbook APIs. +- No durable backend workbook bytes in the core local-first product. +- Typed artifact storage for audits, plans, readiness, replays, citations, and + non-sensitive metadata. +- Existing taskpane planning/apply APIs remain compatible while V7/V8 accuracy + routes mature behind flags. + +### Workbook Worker + +- Node worker boundary for SheetJS/ExcelJS/HyperFormula/JSZip/XML work. +- Open workbook -> session. +- Session -> preserved OOXML export. +- Roundtrip QA, OpenXML package QA, diff, formula graph, simulation, source + evidence, fixture validation, and large workbook analysis. +- Parser abstraction with a second parser/comparator path before GA. +- Strict payload size, extension, signature, and ZIP/OOXML safety checks. + +### Trust Plane + +- Workbook trust report aggregates: + - formula parity + - OOXML package and object fidelity + - source evidence + - operation simulation + - sync conflicts + - rollback/checkpoint availability + - unsupported features + - dependency and security posture + - manual Excel smoke status +- Primary export and live apply are blocked when trust blockers exist. + +## Program Workstreams + +### 1. Spreadsheet Editor Completion + +Build the browser editor into a serious workbook surface. + +Scope: + +- Complete direct grid editing, selection behavior, formula bar, keyboard + navigation, fill/copy/paste, range selection, formula entry, undo/redo, and + sheet tab workflows. +- Expand ribbon actions for Home, Insert, Page Layout, Formulas, Data, Review, + AI, and Settings. +- Make object inspector editors real for tables, comments, notes, validations, + conditional formats, named ranges, protection, freeze panes, charts metadata, + image metadata, and pivot metadata. +- Add dense but calm Shortcut-like visual polish: familiar Excel affordances, + no marketing layout, no empty command-room feel, grid as the primary object. +- Add accessibility and keyboard support: focus traps, aria labels, high + contrast, tab order, screen-reader names, and no clipped controls. +- Add large workbook UX: staged load, lazy panels, recalculation state, visible + analysis scope, warnings before expensive operations, and resumable work. + +Exit gates: + +- A user can complete common analyst workflows without touching dev tools. +- Playwright visual smoke passes desktop and laptop viewports with no clipped + file bar, ribbon, formula bar, status bar, right pane, or grid. +- Keyboard smoke passes enter, tab, shift-tab, escape, ctrl/cmd-Z, ctrl/cmd-Y, + ctrl/cmd-C/V, formula bar focus, sheet switching, and selection commands. + +### 2. Workbook State And Patch Platform + +Make local state durable, migratable, auditable, and boringly reliable. + +Scope: + +- Freeze `WorkbookSessionV1` and `WorkbookPatchV1` with additive-only changes. +- Add schema snapshots and migration tests for every saved IndexedDB version. +- Store operation history, inverse patches, checkpoints, and object edit + history consistently. +- Add range-aware dirty tracking, dependency impact, object impact, and trust + impact to patch application. +- Add conflict-safe patch replay and idempotent queue replay after reload. +- Add compaction for large histories without losing rollback checkpoints. + +Exit gates: + +- Old sessions open without data loss. +- Every supported patch produces an inverse or an explicit rollback warning. +- Reloading mid-sync does not lose queued writes, journal entries, or trust + status. + +### 3. Excel Live Sync GA + +Turn the V1.8/V2.0 sync foundation into production behavior. + +Scope: + +- Persist queue, journal, idempotency keys, retries, and apply outcomes. +- Replace coarse host fingerprint drift with range/object-aware summaries when + host snapshots provide enough data. +- Expand Office operation mapping for all supported patch types. +- Gate mappings on host capability and Excel requirement-set checks. +- Add host event subscriptions for changed, selection, calculated, sheet/table + additions, comment changes, and activation where supported. +- Add "Reload From Excel" as a real host-snapshot import path. +- Add "Keep Browser", "Reload From Excel", and "Stay Paused" reconciliation + flows with clear consequences. +- Add rollback-aware failure reporting from `applyOperationsAtomic`. +- Preserve Excel calculation mode and avoid interfering with native history. + +Exit gates: + +- Office mock tests cover success, host unavailable, protected sheets, partial + failure, rollback warning, external host drift, duplicate replay prevention, + reload from Excel, comments, tables, validations, formats, protection, and + unsupported object warnings. +- Windows Excel smoke captures link, edit, sync, external edit, reconcile, + reload, protected failure, pause/resume, export, reopen, and save. +- No parallel Excel writes. + +### 4. OOXML Fidelity And Export GA + +Make open -> edit -> export -> reopen trustworthy. + +Scope: + +- Make preserve-OOXML the default for any edited original workbook. +- Preserve all untouched parts, relationships, content types, workbook metadata, + custom XML, media, drawings, chart parts, pivot parts, VBA, slicers, themes, + styles, shared strings, calc chain, workbook views, print settings, page + layout, hyperlinks, defined names, and external links where safe. +- Patch supported XML parts with parser/DOM APIs, not string edits. +- Expand object editing for comments/notes, threaded comments where possible, + tables, validations, conditional formats, protections, dimensions, freeze + panes, sheet/workbook names, sheet visibility, defined names, chart metadata, + and image metadata. +- Create blockers for unsupported structural rewrites rather than falling back + silently. +- Add package-level validation for `[Content_Types].xml`, `.rels`, orphan + targets, missing overrides/defaults, duplicate IDs, broken refs, missing + package parts, and invalid ZIP/OOXML signatures. +- Add an independent comparison path so QA is not only "worker reopens its own + output." + +Exit gates: + +- Common edited `.xlsx` fixtures pass without regenerated fallback. +- `.xlsm` fixtures preserve VBA package parts and never execute macros. +- Chart, pivot, image, drawing, slicer, custom XML, and external-link packages + are preserved or explicitly blocked. +- Excel Desktop opens, recalculates, saves, and reopens exported workbooks in + the manual smoke set. + +### 5. Formula And Calculation Parity + +Make formula behavior transparent enough for analyst trust. + +Scope: + +- For a separately licensed custom build, deepen HyperFormula integration for + named ranges, cross-sheet references, structured references where possible, + array formulas, spills, volatile functions, iterative/circular refs, external + links, and unsupported functions. Keep the standard artifact physically free + of that runtime. +- Track source formula, displayed value, calculated value, Excel host value, + stale/dirty state, precedents, dependents, output impact, and errors. +- Add Formula Parity Report V2 with severity, materiality, affected outputs, + recommended Excel verification, and AI-safe alternatives. +- Add formula trace UI that navigates to precedents/dependents and named ranges. +- Add formula repair suggestions that generate `WorkbookPatchV1` only. +- Keep modern Excel gaps explicit. Do not fake dynamic array or external-link + parity. + +Exit gates: + +- Formula-heavy goldens report clean where supported and explain every gap + where unsupported. +- Displayed calculated values update without overwriting source formulas. +- Risky formula edits cannot be applied without formula impact evidence. + +### 6. AI Analyst Engine + +Make AI edits platform-native, evidence-backed, and reversible. + +Scope: + +- Move all manual and AI mutations through `WorkbookPatchV1`. +- Upgrade plan preview with affected ranges, objects, formula impact, source + evidence, sync impact, export impact, rollback status, and checkpoints. +- Require simulation for risky writes before apply. +- Use V7 truth model, formula graph, operation simulation, source evidence, + registry coverage, and repair candidates as the default candidate path only + after readiness gates pass. +- Add workflow templates for DCF, LBO, merger model, operating model, comps, + trading comps, sensitivity tables, debt schedules, monthly FP&A reporting, + KPI dashboards, budget variance, cohort analysis, and data cleanup. +- Add model style systems: banking formatting, color conventions, assumptions, + outputs, checks, scenario tables, sources, comments, and audit sheets. +- Add AI modes: Ask, Audit, Build, Edit, Explain, Repair, Reconcile, Source, + Format, and Compare. +- Add user controls for effort, speed, model/provider, data-use policy, and + auto-plan behavior. + +Exit gates: + +- AI preview never silently drops unsupported operations. +- AI apply can be undone or restored from a checkpoint. +- Benchmark cases prove formula-driven outputs, not hard-coded hidden answers. +- Source-backed tasks cite evidence and expose stale or missing source facts. + +### 7. Source Evidence And Data Integrations + +Move from "spreadsheet editing" to "trusted workbook intelligence." + +Scope: + +- Strengthen document upload, extraction, citation, and source-to-cell mapping. +- Add SEC/EDGAR workflows, filings refresh, PDF table extraction QA, source + refresh plans, stale source detection, and transformation metadata. +- Add connectors in phases: local files, web uploads, SEC, CSV/Parquet, database + OAuth, cloud drives, CRM/ERP exports, and eventually Google Sheets if product + strategy calls for it. +- Track each data transformation as an auditable artifact. +- Add source evidence cards directly in the spreadsheet right pane and taskpane. +- Support private, gitignored fixture sets for customer-like source documents. + +Exit gates: + +- A source-backed model can show exactly which document, page, table, row, and + transformation populated key cells. +- Refreshing source data produces a preview and diff before workbook mutation. +- Source evidence gaps are visible and block high-risk AI-generated outputs. + +### 8. Trust, Audit, Rollback, And Governance + +Make trust a product layer, not a log afterthought. + +Scope: + +- Unify `WorkbookTrustReportV2` with source evidence, simulation, formula + parity, sync, OOXML, rollback, security, dependency, and manual smoke status. +- Add checkpoint policy by operation type and risk level. +- Add full audit export for agent runs and workbook edits. +- Add explainable rollback snapshots for ranges, formulas, formats, comments, + tables, named ranges, sheets, and object metadata where possible. +- Add tamper-evident local audit hashes for sensitive workflows. +- Add review workflows: approve, reject, edit plan, require checkpoint, require + source, require Excel parity, and export blocked. +- Extend the effective AI policy controls for modes, providers, source + retention, external sending, BYOK, stored enterprise provider policy, and + role-mode entitlements into private fixture and UI-admin workflows; export + regenerated fallback and macro-regenerated fallback are now request-gated by + admin policy. + +Exit gates: + +- Every high-risk operation has approval, simulation, checkpoint, and audit + evidence before apply. +- Audit export can reconstruct who/what/when/why/where for AI and manual edits. +- Rollback status is honest: available, partial, manual, or unavailable. + +### 9. Enterprise, Security, Privacy, And Compliance + +Prepare the program for serious institutional users. + +Scope: + +- Define product data classes: workbook bytes, workbook metadata, prompts, + source docs, extracted facts, logs, telemetry, audit artifacts, and replay + hashes. +- Add privacy controls for local-only, no-training, zero-retention provider + routing, BYOK, and custom OpenAI-compatible endpoints. +- Harden API keys, teams, roles, MFA state, scoped permissions, and admin + audit logs. +- Validate the implemented OIDC authorization-code/PKCE and local-pairing + paths against the target IdP, keychain providers, and host matrix. +- Add encryption posture for local and backend artifacts. +- Add dependency risk policy, SBOM, license review, and package update cadence. +- Keep HyperFormula disabled or complete commercial-license approval and key + injection before proprietary shipping. +- Add security tests for malformed workbooks, ZIP bombs, path traversal, + formula injection, CSV injection, macro handling, prompt injection in source + documents, and untrusted external links. + +Exit gates: + +- Security and dependency gates pass without expired allowlists. +- Legal/license review has explicit decisions for HyperFormula, SheetJS, + ExcelJS, Univer, and all copied reference patterns. +- Enterprise policy settings are enforceable and test-covered. + +### 10. Performance, Scale, And Reliability + +Make the platform fast enough for real workbooks and long-running AI flows. + +Scope: + +- Define workbook size tiers and behavior: + - small: instant full analysis + - medium: staged analysis + - large: visible scoped analysis and lazy expansion + - very large: explicit user-selected ranges, background jobs, and host-only + verification where necessary +- Move expensive operations off the UI thread. +- Add worker cancellation, progress, timeout, and resumability. +- Add streaming or chunked paths where needed for ingestion, formula analysis, + QA, and source extraction. +- Add memory budgets and crash recovery for browser and worker. +- Add observability for latency, queue time, apply failures, QA duration, + recalc time, fixture drift, export blockers, and user-visible errors. +- Add offline/reconnect behavior for local-first sessions. + +Exit gates: + +- Large workbook operations show progress and remain cancellable. +- No major UI action blocks interaction for unacceptable periods. +- Reliability metrics are visible in local and CI evidence. + +### 11. Test, Benchmark, And Release Evidence + +Turn "it works on this demo" into durable proof. + +Scope: + +- Contract tests for every public API shape and generated TypeScript client. +- Unit tests for patch/inverse, A1 parsing, object mutation, sync mapping, + storage migration, formula parity, trust reports, and source evidence. +- Worker tests for workbook open/export/QA, package integrity, real OOXML + fixtures, parser comparison, malformed payloads, and dependency isolation. +- Backend tests for `/spreadsheet/*`, task planning, apply gates, readiness, + replay, golden cases, source evidence, audit, and enterprise controls. +- Office mock tests for every supported host operation family. +- Browser tests for editor UX, object inspector, formula trace, sync UI, trust + UI, AI preview/apply/undo, blocked export, fallback export, large workbook + states, and accessibility. +- Playwright visual smoke for the Shortcut-like layout. +- Windows Excel manual smoke, then automated host smoke where feasible. +- Private fixture harness with hand-labeled expected truth, formulas, objects, + source evidence, and export outcomes. + +Exit gates: + +- `scripts/verify-full-stack.ps1` passes. +- `plugin npm run verify` passes. +- `workbook-worker npm test` passes. +- Backend tests pass. +- Dependency risk check passes or has non-expired explicit mitigation. +- Manual Windows Excel smoke evidence is captured for the release candidate. + +## Completion Phases + +### Phase 0: Program Control Plane + +Goal: make the roadmap executable and measurable. + +Deliverables: + +- Program owner map for frontend, Office host, backend, worker, AI, security, + QA, and release. +- Release gate checklist encoded in docs and scripts. +- Backlog labels for phase, workstream, risk, dependency, and acceptance gate. +- Updated capability matrix that distinguishes "foundation", "beta", "GA", + and "deferred". +- Decision records for local-first storage, no backend workbook bytes, macro + execution, HyperFormula license, SheetJS risk, and V7 default policy. + +Exit gate: + +- The team can tell what blocks GA without reading source code. + +### Phase 1: V2 Foundation Hardening + +Goal: make the current V2.0 work credible. + +Deliverables: + +- Backend route tests for `/api/v1/spreadsheet/open`, `/export`, and + `/roundtrip-qa`. +- Type and OpenAPI snapshots for spreadsheet request/response additions. +- IndexedDB migration tests and queue replay tests. +- Worker unavailability tests and user-facing error states. +- Current fixture inventory audit: which fixtures contain real charts, pivots, + images, macros, comments, tables, validations, and protections. +- Manual smoke checklist template. + +Evidence artifacts: + +- `backend/tests/test_spreadsheet_routes.py` for `/api/v1/spreadsheet/*` + success, headers, policy forwarding, and worker failure paths. +- `python scripts/generate_openapi_types.py --check` result for additive route + schema drift. +- `workbook-worker` fixture validation output proving manifest expected objects + match actual OOXML package contents. +- `workbook-worker` package integrity output for preserved exports and + roundtrip QA. +- Parser comparison corpus output from + `python scripts/run_parser_comparison_corpus.py` proving normalized SheetJS + results match ExcelJS/JSZip evidence across generated workbook fixtures, with + mismatches treated as dependency-risk blockers. +- Private fixture manifest validation from + `python -m backend.benchmarks.private_golden_fixtures`; real private + workbooks and `manifest.json` stay gitignored while tracked templates document + hashed fixture IDs and required expectation metadata. +- `plugin/test/spreadsheet-storage.test.ts`, + `plugin/test/spreadsheet-sync-replay.test.ts`, and + `plugin/test/spreadsheet-ui.test.tsx` for IndexedDB migration, durable sync + replay, trust panel, sync panel, formula trace, and object inspector evidence. +- `plugin/test/spreadsheet-editor.smoke.spec.ts` for Playwright desktop/laptop + browser layout, formula-bar interaction, seeded workbook workflow, + formula-trace navigation, undo/redo, object-inspector editing, screenshot, + and owned-shell accessibility smoke evidence. +- `scripts/verify-spreadsheet-phase1.ps1` as the repeatable automated + evidence runner for this phase. +- `docs/WINDOWS_EXCEL_MANUAL_SMOKE.md` as the required manual evidence template + for Excel Desktop behavior. + +Phase 1 exit-gate checklist: + +| Gate | Status | Evidence | +| --- | --- | --- | +| Spreadsheet route tests | Implemented | `backend/tests/test_spreadsheet_routes.py` | +| OpenAPI/type drift check | Implemented | `python scripts/generate_openapi_types.py --check`; response models and export headers are documented | +| IndexedDB migration and replay tests | Implemented | `plugin/test/spreadsheet-storage.test.ts`; `plugin/test/spreadsheet-sync-replay.test.ts` | +| Spreadsheet trust UI tests | Implemented | `plugin/test/spreadsheet-ui.test.tsx` | +| Browser visual/interaction smoke | Implemented | `npm run test:spreadsheet-browser-smoke` in `plugin`; two Chromium viewports cover shell clipping, formula-bar edit, screenshots, and owned-shell axe critical checks | +| Fixture inventory audit | Implemented | `python scripts/run_fixture_inventory_audit.py`; 31 fixtures checked, 11/11 required OOXML feature classes covered | +| Parser comparison corpus | Implemented | `python scripts/run_parser_comparison_corpus.py --fail-on-mismatch`; release bundle fails closed on missing, stale, incomplete, failed, or mismatched corpus evidence | +| Worker package integrity QA | Implemented | `roundtrip-qa-v2.packageIntegrity` checks required package parts, preserved opaque parts, dangling relationships, and content-type coverage | +| Private golden fixture harness | Implemented | `backend/benchmarks/private_golden_fixtures.py`; tracked README/example plus gitignored local `manifest.json` and workbook bytes; release trust and V7 readiness now expose sanitized default-off private-fixture policy snapshots and fail closed when `--require-private-fixtures` is set | +| Worker unavailable/error coverage | Implemented | `backend/tests/test_spreadsheet_routes.py`; `backend/tests/test_operation_surface_v2.py` preserves worker-only async job errors and unavailable reasons | +| Windows Excel smoke template | Implemented | `docs/WINDOWS_EXCEL_MANUAL_SMOKE.md`; `python scripts/scaffold_windows_excel_smoke.py` creates run-specific checklists and evidence slots | +| Windows Excel smoke execution | Blocked until manual run | Attach evidence in the smoke checklist | + +Exit gate: + +- No current V2 foundation behavior is undocumented or untested at its public + boundary. + +### Phase 2: Spreadsheet Editor Beta + +Goal: ship a usable browser spreadsheet editor to internal testers. + +Deliverables: + +- Solid direct edit, formula bar, formatting, rows/columns, sheets, comments, + validations, tables, freeze panes, undo/redo, copy/paste, keyboard navigation, + and selection stats. +- Right-pane formula trace, object inspector, sync journal, trust report, AI + preview, and export fidelity panels. +- Large workbook loading/recalc/sync/export states. +- Visual and accessibility smoke. + +Exit gate: + +- Internal users can open a workbook, make common edits, run QA, export, and + understand every warning without developer help. + +### Phase 3: OOXML Fidelity Beta + +Goal: make exports safe enough for real fixture testing. + +Deliverables: + +- Preserve mode default for edited originals. +- Full package inventory diff and validation. +- Real Excel-authored object fixtures. +- Chart, pivot, image, macro, custom XML, external link, very hidden sheet, + protected workbook, and large workbook goldens. +- Explicit export blockers and regenerated fallback language. + +Exit gate: + +- No unsupported package part disappears silently in the public fixture matrix. + +### Phase 4: Excel Live Sync Beta + +Goal: make browser-to-Excel sync usable for controlled testers. + +Deliverables: + +- Durable queue and journal replay after reload. +- Host event subscriptions and drift detection. +- Capability-gated Office operation mapping. +- Reload from Excel. +- Reconcile UI. +- Office mock matrix. +- Windows Excel manual smoke. + +Exit gate: + +- A linked Excel user can recover from reload, drift, protected sheet failure, + and partial apply failure without losing browser state. + +### Phase 5: Formula Parity And Model Trust + +Goal: make calculations and formula risk visible enough for analyst workflows. + +Deliverables: + +- Formula Parity Report V2. +- Trace UI with navigation. +- Formula impact report in AI previews and manual risky edits. +- Modern Excel gap taxonomy. +- Formula repair suggestions through patches. +- Formula-heavy goldens. + +Exit gate: + +- The product can tell users when browser calculation is trustworthy, advisory, + or blocked. + +### Phase 6: AI Analyst Workflows Beta + +Goal: make AI useful beyond simple patch demos. + +Deliverables: + +- AI modes: Ask, Audit, Build, Edit, Explain, Repair, Source, Reconcile, + Format, Compare. +- Workflow templates for finance and business analysis. +- Source-backed plan previews. +- Checkpoint and simulation requirements for risky plans. +- Review panel and action history that non-developers can understand. +- Benchmark harness for formula-driven financial model outputs. + +Exit gate: + +- AI can complete representative financial modeling and data-cleaning cases + with evidence, preview, rollback, and measurable accuracy. + +### Phase 7: Source Evidence And Integrations + +Goal: make workbook outputs traceable to real sources. + +Deliverables: + +- Source extraction QA. +- Source-to-cell and cell-to-source graph. +- Citation cards in taskpane and spreadsheet pane. +- SEC/EDGAR refresh workflow. +- Database/file connector design and first safe connector path. +- Stale source and refresh diff UI. + +Exit gate: + +- Source-backed workbook cells can be audited back to documents and refreshed + with previewed diffs. + +### Phase 8: Enterprise And Security Readiness + +Goal: make the platform fit for controlled external beta. + +Deliverables: + +- Effective AI policy controls for modes, providers, source retention, external + sending, BYOK, stored enterprise provider policy, and role-mode entitlements + are exposed and enforced at request boundaries, including regenerated export + fallback and macro-regenerated fallback. The taskpane Settings and Controls + drawers now expose the effective policy state and persist provider-policy + toggles for local/cloud routing, BYOK, source documents, and export fallback; + remaining work is broader user/team admin, SSO/MFA/compliance workflows, and + private-fixture policy coverage. +- Provider/data-use policy controls. +- Dependency/license review. +- Security tests for workbook and source-document attack paths. +- Audit exports and tamper-evident local evidence. +- Privacy documentation and data flow diagrams. + +Exit gate: + +- The product has a credible answer for security review, procurement review, + and enterprise pilot constraints. + +### Phase 9: Performance And Reliability RC + +Goal: prove the product can survive real workbooks and real usage. + +Deliverables: + +- Size-tier behavior. +- Background worker cancellation and progress. +- Memory and latency budgets. +- Error recovery for reloads, offline, host disconnect, worker crash, failed + export, and interrupted AI runs. +- Observability dashboards or local reports. +- Regression suite on fixture and private workbook sets. + +Exit gate: + +- Release candidate runs full gates repeatedly without drift, flaky failures, or + severe performance regressions. + +### Phase 10: GA Release + +Goal: ship a completion-grade local-first workbook platform. + +Deliverables: + +- Final compatibility matrix. +- Release notes and migration notes. +- User docs for browser editor, Excel add-in, AI workflows, trust reports, + export fidelity, sync, rollback, source evidence, and enterprise controls. +- Support runbooks for sync failures, export blockers, worker failures, + corrupted workbooks, unsupported formulas, and macro warnings. +- GA smoke evidence bundle. +- Rollback plan for disabling V7 default, sync features, worker preserve export, + and high-risk AI operations. + +Exit gate: + +- Product, engineering, QA, security, and release owners sign off on all gates. + +### Phase 11: Post-GA Expansion + +Goal: broaden reach without destabilizing the core. + +Candidate scopes: + +- Google Sheets parity if strategy requires it. +- Automated Excel host smoke on Windows CI or dedicated lab machines. +- Multi-workbook projects and workspace-level source libraries. +- Collaboration, sharing, and cloud document storage as a separately governed + product line. +- Deeper chart editing, pivot refresh/recalculation, slicers, Power Query, + Power Pivot, and external data connections. +- Desktop helper for trusted local workflows. +- Public API and SDK. +- Skill marketplace and custom workflow builder. + +## Priority Backlog + +### P0: Completion Blockers + +- Backend route coverage for spreadsheet APIs. +- Real Excel-authored fixture expansion. +- Excel Desktop smoke evidence. +- Preserve-mode default and no silent regenerated fallback. +- Package integrity validation. +- Queue replay after reload. +- Host drift reconciliation. +- Formula parity severity model. +- AI plan preview with trust impact. +- Dependency/license decisions. +- Security tests for malformed files and macro handling. + +### P1: Beta Quality + +- Direct grid editing polish. +- Keyboard and clipboard basics. +- Object inspector editing. +- Formula trace navigation. +- Sync journal drawer. +- Trust report panel UX. +- Source evidence cards. +- More Office host capability mocks. +- Large workbook progress/cancel states. +- User docs and smoke runbooks. + +### P2: Competitive Parity + +- Financial model templates and style systems. +- Audit mode. +- Restore/checkpoint branching. +- Workspace/session history. +- Team admin controls. +- Provider/model/effort controls. +- Data connections and OAuth. +- File attachment reuse. +- API keys and automation API. +- File size warnings and large-file AI limits. + +### P3: Differentiators + +- Deterministic finance invariant engine. +- Human-readable model quality score. +- Source refresh impact simulation. +- Local-first trust bundle export. +- Private benchmark harness. +- Analyst workflow macros that produce safe patch sequences. + +## Fixture Strategy + +Public fixtures should cover: + +- Simple workbook. +- Formula-heavy workbook. +- Cross-sheet formulas. +- Named ranges. +- Structured references. +- Dynamic arrays and unsupported formula gaps. +- Comments and notes. +- Threaded comments where feasible. +- Tables. +- Data validations. +- Conditional formats. +- Styles, themes, fills, fonts, borders, number formats. +- Merges. +- Freeze panes. +- Row/column dimensions. +- Sheet visibility and very hidden sheets. +- Sheet/workbook protection. +- Charts with same-sheet and cross-sheet source ranges. +- Images and drawing anchors. +- Pivots and pivot caches. +- Slicers. +- External links. +- Hyperlinks. +- Macros with real `vbaProject.bin`. +- Custom XML. +- Large workbooks. +- Corrupt or malformed package edge cases. + +Private fixtures should cover: + +- Real analyst workbooks with permission. +- Customer-like financial models. +- Dense formatting and banker-style outputs. +- Messy data cleanup cases. +- Source-backed models from PDFs and filings. +- Workbooks known to break Excel add-ins or export paths. + +Fixture acceptance rule: + +- A fixture name must not imply an object is tested unless the OOXML package + actually contains that object and QA asserts it. + +## Verification Matrix + +Automated release-candidate gates: + +- `scripts/verify-full-stack.ps1` +- `scripts/verify-local-accuracy.ps1` +- `scripts/verify-v7-default-prep.ps1` +- `python -m pytest backend/tests -q` +- `python -m pytest backend/tests/contract -q` +- `npm test` in `workbook-worker` +- `npm run verify` in `plugin` +- `npm run test:office-mock` in `plugin` +- fixture open -> patch -> export -> reopen -> QA +- package integrity QA over generated fixtures plus private fixture manifest + validation; populated private workbook execution remains local release evidence +- Playwright visual and interaction smoke for the spreadsheet editor +- dependency and license review +- Docker build and worker smoke + +Manual release-candidate gates: + +- Windows Excel manual smoke using `docs/WINDOWS_EXCEL_MANUAL_SMOKE.md`. +- Evidence links or notes for every Pass, Pass with Warning, Blocked, Fail, or + Not Covered row. + +Windows Excel manual smoke must include: + +- Sideload add-in. +- Link active workbook. +- Edit value, formula, format, comment, table, validation, sheet, and name. +- Sync browser edits to Excel. +- Edit Excel externally and detect drift. +- Keep Browser reconciliation. +- Reload From Excel reconciliation. +- Protected sheet failure. +- Pause/resume. +- Export preserved workbook. +- Reopen export in Excel. +- Save from Excel and reopen again. +- Macro workbook warning and preservation. +- Chart/pivot/image preservation warnings. + +## Release Metrics + +Minimum metrics to track: + +- Open success rate by fixture type. +- Export success rate by fidelity mode. +- Excel Desktop smoke pass rate. +- Missing package part count. +- Formula parity issue count and severity. +- AI plan unsupported operation count. +- Rollback availability rate. +- Sync queue failure rate. +- Host drift detection rate. +- Average open, recalc, QA, export, and sync duration. +- Large workbook memory ceiling. +- Dependency risk status. +- V7 readiness status and replay drift. + +## Strategic Product Bar + +The competitive bar is not just "can edit spreadsheets." Shortcut publicly +markets Excel parity, native add-in workflows, formula-driven outputs, audit and +restore, file fidelity, macros/large files/keyboard shortcuts through the +plugin, team controls, and fast complex work. To compete, this program needs: + +- Excel-native reliability, not just a browser grid. +- Trust and audit as first-class UI. +- Formula-driven AI outputs with source evidence. +- Compatibility proof through real workbooks. +- Strong enterprise/privacy posture. +- Smooth workflows for finance users, not generic spreadsheet demos. +- Fast local-first behavior with clear limits. + +The differentiator should be the trust architecture: every AI edit is a typed +patch, every risky patch is simulated, every output is traceable, every export +has fidelity evidence, and every host write has a recoverable journal. + +## Decisions Required Before GA + +1. HyperFormula license: proprietary posture is fixed; keep the engine disabled + unless a commercial license/key is approved and injected. +2. SheetJS posture: keep the patched parser isolated with clean audit and + corpus-comparison evidence, or replace it before GA if material mismatches + or new unremediated advisories appear. +3. V7 default: stay opt-in until readiness is fully green. +4. Shared runtime: adopt for add-in coordination where supported or keep + isolated runtimes with degraded behaviors. +5. Cloud storage: keep strictly local-first for GA or design a separate + enterprise document-storage product. +6. Macro workflow: preservation/warn only for GA, or trusted desktop runner as + a separate explicitly opted-in feature. +7. Google Sheets: defer until Excel parity and local-first trust are complete. +8. Chart and pivot scope: metadata edit/preserve for GA, full render/recalc + post-GA unless proven earlier. + +## Remaining Work Before GA + +The remaining release work is evidence-led rather than another broad feature +slice: + +1. Run the 54-row Windows Excel Desktop checklist against this exact commit and + archive its screenshots, logs, exports, and reopen/save results. +2. Populate and execute the gitignored customer-like/private fixture corpus, + including true Excel-authored OOXML, and archive the inventory and parser + comparison outputs. +3. Import and pin an anonymized production-like replay corpus, then demonstrate + stable drift against its baseline before considering a V7 default switch. +4. Run the migration/backfill validator against staging or production Postgres, + archive the output with backup/restore evidence, and define reassignment or + import policy for legacy sessions that have no durable owner. +5. Keep HyperFormula disabled or provision an approved commercial key, then + complete the repository-wide proprietary-license and third-party signoff. +6. Exercise per-user OIDC/PKCE, MFA claims, WebSockets, local keychain pairing, + and the signed manifest against the target IdP and Windows/Mac/Web host + matrix before production exposure. +7. Exercise the Windows worker process-tree termination path and retain the + evidence; POSIX/macOS and Linux container paths are already automated. + +The strict release-evidence workflow intentionally fails closed while any of +these required inputs is missing. Synthetic evidence must not be relabeled as +production or Excel Desktop proof. diff --git a/docs/README.md b/docs/README.md new file mode 100644 index 0000000..82d623d --- /dev/null +++ b/docs/README.md @@ -0,0 +1,54 @@ +# Documentation map + +Excel Agent `0.5.0` is a supported private-beta candidate, not a GA release. +Start with the status ledger: it separates repository implementation from +experimental surfaces, design-only work, and promotion blockers. + +## Read by intent + +| Intent | Start here | Truth label | +|---|---|---| +| Understand what exists | [Implementation status](IMPLEMENTATION_STATUS.md) | **Implemented / Experimental / Designed / Blocked** | +| Run a disposable local API | [Getting started](GETTING_STARTED.md) | **Implemented** | +| Understand trust boundaries | [Architecture and authority](ARCHITECTURE.md) | **Implemented**, with vNext boundaries labeled separately | +| Use current API contracts | [OpenAPI cookbook](API_COOKBOOK.md) | **Implemented** default API; opt-in endpoints stay **Experimental** | +| Explore experimental vNext | [vNext clean-break guide](../vnext/README.md) | **Experimental** local contracts, cores, SDKs, and persistence; no deployed service | +| Operate an on-prem deployment | [Operations index](OPERATIONS_INDEX.md) | **Implemented** tooling; live deployment evidence is **Blocked** until supplied | +| Understand future work | [Roadmap index](ROADMAP_INDEX.md) | **Designed** and **Blocked** work is not shipped behavior | + +## Status vocabulary + +- **Implemented** means code and repository-side verification exist. It does + not imply external host, production, legal, signing, or release evidence. +- **Experimental** means explicit opt-in behavior outside the private-beta + default. +- **Designed** means an accepted contract or architecture without an available + end-to-end production runtime. +- **Blocked** means promotion must remain fail-closed pending genuine evidence + or an external decision. + +## Operator routes + +- Administration and recovery: [production runbook](PRODUCTION_RUNBOOK.md) +- Local/self-hosted checks: [self-hosted verification](SELF_HOSTED_VERIFICATION.md) +- Zero-hosted-minutes runners and caches: + [operator guide](ZERO_HOSTED_MINUTES_OPERATIONS.md) +- Exact-SHA release assembly: [release promotion](RELEASE_PROMOTION.md) +- Authenticated air-gap transfer and rehearsal: + [offline distribution](OFFLINE_DISTRIBUTION.md) +- Release blockers and evidence owners: [release checklist](RELEASE_CHECKLIST.md) + +## Documentation contract + +The information architecture is declared in +[`documentation-contract.json`](documentation-contract.json). Run the +network-free validator after documentation changes: + +```sh docs-check=syntax +.venv/bin/python scripts/validate_docs.py +.venv/bin/python -m pytest -q tests/docs +``` + +It verifies the document classification, local links and anchors, canonical +product versions, annotated snippet syntax, runnable examples, and cookbook +operations against FastAPI's generated OpenAPI document. diff --git a/docs/RELEASE_CHECKLIST.md b/docs/RELEASE_CHECKLIST.md new file mode 100644 index 0000000..0d49373 --- /dev/null +++ b/docs/RELEASE_CHECKLIST.md @@ -0,0 +1,122 @@ +# Excel Agent 0.5 Release Checklist + +Complete this checklist from a clean checkout of the exact merge SHA. Record +artifact digests and evidence locations in the signed release record. A checked +box is not evidence by itself. + +## Release identity and change control + +- [ ] `VERSION`, backend, plugin package/lock, and both Office manifests pass + `python scripts/check_version_sync.py` as product `0.5.0` / Office `0.5.0.0`. +- [ ] The working tree is clean; generated files and protocol/capability diffs + have been reviewed like source. +- [ ] Closure changes merged through an approved PR with required CODEOWNERS and + real GitHub Actions results from the merge SHA. +- [ ] All high/medium security findings are closed and independently retested; + there is no reproducible data-loss path. +- [ ] The release owner, security owner, workbook owner, and client owner sign + the release record. Legal counsel approves the proprietary license and all + dual/commercial dependency usage. + +## Automated gates + +- [ ] Backend fast and slow suites pass from a clean environment. +- [ ] Plugin typecheck, lint, Office mocks, unit/browser tests, and production + build pass. +- [ ] Workbook-worker tests, safety smoke, parser comparison, and corpus gates + pass in the packaged boundary. +- [ ] Semantic Ruff checks and the configured mypy closure gate pass. +- [ ] OpenAPI generation, package integrity, version synchronization, SBOM drift, + dependency review, vulnerability audit, secret scan, and license policy pass. +- [ ] Container builds and High/Critical image scans pass; provenance inputs are + pinned and reproducible. + +## Security, retention, and operations + +- [ ] Per-user identity is proven on every production HTTP/WebSocket path; + service-to-service authentication is least privileged and mTLS-verified. +- [ ] Lifecycle tests cover active work, restart, partial file/object-store + failure, zero-hour mode, cache/vector eviction, backup restoration, and signed + deletion receipts. +- [ ] Workbook and PDF parsing reject unsafe packages fail closed within CPU, + memory, wall-clock, page/object/output, and concurrency budgets. +- [ ] No cloud request occurs without policy approval and a matching receipt; + captured requests prove `store: false` for OpenAI. +- [ ] Production capability output excludes incomplete agent/RAG/experimental + routes and every advertised operation is proven or explicitly unavailable. + +## External evidence from this SHA + +- [ ] `release-evidence-index.json` uses schema + `2026-08-closure-release-evidence-index-v2`, records the trusted + `signingKeyId`, and binds every item to the exact lowercase 40-character merge + SHA. Its exact raw bytes verify against the root + `release-evidence-index.ed25519` signature and an out-of-band Ed25519 public + key/key ID supplied by CI. Every restored file is indexed with its allowlisted + repository-relative POSIX path, byte size, and lowercase SHA-256 digest; the + artifact contains no unindexed files or links. +- [ ] All 54 Windows Excel smoke rows have reviewer-identifiable artifacts. +- [ ] Separately build and offline-verify the exact-SHA authenticated + [native-host certification](NATIVE_HOST_CERTIFICATION.md) pack for Windows + Excel Desktop, Mac Excel Desktop, and Excel Web. Each pack must contain every + host-specific row, distinct reviewer/witness role hashes, complete artifact + links, and true no-repair/recalculation/save/coauthor/recovery assertions. A + pending scaffold or verifier test fixture is not evidence. +- [ ] The signed production manifest passes Windows, Mac, and Excel Web. +- [ ] The inventoried private Excel-authored corpus contains at least 250 + workbooks and passes fidelity, mutation, rollback, and no-repair-dialog + checks. +- [ ] Validate `private_corpus_execution` with + `scripts/validate_private_corpus_execution_receipt.py`: the detached + Ed25519 signature, out-of-repository key ID, exact corpus manifest and all + 250 workbook digests, native Excel build/runner/executor hashes, and every + open/recalculate/save/no-repair result bind to the release SHA. +- [ ] Production replay baseline and parser comparison are captured. +- [ ] Validate the anonymized production replay with + `scripts/validate_production_replay_execution_receipt.py`: the detached + Ed25519 signature and out-of-repository key bind the exact merge SHA, raw + corpus digest, source-cohort digest, every sanitized payload digest, + deterministic reruns, privacy scan, baseline/evaluator artifacts, and + provider-free execution. +- [ ] Live PostgreSQL migration, backfill, rollback rehearsal, and persistence + evidence are captured. +- [ ] HyperFormula is disabled in the release profile, or commercial approval + and injected-key evidence are present; no release artifact contains the + literal `gpl-v3` key. +- [ ] CycloneDX SBOM, third-party notices, dependency/license review, image scan, + trust report, and independent penetration-test disposition are present. +- [ ] The strict workflow reports all required evidence IDs as `passed`: + `signed_production_manifest`, `production_manifest_windows`, + `production_manifest_macos`, `production_manifest_excel_web`, + `private_corpus_execution`, `private_corpus_inventory`, `production_replay`, + `parser_corpus`, `postgres_migration`, `postgres_backfill`, + `postgres_rollback`, `postgres_backup_restore`, `hyperformula_posture`, + `sbom`, `trust_report`, `dependency_review`, `container_scan`, + `container_signature`, `provenance_verification`, and + `independent_pentest`, plus `release_signoff` with verified release, + security, workbook, client, and legal owner signatures. + +## Signing and publication + +- [ ] Build installers, containers, manifest, and offline verification bundle + from the merge SHA in the controlled release workflow. +- [ ] Build `excel-agent-release-promotion-v1` only after the signed self-hosted + verification receipt and signed release-evidence index both pass. Verify its + release and security signatures with two distinct out-of-band Ed25519 keys on + a separate machine before any tag or publication. +- [ ] Append and independently verify the dual-approved, hash-chained promotion + disposition described in the + [evidence portal guide](RELEASE_EVIDENCE_PORTAL.md). Bind canary/rollback + slots and exact source digests; generate candidate notes only from the + exact-SHA changelog signed by both owners. Treat the static portal strictly as + a signed redacted derivative, never as evidence or a release claim. +- [ ] Sign artifacts and provenance; verify signatures on a separate machine and + in an air-gapped verification rehearsal. +- [ ] Publish checksums, SBOM, provenance, migration notes, support window, + changelog, known limitations, and evidence index to authorized recipients. +- [ ] Create `release/0.5`, tag the merge SHA as signed `v0.5.0`, and verify the + tag resolves to the artifact source revision. +- [ ] Install the distributed artifacts—not local builds—on the supported host + matrix and complete the final smoke. +- [ ] Record go/no-go decision, rollback owner, monitoring window, and security + contact. Announce clearly that `0.5.0` is a private beta, not GA. diff --git a/docs/RELEASE_EVIDENCE_PORTAL.md b/docs/RELEASE_EVIDENCE_PORTAL.md new file mode 100644 index 0000000..38097c1 --- /dev/null +++ b/docs/RELEASE_EVIDENCE_PORTAL.md @@ -0,0 +1,283 @@ +# Authenticated release evidence portal and promotion ledger + +Status: **Implemented** as offline operator tooling. A portal or ledger entry is +not proof of deployment, publication, production readiness, or a release. The +current private-beta candidate remains **Blocked** until every external gate in +the [release checklist](RELEASE_CHECKLIST.md) is genuinely satisfied. + +This plane adds two derivative records after the signed promotion bundle and +[offline distribution](OFFLINE_DISTRIBUTION.md) already exist: + +1. `promotion_ledger.py` appends a two-person promotion disposition to a + hash-chained, append-only directory. +2. `release_evidence_portal.py` renders a deterministic, content-hashed static + view from that verified ledger and signs its manifest with a third, + portal-only Ed25519 key. + +Both commands run locally and make no network request. They consume the +existing verifiers instead of replacing them. Every operation revalidates the +distribution signature, both promotion approvals, self-hosted verification +receipt, release-evidence index, exact release SHA, toolchain manifest, SBOM, +provenance, checksums, install plan, and every bound artifact. Trusted public +keys and their case-sensitive IDs arrive out of band. + +> **The portal is never release evidence.** It is a redacted operational view +> over evidence. Its signed manifest proves only which portal bytes were +> produced; the underlying signed sources remain authoritative. + +## Trust and storage layout + +Keep all of these directories and keys outside the repository and mutually +disjoint: + +```text +/approved/distribution// authenticated offline distribution +/approved/ledger// append-only promotion dispositions +/approved/portal// signed static derivative +/approved/changelog// dual-approved changelog fragment +/approved/trust/ out-of-band public keys +/approved/keys/ access-controlled private keys +``` + +Release, security, distribution, verification, release-evidence, and portal +key IDs are explicit inputs. Release and security approvals must use distinct +keys. The portal signing key must also be distinct from release, security, and +distribution keys. Private keys, public trust keys, raw evidence payloads, +signer identifiers, signature references, credentials, and private filesystem +paths are never copied into the ledger's public notes or portal output. + +Set the operator paths once. These variables are illustrative and must point +to the controlled host's real locations: + +```sh docs-check=syntax +export REPOSITORY_ROOT="$(pwd)" +export COMMIT_SHA="$(git rev-parse HEAD)" +export DISTRIBUTION_DIR="/approved/distribution/$COMMIT_SHA" +export LEDGER_DIR="/approved/ledger/$COMMIT_SHA" +export PORTAL_DIR="/approved/portal/$COMMIT_SHA" +export CHANGELOG_DIR="/approved/changelog/$COMMIT_SHA" +export TRUST_DIR=/approved/trust +export KEY_DIR=/approved/keys +``` + +## Create the exact-SHA approved changelog + +Candidate release notes are generated only from schema +`excel-agent-approved-changelog-v1`. The fragment must set `approved: true`, +bind the exact 40-character SHA, name the out-of-band release and security key +IDs in that order, contain only allowlisted change categories, and use +canonical JSON. Secret-like text and private paths fail closed. + +Create canonical bytes with the repository's Python runtime: + +```python docs-check=syntax +import json +import os +from pathlib import Path + +release_sha = os.environ["COMMIT_SHA"] +output = Path(os.environ["CHANGELOG_DIR"]) / "approved-changelog.json" +payload = { + "schemaVersion": "excel-agent-approved-changelog-v1", + "releaseSha": release_sha, + "approved": True, + "approvals": [ + {"role": "release", "keyId": "release-owner-k1"}, + {"role": "security", "keyId": "security-owner-k1"}, + ], + "title": "Excel Agent private-beta candidate", + "summary": "Candidate changes approved for offline evaluation.", + "changes": [ + {"category": "Security", "text": "Fail-closed evidence verification."}, + {"category": "Changed", "text": "Deterministic offline operations."}, + ], +} +output.parent.mkdir(parents=True, exist_ok=True) +output.write_text( + json.dumps(payload, sort_keys=True, separators=(",", ":")) + "\n", + encoding="utf-8", +) +``` + +The release and security owners sign those exact bytes independently. Ed25519 +signatures are raw 64-byte files; never sign a reformatted copy. + +```sh docs-check=syntax +openssl pkeyutl -sign -rawin \ + -inkey "$KEY_DIR/release-approval.pem" \ + -in "$CHANGELOG_DIR/approved-changelog.json" \ + -out "$CHANGELOG_DIR/approved-changelog.release.ed25519" +openssl pkeyutl -sign -rawin \ + -inkey "$KEY_DIR/security-approval.pem" \ + -in "$CHANGELOG_DIR/approved-changelog.json" \ + -out "$CHANGELOG_DIR/approved-changelog.security.ed25519" +``` + +No release notes are emitted when approval is absent, a signature differs, a +key ID is stale, the SHA differs, or the text triggers the leakage policy. + +## Append and verify a disposition + +The first disposition normally authorizes a canary without claiming promotion. +The command serializes writers with an adjacent lock, fully verifies all +source bytes before staging, writes one complete `entry-NNNNNN` directory, +atomically publishes it, and verifies the entire chain again. A failed +post-publish check removes only the new entry. Existing entries are never +rewritten. + +```sh docs-check=syntax +.venv/bin/python scripts/promotion_ledger.py append \ + --ledger "$LEDGER_DIR" \ + --distribution "$DISTRIBUTION_DIR" \ + --expected-sha "$COMMIT_SHA" \ + --release-public-key "$TRUST_DIR/release-owner-public.pem" \ + --release-key-id release-owner-k1 \ + --security-public-key "$TRUST_DIR/security-owner-public.pem" \ + --security-key-id security-owner-k1 \ + --verification-public-key "$TRUST_DIR/verification-public.pem" \ + --verification-key-id self-hosted-verification-k1 \ + --release-evidence-public-key "$TRUST_DIR/release-evidence-public.pem" \ + --release-evidence-key-id release-evidence-k1 \ + --distribution-public-key "$TRUST_DIR/distribution-public.pem" \ + --distribution-key-id distribution-k1 \ + --changelog-fragment "$CHANGELOG_DIR/approved-changelog.json" \ + --changelog-release-signature "$CHANGELOG_DIR/approved-changelog.release.ed25519" \ + --changelog-security-signature "$CHANGELOG_DIR/approved-changelog.security.ed25519" \ + --release-private-key "$KEY_DIR/release-approval.pem" \ + --security-private-key "$KEY_DIR/security-approval.pem" \ + --decision approved-for-canary \ + --canary-status pending \ + --canary-reference-id canary-eu1 \ + --rollback-status ready \ + --rollback-reference-id rollback-v1 \ + --json +``` + +Allowed decisions have strict typed slots: + +- `approved-for-canary` requires a pending or not-run canary, a ready rollback, + and no blockers; +- `promoted` requires a passed canary with its artifact digest, a usable + rollback, and no blockers; +- `rejected` requires at least one portable blocker ID; +- `rolled-back` requires a failed canary, an executed rollback with its exact + artifact digest, and at least one blocker. + +Each entry signs the same canonical entry with the release and security keys, +binds all upstream SHA-256 digests, and records the previous canonical entry +hash. Verify from a separate offline operator context before using the head: + +```sh docs-check=syntax +.venv/bin/python scripts/promotion_ledger.py verify \ + --ledger "$LEDGER_DIR" \ + --distribution "$DISTRIBUTION_DIR" \ + --expected-sha "$COMMIT_SHA" \ + --release-public-key "$TRUST_DIR/release-owner-public.pem" \ + --release-key-id release-owner-k1 \ + --security-public-key "$TRUST_DIR/security-owner-public.pem" \ + --security-key-id security-owner-k1 \ + --verification-public-key "$TRUST_DIR/verification-public.pem" \ + --verification-key-id self-hosted-verification-k1 \ + --release-evidence-public-key "$TRUST_DIR/release-evidence-public.pem" \ + --release-evidence-key-id release-evidence-k1 \ + --distribution-public-key "$TRUST_DIR/distribution-public.pem" \ + --distribution-key-id distribution-k1 \ + --json +``` + +An interrupted process can leave the hidden adjacent writer lock. Do not remove +it merely to make the command pass. First prove no writer is active, preserve +the directory for incident review, and verify the existing ledger and source +distribution independently. A missing sequence, extra file, link, changed +signature, re-signed fork, or wrong previous-entry hash blocks the chain. + +## Build the static portal + +Use a clean checkout at `COMMIT_SHA`. The builder refuses repository-contained +or overlapping output and keys. It renders only allowlisted typed summaries: +evidence-class coverage, verified owner roles, artifact kinds/digests/sizes, +exact source bindings, blockers, canary and rollback slots, and candidate notes. +It excludes raw evidence paths and payloads. + +```sh docs-check=syntax +.venv/bin/python scripts/release_evidence_portal.py build \ + --output "$PORTAL_DIR" \ + --ledger "$LEDGER_DIR" \ + --distribution "$DISTRIBUTION_DIR" \ + --expected-sha "$COMMIT_SHA" \ + --release-public-key "$TRUST_DIR/release-owner-public.pem" \ + --release-key-id release-owner-k1 \ + --security-public-key "$TRUST_DIR/security-owner-public.pem" \ + --security-key-id security-owner-k1 \ + --verification-public-key "$TRUST_DIR/verification-public.pem" \ + --verification-key-id self-hosted-verification-k1 \ + --release-evidence-public-key "$TRUST_DIR/release-evidence-public.pem" \ + --release-evidence-key-id release-evidence-k1 \ + --distribution-public-key "$TRUST_DIR/distribution-public.pem" \ + --distribution-key-id distribution-k1 \ + --portal-private-key "$KEY_DIR/portal-signing.pem" \ + --portal-key-id release-portal-k1 \ + --json +``` + +CSS, JavaScript, status, and candidate notes have their content hash in the +filename. `portal-manifest.json` binds every file plus the ledger head, source +digests, template digests, exact SHA, and portal key ID. Its detached raw +signature is `portal-manifest.ed25519`. No clock or host path enters the output, +so identical trusted inputs produce identical bytes. + +## Verify and view without hosted minutes + +Transfer the portal, ledger, distribution, exact-SHA checkout, and public keys +through approved channels. Verify before viewing; the verifier authenticates +the portal signature before parsing, rejects links/traversal/extras, checks +content-hashed names, re-derives every public byte, scans for leakage, and +reruns the full source verification at the end. + +```sh docs-check=syntax +.venv/bin/python scripts/release_evidence_portal.py verify \ + --portal "$PORTAL_DIR" \ + --ledger "$LEDGER_DIR" \ + --distribution "$DISTRIBUTION_DIR" \ + --expected-sha "$COMMIT_SHA" \ + --release-public-key "$TRUST_DIR/release-owner-public.pem" \ + --release-key-id release-owner-k1 \ + --security-public-key "$TRUST_DIR/security-owner-public.pem" \ + --security-key-id security-owner-k1 \ + --verification-public-key "$TRUST_DIR/verification-public.pem" \ + --verification-key-id self-hosted-verification-k1 \ + --release-evidence-public-key "$TRUST_DIR/release-evidence-public.pem" \ + --release-evidence-key-id release-evidence-k1 \ + --distribution-public-key "$TRUST_DIR/distribution-public.pem" \ + --distribution-key-id distribution-k1 \ + --portal-public-key "$TRUST_DIR/portal-public.pem" \ + --portal-key-id release-portal-k1 \ + --json +``` + +The page uses only local, content-hashed files and a restrictive content +security policy. Because browsers block `fetch` from `file:` URLs, optionally +serve the already verified directory on loopback only. This is a local viewer, +not a deployment or externally hosted portal: + +```sh docs-check=syntax +cd "$PORTAL_DIR" +"$REPOSITORY_ROOT/.venv/bin/python" -m http.server 8765 --bind 127.0.0.1 +``` + +Open `http://127.0.0.1:8765/` from the same machine, then stop the process when +review ends. Do not bind a non-loopback address, copy source evidence into the +portal, enable analytics, or add remote assets. If any source or template byte +changes, verification fails; rebuild and obtain a new portal signature rather +than patching the output. + +## Evidence and claim boundary + +The portal deliberately displays `CANDIDATE STATUS ONLY` and `NOT RELEASE +EVIDENCE`. Build success cannot create release notes, approve promotion, prove +a canary, or justify publication. A `promoted` ledger disposition records what +two authorized owners signed about already verified bytes; it does not perform +deployment. Tagging, publication, host validation, production deployment, and +external signing/notarization remain separate manual controls in the +[production runbook](PRODUCTION_RUNBOOK.md) and +[release checklist](RELEASE_CHECKLIST.md). diff --git a/docs/RELEASE_PROMOTION.md b/docs/RELEASE_PROMOTION.md new file mode 100644 index 0000000..6d63d70 --- /dev/null +++ b/docs/RELEASE_PROMOTION.md @@ -0,0 +1,93 @@ +# Offline release promotion + +Release promotion is a separate, fail-closed step after build and evidence +collection. A green build, a GitHub status, or one signature is insufficient. +The promotion bundle binds the exact merge SHA, the complete authenticated +release-evidence tree, the signed self-hosted verification transcript, the +committed SBOM/notices/toolchain lock, and the actual distributable bytes. + +The builder requires these artifact kinds at minimum: + +- `source_archive` +- `office_manifest` +- `backend_image` +- `edge_image` + +Installer, model-pack, symbols, or other deliverables may be added with unique +portable kinds. Each input must be a stable regular file. Container inputs are +OCI archives exported from the already verified image digests; a tag name alone +is not evidence. + +## Build on the controlled promotion host + +Use a clean checkout whose `HEAD` is the exact release SHA. Place every private +key and the output directory outside the repository. The release and security +approval keys must be two distinct Ed25519 keys controlled by different owners. + +```sh +python scripts/build_release_promotion_bundle.py \ + --release-sha "$COMMIT_SHA" \ + --output "/approved/promotions/excel-agent-$COMMIT_SHA" \ + --verification-evidence "/approved/verification/$COMMIT_SHA" \ + --verification-public-key /approved/trust/verification-public.pem \ + --verification-key-id self-hosted-verification-k1 \ + --release-evidence-root "/approved/release-evidence/$COMMIT_SHA" \ + --release-evidence-index "/approved/release-evidence/$COMMIT_SHA/release-evidence-index.json" \ + --release-evidence-signature "/approved/release-evidence/$COMMIT_SHA/release-evidence-index.ed25519" \ + --release-evidence-public-key /approved/trust/release-evidence-public.pem \ + --release-evidence-key-id release-evidence-k1 \ + --release-private-key /approved/keys/release-approval.pem \ + --release-key-id release-owner-k1 \ + --security-private-key /approved/keys/security-approval.pem \ + --security-key-id security-owner-k1 \ + --artifact source_archive=/approved/build/excel-agent-source.tar.zst \ + --artifact office_manifest=/approved/build/manifest.xml \ + --artifact backend_image=/approved/build/backend.oci.tar \ + --artifact edge_image=/approved/build/edge.oci.tar +``` + +The builder independently verifies both inner evidence signatures before it +copies any input. It reads each file through one non-following descriptor, +checks that the source did not change during the copy, hashes every byte, +writes a canonical manifest, signs the same manifest with both approval keys, +and atomically installs the completed directory. Public trust keys are not +embedded. + +## Verify on a separate offline host + +Transfer the bundle plus four public keys through separate approved channels. +The verifier authenticates both promotion signatures before parsing the +manifest, rejects extra/missing/link/traversal files, rehashes every bound byte, +then independently verifies the inner verification receipt and release-evidence +index against the exact expected SHA. + +```sh +python scripts/verify_release_promotion_bundle.py \ + --bundle "/transferred/excel-agent-$COMMIT_SHA" \ + --expected-sha "$COMMIT_SHA" \ + --release-public-key /trusted/release-owner-public.pem \ + --release-key-id release-owner-k1 \ + --security-public-key /trusted/security-owner-public.pem \ + --security-key-id security-owner-k1 \ + --verification-public-key /trusted/verification-public.pem \ + --verification-key-id self-hosted-verification-k1 \ + --release-evidence-public-key /trusted/release-evidence-public.pem \ + --release-evidence-key-id release-evidence-k1 \ + --json +``` + +The verifier must pass on the exact directory that will be distributed. Any +repack, renamed file, new attachment, approval rotation, or artifact rebuild +requires a new bundle and two new signatures. Only after this offline check and +the host installation smoke may the release owner create the signed tag. + +## Record the offline disposition + +After the separately signed offline distribution verifies, follow the +[promotion ledger and evidence portal guide](RELEASE_EVIDENCE_PORTAL.md). The +append-only ledger binds two-person release/security approval, the exact source +digests, canary/rollback slots, blockers, and a previous-entry hash. Candidate +notes are derived only from an exact-SHA changelog fragment signed by both +owners. The separately signed static portal is a redacted derivative for local +operator review; it is not evidence, publication, deployment, or a release +claim. diff --git a/docs/ROADMAP_INDEX.md b/docs/ROADMAP_INDEX.md new file mode 100644 index 0000000..29f90d4 --- /dev/null +++ b/docs/ROADMAP_INDEX.md @@ -0,0 +1,24 @@ +# Roadmap index + +Roadmaps describe intent and sequencing; they do not override the +[implementation status ledger](IMPLEMENTATION_STATUS.md). Historical plans are +retained for decision context and are never evidence that a feature shipped. + +## Current roadmaps + +| Document | Scope | Authority | +|---|---|---| +| [vNext implementation guide](../vnext/README.md) | Current clean-break contracts, cores, SDKs, local persistence, and verification | Implemented/experimental repository truth; no deployed-service claim. | +| [Program completion roadmap](PROGRAM_COMPLETION_ROADMAP.md) | Remaining `0.5.x` product, evidence, and release work | Current planning context; status still comes from the ledger and release checklist. | +| [vNext implementation roadmap](../vnext/docs/IMPLEMENTATION_ROADMAP.md) | Clean-break protocol/core/service program | **Designed/Experimental** unless individually promoted in the ledger. | +| [v0.5 to vNext migration](V05_VNEXT_MIGRATION.md) | One-way migration boundary | Validated stages can enter the neutral lifecycle-owned intake store; domain materialization, hidden dual-write, and reverse conversion remain unavailable. | + +## Archived context + +| Document | Why retained | +|---|---| +| [Flagship V5 implementation plan](FLAGSHIP_V5_IMPLEMENTATION_PLAN.md) | Historical milestone assumptions and acceptance ideas; some baselines and formula-engine claims are superseded. | +| [External codebase utilization notes](EXTERNAL_CODEBASE_RESEARCH.md) | Historical design/research inputs and licensing cautions, not a dependency or readiness claim. | + +No source was deleted or silently rewritten. Each archived document carries a +banner pointing back to this reconciliation page. diff --git a/docs/SELF_HOSTED_VERIFICATION.md b/docs/SELF_HOSTED_VERIFICATION.md new file mode 100644 index 0000000..eccc828 --- /dev/null +++ b/docs/SELF_HOSTED_VERIFICATION.md @@ -0,0 +1,221 @@ +# Self-hosted verification + +Excel Agent does not require GitHub-hosted runner minutes. The checked-in +workflows target only operator-owned self-hosted labels, and +`scripts/verify-local.sh` is the authoritative command graph that can run on a +developer machine, an ephemeral VM, or an isolated release host. + +## Runner pools + +Provision fresh, single-job runners with these exact labels: + +| Pool | Required labels | Intended jobs | +|---|---|---| +| Linux | `self-hosted, linux, x64, excel-agent` | backend, client, worker, vNext, CodeQL, secret scan, containers | +| Windows + Excel | `self-hosted, windows, x64, excel-agent-excel` | commit-bound native Excel and strict release evidence | + +Never attach these labels to a shared workstation. Each job must receive a +clean VM or machine snapshot, a dedicated unprivileged account, an empty +workspace, bounded network access, and automatic destruction after one job. +Do not persist repository credentials, package-manager credentials, signing +keys, evidence trust keys, workbook fixtures, or Actions tokens in a runner +image. + +Workflows reject pull requests whose head repository differs from the base +repository. This prevents unreviewed fork code from executing on an owned +runner. A maintainer can inspect an external contribution, import it onto an +internal branch, and then run the same SHA-bound checks. + +## Local command graph + +First inspect the host: + +```sh +python scripts/toolchain_doctor.py --profile full --require-self-hosted +``` + +Then select a verification tier: + +```sh +scripts/verify-local.sh --profile fast +scripts/verify-local.sh --profile backend +scripts/verify-local.sh --profile client +scripts/verify-local.sh --profile worker +scripts/verify-local.sh --profile vnext +scripts/verify-local.sh --profile full +scripts/verify-local.sh --profile containers +scripts/verify-local.sh --profile release +``` + +The `release` profile intentionally fails while exact-SHA external evidence is +absent. It never generates, substitutes, or relabels native-host, private +corpus, production replay, signing, legal, or penetration-test evidence. + +## Toolchain and cache contract + +- Python 3.12 or newer is required. +- Node.js 22 or newer and npm 10 or newer are required. +- Docker 24 or newer is required for container and pinned Rust/Buf gates. +- Go 1.22 or newer is required for the pinned Actionlint gate. +- PowerShell 7.4 or newer is required on the Windows release host. +- Package installs must use the committed Python, npm, and Cargo locks. Python + installation and cache export must enforce pip `--require-hashes`; every + universal platform artifact accepted by the lock has a SHA-256 digest. +- Cache exports are performance inputs only. A clean cache miss must not change + dependency versions or verification semantics. +- The runner must have enough local storage for no-cache images and browser + binaries, then erase the workspace and caches after the evidence bundle is + retained. + +`toolchain.lock.json` is the machine-readable source of truth for supported +runtimes, the vNext compiler/MSRV split, pinned Rust and Buf images, Actionlint, +the two architecture-specific local Rust tools tags, required Rust components +and target, the explicit WASM-capable package allowlist, native-only package +exclusions, and every dependency lock. Verify it without installing anything: + +```sh +python scripts/verify_toolchain_manifest.py --json +``` + +To seed an approved air-gapped runner without making caches authoritative, +export a cache on a disposable networked host and transfer the resulting +directory through the normal artifact review boundary: + +```sh +scripts/export_dependency_cache.sh --output /absolute/path/outside/repository +(cd /transferred/cache && shasum -a 256 -c SHA256SUMS) +``` + +The exporter builds from committed locks, saves the exact digest-pinned Rust +and Buf images, builds and verifies the Rust tools image for `linux/amd64`, +vendors Cargo sources, downloads Python wheels, warms isolated npm caches, and +hashes every output. Never commit the export or use it to override lock +resolution. + +The derived tools image and tar are not promised to be byte-reproducible: +BuildKit/OCI creation, attestation, and archive metadata can differ across two +otherwise valid no-cache builds. Their authority is the exact base digest, +Dockerfile and toolchain-lock digest, platform, labels, declared components and +target, and verified tool versions. Each concrete exported tar is separately +bound by digest and size in that cache's manifest; offline replay rechecks and +loads those exact bytes. Do not copy a derived image digest into the source +toolchain manifest or compare it across independent exports as a release ID. + +On a clean online Linux runner, the vNext Rust gate builds the matching local +tools image from the exact base digest and then verifies its labels, +architecture, Rust 1.97.1, `rustfmt`, `clippy`, and +`wasm32-unknown-unknown` with container networking disabled before compiling: + +```sh +python scripts/rust_toolchain.py verify-vnext --online +``` + +On an isolated `linux/amd64` runner, consume only an already verified cache: + +```sh +python scripts/rust_toolchain.py verify-vnext \ + --offline-cache /runner-cache/excel-agent +``` + +Offline mode never invokes `docker build` or `docker pull`, reloads the exact +inventory-bound tools archive through a stable file descriptor, rechecks the +cache after load, mounts only the verified Cargo vendor tree, sets Cargo +offline mode, and runs the same formatting, strict Clippy, locked workspace +tests, and locked WASM library checks as online mode. The WASM command names +each manifest-approved portable package; it never applies a blanket workspace +claim. Every active workspace package must be classified as WASM-capable or +native-only, so adding a crate without an explicit posture fails verification. +`excel-agent-service-host` is classified as native-only and remains outside the +WASM compile set as an active root-workspace member. + +## Evidence retention + +For a release candidate, retain the exact source SHA, toolchain-doctor JSON, +command transcript, test reports, image digests, SBOM, vulnerability output, +host/corpus reports, and detached evidence-index signature. Store release trust +keys outside both the repository and artifact bundle. + +GitHub commit statuses may link to an external/self-hosted evidence system, but +the status is only an index. The signed retained artifacts are the authority. + +On a clean candidate checkout, bind the local command graph to its exact SHA +and retain a tamper-evident transcript with an Ed25519 key stored outside both +the repository and output directory: + +```sh +python scripts/run_verification_receipt.py \ + --profile full \ + --output /approved/evidence/excel-agent-$COMMIT_SHA \ + --signing-key /approved/keys/verification-ed25519.pem \ + --signing-key-id self-hosted-verification-k1 +``` + +The command refuses dirty trees, in-repository keys/outputs, non-Ed25519 keys, +and pre-existing output directories. It atomically emits the combined log, a +canonical JSON receipt containing the exact source SHA/toolchain/log digests, +and a raw 64-byte detached signature. A failed verification is retained and +signed as failed; it must never be promoted into a passing commit status. + +Verify a transferred receipt offline against an out-of-band public key and the +commit that an external status claims: + +```sh +python scripts/verify_verification_receipt.py \ + --evidence /transferred/evidence/excel-agent-$COMMIT_SHA \ + --trusted-public-key /approved/trust/verification-ed25519-public.pem \ + --expected-sha "$COMMIT_SHA" \ + --expected-key-id self-hosted-verification-k1 \ + --require-passed --json +``` + +The verifier authenticates the exact canonical receipt bytes before parsing, +then independently checks the source SHA, key ID, pass/exit-code consistency, +toolchain manifest, and complete log digest and size. + +An external status service may dry-run or publish a GitHub commit status only +after that offline verification succeeds. The evidence URL must be HTTPS, +contain the exact commit SHA in its path, and have no credentials, query, or +fragment. Publication is opt-in and reads the API token only from the named +environment variable: + +```sh +python scripts/publish_verification_status.py \ + --evidence /transferred/evidence/excel-agent-$COMMIT_SHA \ + --trusted-public-key /approved/trust/verification-ed25519-public.pem \ + --expected-sha "$COMMIT_SHA" \ + --expected-key-id self-hosted-verification-k1 \ + --repository stag175/excelaiplugin \ + --evidence-url "https://evidence.example/$COMMIT_SHA/receipt" \ + --state success --context excel-agent/self-hosted/full + +# Add --publish only on the isolated status-publisher host with GITHUB_TOKEN set. +``` + +GitHub stores only the status and immutable evidence link. It never receives +the receipt-signing private key and it never runs the verification workload. + +## Native Excel host collection + +The self-hosted command graph and its signed verification receipt do not prove +Excel host behavior. Use the separate +[native-host certification procedure](NATIVE_HOST_CERTIFICATION.md) on approved +Windows Excel Desktop, Mac Excel Desktop, and Excel Web hosts. The repository +provides pending-only scaffolds plus an offline builder/verifier; it performs no +tenant login or host automation and contains no real passing host evidence. + +The operator-owned host run must bind the exact candidate SHA, signed manifest, +add-in and container digests, host/build identities, runner/executor hashes, +every profile row, independently reviewed artifacts, and an external Ed25519 +signature. Missing or failed host packs keep the release profile blocked; never +fall back to hosted minutes or convert a pending scaffold into evidence. + +## Failure and outage behavior + +If the self-hosted pool or external status service is unavailable: + +1. do not fall back to `ubuntu-latest` or `windows-latest`; +2. run the same profile on a clean approved host; +3. retain and review the signed transcript and outputs; +4. post the result to the exact commit only after independent verification; +5. keep merge or release blocked if the required status/evidence cannot be + authenticated. diff --git a/docs/STACK_CAPABILITY_MATRIX.md b/docs/STACK_CAPABILITY_MATRIX.md index 2afb60e..7aef421 100644 --- a/docs/STACK_CAPABILITY_MATRIX.md +++ b/docs/STACK_CAPABILITY_MATRIX.md @@ -1,45 +1,78 @@ # Stack Capability Matrix +The [implementation status ledger](IMPLEMENTATION_STATUS.md) is authoritative. +The older Foundation/Beta/GA vocabulary below is retained for compatibility +analysis only and does not claim GA or supersede **Implemented**, +**Experimental**, **Designed**, and **Blocked** labels. + This matrix freezes the public compatibility surface while the V7 accuracy kernel is built behind flags. +Status vocabulary: + +- Foundation: implemented, but not yet externally proven. +- Beta: covered by automated tests plus captured manual smoke evidence. +- GA: covered by automated gates, Windows Excel evidence, and release-owner signoff. +- Deferred/Gated: intentionally blocked or future scope. + +## vNext clean-break experimental surface + +This table is separate from the supported `/api/v1` compatibility surface. +Generated contracts and locally executable cores do not create a production +service or expand the private-beta API. + +| Surface | Truth label | Current boundary | +| --- | --- | --- | +| Protobuf contracts and TypeScript/Python SDKs | **Implemented contracts / Experimental clients** | Frozen breaking baselines and cross-language fixtures exist. Clients cover current Workbook, Run, Collaboration, and Formula RPCs, but callers own authentication, endpoints, serialization, and transport. | +| Workbook, formula, run, collaboration, and bridge domain cores | **Experimental** | Native/WASM transport-neutral state, proof, recovery, and bounded evaluation slices exist. None is a deployed server, Office adapter, provider, or wholesale Excel authority. | +| OPC package and XLSX semantic cores | **Experimental** | Bounded package validation, exact no-edit bytes, constrained proven existing-cell edits, deterministic recovery, and opaque preservation exist. Native Excel and production export certification remain external gates. | +| Native local service host | **Experimental** | A private-POSIX-root persistence boundary provides authenticated snapshots and exact-generation CAS for the five domain cores. It is native-only and has no network listener, provider, Office integration, remote-filesystem certification, or production claim. | +| Network/Office/provider exposure and production export/calculation promotion | **Designed / Blocked** | Service contracts are not deployed. Commit-matched host/corpus evidence, production certification, and function-family authority evidence remain required. | + ## Compatibility Surface | Surface | Status | Notes | | --- | --- | --- | | `/api/v1/tasks/plan/v2` | Complete | Existing V2 plan clients remain supported. | | `/api/v1/tasks/plan/v3` | Complete | V4/V5/V6/V7 clients preserve this endpoint while richer artifacts are created through additive accuracy routes. | -| `/api/v1/tasks/apply` | Complete | Office.js remains the only live mutation path. | +| `/api/v1/tasks/apply` | Complete | Office.js remains the only live mutation path. Apply and rollback use token-bound, two-phase client acknowledgements; durable state is finalized only after the Office host reports the result. | | `/api/v1/workbooks/map` | Complete | Current semantic map endpoint remains unchanged. | -| `/api/v1/formulas/analyze` | Complete | HyperFormula primary, Univer/Python fallbacks retained. | +| `/api/v1/formulas/analyze` | Complete, fallback-only in standard build | The standard proprietary `0.5.0` build omits HyperFormula and reports bounded Univer/Python analysis as an explicit confidence gap. A separately licensed custom worker can make HyperFormula primary only after distribution, approval, and commercial-key gates pass. | | Workbook worker commands | Complete | Existing commands are preserved. | ## V7 Additive Surface | Capability | Status | Endpoint / Module | | --- | --- | --- | +| Spreadsheet editor V2.0 local-first platform | Foundation, Phase 1 hardening active | Browser sessions now carry additive `WorkbookPlatformStateV2`, formula parity, trust-report, sync-journal, object-history, export-trust, manual-smoke, and large-workbook state; the editor persists sync queues/journals in IndexedDB, durable-replays Excel sync queues with terminal failure guards, stores QA/export artifacts in platform state, exposes tested Formula Trace/Object Inspector/Sync/Trust panels, adds V2 object patch variants, includes patch-backed object inspector editing for tables, comments, validations, conditional formats, chart/image metadata, named ranges, and sheet protection, blocks unsafe primary export from trust blockers, and keeps all workbook bytes local/stateless. Fixture inventory validation now has automated 11/11 real-OOXML feature evidence, and Windows Excel smoke has a run-pack scaffold plus strict artifact validation; captured real Excel Desktop execution remains the active proof gap. | +| Spreadsheet editor V1.8 Excel live sync | Foundation, manual smoke required | Browser spreadsheet sessions now carry additive `ExcelLinkStateV1` metadata, map `WorkbookPatchV1` edits onto the existing Office.js adapter, queue local-first edits for ordered Excel writes, pause on host drift or apply failure, and surface link status, queue depth, conflicts, and reconcile actions in the spreadsheet ribbon/status bar/right pane without adding backend live document storage. | +| Spreadsheet editor V1.7 object fidelity | Foundation, Excel Desktop evidence required | OOXML-preserving export now patches browser-authored notes/comments, tables, validations, conditional formats, sheet protection, freeze panes, dimensions, and workbook defined names while keeping opaque chart/drawing/media/pivot/macro/custom XML parts preserved; the browser fidelity panel shows object-level patched/blocked/lost status. | +| Spreadsheet editor V1.6 fidelity | Foundation, package QA hardening active | Browser export now exposes preserved-vs-regenerated fidelity state, blocks unsafe primary exports, offers an explicit regenerated-copy fallback, and shows roundtrip QA/package preservation status in the spreadsheet pane. | +| Spreadsheet editor V1.5 usability | Foundation | The Shortcut-like spreadsheet surface now routes formula-bar edits, direct Univer grid edits, Home/Insert/Data/Review ribbon actions, row/column/sheet operations, formatting, comments, validations, tables, undo/redo, selection stats, and AI plan application through the additive `WorkbookPatchV1` layer. | +| Shortcut-like spreadsheet engine V1 | Foundation | `plugin/src/spreadsheet/spreadsheet.html` adds a full-page browser spreadsheet editor with Univer UI, local `WorkbookSessionV1` state, patch/undo/redo, IndexedDB persistence, AI plan preview/apply-to-grid, and stateless backend routes `POST /api/v1/spreadsheet/open`, `/export`, and `/roundtrip-qa`. The standard proprietary build deliberately has no browser calculator; formula-bearing workbooks show `calculation_engine_unavailable` and Excel remains authoritative. | +| Spreadsheet worker session commands | Foundation, hardened package boundary | Worker commands `open-workbook-v1`, `export-workbook-v1`, and `roundtrip-qa-v2` convert XLSX/XLSM/CSV payloads into the browser session model, patch supported object XML in preserved exports, delete cleared tables/comments/panes without orphaning owned parts, invalidate stale calculation chains, preserve unrelated opaque OOXML, and report structured fidelity gaps. ZIP structure/path/expansion limits, formula cell/reference budgets, bounded subprocess output, memory limits, timeouts, and process-tree termination fail closed before untrusted workbook work can exhaust the host. | | WorkbookTruthModelV5 | Added behind flag | `POST /api/v1/workbooks/truth-model/v5` builds typed semantic target decisions, analyst-memory hooks, model taxonomy, source-evidence weighting, V2 finance invariant reports, and V7 accuracy reports. | -| FormulaGraphArtifactV5 | Added behind flag | `POST /api/v1/formulas/graph/v5` keeps HyperFormula primary and marks Univer/Python fallbacks as explicit confidence gaps, with structured references, calc-chain hints, material formula failures, and formula graph quality metadata. | +| FormulaGraphArtifactV5 | Added behind flag; licensed calculator gated | `POST /api/v1/formulas/graph/v5` exposes structured references, calc-chain hints, material formula failures, and formula graph quality metadata. Standard-build Univer/Python results remain explicit confidence gaps and cannot silently satisfy a HyperFormula authority gate. | | OperationSimulationV6 | Added behind flag | `POST /api/v1/operations/simulate/v6` is the V7 non-read pre-apply gate with V3 diff previews, execution groups, rollback requirements, checkpoint reasons, finance impact, repair candidates, and silent fallback blocking. | | SourceEvidenceGraphV3 | Added behind flag | `POST /api/v1/source-evidence/v3` adds citation validation, refresh plans, EDGAR workflow placeholders, transformation metadata, and source-backed assumption impact. | | V7WorkbookReview | Experimental | `POST /api/v1/experiments/v7/reviews`, `GET /api/v1/experiments/v7/reviews/{review_id}`, and `/export` persist a read-only local review combining truth V5, formula graph V5, simulation V6, source evidence V3, registry coverage, accuracy score, confidence gaps, checkpoints, repair candidates, recommended actions, V2/V3 versus V7 plan comparison, required inspections, targeted read queue, formula navigation hints, rollback proof status, replay signature, and default-readiness snapshot. | -| V7DefaultReadinessReport | Experimental | `GET /api/v1/experiments/v7/readiness`, `POST /api/v1/experiments/v7/readiness/run`, and `GET /api/v1/experiments/v7/readiness/history` persist the default-prep source of truth across local accuracy, replay drift, formula readiness, registry coverage, rollback coverage, Office mock coverage, worker hardening, dependency risk, golden cases, OpenAPI/type generation, storage parity, taskpane coverage, and `PLAN_ENGINE_VERSION=v2` enforcement. Reports expose opt-in allowance, gate timing, baseline status, readiness history, replay capture policy, and registry readiness breakdown. | -| LocalReplayEvaluation and baselines | Experimental | `GET /api/v1/experiments/v7/replays`, `POST /api/v1/experiments/v7/replays/evaluate`, `POST /api/v1/experiments/v7/replays/capture`, `GET /api/v1/experiments/v7/replays/baselines`, and `POST /api/v1/experiments/v7/replays/baselines` list, evaluate, capture, and pin anonymized local replay artifacts. Evaluation groups deterministic signatures, compares against baselines, reports drift severity, skips unknown scratch artifact types, and rejects raw workbook/prompt content at capture time. | -| V7GoldenCaseEvaluation | Experimental | `GET /api/v1/experiments/v7/golden-cases` and `POST /api/v1/experiments/v7/golden-cases/evaluate` expose public workbook fixture cases as golden-case evidence with expected truth targets, formula facts, simulation gates, dirty ranges, rollback expectations, and repair outcomes. | +| V7DefaultReadinessReport | Opt-in allowed; default switch blocked | `GET /api/v1/experiments/v7/readiness`, `POST /api/v1/experiments/v7/readiness/run`, and `GET /api/v1/experiments/v7/readiness/history` persist the default-prep source of truth across local accuracy, replay drift, formula readiness, registry coverage, rollback coverage, Office mock coverage, worker hardening, dependency risk, golden cases, OpenAPI/type generation, storage parity, taskpane coverage, and `PLAN_ENGINE_VERSION=v2` enforcement. The omitted standard-build HyperFormula runtime is recorded as `not_required`, not as false license approval; formula-authority and external evidence gaps still keep the shipped default on v2. Production/anonymized replay evidence, a separate default-switch change, and Windows Excel release evidence are required before defaulting. | +| LocalReplayEvaluation and baselines | Experimental, typed Postgres projection added | `GET /api/v1/experiments/v7/replays`, `POST /api/v1/experiments/v7/replays/evaluate`, `POST /api/v1/experiments/v7/replays/capture`, `GET /api/v1/experiments/v7/replays/baselines`, and `POST /api/v1/experiments/v7/replays/baselines` list, evaluate, capture, and pin anonymized local replay artifacts. `python -m backend.benchmarks.local_replay_seed` creates deterministic non-production plumbing evidence, `backend/benchmarks/fixtures/replays/synthetic_aggregate_replay_corpus.json` provides a committed non-production importer contract fixture, and `python -m backend.benchmarks.local_replay_corpus --production-evidence --evidence-class ` imports a fail-closed aggregate/anonymized corpus when real production-like evidence exists. Evaluation groups deterministic signatures, compares against baselines, reports drift severity, skips unknown scratch artifact types, and rejects raw workbook/prompt content at capture time; the corpus importer now also rejects filename, path, formula, email, sheet/range, URL, and non-slug raw-content values inside otherwise allowlisted aggregate string fields. Postgres now dual-writes replay artifacts and baselines into typed `local_replay_artifacts` and `local_replay_baselines` tables while preserving JSONB fallback reads. | +| V7GoldenCaseEvaluation | Experimental | `GET /api/v1/experiments/v7/golden-cases` and `POST /api/v1/experiments/v7/golden-cases/evaluate` expose public workbook fixture cases as golden-case evidence with expected truth targets, formula facts, simulation gates, dirty ranges, rollback expectations, and repair outcomes. The evaluation metadata also reports fail-closed private fixture manifest status from `backend.benchmarks.private_golden_fixtures` without exposing private workbook paths or formulas. | | AgentRunV4 | Added behind flag | `POST /api/v1/agent-runs/v4`, execution groups, accuracy report, rollback, retry, resume, repair-plan generation, durable step state, and `GET /api/v1/audit/agent-runs/{run_id}/export` expose V7 run artifacts and audit exports while keeping apply as an Office.js-approved handoff. | | OperationRegistryV3 | Added | `GET /api/v1/operations/registry/v3` exposes checkpoint policy, generated frontend handler keys, simulation support, rollback strategy, V7 readiness metadata, `applyReady`, `safeGated`, host requirements, and readiness reasons. Screenshot and sparkline operations are V7-ready with host fallback behavior; pivots remain safely gated. | | HostCapabilityMatrixV2 | Added | `GET /api/v1/capabilities/office/v2` exposes V7 host capability metadata and registry-backed operation coverage. | | V7 worker commands | Added | Worker now accepts `truth-model-v5`, `formula-graph-v5`, `simulate-operations-v6`, `source-evidence-v3`, `openxml-deep-qa-v2`, `workbook-diff-v3`, and `fixture-validate`. | -| Workbook fixture gate V3 | Expanded | `GET /api/v1/benchmarks/workbook-fixtures/v3` reports 30 generated `.xlsx/.xlsm` fixtures and V7 rollout thresholds. | +| Workbook fixture gate V3 | Expanded | `GET /api/v1/benchmarks/workbook-fixtures/v3` reports 30+ generated `.xlsx/.xlsm` fixtures, including object-fidelity and package-preservation workbooks with notes, tables, validations, conditional formats, protection, freeze panes, named ranges, real chart/drawing parts, pivot parts, and a macro-enabled VBA project stream. Fixture inventory validation is now the automated source of truth for manifest-to-OOXML object claims. | | Taskpane V7 workbench | Added behind flag | The Settings drawer now provides a local persisted `V7 local accuracy mode` opt-in. When enabled, truth, formula graph, simulation, source-evidence, capabilities, and agent-run calls route to V7 surfaces; when disabled, planning stays on the shipped V2 path. The command palette exposes read-only Understand, Trace, Simulate, Repair, and Audit panels plus readiness history, replay baselines, golden cases, execution-group, retry, rollback, and audit-export commands without making V7 default. | -| Local accuracy and readiness gates | Added | `.\scripts\verify-full-stack.ps1`, `python -m backend.benchmarks.local_accuracy`, `python -m backend.benchmarks.local_replay --evaluate`, `python -m backend.benchmarks.v7_readiness`, `python -m backend.benchmarks.v7_golden_cases`, `.\scripts\verify-local-accuracy.ps1`, `.\scripts\verify-v7-default-prep.ps1`, `npm run test:office-mock`, and `npm run test:taskpane` cover deterministic benchmarks, workbook-aware fixtures, anonymized replay artifacts, V7 review contracts, default-prep readiness, frontend workbench hooks, and behavioral Office adapter coverage. | +| Local accuracy and readiness gates | Added | `.\scripts\verify-full-stack.ps1`, `.\scripts\verify-full-stack.ps1 -ReleaseEvidence`, `python -m backend.benchmarks.local_accuracy`, `python -m backend.benchmarks.local_replay --evaluate`, `python -m backend.benchmarks.local_replay_seed`, `python -m backend.benchmarks.local_replay_corpus `, `python -m backend.benchmarks.v7_readiness`, `python -m backend.benchmarks.v7_golden_cases`, `.\scripts\verify-local-accuracy.ps1`, `.\scripts\verify-v7-default-prep.ps1`, `npm run test:office-mock`, `npm run test:frontend-smoke`, `npm run test:spreadsheet-browser-smoke`, and `npm run test:taskpane` cover deterministic benchmarks, workbook-aware fixtures, anonymized replay artifacts, V7 review contracts, default-prep readiness, frontend workbench hooks, browser layout/interaction/workflow smoke, behavioral Office adapter coverage, and strict release-evidence validation. | ## V6 Additive Surface | Capability | Status | Endpoint / Module | | --- | --- | --- | | WorkbookTruthModelV4 | Added behind flag | `POST /api/v1/workbooks/truth-model/v4` builds durable semantic target decisions, resolver scorecards, citation coverage, rollback readiness, repair candidates, and V6 accuracy reports. | -| FormulaGraphArtifactV4 | Added behind flag | `POST /api/v1/formulas/graph/v4` promotes HyperFormula graph output into a pre-apply simulation artifact with clusters, copied-formula regions, dynamic array/spill metadata, named-expression graph, output ranking, and fallback confidence gaps. | +| FormulaGraphArtifactV4 | Added behind flag; licensed calculator gated | `POST /api/v1/formulas/graph/v4` promotes available graph output into a pre-apply simulation artifact with clusters, copied-formula regions, dynamic array/spill metadata, named-expression graph, output ranking, and explicit fallback confidence gaps. | | OperationSimulationV5 | Added behind flag | `POST /api/v1/operations/simulate/v5` is the mandatory V6 pre-apply gate for non-read operations, including diff previews V2, dependency impact, recalculation delta, rollback snapshot requirements, checkpoint reasons, and silent fallback blocking. | | SourceEvidenceGraphV2 | Added behind flag | `POST /api/v1/source-evidence/v2` persists source-to-cell and cell-to-source provenance, citation coverage, stale-fact hooks, and downstream citation risk. | | AgentRunV3 | Added behind flag | `POST /api/v1/agent-runs/v3`, step listing, step retry, rollback, and accuracy-report endpoints expose persisted step artifacts, idempotency metadata, checkpoint status, and rollback hooks. | @@ -62,17 +95,23 @@ kernel is built behind flags. | Formula graph artifact V3 | Added | Formula graphs now include formula clusters, output rankings, centrality, parser confidence, and a formula impact report. | | Workbook truth model V3 | Added | Truth models now include semantic target resolutions, evidence weights, entity counts, ambiguity count, user confirmations, and model sections. | | Operation simulation V4 | Added | Simulations now include formula impact reports, gate reports, rollback snapshot plans, blocking reasons, checkpoint operation IDs, object intersections, and citation impact. | -| Range rollback snapshots | Partial foundation | Taskpane captures pre-mutation range values, formulas, number formats, text, and restore operation hints for range-backed mutations. | +| Two-phase apply and rollback | Hardened foundation | Taskpane snapshots every targeted sheet/range with explicit blank cells, values, formulas, formats, and supported object inverse state. Apply and restore are token-bound; plans move through pending states and are finalized only after a verified Office acknowledgement. Partial or failed restores keep recovery state, while operations without reliable inverse/proof semantics are rejected or explicitly safe-gated. | | Workbook-aware benchmarks | Expanded | `/api/v1/benchmarks/run` supports `mode: workbook-aware` over real ExcelJS-generated `.xlsx` golden fixtures ingested through SheetJS, with synthetic fallback contexts only when fixture files are absent. | ## Known Gaps | Area | Status | Next Step | | --- | --- | --- | -| True atomic Office.js rollback | Partial | Range snapshots exist; object rollback remains manual unless host APIs can snapshot reliably. | -| Postgres parity | Partial | Generic JSONB artifact coverage exists for flagship artifacts; production schema hardening and migrations for typed relational reporting remain future work. | -| Frontend Office.js tests | Active | `npm run test:office-mock` now guards protected-sheet preflight, write rollback snapshots, dirty-range reporting, screenshot and sparkline unsupported-host fallbacks, and V7 opt-in/readiness/replay workbench hooks; next step is broader host-behavior mocks for partial failures, tables, charts, comments, and richer supported-host paths. | -| Workbook-aware benchmark fixtures | Active | Real generated fixtures now cover 30 V7 scenarios; next expansion is externally sourced golden workbooks and gitignored private fixtures with hand-labeled truth models, expected formula graphs, simulation gates, rollback expectations, repair outcomes, and expected audit exports. | -| V7 default switch | Deliberately deferred | V7 can only become default-ready when readiness is `ready`, replay drift is stable, golden cases pass, and backend, worker, plugin, replay, formula, dependency-risk, and manual smoke gates pass. The shipped default remains v2, and any default switch must be a separate small change with rollback instructions. | -| Pivot mutations | Gated | Keep explicit warnings until host support is proven. | -| SheetJS advisories | Tracked risk | Worker now enforces payload size, extension allowlists, and OOXML/ZIP signature checks for `.xlsx/.xlsm/.xlsb`; keep isolation. | +| Office.js recovery boundary | Hardened; unsupported host objects remain gated | Range snapshots are sheet-qualified and include formerly blank cells so rollback can clear newly written values. Office mock coverage proves partial-write recovery and automatic table, named-chart, and comment create/update/delete restoration where the host exposes reliable state. Pivot, unnamed-chart, reply-heavy comment, and other unsafe object mutations remain manual or fail closed rather than being recorded as applied. Windows Excel Desktop execution remains the release proof gap. | +| Postgres parity | Protocol-ready foundation, typed projections, locally live-validated | `PostgresSessionStore` satisfies the storage protocol and persists tenant ownership plus JSONB and typed projections for plans, apply results, audits, documents, V7 readiness/replay, and workbook-intelligence artifacts. A disposable Postgres 16 run proved migrations, all typed backfills, idempotency, first-admin bootstrap, key rotation, and cross-user isolation. `scripts/validate_postgres_migration_backfill.py` remains the release entrypoint; a fresh run and backup/restore archive against the target staging/production topology is still required. Legacy rows without an owner fail closed to non-admins and need an explicit reassignment/import policy if retained. | +| Frontend Office.js tests | Active | `npm run test:office-mock` now guards protected-sheet preflight, write rollback snapshots, partial write failure rollback, automatic table, named-chart, and comment rollback, rollback restore failures, supported table/chart/comment proof, dirty-range reporting, chart create/update/delete/not-found/type aliases, validation clear/error-alert persistence, pivot inventory/safe-gated mutations, screenshot and sparkline unsupported-host fallbacks, and V7 opt-in/readiness/replay workbench hooks; next step is real Excel Desktop evidence. | +| Spreadsheet Excel live sync smoke | Active | V1.8 has host-mock coverage for patch mapping, queue state, persisted replay, range-scoped host drift, force replay, terminal failed-item guards, apply-failure pause, capability filtering for tables, charts, comments, validations, conditional formats, sparklines, pivots, screenshots, and named ranges, plus durable skipped journal results for safe-gated pivot batches; next step is a Windows Excel manual smoke with structured coverage for link, edit, sync, external host edit, reconcile, protected sheet failure, pause/resume, standalone export, and attached evidence artifacts. | +| Local-first workbook platform | Active; browser calculation gated | V2.0 adds persisted sync journals, formula lineage/parity reporting, navigable formula trace, patch-backed object inspector editing, trust reports, QA/export artifacts, IndexedDB migration coverage, a user-visible clear-local-data action, and browser-private session mode. Playwright covers desktop/laptop layout, formula editing, seeded workflows, real Univer mount/input/paste, object editing, and diagnostics. Standard `0.5.0` artifacts omit HyperFormula, so displayed workbook values remain cached and Excel is the calculation authority. Office document settings persist only an opaque per-workbook session identifier; browser/dev fallback is tab-scoped. Next steps are Windows Excel evidence and broader production-style editing coverage. | +| Excel custom functions | Not shipped | Legacy `FINANCEAI.*` stubs were never declared by either manifest or built by Vite, and the AI function depended on a browser-held API key plus a legacy endpoint. They were removed. Reintroducing custom functions requires an explicit shared-runtime design, per-user authentication, supported-client matrix, manifest validation, and Excel Desktop evidence. | +| Enterprise/admin policy | Private-beta implementation; external identity evidence pending | Effective AI policy filters provider routes before cache/health/provider calls and fails closed for documents and agent-run surfaces, so a providerless request cannot escape to a disallowed cloud model. Request-scoped principals and durable owner columns bind sessions, documents, plans, snapshots, agent artifacts, jobs, and WebSockets to their tenant; foreign resources return not found and admins retain an audited bypass. On-prem clients use per-user OIDC authorization-code plus PKCE with signed-token issuer/audience/tenant/MFA checks; local desktop clients use loopback-only one-time pairing. Offline first-admin bootstrap and key rotation remain break-glass controls. Live IdP, keychain-host, and legacy-owner migration evidence are still required. | +| Production add-in distribution | Supported private-beta release candidate | The non-root edge image builds and serves the taskpane, browser spreadsheet, icons, and runtime-rendered HTTPS manifest, strips public source maps, enforces a restrictive CSP, forwards per-user bearer identity, and uses verified mTLS to the backend without placing credentials in browser assets. Development services bind to loopback. Commit-matched Windows/Mac/Web Excel evidence, target-environment TLS/OIDC smoke evidence, licensing, signing, and the external release bundle are still required before GA. | +| Phase 1 evidence bundle | Active, fail-closed release hardening added | `.\scripts\verify-spreadsheet-phase1.ps1` now runs backend route/OpenAPI checks, plugin verify with IndexedDB/sync/UI tests, Playwright browser visual/interaction smoke, worker OOXML tests, and workbook fixture benchmarks; `scripts/generate_release_trust_bundle.py` treats skipped strict smoke artifact validation as a warning that fails in warning-fail mode and now records a sanitized private-golden-fixture policy snapshot. `.\scripts\verify-full-stack.ps1 -ReleaseEvidence` now requires strict smoke validation, production/anonymized replay evidence, and a ready gitignored private fixture manifest before release evidence can pass. | +| Workbook-aware benchmark fixtures | Active, private release gate added | Real generated fixtures now cover 30+ V7 scenarios; `backend.benchmarks.private_golden_fixtures` adds a gitignored private manifest harness for externally sourced/customer-like workbooks with hashed fixture IDs, OOXML workbook package validation, expected truth targets, formula facts, simulation gates, rollback expectations, repair outcomes, and audit expectations. `python -m backend.benchmarks.v7_golden_cases --require-private-fixtures`, `python -m backend.benchmarks.v7_readiness --require-private-fixtures`, and `python scripts/generate_release_trust_bundle.py --require-private-fixtures` fail release/default-switch evidence when that manifest is absent or invalid, while normal readiness remains default-off for private fixture enforcement. Remaining work is to populate the private corpus locally without committing private workbook bytes. | +| V7 default switch | Opt-in allowed, production default blocked | `python -m backend.benchmarks.v7_readiness` permits opt-in use when deterministic gates pass, but dependency/licensing warnings remain visible and `--require-production-replay` blocks release/default-switch readiness unless the pinned replay baseline is production/anonymized corpus evidence. Default-switch blockers distinguish deterministic seed, synthetic contract, sanitized non-production corpus, and production-anonymized replay evidence so local proof can improve without relaxing the shipped-default gate. The shipped default stays v2 until production/anonymized corpus evidence exists and a separate small default-switch change is paired with strict release evidence, Windows Excel Desktop smoke, and rollback instructions. | +| Pivot mutations | Gated with durable skipped replay | Keep explicit warnings until host support is proven; persisted sync replay now records warning-only pivot batches as skipped gate results instead of applied zero-op batches. | +| SheetJS parser posture | Patched and isolated; corpus comparison retained | The worker pins the official SheetJS 0.20.3 distribution, the dependency audit is clean, and no SheetJS advisory exception remains. It also enforces archive, entry, aggregate expansion, path, ZIP-structure, and OOXML signature limits for `.xlsx/.xlsm/.xlsb`; `parser-comparison-snapshot` compares normalized SheetJS output with the independent ExcelJS/JSZip adapter, and `python scripts/run_parser_comparison_corpus.py --fail-on-mismatch` gates the generated workbook corpus. SheetJS remains isolated and active while those audits and corpus comparisons stay clean; a material mismatch or future unremediated advisory reopens the replacement decision. | diff --git a/docs/THIRD_PARTY_NOTICES.md b/docs/THIRD_PARTY_NOTICES.md index 21cea44..6e382b2 100644 --- a/docs/THIRD_PARTY_NOTICES.md +++ b/docs/THIRD_PARTY_NOTICES.md @@ -5,37 +5,110 @@ the `workbook-worker` package and uses other researched projects as references. ## Direct Runtime Dependencies -- SheetJS `xlsx` 0.18.5, Apache-2.0 +- SheetJS `xlsx` 0.20.3, Apache-2.0 - Used in `workbook-worker` for workbook ingestion, formula extraction, metadata, CSV previews, merge/style metadata, and macro/VBA flags. - - Security note: `npm audit` currently reports prototype-pollution/ReDoS - advisories for this package with no npm-published fix. Keep the worker as a - narrow subprocess boundary and re-evaluate if SheetJS publishes a fixed npm - package or if a maintained alternative is chosen. + - Installed from SheetJS's pinned official CDN tarball because the npm + registry package stops at 0.18.5. The lockfile records the HTTPS source and + integrity digest; the worker remains a narrow, payload-limited subprocess. - ExcelJS 4.4.0, MIT - Used in `workbook-worker` for styled XLSX export, tables, formulas, workbook properties, and worksheet formatting. - - Security note: `npm audit` reports a transitive `uuid` advisory through - ExcelJS. The suggested automatic fix is breaking, so this remains a tracked - dependency-review item. -- HyperFormula `hyperformula` 3.2.0, GPL-3.0-only/commercial - - Used as the approved primary formula dependency graph and recalculation engine in the isolated workbook worker. - - V6/V7 use HyperFormula-backed formula graph and simulation artifacts for dependency impact, named expressions, circular references, recalculation warnings, material formula failures, and pre-apply confidence gaps. - - Production deployments should provide `HYPERFORMULA_LICENSE_KEY`; local/dev may use HyperFormula's `gpl-v3` key only where GPL terms are acceptable. + - The transitive `uuid` dependency is overridden to 11.1.1; the resolved + production tree passes `npm audit`. - Univer `@univerjs/engine-formula` 0.21.1, Apache-2.0 - Used as the secondary formula parser/fallback dependency for workbook formula analysis. +- Univer `@univerjs/core` and `@univerjs/preset-sheets-core` 0.22.x, Apache-2.0 + - Used by the browser spreadsheet editor. The project intentionally does not + depend on the `@univerjs/presets` umbrella package, so unrelated Univer Pro + packages are absent from the production dependency graph. +- PyMuPDF 1.28.2, AGPL/commercial dual licensing + - Used for PDF extraction inside the bounded parser worker. + - Proprietary distribution requires documented commercial-license approval + or replacement/disablement before release. The dependency's mere presence + in a lockfile or SBOM is not legal approval. +- `@bufbuild/protobuf` 2.13.0, Apache-2.0 AND BSD-3-Clause + - Used by the generated vNext TypeScript protocol bindings. The exact license + expression and integrity digest are recorded in `vnext/package-lock.json`. +- Python `protobuf` 7.35.1, BSD-3-Clause + - Used by the generated vNext Python protocol bindings and pinned exactly in + `vnext/pyproject.toml`. +- Rust `prost` and `prost-types` 0.14.4, Apache-2.0; `prost-types` also includes + Protocol Buffers code under its original BSD license. + - Used by the generated vNext Rust protocol bindings and checksum-pinned in + `vnext/Cargo.lock`. +- Rust `unicode-casefold` 0.2.0, MIT / Apache-2.0, and + `unicode-normalization` 0.1.25, MIT OR Apache-2.0. + - Used by the vNext workbook core for locale-independent full Unicode case + folding and NFC normalization of Excel sheet-name identity. + - `unicode-normalization` resolves `tinyvec` 1.12.0 and `tinyvec_macros` + 0.1.1; both offer permissive MIT/Apache-2.0/Zlib license choices. Exact + versions, checksums, and reviewed license metadata are recorded in the + vNext Cargo lock and release SBOM. +- Rust `zip` 7.2.0, MIT, and `quick-xml` 0.41.0, MIT. + - Used by the vNext bounded OPC package layer and XLSX semantic XML codec. + `zip` is compiled without its default feature set and enables only the pure-Rust + `deflate-flate2-zlib-rs` path; the package layer rejects unsupported ZIP + methods, encryption, links, ambiguous paths, and resource-budget breaches. + - Reviewed transitive dependencies are `adler2` 2.0.1 (0BSD/MIT/Apache-2.0), + `crc32fast` 1.5.0 (MIT/Apache-2.0), `equivalent` 1.0.2 + (Apache-2.0/MIT), `flate2` 1.1.9 (MIT/Apache-2.0), `hashbrown` 0.17.1 + (MIT/Apache-2.0), `indexmap` 2.14.0 (Apache-2.0/MIT), `memchr` 2.8.3 + (Unlicense/MIT), `miniz_oxide` 0.8.9 (MIT/Zlib/Apache-2.0), + `simd-adler32` 0.3.10 (MIT), `typed-path` 0.12.3 (MIT/Apache-2.0), and + `zlib-rs` 0.6.7 (Zlib). Their exact registry checksums are locked in + `vnext/Cargo.lock` and emitted into the release SBOM. +- Rust `yrs` 0.27.3, MIT. + - Used only by the experimental, transport-neutral vNext collaboration core + for bounded Yrs updates, actor-scoped undo, deterministic compaction, and + digest-verified snapshot recovery. Default features are disabled; both the + pinned native and `wasm32-unknown-unknown` builds are verified. This + dependency does not make the collaboration service production-available. + - All 38 resolved registry dependencies, exact versions, declared license + expressions, and crates.io archive checksums are independently recorded in + `vnext/docs/dependency-reviews/yrs-0.27.3.md` and emitted from the frozen + `vnext/Cargo.lock` into the release SBOM. The graph is permissively licensed; + `r-efi` offers MIT or Apache-2.0 alternatives in addition to + LGPL-2.1-or-later, so this build does not select a copyleft-only term. + +## Optional development and separately licensed builds + +- HyperFormula `hyperformula` 3.2.0, GPL-3.0-only/commercial + - The standard proprietary `0.5.0` plugin and production worker dependency + trees omit HyperFormula. The worker lock retains it only as a development + dependency for optional-adapter contract tests; `npm ci --omit=dev` keeps it + out of the shipped backend image. Browser formula evaluation is unavailable + in this standard build and Excel remains authoritative. + - A custom proprietary build may move the exact package into its production + graph only with separately purchased approval, an injected commercial key, + `HYPERFORMULA_DISTRIBUTED=true`, and regenerated notices/SBOM/evidence. + - The literal `gpl-v3` and other noncommercial sentinel keys are rejected by + runtime configuration, adapters, worker code, and release evidence. + +## Complete inventory and approval + +The deterministic CycloneDX inventory is committed at +`security/sbom.cdx.json` and is regenerated from the backend Python lock, all +npm locks, the exact-pinned vNext Python project/build dependencies, and the vNext +Cargo lock. It is the canonical machine-readable inventory, but it does not +replace source-notice obligations, export review, vulnerability review, or +legal approval. Dependency and license approvals must identify the exact +version and release SHA and must be renewed when either changes. ## Reference Implementations - OpenSheets: Office.js spreadsheet-service patterns. - Office Agents: Excel tool API, dirty ranges, citations, and UI/tool UX. - OfficeCLI: Apache-2.0 schema vocabulary for XLSX operation validation. +- XlsxWriter examples: BSD-2-Clause `vbaProject.bin` sample fixture used only + to generate a macro-enabled workbook preservation test fixture. ## Worker Boundary Notes - The worker remains a subprocess-only spreadsheet engine boundary. - V6/V7 commands preserve the same size limits, extension allowlist, malformed workbook rejection, macro-risk propagation, and no-network design posture used by the existing worker commands. -- Do not mark the SheetJS or ExcelJS advisories as resolved unless dependency versions actually change or a reviewed replacement parser/exporter is adopted. +- Keep SheetJS pinned to an integrity-locked official tarball and re-run the + dependency policy, parser comparison corpus, and worker suite for upgrades. ## Research Only diff --git a/docs/V05_MANAGED_MATERIALIZATION.md b/docs/V05_MANAGED_MATERIALIZATION.md new file mode 100644 index 0000000..c73e427 --- /dev/null +++ b/docs/V05_MANAGED_MATERIALIZATION.md @@ -0,0 +1,77 @@ +# v0.5 encrypted managed-object materialization + +Status: **Implemented for offline operator use**. This is not a runtime domain +importer, production deployment claim, cloud-KMS claim, Marketplace claim, or +evidence that a customer migration has run. + +The materializer consumes an already imported neutral v0.5 bundle and copies +its exact bytes into the opt-in encrypted managed-object store. It does not +register a workbook, document, workflow, CRDT document, job, or legacy model; +it does not mutate the v0.5 source or provide reverse conversion or dual write. + +## Trust inputs + +The operator must receive the exact `import_id` and `binding_sha256` through +the approved external channel and must provide the intended tenant and owner. +The neutral receipt is a canonical SHA-256 integrity digest. It is **not a +digital signature** and is never treated as independent provenance. The reader +revalidates the external binding, content ledger, live owner, derived target +session, lifecycle state, payload bytes, and neutral receipt in one database +snapshot before releasing bytes. + +## Command + +Configure the same `DB_ENGINE` and SQLite `DB_PATH` or PostgreSQL +`DATABASE_URL` used by neutral intake. Configure the opt-in +`MANAGED_OBJECT_STORE_*` root and keys through the deployment secret channel; +do not put keys on the command line. + +```bash +python -m backend.operations.v05_managed_materializer \ + --import-id v05-import- \ + --expected-binding-sha256 \ + --tenant-id \ + --owner-id +``` + +The command registers the encrypted-object purger with the same lifecycle +coordinator and runs inside its active-task fence. Every object write also +checks the durable lifecycle row and owner in the managed metadata +transaction. Exact payload rereads are required, the entire neutral bundle is +re-read immediately before publication, and the deterministic manifest is a +final lifecycle-fenced managed-object write followed by an exact reread. + +Content-addressed writes make retries converge after a crash. Concurrent +operators produce the same payload object IDs, manifest bytes, manifest object +ID, and receipt. Multiple source paths with identical bytes remain distinct +manifest rows and may intentionally reference one managed object. Partial +objects are not a published conversion because the manifest is written last. + +## Conservative in-memory limits + +The reviewed snapshot reader currently returns a bounded in-memory bundle; it +is not a streaming API. It therefore fails closed before fetching payload +bytes when any of these materialization-specific limits is exceeded: + +- 2,048 payloads; +- 16 MiB for one payload; +- 64 MiB total payload bytes. + +These are tighter than neutral archival intake. Raising them requires a +separately reviewed lifecycle-snapshot streaming/callback protocol with +backpressure. The machine-readable success receipt repeats the active limits. +Managed-store object and total-byte quotas are enforced independently. + +## Purge and evidence boundary + +Neutral and managed bytes use the same content-derived target session. A +lifecycle purge first revokes and joins active materialization, invokes the +registered encrypted-object purger, and only then deletes the neutral database +inventory. A verified deletion cannot leave a published manifest or its +payload objects. + +The operator receipt is deterministic and privacy-safe. It reports local +provider and reread status plus content, binding, manifest, source-receipt, and +scope digests. It always states `production_encryption_claim: false` and +`cloud_provider_contacted: false`; external deployment, key custody, host, and +customer-migration evidence remain separate requirements. diff --git a/docs/V05_VNEXT_MIGRATION.md b/docs/V05_VNEXT_MIGRATION.md new file mode 100644 index 0000000..5d23c73 --- /dev/null +++ b/docs/V05_VNEXT_MIGRATION.md @@ -0,0 +1,86 @@ +# v0.5 to vNext clean-break migration + +The migration is one-way and offline. It does not preserve `/api/v1`, the +v0.5 database schema, V2–V7 Python models, or the workbook-worker CLI as vNext +compatibility layers. + +## Export + +Run the exporter in the configured v0.5 environment. Source files are resolved +only from document records beneath `DATA_DIR/documents`; callers cannot provide +source filesystem paths. + +```bash +python scripts/export_v05_migration_bundle.py \ + --owner user_123 \ + --session session_abc \ + --output ./excel-agent-v05.zip +``` + +The deterministic ZIP contains canonical JSON session descriptors, neutral +saved-workflow recipes marked `requires_review`, and content-addressed managed +document blobs. Parsed document derivatives, conversations, tasks, audit and +replay data, caches/indexes, enterprise configuration, credentials, and legacy +model artifacts are excluded. Known secret-bearing workflow fields and token +patterns are redacted. Workflow step parameters are additionally restricted to +the documented portable subset for each legacy step kind; unknown configuration +is omitted and recorded in the redaction ledger instead of being guessed safe. +The companion `.sha256` file covers the exact ZIP bytes; +transport that digest through a trusted or signed release channel. + +## Validate and stage + +```bash +python vnext/scripts/import_v05_migration_bundle.py \ + --bundle ./excel-agent-v05.zip \ + --owner user_123 +``` + +The importer verifies the external archive digest, schema version, owner scope, +canonical paths, duplicate entries and JSON keys, entry and total size limits, +compression ratio, every payload digest, content-addressed blob paths, record +links, and the internal content ledger. It rejects traversal, symlinks, +unlisted data, unsupported versions, stale owner scope, and tampering. +Before staging, it independently re-applies the portable-field policy and scans +the remaining workflow text for credential keys, provider token shapes, +credential-bearing URLs, private keys, JWTs, and high-entropy opaque secrets. + +Success creates only +`vnext/data/migration-inbox//` with a +`validated_pending_import` receipt. It does not mutate a workbook, database, +CRDT document, object store, or live workflow registry. + +## Consume into the neutral intake store + +After transporting both verified digests through the approved channel, the +operator may copy the exact staged bytes into the configured lifecycle-owned +neutral intake store: + +```bash +python -m backend.operations.v05_import_consumer \ + --staging-path ./vnext/data/migration-inbox/ \ + --expected-archive-sha256 \ + --expected-content-sha256 \ + --target-owner-id user_123 +``` + +The consumer revalidates the canonical manifest, stage receipt, every record +and byte digest, owner binding, safe filesystem identity, and closed file +inventory before writing. SQLite and PostgreSQL implementations use the same +content-addressed lifecycle session, compare-and-set state machine, immutable +payload ledger, and idempotent receipt. A crash after the import claim resumes +the same attempt; a lifecycle purge removes the neutral payloads. + +This intake is deliberately not workbook materialization. It creates no +runtime workbook, document, CRDT state, or workflow, does not modify the v0.5 +source, and offers no reverse conversion or hidden dual write. + +## Materialize encrypted neutral objects + +The implemented, operator-only +[managed-object materializer](V05_MANAGED_MATERIALIZATION.md) can copy an +authenticated neutral import into tenant-, owner-, and target-session-scoped +encrypted managed objects. It publishes a deterministic content-addressed +manifest last and creates no runtime workbook, document row, CRDT state, or +executable workflow. A separately reviewed vNext domain importer is still +required before migration can be called complete. diff --git a/docs/WINDOWS_EXCEL_MANUAL_SMOKE.md b/docs/WINDOWS_EXCEL_MANUAL_SMOKE.md new file mode 100644 index 0000000..4b40871 --- /dev/null +++ b/docs/WINDOWS_EXCEL_MANUAL_SMOKE.md @@ -0,0 +1,180 @@ +# Windows Excel Manual Smoke Checklist + +This checklist captures manual Excel Desktop evidence for spreadsheet platform +releases. Use it for Phase 1 evidence and expand it for later live-sync release +candidates. + +## Run Metadata + +| Field | Value | +| --- | --- | +| Date | | +| Tester | | +| Git commit | | +| Windows version | | +| Excel version/build | | +| Add-in manifest | | +| Backend command | | +| Plugin command | | +| Worker command | | +| Test fixture set | | +| Evidence bundle path | | +| Screenshot/video directory | | +| Excel prompt/log evidence | | +| Preflight trust bundle ID | | +| Release owner signoff | | +| Overall result | Pass / Pass with Warning / Blocked / Fail | + +## Status Model + +- Pass: behavior works and evidence is attached. +- Pass with Warning: user-visible warning is correct and no silent loss occurs. +- Blocked: product blocks the action intentionally. +- Fail: behavior is wrong, silent, corrupting, or undocumented. +- Not Covered: not part of this smoke pass. + +## Validation Command + +Create a run-specific evidence pack before starting manual Excel Desktop smoke: + +```powershell +python scripts\scaffold_windows_excel_smoke.py +``` + +The scaffold creates a dated checklist copy, evidence directories, and an +evidence-slot manifest plus `run-metadata.json`. It does not mark any manual +evidence as complete. + +Use the validator while filling the checklist: + +```powershell +python scripts\validate_windows_excel_smoke.py --smoke-doc data\release-trust\\WINDOWS_EXCEL_MANUAL_SMOKE.md +``` + +Before release signoff, require strict evidence validation. Completed rows must +have a status, evidence notes, and at least one attachment reference; local +artifacts must exist, be non-placeholder, and live under the declared Evidence +bundle path: + +```powershell +python scripts\validate_windows_excel_smoke.py --smoke-doc data\release-trust\\WINDOWS_EXCEL_MANUAL_SMOKE.md --require-existing-artifacts +``` + +During manual smoke, record only non-strict automated preflight output. Attach +the local JSON from the `Preflight trust bundle snapshot` row under the evidence +bundle; strict validation rejects missing, stale, or wrong-commit preflight JSON. +The final strict release trust bundle is generated after this checklist and its +referenced artifacts validate, so it is not a prerequisite field in this table. + +After the checklist is complete, run the strict full-stack release evidence mode +to validate artifacts and generate the final release trust bundle: + +```powershell +.\scripts\verify-full-stack.ps1 -ReleaseEvidence -SmokeDocPath data\release-trust\\WINDOWS_EXCEL_MANUAL_SMOKE.md +``` + +The validator checks checklist completeness, required attachment references, +referenced artifacts, local artifact digests, scaffolded run metadata/evidence +slot manifests when present, and whether local artifacts are inside the declared +evidence bundle. Real Windows Excel Desktop evidence still has to be captured +manually. + +## Automated Preflight + +| Gate | Command | Result | Evidence / Notes | +| --- | --- | --- | --- | +| Full stack preflight | `.\scripts\verify-full-stack.ps1` | | | +| OpenAPI types | `python scripts\generate_openapi_types.py --check` | | | +| Plugin verify | `Push-Location plugin; npm run verify; Pop-Location` | | | +| Worker tests | `Push-Location workbook-worker; npm test; Pop-Location` | | | +| Parser comparison corpus | `python scripts\run_parser_comparison_corpus.py --fail-on-mismatch` | | | +| Dependency risk | `python scripts\check_dependency_risks.py` | | | +| Smoke checklist progress snapshot | `python scripts\validate_windows_excel_smoke.py` | | | +| Preflight trust bundle snapshot | `python scripts\generate_release_trust_bundle.py --mode metadata-only --label windows-excel-preflight` | | | +| Known skipped gates | | | | + +## Required Evidence Coverage + +| ID | Evidence Area | Required Capture | Status | Evidence / Notes | +| --- | --- | --- | --- | --- | +| EXCEL-ENV | Excel Desktop environment | Windows version, Excel version/build, manifest, commands, and fixture set recorded in Run Metadata | | | +| EXCEL-PREFLIGHT | Automated gate output | Non-strict full stack preflight, OpenAPI, plugin, worker, dependency risk, parser corpus, smoke progress, and preflight trust bundle outputs linked | | | +| EXCEL-FIXTURES | Fixture matrix | Every required workbook fixture row has a result and evidence path or note | | | +| EXCEL-LINK | Sideload and link proof | Sideload, open, taskpane load, link, and stable host identity evidence attached | | | +| EXCEL-SYNC | Edit/sync proof | Value, formula, format, comment/note, table, validation, and sheet/name sync evidence attached | | | +| EXCEL-RECONCILE | Drift and reconcile proof | Host drift detection plus Keep Browser, Reload From Excel, and Stay Paused evidence attached | | | +| EXCEL-RECOVERY | Failure recovery proof | Protected-sheet failure, apply details, resume behavior, and reload mid-queue evidence attached | | | +| EXCEL-EXPORT | Export fidelity proof | Preserved export, headers, Excel reopen/save/reopen, macro, and object preservation evidence attached | | | +| EXCEL-ARTIFACTS | Artifact bundle | Preflight trust JSON/Markdown, screenshots or video, logs, and any repair prompts stored under Evidence bundle path | | | + +## Fixture Matrix + +| Fixture | Contains | Open | Link | Edit | Sync | Export | Reopen | Save/Reopen | Result | Evidence | +| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | +| Simple `.xlsx` | Values/formulas/styles | | | | | | | | | | +| Object fidelity `.xlsx` | Table/comment/validation/CF/protection/name | | | | | | | | | | +| Macro `.xlsm` | VBA project | | | | | | | | | | +| Chart workbook | Chart package parts | | | | | | | | | | +| Pivot workbook | Pivot package parts | | | | | | | | | | +| Image/drawing workbook | Drawing/media parts | | | | | | | | | | +| Protected workbook | Sheet/workbook protection | | | | | | | | | | +| Large workbook | Large row/cell count | | | | | | | | | | + +## Sideload And Link + +| Step | Expected Result | Status | Evidence / Notes | +| --- | --- | --- | --- | +| Run `npm run sideload` from `plugin` | Add-in sideloads without certificate or manifest error | | | +| Open a normal `.xlsx` in Excel Desktop | Workbook opens without repair dialog | | | +| Open the taskpane | Taskpane loads and can reach backend | | | +| Link active workbook | Link mode shows `linked`, queue depth is 0, no initial conflict | | | +| Check host | Workbook identity/fingerprint remains stable | | | + +## Browser Edit To Excel Sync + +| Patch Type | Action | Expected Result | Status | Evidence / Notes | +| --- | --- | --- | --- | --- | +| Value | Edit one value in browser grid | Excel receives value after sync | | | +| Formula | Edit one formula in browser grid/formula bar | Excel receives formula and recalculates | | | +| Format | Apply bold/fill/number format | Excel receives visible format where supported | | | +| Comment/note | Add or edit comment | Excel receives comment, or visible warning explains unsupported path | | | +| Table | Create/update table metadata | Excel receives table change, or visible warning explains unsupported path | | | +| Validation | Add validation | Excel receives validation, or visible warning explains unsupported path | | | +| Sheet/name | Rename sheet or named range | Excel receives change where supported | | | + +## Host Drift And Reconcile + +| Step | Expected Result | Status | Evidence / Notes | +| --- | --- | --- | --- | +| Make an external Excel edit | Browser detects host drift before overwrite | | | +| Choose Keep Browser | Queued browser edits replay to Excel and journal records outcome | | | +| Repeat drift | Conflict appears again with range/object summary | | | +| Choose Reload From Excel | Browser session reloads from host snapshot and clears queue | | | +| Choose Stay Paused | Queue remains untouched and sync stays paused | | | + +## Failure And Recovery + +| Step | Expected Result | Status | Evidence / Notes | +| --- | --- | --- | --- | +| Attempt write to protected sheet | Sync pauses with protected-sheet failure | | | +| Review apply result | Rollback/apply details are visible | | | +| Resume after unprotecting or changing target | Queue can complete or remain safely paused | | | +| Reload browser/taskpane mid-queue | Queue and journal survive reload, or row is marked Not Covered for Phase 1 | | | + +## Export Fidelity + +| Step | Expected Result | Status | Evidence / Notes | +| --- | --- | --- | --- | +| Export preserved workbook | Primary export uses preserved OOXML when original bytes exist | | | +| Check headers where visible | `X-Spreadsheet-Fidelity-Mode`, `X-Spreadsheet-Fidelity-Status`, and `X-Spreadsheet-Unsupported-Preserved` are correct | | | +| Reopen export in Excel Desktop | No repair dialog, missing sheets, or obvious corruption | | | +| Save from Excel | Save completes without repair prompts | | | +| Reopen saved copy | Workbook still opens cleanly | | | +| Macro workbook export | VBA is preserved, macro execution remains blocked/warned in browser | | | +| Chart/pivot/image workbook export | Objects are preserved or warnings/blockers are explicit | | | + +## Blockers And Follow-Up + +| ID | Severity | Area | Description | Owner | Target Fix | +| --- | --- | --- | --- | --- | --- | +| | | | | | | diff --git a/docs/ZERO_HOSTED_MINUTES_OPERATIONS.md b/docs/ZERO_HOSTED_MINUTES_OPERATIONS.md new file mode 100644 index 0000000..9a09b0f --- /dev/null +++ b/docs/ZERO_HOSTED_MINUTES_OPERATIONS.md @@ -0,0 +1,200 @@ +# Zero-hosted-minutes operator plane + +Excel Agent verification is designed to run on contributor machines or +operator-owned, single-job runners. Nothing in this plane falls back to a +GitHub-hosted label, and a cache miss never relaxes a committed dependency +lock. + +## Fresh contributor check + +Run the bootstrap from a fresh checkout before installing dependencies: + +```sh +scripts/bootstrap-contributor.sh --profile fast +``` + +On Windows: + +```powershell +.\scripts\bootstrap-contributor.ps1 -Profile fast +``` + +The wrapper contains no copied runtime versions. It delegates first to +`scripts/verify_toolchain_manifest.py`, whose source of truth is +`toolchain.lock.json`, and then to `scripts/toolchain_doctor.py`. Use +`--plan-only --json` (or `-PlanOnly -Json`) to inspect the exact delegated +commands without running them. Passing `--dependency-cache PATH` additionally +verifies that cache against the checkout; it does not silently install or +restore anything. + +## Export and import an offline cache + +Build the transfer on a disposable networked host. The target must be an empty +directory outside the checkout: + +```sh +scripts/export_dependency_cache.sh --output /approved-transfer/excel-agent-cache +``` + +The exporter uses the Python hash lock, all npm locks, the Cargo lock, the +pinned Rust and Buf image digests, and the pinned Actionlint Go module. It +builds an architecture-specific Rust tools image from the exact pinned base, +installs only the manifest-declared `clippy`, `rustfmt`, and +`wasm32-unknown-unknown` components, verifies them without network access, and +stores that image beside the unchanged base image. The same manifest names +every WASM-capable vNext package and the native-only `service-host` exclusion; +an unclassified workspace member is a hard failure. It +preserves dependency locks below `locks/` and emits +`dependency-cache-manifest.json`. The manifest binds every regular file and +directory to: + +- its exact SHA-256 digest and size; +- the exact `toolchain.lock.json` digest; and +- the exact dependency-lock set and digests from the checkout. + +Independent valid no-cache builds may produce different derived image/tar +bytes because BuildKit/OCI creation, attestation, and archive metadata are not +the release authority. The verified base digest, Dockerfile/toolchain-lock +digest, architecture, labels, component/target versions, and tool versions are +authoritative. The cache manifest then binds the one concrete exported tools +tar by digest and size, and offline consumption rechecks those exact bytes +before loading them. Never invent or source-pin a universal derived-image +digest. + +At the receiving checkout, verify before using the transfer: + +```sh +python scripts/dependency_cache.py verify \ + --cache /received/excel-agent-cache --json +``` + +Verification rejects missing or extra files/directories, digest or size drift, +foreign toolchain/lock state, absolute or traversing manifest paths, symlinks, +and non-regular filesystem objects. A plain `shasum -c` is useful transport +diagnostics but is not a substitute for this semantic verification. + +Restore into an absent destination, or opt in explicitly to replacement: + +```sh +python scripts/dependency_cache.py restore \ + --source /received/excel-agent-cache \ + --destination /runner-cache/excel-agent --json + +python scripts/dependency_cache.py restore \ + --source /received/excel-agent-cache \ + --destination /runner-cache/excel-agent --replace --json +``` + +Restore verifies the source, copies to a sibling staging directory, verifies +the staged copy, and then atomically swaps it into place. A corrupt transfer +does not modify an existing destination. Concurrent restores to the same name +are rejected. + +After restoring, emit the explicit offline-consumption environment: + +```sh +python scripts/dependency_cache.py environment \ + --cache /runner-cache/excel-agent --shell sh > /tmp/excel-agent-cache.env +. /tmp/excel-agent-cache.env +``` + +PowerShell and JSON renderings are also available with `--shell powershell` +and `--shell json`. The environment sets pip no-index/find-links, npm offline +mode and per-package cache locations, Go proxy-off/cache locations, Cargo +offline/vendor paths, and the saved Rust base/tools and Buf image paths. The +Rust runner loads the verified tools archive itself and refuses to build, pull, +or enable networking in offline mode: + +```sh +python scripts/rust_toolchain.py verify-vnext \ + --offline-cache /runner-cache/excel-agent +``` + +The same path is available through +`scripts/verify-local.sh --profile vnext --dependency-cache PATH`. Keep the +exact base digest and component/target declarations in `toolchain.lock.json` +authoritative. The runner compiles only the explicit WASM package allowlist, +while formatting, Clippy, and tests still cover the full workspace. A derived +local image tag is never a release identity. + +The cache is untrusted executable input even when all digests match its own +manifest. Transfer it through the normal artifact review boundary, bind it to +an approved source SHA, and never import caches from pull requests or unknown +authors. + +## Clean-workspace gate + +Before a runner checkout is materialized, require an absent or empty directory: + +```sh +python scripts/enforce_clean_workspace.py empty \ + --path /approved-runner-base/excel-agent-job-123 --create --json +``` + +After checkout, bind verification to a full commit SHA and reject tracked or +untracked changes: + +```sh +python scripts/enforce_clean_workspace.py git \ + --path /checkout --expected-sha "$COMMIT_SHA" --json +``` + +The command rejects symlink roots and nested Git working directories. Do not +"clean" an unexpected directory in place; discard the ephemeral machine. + +## Ephemeral Linux runner + +Stage an official Actions runner archive and obtain its SHA-256 digest through +an independent operator channel. Use a short-lived registration token in an +environment variable, never a checked-in file: + +```sh +export EXCEL_AGENT_RUNNER_TOKEN='short-lived-registration-token' +infra/ci/bootstrap-linux-runner.sh \ + --runner-archive /staging/actions-runner-linux-x64.tar.gz \ + --archive-sha256 "$RUNNER_ARCHIVE_SHA256" \ + --url https://github.com/OWNER/REPOSITORY +``` + +The bootstrap refuses root, validates a credential-free HTTPS registration +URL, verifies the archive digest, safely extracts without links or traversal, +uses the fixed `self-hosted,linux,x64,excel-agent` label contract, and passes +`--ephemeral --disableupdate`. It runs one job and invokes +`cleanup-linux-runner.sh` on success, failure, or signal. Cleanup is destructive +only after the exact root/base tuple is authenticated by +`.excel-agent-ephemeral-runner.json`. + +## Ephemeral Windows + Excel runner + +Use a dedicated, non-administrator Windows account on a disposable VM with +desktop Microsoft Excel installed. In PowerShell: + +```powershell +$env:EXCEL_AGENT_RUNNER_TOKEN = 'short-lived-registration-token' +.\infra\ci\Bootstrap-WindowsExcelRunner.ps1 ` + -RunnerArchive C:\Staging\actions-runner-win-x64.zip ` + -ArchiveSha256 $env:RUNNER_ARCHIVE_SHA256 ` + -Url https://github.com/OWNER/REPOSITORY +``` + +The script verifies the same one-job configuration contract, digest and safe +archive extraction, proves that the Excel COM application can start and report +a version, registers fixed +`self-hosted,windows,x64,excel-agent-excel` labels, and always calls +`Cleanup-WindowsExcelRunner.ps1`. It never installs the runner as a service. + +## Configuration authority and failure behavior + +`infra/ci/runner-contract.json` is the machine-readable authority for labels, +token source, one-job lifetime, no-service policy, clean workspace, and +mandatory cleanup. Validate it locally with: + +```sh +python scripts/runner_contract.py verify-contract --json +``` + +If registration, cache verification, Excel detection, archive verification, +or cleanup-target validation fails, stop. Never substitute `ubuntu-latest` or +`windows-latest`, reuse a partially configured runner, weaken a digest check, +or retain the job workspace for the next run. Preserve only reviewed evidence +outside the runner root and destroy the VM or machine snapshot. diff --git a/docs/assets/browser-workbook-preview.png b/docs/assets/browser-workbook-preview.png new file mode 100644 index 0000000..24e8ab5 Binary files /dev/null and b/docs/assets/browser-workbook-preview.png differ diff --git a/docs/documentation-contract.json b/docs/documentation-contract.json new file mode 100644 index 0000000..ff457a8 --- /dev/null +++ b/docs/documentation-contract.json @@ -0,0 +1,72 @@ +{ + "schemaVersion": 1, + "activeDocuments": [ + "README.md", + "CONTRIBUTING.md", + "docs/README.md", + "docs/API_COOKBOOK.md", + "docs/ARCHITECTURE.md", + "docs/GETTING_STARTED.md", + "docs/IMPLEMENTATION_STATUS.md", + "docs/NATIVE_HOST_CERTIFICATION.md", + "docs/OFFLINE_DISTRIBUTION.md", + "docs/OPERATIONS_INDEX.md", + "docs/PRODUCTION_RUNBOOK.md", + "docs/PROGRAM_COMPLETION_ROADMAP.md", + "docs/RELEASE_CHECKLIST.md", + "docs/RELEASE_EVIDENCE_PORTAL.md", + "docs/RELEASE_PROMOTION.md", + "docs/ROADMAP_INDEX.md", + "docs/SELF_HOSTED_VERIFICATION.md", + "docs/STACK_CAPABILITY_MATRIX.md", + "docs/THIRD_PARTY_NOTICES.md", + "docs/V05_MANAGED_MATERIALIZATION.md", + "docs/V05_VNEXT_MIGRATION.md", + "docs/WINDOWS_EXCEL_MANUAL_SMOKE.md", + "docs/ZERO_HOSTED_MINUTES_OPERATIONS.md", + "vnext/README.md", + "vnext/crates/service-host/README.md", + "vnext/docs/IMPLEMENTATION_ROADMAP.md", + "vnext/examples/README.md", + "vnext/sdk/README.md" + ], + "historicalDocuments": [ + "docs/EXTERNAL_CODEBASE_RESEARCH.md", + "docs/FLAGSHIP_V5_IMPLEMENTATION_PLAN.md" + ], + "runnableExamples": [ + { + "path": "examples/no-cloud-local-quickstart.sh", + "language": "sh", + "executable": true + }, + { + "path": "examples/openapi_inspect.py", + "language": "python", + "executable": true + }, + { + "path": "vnext/scripts/verify-sdk-examples.sh", + "language": "sh", + "executable": true + } + ], + "requiredStatusLabels": [ + "Implemented", + "Experimental", + "Designed", + "Blocked" + ], + "requiredOpenApiOperations": [ + "DELETE /api/v1/sessions/{session_id}", + "GET /api/v1/capabilities/office", + "GET /api/v1/contracts/operation-registry", + "GET /api/v1/sessions/{session_id}/timeline", + "POST /api/v1/contracts/operation-plan-v2/validate", + "POST /api/v1/sessions/{session_id}/rollback", + "POST /api/v1/sessions/{session_id}/rollback/result", + "POST /api/v1/tasks/apply", + "POST /api/v1/tasks/apply/result", + "POST /api/v1/tasks/plan" + ] +} diff --git a/examples/no-cloud-local-quickstart.sh b/examples/no-cloud-local-quickstart.sh new file mode 100755 index 0000000..52155b1 --- /dev/null +++ b/examples/no-cloud-local-quickstart.sh @@ -0,0 +1,77 @@ +#!/bin/sh +# Exercise a supported API surface with disposable local state and no cloud model. +set -eu + +REPOSITORY_ROOT=$(CDPATH= cd -- "$(dirname -- "$0")/.." && pwd) +if [ -n "${EXCEL_AGENT_PYTHON:-}" ]; then + PYTHON=$EXCEL_AGENT_PYTHON +elif [ -x "$REPOSITORY_ROOT/.venv/bin/python" ]; then + PYTHON=$REPOSITORY_ROOT/.venv/bin/python +else + PYTHON=python3 +fi +if [ -n "${EXCEL_AGENT_QUICKSTART_PORT:-}" ]; then + PORT=$EXCEL_AGENT_QUICKSTART_PORT +else + PORT=$("$PYTHON" -c 'import socket; s=socket.socket(); s.bind(("127.0.0.1", 0)); print(s.getsockname()[1]); s.close()') +fi +QUICKSTART_ROOT=$(mktemp -d "${TMPDIR:-/tmp}/excel-agent-quickstart.XXXXXX") +SERVER_PID= + +cleanup() { + if [ -n "$SERVER_PID" ]; then + kill "$SERVER_PID" 2>/dev/null || true + wait "$SERVER_PID" 2>/dev/null || true + fi + rm -rf "$QUICKSTART_ROOT" +} +trap cleanup EXIT HUP INT TERM + +mkdir -p "$QUICKSTART_ROOT/data" +export CONFIG_PROFILE=dev +export DATA_DIR="$QUICKSTART_ROOT/data" +export DB_PATH="$QUICKSTART_ROOT/data/sessions.db" +export AUTH_ENABLED=true +export API_KEY=local-disposable-quickstart-key +export STATIC_KEY_ADMIN_ENABLED=true +export DEFAULT_PROVIDER=ollama +export OLLAMA_BASE_URL=http://127.0.0.1:11434 +unset ANTHROPIC_API_KEY OPENAI_API_KEY + +cd "$REPOSITORY_ROOT" +"$PYTHON" -m uvicorn backend.main:app --host 127.0.0.1 --port "$PORT" \ + >"$QUICKSTART_ROOT/server.log" 2>&1 & +SERVER_PID=$! + +attempt=0 +until curl --fail --silent "http://127.0.0.1:$PORT/health" \ + >"$QUICKSTART_ROOT/health.json" 2>/dev/null; do + attempt=$((attempt + 1)) + if [ "$attempt" -ge 30 ]; then + printf '%s\n' "Backend did not become ready; log follows:" >&2 + sed -n '1,200p' "$QUICKSTART_ROOT/server.log" >&2 + exit 1 + fi + sleep 1 +done + +curl --fail --silent --show-error \ + -H "X-API-Key: $API_KEY" \ + "http://127.0.0.1:$PORT/api/v1/contracts/operation-registry" \ + >"$QUICKSTART_ROOT/registry.json" + +"$PYTHON" - "$QUICKSTART_ROOT/health.json" "$QUICKSTART_ROOT/registry.json" <<'PY' +import json +import sys + +health = json.load(open(sys.argv[1], encoding="utf-8")) +registry = json.load(open(sys.argv[2], encoding="utf-8")) +assert health["status"] == "alive" +assert isinstance(registry.get("schema_version"), str) +assert registry["operations"] +print(json.dumps({"health": health["status"], "operation_count": len(registry["operations"])}, indent=2)) +PY + +"$PYTHON" examples/openapi_inspect.py \ + --operation GET /api/v1/contracts/operation-registry >/dev/null +printf '%s\n' "No-cloud API quickstart passed; disposable state will now be removed." diff --git a/examples/openapi_inspect.py b/examples/openapi_inspect.py new file mode 100755 index 0000000..ab2d657 --- /dev/null +++ b/examples/openapi_inspect.py @@ -0,0 +1,86 @@ +#!/usr/bin/env python3 +"""Inspect the current FastAPI OpenAPI document without copying schemas.""" + +from __future__ import annotations + +import argparse +import contextlib +import io +import json +import sys +from pathlib import Path +from typing import Any, Sequence + + +ROOT = Path(__file__).resolve().parents[1] +sys.path.insert(0, str(ROOT)) + + +def generated_openapi() -> dict[str, Any]: + captured = io.StringIO() + with contextlib.redirect_stdout(captured), contextlib.redirect_stderr(captured): + from backend.main import app + + schema = app.openapi() + if not isinstance(schema, dict) or not isinstance(schema.get("paths"), dict): + raise ValueError("FastAPI did not generate an OpenAPI path inventory") + return schema + + +def operation(schema: dict[str, Any], method: str, path: str) -> dict[str, Any]: + paths = schema["paths"] + if not isinstance(paths, dict): + raise ValueError("OpenAPI path inventory is invalid") + path_item = paths.get(path) + value = path_item.get(method.lower()) if isinstance(path_item, dict) else None + if not isinstance(value, dict): + raise ValueError(f"unknown OpenAPI operation: {method.upper()} {path}") + return value + + +def component(schema: dict[str, Any], name: str) -> dict[str, Any]: + components = schema.get("components") + schemas = components.get("schemas") if isinstance(components, dict) else None + value = schemas.get(name) if isinstance(schemas, dict) else None + if not isinstance(value, dict): + raise ValueError(f"unknown OpenAPI component schema: {name}") + return value + + +def _parser() -> argparse.ArgumentParser: + parser = argparse.ArgumentParser(description=__doc__) + group = parser.add_mutually_exclusive_group() + group.add_argument("--operation", nargs=2, metavar=("METHOD", "PATH")) + group.add_argument("--schema", metavar="NAME") + group.add_argument("--list", action="store_true") + return parser + + +def main(argv: Sequence[str] | None = None) -> int: + args = _parser().parse_args(argv) + try: + schema = generated_openapi() + if args.operation: + payload: object = operation(schema, args.operation[0], args.operation[1]) + elif args.schema: + payload = component(schema, args.schema) + else: + paths = schema["paths"] + if not isinstance(paths, dict): + raise ValueError("OpenAPI path inventory is invalid") + payload = [ + f"{method.upper()} {path}" + for path, path_item in sorted(paths.items()) + if isinstance(path_item, dict) + for method in sorted(path_item) + if method.lower() in {"get", "post", "put", "patch", "delete"} + ] + except ValueError as exc: + print(str(exc), file=sys.stderr) + return 1 + print(json.dumps(payload, indent=2, sort_keys=True)) + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/infra/ci/Bootstrap-WindowsExcelRunner.ps1 b/infra/ci/Bootstrap-WindowsExcelRunner.ps1 new file mode 100644 index 0000000..5f81f6d --- /dev/null +++ b/infra/ci/Bootstrap-WindowsExcelRunner.ps1 @@ -0,0 +1,94 @@ +[CmdletBinding()] +param( + [Parameter(Mandatory = $true)] + [string]$RunnerArchive, + [Parameter(Mandatory = $true)] + [ValidatePattern("^[0-9a-fA-F]{64}$")] + [string]$ArchiveSha256, + [string]$Url = $env:EXCEL_AGENT_RUNNER_URL, + [string]$RunnerBase = "C:\ExcelAgentRunners", + [string]$Name = "excel-agent-windows-$([Environment]::MachineName)-$PID", + [string]$Python = "python" +) + +$ErrorActionPreference = "Stop" +Set-StrictMode -Version Latest +$RepositoryRoot = Split-Path -Parent (Split-Path -Parent $PSScriptRoot) +$RunnerContract = Join-Path $RepositoryRoot "scripts\runner_contract.py" +$WorkspaceEnforcer = Join-Path $RepositoryRoot "scripts\enforce_clean_workspace.py" +$CleanupScript = Join-Path $PSScriptRoot "Cleanup-WindowsExcelRunner.ps1" + +if (-not $env:EXCEL_AGENT_RUNNER_TOKEN) { + throw "EXCEL_AGENT_RUNNER_TOKEN must contain a short-lived registration token" +} +$RegistrationToken = $env:EXCEL_AGENT_RUNNER_TOKEN +Remove-Item Env:EXCEL_AGENT_RUNNER_TOKEN -ErrorAction SilentlyContinue +if (-not $Url) { + throw "-Url or EXCEL_AGENT_RUNNER_URL is required" +} +$Identity = [Security.Principal.WindowsIdentity]::GetCurrent() +$Principal = [Security.Principal.WindowsPrincipal]::new($Identity) +if ($Principal.IsInRole([Security.Principal.WindowsBuiltInRole]::Administrator)) { + throw "Run the ephemeral runner as a dedicated unprivileged account" +} + +& $Python $RunnerContract verify-contract | Out-Null +if ($LASTEXITCODE -ne 0) { throw "Runner configuration contract failed" } +& $Python $RunnerContract validate-url --url $Url | Out-Null +if ($LASTEXITCODE -ne 0) { throw "Runner URL contract failed" } +& $Python $RunnerContract verify-archive --archive $RunnerArchive --sha256 $ArchiveSha256 | Out-Null +if ($LASTEXITCODE -ne 0) { throw "Runner archive digest verification failed" } + +New-Item -ItemType Directory -Path $RunnerBase -Force | Out-Null +$RunnerBase = (Resolve-Path -LiteralPath $RunnerBase).Path +$RunnerRoot = Join-Path $RunnerBase $Name +& $Python $WorkspaceEnforcer empty --path $RunnerRoot --create | Out-Null +if ($LASTEXITCODE -ne 0) { throw "Runner workspace is not clean" } +& $Python $RunnerContract write-marker --runner-root $RunnerRoot --runner-base $RunnerBase --platform windowsExcel | Out-Null +if ($LASTEXITCODE -ne 0) { throw "Runner cleanup marker creation failed" } + +$ExitCode = 1 +$LocationPushed = $false +try { + & $Python $RunnerContract extract-archive --archive $RunnerArchive --destination $RunnerRoot --archive-type zip --sha256 $ArchiveSha256 | Out-Null + if ($LASTEXITCODE -ne 0) { throw "Runner archive extraction failed" } + + $Excel = $null + try { + $Excel = New-Object -ComObject Excel.Application + if (-not $Excel.Version) { throw "Microsoft Excel version could not be read" } + } + finally { + if ($null -ne $Excel) { + $Excel.Quit() + [Runtime.InteropServices.Marshal]::FinalReleaseComObject($Excel) | Out-Null + } + } + + Push-Location $RunnerRoot + $LocationPushed = $true + $Config = Join-Path $RunnerRoot "config.cmd" + $Run = Join-Path $RunnerRoot "run.cmd" + if (-not (Test-Path -LiteralPath $Config -PathType Leaf)) { throw "Runner archive is missing config.cmd" } + if (-not (Test-Path -LiteralPath $Run -PathType Leaf)) { throw "Runner archive is missing run.cmd" } + & $Config --unattended --ephemeral --disableupdate --url $Url ` + --token $RegistrationToken --name $Name ` + --labels "self-hosted,windows,x64,excel-agent-excel" --work _work + if ($LASTEXITCODE -ne 0) { throw "Runner configuration failed" } + & $Run + $ExitCode = $LASTEXITCODE +} +finally { + if ($LocationPushed) { + Pop-Location -ErrorAction SilentlyContinue + } + $env:EXCEL_AGENT_RUNNER_TOKEN = $RegistrationToken + try { + & $CleanupScript -RunnerRoot $RunnerRoot -RunnerBase $RunnerBase -Python $Python + } + finally { + Remove-Item Env:EXCEL_AGENT_RUNNER_TOKEN -ErrorAction SilentlyContinue + $RegistrationToken = $null + } +} +exit $ExitCode diff --git a/infra/ci/Cleanup-WindowsExcelRunner.ps1 b/infra/ci/Cleanup-WindowsExcelRunner.ps1 new file mode 100644 index 0000000..b1a34f5 --- /dev/null +++ b/infra/ci/Cleanup-WindowsExcelRunner.ps1 @@ -0,0 +1,37 @@ +[CmdletBinding()] +param( + [Parameter(Mandatory = $true)] + [string]$RunnerRoot, + [string]$RunnerBase = "C:\ExcelAgentRunners", + [string]$Python = "python" +) + +$ErrorActionPreference = "Stop" +Set-StrictMode -Version Latest +$RepositoryRoot = Split-Path -Parent (Split-Path -Parent $PSScriptRoot) +$RunnerContract = Join-Path $RepositoryRoot "scripts\runner_contract.py" + +# Destructive cleanup is authorized only after the exact root/base/platform +# tuple is authenticated by .excel-agent-ephemeral-runner.json. +& $Python $RunnerContract validate-cleanup-target ` + --runner-root $RunnerRoot --runner-base $RunnerBase --platform windowsExcel | Out-Null +if ($LASTEXITCODE -ne 0) { throw "Runner cleanup target validation failed" } + +$Config = Join-Path $RunnerRoot "config.cmd" +if ((Test-Path -LiteralPath $Config -PathType Leaf) -and $env:EXCEL_AGENT_RUNNER_TOKEN) { + Push-Location $RunnerRoot + try { + & $Config remove --unattended --token $env:EXCEL_AGENT_RUNNER_TOKEN + } + catch { + Write-Warning "Runner deregistration failed; ephemeral registration still expires after one job: $_" + } + finally { + Pop-Location + Remove-Item Env:EXCEL_AGENT_RUNNER_TOKEN -ErrorAction SilentlyContinue + } +} + +& $Python $RunnerContract cleanup-runner ` + --runner-root $RunnerRoot --runner-base $RunnerBase --platform windowsExcel | Out-Null +if ($LASTEXITCODE -ne 0) { throw "Runner cleanup failed closed" } diff --git a/infra/ci/bootstrap-linux-runner.sh b/infra/ci/bootstrap-linux-runner.sh new file mode 100755 index 0000000..81cd154 --- /dev/null +++ b/infra/ci/bootstrap-linux-runner.sh @@ -0,0 +1,103 @@ +#!/bin/sh +# Configure and execute exactly one ephemeral Linux Actions runner job. +set -eu +umask 077 + +REPOSITORY_ROOT=$(CDPATH= cd -- "$(dirname -- "$0")/../.." && pwd) +PYTHON=${EXCEL_AGENT_PYTHON:-python3} +RUNNER_BASE=/var/tmp/excel-agent-runners +RUNNER_NAME="excel-agent-linux-$(hostname | tr -cd 'A-Za-z0-9._-' | cut -c1-40)-$$" +RUNNER_ARCHIVE= +ARCHIVE_SHA256= +RUNNER_URL=${EXCEL_AGENT_RUNNER_URL:-} + +usage() { + cat <<'EOF' +Usage: infra/ci/bootstrap-linux-runner.sh \ + --runner-archive FILE --archive-sha256 SHA256 --url HTTPS_URL \ + [--runner-base DIRECTORY] [--name excel-agent-NAME] + +EXCEL_AGENT_RUNNER_TOKEN must contain a short-lived registration token. +The runner archive is operator-staged and must have an independently obtained +SHA-256 digest. The runner is configured with --ephemeral --disableupdate, +executes one job, and is removed by cleanup-linux-runner.sh on every exit path. +EOF +} + +while [ "$#" -gt 0 ]; do + case "$1" in + --runner-archive) [ "$#" -ge 2 ] || exit 2; RUNNER_ARCHIVE=$2; shift 2 ;; + --archive-sha256) [ "$#" -ge 2 ] || exit 2; ARCHIVE_SHA256=$2; shift 2 ;; + --url) [ "$#" -ge 2 ] || exit 2; RUNNER_URL=$2; shift 2 ;; + --runner-base) [ "$#" -ge 2 ] || exit 2; RUNNER_BASE=$2; shift 2 ;; + --name) [ "$#" -ge 2 ] || exit 2; RUNNER_NAME=$2; shift 2 ;; + -h|--help) usage; exit 0 ;; + *) echo "Unknown argument: $1" >&2; usage >&2; exit 2 ;; + esac +done + +[ "$(id -u)" -ne 0 ] || { echo "Run the ephemeral runner as a dedicated unprivileged account" >&2; exit 1; } +[ -n "$RUNNER_ARCHIVE" ] || { echo "--runner-archive is required" >&2; exit 2; } +[ -n "$ARCHIVE_SHA256" ] || { echo "--archive-sha256 is required" >&2; exit 2; } +[ -n "$RUNNER_URL" ] || { echo "--url or EXCEL_AGENT_RUNNER_URL is required" >&2; exit 2; } +[ -n "${EXCEL_AGENT_RUNNER_TOKEN:-}" ] || { echo "EXCEL_AGENT_RUNNER_TOKEN is required" >&2; exit 2; } +REGISTRATION_TOKEN=$EXCEL_AGENT_RUNNER_TOKEN +unset EXCEL_AGENT_RUNNER_TOKEN + +"$PYTHON" "$REPOSITORY_ROOT/scripts/runner_contract.py" verify-contract >/dev/null +"$PYTHON" "$REPOSITORY_ROOT/scripts/runner_contract.py" validate-url --url "$RUNNER_URL" >/dev/null +"$PYTHON" "$REPOSITORY_ROOT/scripts/runner_contract.py" verify-archive \ + --archive "$RUNNER_ARCHIVE" --sha256 "$ARCHIVE_SHA256" >/dev/null + +mkdir -p "$RUNNER_BASE" +RUNNER_BASE=$(CDPATH= cd -- "$RUNNER_BASE" && pwd) +RUNNER_ROOT=$RUNNER_BASE/$RUNNER_NAME +"$PYTHON" "$REPOSITORY_ROOT/scripts/enforce_clean_workspace.py" empty \ + --path "$RUNNER_ROOT" --create >/dev/null +"$PYTHON" "$REPOSITORY_ROOT/scripts/runner_contract.py" write-marker \ + --runner-root "$RUNNER_ROOT" --runner-base "$RUNNER_BASE" --platform linux >/dev/null + +RUNNER_PID= +cleanup() { + case "$RUNNER_PID" in + ''|*[!0-9]*) ;; + *) + kill -TERM -- "-$RUNNER_PID" 2>/dev/null || kill -TERM "$RUNNER_PID" 2>/dev/null || true + ;; + esac + EXCEL_AGENT_RUNNER_TOKEN=$REGISTRATION_TOKEN \ + "$REPOSITORY_ROOT/infra/ci/cleanup-linux-runner.sh" \ + --runner-root "$RUNNER_ROOT" --runner-base "$RUNNER_BASE" || true +} +trap cleanup EXIT HUP INT TERM + +"$PYTHON" "$REPOSITORY_ROOT/scripts/runner_contract.py" extract-archive \ + --archive "$RUNNER_ARCHIVE" --destination "$RUNNER_ROOT" --archive-type tar.gz \ + --sha256 "$ARCHIVE_SHA256" >/dev/null + +cd "$RUNNER_ROOT" +[ -x ./config.sh ] || { echo "Runner archive does not contain executable config.sh" >&2; exit 1; } +[ -x ./run.sh ] || { echo "Runner archive does not contain executable run.sh" >&2; exit 1; } + +set +x +./config.sh \ + --unattended \ + --ephemeral \ + --disableupdate \ + --url "$RUNNER_URL" \ + --token "$REGISTRATION_TOKEN" \ + --name "$RUNNER_NAME" \ + --labels "self-hosted,linux,x64,excel-agent" \ + --work _work + +set +e +if command -v setsid >/dev/null 2>&1; then + setsid ./run.sh & +else + ./run.sh & +fi +RUNNER_PID=$! +wait "$RUNNER_PID" +RUNNER_STATUS=$? +set -e +exit "$RUNNER_STATUS" diff --git a/infra/ci/cleanup-linux-runner.sh b/infra/ci/cleanup-linux-runner.sh new file mode 100755 index 0000000..ac7fc65 --- /dev/null +++ b/infra/ci/cleanup-linux-runner.sh @@ -0,0 +1,36 @@ +#!/bin/sh +# Remove only an exact, marker-bound ephemeral Linux runner directory. +set -eu + +REPOSITORY_ROOT=$(CDPATH= cd -- "$(dirname -- "$0")/../.." && pwd) +PYTHON=${EXCEL_AGENT_PYTHON:-python3} +RUNNER_ROOT= +RUNNER_BASE=/var/tmp/excel-agent-runners + +usage() { + echo "Usage: infra/ci/cleanup-linux-runner.sh --runner-root DIRECTORY [--runner-base DIRECTORY]" +} + +while [ "$#" -gt 0 ]; do + case "$1" in + --runner-root) [ "$#" -ge 2 ] || exit 2; RUNNER_ROOT=$2; shift 2 ;; + --runner-base) [ "$#" -ge 2 ] || exit 2; RUNNER_BASE=$2; shift 2 ;; + -h|--help) usage; exit 0 ;; + *) echo "Unknown argument: $1" >&2; usage >&2; exit 2 ;; + esac +done + +[ -n "$RUNNER_ROOT" ] || { echo "--runner-root is required" >&2; exit 2; } +"$PYTHON" "$REPOSITORY_ROOT/scripts/runner_contract.py" validate-cleanup-target \ + --runner-root "$RUNNER_ROOT" --runner-base "$RUNNER_BASE" --platform linux >/dev/null + +if [ -x "$RUNNER_ROOT/config.sh" ] && [ -n "${EXCEL_AGENT_RUNNER_TOKEN:-}" ]; then + set +x + (cd "$RUNNER_ROOT" && ./config.sh remove --unattended --token "$EXCEL_AGENT_RUNNER_TOKEN") || true + unset EXCEL_AGENT_RUNNER_TOKEN +fi + +# Reauthenticate .excel-agent-ephemeral-runner.json and delete through the +# marker-bound descriptor-safe primitive. +"$PYTHON" "$REPOSITORY_ROOT/scripts/runner_contract.py" cleanup-runner \ + --runner-root "$RUNNER_ROOT" --runner-base "$RUNNER_BASE" --platform linux >/dev/null diff --git a/infra/ci/runner-contract.json b/infra/ci/runner-contract.json new file mode 100644 index 0000000..92c0bf7 --- /dev/null +++ b/infra/ci/runner-contract.json @@ -0,0 +1,41 @@ +{ + "schemaVersion": "excel-agent-ephemeral-runner-v1", + "noHostedFallback": true, + "registration": { + "disableUpdate": true, + "ephemeral": true, + "maximumJobs": 1, + "serviceInstallationAllowed": false, + "tokenEnvironment": "EXCEL_AGENT_RUNNER_TOKEN" + }, + "workspace": { + "cleanupRequired": true, + "markerFile": ".excel-agent-ephemeral-runner.json", + "rejectSymlinks": true, + "requireEmpty": true + }, + "platforms": { + "linux": { + "archiveType": "tar.gz", + "excelRequired": false, + "labels": [ + "self-hosted", + "linux", + "x64", + "excel-agent" + ], + "runnerBaseDefault": "/var/tmp/excel-agent-runners" + }, + "windowsExcel": { + "archiveType": "zip", + "excelRequired": true, + "labels": [ + "self-hosted", + "windows", + "x64", + "excel-agent-excel" + ], + "runnerBaseDefault": "C:\\ExcelAgentRunners" + } + } +} diff --git a/infra/ci/rust-tools.Dockerfile b/infra/ci/rust-tools.Dockerfile new file mode 100644 index 0000000..8275929 --- /dev/null +++ b/infra/ci/rust-tools.Dockerfile @@ -0,0 +1,31 @@ +ARG RUST_BASE_IMAGE=scratch +FROM ${RUST_BASE_IMAGE} + +ARG RUST_TOOLCHAIN +ARG RUST_COMPONENTS +ARG RUST_TARGETS +ARG RUST_TOOLCHAIN_LOCK_SHA256 +ARG RUST_TOOLS_DOCKERFILE_SHA256 +ARG RUST_BASE_IMAGE + +RUN set -eu; \ + test "$(rustc --version | awk '{print $2}')" = "$RUST_TOOLCHAIN"; \ + for component in $RUST_COMPONENTS; do \ + rustup component add --toolchain "$RUST_TOOLCHAIN" "$component"; \ + done; \ + for target in $RUST_TARGETS; do \ + rustup target add --toolchain "$RUST_TOOLCHAIN" "$target"; \ + done; \ + rustfmt --version; \ + cargo clippy --version; \ + for target in $RUST_TARGETS; do \ + rustup target list --toolchain "$RUST_TOOLCHAIN" --installed | grep -F -x "$target"; \ + done; \ + rm -rf /usr/local/rustup/downloads/* /usr/local/rustup/tmp/* + +LABEL com.excel-agent.rust.base-image="$RUST_BASE_IMAGE" \ + com.excel-agent.rust.components="$RUST_COMPONENTS" \ + com.excel-agent.rust.targets="$RUST_TARGETS" \ + com.excel-agent.rust.toolchain="$RUST_TOOLCHAIN" \ + com.excel-agent.rust.tools-dockerfile-sha256="$RUST_TOOLS_DOCKERFILE_SHA256" \ + com.excel-agent.toolchain-lock-sha256="$RUST_TOOLCHAIN_LOCK_SHA256" diff --git a/infra/distribution/Install-Windows.ps1 b/infra/distribution/Install-Windows.ps1 new file mode 100644 index 0000000..e369877 --- /dev/null +++ b/infra/distribution/Install-Windows.ps1 @@ -0,0 +1,18 @@ +[CmdletBinding()] +param( + [Parameter(ValueFromRemainingArguments = $true)] + [string[]]$RemainingArguments, + [string]$Python = "python" +) + +$ErrorActionPreference = "Stop" +Set-StrictMode -Version Latest +$Identity = [Security.Principal.WindowsIdentity]::GetCurrent() +$Principal = [Security.Principal.WindowsPrincipal]::new($Identity) +if ($Principal.IsInRole([Security.Principal.WindowsBuiltInRole]::Administrator)) { + throw "Run the offline installer as a dedicated unprivileged account" +} +$DistributionRoot = $PSScriptRoot +$Installer = Join-Path $DistributionRoot "tools\offline_install.py" +& $Python $Installer install --distribution $DistributionRoot @RemainingArguments +exit $LASTEXITCODE diff --git a/infra/distribution/Uninstall-Windows.ps1 b/infra/distribution/Uninstall-Windows.ps1 new file mode 100644 index 0000000..e5e58ef --- /dev/null +++ b/infra/distribution/Uninstall-Windows.ps1 @@ -0,0 +1,18 @@ +[CmdletBinding()] +param( + [Parameter(ValueFromRemainingArguments = $true)] + [string[]]$RemainingArguments, + [string]$Python = "python" +) + +$ErrorActionPreference = "Stop" +Set-StrictMode -Version Latest +$Identity = [Security.Principal.WindowsIdentity]::GetCurrent() +$Principal = [Security.Principal.WindowsPrincipal]::new($Identity) +if ($Principal.IsInRole([Security.Principal.WindowsBuiltInRole]::Administrator)) { + throw "Run the offline uninstaller as a dedicated unprivileged account" +} +$DistributionRoot = $PSScriptRoot +$Installer = Join-Path $DistributionRoot "tools\offline_install.py" +& $Python $Installer uninstall @RemainingArguments +exit $LASTEXITCODE diff --git a/infra/distribution/install.sh b/infra/distribution/install.sh new file mode 100755 index 0000000..5466357 --- /dev/null +++ b/infra/distribution/install.sh @@ -0,0 +1,8 @@ +#!/bin/sh +# Thin offline wrapper; authentication and transactional state live in Python. +set -eu + +DISTRIBUTION_ROOT=$(CDPATH= cd -- "$(dirname -- "$0")" && pwd) +PYTHON=${EXCEL_AGENT_PYTHON:-python3} +exec "$PYTHON" "$DISTRIBUTION_ROOT/tools/offline_install.py" install \ + --distribution "$DISTRIBUTION_ROOT" "$@" diff --git a/infra/distribution/uninstall.sh b/infra/distribution/uninstall.sh new file mode 100755 index 0000000..6f5c6c5 --- /dev/null +++ b/infra/distribution/uninstall.sh @@ -0,0 +1,7 @@ +#!/bin/sh +# Idempotent receipt-bound removal; no network or updater is invoked. +set -eu + +DISTRIBUTION_ROOT=$(CDPATH= cd -- "$(dirname -- "$0")" && pwd) +PYTHON=${EXCEL_AGENT_PYTHON:-python3} +exec "$PYTHON" "$DISTRIBUTION_ROOT/tools/offline_install.py" uninstall "$@" diff --git a/infra/mtls/README.md b/infra/mtls/README.md new file mode 100644 index 0000000..2ce6a29 --- /dev/null +++ b/infra/mtls/README.md @@ -0,0 +1,18 @@ +# Edge-to-backend mTLS material + +Production Compose requires files provisioned here by the deployment secret +manager. Never commit certificate private keys or copy them into an image. + +- `ca.crt`: private deployment CA chain trusted by both services. +- `backend-server.crt` / `backend-server.key`: backend server identity. Its SAN + must contain the Compose DNS name `backend`; EKU must allow server auth. +- `edge-client.crt` / `edge-client.key`: edge transport identity. EKU must allow + client auth. It grants network access only and carries no application role. +- `health-client.crt` / `health-client.key`: independent client-auth identity + used only by the container health probe. + +Issue the three leaf identities separately, use short lifetimes, restrict keys +to their named containers, and rotate them through the deployment secret +manager. A request still needs a valid user OIDC bearer token (or a directly +provisioned service API key when bypassing the public edge); mTLS never grants +administrator access. diff --git a/infra/nginx/40-render-production-manifest.sh b/infra/nginx/40-render-production-manifest.sh new file mode 100755 index 0000000..1138627 --- /dev/null +++ b/infra/nginx/40-render-production-manifest.sh @@ -0,0 +1,65 @@ +#!/bin/sh +set -eu + +deploy_origin=${DEPLOY_ORIGIN:?DEPLOY_ORIGIN must be the public HTTPS origin, for example https://excel.example.com} +manifest_template=${MANIFEST_TEMPLATE:-/etc/nginx/manifest.production.xml.template} +manifest_output=${MANIFEST_OUTPUT:-/usr/share/nginx/html/manifest.xml} +manifest_runtime_dir=${MANIFEST_RUNTIME_DIR:-/tmp/nginx-runtime} +nginx_template=${NGINX_TEMPLATE:-/etc/nginx/nginx.conf.template} +nginx_output=${NGINX_OUTPUT:-/tmp/nginx-runtime/nginx.conf} +oidc_connect_origins=${OIDC_BROWSER_CONNECT_ORIGINS:?OIDC_BROWSER_CONNECT_ORIGINS must list the exact HTTPS origins used for OIDC discovery, token exchange, and JWKS} +umask 077 +manifest_rendering=$(mktemp "${manifest_runtime_dir}/manifest.XXXXXX") +nginx_rendering=$(mktemp "${manifest_runtime_dir}/nginx.XXXXXX") +trap 'rm -f "$manifest_rendering" "$nginx_rendering"' EXIT HUP INT TERM + +if ! printf '%s\n' "$deploy_origin" | grep -Eq '^https://[A-Za-z0-9.-]+(:[0-9]{1,5})?$'; then + echo "DEPLOY_ORIGIN must be an HTTPS origin without a path or trailing slash" >&2 + exit 1 +fi + +if printf '%s\n' "$deploy_origin" | grep -Eiq '^https://(localhost|127\.|0\.0\.0\.0|\[?::1)'; then + echo "DEPLOY_ORIGIN must not use a loopback or localhost address" >&2 + exit 1 +fi + +for oidc_origin in $oidc_connect_origins; do + if ! printf '%s\n' "$oidc_origin" | grep -Eq '^https://[A-Za-z0-9.-]+(:[0-9]{1,5})?$'; then + echo "OIDC_BROWSER_CONNECT_ORIGINS entries must be exact HTTPS origins without paths or wildcards" >&2 + exit 1 + fi + if printf '%s\n' "$oidc_origin" | grep -Eiq '^https://(localhost|127\.|0\.0\.0\.0|\[?::1)'; then + echo "OIDC_BROWSER_CONNECT_ORIGINS must not contain local origins" >&2 + exit 1 + fi +done + +if [ -z "$oidc_connect_origins" ]; then + echo "OIDC_BROWSER_CONNECT_ORIGINS must contain at least one exact HTTPS origin" >&2 + exit 1 +fi + +envsubst '${DEPLOY_ORIGIN}' \ + < "$manifest_template" \ + > "$manifest_rendering" + +if grep -Eq '\$\{|localhost' "$manifest_rendering"; then + rm -f "$manifest_rendering" + echo "Production manifest contains an unresolved or local URL" >&2 + exit 1 +fi + +cat "$manifest_rendering" > "$manifest_output" + +OIDC_BROWSER_CONNECT_ORIGINS=$oidc_connect_origins \ + envsubst '${OIDC_BROWSER_CONNECT_ORIGINS}' \ + < "$nginx_template" \ + > "$nginx_rendering" +if grep -Eq '\$\{' "$nginx_rendering"; then + echo "Production nginx configuration contains an unresolved variable" >&2 + exit 1 +fi +cat "$nginx_rendering" > "$nginx_output" +rm -f "$manifest_rendering" +rm -f "$nginx_rendering" +trap - EXIT HUP INT TERM diff --git a/infra/nginx/Dockerfile b/infra/nginx/Dockerfile new file mode 100644 index 0000000..527bf40 --- /dev/null +++ b/infra/nginx/Dockerfile @@ -0,0 +1,52 @@ +# syntax=docker/dockerfile:1.7 + +FROM node:22.23.2-alpine3.24@sha256:c610fcdfb1d5b4740dd70c284ed3cb16bb857e0f7166196e36a5501df7a3aa32 AS plugin-build + +WORKDIR /build/plugin + +COPY plugin/package.json plugin/package-lock.json ./ +RUN npm ci + +COPY plugin/vite.config.js plugin/tsconfig.json ./ +COPY plugin/src ./src +COPY plugin/assets ./assets + +# Production API and WebSocket requests are same-origin. No backend credential +# is accepted as a build argument or compiled into these browser assets. +RUN npm run build \ + && test -s dist/src/taskpane/taskpane.html \ + && test -s dist/src/spreadsheet/spreadsheet.html + + +FROM nginx:1.30.4-alpine3.24@sha256:97d490c12ba55b4946b01546d1c3ed324e8d41ab1c9fcb2a616aa470620e5b46 AS edge + +COPY infra/nginx/nginx.conf /etc/nginx/nginx.conf.template +COPY infra/nginx/manifest.production.xml.template /etc/nginx/manifest.production.xml.template +COPY infra/nginx/40-render-production-manifest.sh /docker-entrypoint.d/40-render-production-manifest.sh + +COPY --from=plugin-build /build/plugin/dist/ /usr/share/nginx/html/ +COPY --from=plugin-build /build/plugin/assets/ /usr/share/nginx/html/assets/ + +RUN chmod 0555 /docker-entrypoint.d/40-render-production-manifest.sh \ + && rm -f /etc/nginx/conf.d/default.conf \ + && find /usr/share/nginx/html -type f -name '*.map' -delete \ + && install -d -m 0700 -o nginx -g nginx \ + /tmp/nginx-runtime \ + /tmp/nginx-runtime/client_temp \ + /tmp/nginx-runtime/proxy_temp \ + /tmp/nginx-runtime/fastcgi_temp \ + /tmp/nginx-runtime/uwsgi_temp \ + /tmp/nginx-runtime/scgi_temp \ + && touch /tmp/nginx-runtime/manifest.xml \ + && chown nginx:nginx /tmp/nginx-runtime/manifest.xml \ + && ln -sf /tmp/nginx-runtime/manifest.xml /usr/share/nginx/html/manifest.xml \ + && test -s /usr/share/nginx/html/src/taskpane/taskpane.html \ + && test -s /usr/share/nginx/html/src/spreadsheet/spreadsheet.html \ + && test -s /usr/share/nginx/html/assets/icon-32.png \ + && test -s /usr/share/nginx/html/assets/icon-80.png + +USER nginx + +EXPOSE 8080 8443 + +CMD ["nginx", "-c", "/tmp/nginx-runtime/nginx.conf", "-g", "daemon off;"] diff --git a/infra/nginx/certs/README.md b/infra/nginx/certs/README.md new file mode 100644 index 0000000..a1ece52 --- /dev/null +++ b/infra/nginx/certs/README.md @@ -0,0 +1,5 @@ +# Public TLS material + +Provision `server.crt` and `server.key` for the exact production add-in origin +through the deployment secret manager. The files are ignored by Git and must +not be copied into the image or release bundle. diff --git a/infra/nginx/manifest.production.xml.template b/infra/nginx/manifest.production.xml.template new file mode 100644 index 0000000..761e3d3 --- /dev/null +++ b/infra/nginx/manifest.production.xml.template @@ -0,0 +1,93 @@ + + + a1b2c3d4-e5f6-7890-abcd-ef1234567890 + 0.5.0.0 + Internal Finance AI + en-US + + + + + + + + ${DEPLOY_ORIGIN} + + + + + + + + + + + + + + + + + ReadWriteDocument + + + + + + + + +