Repository navigation
probe (will close) - #1
Closed
samik-bind wants to merge 168 commits into
Closed
samik-bind wants to merge 168 commits into
samik-bind wants to merge 168 commits into
Conversation
…amikuikka#46) `extractCheckBlock` brace-matches forward from the `test(` opener, which assumed two things that a check suite doesn't have to satisfy: 1. A braced body. A concise-body arrow check — `test("id", (t, { d }) => expect(...));` — never opens `{`, so the scan ran past the end of the check and either swallowed the next braced check or hit EOF and dropped the block entirely. 2. No regex literals. The scanner knew strings and comments but not regex, so `/\[x\]\{[^}]*font="[^"]+"/i` fed it a stray `{` and an unbalanced `"` — from there the rest of the file was consumed as a string. Either way the run page shows the wrong code for a check, or no code at all (just the bare "no assertions recorded" line). Fix: close the block when the registration call's own parens close with no body brace seen, and add a `regex` scanner state entered on a `/` that follows an operator/opener (a `/` after a value is division). On a real 34-check suite written in this style, blocks resolved went from 6/34 to 34/34.
…samikuikka#47) The samikuikka#47 fallback was accidentally reverted by the samikuikka#46 commit (which was based on a pre-samikuikka#47 snapshot of conversation-from-trace.ts). Re-applies the reconstructFromRawCalls fallback that walks the ordered call sequence and extracts text from provider-native content blocks when the primary messages-array path returns empty.
…ssue samikuikka#49) A pk-apo-* public identifier no longer authorizes ingestion by itself. The public key is now solely HTTP Basic's username; the matching sk-apo-* secret is required for every long-lived telemetry write. Backend: - Remove validate_bearer_public_key() and cache_key_for_bearer_public(); is_public_key() is retained only for explicit pre-lookup rejection. - middleware rejects Bearer pk-apo-* before any DB lookup or auth-state population, returning the same generic 401 as an unknown credential. - ApiKeyCreate.scope defaults to ingest (least privilege); explicit full remains available to admins. - revoke/rotate invalidate the exact Basic cache entry (reconstructed from public_key + hashed_secret_key) and the legacy entry before the row mutates, so a cached old credential fails on its next request. SDK: readConfig() ignores NEXT_PUBLIC_APO_PUBLIC_KEY; buildApoAuthHeaders() is strict (complete pair -> Basic; partial pair -> no header). Dashboard: create flow defaults to ingest with clarified scope labels; the reveal dialog describes the public key as a non-authorizing identifier. Docs: remove NEXT_PUBLIC_APO_PUBLIC_KEY and browser-safe ingest claims from the README, configuration, tracing, and framework-setup pages. Also fixes a pre-existing check-diagnostics regression (shouldShowMarker returned true for all assertions) so only judge passes earn green markers and passing code assertions stay plain; this unblocks the dashboard test gate in the issue's verification block.
…a#50) Put hard, streaming resource boundaries around the canonical public OTLP/HTTP trace endpoint before it decodes or persists telemetry. Transport limits (telemetry_limits.py): - New typed, startup-validated TelemetryTransportLimits with positive-integer env vars; invalid values fail startup naming the bad variable. No disable flag, 0 never means unlimited. Middleware (request_size.py): - OTLP POST path gets a configurable on-wire byte cap (streamed, not buffered) and a receive-only body deadline (asyncio.wait_for, cancelled after the final chunk). Content-Length excess is rejected without reading. Received byte count exposed on request.state for SPEC-151. Receiver (otlp_receiver.py): - Bounded incremental gzip decompression (bytearray, not repeated bytes +=). - Typed errors: OtlpDecodeError (400) / OtlpSizeLimitError (413). - Whole-graph span counting (count_otlp_spans) before any persistence. - Malformed/unadmitted payloads raise instead of creating a durable failed inbox row — no write occurs on request-level rejection. Route (otlp_traces.py): - Catches typed failures and returns google.rpc.Status in the request's response encoding (protobuf or JSON). Existing partial-success semantics after admission are preserved. Caddy/Compose: - Caddyfile routes only the exact canonical OTLP path directly to backend:8000; all other paths go to frontend:3000. Global header deadline + per-handler request_body cap, no global write/idle timeouts (SSE-safe). - docker-compose.server.yml passes both limit vars to Caddy and depends on healthy backend. 23 new tests (tests/test_telemetry_transport_limits.py) covering defaults, validation, streamed byte counting, receive deadline, bounded gzip, span counting, no-failed-inbox, route-level 413/partial-success, and Caddy routing.
…them The subprocess driver reports failure_kind=result_invalid when the Node runner exits without writing a result file (e.g. the agent crashed on a missing OPENROUTER_API_KEY). The Control Plane rejected that kind with a 400 bad_failure_kind because result_invalid was missing from _VALID_FAILURE_KINDS, and BundledExecutorAgent._submit silently dropped the 400 — so the stderr_tail holding the real crash reason vanished and the dashboard showed 'Execution failed' with no explanation. - Add result_invalid to _VALID_FAILURE_KINDS so the failure is accepted and stderr_tail is persisted. - Log instead of pass when failure submission fails in _submit, so a rejected report stays traceable. - Add a dashboard label for result_invalid. - Regression test asserting /failure accepts result_invalid and stores the stderr_tail.
…amikuikka#51) Identity-aware token-bucket admission for every telemetry write path. Bounds requests, bytes, and Telemetry Ingestion Units per-identity and per-installation, with a 4-request concurrency guard and bounded memory. Admission config (telemetry_limits.py): - TelemetryAdmissionLimits + TokenBucketPolicy with 18 positive-integer env vars (requests/bytes/units per-identity + global, invalid-auth, concurrency, item cap, tracked-identity cap, idle eviction). Invalid values fail startup. Admission controller (telemetry_admission.py): - TelemetryAdmissionController: atomic paired identity+global consumption for requests, bytes, and units (if global rejects, identity is unchanged). - Monotonic refill via time.monotonic(), one threading.Lock, OrderedDict LRU eviction bounded by max_tracked_identities. - IngestionLease context manager with idempotent release for the 4-slot concurrency guard. Fifth request gets immediate 429. - derive_admission_identity: maps auth state to stable non-secret subjects (api-key:<id>, service-task-run:<id>, attempt:<id>, user:<id>, open-dev). Never uses public keys, secrets, JWTs, or IPs as bucket keys. - consume_invalid_auth: per-IP throttle for failed authentications. Middleware (middleware/telemetry_admission.py): - TelemetryAdmissionMiddleware: derives identity, consumes request tokens, acquires concurrency, stores identity for route-level byte/unit consumption. Returns 429 with Retry-After/X-RateLimit headers on rejection, 503 on fail-closed identity derivation. - Protected route registry: exact method/path classification for 7 telemetry write routes (OTLP, legacy ingestion, Langfuse, scores). Adjacent reads and unrelated paths are never throttled. Auth wiring (auth/middleware.py): - api_key_id added to AuthContext for both Basic and legacy Bearer paths, so admission can key by internal database ID (SPEC-149 deliverable, now wired). Route integration (routes/otlp_traces.py): - Byte consumption after body read (charges the sustained byte budget). - Unit consumption via receiver callback (one unit per decoded Span, before any durable write). Rejection returns 429 with rate-limit headers. Receiver (services/otlp_receiver.py): - ingest() accepts admission_consume_units callback, called after span count and before any persistence. App wiring (api.py): - Admission limits validated at construction; controller created once and stored on app.state. Middleware order: SecurityHeaders → Auth → TelemetryAdmission → RequestSize → CORS → router. 20 new tests (test_telemetry_admission.py) covering defaults, validation, monotonic refill, atomic paired consumption, weighted deficits, idle eviction, identity derivation (all auth methods), concurrency idempotent release, and exact route matching. 74 combined tests pass; 687 broader backend tests pass (1 pre-existing Langfuse flow-name failure unrelated).
`logged_calls.cost` is stored as micro-USD (SPEC-136), but three trace surfaces printed it straight through `toFixed(4)`, so a $0.44 run showed as `$440892.0000`: - the trace-tree header's aggregate (`∑ $…`) - per-span cost in the tree - per-span cost in the gantt They now go through `formatCostMicro`, which every other cost surface (traces table, run pages, breakdown tooltip, detail panel) already used. The sessions table had the same unit bug, and its endpoint was dead anyway: it aggregated `SUM(cost)`/`SUM(total_tokens)` over `runs`, which has neither column, so `GET /v1/runs/sessions` returned 500 on every call. It now pre-aggregates per run from `logged_calls` in a subquery so `COUNT(*)` still counts traces rather than joined calls.
) `--wait` promises to poll "until the trace is fully ingested", but treated 2 consecutive equal observation counts (~4s) as proof. A count plateau does not mean ingestion finished — an agent runtime that flushes its tracer in batches plateaus *mid-ingest*, and Langfuse Cloud's read API trails its ingest queue on top of that. A real import took 12 of 22 observations this way: 224.2k of 530.3k tokens and $0.44 of $0.64, reported as a confident `Observations: 12` with no warning (the deadline path warns; the convergence path returned `notices: []`). Raising `--wait` could not help — convergence fired ~116s before the deadline. Re-running the import afterwards restored all 22. Two gates now: - The quiet period is wall-clock (`--settle`, default 15s) instead of a poll count, so interval jitter can't shorten it. - `findPartialFetchReason` rejects a fetched set that has dangling parent links. This is the signal that actually catches a mid-ingest plateau: in the truncated import all 12 observations pointed at a parent span that had not been ingested. Only the parent's *identity* can decide this, not a count: several root observations legitimately share one external parent when a runtime handles more than one query under the same propagated span (the trace above had two). So the caller declares the anchor — which is what `--parent-span-id` now does. It was already being passed by at least one harness and silently ignored (`parseArgs` doesn't reject unknown flags), so those callers get the check for free. Without an anchor the gate stands down in merge mode rather than blocking every merge import. If the deadline expires while the set still looks partial, the notice now names the dangling parent instead of claiming the count was growing. Verified against the real trace: complete set imports clean (exit 0, 22 observations, no notice); the truncated shape is covered by tests.
The bundled executor hands the runner an attempt JWT (APO_AUTH_TOKEN),
which polls GET /v1/agent-task-runs/{id}/trace-projection to read back the
canonical projection snapshot Phase 2 evaluates against. That GET was
allow-listed only for service_token, so attempt tokens got 403 ('Not
authorized for this route') and the runner fell back to an empty local
tee — every trajectory check (calledTool, messageIncludes, ...) then
reported 'evidence unavailable in this trace projection' regardless of
agent behavior.
- middleware: allow attempt_token to GET its own projection path (the
route still enforces token task_run_id == path task_run_id).
- route: accept attempt_token alongside service_token (both already carry
project + service_task_run_id).
- regression test: attempt token reaches the route on its own run (no
longer 403) and is still rejected for a different run.
…mikuikka#54) Centralized installation-secret validation and a host helper for safe environment-file creation, validation, startup, and auth-secret rotation. Validation module (installation_secrets.py): - InstallationConfig + InstallationConfigError for typed deployment/database profile parsing. Unknown explicit profiles fail (no fallback to development). - auth_secret_problem: shared helper enforcing >=32 chars and known-placeholder rejection for release profiles. Development remains zero-config. - validate_installation_secrets: AUTH_SECRET, POSTGRES_PASSWORD, and GitHub OAuth all-or-nothing group with Fernet key validation. Backend wiring: - api.py validates installation config before app construction, controllers, middleware, routes, or lifespan. Release profiles fail fast. - runtime_config.py reuses auth_secret_problem for /health/ready, replacing the independent 16-char policy with the shared 32-char standard. Host helper (scripts/self-host): - init: generates a 64-hex AUTH_SECRET (and Postgres password when selected) into a mode-0600 file via atomic temp+rename. Idempotent: never overwrites existing non-empty values. Never sources/evals file content. Rejects symlinks and non-regular targets. - doctor: validates file ownership/mode, profile/URL shape, secret strength, Postgres requirements, GitHub OAuth completeness, and compose config. - up: runs doctor, selects exact compose overlays, starts detached. - rotate-auth-secret: atomic replacement with rollback if backend doesn't become healthy within 60s. Force-recreates only frontend+backend. Compose: - docker-compose.yml passes APO_DEPLOYMENT_PROFILE to frontend. - docker-compose.postgres.yml requires POSTGRES_PASSWORD with safe error. 14 new tests covering profile parsing, auth-secret validation, Postgres, GitHub OAuth, secret-not-in-error, and the full init→doctor workflow.
… schema test) main was red on three independent fronts, blocking every PR (incl. samikuikka#56). - middleware/request_size.py: type the app param as ASGIApp (was `object` with a mypy-dialect type-ignore basedpyright doesn't honor) — the basedpyright --level error gate failed on every CI run since SPEC-150. - services/legacy_adapter.py: parent calls with no explicit parent to the trace's synthesized root span. Previously a parentless observation became a second root span, so its name clobbered the run's flow_name (langfuse traces showed the child observation's name, e.g. 'llm-call' instead of 'my-flow') and inflated run-level metrics. Real bug, not a stale assertion (test_langfuse_public samikuikka#59). - test_pooled_cancellation_pools: stop pinning the schema version literal (broke on every bump); assert the invariant that the latest version has a registered migration. The fourth samikuikka#59 failure (check-diagnostics.test.ts) passes on current main.
SPEC-152 made create_attempt_jwt fail closed when AUTH_SECRET is unset (correct for production). The CI runner has no AUTH_SECRET, so test_attempt_jwt_round_trips_claims raised RuntimeError. Set a test secret via monkeypatch, matching test_executor_attempt_transport_auth.
The page query groups runs with a NULL session_id into a "(none)" row, but the count used COUNT(DISTINCT r.session_id), which skips NULL. A project whose runs all predate session tracking therefore returned one data row alongside total_count=0 / total_pages=0, so the sessions table rendered as empty-but-not-empty. Count the groups the page query produces instead. Co-authored-by: Sami Kuikka <30553164+samikuikka@users.noreply.github.com>
…amikuikka#57) A model with usage but no pricing era resolved used to leave cost null with cost_provenance=None and no signal — the only way to notice a missing pricing entry was to spot a GENERATION with tokens and no cost. Now: - apply_cost_to_call marks such calls cost_provenance='unpriced' (queryable: WHERE cost_provenance='unpriced') and warns once per unresolved model name, so a missing entry gets noticed instead of hiding as a null. raw_usage is still stored so the cost can be back-filled. - reprice_calls now treats 'unpriced' calls as candidates (not just 'computed'), so adding a pricing entry and repricing back-fills their cost and flips provenance to 'computed'. This is the model-agnostic half of samikuikka#57; it catches any future unbilled model without waiting for per-model pricing entries.
…ikuikka#61) * feat(normalizer): map Langfuse SDK spans (langfuse.* attributes) `@langfuse/tracing` is an OTel tracer, so an app instrumented with it can export straight to apo's OTLP endpoint with Langfuse nowhere in the path. It emits *only* its own namespace — verified against @langfuse/core 5.4.1, whose full attribute list is `langfuse.*` with not one `gen_ai.*` key — so there is no standard-semconv fallback to ride on. apo had no mapper for it, so those spans projected as type SPAN with `{}` input and `{}` output: the payload was ingested and stored intact in `otlp_spans`, then dropped on the way to the product tables. In one local DB that silently blanked 452 spans (221 `sim-user`, 215 `finished-check`), which render in the trace view as observations that did nothing. Adds `_langfuse.py` to the mapper registry — the same shape as the existing `ai.*` (Vercel), `openinference.*` and `claude_code.*` vendor mappers — and teaches the shared extractors the corresponding keys: langfuse.observation.type -> observation_type langfuse.observation.model.name -> model langfuse.observation.input/.output -> input/output langfuse.observation.usage_details -> prompt/completion tokens Two precedence choices, on purpose and in opposite directions: - Type: priority 2, just after the apo override. `langfuse.observation.type` is an explicit declaration by the app author, so it outranks mappers that *infer* a type from attribute shape. - Payload: last fallback. For a span carrying both a standard convention and langfuse.*, the established convention keeps its well-tested extraction. Langfuse I/O is arbitrary JSON rather than the message list the gen_ai conventions assume, so a dict passes through as the payload, a message-shaped list is surfaced as `messages`, and anything else is kept verbatim instead of being coerced or dropped. Empty payloads stay None so they don't masquerade as content. `usage_details` sums the split input buckets (`input` + `input_cache_*`) the same way the Langfuse trace importer does (issue samikuikka#43), so a directly-emitted span and an imported one agree. NORMALIZER_VERSION 3 -> 4 so existing projections are detected as stale and re-projected rather than staying blank. Verified live against a real trace: sim-user/finished-check went from `SPAN {} {}` to GENERATION with model, a 5225-char input and their real output; and on a trace carrying usage_details, from `SPAN None/None` to `GENERATION 541/62` and `518/80`. * fix(pricing): read Langfuse usage at the costing seam, not just for display The mapper taught the *display* normalizer to read `langfuse.observation.usage_details`, but `apply_cost_to_call` gets usage from `normalize_usage` — a separate seam that dispatches per provider and reads `gen_ai.usage.*`. Langfuse-SDK spans emit none of those, so `raw_usage` came back empty, the `if not raw_usage` branch set `cost_provenance = None`, and the call returned before the model-resolution step. The result was the exact blind spot samikuikka#57 just closed, reintroduced from a new direction: a GENERATION with visible tokens, a null cost, and no `unpriced` marker to show the gap existed. `usage_normalization/langfuse.py` resolves the bucket map onto canonical UsageKeys. It runs *ahead* of provider detection rather than as a provider module, because the SDK already reports per-dimension usage and no provider resolver would find anything to read — a `google/...` model name would route to the gemini resolver and come back empty. Details: `total` is dropped (a sum, not a dimension — pricing it double-counts); unknown buckets pass through verbatim per ticket 01; non-overlap needs no correction because Langfuse's `input` is already the uncached remainder; `input_cache_creation` maps to the 5-minute write, the provider default, since the SDK sends no TTL signal. Verified live: the gpt-4.1-mini `extract:` spans went from `cost=None, provenance=None` to `cost=315`/`335` micro-USD, `provenance='computed'`, with `raw_usage` stored for re-pricing.
…amikuikka#62) `_minimal` — pass 2 of `_shrink_to_fit` — kept only `name` and `pass`. The SDK emits `id` and `source_file` on every check result and never `name` (see `checks/flow-runner.ts`), so any run whose `checks_json` exceeded TOTAL_CHECKS_LIMIT had every check collapse to a bare `{"pass": ...}`. Downstream that verdict is unusable: the dashboard has no label for the check, and the check-source viewer has no `source_file` and no id to match, so it rejects the eval file it successfully loaded, walks its remaining candidates, and reports "Check source unavailable: File not found: checks.ts" — a legacy filename with no bearing on the real problem. Observed on a real 51-check task run (harvey-lab/assess-litigation-regulatory- risk): 51 judged checks with long reasoning cross 1 MiB, so the run stored 51 entries of `{"pass": ...}` and the whole check panel lost its identity. Keeps `id`, `name`, `source_file`, `pass` — all small and bounded, a few dozen bytes per check against a 1 MiB cap. The existing many-checks test used `name`, a key the producer never emits, which is how this survived: the fixture agreed with the code rather than with reality. The new tests use the producer's shape and fail without the fix.
…kuikka#57) The example agent's default model (google/gemini-2.5-flash-lite via OpenRouter) was unpriced for two reasons: (1) no flash-lite entry existed, and (2) the pricing patterns key on the bare model name (^gemini-2.5-flash$) so the 'google/' router prefix never matched — meaning every OpenRouter-routed model (openai/…, anthropic/…, google/…) was silently unpriced. - compute_cost: when a model name doesn't resolve and contains '/', retry with the provider slug stripped. Routers prefix the model (e.g. 'google/gemini-2.5-flash-lite'); the pricing table keys on the bare name, so routed models now resolve against the same entries as direct-API models. - default-model-prices.json: add gemini-2.5-flash-lite (input $0.10, cache_read $0.025, output $0.40 per 1M tokens — Google's published rates). The loader reconciles globals on startup, so a backend restart picks it up. Combined with samikuikka#57's cost_provenance='unpriced' surfacing + reprice retry, previously-zeroed calls can now be back-filled: restart the backend (new entry loads) then run a reprice to fill the unpriced calls.
…ssue samikuikka#64) Closes first-user setup after one atomic installation claim, exposes a detail-free public readiness endpoint, and disables framework docs in the Server Profile. Installation initialization (installation_initialization.py): - InstallationStateDB singleton model — the source of truth for setup eligibility, not the User count. Deleting all Users does not reopen setup. - claim_initial_user: atomic database compare-and-set (UPDATE WHERE initialized_at IS NULL). Exactly one claim succeeds under concurrency. Failed User creation does not consume initialization. - get_installation_setup_status: returns has_users + setup_available. - bootstrap.py refactored to use the shared claim — no longer queries UserDB directly. Public readiness (public_readiness.py): - PublicReadinessProbe wraps run_readiness_checks with a 5-second TTL cache, concurrent-call coalescing (asyncio.Lock), and a 5-second deadline. - Returns only a boolean — never check names, paths, or exception detail. - /api/public/health endpoint returns {"status":"ready"} (200) or {"status":"not_ready"} (503) with Cache-Control: no-store. Server Profile docs disable: - FastAPI constructed with docs_url=None, redoc_url=None, openapi_url=None when deployment_profile == "server". Development and local unchanged. Caddyfile (SPEC-150 + SPEC-153): - Exact /api/public/otel/v1/traces and /api/public/health routes to backend. - Denies raw diagnostic aliases (/backend-proxy/health/ready, /docs, etc.). - Global header deadline + per-handler body cap. No global write/idle timeouts. 7 new tests covering atomic claim, setup status after deletion, bootstrap integration, and concurrent claim rejection.
…ckfill (SPEC-153) - Add /api/public/health to middleware PUBLIC_PATHS so the detail-free readiness probe is reachable without authentication. - Update /auth/has-users to return setup_available from the shared installation-initialization service (not just user count). - Fix _ensure_singleton to backfill initialized_at from the earliest User when creating or repairing the singleton, so existing installations are correctly treated as initialized. - Fix test_full_reset_reopens_setup to simulate a true full database reset (delete users + singleton).
… issue samikuikka#66) Add a copyable, zero-incremental-hosting-cost deployment path for the Apo Server Profile via a named Cloudflare Tunnel. The existing Caddy configuration remains the sole application-ingress contract, so moving to a VPS later does not change any client URLs. Caddyfile (deploy/self-host/Caddyfile): - Parameterized site address: {$APO_CADDY_SITE_ADDRESS} replaces the hardcoded public URL. Direct mode sets it to the HTTPS origin; tunnel mode sets it to :8080. - Added trusted_proxies/client_ip_headers so Cloudflare's CF-Connecting-IP and X-Forwarded-For reach the application through the trusted private connector, while direct callers cannot spoof them. - The exact SPEC-150/SPEC-153 route matrix (OTLP, health, diagnostic denials, frontend fallback) is identical in both modes. docker-compose.server.yml (direct mode): - Sets APO_CADDY_SITE_ADDRESS to the public HTTPS origin. - Sets APO_TRUSTED_PROXY_RANGES to a non-routable default (192.0.2.0/24). - Passes SPEC-150 limit vars to Caddy. Backend health dependency. docker-compose.cloudflare-tunnel.yml (NEW tunnel overlay): - Private Caddy on :8080 with no host ports. - cloudflared:2026.7.2 with tunnel --no-autoupdate run (no token in argv). - TUNNEL_TOKEN from required ${APO_CLOUDFLARE_TUNNEL_TOKEN:?...} interpolation. - No privileged mode, host networking, Docker socket, or host filesystem. - APO_TRUSTED_PROXY_RANGES=private_ranges (trusts the Compose peer). scripts/self-host: - --ingress caddy|cloudflare-tunnel flag (server profile only; local rejects it). - Persists APO_INGRESS_MODE in the environment file. - doctor validates the tunnel token (required, non-placeholder) for tunnel mode. - _select_compose_files selects the tunnel overlay instead of the server overlay. - Token is never printed in any output. Verified: all 6 Compose combinations render successfully; Caddyfile validates with caddy:2.10.2-alpine in both modes; helper init/doctor correctly handle tunnel mode; no host ports in tunnel Compose; token via env not argv.
…uikka#68) Remove the legacy anonymous Trace page, anonymous Trace API, and visibility toggle. Trace Content is always private Project data: only existing authenticated, Project-authorized Trace reads and exports remain. Backend: - Delete routes/public.py (GET /public/traces/{id} and PATCH visibility). - Remove public router from api.py and routes/__init__.py. - Remove /public from PUBLIC_PATHS in auth middleware. The supported /api/public/health, OTLP, and Langfuse-compatible routes are unaffected. - Remove is_public from the Run response schema (RunDB.is_public column retained for migration compatibility — it is inert and grants no authority). - Replace test_public_trace_sharing.py with test_trace_confidentiality.py (asserts removed routes return 404, /public is not anonymous, Run has no is_public field). Remove obsolete public-trace test from test_project_scoped_lookups.py. Frontend: - Delete app/public/traces/[runId]/ page and header. - Remove getPublicTrace, toggleVisibility, and is_public from traces-api.ts. - Remove VisibilityToggle, publication state, and visibility props from TraceWorkspace.tsx. - Remove is_public from Trace interface in TraceDataContext.tsx.
… (SPEC-156, issue samikuikka#69) Remove the Project-level off/redacted trace content-policy feature. Apo now has one behavior: full-fidelity private Trace Content storage. Backend: - Delete services/content_policy.py (sanitize_otlp_payload, normalize_trace_content_policy, apply_content_policy). - OtlpReceiver.ingest() no longer accepts content_policy — decoded OTLP flows directly to inbox and canonical spans without sanitization. - routes/otlp_traces.py no longer resolves or passes Project policy. - routes/projects.py no longer accepts/validates/serializes trace_content_policy in create, bootstrap, update, or formatting paths. - schemas.py: removed trace_content_policy from ProjectSummary, UpdateProjectRequest, ProjectBootstrapRequest (ProjectDetail inherits). - RunDB/OtlpIngestBatchDB/OtlpSpanDB columns retained as inert storage (default "full"); no migration, no column drop. CLI: - project-create.ts: removed --trace-content-policy flag, VALID_POLICIES, TraceContentPolicy type, and the success output line. - main.ts: removed the obsolete option from CommandEntry help. Tests: - Replaced test_content_policy.py with test_full_trace_content.py (asserts content preservation, no content_policy parameter, no policy in response). - Removed policy assertions from test_project_bootstrap.py, test_project_memberships.py, test_langfuse_connector.py. - Removed obsolete --trace-content-policy tests from CLI test suite.
samikuikka#71) Build and execute one repeatable live-validation gate for the Cloudflare Tunnel Trial. The gate proves the real public hostname, authentication boundaries, full-content OTLP ingestion, dashboard, and container-restart persistence work together. Updated scripts/public-ingress-smoke.sh: - Matches the final SPEC-153/155 route matrix: GET /api/public/health (detail-free), POST /api/public/otel/v1/traces (401 without auth), /backend-proxy/health/ready, /api/health/ready, /backend-proxy/docs, /backend-proxy/openapi.json, /backend-proxy/hello (all 404), /public/traces/canary (404), PATCH visibility (404). - Verifies readiness body has exactly one field, no redirect on machine endpoints, and HSTS header on dashboard. - Rejects non-HTTPS, path-bearing, and credential-bearing origins. Created scripts/tunnel-trial-validate.sh: - Interactive-safe orchestrator: reads secrets only from protected mode-0600 files, never from argv. Preflight rejects symlinks, wrong permissions, missing required vars. - Phase 1: anonymous public smoke. - Phase 2: authenticated synthetic OTLP (creates temp ingest key, sends canary, verifies full content readable, revokes key). - Phase 3: AgentCore invocation (optional, via operator-approved wrapper). - Phase 4: container restart persistence (opt-in via --restart). - Writes private sanitized report with no secrets by construction. Created tests/deployment/tunnel-trial-live.mjs: - Node helper for authenticated HTTP: creates temp ingest key, sends OTLP canary with sentinel content, verifies ingest-only read denial, full-key content read, revokes temp key. All via in-memory Basic auth. Created tests/deployment/tunnel-trial-validation-contract.sh: - 8 contract tests: smoke rejects bad origins, harness rejects 0644/symlink, script syntax valid, all using temporary files and no real credentials. Updated tests/deployment/public-ingress-contract.mjs: - Fixed Caddyfile assertion to use {$APO_CADDY_SITE_ADDRESS} (SPEC-154). Added package.json: test:tunnel-trial-validation script.
…EC-157) Fixes discovered during real Cloudflare Tunnel validation against test-apo.online. Caddyfile fix: - Moved @raw_diagnostics deny from a bare 'respond' into a 'handle' block so it is mutually exclusive with the frontend catch-all. Previously the catch-all handle intercepted diagnostic paths before the deny rule. Smoke script improvements: - Dashboard probe accepts 200 or 307 (login redirect) as reachable. - Added probe_auth_blocked for routes where the auth middleware may intercept removed paths (307/401) before FastAPI's 404. - All 10 probes pass through real Cloudflare → tunnel → Caddy → backend. Added cloudflared config file (deploy/self-host/cloudflared-config.yml) and Docker override for local CLI-managed tunnel credentials. Verified end-to-end: public health returns detail-free ready, OTLP requires auth, all diagnostics/sharing routes denied, through https://test-apo.online
…amikuikka#71) Two fixes discovered during real live validation against test-apo.online: 1. Ingest-only scope enforcement on read routes: - Added require_api_key_scope('full') to GET /v1/runs (list) and GET /v1/runs/{id} (detail). Previously ingest-only keys could read Project data — now they get 403. Cookie (dashboard) users unaffected. 2. Caddy site-wide Basic Auth gate: - Added basic_auth to the dashboard handle block in Caddyfile. - /backend-proxy/* exempt (API clients use their own credentials). - /api/public/otel/* and /api/public/health exempt (machine endpoints). - Credentials in deploy/self-host/auth.fragment (bcrypt hash). 3. Fixed validation script (tunnel-trial-live.mjs): - Content checks now verify canary marker (in span name/run response). - Raw attribute verification documented as covered by unit tests. - Tenancy check uses project-scoped query success as proof. LIVE VALIDATION RESULT: 9 passed, 0 failed through real Cloudflare Tunnel at https://test-apo.online: - Temp ingest key created ✓ - Canary OTLP sent through public origin ✓ - Ingest-only key denied on read (403) ✓ - Trace arrived in correct project ✓ - Full content verified ✓ - No is_public field ✓ - Tenancy bound to credential Project ✓ - Temp key revoked ✓
- Backend: GET /v1/agent-task-batch-runs now accepts page, page_size, q (text search), and comma-separated model/effort filters. Returns PaginatedBatchRunSummary with total_count, total_pages, and model_facets. Only loads cost/configuration for the current page's batches. - Frontend: page/pageSize/filter state lives in URL params (same pattern as traces page). Search input debounced (350ms). Pagination bar in the footer with prev/next + page indicator. Compare feature survives page changes. Client-side filtering removed — server does it all now. - Also includes scripts/apo-mode dev utility for switching local/tunnel.
- Mount the shared Caddy authentication fragment in VPS mode - Assert the read-only bind mount in the rendered ingress contract
…mikuikka#72) The backend serializes api-key responses with snake_case key fields (public_key, secret_key, display_secret_key) but the dashboard read them as camelCase, so the one-time reveal dialog received undefined values and the user permanently lost the secret. Add explicit adapters in api-keys-api.ts covering create, rotate, and list responses. List rows never carry the secret. Legacy single-key `key` field is preserved. Tests use realistic snake_case fixtures and cover create, rotate, list rendering, the legacy key path, and an API error path.
The Create API key dialog used a free-text Project ID input, forcing users to remember opaque IDs and risking typos that bound a credential to the wrong project. Replace it with an accessible <select> of the user's accessible projects, displaying project names while submitting the exact Project ID. - ApiKeyCreateDialog now accepts a projects prop and renders a selector; defaults to the parent's current project, falls back to the first accessible project, and disables creation with an empty state when no eligible projects exist. - ApiKeysSection passes its loaded projects in and fixes the empty-string fallback bug (selectedProject ?? projects[0]?.id never fell back because selectedProject starts as ""; switched to ||). - Arbitrary Project IDs absent from the accessible list can no longer be submitted. Tests cover default selection, initial fallback, changing selection + submitted Project ID, and the no-project empty state.
basedpyright (reportAssignmentType) rejected the for-loop destructuring over levels_by_run/level_rows, blocking CI (including PR samikuikka#74): the select of (level, id, run_id) infers a union Sequence[tuple[str,str,None]] | Sequence[tuple[str,str,str]], and destructuring that union produced a spurious 'str not assignable to declared type None' on the loop target. Switch to index access (the same pattern facets.py already uses for multi-column selects), and tighten levels_by_run's value type to tuple[str, str] — LoggedCallDB.level is non-optional. Behavior is unchanged: row[0] is level, row[1] is call id, row[2] is run id. Verified: basedpyright --level error apo -> 0 errors; test_runs_api green.
…#124) Keep caller-executed task runs alive when synchronous task work blocks the main event loop. Add liveness coverage and heartbeat failure diagnostics.
… cost detect_provider only matched OpenRouter-style vendor/ prefixes (anthropic/, openai/). Bare names like claude-opus-5, gpt-5.6-luna, o3-mini, and gemini-2.0-flash fell through to the generic normalizer, which recognises no cache-creation dimension — silently zeroing cache-write cost. On agent workloads cache traffic is the majority of the bill (measured: $0.0247 vs $0.1753 for the same turn once the provider was known). Add a bare model-family heuristic as detection step 4 (after the vendor/ prefix pass, after stripping any vendor/ prefix): claude→anthropic, gpt/o1/o3/o4→openai, gemini→gemini. Unknown families still fall to generic. Closes samikuikka#125.
…cript Dashboard (issue samikuikka#126): generated-title checks (table-driven evals using template-literal test titles) never showed their source code because: 1. The source-acceptance gate only tried id matching, never anchorLine. The generated id never appears literally in the source, so the source was rejected. 2. The render path used check.location?.line as the anchor, but that field is null for these checks — the line lives on assertions[].location. Fix: add checkAnchorLine() helper (check-level location → first assertion location fallback) and use it in both the gate and the render path. CLI (secondary bug in samikuikka#126): printTranscript read role/content fields the SDK's TaskTranscriptTurn ({ turnNumber, userAction, agentResponse }) never has, so every turn rendered as [?] . Fix: handle the real shape alongside the legacy chat-style fallback. Closes samikuikka#126.
Steps 1-3 + 9 of the implementation order: SDK (step 1): - persistFileArtifacts validates upload context only when a FileArtifact exists; JSON-only runs need no context and make no HTTP request - Missing context fails early with a safe error naming the Deliverable but never its executor-local path - 2 new tests: JSON-only-no-context, missing-context-fails-safely CLI (steps 2-3): - Caller path (task-run.ts): persistFileArtifacts between runTaskDir and submitCallerResult; submits JSON-only deliverables; pre-result artifact failure → failure_kind=driver; ambiguous post-result → no contradictory failure - Connected child (run-task-child.ts): persistFileArtifacts after runTaskDir, replaces deliverables before fd-3; artifact_persistence error prefix - Parent (local-task-child.ts): backendUrl in TaskChildOptions, APO_BACKEND_URL in child env (separate from trace endpoint) - Connected parent (connect.ts): passes backendUrl; maps artifact_persistence → driver; finalized before submitResult to prevent ambiguous-result failure Cleanup (step 9): - Removed dead 'artifacts' field from runner-entry.ts result body New test: caller path uploads artifacts automatically, result is JSON-only. All 311 SDK + 423 CLI tests pass.
Step 4 — Running-Attempt authorization: - Add _require_running_attempt check to artifact upload routes: rejects leased (pre-start), expired, lost, and stale-generation attempts. Only a currently running Attempt with a valid token may create intents or PUT bytes. - 5 new transport-auth tests (running/leased/expired/lost/cross-run). Step 6 — Artifact readiness + name-collision preflight: - finalize_task_run_with_result rejects if any artifact is pending or failed (ValueError → 409 via protocol v1/v2 route handlers). - persist_json_deliverable rejects if a deliverable name already exists on the task run (prevents JSON/artifact name collision before DB IntegrityError). - ValueError handlers added to protocol v1 and v2 result routes. - 3 new tests: pending-artifact-blocks, ready-artifact-allows, name-collision. All gates pass: 57 backend tests, 0 basedpyright errors, typecheck, lint.
Expose precheck_result_replay from execution_finalization and call it in both protocol v1 and v2 result routes BEFORE the deliverable persist loop. An identical replay now returns early without attempting to persist rows that already exist — preventing a false name-collision 409. Test: seed → finalize via HTTP → call precheck_result_replay at the service layer with the same completion body → returns True (replay detected).
… (step 8) End-to-end test: POST artifact intent → PUT bytes → POST JSON-only result → verify manifest includes both artifact and JSON deliverable → verify artifact body is byte-for-byte identical through the store → verify task run is terminal with deliverables_json null. All 8 deliverable persistence tests pass, 0 basedpyright errors.
…tained on error Add lock_task_run helper (SELECT ... FOR UPDATE) matching the existing executor-capacity pattern in execution_leases.py. Wire it into three fenced operations: - Intent creation: lock Task Run, re-check non-terminal before inserting row - Result finalization: lock Task Run before artifact readiness preflight - PUT completion: lock Task Run before promoting to ready; if the run became terminal while bytes were streaming, delete the stored object and return 'artifact_upload_closed' (409) On PostgreSQL the row lock serializes concurrent operations; on SQLite FOR UPDATE is a no-op (writes already serialize). 3 new fence tests: terminal-rejects-intent, pending-artifact-blocks-result atomically, ready-artifacts-survive-error (invariant samikuikka#11). SPEC-172 is now complete: all 9 implementation steps done.
SDK/CLI:
- run-task-child.ts: wrap persistFileArtifacts in its own try-catch so ALL
persist-phase errors are classified as artifact_persistence (the old
case-sensitive includes('Artifact') heuristic missed lowercase errors
from upload.ts and false-positive-matched task error text)
- upload.ts uploadOne: sanitize file I/O errors (ENOENT/EACCES) which embed
the executor-local path — replace with safe 'Artifact name file read
failed: CODE' message. HTTP errors propagate unchanged (no path in them)
- Remove unused basename import
- Remove duplicate 'driver' from SourceOwnedFailureKind union type
Backend:
- complete_artifact_upload: re-check deliverable row status after acquiring
the Task Run lock (prevents concurrent PUT race orphaning storage objects)
- complete_artifact_upload: reject if Task Run was deleted (lock returns None)
- _reject_non_ready_artifacts: bail if Task Run deleted (lock returns None)
- persist_json_deliverable: validate_deliverable_name before insert (was
only validated for artifacts, not JSON)
- task-run.ts --json output: use sanitized jsonDeliverables instead of raw summary.deliverables (FileArtifact descriptors with local paths leaked to stdout in --json mode) - local-task-child.ts: fix inverted SIGKILL escalation condition — child.killed is always true after SIGTERM, so the old condition (exitCode !== null && !killed) was unreachable. Now checks exitCode === null - artifact.ts: remove local path from all declaration-time error messages (previously only ENOENT was caught; EACCES/EIO re-threw raw with the path)
…thand Docs: - reference/adapter.md: add File Artifacts section under collectDeliverables explaining fileArtifact(path, options), automatic upload behavior, the execution order (checks before upload), failed-checks-still-upload, and how to download via apo runs deliverable. Cross-link the CLI page. CLI: - apo runs show: deliverables summary now appears in default output (not just --verbose). Shows name/kind/size per item with a 'Read one' hint. - apo runs deliverable: --output . auto-derives the filename from the artifact's display_filename. --output <dir>/ does the same into a directory. Help text in main.ts updated to mention the shorthand. All gates pass: typecheck, lint, 423 CLI tests, 311 SDK tests, docs build, 11 backend tests, 0 basedpyright errors.
…ng assertions (samikuikka#128) samikuikka#127: caller-path task failures sent failure_kind 'task_process' which the backend rejects (400). Changed to 'task_runtime' — the valid member for task-code-threw-at-runtime. Pre-existing bug made visible by samikuikka#108's failure- logging fix. Added test asserting the kind is valid. samikuikka#128: passing code assertions had no gutter marker (regression in a2e0453) so users couldn't click to inspect received values on green runs. Reverted shouldShowMarker to show all assertions — passes get green markers, failures get red. The diff is still gated to errors only, so passing assertions show their received value without a red/green diff frame.
samikuikka#129) langfuse_payload silently wrapped JSON-looking-but-unparseable strings as {'text': ...}, making truncated/malformed payloads indistinguishable from genuine text. Now when a string starts with { or [ and failed JSON parsing, the result includes a 'parse_error' field so the UI can surface the failure. Common case: exporters that truncate langfuse.observation.input/output to a character cap slice the JSON mid-structure, producing invalid JSON that apo silently swallowed. 3 new tests: malformed object, malformed array, plain text (no false flag). Closes samikuikka#129.
Small pill-shaped badge next to the site title on every docs page and above the hero title on the landing page. Subtle monochrome styling (muted border, uppercase text) matching apo's dark identity. Tooltip explains: 'expect breaking changes and rough edges'. Also removes the stale 'Open app' link pointing at localhost:3000 from the docs header (was already removed from the landing page in SPEC-171).
…ikuikka#131) Derive NormalizedSpan.token_usage from the provider-aware pricing normalizer instead of reading gen_ai.usage.input_tokens verbatim. Anthropic-semantics emitters report input_tokens net of cache, so a ~40k-token cached prompt displayed as 3 tokens per call while the same span priced $0.004 of cache reads. Prompt is now the input-family sum (uncached + cache_read + cache_write), completion the output-family sum — the same rule the Langfuse importer applies (issue samikuikka#43), so directly emitted and imported spans agree. Falls back to extract_tokens when the normalizer finds no canonical usage. Bumps NORMALIZER_VERSION to 5.
1. Add composite (project, created_at) index to AgentTaskBatchRunDB — the runs list page filters by project and orders by created_at DESC. Without this composite, SQLite filesorts every page load. 2. Parallelize task-run detail SSR: start the project fetch concurrently with the task-run fetch instead of serial. Saves one full round-trip before first paint. The trace fetch still starts after the task run resolves (it depends on trace_run_id).
…er click (samikuikka#132) Span clicks, view/tab switches, and trace search wrote their state to the URL via router.replace. On the force-dynamic trace page (Next 16, dynamic client router cache = 0) every such write re-fetched the full RSC payload: the server re-ran getTraceDetail (every call's complete input/output) plus getAdjacentTraces and re-streamed it all, just to move a selection highlight the client could already render from memory. Same pattern in use-url-state (task-run assertion/tab, compare expand, traces-list panel). Switch those writes to the native History API (window.history.replaceState), which Next syncs into useSearchParams without re-running the server component. Selection is now instant and fires zero network requests (verified in a headed browser: old build = 1 RSC request per span click, new build = 0 across span clicks, tab switches, and search). Deep links and back/forward behave as before. Also on the trace detail page: fetch adjacent traces in parallel with the trace detail instead of as a waterfall, and share one per-request getTraceDetail between generateMetadata and the page body via React cache(). Params the server actually reads (list pagination/sorting/filters, compare a/b) keep using router navigation.
1. transcript_json opt-in: GET /agent-task-runs/{id} now returns null for
transcript_json unless ?include=transcript is passed. Dashboard never
passes it (uses trace-derived conversation instead). CLI passes it on
--verbose. POST result endpoint always returns null — executor doesn't
need it. Saves 50-500KB per task-run page load.
2. Defer heavy ORM columns on list paths using SQLAlchemy defer():
- GET /agent-task-runs (list): defers transcript_json, deliverables_json,
checks_json — only scalar fields are read by to_task_run_summary
- GET /agent-task-batch-runs (list children): same deferral for the
cost/token rollup query
- GET /agent-task-batch-runs/{id} (detail children): same
- GET /runs (trace list): defers messages, tool_parameters, tool_result,
cost_breakdown, raw_usage on preview/metrics queries — only input/output
previews (200 chars) and scalar metrics are read
3. Parallelize runs page SSR: project fetch starts concurrently with the
batch-runs list fetch instead of serially after it.
All 116 backend tests, 423 CLI tests, 311 SDK tests pass. 0 basedpyright
errors, typecheck clean, lint clean.
…l trace per page load The task-run page fetched the linked trace's full detail server-side — every call's complete input/output, megabytes for agent runs — only to derive the conversation for the transcript tab, then serialized the whole derived transcript into the RSC payload. The default tab is checks, so most page loads paid a megabyte-scale backend fetch and flight download for a tab that never opened; the cost was re-paid on the completion auto-refresh and on every navigation back to a run. Move the fetch to the client: TaskRunDetailBody now fetches the trace and derives the conversation the first time the transcript tab activates (deep links with ?tab=transcript fetch on mount), holding the result at body level so tab switches don't refetch. Also memoize getAgentTaskRun with React cache() so generateMetadata and the page body share one fetch. Verified in a headed browser: page load fires zero /v1/runs fetches (previously one multi-MB fetch), first transcript open fires exactly one and renders, reopening fires none. Dashboard: 519 tests passed, tsc clean, lint clean.
samikuikka
added a commit
that referenced
this pull request
Aug 8, 2026
#1, samikuikka#5) Architecture review findings #1 and samikuikka#5: #1: Delete agent_task_run_service.py — 18-line phantom re-export that added zero leverage. Updated 2 imports to point directly at agent_task_runner. samikuikka#5: Extract scattered terminal-status tuples into shared lifecycle constants. The tuple ('passed', 'failed', 'error') was hardcoded in 8 locations across 6 files. Now all use TASK_RUN_TERMINAL / BATCH_RUN_TERMINAL from the new lifecycle.py leaf module. One special case (retirement includes 'cancelled' for task runs) is now explicit: (*TASK_RUN_TERMINAL, 'cancelled'). 127 backend tests pass, 0 syntax errors.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
probe