feat(server): derive span→turn attribution from the audio timeline#93
Merged
Conversation
LukasPoque
force-pushed
the
feat/metrics-eval
branch
from
7cad213 to
2d3aa13
Compare
basilebong
force-pushed
the
feat/metrics-eval
branch
from
2d3aa13 to
d195f58
Compare
Replace stored tool_calls/model_usage turn_idx with eval/read-time attribution: each row's wall-clock started_at is mapped onto the audio timeline whose origin is replays.recording_started_at (driver sends it via the X-Recording-Started-At header on audio upload). Turn windows are clamped to tile the timeline so every offset falls in exactly one turn, even under barge-in. Model TTFT moves from a per-turn metric to an optional model_usage.ttft_ms attribute (gen_ai.response.time_to_first_chunk), with a new max_ttft_ms assertion. - timeline.ts: shared audioOffsetMs + clampedTurnWindows, used by both the server evaluator and the client trace tree — one attribution rule, no client/server boundary divergence - validate the anchor with a Date.parse gate so ISO forms Date.parse rejects can't be stored as a live-but-unplaceable anchor - client consumes server-derived audio_offset_ms, renders ttft_ms, and drops the started_at fallback (unanchored replays render untimed) - orchestrator warns when a runtime produces audio without the anchor - migration 0002 drops *.turn_idx + replay_metrics.ttft_ms, adds model_usage.ttft_ms + replays.recording_started_at
basilebong
force-pushed
the
feat/metrics-eval
branch
from
d195f58 to
84457c9
Compare
The tiling attribution windows must be built from each turn's voiceEndMs (the voice-active boundary) per spec 0001 §3.4 and the clampedTurnWindows contract, not turnEndMs. deriveTurns sets both columns to the same value today so this is behaviour-preserving, but they are stored separately for future overlap handling; using turnEndMs would silently mis-attribute once they diverge, uncaught by fixtures that set them equal. Fixed in both the server evaluator and the client trace tree so the two never drift. Also hoist the per-row Number.isFinite(originMs) check out of the buildReplayDetail offset closure, computed once per payload now. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
hono <4.12.25 carries three advisories (GHSA-j6c9-x7qj-28xf, GHSA-rv63-4mwf-qqc2, GHSA-wgpf-jwqj-8h8p): body-limit-middleware bypasses on AWS Lambda and a Lambda@Edge header-dropping adapter. xray deploys on Bun.serve as a single image, never AWS Lambda / Lambda@Edge, so none are reachable here, but pnpm audit --audit-level=moderate blocks the supply-chain CI gate. Bump the direct dep to 4.12.25 and add surgical pnpm overrides for the second copy bunqueue's embedded MCP server pulls (@modelcontextprotocol/sdk>hono and >@hono/node-server>hono, both stuck at 4.12.21). 4.12.25 was published 2026-06-09, past the 7-day cooldown, so no cooldown bypass. Audit gate passes locally; 735 tests green. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Turn membership for
tool_calls/model_usageis no longer a storedturn_idx— it's derived at eval/read time by mapping each row's wall-clockstarted_atonto the audio timeline whose origin isreplays.recording_started_at(the driver sends it as theX-Recording-Started-Atheader on audio upload). Model TTFT moves from a per-turn metric to an optionalmodel_usage.ttft_msattribute (gen_ai.response.time_to_first_chunk), with a newmax_ttft_msassertion.Spec:
docs/specs/0001-timeline-clock-alignment.md.Key points
timeline.ts(audioOffsetMs+clampedTurnWindows+offsetInTurnWindow) is shared by the server evaluator and the client trace tree. Windows are clamped with a monotonic cursor so they tile the timeline — every offset falls in exactly one turn, even under barge-in (overlapping VAD segments).Date.parsegate. Rejects ISO forms valibot'sisoTimestamp()accepts butDate.parsereturns NaN for (…+02, space before offset), which would otherwise persist as a live-but-unplaceable anchor and make tool/ttft assertions report a misleading pass/fail instead oferrored.audio_offset_ms, rendersttft_ms, and drops thestarted_atfallback — unanchored replays render Untimed (spec §3.1), never silently skewed.RuntimeResult.recording_started_at_epoch; the orchestrator warns when a runtime produces audio without it.*.turn_idx+replay_metrics.ttft_ms, addsmodel_usage.ttft_ms+replays.recording_started_at.Testing
bun test— green (pre-existing environmental network/SSE flakes aside, identical onmain); +30 net new tests covering clamped windows, TTFT selection, anchor validation, and the populated-DB migration upgrade.tsc --noEmitclean, biome clean.pyright --strict0 errors,pytest93 passed.Follow-up
snapshot/xray.dbfrom a fresh example run (stale migration baseline + missing anchor; pre-existing, out of scope here).