Repository navigation
Conversation
Keep file-size limits in the style checker instead of a manually updated JSON table. Bound existing large paths by HEAD, the comparison baseline and their committed history, so shrinking automatically lowers limits. Keep new paths within the default ceiling and retire deleted exemptions. Fail on incomplete baseline history instead of silently skipping checks. Remove the obsolete ratchet command and document the Git requirements. The hotspot ranking reads the limits the checker derives from HEAD. Change-Id: Ic35d28e897f699fdb57efa769b73d80d3d0eca50
The face detector suite looked only in the default target directory, so a valid custom build lost server coverage and the setup case tried to navigate to an empty URL. Clean-target gate runs then failed for a test harness error rather than a face detector regression. Use the selected Cargo target directory for both the executable and the suite's temporary files. Guard every case when the server is unavailable, keeping missing builds explicit skips while custom builds exercise the real server. Validate absolute and relative target paths, the missing-build skip path, and the Rust gate. Change-Id: Ic7d77d9cf7230ac4110eed5600bea5170685c9bf
The report test file has reached its size ceiling. Its scripted recovery helper needs focused timing coverage for the rejoin boundary, so give that support a separate file without increasing the existing budget. Move the helper unchanged through an include at the same lexical scope, keeping its callers and test names stable. The discovered tests and report behavior stay unchanged. Change-Id: I1a9fd5478d586eb35ce0c611318f2a33efc699f5
A regenerated report can finish while its candidate is away and remain held until the candidate returns within the rejoin grace. The browser fallback could finalize the provisional failure before that retained result was ready for delivery. Include the final rejoin grace in the browser retry wait and its generated wire fixture. Document the configured generation, rejoin and delivery waits, including the limit that SDK publication and host scheduling can still stall. Capture the recovery holder's return time inside its future so the scripted sender cannot conceal an early return. Cover the late rejoin in the real holder and assert that the page retains its room before accepting the final packet. Extend the cross-language budget check to include the rejoin grace. Change-Id: I99b65f6435278fccf593ff3aebc6b69f7ae98f0c
A write canceled before reaching Gemini must retain the complete input that was constructed for it. The final text-write matrices checked transport, usage and cleanup without checking constructed counts or bytes. The prefix matrix checked the count but left bytes unchecked. Compare each canceled text write with a completed wire reference from the same fixture. Check the exact count and UTF-8 bytes, including the clock frame, and keep the prefix matrix's listening context charge. Account for the added reference row in usage and ingress assertions. Change-Id: I6c0ecb9059579506a8b514ff6707f94705b7052d
Report packet counts could hide duplicate bounded generation after an interview ended. Observe entry to the real bounded generator by room scope, route its report requests to a local fixture, and assert generation separately from provider attempts and native delivery receipts. Cover guarded deadline reentry and locally ready End, deadline and room closure schedules through the SDK. The ended-guard mutation must reach two real generations before the cardinality assertion fails. Keep provider calls local and the instrumentation limited to test builds; these fixtures do not claim arbitrary native arrival ordering or production clock timing. Change-Id: I39c0b5d9c56160866e686d428f398dc6e154a103
Keep one test-only owner for child selection, pipe draining, proxy guards and the existing timeout. Recovery retains its local adapters. Mount the helper once at the crate root so callers share the implementation without compiling duplicate modules or exposing a production API. Change-Id: I110079f7ff749d4873698fd5049788202020b929
Content-free timing telemetry cannot establish which tool answers actually executed, and its silence and discard gates can omit observations. Capture the final executed answers in an admitted test- only collector before board resends or socket writes can suspend or be cancelled. Bound serialized metadata and results before retaining copies, latch scope or identity failures, and keep socket changes within the admitted session. Leave send completion unknown; this acquisition leaf does not establish provider receipt, completed replies or long-run interview qualification. Change-Id: I1d1f6824a6b2985d14a551d4e25504dfe526c8e6
Remove the recovery driver's cached byte total and the four persistent updates needed to keep it synchronized with retained contexts. Derive a local total from the bounded queue when admitting another summary. Keep both retention limits, FIFO eviction, omission accounting and prepare-before-write behavior. Observe retained text in the fixtures and remove only cache bookkeeping and tautological assertions. Existing behavioral oracles catch both a changed bound and an ignored byte total. Change-Id: Ia8a15176d1eaec35533172ee39c59da7d55132c4
Let an isolated fixture request and inspect small buffers on one authored loopback connection while retaining the normal TCP and WebSocket path. Scope the registration to its exact URL and refuse a second dial. Use borrowed socket inspection for connected streams. Add only a test dependency on the existing locked socket2 version, with no production dependency or transport-default change. Change-Id: Iccba5ccc5feec9f796ed0c80ca61d7d5d1abb66a
Use measured small sender and receiver buffers to exercise a failed listening-note write with a smaller payload. The large synthetic note spent most of the local test deadline preparing its message. Keep the original deadline, cold-brief and reply-debt assertions, and peer cleanup. Run the fixture in its own process to isolate socket state. Change-Id: I9944fb2e7da25a034d1cc1bb0ec7028b094d0274
Keep old-socket shutdown and the bounded peer join beside the recovery rig that owns them. Reuse that helper when finishing a rig so the same lifecycle rule is maintained in one place. Retain the two-second join deadline, every captured frame and the expected-frame assertion. Scenario setup and recovery assertions stay with their existing tests. Change-Id: I587d45342261e5596bd4510a63ae76b1445410e6
Both packet paths already share the reaction decision, but each chooses the clock passed to it. Pin that choice independently of code changes and previous interjections, so a wrong source cannot silently change when the interviewer replies. Keep the existing decision/effect split and its threshold tests. This adds transport-free coverage without another clock or transport layer. Change-Id: Ibaf6ddb43eef18504b5269da1db5395a23593adb
Keep packet fields, driver state and internal operations private to recovery and its descendants. Their callers already live within that boundary, so the wider visibility exposes more than the parent needs. Retain the types and entry points used by the parent and sibling tests. This changes no recovery behavior or test placement. Change-Id: Icca3b1672d7fb1fc89cfb9dcc65f43bf4e64f0fd
The test runner executes after hooks in registration order. Removing a bundle before its server stops races the reviewer heartbeat and can fail with ENOTEMPTY while the remaining processes are still active. Own each fixture writer from spawn through close, including the wait process. Close the page, stop and join writers, then remove the bundle. Keep cleanup active when page closing fails and retain existing limits. Change-Id: I90ebd11c1d7e12e1507b809d07b4eeea1e5e65fd
Keep the session event loop and its test file focused on routing by moving the existing tool-batch execution into a child module and the tool-dispatch timing case into an included fragment. Preserve the bodies, callers and test names so later completion tracking has a separate review boundary. No tool execution or send behavior changes, and no visibility widens. Change-Id: Ib3d798d148fb493c358ccca1d8cd1a2e39072c42
Keep test acquisition results tied to the capture and batch admitted before the owner awaits a send. Record returned success and failure even when timing logs are disabled, and leave a canceled await unknown. Preserve that original association when another capture is installed. Reject completion that exceeds the byte budget or follows a latched failure. Cover owner transitions, cancellation and exact retained bytes. Change-Id: Ie2c78ac5e0662bf9778c422bb5284f001988f1b3
A long interview reported its decoded signal as not assessed even when a replacement swap had been installed, because only the soak flow ran the media probe. Install the probe for the long flow, mark each installed swap once, and carry the judged result into the report as a separate decoded signal status. The status is verified, not required, failed or not assessed. It never changes the exit code or the qualification. A verdict whose rows are unusable stays not assessed, and so does one whose replies were not all placed, since the audio judge skips such swaps and a verified neighbor would hide the gap. The swap marker and the settle and judge step move to swap-audio-owner.cjs, shared with the soak and tested there against a stand-in page. The generic whole-track assertion stays off for this flow, and the probe is stopped on every failure path. Change-Id: I59eb707be63dd2f41ca807913321c18cb0916a2a
The server's launch configuration lived in the binary crate, so no test fixture could build a server from the same constructors that codetrial web uses. A fixture that copied them would only resemble the product. Move the flag, environment and config-file handling, the provider pool and server config constructors, and their startup refusals into a library module, and import them back into the binary. The moved code is unchanged apart from paths and visibility, and the binary tests still reach it through the same names. Point the source inspection test, the desktop note and the review routing at the new file. Change-Id: If153d909e1f6198d1dd73138115e096f7229e486
A browser-server fixture cannot know the room name or session id in advance, and a capture bound to a fixed scope would either need a forced room name or accept a batch from whichever session spoke first. Add a test-only observer that starts unarmed. The dispatcher's accepted assignment names the one expected room, and the first latency bind for it latches a nonzero session, then builds and installs the capture. A bind before any assignment, a second room, session zero, a different later session or an occupied slot fails the observer for good and removes the capture. Removing the observer takes its capture with it, clearing only the dispatch hook it installed, and events already in flight afterwards do nothing. Dispatch behavior is unchanged, and nothing here compiles outside test builds. Change-Id: Ica3ce1a074d5ad5e382c4e1e92257bed30f88ca9
A fixture that hosts the web service must run the same startup steps and serve it the way codetrial web does, in the same order, or its parity with the product is only a claim. run_web interleaved those steps with the socket bind and kept its serving tail inline, so a fixture could call neither. Move the steps before the bind, the steps that need the bound address, the local agent config load and the serving tail into library functions, and call them from run_web in the original order. The product passes a shutdown that never resolves and no drain bound, and behaves as before; a caller can stop the server with its own signal and bound how long open connections may hold it up. Run the phases directly in tests and extend the source check on environment reads to cover the new bodies. Change-Id: I84dada94432bb616dc780099b06447fd6f6832a6
A browser run needs the real web service and local interviewer hosted where the arming observer can see their dispatch and bind boundaries, and a report of what it saw once the run stops. Add an in-crate fixture built on the shared startup phases and serving function. A launcher names a config and one directory; the account database, ready file, stop file and report live there, so the database can never default to the operator's own. It clears the previous run's files first, publishes by rename, bounds the connection drain and writes the report even when the server ends badly. The ignored entry refuses to run without an exact test filter, since the observer is process wide. Tests cover the service, the recorded evidence, the drain bound and the path rules without credentials. It binds loopback only, and takes its drain from the launcher like its settle. Change-Id: Iee523e791c35a1f3cea47b506a90f8793712746d
A stopping server that waits for its rooms to finish needs to know that no new room can arrive meanwhile. Closing the listener does not end a request already in flight, and checking that the dispatcher is idle is a separate decision from admitting the next room. Add a dispatcher method that closes admission and implement it in the local dispatcher under the lock every admission already takes, so closing, refusing and the idle check agree. Rooms already admitted keep running and still count as live. Other dispatchers keep the default, which does nothing. A start refused after the close tells the candidate the server is shutting down, not that it is full. Change-Id: I3cf51cea14c2d0b35f1ad49c0ddbbeb2e08772c8
A fixture that snapshots evidence when the server stops would otherwise cut off an interview that was still writing its report, because the runtime drops and ends every task it holds. Let a caller bound how long admitted rooms and reports may take after the last connection has gone. The server closes admission when its shutdown begins, waits for the dispatcher to go idle, and reports rooms still running at the bound together with any error the shutdown had. codetrial web passes no bound and its shutdown never begins, so it behaves as before. The serving fixture passes the bound, set by the launcher and defaulting to two minutes, and records unfinished rooms. Change-Id: I43347d1e6801fefbefe44c415c98fc9704a88993
A browser interview needs the in-process serving fixture as its server so that the evidence its observer recorded survives the run, and the paid window must not be spent building anything. Add a fixture mode to browser-check.sh. Given an exact test name, it runs the pinned test executable as the server in a session of its own, checks first that the pin lists that one ignored test, and keeps the fixture's report where the caller named it. Cleanup asks the fixture to stop and waits for it to drain and settle instead of killing it, treats a failed exit or a missing report as a failure, and prints the end of its log when no report exists. The stop wait must fit inside the manifest's cleanup reserve, and the fixture's drain and settle are derived from it. The budget runner tells the worker when its reap begins, and the fixture's own deadline falls before it. Change-Id: Iafc68481e36b64f9aefbf36e90ac7378e2cbd8e0
A tool call can be cancelled after the handler receives it and before it executes, at the acknowledgement or board awaits, so evidence captured only at execution cannot tell a lost call from one never made. Add test-only receipts recorded at the handler's entry, before every await that can cancel the batch. A receipt keeps the call's id, name, socket, clock and whether it arrived under a hold or discard, and is linked to the executed batch only on the same session and socket. Receipts are scoped like batches, kept all or nothing, and counted against the byte limit apart from the batches. Received, executed and locally sent stay separate facts, and none is the provider's receipt. Change-Id: I7c5f956568c5a9cc1f9e13d4a8ca2015c85a39ee
The test-only acquisition capture kept tool receipts and answers but nothing of what the interviewer said around them, nor where a reply ended. Record the newest view of each interviewer transcript segment as the page was sent it, keyed by session, socket and segment id, with where it first appeared and at its newest view (receipts and batches seen before it), how many views it had, and whether a view rewrote or reopened what the page showed. A departed candidate's final segment, published without an identity, is not taken for a reply. Also record each provider TurnComplete and Interrupted with whether the room loop took it or swallowed it, the socket it arrived on, and how many receipts, batches and replies preceded it, so a reply can be bounded by the ending that closes it. Retention is bounded and fails closed like the receipts. Each new view is charged only for its own change against a running byte count, as the receipts already were, so a long reply does not serialize every earlier one again under the lock. Change-Id: Ifd1f0b7a21ae7aa39a3bb18e7f8ae6e539961571
The acquisition report holds ordered facts: tool receipts, executed batches, interviewer segments and turn endings, each with its socket and generation. Add the consumer that turns them into the evidence judgeCheckpoint reads. A reply is credited with an ending or a tool call only when the facts leave one reading: same socket, same generation, nothing sealed it early, nothing discarded. Otherwise the checkpoint is not_assessed with the reason, as it is for unusable clocks, a rewritten reply and one that began before the question's estimated end. Only an interview whose every checkpoint passed is verified. The report is a fixture written by a Rust test from the real handlers, so the consumer is tested on producer output, not on a copy of it. Change-Id: I85b1a3de2a4a89bae7dad75a58bf83fce9d3a194
The reply-content verdict needs the serving fixture's report, which only exists after the fixture has stopped, so the browser check runs the judge from its exit handler once that report is kept. It writes reply_content and its reasons into the capture file and leaves the exit status, the qualification and every other field as the run left them. The long report now records the schedule origin and the surface the schedule was generated for, so the schedule can be rebuilt and the interview clock read against it. A run that did not complete cleanly, a clock not anchored to the server, a fixture that did not end ok or a schedule that differs from the planned one is not assessed, and so is a tone fixture run, which asked the model nothing. A fixture report that is there but unreadable is named as such. A capture left by an earlier run, or one that shares a path with the kept report, is refused up front. Change-Id: I10edd33a999a267d3f7df8dab095e597f7ef449a
A judged swap row said verified or failed but not on what. A reader holding only the report could not audit the verdict, and the page was gone by then. Each row now carries the samples behind it, cut to the fields the judgment read, with long text shortened and at most four per swap. When a swap has more, the sample that decided the verdict is kept and the rest are counted. A row covered by a later swap names that swap's sample. The long report carries up to 512 rows and counts what it cuts. Change-Id: Ifb19ad36a7f4dcd03d6e1df01e2a0baf2a468b7e
When the serving fixture stopped, a missing report left the report path empty, the server log went with the scratch directory, and a report of the wrong shape was accepted because it merely existed. Move that step into fixture-server.sh as keep. It saves the server log and the report (both owner only, neither over a file already there), writes a report that says so when the fixture left none, and refuses a report that is not an object, names no outcome other than ok, or lacks the lists the judge reads. Any of these fails the run. A capture the observer could not admit is left to the judge to name. The SDK fixture launcher now pins its test name through the same check as the browser one. Change-Id: I7c4cd9eaef736d6828eae7f8ef04262cfb4c4e79
A launcher that cannot reach the stop file used to end the fixture with TERM and get no report. The fixture now treats a TERM that arrives once it is serving as the same request as the stop file: drain, settle and write the report. A TERM in the first moments, before the handler is registered, still ends it without one, so the browser check keeps the fixture in its own session and asks with the stop file. After the stop file a TERM is ignored and the escalation is the kill. Taking a TERM needs Unix signals and its test sends one with kill, so the module is built on Unix only and says so in its documentation. Nothing runs these tests elsewhere today, so no coverage is lost. Change-Id: I8c9296b01398e656f46e9d32e42b5a3a9a494354
The fixture path from the manifest to the exit handler was covered only in pieces. Run browser-check.sh through it with a stand-in fixture server and a stand-in node for the browser worker only: the report and a private log are kept, the verdict lands in the capture, a judgment that fails or cannot run leaves the run's status alone, and a fixture that wrote nothing fails the run and says so. No browser, no credentials. Change-Id: I4e4e4a5a7bae996a7599218eca5de1f99540ca97
The first paid pilot's content judge failed all 23 checkpoints, and reading what was said against what was required showed the answer key at fault. An inspect question asks about the duplicate case but the key also demanded the revision marker, and the final question asks for the open question and the latest revision but the key also demanded the stage. The constraints rubric did not accept "10 thousand", which is what the interviewer said, and checking the duplicate-case rubric against the pilot's replies showed it missed "a previous, different position". Check the revision where it belongs, in what the tool read, and let a reply state the current one without being faulted. Widen the rubrics to the plain ways of saying the same thing, with negatives so a number that merely contains the token is not the bound. The catalogue changes, so the version, the review ids and the pins move on purpose, and every clip manifest of the old version is stale. Change-Id: I7ce3c888889078f9c32015050069f8f97c54fe69
The whiteboard schedule could never pass: it wanted a revision marker in what read_board returned, but the tool gives the model a count and an image, never text, so a marker cannot be read from a board. Judge a board revision by the stroke count the tool reports. The page draws four strokes per revision and the board keeps them, so revision n reads 4n strokes. The whiteboard questions no longer ask for a revision, and a reply is not faulted for mentioning a count. The wording the judge reads is pinned on the Rust side, since rewording it would break the judge without any test there noticing. Say in the development notes what a long-interview reply is held to. Change-Id: Ida7f577f294b3c73da75aee13bff7bb8d068c423
Each test that reruns itself in a child process spelled its own name as a string beside a module path lookup, so a renamed test kept a name that selected no test until its child ran. Derive the name in the shared isolation helper from the path of an item declared inside the test, and use it at every site. Change-Id: I0bf0c4a49ce94a540e5453cd19b3a3f59f00f542
![cubic-dev-ai[bot]](https://avatars.githubusercontent.com/in/1082092?s=60&v=4){kind=small}
There was a problem hiding this comment.
15 issues found across 90 files
Prompt for AI agents (unresolved issues)
Check if these issues are valid — if so, understand the root cause of each and fix them. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. If appropriate, use sub-agents to investigate and fix each issue separately.
<file name="docs/latency-observability.md">
<violation number="1" location="docs/latency-observability.md:229">
P2: The credential-free command above does not run the new serving-fixture evidence tests, so it validates the structural driver but not the `reply_content` verdict documented here. Add `tests/browser/long-interview-evidence.test.js` to that command.</violation>
</file>
<file name="tests/unit/livekit/recovery_parts/acquisition_tool_sends.rs">
<violation number="1" location="tests/unit/livekit/recovery_parts/acquisition_tool_sends.rs:651">
P2: `zero_clocks` erases `retained_calls` even though it is a call count, not a clock-dependent byte count. Restrict normalization to `retained_bytes` and `retained_receipt_bytes`, then regenerate the fixture so the real report retains `retained_calls: 1`.</violation>
</file>
<file name="tests/unit/livekit/recovery_parts/report_generation_cardinality.rs">
<violation number="1" location="tests/unit/livekit/recovery_parts/report_generation_cardinality.rs:39">
P2: The test polls `future` a second time without checking whether the first poll returned `Ready`; an immediately refused loopback connection can complete the report future on that first poll, making the second poll invalid and flaky. Remove the second poll or retain the future’s `Poll` result and only poll again while it is pending.</violation>
</file>
<file name="tests/unit/livekit/recovery_live_parts/report_generation_cardinality.rs">
<violation number="1" location="tests/unit/livekit/recovery_live_parts/report_generation_cardinality.rs:297">
P2: The `AllReady` scenario sends `end_interview` even though it declares and asserts a `Deadline` terminal. Exclude `AllReady` from this final-action branch so the deadline scenario does not receive a queued End event before startup.</violation>
</file>
<file name="tests/stubs/interview-run-budget/stand-in-fixture.py">
<violation number="1" location="tests/stubs/interview-run-budget/stand-in-fixture.py:46">
P2: The stand-in ignores the launcher's fixture deadline and always self-terminates after 120 seconds. Use `CODETRIAL_FIXTURE_DEADLINE_S` here so a missing stop file is bounded by the same deadline that the launcher and report contract calculate.</violation>
</file>
<file name="tests/test_check_rust_style.py">
<violation number="1" location="tests/test_check_rust_style.py:293">
P2: This test does not exercise the missing-baseline case it names because the temporary root is not a Git repository. Initialize the fixture history before calling `run`, so the assertion verifies rejection of an absent ref rather than rejection of the entire repository context.</violation>
<violation number="2" location="tests/test_check_rust_style.py:324">
P2: This test never crosses the default ceiling, so it cannot prove that a sub-default historical size is clamped to 1,500 lines. Use 1,501 lines and assert a finding over 1,500 to distinguish the clamp from an incorrectly inherited 1,400-line limit.</violation>
</file>
<file name="tests/browser/soak-judge.test.js">
<violation number="1" location="tests/browser/soak-judge.test.js:1224">
P2: The whitelist test does not cover `covered_by.sample`, which is a separate report-retained payload. A regression that copies `pageSecret` or an overlong `reason` into a carried sample would pass these tests; add a covered-swap case with page-supplied fields and assert that `covered_by.sample` is filtered and bounded too.</violation>
</file>
<file name="tests/unit/livekit/probe/serving.rs">
<violation number="1" location="tests/unit/livekit/probe/serving.rs:236">
P2: A stale stop file can make this process exit without clearing an earlier `out.json`, and the keeper will then copy and accept that old successful report as this run’s result. Clear the ready/report artifacts before rejecting the stale stop request, using the same error-checked cleanup as `serve`.</violation>
</file>
<file name="src/launch.rs">
<violation number="1" location="src/launch.rs:266">
P2: `is_cold_start` accepts complete `LIVEKIT_*` environment credentials, but this unconditional file lookup still rejects launches without a config file. Permit environment-only launches to load values without requiring a file.</violation>
<violation number="2" location="src/launch.rs:589">
P1: A malformed `CODETRIAL_TRUSTED_PROXY_HOPS` becomes zero, so a loopback server behind a proxy bypasses the public-session-secret refusal. Reject invalid values or fail closed as proxy-reachable instead of silently disabling the guard.</violation>
</file>
<file name="docs/provider-cost-and-degradation.md">
<violation number="1" location="docs/provider-cost-and-degradation.md:516">
P2: This overstates shutdown admission: a request for an already-live pinned room can still return `Existing` after `close_admission`, so that candidate is not told the server is shutting down. Qualify this as a new-room start, or change the dispatcher if shutdown must reject reuse too.</violation>
</file>
<file name="scripts/long-interview-evidence.cjs">
<violation number="1" location="scripts/long-interview-evidence.cjs:219">
P2: `toolCall` throws when a readable acquisition report contains a batch without a `calls` array, such as after schema drift or partial corruption. Validate nested batch/receipt records before judging, or turn this malformed evidence into an explicit `not_assessed` verdict instead of leaving the merge to an uncaught exception.</violation>
</file>
<file name="scripts/fixture-server.sh">
<violation number="1" location="scripts/fixture-server.sh:102">
P1: `stop` accepts PID `0`, so an unwritable stop directory makes `kill -TERM 0` signal the caller's entire process group instead of the fixture. Reject zero before any kill operation.</violation>
</file>
<file name="scripts/browser-check.sh">
<violation number="1" location="scripts/browser-check.sh:326">
P2: Fixture mode accepts non-`long-interview` flows even though the evidence judge only processes long-interview captures. A default `voice` run can therefore complete with a fixture report but no reply-content verdict; reject `BROWSER_CHECK_FIXTURE_TEST` unless `BROWSER_CHECK_FLOW=long-interview`.</violation>
</file>
Reply with feedback, questions, or to request a fix.
View guided diff | Turn on auto-fix | Re-trigger cubic
| fn trusted_proxy_hops(values: &BTreeMap<String, String>) -> u32 { | ||
| nonempty(values, "CODETRIAL_TRUSTED_PROXY_HOPS") | ||
| .and_then(|value| value.parse().ok()) | ||
| .unwrap_or(0) |
There was a problem hiding this comment.
P1: A malformed CODETRIAL_TRUSTED_PROXY_HOPS becomes zero, so a loopback server behind a proxy bypasses the public-session-secret refusal. Reject invalid values or fail closed as proxy-reachable instead of silently disabling the guard.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At src/launch.rs, line 589:
<comment>A malformed `CODETRIAL_TRUSTED_PROXY_HOPS` becomes zero, so a loopback server behind a proxy bypasses the public-session-secret refusal. Reject invalid values or fail closed as proxy-reachable instead of silently disabling the guard.</comment>
<file context>
@@ -0,0 +1,1073 @@
+fn trusted_proxy_hops(values: &BTreeMap<String, String>) -> u32 {
+ nonempty(values, "CODETRIAL_TRUSTED_PROXY_HOPS")
+ .and_then(|value| value.parse().ok())
+ .unwrap_or(0)
+}
+
</file context>
| pid=$2 | ||
| wait_seconds=$3 | ||
| case $pid in | ||
| '' | *[!0-9]*) usage ;; |
There was a problem hiding this comment.
P1: stop accepts PID 0, so an unwritable stop directory makes kill -TERM 0 signal the caller's entire process group instead of the fixture. Reject zero before any kill operation.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At scripts/fixture-server.sh, line 102:
<comment>`stop` accepts PID `0`, so an unwritable stop directory makes `kill -TERM 0` signal the caller's entire process group instead of the fixture. Reject zero before any kill operation.</comment>
<file context>
@@ -0,0 +1,185 @@
+ pid=$2
+ wait_seconds=$3
+ case $pid in
+ '' | *[!0-9]*) usage ;;
+ esac
+ case $wait_seconds in
</file context>
| are `not_assessed` by this driver. Separate evidence is required before | ||
| qualifying a build. `exposure_s` ends at the harness's observation boundary; | ||
| `not_evaluated`. Checkpoint `reply_content` is `not_assessed` by this driver; | ||
| in serving-fixture mode `scripts/browser-check.sh` judges it from the |
There was a problem hiding this comment.
P2: The credential-free command above does not run the new serving-fixture evidence tests, so it validates the structural driver but not the reply_content verdict documented here. Add tests/browser/long-interview-evidence.test.js to that command.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At docs/latency-observability.md, line 229:
<comment>The credential-free command above does not run the new serving-fixture evidence tests, so it validates the structural driver but not the `reply_content` verdict documented here. Add `tests/browser/long-interview-evidence.test.js` to that command.</comment>
<file context>
@@ -225,9 +225,12 @@ node --test tests/browser/soak-judge.test.js tests/browser/long-interview-driver
-are `not_assessed` by this driver. Separate evidence is required before
-qualifying a build. `exposure_s` ends at the harness's observation boundary;
+`not_evaluated`. Checkpoint `reply_content` is `not_assessed` by this driver;
+in serving-fixture mode `scripts/browser-check.sh` judges it from the
+fixture's report once the fixture has stopped and writes the verdict into the
+capture. Post-swap `decoded_signal` (`verified`, `not_required`, `failed` or
</file context>
| serde_json::Value::Object(map) => { | ||
| for (key, inner) in map.iter_mut() { | ||
| // Byte counts embed the digits of the clock they carry. | ||
| if key == "emitted_unix_ms" || key.starts_with("retained_") { |
There was a problem hiding this comment.
P2: zero_clocks erases retained_calls even though it is a call count, not a clock-dependent byte count. Restrict normalization to retained_bytes and retained_receipt_bytes, then regenerate the fixture so the real report retains retained_calls: 1.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At tests/unit/livekit/recovery_parts/acquisition_tool_sends.rs, line 651:
<comment>`zero_clocks` erases `retained_calls` even though it is a call count, not a clock-dependent byte count. Restrict normalization to `retained_bytes` and `retained_receipt_bytes`, then regenerate the fixture so the real report retains `retained_calls: 1`.</comment>
<file context>
@@ -0,0 +1,673 @@
+ serde_json::Value::Object(map) => {
+ for (key, inner) in map.iter_mut() {
+ // Byte counts embed the digits of the clock they carry.
+ if key == "emitted_unix_ms" || key.starts_with("retained_") {
+ *inner = 0.into();
+ } else {
</file context>
| scope.entries().is_empty(), | ||
| "constructing a lazy generation does not start it" | ||
| ); | ||
| std::future::poll_fn(|cx| { |
There was a problem hiding this comment.
P2: The test polls future a second time without checking whether the first poll returned Ready; an immediately refused loopback connection can complete the report future on that first poll, making the second poll invalid and flaky. Remove the second poll or retain the future’s Poll result and only poll again while it is pending.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At tests/unit/livekit/recovery_parts/report_generation_cardinality.rs, line 39:
<comment>The test polls `future` a second time without checking whether the first poll returned `Ready`; an immediately refused loopback connection can complete the report future on that first poll, making the second poll invalid and flaky. Remove the second poll or retain the future’s `Poll` result and only poll again while it is pending.</comment>
<file context>
@@ -0,0 +1,170 @@
+ scope.entries().is_empty(),
+ "constructing a lazy generation does not start it"
+ );
+ std::future::poll_fn(|cx| {
+ let _ = std::future::Future::poll(future.as_mut(), cx);
+ std::task::Poll::Ready(())
</file context>
| } | ||
|
|
||
| fn serve_until_stop_file(settings: &Settings) -> Result<(), String> { | ||
| if settings.stop().exists() { |
There was a problem hiding this comment.
P2: A stale stop file can make this process exit without clearing an earlier out.json, and the keeper will then copy and accept that old successful report as this run’s result. Clear the ready/report artifacts before rejecting the stale stop request, using the same error-checked cleanup as serve.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At tests/unit/livekit/probe/serving.rs, line 236:
<comment>A stale stop file can make this process exit without clearing an earlier `out.json`, and the keeper will then copy and accept that old successful report as this run’s result. Clear the ready/report artifacts before rejecting the stale stop request, using the same error-checked cleanup as `serve`.</comment>
<file context>
@@ -0,0 +1,604 @@
+}
+
+fn serve_until_stop_file(settings: &Settings) -> Result<(), String> {
+ if settings.stop().exists() {
+ return Err(format!(
+ "{} already exists and would end the run before it began",
</file context>
|
|
||
| pub fn load_values(options: &CliOptions) -> Result<BTreeMap<String, String>, String> { | ||
| let mut values = environment(options); | ||
| let path = primary_config_path(options)?; |
There was a problem hiding this comment.
P2: is_cold_start accepts complete LIVEKIT_* environment credentials, but this unconditional file lookup still rejects launches without a config file. Permit environment-only launches to load values without requiring a file.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At src/launch.rs, line 266:
<comment>`is_cold_start` accepts complete `LIVEKIT_*` environment credentials, but this unconditional file lookup still rejects launches without a config file. Permit environment-only launches to load values without requiring a file.</comment>
<file context>
@@ -0,0 +1,1073 @@
+
+pub fn load_values(options: &CliOptions) -> Result<BTreeMap<String, String>, String> {
+ let mut values = environment(options);
+ let path = primary_config_path(options)?;
+ crate::config::apply_config_file(&mut values, crate::config::read_config_file(&path)?);
+ apply_options(&mut values, options);
</file context>
| `dispatch_refused ... reason=finalizing` rather than `at_capacity`; it cannot | ||
| reuse that agent as an interviewer for another candidate. Provider projects are | ||
| reuse that agent as an interviewer for another candidate. A server that has | ||
| begun to stop refuses every new start as `reason=closed`, and the candidate is |
There was a problem hiding this comment.
P2: This overstates shutdown admission: a request for an already-live pinned room can still return Existing after close_admission, so that candidate is not told the server is shutting down. Qualify this as a new-room start, or change the dispatcher if shutdown must reject reuse too.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At docs/provider-cost-and-degradation.md, line 516:
<comment>This overstates shutdown admission: a request for an already-live pinned room can still return `Existing` after `close_admission`, so that candidate is not told the server is shutting down. Qualify this as a new-room start, or change the dispatcher if shutdown must reject reuse too.</comment>
<file context>
@@ -511,7 +512,9 @@ completed interviews awaiting recovery; admission still refuses excess
`dispatch_refused ... reason=finalizing` rather than `at_capacity`; it cannot
-reuse that agent as an interviewer for another candidate. Provider projects are
+reuse that agent as an interviewer for another candidate. A server that has
+begun to stop refuses every new start as `reason=closed`, and the candidate is
+told it is shutting down; admitted rooms keep running. Provider projects are
rotated and their LiveKit connection-minute quota is refreshed in the
</file context>
| /// The evidence for one cited call, or the reason it cannot be read. | ||
| function toolCall(capture, receipt, cp, key, options) { | ||
| const batch = capture.batches.find((b) => b.sequence === receipt.batch); | ||
| const call = batch?.calls.find((candidate) => candidate.id === receipt.id); |
There was a problem hiding this comment.
P2: toolCall throws when a readable acquisition report contains a batch without a calls array, such as after schema drift or partial corruption. Validate nested batch/receipt records before judging, or turn this malformed evidence into an explicit not_assessed verdict instead of leaving the merge to an uncaught exception.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At scripts/long-interview-evidence.cjs, line 219:
<comment>`toolCall` throws when a readable acquisition report contains a batch without a `calls` array, such as after schema drift or partial corruption. Validate nested batch/receipt records before judging, or turn this malformed evidence into an explicit `not_assessed` verdict instead of leaving the merge to an uncaught exception.</comment>
<file context>
@@ -0,0 +1,586 @@
+/// The evidence for one cited call, or the reason it cannot be read.
+function toolCall(capture, receipt, cp, key, options) {
+ const batch = capture.batches.find((b) => b.sequence === receipt.batch);
+ const call = batch?.calls.find((candidate) => candidate.id === receipt.id);
+ if (!call) return { why: `call ${receipt.id} has no executed batch` };
+ if (batch.local_send_completed == null) {
</file context>
| --manifest "$BROWSER_CHECK_RUN_MANIFEST" candidate) || exit 2 | ||
| fi | ||
|
|
||
| if [ -n "$FIXTURE_TEST" ]; then |
There was a problem hiding this comment.
P2: Fixture mode accepts non-long-interview flows even though the evidence judge only processes long-interview captures. A default voice run can therefore complete with a fixture report but no reply-content verdict; reject BROWSER_CHECK_FIXTURE_TEST unless BROWSER_CHECK_FLOW=long-interview.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At scripts/browser-check.sh, line 326:
<comment>Fixture mode accepts non-`long-interview` flows even though the evidence judge only processes long-interview captures. A default `voice` run can therefore complete with a fixture report but no reply-content verdict; reject `BROWSER_CHECK_FIXTURE_TEST` unless `BROWSER_CHECK_FLOW=long-interview`.</comment>
<file context>
@@ -278,6 +323,82 @@ if [ -n "$BUDGET_VARS" ]; then
--manifest "$BROWSER_CHECK_RUN_MANIFEST" candidate) || exit 2
fi
+if [ -n "$FIXTURE_TEST" ]; then
+ if [ -z "$PREBUILT_BIN" ] || [ "$BROWSER_CHECK_AGENT" != dispatch ]; then
+ echo "BROWSER_CHECK_FIXTURE_TEST needs a budgeted run (its pinned executable is the" >&2
</file context>
Suggested change
| if [ -n "$FIXTURE_TEST" ]; then | |
| if [ -n "$FIXTURE_TEST" ]; then | |
| if [ "$BROWSER_CHECK_FLOW" != long-interview ]; then | |
| echo "BROWSER_CHECK_FIXTURE_TEST applies to BROWSER_CHECK_FLOW=long-interview." >&2 | |
| exit 2 | |
| fi |
jserv
added
the
explained
Moving the web launch code into the library put it in the mutation gate's diff, and seven mutants survived: nothing called extend_pool, the concurrency cap, the interview length or the compiler switch with a value to check. Test each with real values, and return the startup pool lines from a function so the empty-pool case can be asserted. Change-Id: Ifb19b027f8196dd5b8547b95083b9732e88e4b7f
The running total was only ever reduced once per new context in the tests, so dividing it instead of subtracting survived. Hold the bound where room for one context takes two evictions. Change-Id: I7bf22cbad6a821d29f8090d0a225ef380ede2016
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A paid long-interview run could only be judged from what the page saw. This branch lets it be judged from what the interviewer actually received and said. A test-only acquisition capture records tool receipts, executed batches, interviewer segments and turn endings, each with its socket and generation. An in-process serving fixture hosts the real web service and interviewer from the pinned test binary, behind the same startup and serving functions
codetrial webnow shares fromsrc/launch.rs, and writes that capture as a report when it stops.scripts/browser-check.shruns the fixture, keeps its report and log (owner only), and after it stops judges reply content and decoded swap audio into the capture. Neither verdict changes the run's exit code or its qualification.Along the way the Rust file ceilings are derived from Git history instead of a hand-kept JSON table (
scripts/ceiling-hotspots.pyreads the same limits), and several recovery, report and handoff tests are tightened.Verified with
./scripts/test.shat the head: exit 0, Rust 2776 passed and 29 ignored, browser and JS 3034 of 3034; desktop clippy and cargo-deny were skipped because their tools are not installed here. Every commit passescargo check --all-targets, and no.cargo/mutants.tomlexemption is added.Left out: no paid long-interview run has been made with this yet. A reply that starts before a question's estimated end, which comes from its word count, is
not_assessedrather than judged, because the capture does not record where a clip's speech really ends; carrying the clip manifest's speech boundaries into the capture is the follow-up that would let such a checkpoint pass.Summary by cubic
Judges a paid long-interview run from the serving fixture's acquisition report instead of only what the page saw. Reply content and decoded swap audio verdicts are written into the capture after the fixture stops, and neither changes the run's exit code or its qualification.
Rust file-size ceilings are now derived from Git history rather than the hand-kept
scripts/rust-style-ceilings.jsontable, several recovery, report, and handoff tests are tightened, and the launch helpers moved into the library are now exercised in the mutation gate.Serving fixture
codetrial webuses (src/launch.rs).scripts/browser-check.shruns the pinned test binary as the fixture, keeps its report and log, and judges the capture once it stops.Ceilings
scripts/check-rust-style.pynow derives limits from HEAD and the baseline history;scripts/ceiling-hotspots.pyreads the same limits.scripts/rust-style-ceilings.json; the--ratchetcommand is removed.Written for commit af9ca58. Summary will update on new commits.