fix: bound remote tool calls with timeout; remove dead live mode (#19) - #20
Merged
Conversation
(B) No timeout / abort on remote MCP calls: Add ToolCallTimeoutError, DEFAULT_TIMEOUT_MS (30s) and a withTimeout() wrapper in src/benchmark.ts. The wrapper uses an AbortController plus a timer so the returned promise always settles even if the underlying call hangs forever, and aborts args.signal on timeout. runBenchmark now routes both fetchStats and runQuery through a boundedCaller, so a hung nexus-agents server can no longer make the benchmark hang. Adds an optional BenchmarkConfig.timeoutMs (defaults to DEFAULT_TIMEOUT_MS). (C) Broken "live mode" (verified-dead removal): src/run-live.ts imported a non-existent './live-bridge.js' and always exited 1; createLiveCaller in src/live-caller.ts was referenced nowhere (not re-exported from src/index.ts). Remove both files, the "live" package.json script, and the README live-mode section. tsx is no longer a direct devDependency. Tests: append withTimeout coverage to src/benchmark.test.ts — timeout rejection, fast pass-through, underlying-error propagation, and signal-abort-on-timeout. Fixes #19 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #19. Addresses the two confirmed harness defects (B and C). Defects A (error-swallowing) and D (dead-scaffolding) were verified NOT present and left untouched.
(B) No timeout / abort on remote MCP calls
ToolCaller.callwas awaited unbounded byfetchStatsandrunQuery, so a hung nexus-agents server made the benchmark hang forever — includingrunQuery, which measuresdurationMsaround the call.DEFAULT_TIMEOUT_MS(30s),ToolCallTimeoutError, and awithTimeout(caller, timeoutMs?)wrapper insrc/benchmark.ts. It combines anAbortControllerwith a timer so the returned promise always settles, and abortsargs.signalon timeout.runBenchmarknow routes bothfetchStatsandrunQuerythrough aboundedCaller, bounding every remote call in the benchmark path.BenchmarkConfig.timeoutMs(defaults toDEFAULT_TIMEOUT_MS). Helper signatures unchanged, so existing tests keep passing.(C) Broken "live mode" (verified-dead removal)
src/run-live.tsimported a non-existent./live-bridge.jsand alwaysprocess.exit(1);createLiveCallerinsrc/live-caller.tswas referenced nowhere (not re-exported fromsrc/index.ts).src/run-live.tsandsrc/live-caller.ts."live"script frompackage.json(fixed trailing comma) and dropped the now-unusedtsxdirect devDependency; lockfile regenerated.README.md.Tests
Appended a
withTimeoutsuite tosrc/benchmark.test.ts:ToolCallTimeoutErrorwhen the underlying call hangs (10ms timeout)args.signal.aborted === true)CI (all green locally)
pnpm install --frozen-lockfile✅pnpm typecheck✅pnpm test✅ (49 tests, 24 in benchmark)pnpm build✅🤖 Generated with Claude Code