fix(benchmarks): split BENCHMARK_HUB_DATA_DIR on the platform path delimiter - #471
Open
Tyagiquamar wants to merge 1 commit into
Open
Conversation
Author
|
Hi @lluisinthedesert, gentle review ping when convenient. #471, #472, and #473 are three small platform-compatibility fixes that are ready for review. Happy to make any changes you'd like. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
BENCHMARK_HUB_DATA_DIRis documented as a multi-root list, but every consumer parsed it by splitting on":"— the POSIX path-list separator:src/benchmark-hub-core.ts(slugRoots)src/benchmarks-mcp.ts(configureBenchmarksMcpRoots,hubPrimaryRoot)On Windows there is no way to spell a colon-separated list of absolute directories: an absolute path like
C:\Users\me\.understudy\benchmarkssplits into"C"and"\Users\me\.understudy\benchmarks". The first fragment becomes thedataroot (resolving to a nonexistent<cwd>\Cdirectory) and the real directory lands under thedata2prefix. Every slug lookup then fails withunknown benchmark slug: data--<name> (use list_benchmarks)— for the MCP server, the CLI, and the hub loaders alike.I found this by running the test suite on Windows: ~31 tests across
tests/benchmarks-mcp.test.mjs,tests/experiment-lineage.test.mjs, andtests/benchmark-extension.test.mjsfailed with exactly this signature; the fixture setsBENCHMARK_HUB_DATA_DIRto a single absolute temp dir, which is enough to trigger the mis-split.Fix
Parse and join the env var with
path.delimiter(:on POSIX,;on Windows). Behavior on POSIX is byte-for-byte unchanged; on Windows, single absolute paths now parse correctly and the documented multi-root list works with the platform separator. Also updated the contract comment/README wording and one test assertion that hardcoded the:join.Testing
On Windows 11, Node 24 (before → after):