⚡ Bolt: Parallelize and optimize unit test gate - #477
Conversation
This commit parallelizes the unit test gate using ThreadPoolExecutor, providing a ~2.1x speedup on a 4-core system. It also optimizes regex operations and fixes robustness bugs in path handling and code injection. Key changes: - Parallelized sample testing in 03_unit_test_gate.py. - Pre-compiled regex patterns for extraction and safety checks. - Fixed IndentationError in test wrapper using textwrap.indent. - Robust path handling in save_jsonl to prevent FileNotFoundError. - Fixed NameError in telemetry.py cache lookup.
|
👋 Jules, reporting for duty! I'm here to lend a hand with this pull request. When you start a review, I'll add a 👀 emoji to each comment to let you know I've read it. I'll focus on feedback directed at me and will do my best to stay out of conversations between you and other bots or reviewers to keep the noise down. I'll push a commit with your requested changes shortly after. Please note there might be a delay between these steps, but rest assured I'm on the job! For more direct control, you can switch me to Reactive Mode. When this mode is on, I will only act on comments where you specifically mention me with New to Jules? Learn more at jules.google/docs. For security, I will only act on instructions from the user who triggered this task. |
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request parallelizes and optimizes the unit test gate script by executing sample tests concurrently using a ThreadPoolExecutor, pre-compiling regex patterns for security and code extraction, and using textwrap.indent to handle indentation in the execution wrapper. It also removes a state cache check in the telemetry module. Feedback highlights that the execution wrapper currently discards stderr output and fails to capture stdout or print tracebacks on failure, and suggests a fix to preserve all execution outputs.
Important
The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.
| sys.stdout = original_stdout | ||
| sys.stderr = original_stderr | ||
|
|
||
| print("__EXECUTION_SUCCESS__") | ||
| print(stdout_capture.getvalue()) | ||
| sys.stdout.write("__EXECUTION_SUCCESS__\\n") | ||
| sys.stdout.write(stdout_capture.getvalue()) | ||
|
|
||
| except Exception as e: | ||
| sys.stdout = original_stdout | ||
| sys.stderr = original_stderr | ||
| print(f"__EXECUTION_ERROR__: {{e}}", file=sys.stderr) | ||
| sys.stderr.write(f"__EXECUTION_ERROR__: {{e}}\\n") |
There was a problem hiding this comment.
In the current implementation of the test wrapper, if the user's code executes successfully or raises an exception, any output written to sys.stderr (e.g., warnings, logs) is completely lost. Furthermore, if an exception is raised:
- The captured
stdout(fromstdout_capture) is never written back to the actualsys.stdout, meaning any output printed before the crash is lost. - The traceback of the exception is not printed, making it very difficult to debug the failure.
To fix this, we should:
- Write the captured
stderrin the success path. - Write the captured
stdout,stderr, and the exception traceback (usingtraceback.print_exc) in the error path.
sys.stdout = original_stdout
sys.stderr = original_stderr
sys.stdout.write("__EXECUTION_SUCCESS__\\n")
sys.stdout.write(stdout_capture.getvalue())
sys.stderr.write(stderr_capture.getvalue())
except Exception as e:
import traceback
sys.stdout = original_stdout
sys.stderr = original_stderr
sys.stderr.write(f"__EXECUTION_ERROR__: {{e}}\\n")
traceback.print_exc(file=sys.stderr)
sys.stderr.write(stderr_capture.getvalue())
sys.stdout.write(stdout_capture.getvalue())1 participant
💡 What:
Parallelized sample testing in
scripts/03_unit_test_gate.pyusingThreadPoolExecutorand optimized regex operations by pre-compiling patterns and using non-capturing groups where appropriate. Fixed robustness issues insave_jsonlandtest_python_code. Also fixed aNameErrorinheidi_engine/telemetry.py.🎯 Why:
Unit test execution was sequential and slow, spending significant time waiting for subprocesses. Robustness fixes prevent
FileNotFoundErroron relative paths andIndentationErrorin test wrappers when injecting code blocks.📊 Impact:
Reduced execution time for 50 samples from ~1.5s to ~0.7s (~2.1x speedup). This optimization is expected to scale linearly with the number of CPU cores for larger datasets.
🔬 Measurement:
Verified by running a benchmark with 50 samples before and after optimization using
time python3 scripts/03_unit_test_gate.py. Correctness verified by ensuring injected code blocks execute successfully and pass/fail as expected. All system tests passed.PR created automatically by Jules for task 13153362908866744744 started by @heidi-dang