experiment(rlm): use Monty as the default interpreter - #76
Draft
isaacbmiller wants to merge 1 commit into
Draft
Conversation
Amp-Thread-ID: https://ampcode.com/threads/T-019fb0dc-8d11-7510-abc9-4bc180e4e6e4 Co-authored-by: Isaac Miller <isaac@cmpnd.ai>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose
Measure the CI and test-suite impact of replacing RLM's default Deno/Pyodide interpreter with a first-party Monty interpreter.
This is an experiment, not a merge proposal. The temporary base branch is pinned to Stanford DSPy main at
c69136b2so the diff and CI comparison stay focused.Changes
CodeInterpreterwith persistent sessions, host tools, typedSUBMIT, lifecycle/error handling, mounts, and resource-limit passthroughMontyInterpreterthe default fordspy.RLMpydantic-monty>=0.0.19a required dependency for this experimentPythonInterpreterand its explicit Deno integration testsLocal validation
130 passed, 18 skippedfor RLM + Monty interpreter tests in 16.58s1131 passed, 352 skipped, 2 xfailedfor the normal repository suite in 39.85s with xdistgit diff --checkpassCI result
The experiment matrix passed on Python 3.10–3.14. Compared with the exact-base upstream run:
Conclusion: total CI time is effectively neutral in this single-run sample; differences by Python version are dominated by normal hosted-runner noise. Monty adds no visible critical-path regression despite moving Monty-backed RLM integration cases into the normal test phase.
The dependency-range workflow's minimum-direct job also passed. Its latest-dependencies job failed in LiteLLM startup because the latest FastAPI removed
get_flat_dependant; the same exact-base upstream dependency run failed independently with the same timeout, so this is not attributable to Monty.