Skip to content

experiment(rlm): use Monty as the default interpreter - #76

Draft
isaacbmiller wants to merge 1 commit into
monty-ci-basefrom
experiment/rlm-monty-ci
Draft

experiment(rlm): use Monty as the default interpreter#76
isaacbmiller wants to merge 1 commit into
monty-ci-basefrom
experiment/rlm-monty-ci

Conversation

@isaacbmiller

@isaacbmiller isaacbmiller commented Aug 3, 2026

Copy link
Copy Markdown

Purpose

Measure the CI and test-suite impact of replacing RLM's default Deno/Pyodide interpreter with a first-party Monty interpreter.

This is an experiment, not a merge proposal. The temporary base branch is pinned to Stanford DSPy main at c69136b2 so the diff and CI comparison stay focused.

Changes

  • add a Monty-backed CodeInterpreter with persistent sessions, host tools, typed SUBMIT, lifecycle/error handling, mounts, and resource-limit passthrough
  • make MontyInterpreter the default for dspy.RLM
  • make pydantic-monty>=0.0.19 a required dependency for this experiment
  • move RLM integration coverage from the Deno-only phase into the normal test phase using the Monty default
  • retain PythonInterpreter and its explicit Deno integration tests

Local validation

  • 130 passed, 18 skipped for RLM + Monty interpreter tests in 16.58s
  • 1131 passed, 352 skipped, 2 xfailed for the normal repository suite in 39.85s with xdist
  • focused Ruff checks and git diff --check pass

CI result

The experiment matrix passed on Python 3.10–3.14. Compared with the exact-base upstream run:

Job Deno default baseline Monty default experiment Delta
Python 3.10 198s 200s +2s
Python 3.11 205s 185s -20s
Python 3.12 201s 194s -7s
Python 3.13 163s 195s +32s
Python 3.14 208s 210s +2s
Median test job 201s 195s -6s
Mean test job 195.0s 196.8s +1.8s
Workflow wall time 211s 215s +4s

Conclusion: total CI time is effectively neutral in this single-run sample; differences by Python version are dominated by normal hosted-runner noise. Monty adds no visible critical-path regression despite moving Monty-backed RLM integration cases into the normal test phase.

The dependency-range workflow's minimum-direct job also passed. Its latest-dependencies job failed in LiteLLM startup because the latest FastAPI removed get_flat_dependant; the same exact-base upstream dependency run failed independently with the same timeout, so this is not attributable to Monty.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant