runtime: Python 3.14.6 embedded; generated apps follow the interpreter that runs them; the runtime keeps ensurepip so they can - #154
Draft
kai-openswarm wants to merge 8 commits into
Conversation
… a window, so reducer tests run under node:test Both modules touched window at import time (port/host derivation, the fetch interceptor install, the debug handle), so any node:test file that imports a reducer importing API_BASE died with 'window is not defined' before its first assertion; fetchSessionsStrip.test.ts has been red that way since the resilience work landed, unnoticed because nothing runs these tests in CI. In a renderer (window present) nothing changes: same port/host, same interceptor, same handle. Without one the module answers with the defaults and installs nothing.
Nothing ran any of them in CI: the 235-file backend pytest suite, the 22 renderer node:test files and the edge suite were run by hand, one file at a time, so a regression only surfaced when someone happened to run the right one. Three small workflows, hosted ubuntu, path-filtered, read-only token: - backend-tests: pytest on Python 3.13 from the locked requirements, plus a completion assertion (junit testcase count == collect-only count) so a test process that dies mid-run can never read as green - frontend-tests: tsc --noEmit + node:test via tsx over src/**/*.test.ts(x), through frontend/scripts/run-tests.mjs (the runner the tests already name) - edge-tests: pytest for openswarm-edge All three are green on the current tree: 2951 backend tests, 143 frontend tests across 22 files, 14 edge tests.
Two of five hosted runs of the backend suite stalled at 99% until the job cap with no summary and no junit: one test blocked forever on a bare ws.receive_json() (fixed on its own in a separate change). A CI lane should never depend on every test being unable to hang, so add pytest-timeout to the dev requirements and run the suite with --timeout=300. On Linux the default signal method fails just the offending test and the run continues, so the report and the "every collected test ran" assertion stay meaningful.
test_ws_endpoint_streams_a_full_turn_end_to_end read the socket with a bare
ws.receive_json() in a 40-iteration loop and broke only on the assistant reply.
When the loop ends early for any reason (fewer than 40 events, no reply), the
next receive blocks forever and the whole pytest run stalls at 99% until the
job cap. On hosted runners it does exactly that intermittently, on Linux and
Windows alike: the turn path's configure_provider_env decides whether 9Router
needs reviving from provider evidence earlier tests may leave behind, and that
revival spawns/installs the router behind a module-level asyncio.Lock; the
background turn-label aux call reaches the same machinery. Neither is part of
this test's contract ("SDK and WS auth mocked, everything else real").
Pin both out with monkeypatch, bound every receive at 5s (a regression now
fails this test instead of hanging the runner), and wait for the turn's
completed status before asserting on session.messages so the assertion cannot
race the loop's tail.
Same class as the config/backendConnection change: safeMode.ts read `window` at import, and dashboardLayoutSlice imports it, so any reducer test that imports the slice died under node:test before it ran. Guard the read; in a renderer nothing changes.
….14.6, 20260623) The bundled interpreter moves from 3.13.2 (indygreg 20250212) to 3.14.6 from the astral-sh python-build-standalone release 20260623, on both build scripts: - build-python-env.sh / build-python-env-win.ps1: 3.14 pins, the astral-sh download URL (with the '+' in the asset name URL-encoded), and the archive is now pinned by SHA-256 per architecture, so a moved or replaced release asset cannot ship a different interpreter. Every lib/python3.13 path becomes 3.14. The macOS Python.app launcher (LSUIElement=1, hides the Dock icon) is kept; the 3.14 launcher is statically linked and carries no libpython load command, so the dylib-path rewrite runs only when a build actually links it. Verified: the wrapper's self-test passes on the built env. - backend/requirements.lock: recompiled with --python-version 3.14. Every pinned version stays exactly as it was; only the header and two environment markers change, and the hashes cover cp314 wheels. - The workflows' setup-python pins, build-app.sh's env check, the webapp template's interpreter candidates, and the two site-packages paths in electron/main.js follow. Proof: the backend suite on 3.14.6 from this lock with the current dev pins — 2951 passed / 15 skipped (the same numbers as 3.13; the deprecation warnings it adds come from pytest-asyncio 0.25's asyncio-policy use); a packaged macOS app built with the 3.14 env: upstream smoke 5/5 + the characterization suite 19/19. Not in this change: the openswarm-edge base image (still python:3.13-slim; bump with the edge lane), and dev-dependency bumps (pytest 9 / pytest-asyncio 1.x silence the deprecations but change loop scoping and fail five runtime-cleanup tests as written; a separate change).
…runs it; the warm cache is keyed by that interpreter; importing the backend starts nothing The previous commit moves the embedded interpreter. Every generated app's backend runs on it: run.sh receives it as OPENSWARM_PYTHON and creates the workspace's .venv with it, and the shared warm venv (copied into new workspaces by backend_init.sh) is built by it. Neither noticed the interpreter changing: - run.sh reused any existing .venv (its fast path never looked inside), so after the update every app's venv still pointed at the previous interpreter and its backend never came up again; - the warm cache was keyed by the template's dependencies alone, so new apps kept receiving a venv built by the previous interpreter. Now a .venv is reused only when it is ours and its interpreter is the one that will run it; otherwise it is rebuilt — and only when it is ours: a .venv the template did not create, or a symlink/junction standing in for one, is preserved untouched with a message, and nothing is ever `rm -rf`ed. The rules live in one stdlib-only helper the template ships (backend/config/python_runtime_guard.py): ownership markers, identity (version, implementation, ABI tag, platform, machine) proven by asking the venv's interpreter, `pyvenv.cfg` checked before that interpreter is ever run, reparse points refused. Venvs the previous template wrote (their `.openswarm_installed` sentinel is present) are adopted, then judged like any other. Nothing pins a version: the comparison is against the running interpreter, so the next runtime bump needs no code change here. The warm cache (backend/apps/outputs/python_venv/) is keyed by the template's dependencies and the interpreter's identity, built in a staging directory and published atomically, owned by markers, and only reported complete after its interpreter proves it is ours. It installs exactly what webapp_template/backend/pyproject.toml declares (parsed, not a duplicate list). backend_init.sh copies only an owned, complete cache, through a staging directory; run.sh re-verifies the copy with the interpreter that will run it. backend_init.sh runs from the agent's shell, which has no OPENSWARM_PYTHON, so it uses whatever Python 3 that shell has (macOS's own 3.9 included) for its two stdlib-only jobs — the structural cache check and picking a free port (which no longer hard-codes `python3`, absent on Windows). Importing backend.main is now side-effect free: the App Builder's warmup (template node_modules + backend venv) is registered with the health route and runs as a background task after the first successful /api/health/check, so a venv build and a pip install cannot compete with the server's own start-up on a cold machine, overlapping health probes cannot start two warmers, and importing the module in tests or an import probe starts nothing. OPENSWARM_BACKEND_IMPORT_ONLY=1 (the shell's import probe) and OPENSWARM_DISABLE_TEMPLATE_WARM_CACHE=1 skip the registration. The health.py hook is the same one openswarm-ai#152 adds, so either PR can merge first. Fixed in passing, both found while proving this: values written into a workspace's .env are shell-quoted, so an install path with a space (or backslashes) no longer breaks `source .env` in the template's scripts (and a re-seed no longer runs a Windows path through re.sub's escape processing); a `python -m venv` that fails half-way (a runtime without ensurepip, a full disk) now cleans up what it created instead of leaving a .venv the next start must not touch. Tests: python_venv (identity/ownership/atomic publish/lock), the guard's CLI on real venvs and through run.sh (unowned preserved, other-interpreter rebuilt, failed creation cleaned up, system-python cache check), the .env round trip through bash and read_env_value, and the import/health behaviour in a subprocess. Full backend suite: 2994 passed / 16 skipped on 3.14 and on 3.13.
…er creates with it actually get created Since bf6d7a9 the App Builder creates every generated app's backend venv, and the shared warm venv, with the bundled interpreter (run.sh receives it as OPENSWARM_PYTHON; view_builder_templates uses sys.executable). Since 9001cb8 the macOS build script strips `ensurepip` from that interpreter, on the reasoning that the backend never installs packages at runtime — true when written, eight days before bf6d7a9 made it false. `python -m venv` bootstraps pip into the new environment from ensurepip's bundled wheel, so with ensurepip gone the venv step fails and no generated backend can start in the packaged macOS app. The Windows script listed `lib\python3.14\ensurepip` for removal, which never matched the Windows `Lib\` layout, so Windows kept ensurepip by accident — which is why the feature looked fine there. Keep ensurepip on both platforms (~1.8 MB; the parent's own pip stays stripped — the venvs get theirs from ensurepip's wheel), correct the two comments that still say nothing calls `python -m venv`, and add a self-check after every trim: the trimmed interpreter must create a venv that has pip, or the build fails, so a future strip cannot take this away silently. Rebuilt the arm64 env with this script: the self-check passes, `python3 -m venv` from the trimmed runtime yields a venv with pip 26.1.2, `import pip` in the parent still fails as intended, size 470 → 472 MB.
kai-openswarm
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
1.
build: the embedded runtime is Python 3.14 (python-build-standalone 3.14.6, 20260623)The bundled interpreter moves from 3.13.2 (indygreg 20250212) to 3.14.6 from the astral-sh python-build-standalone release 20260623, on both build scripts:
scripts/build-python-env.sh/build-python-env-win.ps1: 3.14 pins, the astral-sh download URL (with the+in the asset name URL-encoded), and the archive is now pinned by SHA-256 per architecture, so a moved or replaced release asset cannot ship a different interpreter. Everylib/python3.13path becomes 3.14. The macOSPython.applauncher (LSUIElement=1, hides the Dock icon) is kept; the 3.14 launcher is statically linked and carries no libpython load command, so the dylib-path rewrite runs only when a build actually links it — the wrapper's self-test passes on the built env.backend/requirements.lock: recompiled with--python-version 3.14. Every pinned version stays exactly as it was; only the header and two environment markers change, and the hashes already cover cp314 wheels.setup-pythonpins,build-app.sh's env check, the webapp template's interpreter candidates, the two site-packages paths inelectron/main.js, and the README follow.2.
outputs: a generated app's backend venv follows the interpreter that runs it; the warm cache is keyed by that interpreter; importing the backend starts nothingWhy it is in this PR: commit 1 alone would break every existing generated app on update.
run.shreceives the bundled interpreter asOPENSWARM_PYTHONand creates the workspace's.venvwith it, but its fast path reused any existing.venvblindly — after the update those venvs still point at the removed 3.13 interpreter and the backend never comes up. The shared warm venv (copied into new workspaces bybackend_init.sh) was keyed by the template's dependencies alone, so new apps kept receiving a venv built by the old interpreter.Now a
.venvis reused only when it is ours and its interpreter is the one that will run it; otherwise it is rebuilt — and only when it is ours: a.venvthe template did not create, or a symlink/junction standing in for one, is preserved untouched with a message, and nothing is everrm -rfed. The rules live in one stdlib-only helper the template ships,backend/config/python_runtime_guard.py(identity/verify/verify-cache/claim-created/remove-owned): ownership markers, identity (version, implementation, ABI cache tag, platform, machine) proven by asking the venv's interpreter,pyvenv.cfgchecked before that interpreter is ever run, reparse points refused. Venvs the previous template wrote (their.openswarm_installedsentinel is present) are adopted, then judged like any other. Nothing pins a version — the comparison is against the running interpreter, so the next runtime bump needs no code change here.backend/apps/outputs/python_venv/(identity / safety / cache): the warm cache is keyed by the template's dependencies and the interpreter's identity, built in a staging directory and published atomically, owned by markers, and only reported complete after its interpreter proves it is ours. It installs exactly whatwebapp_template/backend/pyproject.tomldeclares (parsed withtomllib— no more duplicated dependency list to keep in sync).backend_init.shcopies only an owned, complete cache, through a staging directory;run.shre-verifies the copy with the interpreter that will run it.backend_init.shruns from the agent's shell, which has noOPENSWARM_PYTHON, so it uses whatever Python 3 that shell has (macOS's own 3.9 included) for its two stdlib-only jobs — the structural cache check and picking a free port (which no longer hard-codespython3, absent on Windows).backend.mainis side-effect free. The App Builder's warmup (template node_modules + backend venv) is registered with the health route and runs as a background task after the first successful/api/health/check, so a venv build and a pip install cannot compete with the server's own start-up on a cold machine, overlapping health probes cannot start two warmers, and importing the module in tests or an import probe starts nothing.OPENSWARM_BACKEND_IMPORT_ONLY=1(the shell's import probe) andOPENSWARM_DISABLE_TEMPLATE_WARM_CACHE=1skip the registration. Thehealth.pyhook is byte-identical to the one backend: agents/service/workflows hardening, plus the hosting seam the routes now ask #152 adds, so either PR can merge first..envare shell-quoted, so an install path with a space (or backslashes) no longer breakssource .envin the template's scripts (and a re-seed no longer runs a Windows path throughre.sub's escape processing); apython -m venvthat fails half-way cleans up what it created instead of leaving a.venvthe next start must not touch.3.
build: the bundled Python keeps ensurepip, so the venvs the App Builder creates with it actually get createdFound by trying commit 2 on the packaged runtime: since bf6d7a9 the App Builder creates every generated backend's venv (and the warm venv) with the bundled interpreter, and since 9001cb8 the macOS build script strips
ensurepipfrom it — true-when-written reasoning ("the backend never installs packages at runtime") that bf6d7a9 made false eight days later.python -m venvbootstraps pip into the new environment from ensurepip's bundled wheel, so with it gone the venv step fails and no generated backend can start in the packaged macOS app. The Windows script listedlib\python3.14\ensurepipfor removal, which never matched theLib\layout, so Windows kept it by accident — which is why the feature looked fine there. Keepensurepipon both platforms (~1.8 MB; the parent's own pip stays stripped — the venvs get theirs from ensurepip's wheel), correct the two comments that still say nothing callspython -m venv, and add a self-check after every trim: the trimmed interpreter must create a venv that has pip, or the build fails.Proof
run.sh(unowned preserved, other-interpreter rebuilt, failed creation cleaned up, system-python cache check),python_venv(identity/ownership/atomic publish/process lock), the.envround trip through bash andread_env_value, and the import/health behaviour in a subprocess.backend_init.sh(system python 3.9) copies it →run.shunder 3.13 verifies, installs, serves 200 → the same workspace under 3.14:stale: pyvenv.cfg was not written by Python 3.14.6 … Discarding the OpenSwarm-generated virtual environment … Creating virtual environment … Installing dependencies …→ serves 200 under 3.14.6 → restart = fast path; per-interpreter cache directories distinct and each intact; a symlinked.venvpreserved.python3 -m venvfrom the trimmed runtime yields a venv with pip 26.1.2;import pipin the parent still fails as intended; 470 → 472 MB); a packaged macOS app with that env and this backend: upstream smoke 5/5 + the characterization suite (e2e: characterization suite for the 1.7.4-1.7.7 release notes (packaged app, no provider key) #146) 19/19, and the backend log showsGET /api/health/check 200→backend-http-ready→ the warmup starting, in that order.Not in this change: the
openswarm-edgebase image (stillpython:3.13-slim; bump with the edge lane), and dev-dependency bumps (pytest 9 / pytest-asyncio 1.x silence the deprecations but change loop scoping and fail five runtime-cleanup tests as written; a separate change).