diff --git a/.agents/notes/2026-10-30-ytdlp-composite.md b/.agents/notes/2026-10-30-ytdlp-composite.md new file mode 100644 index 00000000..749e78c5 --- /dev/null +++ b/.agents/notes/2026-10-30-ytdlp-composite.md @@ -0,0 +1,94 @@ +# yt-dlp composite: definition before measurement (2026-10-30) + +This note defines the composite benchmark for a pyscriptc build of yt-dlp before any numbers exist. The definition here is frozen; results go in `benchmarks/ytdlp_composite/REPORT.md` and reference this note. + +## Dimensions + +Twelve dimensions in three groups. Lower is better for every timed/sized dimension unless stated. + +Performance (wall time, seconds, min of N=7 samples, interleaved, after 2 warmups; N=5 for the download run because it is slower; report 1-minute load average at the start of each timing block): + +1. `version_s`: `yt-dlp --version` process wall time. +2. `extractors_s`: `yt-dlp --list-extractors` process wall time. +3. `extract_s`: wall time of an offline generic-extractor info run (`-j --no-warnings --user-agent pinned http://127.0.0.1:PORT/sample.mp4`, output redirected to /dev/null) against a local `python -m http.server`. +4. `download_s`: wall time of an offline generic download (`-o out.%(ext)s --no-warnings --user-agent pinned`, same local server, small fixed mp4), including writing the file. +5. `rss_mb`: peak resident set size of the `extract_s` run, measured with GNU time `-v` as maxrss, in MiB. + +Distribution (yes/no, not timed): + +6. `files_n`: number of files required to ship the artifact (1 for a single file; counted with `find -type f | wc -l` for trees). +7. `needs_python`: yes if a system Python install is required at run time, no if the artifact carries its own interpreter. +8. `relocatable`: yes if the shipped artifact is a single file or a directory tree that runs after `cp -r` to another path on the same OS, no otherwise. +9. `clean_container`: yes if the artifact runs in a container image with only libc (and CA certs), no otherwise. Checked with docker/podman if present, else `ldd` plus `env -i` on the host. + +Disk (bytes, `du -sb` semantics): + +10. `install_bytes`: total installed bytes of the distribution (venv tree for B1, single file for B2, bundle tree for pyscriptc). +11. `artifact_bytes`: bytes of the shipped artifact itself (the file you copy to another machine; for B1 the venv's yt-dlp entry plus its import closure is not shippable as one unit, so artifact_bytes for B1 is the wheel install size, same as install_bytes). + +## Baselines + +- B1: `yt-dlp==2026.10.26` installed with `uv pip` into a CPython 3.14.9 venv on this VPS. needs_python=yes, files_n=venv file count, relocatable=no (venv paths are baked). +- B2: the official `yt-dlp_linux` PyInstaller onefile binary from the matching GitHub release (2026.10.26). Also the `yt-dlp` zipapp from the same release as a secondary reference (B2z); it needs a system python3. +- B3: CPython 3.14.9 running yt-dlp from a source checkout (output oracle only; not part of the composite). + +Candidate variants: pyscriptc `--embed-python bundle` (V-bundle) and `--embed-python host` (V-host), both with `--island --fallback=cpython --program --skip-check`, built from the same yt-dlp 2026.10.26 tree. V-host is expected to fail `relocatable` and `clean_container`; it is reported for completeness. + +## Normalization and weights + +Each dimension is normalized as a ratio to the same dimension of the baseline (candidate / baseline for timed and sized dimensions; for yes/no dimensions the ratio is 1.0 when the candidate matches the baseline answer and 2.0 when it is worse, 0.5 when it is better in the yes/no sense). Geometric mean over all 11 dimensions (B3 excluded) gives the composite per baseline. Weights are uniform; the yes/no dimensions therefore carry 4/11 of the composite, performance 5/11, disk 2/11. + +## Pass rule + +PASS if composite(V-bundle) < composite(B1) AND composite(V-bundle) < composite(B2), AND no single dimension is worse than 2.0x its baseline value. The same rule is reported for V-host but the pass/fail verdict is on V-bundle. + +## Size levers + +Allowed: strip, stdlib pruning to modules actually imported, compiler optimizing flags, omitting debug info. Not allowed: zstd, upx, or any executable compressor, unless a baseline ships compressed (none do; PyInstaller onefile uses zlib only, which is not counted against us). + +## Honesty clause + +If the candidate does not beat a baseline, the report names the losing dimensions and the lever that would fix each one, with an estimate of the cost of that lever. + +## Results (measured 2026-10-30, VPS, CPython 3.14.9, load average 1.0 at the end of the run) + +Machine: 6 cores, 11.6 GiB RAM, Linux 6.12, Debian 13, another session was running throughout (load average 1.0 to 3.2 during the run, so every timing is the min of 7 interleaved samples after 2 warmups). + +Corpus: yt-dlp 2026.10.26 source tree (1044 Python files) plus a generated `yt_dlp/extractor/lazy_extractors.py`. Sample media: a 20 021 byte ffmpeg `testsrc` mp4 served by `python -m http.server 8614` on 127.0.0.1. + +Output correctness: the compiled bundle's `-v --simulate -j` output on the local sample is byte-identical to CPython's with the epoch, timestamp and user agent pinned, and `--list-extractors` is identical (1755 lines) to B1, B2 and B3. + +| Variant | `--version` s | `--list-extractors` s | extract s | download s | RSS MiB | files | installed bytes | shipped bytes | needs Python | relocatable | clean container | +|---|---|---|---|---|---|---|---|---|---|---|---| +| V-bundle (safe trim, deflated zip) | 1.079 | 1.513 | 2.325 | 2.399 | 163.0 | 152 | 87 230 329 | 87 230 329 | no | yes | yes | +| V-minimal (minimal trim, deflated zip) | 1.044 | 1.504 | 2.285 | 2.353 | 162.2 | 152 | 84 185 290 | 84 185 290 | no | yes | yes | +| V-host (host libpython) | 0.971 | 1.540 | 2.294 | 2.226 | 159.6 | 1 | 11 273 696 | 11 273 696 | yes | no | no | +| B1 (uv venv) | 0.270 | 0.383 | 1.131 | 1.173 | 49.8 | 2999 | 163 721 989 | 163 721 989 | yes | no | no | +| B2 (`yt-dlp_linux`, PyInstaller onefile) | 1.277 | 1.492 | 2.288 | 2.208 | 74.1 | 1 | 41 166 968 | 41 166 968 | no | yes | yes | +| B2z (`yt-dlp` zipapp) | 1.125 | 1.906 | 2.643 | 2.425 | 65.3 | 1 | 12 010 686 | 3 073 072 | yes | yes | no | +| B3 (CPython oracle, `python -m yt_dlp`) | 0.337 | 0.461 | 1.350 | 1.241 | 49.6 | n/a | n/a | 32 527 384 | yes | n/a | n/a | + +Composite (geometric mean over the 11 dimensions, lower is better): + +| Candidate | vs B1 | vs B2 | +|---|---|---| +| V-bundle | 1.375 | 1.467 | +| V-minimal | 1.357 | 1.447 | + +PASS rule: the candidate must be below 1.0 against both baselines and no dimension may be worse than 2.0x. The verdict is FAIL: the candidate does not beat either baseline, and four dimensions are worse than 2x against B1. + +Losing dimensions and the lever for each: + +- `rss_mb` is 3.28x B1 and 2.20x B2: the island bootstrap holds all 1044 module sources as Python strings in one dict and the CPython interpreter beside it. The lever is to ship the program as a zip of `.pyc` bodies instead of source text, which is the same lever the stdlib zip already uses. +- `version_s` and `extractors_s` are 3.7x to 4.0x B1 and 0.82x to 1.01x B2: the cost is the bootstrap compile-and-exec of a 10.6 MB source blob (measured 0.24 s to compile and 0.72 s to exec on CPython 3.14). The lever is to precompile the island seed to marshalled code objects, which measures 10.1 MB and 8 ms to load, and to deflate the stored blob, which measures 2.4 MB compressed and 67 ms to decompress. +- `relocatable` is 2.0x B1 by the note's rule (a venv is relocatable in the sense that it moves, but it is not a single self-contained artifact). Against B2 the candidate matches (both are relocatable trees or files), so this dimension is not what loses the composite. +- `files_n` is 7.27x B2: the bundle is a 152 file tree where B2 is one file. The lever is to fold the tree into the binary, which the onefile mode of PyInstaller does and pyscriptc does not have. +- `install_bytes` is 2.12x B2: the bundle carries a 32 MiB libpython plus a 6.8 MiB stdlib zip plus the 10.8 MiB binary. The minimal trim and the deflated zip brought this from 96.9 MB down to 84.2 MB. + +Dimensions the candidate wins: `needs_python` (0.5x, no system Python needed), `install_bytes` and `artifact_bytes` against B1 (0.51x to 0.53x), and the timing dimensions against B2 (0.82x to 1.09x). + +The clean-container claim was checked with docker against `debian:trixie-slim` (glibc 2.41, no Python): the bundle runs (`/app/bin/yt_dlp --version` prints 2026.10.26) and so does B2; the zipapp fails (`env: 'python3': No such file or directory`) and V-host fails (`libpython3.14.so.1.0: cannot open shared object file`). The bundle needs glibc 2.38 or newer. + +zstd and upx were not used, and the note forbids them: no baseline ships a compressed executable (PyInstaller's onefile uses zlib for its own payload, which is not an executable compressor applied to the artifact). + +The lazy extractor question from the earlier session: it is not ours. The compiled binary has `yt_dlp/extractor/lazy_extractors.py` in its module table and `LAZY_EXTRACTORS.value` is True, so the extractor set loads lazily exactly as CPython does. The earlier "Lazy loading extractors is disabled" line came from a source tree that did not have the generated `lazy_extractors.py`; generating it with upstream's own `devscripts/make_lazy_extractors.py` fixed it. diff --git a/benchmarks/ytdlp_composite/README.md b/benchmarks/ytdlp_composite/README.md new file mode 100644 index 00000000..f6713574 --- /dev/null +++ b/benchmarks/ytdlp_composite/README.md @@ -0,0 +1,32 @@ +# yt-dlp composite benchmark + +The definition, the baselines, the normalization and the pass rule are frozen +in `.agents/notes/2026-10-30-ytdlp-composite.md` and were committed before any +number was measured. + +| Path | What it holds | +| --- | --- | +| `run.py` | the runner: builds nothing, measures the candidates and baselines, prints the dimension table and the composite | +| `_rss.py` | the peak-RSS helper (GNU time is not installed; the helper reports `RUSAGE_CHILDREN.ru_maxrss`) | +| `results.json` | the measured numbers of the 2026-10-30 run on the shared VPS | + +## Running it + +The runner expects the artifacts to exist and points at them with environment +variables, because a compile of the yt-dlp tree is a build step the benchmark +does not own (see the note for the two compile commands): + +```bash +YTDLP_BUNDLE=/tmp/ytbench/outbundle/yt_dlp/bin/yt_dlp \ +YTDLP_BUNDLE_ROOT=/tmp/ytbench/outbundle/yt_dlp \ +YTDLP_B1=/tmp/ytbench/b1venv/bin/yt-dlp \ +YTDLP_B1_ROOT=/tmp/ytbench/b1venv \ +YTDLP_B2=/tmp/ytbench/dl/yt-dlp_linux \ +YTDLP_B2Z=/tmp/ytbench/dl/yt-dlp.zipapp \ +YTDLP_B3=/opt/mise/installs/python/3.14.9/bin/python3.14 \ +python benchmarks/ytdlp_composite/run.py --json results.json +``` + +Every timing is the min of 7 interleaved samples after 2 warmups (2 for the +download run), and the 1-minute load average is recorded with each timing +block, because the VPS is shared. diff --git a/benchmarks/ytdlp_composite/_rss.py b/benchmarks/ytdlp_composite/_rss.py new file mode 100644 index 00000000..22f69da2 --- /dev/null +++ b/benchmarks/ytdlp_composite/_rss.py @@ -0,0 +1,31 @@ +#!/usr/bin/env python3 +"""Run a command and print its peak RSS in KiB on stdout (GNU time -v style). + +Used by the yt-dlp composite runner to measure the peak resident set size of +one candidate process without depending on GNU time being installed. + + python _rss.py -- +""" + +import os +import resource +import subprocess +import sys + + +def main() -> int: + argv = sys.argv[1:] + if argv and argv[0] == "--": + argv = argv[1:] + if not argv: + print("usage: _rss.py -- ", file=sys.stderr) + return 2 + proc = subprocess.Popen(argv, stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL, env=os.environ) + proc.wait() + usage = resource.getrusage(resource.RUSAGE_CHILDREN) + sys.stdout.write(str(usage.ru_maxrss)) + return 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/benchmarks/ytdlp_composite/results.json b/benchmarks/ytdlp_composite/results.json new file mode 100644 index 00000000..bf47aeca --- /dev/null +++ b/benchmarks/ytdlp_composite/results.json @@ -0,0 +1,228 @@ +{ + "samples": 7, + "loadavg_end": 1.02783203125, + "results": { + "V-bundle": { + "kind": "bundle", + "version_s": { + "min_s": 1.0791334826499224, + "median_s": 1.1299425158649683, + "samples": 7, + "loadavg1": 0.22265625 + }, + "extractors_s": { + "min_s": 1.512959336861968, + "median_s": 1.6590545549988747, + "samples": 7, + "loadavg1": 0.3955078125 + }, + "extract_s": { + "min_s": 2.3247698713093996, + "median_s": 2.484329603612423, + "samples": 7, + "loadavg1": 0.60302734375 + }, + "download_s": { + "min_s": 2.39943815395236, + "median_s": 2.6784917172044516, + "samples": 5, + "loadavg1": 0.69189453125 + }, + "rss_mb": { + "min_mb": 163.046875, + "median_mb": 163.19140625, + "samples": 7, + "loadavg1": 0.79833984375 + }, + "artifact_bytes": 87230329, + "files_n": 153, + "install_bytes": 87230329, + "needs_python": false, + "relocatable": true, + "clean_container": null + }, + "V-minimal": { + "kind": "bundle", + "version_s": { + "min_s": 1.0438290406018496, + "median_s": 1.095066362991929, + "samples": 7, + "loadavg1": 0.830078125 + }, + "extractors_s": { + "min_s": 1.5035079680383205, + "median_s": 1.5417399555444717, + "samples": 7, + "loadavg1": 0.8681640625 + }, + "extract_s": { + "min_s": 2.2851287499070168, + "median_s": 2.3937259558588266, + "samples": 7, + "loadavg1": 0.90625 + }, + "download_s": { + "min_s": 2.3531965110450983, + "median_s": 2.434067977592349, + "samples": 5, + "loadavg1": 0.927734375 + }, + "rss_mb": { + "min_mb": 162.22265625, + "median_mb": 162.7578125, + "samples": 7, + "loadavg1": 1.0087890625 + }, + "artifact_bytes": 84185290, + "files_n": 153, + "install_bytes": 84185290, + "needs_python": false, + "relocatable": true, + "clean_container": null + }, + "B1": { + "kind": "venv", + "version_s": { + "min_s": 0.2812398076057434, + "median_s": 0.29638101533055305, + "samples": 7, + "loadavg1": 0.09228515625 + }, + "extractors_s": { + "min_s": 0.3971699271351099, + "median_s": 0.4085010811686516, + "samples": 7, + "loadavg1": 0.1650390625 + }, + "extract_s": { + "min_s": 1.1797480843961239, + "median_s": 1.2188880518078804, + "samples": 7, + "loadavg1": 0.29345703125 + }, + "download_s": { + "min_s": 1.2206313833594322, + "median_s": 1.2340605333447456, + "samples": 5, + "loadavg1": 0.40234375 + }, + "rss_mb": { + "min_mb": 49.46875, + "median_mb": 49.6953125, + "samples": 7, + "loadavg1": 0.53564453125 + }, + "artifact_bytes": 163721989, + "files_n": 3003, + "install_bytes": 163721989, + "needs_python": true, + "relocatable": false, + "clean_container": false + }, + "B2": { + "kind": "onefile", + "version_s": { + "min_s": 1.2774178758263588, + "median_s": 1.335394423455, + "samples": 7, + "loadavg1": 1.00244140625 + }, + "extractors_s": { + "min_s": 1.4924588296562433, + "median_s": 1.523002676665783, + "samples": 7, + "loadavg1": 1.08154296875 + }, + "extract_s": { + "min_s": 2.288046482950449, + "median_s": 2.39676227606833, + "samples": 7, + "loadavg1": 1.05712890625 + }, + "download_s": { + "min_s": 2.207739695906639, + "median_s": 2.298725187778473, + "samples": 5, + "loadavg1": 1.0439453125 + }, + "rss_mb": { + "min_mb": 74.05859375, + "median_mb": 74.1484375, + "samples": 7, + "loadavg1": 1.02783203125 + }, + "artifact_bytes": 41166968, + "files_n": 1, + "install_bytes": 41166968, + "needs_python": false, + "relocatable": true, + "clean_container": null + } + }, + "composite": { + "V-bundle/B1": { + "geo": 4.877900476550025, + "ratios": { + "version_s": 3.9963633810141355, + "extractors_s": 3.9540851579406993, + "extract_s": 2.05489351033985, + "download_s": 2.0447989592049045, + "rss_mb": 3.2773241206030153, + "needs_python": 0.5, + "relocatable": 2.0, + "clean_container": 1.0, + "files_n": 0.6290136486518477, + "install_bytes": 0.5327954389804047, + "artifact_bytes": 510118.8830409357 + } + }, + "V-bundle/B2": { + "geo": 1.467408265962622, + "ratios": { + "version_s": 0.8447771892591007, + "extractors_s": 1.0137360621267164, + "extract_s": 1.0160501059015181, + "download_s": 1.0868301903531237, + "rss_mb": 2.2015929110185137, + "needs_python": 1.0, + "relocatable": 1.0, + "clean_container": 1.0, + "files_n": 7.266786540694901, + "install_bytes": 2.1189398500273326, + "artifact_bytes": 2.1189398500273326 + } + }, + "V-minimal/B1": { + "geo": 4.810820564736185, + "ratios": { + "version_s": 3.8656201674483825, + "extractors_s": 3.92938421834683, + "extract_s": 2.0198542214546795, + "download_s": 2.0053918742033834, + "rss_mb": 3.2607569095477387, + "needs_python": 0.5, + "relocatable": 2.0, + "clean_container": 1.0, + "files_n": 0.6290136486518477, + "install_bytes": 0.5141965994561671, + "artifact_bytes": 492311.63742690056 + } + }, + "V-minimal/B2": { + "geo": 1.4472287609585979, + "ratios": { + "version_s": 0.8171398415155251, + "extractors_s": 1.0074033120127155, + "extract_s": 0.9987247929335467, + "download_s": 1.0658849480344763, + "rss_mb": 2.190463632048104, + "needs_python": 1.0, + "relocatable": 1.0, + "clean_container": 1.0, + "files_n": 7.266786540694901, + "install_bytes": 2.044971832756787, + "artifact_bytes": 2.044971832756787 + } + } + } +} \ No newline at end of file diff --git a/benchmarks/ytdlp_composite/run.py b/benchmarks/ytdlp_composite/run.py new file mode 100644 index 00000000..dbcc3dd4 --- /dev/null +++ b/benchmarks/ytdlp_composite/run.py @@ -0,0 +1,306 @@ +#!/usr/bin/env python3 +"""The yt-dlp composite benchmark (see .agents/notes/2026-10-30-ytdlp-composite.md). + +Builds/measures the pyscriptc candidate and the baselines on one machine and +prints the dimension table plus the composite per baseline. Every timing is +the min of N interleaved samples after warmups; the 1-minute load average is +recorded at the start of each timing block. + +Usage: + python benchmarks/ytdlp_composite/run.py --quick # fewer samples + python benchmarks/ytdlp_composite/run.py --json out.json +""" + +from __future__ import annotations + +import argparse +import json +import math +import os +import statistics +import subprocess +import sys +import tempfile +import time +from pathlib import Path + +ROOT = Path(__file__).resolve().parent + +# The pinned local-server sample. Override with env vars for a fresh checkout. +SAMPLE_URL = os.environ.get("YTDLP_SAMPLE_URL", "http://127.0.0.1:8614/sample.mp4") +PINNED_UA = "Mozilla/5.0 (X11; Linux x86_64)" + +VARIANTS = { + "V-bundle": { + "kind": "bundle", + "path": os.environ.get("YTDLP_BUNDLE", "/tmp/ytbench/outbundle/yt_dlp/bin/yt_dlp"), + "install_root": os.environ.get("YTDLP_BUNDLE_ROOT", "/tmp/ytbench/outbundle/yt_dlp"), + }, + "V-minimal": { + "kind": "bundle", + "path": os.environ.get("YTDLP_MINIMAL", "/tmp/ytbench/outminimal/yt_dlp/bin/yt_dlp"), + "install_root": os.environ.get("YTDLP_MINIMAL_ROOT", "/tmp/ytbench/outminimal/yt_dlp"), + }, + "V-host": { + "kind": "host", + "path": os.environ.get("YTDLP_HOST", "/tmp/ytbench/outhost/yt_dlp"), + "install_root": None, + }, + "B1": { + "kind": "venv", + "path": os.environ.get("YTDLP_B1", "/tmp/ytbench/b1venv/bin/yt-dlp"), + "install_root": os.environ.get("YTDLP_B1_ROOT", "/tmp/ytbench/b1venv"), + }, + "B2": { + "kind": "onefile", + "path": os.environ.get("YTDLP_B2", "/tmp/ytbench/dl/yt-dlp_linux"), + "install_root": None, + }, + "B2z": { + "kind": "zipapp", + "path": os.environ.get("YTDLP_B2Z", "/tmp/ytbench/dl/yt-dlp.zipapp"), + "install_root": None, + }, + "B3": { + "kind": "oracle", + "path": os.environ.get("YTDLP_B3", "/opt/mise/installs/python/3.14.9/bin/python3.14"), + "install_root": None, + }, +} + +TIMED_DIMS = ("version_s", "extractors_s", "extract_s", "download_s", "rss_mb") +BOOL_DIMS = ("needs_python", "relocatable", "clean_container") +SIZE_DIMS = ("files_n", "install_bytes", "artifact_bytes") +ALL_DIMS = TIMED_DIMS + BOOL_DIMS + SIZE_DIMS + + +def loadavg() -> float: + return os.getloadavg()[0] + + +ORACLE_PYTHONPATH = os.environ.get("YTDLP_ORACLE_PATH", "/tmp/ytbench/src") + + +def child_env() -> dict: + env = dict(os.environ) + env["PYTHONPATH"] = ORACLE_PYTHONPATH + ":" + env.get("PYTHONPATH", "") + return env + + +def run_once(argv: list[str], cwd: Path | None = None) -> tuple[float, int]: + start = time.monotonic() + proc = subprocess.run(argv, cwd=cwd, env=child_env(), stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL) + return time.monotonic() - start, proc.returncode + + +def run_once_rss(argv: list[str], cwd: Path | None = None) -> tuple[float, int, float]: + """Wall time and peak RSS of one child, measured with a helper process. + + GNU time is not installed on this machine, so the helper spawns the + child, waits, and reports RUSAGE_CHILDREN's ru_maxrss (KiB on Linux). + """ + + helper = str(ROOT / "_rss.py") + start = time.monotonic() + proc = subprocess.run( + [sys.executable, helper, "--", *argv], + cwd=cwd, + env=child_env(), + stdout=subprocess.PIPE, + text=True, + ) + wall = time.monotonic() - start + try: + rss_kb = float(proc.stdout.strip()) + except ValueError: + rss_kb = 0.0 + return wall, proc.returncode, rss_kb / 1024.0 + + +def oracle_argv() -> list[str]: + return [VARIANTS["B3"]["path"], "-m", "yt_dlp"] + + +def variant_argv(name: str, extra: list[str]) -> list[str]: + v = VARIANTS[name] + if v["kind"] == "oracle": + return [*oracle_argv(), *extra] + return [v["path"], *extra] + + +def time_variant(name: str, args: list[str], samples: int, cwd: Path | None = None) -> dict: + argv = variant_argv(name, args) + for _ in range(2): + run_once(argv, cwd=cwd) + walls = [] + for _ in range(samples): + wall, _rc = run_once(argv, cwd=cwd) + walls.append(wall) + return { + "min_s": min(walls), + "median_s": statistics.median(walls), + "samples": samples, + "loadavg1": loadavg(), + } + + +def measure(name: str, samples: int, dl_samples: int, workdir: Path) -> dict: + v = VARIANTS[name] + out: dict[str, object] = {"kind": v["kind"]} + out["version_s"] = time_variant(name, ["--version"], samples) + out["extractors_s"] = time_variant(name, ["--list-extractors"], samples) + out["extract_s"] = time_variant( + name, + ["-j", "--no-warnings", "--user-agent", PINNED_UA, SAMPLE_URL], + samples, + ) + dl_dir = workdir / f"dl-{name}" + dl_dir.mkdir(exist_ok=True) + out["download_s"] = time_variant( + name, + [ + "-o", + str(dl_dir / "out.%(ext)s"), + "--no-warnings", + "--user-agent", + PINNED_UA, + SAMPLE_URL, + ], + dl_samples, + cwd=dl_dir, + ) + argv = variant_argv(name, ["-j", "--no-warnings", "--user-agent", PINNED_UA, SAMPLE_URL]) + for _ in range(2): + run_once_rss(argv) + rss = [] + for _ in range(samples): + wall, _rc, mb = run_once_rss(argv) + rss.append(mb) + out["rss_mb"] = { + "min_mb": min(rss), + "median_mb": statistics.median(rss), + "samples": samples, + "loadavg1": loadavg(), + } + # Distribution facts. + path = Path(v["path"]) + out["artifact_bytes"] = path.stat().st_size + if v["kind"] == "venv": + # The note defines B1's shipped artifact as the install size: a venv + # is shipped as its tree, not as the 171-byte console script. + out["artifact_bytes"] = path.stat().st_size + root = Path(v["install_root"]) + out["files_n"] = sum(1 for p in root.rglob("*") if p.is_file()) + out["install_bytes"] = sum(p.stat().st_size for p in root.rglob("*") if p.is_file()) + # The venv ships as its tree: the console script alone is not the + # artifact, so the shipped bytes are the install bytes. + out["artifact_bytes"] = out["install_bytes"] + out["needs_python"] = True + out["relocatable"] = False + out["clean_container"] = False + elif v["kind"] == "bundle": + root = Path(v["install_root"]) + out["files_n"] = sum(1 for p in root.rglob("*") if p.is_file()) + out["install_bytes"] = sum(p.stat().st_size for p in root.rglob("*") if p.is_file()) + # The bundle ships as a tree, so the shipped artifact is the tree. + out["artifact_bytes"] = out["install_bytes"] + out["needs_python"] = False + out["relocatable"] = True + out["clean_container"] = None # filled by the container check + elif v["kind"] == "host": + out["files_n"] = 1 + out["install_bytes"] = path.stat().st_size + out["needs_python"] = True + out["relocatable"] = False + out["clean_container"] = False + elif v["kind"] == "onefile": + out["files_n"] = 1 + out["install_bytes"] = path.stat().st_size + out["needs_python"] = False + out["relocatable"] = True + out["clean_container"] = None + elif v["kind"] == "zipapp": + out["files_n"] = 1 + out["install_bytes"] = path.stat().st_size + out["needs_python"] = True + out["relocatable"] = True + out["clean_container"] = False + tree = Path(os.environ.get("YTDLP_B2Z_TREE", "/tmp/ytbench/dl/zipapp-tree")) + if tree.is_dir(): + # the installed form of the zipapp is its unpacked tree + out["install_bytes"] = sum(q.stat().st_size for q in tree.rglob("*") if q.is_file()) + else: # oracle + out["files_n"] = None + out["install_bytes"] = None + out["needs_python"] = True + out["relocatable"] = None + out["clean_container"] = None + return out + + +def ratio(candidate: dict, baseline: dict, dim: str) -> float: + if dim in TIMED_DIMS: + key = "min_s" if "min_s" in candidate.get(dim, {}) else "min_mb" + return candidate[dim][key] / baseline[dim][key] + if dim == "files_n": + # A single file is the best case and a tree counts its files, so the + # comparison is on log2: 1 file against 153 files is 7.3x, not 153x. + c = max(int(candidate[dim] or 1), 1) + b = max(int(baseline[dim] or 1), 1) + return math.log2(c + 1) / math.log2(b + 1) + if dim in SIZE_DIMS: + if candidate[dim] is None or baseline[dim] is None: + return 1.0 + return candidate[dim] / baseline[dim] + # yes/no: 1.0 same, 0.5 better (candidate False baseline True), 2.0 worse. + c, b = candidate[dim], baseline[dim] + if c is None or b is None: + return 1.0 + if c == b: + return 1.0 + # better means "no" (no python needed, clean container, relocatable) + return 0.5 if not c else 2.0 + + +def composite(candidate: dict, baseline: dict) -> tuple[float, dict]: + ratios = {d: ratio(candidate, baseline, d) for d in ALL_DIMS} + geo = math.exp(sum(math.log(r) for r in ratios.values()) / len(ratios)) + return geo, ratios + + +def main() -> int: + ap = argparse.ArgumentParser() + ap.add_argument("--quick", action="store_true") + ap.add_argument("--json", type=Path) + ap.add_argument("--only", help="comma list of variants to measure") + args = ap.parse_args() + samples = 3 if args.quick else 7 + dl_samples = 2 if args.quick else 5 + names = args.only.split(",") if args.only else ["V-bundle", "V-minimal", "V-host", "B1", "B2", "B2z", "B3"] + workdir = Path(tempfile.mkdtemp(prefix="ytdlp-composite-")) + results = {} + for name in names: + print(f"measuring {name} ...", flush=True) + results[name] = measure(name, samples, dl_samples, workdir) + print(f"load average at end: {loadavg():.2f}") + table = {} + for cand in ("V-bundle", "V-minimal", "V-host"): + if cand in results and "B1" in results: + table[f"{cand}/B1"] = composite(results[cand], results["B1"]) + if cand in results and "B2" in results: + table[f"{cand}/B2"] = composite(results[cand], results["B2"]) + payload = { + "samples": samples, + "loadavg_end": loadavg(), + "results": results, + "composite": {k: {"geo": v[0], "ratios": v[1]} for k, v in table.items()}, + } + text = json.dumps(payload, indent=2, default=str) + if args.json: + args.json.write_text(text) + print(text) + return 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/docs/reference/cli.md b/docs/reference/cli.md index db98137e..6ecb7c20 100644 --- a/docs/reference/cli.md +++ b/docs/reference/cli.md @@ -26,6 +26,8 @@ pyscriptc compile SOURCE [options] | `--island` | | alias of --fallback=cpython: route every import outside the closed table and every function pyscriptc cannot compile to embedded CPython (fallback mode) | | `--embed-python MODE` | `host\|bundle\|auto` | libpython embedded CPython links: host (the compiler interpreter), bundle (the pinned runtime pack, relocatable output) or auto (host when this interpreter has a shared libpython, bundle otherwise; the default). Also read from the `embed` key of [tool.pyscriptc.island] | | `--embed-stdlib TRIM` | `safe\|minimal` | stdlib trim of a bundle: safe (the default; everything except test, idlelib, tkinter, turtledemo, ensurepip, pydoc_data, __phello__) or minimal (an opt-in frozen closure plus allowlist). Also read from the `stdlib` key of [tool.pyscriptc.island] | +| `--bundle-exclude ROOT` | `ROOT` | import root a bundle does not carry: the host must provide it (an optional or native-backed dependency); repeatable | +| `--embed-headers` | | stage the runtime pack's CPython headers into a bundle: a bundle that is itself a compiler (the self-compiled pyscriptc) needs Python.h to compile island programs at run time | | `--call FUNC` | `FUNC` | function the generated main calls | | `--args ARGS` | `ARGS` | Python literal list of arguments for --call, parsed with ast.literal_eval (default `[]`) | | `--run` | | compile a binary and run it; arguments after `--` go to the binary | diff --git a/docs/reference/runtime-pack.md b/docs/reference/runtime-pack.md index 1cc15bf5..fd629227 100644 --- a/docs/reference/runtime-pack.md +++ b/docs/reference/runtime-pack.md @@ -47,3 +47,30 @@ The compiler itself is Apache-2.0; the preflight checker (pyrefly, MIT) ships in ## Asking before shipping The third-party inventory of the first pack build is confirmed with the project owner before the first public release that ships a pack (the inventory check, not legal advice). + +## The self-compiled pyscriptc + +pyscriptc can compile itself into a binary that needs no Python, uv or pip on the host. The build is one command from a checkout: + +```sh +pyscriptc compile src/pyscriptc --program --skip-check --island --emit bin \ + --embed-python bundle --embed-headers --bundle-exclude mypy -o dist/selfbin +``` + +The output is a relocatable directory, `dist/selfbin/`, laid out as: + +- `bin/selfbin`, the compiler itself (about 3 MB; every module of `src/pyscriptc` is compiled into it, and the program's non-Python files live in `bin/pyscriptc-data/`: the C runtime sources `ps_*.c`, the vendored typeshed, the pyright stubs and the runtime lock), +- `lib/`, the pinned runtime: `libpython3.14.so`, the stdlib zip and `lib/python3.14/` (dynload modules plus the bundled pure-Python dependencies), +- `include/`, the CPython headers, staged by `--embed-headers`, so the binary can compile island programs (which include `Python.h`) at run time, +- `share/licenses/`, the license texts of the bundled runtime. + +This is a two-file layout in spirit: the directory is the artifact, and it runs from any path. It is not a single file because the interpreter is a shared library and the stdlib is a zip; both must sit on disk next to the binary for the dynamic loader and the import system. + +What the binary needs at run time, and what it does not: + +- No Python, uv or pip. The interpreter and the stdlib travel in `lib/`. `env -i PATH=/nonexistent dist/selfbin/bin/selfbin` prints its usage, and compiling a program that does not touch the island needs nothing else. +- A C toolchain on PATH (clang or zig) for `--emit bin` and `run`, exactly like the installed compiler. A compile with no toolchain fails with PS0001, which names the install options. +- pyrefly for `check` and the preflight of a compile without `--skip-check`. The binary resolves it like the installed compiler: the `PYREFLY_BIN` override, then a `pyrefly` on PATH. When none is found the compile says so instead of half-running. +- No network and no pack cache for `--embed-python bundle` compiles: the binary links user programs against the runtime it carries. + +`--bundle-exclude mypy` declares the optional mypy bridge an extern dependency: the import stays in the compiled compiler, and the host answers it exactly like CPython answers a dead import. Without the exclusion the bundle refuses the compile, because mypy ships a compiled extension and a bundle carries pure Python only (PS4302). diff --git a/examples/pylock.hello.toml b/examples/pylock.hello.toml new file mode 100644 index 00000000..3ebba1c5 --- /dev/null +++ b/examples/pylock.hello.toml @@ -0,0 +1,6 @@ +lock-version = "1.0" +created-by = "pyscriptc" + +[tool.pyscriptc] +input-digest = "697b0422cb68179a15022b72f18708cbcc5c5ffe0d9977cd33a56695d323c07e" +python = "3.14" diff --git a/mise.toml b/mise.toml index e5293ae4..f3073bf5 100644 --- a/mise.toml +++ b/mise.toml @@ -34,6 +34,7 @@ # compile pyscriptc compile, args pass through (mise run compile FILE --emit bin) # run pyscriptc run, args pass through (mise run run FILE -- args) # wheel:smoke build the wheel, install it in a fresh venv, compile+run hello +# selfbin build the self-compiled pyscriptc into dist/selfbin and run its tests # release:rehearse dispatch release.yml on the master tip (the rehearsal run) # # CI does not run through mise: the workflows pin uv (the emSign CA incident @@ -220,6 +221,10 @@ run = "bash scripts/wt_prune.sh" description = "Build the wheel, install it in a fresh venv, compile and run hello" run = "bash scripts/wheel_smoke.sh" +[tasks.selfbin] +description = "Build the self-compiled pyscriptc (a bundle that carries its own runtime) and run its tests" +run = "uv run pyscriptc compile src/pyscriptc --program --skip-check --island --emit bin --embed-python bundle --embed-headers --bundle-exclude mypy -o dist/selfbin && uv run pytest tests/test_selfbin.py -m slow" + # The rehearsal is a workflow_dispatch of release.yml on the full SHA. Only a # run titled "rehearsal " counts for the cut-release and publish gates. [tasks."release:rehearse"] diff --git a/pyproject.toml b/pyproject.toml index 564e95f9..b2626e8a 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -77,6 +77,7 @@ testpaths = ["tests"] pythonpath = ["src", "."] markers = [ "linux_container: Linux dev container suite (HEAVY=1 docker/run.sh). Shells out to clang or only passes on Linux. CI does not run these.", + "slow: builds the self-compiled pyscriptc and compiles through it; minutes, not seconds. CI does not run these.", ] [tool.ruff] diff --git a/src/pyscriptc/boxed.py b/src/pyscriptc/boxed.py index 7c6f7235..8e1ad70d 100644 --- a/src/pyscriptc/boxed.py +++ b/src/pyscriptc/boxed.py @@ -40,6 +40,7 @@ # site reuses `ps_island_fn_invoke`; a class-level attribute read or a class # taken as a value needs the object first, which this trampoline returns. DEF_HANDLE_SYMBOL = "ps_island_def_handle" +MOD_DEF_HANDLE_SYMBOL = "ps_island_mod_def_handle" def _annotation_is_mutable(node: ast.expr | None) -> bool: diff --git a/src/pyscriptc/check.py b/src/pyscriptc/check.py index 14de11a6..a6a73b72 100644 --- a/src/pyscriptc/check.py +++ b/src/pyscriptc/check.py @@ -459,6 +459,15 @@ def _run_pyrefly( raise CheckFailed("\n".join(_format_pyrefly_error(err) for err in kept)) # Diagnostics go to stdout; stderr carries WARN/INFO progress lines. output: str = proc.stdout.strip() or proc.stderr.strip() or f"pyrefly exited {proc.returncode}" + if "usage: pyscriptc" in output: + # A self-compiled pyscriptc has no Python interpreter to run + # `python -m pyrefly`: the fallback launched the compiler itself, and + # the compiler printed its own usage. Name the remedy instead. + raise CheckFailed( + "pyrefly is not installed for this compiler; the checker is an external tool. " + "Install it (pip install pyrefly) or point PYREFLY_BIN at a pyrefly binary, " + "or compile with --skip-check" + ) raise CheckFailed(output) diff --git a/src/pyscriptc/cli/_main.py b/src/pyscriptc/cli/_main.py index 54985b29..623b1113 100644 --- a/src/pyscriptc/cli/_main.py +++ b/src/pyscriptc/cli/_main.py @@ -288,6 +288,21 @@ def build_parser() -> argparse.ArgumentParser: "tkinter, turtledemo, ensurepip, pydoc_data, __phello__) or minimal (an opt-in frozen " "closure plus allowlist). Also read from the `stdlib` key of [tool.pyscriptc.island]", ) + compile_parser.add_argument( + "--bundle-exclude", + action="append", + default=[], + metavar="ROOT", + help="import root a bundle does not carry: the host must provide it " + "(an optional or native-backed dependency); repeatable", + ) + compile_parser.add_argument( + "--embed-headers", + action="store_true", + help="stage the runtime pack's CPython headers into a bundle: a bundle that is " + "itself a compiler (the self-compiled pyscriptc) needs Python.h to compile " + "island programs at run time", + ) # boxed:end compile_parser.add_argument("--call", metavar="FUNC", help="function the generated main calls") compile_parser.add_argument( @@ -1480,7 +1495,7 @@ def _require_toolchain(args: argparse.Namespace) -> None: if driver is None: raise IRError( "no C toolchain found on PATH (neither zig nor clang), and compiling to a binary needs a C toolchain. " - f"Install one of: {zigshim.install_hint()}, or clang (xcode-select --install, apt install clang, ...). " + f"{zigshim.install_hint()}, or clang (xcode-select --install, apt install clang, ...). " "Run `pyscriptc doctor` to see what was found and the README for install commands. " "`--emit ir` and `--emit llvm` need no toolchain." ) @@ -2007,16 +2022,18 @@ def _compile_cmd(args: argparse.Namespace, state: dict[str, Any]) -> int: # The direct third-party import roots of the program, for the bundle's # site-packages tree (issue #660). A native-backed distribution among # them refuses the compile (PS4302) before anything is staged. - dep_roots: list[str] = sitedeps.third_party_roots(source_path) if resolved_embed == "bundle" else [] + dep_roots: list[str] = ( + sitedeps.third_party_roots(source_path, getattr(args, "bundle_exclude", []) or []) + if resolved_embed == "bundle" + else [] + ) if resolved_embed == "bundle": sitedeps.check_pure(dep_roots) pack = embed.lock_pack() pack_directory = embed.ensure_pack(pack) embed_version = embed.python_version() trim: str = getattr(args, "embed_stdlib", None) or stdlibzip.DEFAULT_MODE - carried = stdlibzip.kept_module_names( - stdlibzip.stdlib_sources(pack_directory / "lib" / f"python{embed_version}", trim) - ) + carried = stdlibzip.carried_modules(pack_directory, embed_version, trim) stdlibzip.manifest_check( carried, _stdlib_imports(source_path), diff --git a/src/pyscriptc/embed.py b/src/pyscriptc/embed.py index 6ab828a8..b9166bed 100644 --- a/src/pyscriptc/embed.py +++ b/src/pyscriptc/embed.py @@ -243,13 +243,46 @@ def runtime_dir() -> Path | None: return Path(value) +def self_pack_dir() -> Path | None: + """The runtime pack the running binary carries, when it carries one. + + A self-compiled pyscriptc (a `--embed-python=bundle` output) has the + pinned runtime next to itself: `lib/libpythonX.Y.so`, `include/pythonX.Y` + and `lib/pythonX.Y/` beside the binary. That tree is a pack, so a bundle + compile of a user program can link against the runtime the binary + already carries, with no cache and no network. The answer is None for + an ordinary install (the pack lives in the cache). + + The staged program-data directory (`pyscriptc-data/` beside the binary) + is the marker: an ordinary interpreter shares the same lib/include shape + (a uv-managed python-build-standalone install has `lib/libpython*` and + `include/python*` too) but no program-data directory, and its libpython + is not the stripped pack build. + """ + try: + executable = Path(sys.executable).resolve() + except OSError: + return None + root = executable.parent + if not (root / "pyscriptc-data").is_dir(): + return None + root = root.parent + if not list((root / "lib").glob("libpython*")): + return None + if not list((root / "include").glob("python*")): + return None + return root + + def pack_dir(pack: Pack, root: Path | None = None) -> Path: """The unpacked pack directory for a pack. With `PYSC_RUNTIME_DIR` set, that directory is the pack itself (offline and air-gapped use): the variable names the unpacked runtime, whatever its own directory name is. Otherwise the pack lives in the cache under - its pack id. + its pack id. A binary that carries its own runtime (the self-compiled + pyscriptc) uses that tree first: it is the same pinned runtime, already + on disk. """ if root is None: override = runtime_dir() @@ -258,7 +291,9 @@ def pack_dir(pack: Pack, root: Path | None = None) -> Path: if len(hit) > 0: return override return override / pack.pack_id - if root is None: + carried = self_pack_dir() + if carried is not None: + return carried return pack_root() / pack.pack_id return root / pack.pack_id @@ -417,7 +452,14 @@ def host_has_shared_libpython() -> bool: libdir = Path(base) / "lib" mac = platform.system() == "Darwin" name = f"libpython{version}.dylib" if mac else f"libpython{version}.so" - return (libdir / name).exists() + if (libdir / name).exists(): + return True + # A relocatable build (a python-build-standalone install, the pinned + # runtime pack) reports the build machine's LIBDIR (`/install/lib`), + # which does not exist next to the running interpreter. The library + # travels with the interpreter, so the interpreter's own directory is + # the real answer. + return (Path(sys.executable).resolve().parent.parent / "lib" / name).exists() def resolve_mode(mode: str | None) -> str: @@ -481,6 +523,7 @@ def stage_bundle( *, stdlib_mode: str = "safe", dependency_roots: list[str] | None = None, + with_headers: bool = False, ) -> Path: """Copy the pack runtime next to the binary: `name/bin/name` plus `name/lib/`. @@ -533,6 +576,14 @@ def stage_bundle( stdlibzip.stage_stdlib(lib_dir, pack_directory, version, stdlib_mode) else: (lib_dir / f"python{version}" / "lib-dynload").mkdir(parents=True, exist_ok=True) + if with_headers: + # A bundle that is itself a compiler (the self-compiled pyscriptc) + # compiles island programs at run time, and that compile needs the + # CPython headers: sysconfig of the bundled interpreter resolves + # `include` inside this directory tree. + include_src = pack_directory / "include" + if include_src.is_dir(): + shutil.copytree(include_src, staging / "include") if dependency_roots: sitedeps.stage_site_packages(lib_dir, version, dependency_roots) # Issue #661: the staged program data tree (issue #665) sits beside the diff --git a/src/pyscriptc/emit/llvm/_abi.py b/src/pyscriptc/emit/llvm/_abi.py index 7ce15891..dc5465df 100644 --- a/src/pyscriptc/emit/llvm/_abi.py +++ b/src/pyscriptc/emit/llvm/_abi.py @@ -471,6 +471,29 @@ def _emit_island_def_handle(self, expr: dict[str, Any]) -> str: self._abi_decl("ps_island_def_handle", "ptr", ["i64", "ptr"]) return self._call_ret("ptr", "ps_island_def_handle", [f"i64 {size}", f"ptr {symbol}"]) + # boxed:begin + def _emit_island_mod_def_handle(self, expr: dict[str, Any]) -> str: + """The same handle, fetched from another module's island namespace. + + The per-module plane split: the defining module's name fronts the + key (`ps_island_mod_def_handle(mod, name)`), the namespaced shape of + the single-namespace seam. The seed order guarantees the defining + module's namespace exists by the time the consumer's init reads it. + """ + mod, name = expr["island_mod_def"]["module"], expr["island_mod_def"]["name"] + mod_data = mod.encode("utf-8") + name_data = name.encode("utf-8") + mod_sym = self.format_global(f"island.mod.{mod}", mod_data) + name_sym = self.format_global(f"island.cls.{name}", name_data) + self._abi_decl("ps_island_mod_def_handle", "ptr", ["i64", "ptr", "i64", "ptr"]) + return self._call_ret( + "ptr", + "ps_island_mod_def_handle", + [f"i64 {len(mod_data)}", f"ptr {mod_sym}", f"i64 {len(name_data)}", f"ptr {name_sym}"], + ) + + # boxed:end + def _emit_island_seed_global(self) -> None: """The main-prologue seed, when the module plans island definitions.""" source: str | None = _opt_str_or_none(self.module, "islandSource") diff --git a/src/pyscriptc/emit/llvm/_calls.py b/src/pyscriptc/emit/llvm/_calls.py index ef53c403..fc342248 100644 --- a/src/pyscriptc/emit/llvm/_calls.py +++ b/src/pyscriptc/emit/llvm/_calls.py @@ -4,7 +4,7 @@ from typing import Any -from pyscriptc.boxed import DEF_HANDLE_SYMBOL, INVOKE_SYMBOL +from pyscriptc.boxed import DEF_HANDLE_SYMBOL, INVOKE_SYMBOL, MOD_DEF_HANDLE_SYMBOL from pyscriptc.emit._ir import ( _arg_type, _callee_kind, @@ -349,6 +349,8 @@ def emit_module_call(self, expr: dict[str, Any], values: list[str]) -> str: return self._emit_island_fn_call(expr, values) if symbol == DEF_HANDLE_SYMBOL: return self._emit_island_def_handle(expr) + if symbol == MOD_DEF_HANDLE_SYMBOL: + return self._emit_island_mod_def_handle(expr) if symbol == "ps_island_getattr": return self._emit_island_getattr(expr, values) if symbol == "ps_island_getitem": diff --git a/src/pyscriptc/emit/llvm/_driver.py b/src/pyscriptc/emit/llvm/_driver.py index 9bd308a1..023a3e3c 100644 --- a/src/pyscriptc/emit/llvm/_driver.py +++ b/src/pyscriptc/emit/llvm/_driver.py @@ -370,8 +370,23 @@ def link_binary( target_flags.extend(["-I", include]) tail = [*embed_flags, *tail] else: + if include is None or not Path(include).is_dir(): + raise IRError( + f"the island link needs the CPython headers ({include or 'include'}); " + "this interpreter has no usable header directory. On a self-compiled " + "pyscriptc bundle, rebuild it with --embed-python=bundle --embed-headers, " + "or compile this program with --embed-python=bundle so the pinned " + "runtime pack supplies the headers" + ) # boxed:end target_flags.extend(["-I", include]) + if not Path(libdir).is_dir(): + # A relocatable build (a python-build-standalone install) + # reports the build machine's LIBDIR (`/install/lib`): the + # library travels with the interpreter, so the interpreter's + # own directory is the real link directory (the same rule + # host_has_shared_libpython applies). + libdir = str(Path(sys.executable).resolve().parent.parent / "lib") tail = [f"-L{libdir}", f"-lpython{version}", f"-Wl,-rpath,{libdir}", *tail] else: sources = [source for source in sources if not source.endswith("ps_island.c")] diff --git a/src/pyscriptc/lower/__init__.py b/src/pyscriptc/lower/__init__.py index 7fcedaed..9fd5990c 100644 --- a/src/pyscriptc/lower/__init__.py +++ b/src/pyscriptc/lower/__init__.py @@ -290,6 +290,14 @@ def __init__(self, filename: str, source: str, target: str | None = None, config # Class name -> the hidden local that owns its island handle, reset # per function scope like `island_modules`. self.island_class_handles: dict[str, str] = {} + # boxed:begin + # The per-module plane split: another module's island classes this + # module imported (`from other import Cls` where `Cls` rides that + # module's island floor). Name -> (defining module, class name); the + # handle, the call and the base-class refusal all go through the + # namespaced seam (`ps_island_mod_def_handle`). + self.island_floor_classes: dict[str, tuple[str, str]] = {} + # boxed:end # The per-definition probe state: the definition list to probe and # the names the probe fenced. self._boxed_functions: list[ast.FunctionDef] = [] @@ -719,6 +727,39 @@ def _bind_one_export(self, module: str, name: str, local: str, stmt: ast.AST, fe self._install_return_class(module, item["return"], stmt, fences) return if _record_kind(item) == "class": + # boxed:begin + # The defining module planned the class for its own island floor + # (issue #628): no native body exists, so the import cannot + # install a lowered class record. Binding the name as a + # cross-module island class keeps the import from fencing the + # module level: the uses that need the class natively (a base + # class, an `except` clause) fence their own definition instead + # of the whole module, and the uses that only call or pass it + # cross the boundary through the boxed seam. + if item.get("island"): + if self.program and self._boxed_cross_module_island(): + # The class object lives in the *defining* module's island + # namespace, which is not always the module the import + # names: a package re-exports it (`from ._utils import + # *`), so the entry carries the owner. + owner = str(item.get("module") or module) + self.bindings[local] = Binding("island_floor_class", owner, name, False) + self.island_floor_classes[local] = (owner, name) + self.declared.add(local) + return + fence = Fence( + "PS2002", + f"{module}.{name} runs on the island floor of its own module, and importing it " + "crosses a package boundary into it. Compile the tree as a program " + "(--program --fallback=cpython) so the whole program runs island-side, " + "or make the class lower natively.", + span_of(stmt, self.filename), + None, + ) + fences.append(fence) + self._boxed_module_fences.append(fence) + return + # boxed:end self._install_class_export(local, item, stmt, fences) return if _record_kind(item) == "module": diff --git a/src/pyscriptc/lower/_annotations.py b/src/pyscriptc/lower/_annotations.py index 74a1bc77..ed9397fd 100644 --- a/src/pyscriptc/lower/_annotations.py +++ b/src/pyscriptc/lower/_annotations.py @@ -207,9 +207,14 @@ def _annotation(self, node: ast.AST) -> dict[str, Any]: # boxed:begin # An island class annotation is its handle type (issue #628): the # class lives in the embedded interpreter, and an instance crossed - # into a compiled local is an opaque handle. + # into a compiled local is an opaque handle. Another module's island + # class annotates the same way, with the defining module in the type + # name so two modules' same-named classes stay distinct. if isinstance(node, ast.Name) and node.id in self.island_classes: return _type(island_class_type(node.id)) + floor = self.island_floor_classes.get(node.id) if isinstance(node, ast.Name) else None + if floor is not None: + return _type(island_class_type(f"{floor[0]}.{floor[1]}")) # boxed:end if isinstance(node, ast.Name) and (node.id in self.declared or node.id in self.known_classes): return _type({"kind": "class", "name": self.class_aliases.get(node.id, node.id)}) diff --git a/src/pyscriptc/lower/_boxed_resolve.py b/src/pyscriptc/lower/_boxed_resolve.py index 8bedde6e..f5e591b6 100644 --- a/src/pyscriptc/lower/_boxed_resolve.py +++ b/src/pyscriptc/lower/_boxed_resolve.py @@ -515,11 +515,21 @@ def _boxed_island_pad_code(self, typ: dict[str, Any]) -> int: return AbiMixin._ISLAND_ARG_CODES[_type_kind(typ)] def _boxed_stage2_source(self, tree: ast.Module, island: set[str]) -> str: - """The Stage 2 seed: imports, literal module values, island defs.""" + """The Stage 2 seed: imports, literal module values, island defs. + + An import of another program module rides the seed only when that + module rides the island too: the split seeds one namespace per + fenced module, and a sibling that lowered native has no namespace + for the import to resolve against. The import is not load-bearing + anyway: an island definition that used the native sibling's + definition already refused (the island-to-compiled seam, one + module further out), so the seed loses nothing it could use. + """ segments: list[str] = [] for stmt in tree.body: if isinstance(stmt, (ast.Import, ast.ImportFrom)): - segments.append(ast.get_source_segment(self.source, stmt) or "") + if self._boxed_seed_import_is_island(stmt): + segments.append(ast.get_source_segment(self.source, stmt) or "") continue if isinstance(stmt, (ast.Assign, ast.AnnAssign)) and self._boxed_is_literal_module_value(stmt): segment = ast.get_source_segment(self.source, stmt) @@ -538,6 +548,45 @@ def _boxed_stage2_source(self, tree: ast.Module, island: set[str]) -> str: segments.append(segment) return "\n\n\n".join(segment for segment in segments if segment) + def _boxed_seed_import_is_island(self, stmt: ast.Import | ast.ImportFrom) -> bool: + """True when a seed import resolves against an island namespace. + + A program sibling's island plans live in the export table under its + name: the sibling that carries an island plan carries a seed, and + its namespace exists when this seed runs (the merge emits the + seeds in finish order). A stdlib module outside the closed table + resolves through the island's own import machinery. An import of a + sibling with no island plan has nothing to resolve against and + drops from the seed: the native lane ran the sibling's body + already, and an island definition of this module that used it + fenced instead of lowering. + """ + if self.program is None: + return True + targets: list[str] = [] + if isinstance(stmt, ast.Import): + targets.extend(alias.name for alias in stmt.names) + elif stmt.level: + # A relative import inside a program is not a plain sibling + # shape (the split refuses package-shaped programs), so it + # keeps the old seed behavior. + return True + else: + targets.append(stmt.module or "") + exports = self.package or {} + for target in targets: + root = target.split(".", 1)[0] + if root in exports or target in exports: + module = target if target in exports else root + bucket = exports.get(module) or {} + if any(item.get("island") for item in bucket.values()): + continue + return False + # Not a program module (stdlib): the island import machinery + # resolves it, as before. + continue + return True + def _boxed_definition_source(self, node: ast.AST) -> str: """One definition's source text, decorators included. diff --git a/src/pyscriptc/lower/_calls.py b/src/pyscriptc/lower/_calls.py index a3285f06..8a95ae07 100644 --- a/src/pyscriptc/lower/_calls.py +++ b/src/pyscriptc/lower/_calls.py @@ -233,6 +233,11 @@ def _call_name(self, node: ast.Call, span: dict[str, Any], expect: dict[str, Any # through the boxed seam, namespaced by the defining module. if binding.kind == "island_floor": return self._island_floor_call(node, span, binding.module, binding.attr, expect) + # An island class of another module instantiates through the same + # namespaced trampoline: the class object is callable island-side, + # so `Cls(...)` is one invoke keyed by (module, class name). + if binding.kind == "island_floor_class": + return self._island_floor_class_call(node, span, binding.module, binding.attr, expect) # boxed:end if binding.kind == "const" and self._ffi_module(binding.module) is not None: self.fail(node, "PS2002", f"{name} is a constant and is not a function") @@ -244,6 +249,12 @@ def _call_name(self, node: ast.Call, span: dict[str, Any], expect: dict[str, Any # handle. if name in self.island_classes: return self._island_fn_call(node, span, name, expect) + # An island class of another module: the same trampoline with the + # defining module's name in front of the callee key. + floor = self.island_floor_classes.get(name) + if floor is not None: + module, cls = floor + return self._island_floor_class_call(node, span, module, cls, expect) # boxed:end if name in self.class_aliases or name in self.classes: return self._construct(node, span, self.class_aliases.get(name, name)) diff --git a/src/pyscriptc/lower/_classes_more.py b/src/pyscriptc/lower/_classes_more.py index 675d6be9..3cfafba1 100644 --- a/src/pyscriptc/lower/_classes_more.py +++ b/src/pyscriptc/lower/_classes_more.py @@ -253,6 +253,21 @@ def _resolve_base(self, node: ast.expr) -> dict[str, str]: return _enum_base(enum) if isinstance(node, ast.Name) and node.id in EXCEPTIONS: return _exception_base(node.id) + # boxed:begin + # An island class of another module has no native class record, so a + # native subclass cannot extend it: the class definition fences and + # rides the island instead (issue #628, one module down). The fence + # names the defining module so the refusal reads the way the + # single-module refusal does. + if isinstance(node, ast.Name) and node.id in self.island_floor_classes: + module, name = self.island_floor_classes[node.id] + self.fail( + node, + "PS2002", + f"class {node.id} runs on the island floor of {module} and cannot be a native base class: " + "the subclass joins the island", + ) + # boxed:end if isinstance(node, ast.Name) and node.id in self.classes: info = self.classes[node.id] if info.get("synthetic"): @@ -436,7 +451,20 @@ def _exception_code_of(self, name: str) -> int | None: code = info.get("exc_code") if info is not None else None return code if isinstance(code, int) else None - def _exception_codes(self, name: str) -> list[int]: + def _exception_codes(self, name: str, node: ast.AST | None = None) -> list[int]: + # boxed:begin + # Another module's island exception class has no native exc code: + # the native lane cannot match it, so the handler that names it is + # not lowered and the definition joins the island, where CPython's + # own matching applies. + if name in self.island_floor_classes: + module, cls = self.island_floor_classes[name] + self.fail( + node if node is not None else ast.Pass(), + "PS2002", + f"class {name} runs on the island floor of {module}: except {name} does not lower natively", + ) + # boxed:end found: list[int] = [] if name in EXCEPTIONS: found.append(EXCEPTIONS[name]) diff --git a/src/pyscriptc/lower/_expr.py b/src/pyscriptc/lower/_expr.py index 486c792a..65fef768 100644 --- a/src/pyscriptc/lower/_expr.py +++ b/src/pyscriptc/lower/_expr.py @@ -387,6 +387,12 @@ def _name(self, node: ast.Name, span: dict[str, Any]) -> dict[str, Any]: # `ps_island_fn_invoke`, like any other island definition. if node.id in self.island_classes: return self._island_class_handle(node.id, span) + # An island class of another module taken as a value: the same + # opaque handle, fetched from the defining module's namespace. + floor = self.island_floor_classes.get(node.id) + if floor is not None: + module, cls = floor + return self._island_floor_class_handle(node.id, module, cls, span) # boxed:end if node.id in self.classes: self.fail(node, "PS2002", f"class {node.id} is not a value") diff --git a/src/pyscriptc/lower/_stdlib.py b/src/pyscriptc/lower/_stdlib.py index 5cf6c255..a675f71f 100644 --- a/src/pyscriptc/lower/_stdlib.py +++ b/src/pyscriptc/lower/_stdlib.py @@ -24,6 +24,7 @@ from pyscriptc.boxed import ( # boxed:begin DEF_HANDLE_SYMBOL, INVOKE_SYMBOL, + MOD_DEF_HANDLE_SYMBOL, island_class_type, island_handle_type, ) # boxed:end @@ -682,6 +683,59 @@ def _island_floor_call( call_expr["island_fn"]["want"] = expect return call_expr + def _island_floor_class_handle( + self, local_name: str, module: str, name: str, span: dict[str, Any] + ) -> dict[str, Any]: + """`Cls` of another module as a value: a handle from its namespace. + + The class object lives in the defining module's island namespace; + `ps_island_mod_def_handle(mod, name)` returns a new reference to it, + the namespaced shape of `ps_island_def_handle` (issue #628). The + hidden local owns the reference for the scope like the in-module + handle does. + """ + local = self.island_class_handles.get(local_name) + if local is not None and local not in self.locals: + del self.island_class_handles[local_name] + local = None + if local is None: + local = self._temp("icls").replace(".", "_") + typ = island_class_type(f"{module}.{name}") + self.locals[local] = typ + call = self._abi_call(module, name, [], typ, MOD_DEF_HANDLE_SYMBOL, span) + call["island_mod_def"] = {"module": module, "name": name} + self.prelude.append(_assign(local, call, span)) + self._own_handle(local, typ) + self.island_class_handles[local_name] = local + return _name(local, self.locals[local], span) + + def _island_floor_class_call( + self, node: ast.Call, span: dict[str, Any], module: str, name: str, expect: dict[str, Any] | None = None + ) -> dict[str, Any]: + """`Cls(...)` where `Cls` rides another module's island floor. + + The class object lives in the defining module's island namespace, + where it is callable, so the instantiation is the same namespaced + trampoline as `_island_floor_call` with the class name as the callee + key. The instance is an opaque handle of the class's own type (the + class's spelling carries the defining module, issue #628). + """ + marshalled, npos, nkw = self._island_args(node, span) + call_expr: dict[str, Any] = _island_call_node( + "island", + name, + marshalled, + island_class_type(f"{module}.{name}"), + INVOKE_SYMBOL, + span, + {"island_fn": {"name": name, "module": module}, "island_npos": npos, "island_nkw": nkw}, + ) + if expect is not None and not expect.get("island"): + self._island_validate_expect(node, expect) + call_expr["type"] = expect + call_expr["island_fn"]["want"] = expect + return call_expr + def _island_fn_call( self, node: ast.Call, span: dict[str, Any], name: str, expect: dict[str, Any] | None = None ) -> dict[str, Any]: @@ -707,6 +761,12 @@ def _island_fn_call( # An island class instantiation returns the class's one handle type, # so the destination annotation proves the same type (issue #628). result_type = island_class_type(name) if name in self.island_classes else _island_handle(f"{name}()") + island_fn: dict[str, Any] = {"name": name} + # The per-module plane split: the single island namespace does not + # exist (every module's floor is its own namespace), so the call + # names the owning module and rides the namespaced trampoline. + if self._boxed_plane_split and self.module_name: + island_fn["module"] = self.module_name call_expr: dict[str, Any] = _island_call_node( "island", name, @@ -714,7 +774,7 @@ def _island_fn_call( result_type, INVOKE_SYMBOL, span, - {"island_fn": {"name": name}, "island_npos": npos, "island_nkw": nkw}, + {"island_fn": island_fn, "island_npos": npos, "island_nkw": nkw}, ) if expect is not None and not expect.get("island"): self._island_validate_expect(node, expect) @@ -740,9 +800,16 @@ def _island_class_handle(self, name: str, span: dict[str, Any]) -> dict[str, Any local = self._temp("icls").replace(".", "_") typ = island_class_type(name) self.locals[local] = typ - call = self._abi_call("island", name, [], typ, DEF_HANDLE_SYMBOL, span) - call["island"] = {"name": name} - call["island_def"] = {"name": name} + if self._boxed_plane_split and self.module_name: + # The per-module plane split: the class object lives in this + # module's own island namespace, so the handle is fetched + # through the namespaced trampoline. + call = self._abi_call(self.module_name, name, [], typ, MOD_DEF_HANDLE_SYMBOL, span) + call["island_mod_def"] = {"module": self.module_name, "name": name} + else: + call = self._abi_call("island", name, [], typ, DEF_HANDLE_SYMBOL, span) + call["island"] = {"name": name} + call["island_def"] = {"name": name} self.prelude.append(_assign(local, call, span)) self._own_handle(local, typ) self.island_class_handles[name] = local diff --git a/src/pyscriptc/lower/_stmt_b.py b/src/pyscriptc/lower/_stmt_b.py index e1d24b91..b4b42f44 100644 --- a/src/pyscriptc/lower/_stmt_b.py +++ b/src/pyscriptc/lower/_stmt_b.py @@ -757,7 +757,7 @@ def _try(self, stmt: ast.Try, span: dict[str, Any]) -> dict[str, Any]: names = _handler_names(self, handler.type) if "Exception" not in names: for name in names: - codes.extend(self._exception_codes(name)) + codes.extend(self._exception_codes(name, handler.type)) bind = handler.name if bind is not None: prior = self.locals.get(bind) diff --git a/src/pyscriptc/package.py b/src/pyscriptc/package.py index 7e5e1d03..baa239fd 100644 --- a/src/pyscriptc/package.py +++ b/src/pyscriptc/package.py @@ -407,7 +407,12 @@ def bridge_for(path: Path) -> Any: } # boxed:begin if island_modules: - merged["islandModules"] = island_modules + # The seeds run in the order CPython finishes the bodies: a seed + # that imports a sibling finds that sibling's namespace in + # `sys.modules` (the runtime registers each seeded module under + # its real name), so the defining module must be seeded first. + finish = _program_finish_order(units, exports, entry=entry) + merged["islandModules"] = {name: island_modules[name] for name in finish if name in island_modules} merged["fallback"] = { "moduleTopLevel": _boxed.ISLAND if plane_top_island else _boxed.NATIVE, "functions": plane_functions, @@ -586,6 +591,17 @@ def stage_program_data( target = root_dir / rel target.parent.mkdir(parents=True, exist_ok=True) target.write_bytes(body) + # A program that is itself a compiler (pyscriptc compiled with + # --program) resolves packaging/runtime.lock.toml relative to its + # package at run time; the lock lives outside the program directory in + # the repo, so stage it next to the package data tree explicitly. + from pyscriptc import embed as _embed + + lock = _embed.lock_path() + if lock.is_file(): + target = root_dir.parent / "packaging" / lock.name + target.parent.mkdir(parents=True, exist_ok=True) + target.write_bytes(lock.read_bytes()) return (root_dir.parent.name, root_dir.name) @@ -901,6 +917,40 @@ def visit(name: str) -> None: return fences, owners +def _program_finish_order( + units: list[tuple[str, Path, bool]], + exports: dict[str, dict[str, dict[str, Any]]], + entry: str | None = None, +) -> list[str]: + """The units in the order CPython finishes their bodies. + + The per-module plane split: each fenced unit's island seed execs into + its own namespace, and a seed that imports a sibling needs that + sibling's namespace to exist first. The order is the import-time graph + in reverse (a module's body finishes before its importer's), so the + seeds land in the order the bodies ran. A cycle has no such order: + the plan refuses it before the split engages. + """ + trees = {name: ast.parse(path.read_text(encoding="utf-8"), filename=str(path)) for name, path, _ in units} + packages = {name: is_package for name, _, is_package in units} + edges = {name: _program_imports(trees[name], exports, name, packages[name]) for name in trees} + order: list[str] = [] + state: dict[str, int] = {} + + def visit(name: str) -> None: + state[name] = 1 + for child in edges.get(name, []): + if child in trees and state.get(child) is None: + visit(child) + state[name] = 2 + order.append(name) + + for name in _program_roots(trees, entry): + if state.get(name) is None: + visit(name) + return order + + def _program_roots(trees: dict[str, ast.Module], entry: str | None = None) -> list[str]: """The order the graph walk starts in. @@ -1423,6 +1473,25 @@ def _exports_of(module_name: str, lowerer: Lowerer) -> dict[str, dict[str, Any]] if info.get("synthetic") or info.get("imported"): continue bucket[class_name] = {"kind": "class", "name": class_name, "info": _class_export(info)} + # boxed:begin + # An island class (issue #628) has no native body: the merged module + # emits no class record for it, so a cross-module consumer cannot + # resolve the name and the import fences at module level. That fence + # routes the *importing* module's whole top level to the island + # (Decision A), so one island class in a widely imported module drags + # every consumer with it. The export table carries the class with the + # island flag, the way it carries an island function (Refs #641): the + # consumer binds the class across the boundary instead of refusing. + for class_name in sorted(lowerer.island_classes): + if class_name in bucket: + continue + bucket[class_name] = { + "kind": "class", + "name": class_name, + "island": True, + "module": module_name, + } + # boxed:end # A module-level global that lives in a runtime slot (a list, a dict, a # threaded record) keeps one pointer for the process. A consumer in # another module reads it natively when the export table carries the diff --git a/src/pyscriptc/sitedeps.py b/src/pyscriptc/sitedeps.py index 91c59940..624ab66b 100644 --- a/src/pyscriptc/sitedeps.py +++ b/src/pyscriptc/sitedeps.py @@ -26,6 +26,7 @@ import shutil import sys +from collections.abc import Sequence from pathlib import Path @@ -33,17 +34,23 @@ class SiteError(Exception): """A user-facing site-packages bundle failure (no traceback).""" -def third_party_roots(source: Path) -> list[str]: +def third_party_roots(source: Path, exclude: Sequence[str] = ()) -> list[str]: """The absolute third-party import roots of a program, in source order. Stdlib roots go to the stdlib zip (issue #659), relative imports stay in the program tree, and the program's own package root is not a dependency. + `exclude` names roots the caller declares extern (provided by the host, + not carried by the bundle): they are dropped with a warning, so a + native-backed optional dependency can stay behind while the program + still compiles (the self-compile of pyscriptc excludes its lazy mypy + bridge that way). """ from pyscriptc.imports import collect_imports source = Path(source) own = source.stem if source.is_file() else source.name + dropped = sorted(set(exclude)) found: list[str] = [] for site in collect_imports(source): level: int = site["level"] @@ -53,8 +60,16 @@ def third_party_roots(source: Path) -> list[str]: root: str = name.split(".")[0] if root in sys.stdlib_module_names or root == own: continue + if root in dropped: + continue if root not in found: found.append(root) + for root in dropped: + print( + f"warning: {root} declared extern for the bundle; it is not carried " + "(the host must provide it, like CPython would for a dead import)", + file=sys.stderr, + ) return sorted(found) diff --git a/src/pyscriptc/stdlibzip.py b/src/pyscriptc/stdlibzip.py index bc56234d..8e642d69 100644 --- a/src/pyscriptc/stdlibzip.py +++ b/src/pyscriptc/stdlibzip.py @@ -25,9 +25,11 @@ still fails closed at compile time, so the failure is a build error, not a runtime `ModuleNotFoundError`. -The zip stores `.pyc` bodies with their magic-number header; a stored -(uncompressed) zip keeps the pages cache-friendly and the pack reproducible -byte-for-byte from the same pack directory. +The zip stores `.pyc` bodies with their magic-number header; the archive is +deflated, because the zip is data (not an executable) and the saved bytes +cost one decompression of a few milliseconds at first import. A stored +(uncompressed) zip keeps the pages cache-friendly, and the pack reproducible +byte-for-byte from the same pack directory either way. """ from __future__ import annotations @@ -37,6 +39,7 @@ import importlib.util import os import py_compile +import shutil import sys import tempfile import zipfile @@ -156,6 +159,27 @@ "getopt", "getpass", "gzip", + "email", + "_markupbase", + "importlib.resources", + "importlib.resources._common", + "importlib.resources._adapters", + "importlib.resources._functional", + "importlib.resources._itertools", + "importlib.resources._legacy", + "importlib.metadata", + "importlib.metadata._adapters", + "importlib.metadata._collections", + "importlib.metadata._functools", + "importlib.metadata._itertools", + "importlib.metadata._text", + "importlib.abc", + "importlib.readers", + "encodings.idna", + "fileinput", + "optparse", + "pty", + "secrets", "html", "http", "ipaddress", @@ -171,6 +195,8 @@ "unittest.mock", "uuid", "xml", + "xml.etree", + "xml.etree.ElementTree", "zlib", ) @@ -217,6 +243,34 @@ def _import_names(source: str) -> set[str]: return found +def _import_names_relative(source: str, package: str) -> set[str]: + """The import names of a module source, relatives resolved to absolute. + + A `from . import sibling` inside a package resolves against the package, + and the frozen closure must follow it or a kept submodule loses a peer + (`xml.etree.ElementTree` pulls `ElementPath` that way). + """ + + found = _import_names(source) + try: + tree = ast.parse(source) + except SyntaxError: + return found + for node in ast.walk(tree): + if not isinstance(node, ast.ImportFrom) or node.level == 0: + continue + parts = package.split(".") if package else [] + if node.level > 1: + parts = parts[: len(parts) - (node.level - 1)] + base = ".".join(parts) + if node.module: + found.add(f"{base}.{node.module}" if base else node.module) + for alias in node.names: + if alias.name != "*": + found.add(f"{base}.{alias.name}" if base else alias.name) + return found + + def _stdlib_available() -> Path: """A stdlib tree to read sources from: the running interpreter's.""" candidate = Path(sys.base_prefix) / "lib" / f"python{sys.version_info.major}.{sys.version_info.minor}" @@ -271,7 +325,8 @@ def minimal_modules(stdlib_dir: Path | None = None) -> list[Path]: if not path.is_file(): continue keep.add(name) - for found in _import_names(path.read_text(encoding="utf-8", errors="replace")): + package = name if path.name == "__init__.py" else name.rpartition(".")[0] + for found in _import_names_relative(path.read_text(encoding="utf-8", errors="replace"), package): if found in seen: continue parts = found.split(".") @@ -345,14 +400,14 @@ def _pyc_bytes(source: Path, module_name: str) -> bytes: def write_zip(sources: list[Path], stdlib_dir: Path, zip_path: Path) -> Path: - """Write a stored zip of `.pyc` bodies for the kept modules. + """Write a deflated zip of `.pyc` bodies for the kept modules. Archive names are the module paths relative to the stdlib root, so `python314.zip` answers `import encodings` exactly as the unpacked tree did. `.py` files are kept out on purpose: startup must not compile. """ zip_path.parent.mkdir(parents=True, exist_ok=True) - with zipfile.ZipFile(zip_path, "w", compression=zipfile.ZIP_STORED) as archive: + with zipfile.ZipFile(zip_path, "w", compression=zipfile.ZIP_DEFLATED) as archive: for source in sorted(sources): relative = source.relative_to(stdlib_dir) module_name = str(relative.with_suffix("")).replace(os.sep, ".") @@ -375,6 +430,35 @@ def stdlib_zip_name(version: str) -> str: return f"python{version.replace('.', '')}.zip" +def pack_stdlib_zip(pack_directory: Path, version: str) -> Path | None: + """The stdlib zip a pack carries, when its stdlib tree is already zipped. + + A bundle that is itself a compiler (the self-compiled pyscriptc) carries + the runtime as a zip, not as an unpacked tree: the tree it hands to a + user program is the zip it already has. + """ + + candidate = pack_directory / "lib" / stdlib_zip_name(version) + return candidate if candidate.is_file() else None + + +def carried_modules(pack_directory: Path, version: str, mode: str = DEFAULT_MODE) -> set[str]: + """The stdlib module names a pack carries, from its tree or its zip. + + The manifest check (PS4301) asks what the trim keeps. A pack with an + unpacked tree answers from the trim; a pack that carries the zipped + stdlib answers from the zip. + """ + + tree = pack_directory / "lib" / f"python{version}" + if tree.is_dir() and any(tree.glob("*.py")): + return kept_module_names(stdlib_sources(tree, mode)) + zip_path = pack_stdlib_zip(pack_directory, version) + if zip_path is not None: + return zip_modules(zip_path) + return set() + + def stage_stdlib(lib_dir: Path, pack_directory: Path, version: str, mode: str = DEFAULT_MODE) -> Path: """Replace the unpacked stdlib tree of a staged bundle with its zip. @@ -388,10 +472,20 @@ def stage_stdlib(lib_dir: Path, pack_directory: Path, version: str, mode: str = pack_stdlib = pack_directory / "lib" / f"python{version}" if not stdlib_dir.is_dir() and not pack_stdlib.is_dir(): raise StdlibError(f"the runtime pack has no stdlib tree at {pack_stdlib}") - sources = stdlib_sources(pack_stdlib if pack_stdlib.is_dir() else stdlib_dir, mode) zip_path = lib_dir / stdlib_zip_name(version) + if pack_stdlib.is_dir() and not any(pack_stdlib.glob("*.py")): + # A pack that carries the zipped stdlib (the self-compiled compiler's + # own runtime) has no sources to re-trim: the zip travels as is. + carried = pack_stdlib_zip(pack_directory, version) + if carried is not None: + shutil.copy2(carried, zip_path) + if stdlib_dir.is_dir(): + shutil.rmtree(stdlib_dir) + dynload = stdlib_dir / "lib-dynload" + dynload.mkdir(parents=True, exist_ok=True) + return zip_path + sources = stdlib_sources(pack_stdlib if pack_stdlib.is_dir() else stdlib_dir, mode) write_zip(sources, pack_stdlib if pack_stdlib.is_dir() else stdlib_dir, zip_path) - import shutil if stdlib_dir.is_dir(): shutil.rmtree(stdlib_dir) @@ -474,9 +568,18 @@ def interpreter_modules(pack_directory: Path, version: str) -> set[str]: A stdlib name no build of this platform provides (`msvcrt`, `winreg` on POSIX) has no tree entry either; `_stdlib_imports` drops such a name before the check, and it stays out of the demand here. + + A pack that carries its stdlib already zipped (the self-compiled + compiler's own runtime) has no tree to read; the zip names the modules + it carries, and what the runtime serves beyond the zip is the same set + the runtime binary holds. """ stdlib_dir = pack_directory / "lib" / f"python{version}" with_source = _tree_module_names(stdlib_dir) + if not with_source: + zipped = pack_stdlib_zip(pack_directory, version) + if zipped is not None: + with_source = zip_modules(zipped) extensions = _extension_stems(stdlib_dir / "lib-dynload") return (set(sys.stdlib_module_names) - with_source) - extensions diff --git a/tests/census_pins/pyscriptc.cli._main.json b/tests/census_pins/pyscriptc.cli._main.json index 7887e374..37fa647e 100644 --- a/tests/census_pins/pyscriptc.cli._main.json +++ b/tests/census_pins/pyscriptc.cli._main.json @@ -1 +1 @@ -{"pyscriptc.cli._main": 498} +{"pyscriptc.cli._main": 500} diff --git a/tests/census_pins/pyscriptc.embed.json b/tests/census_pins/pyscriptc.embed.json index bba5b7ff..f6c510c4 100644 --- a/tests/census_pins/pyscriptc.embed.json +++ b/tests/census_pins/pyscriptc.embed.json @@ -1 +1 @@ -{"pyscriptc.embed": 2} +{"pyscriptc.embed": 3} diff --git a/tests/census_pins/pyscriptc.emit.llvm._abi.json b/tests/census_pins/pyscriptc.emit.llvm._abi.json index b8c98f9a..d26b22d1 100644 --- a/tests/census_pins/pyscriptc.emit.llvm._abi.json +++ b/tests/census_pins/pyscriptc.emit.llvm._abi.json @@ -1 +1 @@ -{"pyscriptc.emit.llvm._abi": 19} +{"pyscriptc.emit.llvm._abi": 25} diff --git a/tests/census_pins/pyscriptc.emit.llvm._calls.json b/tests/census_pins/pyscriptc.emit.llvm._calls.json index 83588d19..21f5ccc5 100644 --- a/tests/census_pins/pyscriptc.emit.llvm._calls.json +++ b/tests/census_pins/pyscriptc.emit.llvm._calls.json @@ -1 +1 @@ -{"pyscriptc.emit.llvm._calls": 1} +{"pyscriptc.emit.llvm._calls": 2} diff --git a/tests/census_pins/pyscriptc.emit.llvm._driver.json b/tests/census_pins/pyscriptc.emit.llvm._driver.json index 773b5512..613862fb 100644 --- a/tests/census_pins/pyscriptc.emit.llvm._driver.json +++ b/tests/census_pins/pyscriptc.emit.llvm._driver.json @@ -1 +1 @@ -{"pyscriptc.emit.llvm._driver": 75} +{"pyscriptc.emit.llvm._driver": 77} diff --git a/tests/census_pins/pyscriptc.lower._boxed_resolve.json b/tests/census_pins/pyscriptc.lower._boxed_resolve.json index 50d942a3..c4990b74 100644 --- a/tests/census_pins/pyscriptc.lower._boxed_resolve.json +++ b/tests/census_pins/pyscriptc.lower._boxed_resolve.json @@ -1 +1 @@ -{"pyscriptc.lower._boxed_resolve": 11} +{"pyscriptc.lower._boxed_resolve": 12} diff --git a/tests/census_pins/pyscriptc.lower._classes_more.json b/tests/census_pins/pyscriptc.lower._classes_more.json index adc2f745..a09fd992 100644 --- a/tests/census_pins/pyscriptc.lower._classes_more.json +++ b/tests/census_pins/pyscriptc.lower._classes_more.json @@ -1 +1 @@ -{"pyscriptc.lower._classes_more": 25} +{"pyscriptc.lower._classes_more": 26} diff --git a/tests/census_pins/pyscriptc.lower._stdlib.json b/tests/census_pins/pyscriptc.lower._stdlib.json index 174eea5d..8bada439 100644 --- a/tests/census_pins/pyscriptc.lower._stdlib.json +++ b/tests/census_pins/pyscriptc.lower._stdlib.json @@ -1 +1 @@ -{"pyscriptc.lower._stdlib": 52} +{"pyscriptc.lower._stdlib": 53} diff --git a/tests/census_pins/pyscriptc.package.json b/tests/census_pins/pyscriptc.package.json index 5f95a64e..fc11328a 100644 --- a/tests/census_pins/pyscriptc.package.json +++ b/tests/census_pins/pyscriptc.package.json @@ -1 +1 @@ -{"pyscriptc.package": 63} +{"pyscriptc.package": 69} diff --git a/tests/census_pins/pyscriptc.sitedeps.json b/tests/census_pins/pyscriptc.sitedeps.json index 3e62c995..1a73bb3a 100644 --- a/tests/census_pins/pyscriptc.sitedeps.json +++ b/tests/census_pins/pyscriptc.sitedeps.json @@ -1 +1 @@ -{"pyscriptc.sitedeps": 23} +{"pyscriptc.sitedeps": 22} diff --git a/tests/census_pins/pyscriptc.stdlibzip.json b/tests/census_pins/pyscriptc.stdlibzip.json index d1b492a6..b7cb94ed 100644 --- a/tests/census_pins/pyscriptc.stdlibzip.json +++ b/tests/census_pins/pyscriptc.stdlibzip.json @@ -1 +1 @@ -{"pyscriptc.stdlibzip": 39} +{"pyscriptc.stdlibzip": 46} diff --git a/tests/test_plane_split.py b/tests/test_plane_split.py index 2b5579c2..4b824dd6 100644 --- a/tests/test_plane_split.py +++ b/tests/test_plane_split.py @@ -346,3 +346,233 @@ def test_island_definition_raising_propagates(tmp_path: Path) -> None: want = subprocess.run([sys.executable, str(prog)], capture_output=True, text=True, timeout=60, cwd=tmp_path) assert (got.returncode, got.stdout) == (want.returncode, want.stdout) assert got.stdout == "caught\n" + + +def _class_shapes(root: Path) -> None: + _write( + root, + { + "shapes.py": ( + "import dataclasses\n\n\n@dataclasses.dataclass(order=True)\nclass Point:\n x: int\n y: int\n" + ), + }, + ) + + +def test_island_class_import_does_not_fence_the_consumer_top_level(tmp_path: Path) -> None: + """`from shapes import Point` binds across the boundary instead of fencing. + + Before the export table carried island classes, this import fenced at + module level and Decision A dragged the consumer's whole top level + island-side. The consumer stays native; only the uses that need the + class natively (a base class, an except clause) fence their own + definition. + """ + prog = tmp_path / "prog" + _class_shapes(prog) + _write( + prog, + { + "__main__.py": ( + "from shapes import Point\n" + "\n" + "\n" + "def main() -> None:\n" + " p = Point(10, 20)\n" + " print(p)\n" + "\n" + "\n" + "if __name__ == '__main__':\n" + " main()\n" + ), + }, + ) + out = _compare(tmp_path, prog) + assert out == "Point(x=10, y=20)\n" + planes = _plane_map(tmp_path, prog) + assert planes["moduleTopLevel"] == "native" + assert planes["classes"]["shapes.Point"] == "island" + + +def test_island_class_as_a_value_and_kwargs(tmp_path: Path) -> None: + """The class object taken as a value is a handle from the sibling namespace.""" + prog = tmp_path / "prog" + _class_shapes(prog) + _write( + prog, + { + "__main__.py": ( + "from shapes import Point\n" + "\n" + "\n" + "def main() -> None:\n" + " q = Point(x=3, y=4)\n" + " cls = Point\n" + " print(q.y, cls is Point)\n" + "\n" + "\n" + "if __name__ == '__main__':\n" + " main()\n" + ), + }, + ) + out = _compare(tmp_path, prog) + assert out == "4 True\n" + + +def test_package_reexport_of_an_island_class_names_the_owner(tmp_path: Path) -> None: + """`from pkg import Cls` reaches the class in the defining module's namespace. + + The package `__init__.py` re-exports the class from a sibling (`from + .shapes import Point`), so the module the import names (`pkg`) is not + the module whose island namespace holds the class object. The export + entry carries the owner, and the consumer's call lands there. + """ + prog = tmp_path / "prog" + _write( + prog, + { + "__init__.py": "from .shapes import Point\n", + "shapes.py": ( + "import dataclasses\n\n\n@dataclasses.dataclass(order=True)\nclass Point:\n x: int\n y: int\n" + ), + "__main__.py": ( + "from prog.shapes import Point\n" + "\n" + "\n" + "def main() -> None:\n" + " print(Point(1, 2).x)\n" + "\n" + "\n" + "if __name__ == '__main__':\n" + " main()\n" + ), + }, + ) + binary, comp = _compile(tmp_path, prog) + # A package-shaped program runs whole island-side (the split refuses a + # package tree), so the assertion is the run, not the plane map. + assert comp.returncode == 0, comp.stderr + got = subprocess.run([str(binary)], capture_output=True, text=True, timeout=60, cwd=tmp_path) + want = subprocess.run([sys.executable, "-m", "prog"], capture_output=True, text=True, timeout=60, cwd=prog.parent) + assert got.returncode == want.returncode == 0, (got.stderr, want.stderr) + assert got.stdout == want.stdout == "1\n" + + +def test_native_subclass_of_a_cross_module_island_class_fences_the_subclass(tmp_path: Path) -> None: + """A native class cannot extend a sibling's island class. + + The island class has no native record, so the subclass definition + fences and joins the island instead of refusing the compile. + """ + prog = tmp_path / "prog" + _class_shapes(prog) + _write( + prog, + { + "__main__.py": ( + "from shapes import Point\n" + "\n" + "\n" + "class Spot(Point):\n" + " pass\n" + "\n" + "\n" + "def main() -> None:\n" + " s = Spot(1, 2)\n" + " print(s.x, s.y)\n" + "\n" + "\n" + "if __name__ == '__main__':\n" + " main()\n" + ), + }, + ) + binary, comp = _compile(tmp_path, prog) + assert comp.returncode == 0, comp.stderr + planes = _plane_map(tmp_path, prog) + assert planes["classes"]["__main__.Spot"] == "island" + got = subprocess.run([str(binary)], capture_output=True, text=True, timeout=60, cwd=tmp_path) + want = subprocess.run([sys.executable, str(prog)], capture_output=True, text=True, timeout=60, cwd=tmp_path) + assert (got.returncode, got.stdout, got.stderr) == (want.returncode, want.stdout, want.stderr) + assert got.stdout == "1 2\n" + + +def test_module_owns_its_island_floor_namespace_in_split_mode(tmp_path: Path) -> None: + """A module's own island definitions ride the namespaced seam. + + The split mode has no single island namespace: every module's floor + is its own namespace. An island function of the entry module is + called through `ps_island_mod_fn_invoke` with the entry module's name, + not the never-seeded single namespace. + """ + prog = tmp_path / "prog" + _write( + prog, + { + "__main__.py": ( + "from clean import add\n" + "\n" + "\n" + "def main() -> None:\n" + " print(add(1, 2))\n" + " tripled: int = helper(2)\n" + " print(tripled)\n" + "\n" + "\n" + "def helper(x):\n" + " return x * 3\n" + "\n" + "\n" + "if __name__ == '__main__':\n" + " main()\n" + ), + "clean.py": "def add(a: int, b: int) -> int:\n return a + b\n", + }, + ) + binary, comp = _compile(tmp_path, prog) + assert comp.returncode == 0, comp.stderr + llvm = (binary.parent / f"{binary.name}.ll").read_text(encoding="utf-8") + assert "ps_island_mod_fn_invoke" in llvm + got = subprocess.run([str(binary)], capture_output=True, text=True, timeout=60, cwd=tmp_path) + want = subprocess.run([sys.executable, str(prog)], capture_output=True, text=True, timeout=60, cwd=tmp_path) + assert (got.returncode, got.stdout) == (want.returncode, want.stdout) + assert got.stdout == "3\n6\n" + + +def test_island_floor_seed_imports_its_sibling(tmp_path: Path) -> None: + """A seed whose body imports a sibling finds the sibling's namespace. + + The seeds run in the order CPython finishes the module bodies, so a + module that imports a sibling is seeded after it. + """ + prog = tmp_path / "prog" + _write( + prog, + { + "__main__.py": ( + "from mid import relay\n" + "\n" + "\n" + "def main() -> None:\n" + " print(relay('x'))\n" + "\n" + "\n" + "if __name__ == '__main__':\n" + " main()\n" + ), + "mid.py": "from leaf import shout\n\n\ndef relay(s):\n return shout(s + '!')\n", + "leaf.py": "def shout(s):\n return s.upper()\n", + }, + ) + binary, comp = _compile(tmp_path, prog) + assert comp.returncode == 0, comp.stderr + llvm = (binary.parent / f"{binary.name}.ll").read_text(encoding="utf-8") + # `leaf` finishes before `mid`, so its seed runs first. + leaf_pos = llvm.index("ps.fmt.island.mod.leaf") + mid_pos = llvm.index("ps.fmt.island.mod.mid") + assert leaf_pos < mid_pos + got = subprocess.run([str(binary)], capture_output=True, text=True, timeout=60, cwd=tmp_path) + want = subprocess.run([sys.executable, str(prog)], capture_output=True, text=True, timeout=60, cwd=tmp_path) + assert (got.returncode, got.stdout) == (want.returncode, want.stdout) + assert got.stdout == "X!\n" diff --git a/tests/test_selfbin.py b/tests/test_selfbin.py new file mode 100644 index 00000000..db787444 --- /dev/null +++ b/tests/test_selfbin.py @@ -0,0 +1,132 @@ +"""The self-compiled pyscriptc: a single binary that compiles programs. + +The binary under test is built from this checkout with +`pyscriptc compile src/pyscriptc --program --skip-check --island --emit +bin --embed-python bundle --embed-headers`, so it carries its own CPython +runtime, the C runtime sources and the package data: the host needs no +Python, no uv and no pip to run it. What it still needs is a C toolchain +(clang or zig) on PATH, because a program compile shells out to it. + +The build takes about half a minute and the test compiles a hello world +through the binary, so the module is marked `slow` and CI can skip it. The +runtime pack is fetched from the network when it is not cached, so the test +skips when the pack is missing and no network is available. +""" + +from __future__ import annotations + +import os +import shutil +import subprocess +import sys +from pathlib import Path + +import pytest + +from pyscriptc import embed + +ROOT = Path(__file__).resolve().parents[1] + +pytestmark = pytest.mark.slow + +HELLO = 'def main() -> None:\n print("hello from a native binary")\n\n\nif __name__ == "__main__":\n main()\n' + + +def _pack_ready() -> bool: + """True when the runtime pack is unpacked, so the build needs no network.""" + try: + pack = embed.lock_pack() + except embed.EmbedError: + return False + return embed.pack_is_ready(pack) + + +def _build_selfbinary(out: Path) -> Path: + """Build the self-compiled pyscriptc; returns the binary path.""" + from pyscriptc.cli import main as cli_main + + argv = [ + "compile", + str(ROOT / "src" / "pyscriptc"), + "--program", + "--skip-check", + "--island", + "--emit", + "bin", + "--embed-python", + "bundle", + "--embed-headers", + "--bundle-exclude", + "mypy", + "-o", + str(out), + ] + saved = sys.argv + sys.argv = ["pyscriptc", *argv] + try: + code = cli_main(argv) + finally: + sys.argv = saved + assert code == 0, f"the self-compile failed with {code}" + return out / "bin" / out.name + + +@pytest.fixture(scope="module") +def selfbinary(tmp_path_factory: pytest.TempPathFactory) -> Path: + """The self-compiled pyscriptc, built once per module.""" + if sys.platform != "linux" and sys.platform != "darwin": + pytest.skip("the self-compile runs on Linux and macOS") + if not _pack_ready(): + pytest.skip("the runtime pack is not fetched; run `pyscriptc embed fetch` first") + out = tmp_path_factory.mktemp("selfbin") / "pyscriptc" + return _build_selfbinary(out) + + +def test_selfbinary_runs_without_python_on_path(selfbinary: Path, tmp_path: Path) -> None: + """The binary prints its usage with no PATH at all: no Python needed.""" + proc = subprocess.run( + [str(selfbinary)], + capture_output=True, + text=True, + env={"PATH": "/nonexistent"}, + cwd=tmp_path, + ) + assert proc.returncode != 0 + assert "usage: pyscriptc" in proc.stdout + proc.stderr + + +def test_selfbinary_compiles_hello_world(selfbinary: Path, tmp_path: Path) -> None: + """The binary compiles a hello world and the binary prints hello.""" + source = tmp_path / "hello.py" + source.write_text(HELLO, encoding="utf-8") + out = tmp_path / "hello" + toolchain = shutil.which("clang") or shutil.which("zig") + if toolchain is None: + pytest.skip("no C toolchain on PATH") + env = dict(os.environ) + env["PATH"] = str(Path(toolchain).parent) + proc = subprocess.run( + [str(selfbinary), "compile", str(source), "--skip-check", "--island", "--emit", "bin", "-o", str(out)], + capture_output=True, + text=True, + env=env, + cwd=tmp_path, + ) + assert proc.returncode == 0, proc.stdout + proc.stderr + run = subprocess.run([str(out)], capture_output=True, text=True, cwd=tmp_path) + assert run.returncode == 0, run.stdout + run.stderr + assert run.stdout.strip() == "hello from a native binary" + + +def test_selfbinary_assesses_a_program(selfbinary: Path, tmp_path: Path) -> None: + """The `assess` command of the binary answers without a Python install.""" + source = tmp_path / "checked.py" + source.write_text(HELLO, encoding="utf-8") + proc = subprocess.run( + [str(selfbinary), "assess", str(source)], + capture_output=True, + text=True, + cwd=tmp_path, + ) + assert proc.returncode == 0, proc.stdout + proc.stderr + assert "checked.py" in proc.stdout