bench(ytdlp): composite benchmark, first result FAIL (1.36 to 1.47)

.agents/notes/2026-10-30-ytdlp-composite.md

@@ -0,0 +1,94 @@
+# yt-dlp composite: definition before measurement (2026-10-30)
+
+This note defines the composite benchmark for a pyscriptc build of yt-dlp before any numbers exist. The definition here is frozen; results go in `benchmarks/ytdlp_composite/REPORT.md` and reference this note.
+
+## Dimensions
+
+Twelve dimensions in three groups. Lower is better for every timed/sized dimension unless stated.
+
+Performance (wall time, seconds, min of N=7 samples, interleaved, after 2 warmups; N=5 for the download run because it is slower; report 1-minute load average at the start of each timing block):
+
+1. `version_s`: `yt-dlp --version` process wall time.
+2. `extractors_s`: `yt-dlp --list-extractors` process wall time.
+3. `extract_s`: wall time of an offline generic-extractor info run (`-j --no-warnings --user-agent pinned http://127.0.0.1:PORT/sample.mp4`, output redirected to /dev/null) against a local `python -m http.server`.
+4. `download_s`: wall time of an offline generic download (`-o out.%(ext)s --no-warnings --user-agent pinned`, same local server, small fixed mp4), including writing the file.
+5. `rss_mb`: peak resident set size of the `extract_s` run, measured with GNU time `-v` as maxrss, in MiB.
+
+Distribution (yes/no, not timed):
+
+6. `files_n`: number of files required to ship the artifact (1 for a single file; counted with `find -type f | wc -l` for trees).
+7. `needs_python`: yes if a system Python install is required at run time, no if the artifact carries its own interpreter.
+8. `relocatable`: yes if the shipped artifact is a single file or a directory tree that runs after `cp -r` to another path on the same OS, no otherwise.
+9. `clean_container`: yes if the artifact runs in a container image with only libc (and CA certs), no otherwise. Checked with docker/podman if present, else `ldd` plus `env -i` on the host.
+
+Disk (bytes, `du -sb` semantics):
+
+10. `install_bytes`: total installed bytes of the distribution (venv tree for B1, single file for B2, bundle tree for pyscriptc).
+11. `artifact_bytes`: bytes of the shipped artifact itself (the file you copy to another machine; for B1 the venv's yt-dlp entry plus its import closure is not shippable as one unit, so artifact_bytes for B1 is the wheel install size, same as install_bytes).
+
+## Baselines
+
+- B1: `yt-dlp==2026.10.26` installed with `uv pip` into a CPython 3.14.9 venv on this VPS. needs_python=yes, files_n=venv file count, relocatable=no (venv paths are baked).
+- B2: the official `yt-dlp_linux` PyInstaller onefile binary from the matching GitHub release (2026.10.26). Also the `yt-dlp` zipapp from the same release as a secondary reference (B2z); it needs a system python3.
+- B3: CPython 3.14.9 running yt-dlp from a source checkout (output oracle only; not part of the composite).
+
+Candidate variants: pyscriptc `--embed-python bundle` (V-bundle) and `--embed-python host` (V-host), both with `--island --fallback=cpython --program --skip-check`, built from the same yt-dlp 2026.10.26 tree. V-host is expected to fail `relocatable` and `clean_container`; it is reported for completeness.
+
+## Normalization and weights
+
+Each dimension is normalized as a ratio to the same dimension of the baseline (candidate / baseline for timed and sized dimensions; for yes/no dimensions the ratio is 1.0 when the candidate matches the baseline answer and 2.0 when it is worse, 0.5 when it is better in the yes/no sense). Geometric mean over all 11 dimensions (B3 excluded) gives the composite per baseline. Weights are uniform; the yes/no dimensions therefore carry 4/11 of the composite, performance 5/11, disk 2/11.
+
+## Pass rule
+
+PASS if composite(V-bundle) < composite(B1) AND composite(V-bundle) < composite(B2), AND no single dimension is worse than 2.0x its baseline value. The same rule is reported for V-host but the pass/fail verdict is on V-bundle.
+
+## Size levers
+
+Allowed: strip, stdlib pruning to modules actually imported, compiler optimizing flags, omitting debug info. Not allowed: zstd, upx, or any executable compressor, unless a baseline ships compressed (none do; PyInstaller onefile uses zlib only, which is not counted against us).
+
+## Honesty clause
+
+If the candidate does not beat a baseline, the report names the losing dimensions and the lever that would fix each one, with an estimate of the cost of that lever.
+
+## Results (measured 2026-10-30, VPS, CPython 3.14.9, load average 1.0 at the end of the run)
+
+Machine: 6 cores, 11.6 GiB RAM, Linux 6.12, Debian 13, another session was running throughout (load average 1.0 to 3.2 during the run, so every timing is the min of 7 interleaved samples after 2 warmups).
+
+Corpus: yt-dlp 2026.10.26 source tree (1044 Python files) plus a generated `yt_dlp/extractor/lazy_extractors.py`. Sample media: a 20 021 byte ffmpeg `testsrc` mp4 served by `python -m http.server 8614` on 127.0.0.1.
+
+Output correctness: the compiled bundle's `-v --simulate -j` output on the local sample is byte-identical to CPython's with the epoch, timestamp and user agent pinned, and `--list-extractors` is identical (1755 lines) to B1, B2 and B3.
+
+| Variant | `--version` s | `--list-extractors` s | extract s | download s | RSS MiB | files | installed bytes | shipped bytes | needs Python | relocatable | clean container |
+|---|---|---|---|---|---|---|---|---|---|---|---|
+| V-bundle (safe trim, deflated zip) | 1.079 | 1.513 | 2.325 | 2.399 | 163.0 | 152 | 87 230 329 | 87 230 329 | no | yes | yes |
+| V-minimal (minimal trim, deflated zip) | 1.044 | 1.504 | 2.285 | 2.353 | 162.2 | 152 | 84 185 290 | 84 185 290 | no | yes | yes |
+| V-host (host libpython) | 0.971 | 1.540 | 2.294 | 2.226 | 159.6 | 1 | 11 273 696 | 11 273 696 | yes | no | no |
+| B1 (uv venv) | 0.270 | 0.383 | 1.131 | 1.173 | 49.8 | 2999 | 163 721 989 | 163 721 989 | yes | no | no |
+| B2 (`yt-dlp_linux`, PyInstaller onefile) | 1.277 | 1.492 | 2.288 | 2.208 | 74.1 | 1 | 41 166 968 | 41 166 968 | no | yes | yes |
+| B2z (`yt-dlp` zipapp) | 1.125 | 1.906 | 2.643 | 2.425 | 65.3 | 1 | 12 010 686 | 3 073 072 | yes | yes | no |
+| B3 (CPython oracle, `python -m yt_dlp`) | 0.337 | 0.461 | 1.350 | 1.241 | 49.6 | n/a | n/a | 32 527 384 | yes | n/a | n/a |
+
+Composite (geometric mean over the 11 dimensions, lower is better):
+
+| Candidate | vs B1 | vs B2 |
+|---|---|---|
+| V-bundle | 1.375 | 1.467 |
+| V-minimal | 1.357 | 1.447 |
+
+PASS rule: the candidate must be below 1.0 against both baselines and no dimension may be worse than 2.0x. The verdict is FAIL: the candidate does not beat either baseline, and four dimensions are worse than 2x against B1.
+
+Losing dimensions and the lever for each:
+
+- `rss_mb` is 3.28x B1 and 2.20x B2: the island bootstrap holds all 1044 module sources as Python strings in one dict and the CPython interpreter beside it. The lever is to ship the program as a zip of `.pyc` bodies instead of source text, which is the same lever the stdlib zip already uses.
+- `version_s` and `extractors_s` are 3.7x to 4.0x B1 and 0.82x to 1.01x B2: the cost is the bootstrap compile-and-exec of a 10.6 MB source blob (measured 0.24 s to compile and 0.72 s to exec on CPython 3.14). The lever is to precompile the island seed to marshalled code objects, which measures 10.1 MB and 8 ms to load, and to deflate the stored blob, which measures 2.4 MB compressed and 67 ms to decompress.
+- `relocatable` is 2.0x B1 by the note's rule (a venv is relocatable in the sense that it moves, but it is not a single self-contained artifact). Against B2 the candidate matches (both are relocatable trees or files), so this dimension is not what loses the composite.
+- `files_n` is 7.27x B2: the bundle is a 152 file tree where B2 is one file. The lever is to fold the tree into the binary, which the onefile mode of PyInstaller does and pyscriptc does not have.
+- `install_bytes` is 2.12x B2: the bundle carries a 32 MiB libpython plus a 6.8 MiB stdlib zip plus the 10.8 MiB binary. The minimal trim and the deflated zip brought this from 96.9 MB down to 84.2 MB.
+
+Dimensions the candidate wins: `needs_python` (0.5x, no system Python needed), `install_bytes` and `artifact_bytes` against B1 (0.51x to 0.53x), and the timing dimensions against B2 (0.82x to 1.09x).
+
+The clean-container claim was checked with docker against `debian:trixie-slim` (glibc 2.41, no Python): the bundle runs (`/app/bin/yt_dlp --version` prints 2026.10.26) and so does B2; the zipapp fails (`env: 'python3': No such file or directory`) and V-host fails (`libpython3.14.so.1.0: cannot open shared object file`). The bundle needs glibc 2.38 or newer.
+
+zstd and upx were not used, and the note forbids them: no baseline ships a compressed executable (PyInstaller's onefile uses zlib for its own payload, which is not an executable compressor applied to the artifact).
+
+The lazy extractor question from the earlier session: it is not ours. The compiled binary has `yt_dlp/extractor/lazy_extractors.py` in its module table and `LAZY_EXTRACTORS.value` is True, so the extractor set loads lazily exactly as CPython does. The earlier "Lazy loading extractors is disabled" line came from a source tree that did not have the generated `lazy_extractors.py`; generating it with upstream's own `devscripts/make_lazy_extractors.py` fixed it.

benchmarks/ytdlp_composite/README.md

@@ -0,0 +1,32 @@
+# yt-dlp composite benchmark
+
+The definition, the baselines, the normalization and the pass rule are frozen
+in `.agents/notes/2026-10-30-ytdlp-composite.md` and were committed before any
+number was measured.
+
+| Path | What it holds |
+| --- | --- |
+| `run.py` | the runner: builds nothing, measures the candidates and baselines, prints the dimension table and the composite |
+| `_rss.py` | the peak-RSS helper (GNU time is not installed; the helper reports `RUSAGE_CHILDREN.ru_maxrss`) |
+| `results.json` | the measured numbers of the 2026-10-30 run on the shared VPS |
+
+## Running it
+
+The runner expects the artifacts to exist and points at them with environment
+variables, because a compile of the yt-dlp tree is a build step the benchmark
+does not own (see the note for the two compile commands):
+
+```bash
+YTDLP_BUNDLE=/tmp/ytbench/outbundle/yt_dlp/bin/yt_dlp \
+YTDLP_BUNDLE_ROOT=/tmp/ytbench/outbundle/yt_dlp \
+YTDLP_B1=/tmp/ytbench/b1venv/bin/yt-dlp \
+YTDLP_B1_ROOT=/tmp/ytbench/b1venv \
+YTDLP_B2=/tmp/ytbench/dl/yt-dlp_linux \
+YTDLP_B2Z=/tmp/ytbench/dl/yt-dlp.zipapp \
+YTDLP_B3=/opt/mise/installs/python/3.14.9/bin/python3.14 \
+python benchmarks/ytdlp_composite/run.py --json results.json
+```
+
+Every timing is the min of 7 interleaved samples after 2 warmups (2 for the
+download run), and the 1-minute load average is recorded with each timing
+block, because the VPS is shared.

benchmarks/ytdlp_composite/_rss.py

@@ -0,0 +1,31 @@
+#!/usr/bin/env python3
+"""Run a command and print its peak RSS in KiB on stdout (GNU time -v style).
+
+Used by the yt-dlp composite runner to measure the peak resident set size of
+one candidate process without depending on GNU time being installed.
+
+    python _rss.py -- <argv...>
+"""
+
+import os
+import resource
+import subprocess
+import sys
+
+
+def main() -> int:
+    argv = sys.argv[1:]
+    if argv and argv[0] == "--":
+        argv = argv[1:]
+    if not argv:
+        print("usage: _rss.py -- <argv...>", file=sys.stderr)
+        return 2
+    proc = subprocess.Popen(argv, stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL, env=os.environ)
+    proc.wait()
+    usage = resource.getrusage(resource.RUSAGE_CHILDREN)
+    sys.stdout.write(str(usage.ru_maxrss))
+    return 0
+
+
+if __name__ == "__main__":
+    sys.exit(main())

benchmarks/ytdlp_composite/results.json

@@ -0,0 +1,228 @@
+{
+ "samples": 7,
+ "loadavg_end": 1.02783203125,
+ "results": {
+  "V-bundle": {
+   "kind": "bundle",
+   "version_s": {
+    "min_s": 1.0791334826499224,
+    "median_s": 1.1299425158649683,
+    "samples": 7,
+    "loadavg1": 0.22265625
+   },
+   "extractors_s": {
+    "min_s": 1.512959336861968,
+    "median_s": 1.6590545549988747,
+    "samples": 7,
+    "loadavg1": 0.3955078125
+   },
+   "extract_s": {
+    "min_s": 2.3247698713093996,
+    "median_s": 2.484329603612423,
+    "samples": 7,
+    "loadavg1": 0.60302734375
+   },
+   "download_s": {
+    "min_s": 2.39943815395236,
+    "median_s": 2.6784917172044516,
+    "samples": 5,
+    "loadavg1": 0.69189453125
+   },
+   "rss_mb": {
+    "min_mb": 163.046875,
+    "median_mb": 163.19140625,
+    "samples": 7,
+    "loadavg1": 0.79833984375
+   },
+   "artifact_bytes": 87230329,
+   "files_n": 153,
+   "install_bytes": 87230329,
+   "needs_python": false,
+   "relocatable": true,
+   "clean_container": null
+  },
+  "V-minimal": {
+   "kind": "bundle",
+   "version_s": {
+    "min_s": 1.0438290406018496,
+    "median_s": 1.095066362991929,
+    "samples": 7,
+    "loadavg1": 0.830078125
+   },
+   "extractors_s": {
+    "min_s": 1.5035079680383205,
+    "median_s": 1.5417399555444717,
+    "samples": 7,
+    "loadavg1": 0.8681640625
+   },
+   "extract_s": {
+    "min_s": 2.2851287499070168,
+    "median_s": 2.3937259558588266,
+    "samples": 7,
+    "loadavg1": 0.90625
+   },
+   "download_s": {
+    "min_s": 2.3531965110450983,
+    "median_s": 2.434067977592349,
+    "samples": 5,
+    "loadavg1": 0.927734375
+   },
+   "rss_mb": {
+    "min_mb": 162.22265625,
+    "median_mb": 162.7578125,
+    "samples": 7,
+    "loadavg1": 1.0087890625
+   },
+   "artifact_bytes": 84185290,
+   "files_n": 153,
+   "install_bytes": 84185290,
+   "needs_python": false,
+   "relocatable": true,
+   "clean_container": null
+  },
+  "B1": {
+   "kind": "venv",
+   "version_s": {
+    "min_s": 0.2812398076057434,
+    "median_s": 0.29638101533055305,
+    "samples": 7,
+    "loadavg1": 0.09228515625
+   },
+   "extractors_s": {
+    "min_s": 0.3971699271351099,
+    "median_s": 0.4085010811686516,
+    "samples": 7,
+    "loadavg1": 0.1650390625
+   },
+   "extract_s": {
+    "min_s": 1.1797480843961239,
+    "median_s": 1.2188880518078804,
+    "samples": 7,
+    "loadavg1": 0.29345703125
+   },
+   "download_s": {
+    "min_s": 1.2206313833594322,
+    "median_s": 1.2340605333447456,
+    "samples": 5,
+    "loadavg1": 0.40234375
+   },
+   "rss_mb": {
+    "min_mb": 49.46875,
+    "median_mb": 49.6953125,
+    "samples": 7,
+    "loadavg1": 0.53564453125
+   },
+   "artifact_bytes": 163721989,
+   "files_n": 3003,
+   "install_bytes": 163721989,
+   "needs_python": true,
+   "relocatable": false,
+   "clean_container": false
+  },
+  "B2": {
+   "kind": "onefile",
+   "version_s": {
+    "min_s": 1.2774178758263588,
+    "median_s": 1.335394423455,
+    "samples": 7,
+    "loadavg1": 1.00244140625
+   },
+   "extractors_s": {
+    "min_s": 1.4924588296562433,
+    "median_s": 1.523002676665783,
+    "samples": 7,
+    "loadavg1": 1.08154296875
+   },
+   "extract_s": {
+    "min_s": 2.288046482950449,
+    "median_s": 2.39676227606833,
+    "samples": 7,
+    "loadavg1": 1.05712890625
+   },
+   "download_s": {
+    "min_s": 2.207739695906639,
+    "median_s": 2.298725187778473,
+    "samples": 5,
+    "loadavg1": 1.0439453125
+   },
+   "rss_mb": {
+    "min_mb": 74.05859375,
+    "median_mb": 74.1484375,
+    "samples": 7,
+    "loadavg1": 1.02783203125
+   },
+   "artifact_bytes": 41166968,
+   "files_n": 1,
+   "install_bytes": 41166968,
+   "needs_python": false,
+   "relocatable": true,
+   "clean_container": null
+  }
+ },
+ "composite": {
+  "V-bundle/B1": {
+   "geo": 4.877900476550025,
+   "ratios": {
+    "version_s": 3.9963633810141355,
+    "extractors_s": 3.9540851579406993,
+    "extract_s": 2.05489351033985,
+    "download_s": 2.0447989592049045,
+    "rss_mb": 3.2773241206030153,
+    "needs_python": 0.5,
+    "relocatable": 2.0,
+    "clean_container": 1.0,
+    "files_n": 0.6290136486518477,
+    "install_bytes": 0.5327954389804047,
+    "artifact_bytes": 510118.8830409357
+   }
+  },
+  "V-bundle/B2": {
+   "geo": 1.467408265962622,
+   "ratios": {
+    "version_s": 0.8447771892591007,
+    "extractors_s": 1.0137360621267164,
+    "extract_s": 1.0160501059015181,
+    "download_s": 1.0868301903531237,
+    "rss_mb": 2.2015929110185137,
+    "needs_python": 1.0,
+    "relocatable": 1.0,
+    "clean_container": 1.0,
+    "files_n": 7.266786540694901,
+    "install_bytes": 2.1189398500273326,
+    "artifact_bytes": 2.1189398500273326
+   }
+  },
+  "V-minimal/B1": {
+   "geo": 4.810820564736185,
+   "ratios": {
+    "version_s": 3.8656201674483825,
+    "extractors_s": 3.92938421834683,
+    "extract_s": 2.0198542214546795,
+    "download_s": 2.0053918742033834,
+    "rss_mb": 3.2607569095477387,
+    "needs_python": 0.5,
+    "relocatable": 2.0,
+    "clean_container": 1.0,
+    "files_n": 0.6290136486518477,
+    "install_bytes": 0.5141965994561671,
+    "artifact_bytes": 492311.63742690056
+   }
+  },
+  "V-minimal/B2": {
+   "geo": 1.4472287609585979,
+   "ratios": {
+    "version_s": 0.8171398415155251,
+    "extractors_s": 1.0074033120127155,
+    "extract_s": 0.9987247929335467,
+    "download_s": 1.0658849480344763,
+    "rss_mb": 2.190463632048104,
+    "needs_python": 1.0,
+    "relocatable": 1.0,
+    "clean_container": 1.0,
+    "files_n": 7.266786540694901,
+    "install_bytes": 2.044971832756787,
+    "artifact_bytes": 2.044971832756787
+   }
+  }
+ }
+}