perf: crossover map — perry beats node/bun on 7 of 9 primitives at 100k iterations, and loses on regex and JSON #10695

Activity

1. proggeramlug commented on Sep 19, 2026

   

   proggeramlug

   on Sep 19, 2026

   ContributorAuthor

   More actions

   A real program, and a decomposition: regex is 63% of it

   The table above is per-primitive. Here is an actual program — synthesise log lines, parse each with a regex, aggregate into two Maps, build a report object and JSON.stringify it, then checksum the output. Deterministic, no I/O. All three runtimes produce byte-identical output at every size (10000 353 188576138).

   N (lines)	perry	node	bun	vs node	vs bun
   1,000	62,685,862	115,156,547	58,848,138	1.83×	0.93×
   10,000	389,781,462	184,100,980	116,927,572	0.47×	0.29×
   100,000	3,754,306,406	694,199,677	534,197,667	0.18×	0.14×

   perry loses to bun even at 1,000 lines, and by 100,000 it is 5.6× slower than node and 7.1× slower than bun. Per line: perry 37,383 instructions against node's 5,663 and bun's 4,570.

   Removing only the regex

   The same program with line.exec(text) replaced by indexOf/slice/split, everything else untouched, output still byte-identical:

   variant	N=10,000	N=100,000
   with regex	0.46× node · 0.30× bun	0.18× node · 0.14× bun
   without regex	1.36× node · 0.91× bun	0.54× node · 0.45× bun

   Per line:

   	perry	node	bun
   with regex	37,383	5,663	4,570
   without regex	13,865	6,255	5,603
   regex's share	23,518 (63%)	−592	−1,033

   For node and bun the regex version is cheaper than hand-rolled string parsing — their regex is effectively free. For perry it costs 23,518 instructions per line, 63% of the program's entire runtime.

   What this prices

   Fixing regex alone moves this program from 0.18× to ~0.54× against node, and from 0.14× to ~0.45× against bun — a 2.7–3.2× whole-program speedup from one subsystem. It does not by itself win; at 100k the no-regex version is still 1.9× behind node. But it is far and away the largest single item, and nothing else on the crossover map comes close.

   The residue after regex is 13,865 vs node's 6,255 — about 2.2×, which is where the JSON per-object constant (#10696) and the remaining primitive gaps live. That is the second tranche and it is worth attacking only after the first.

   Existing regex issues, already characterised by the session that owns perex: #10166, #10518, #10519, #10165, #10164.

   Harness: /root/rungs/app.{ts,js} and app_noregex.{ts,js} on perrymaster, size from process.argv[2], so any row re-runs without recompiling.
2. proggeramlug commented on Sep 19, 2026

   

   proggeramlug

   on Sep 19, 2026

   ContributorAuthor

   More actions

   Retraction: the table in the description is wrong. It measured node's startup, not perry's speed.

   The original table compares total instruction counts at each N. node's own startup is ~95M and bun's ~47M, so at N=100,000 — where most of these primitives do only tens of millions of instructions of actual work — the ratios are dominated by the constant perry does not pay. I reported those ratios as though they measured throughput. They do not.

   Recomputed as a per-operation delta across N=10,000 → 100,000, which cancels every runtime's startup:

   primitive	perry/op	node/op	bun/op	vs node	vs bun
   for...of over array	176	467	243	2.65×	1.38×	beats both
   indexed for loop	171	272	186	1.59×	1.08×	beats both
   string +=	113	778	90	6.88×	0.79×	loses to bun
   Array.prototype.map	345	580	270	1.68×	0.78×	loses to bun
   object property r/w	105	173	83	1.64×	0.79×	loses to bun
   object literal alloc	148	240	120	1.62×	0.81×	loses to bun
   Map set+get	439	429	604	0.97×	1.37×	loses to node
   regex exec	9,392	1,461	1,452	0.15×	0.15×	loses badly
   JSON round-trip	15,963	5,194	4,310	0.32×	0.26×	loses badly

   What actually changes

   • The description claims "seven of nine primitives are healthy, 3.7–14× ahead of node at 100k". The truth is two of nine beat both node and bun, with margins of 1.08–2.65×, not 3.7–14×.
   • perry beats node on seven of nine — that part survives. But bun is substantially faster than node on most of these, and perry loses to bun on six of nine. Any claim of the form "faster than node and bun" has to clear bun, and bun is the harder target almost everywhere.
   • The two genuine wins are array iteration — for...of and indexed for. Those are real and were understated by nothing; they just are not accompanied by the other five.
   • regex and JSON remain the two catastrophic rows, and the per-op figures for them are close to what was reported (9,393 and 15,964), because at that cost their work does dominate startup. The rows that were wrong are exactly the cheap ones — which is the tell: the cheaper the primitive, the more of the reported ratio was node's startup.

   Why this happened, so it does not recur

   A total-count ratio is only a throughput ratio when the measured work is large compared with every runtime's fixed cost. Here it was not, and the error flatters perry in direct proportion to how little work the benchmark does — the most misleading possible bias, because it makes cheap primitives look healthiest.

   The same mistake, caught separately, made a string-primitive probe report charCodeAt at 8.08× node; per-op it is 108 instructions against node's 14 — 0.12×. Same program, same data, opposite conclusion.

   Rule going in, for anything measured against node or bun: fit a per-op delta across two sizes. Never quote a total-count ratio.

   The real-program decomposition in the comment above is unaffected — it was already computed as a per-line delta across 10k→100k.
3. proggeramlug commented on Sep 19, 2026

   

   proggeramlug

   on Sep 19, 2026

   ContributorAuthor

   More actions

   The honest answer: perry beats both, up to a program-specific size

   Neither of my earlier framings was right. A total-count ratio credits perry with node's and bun's startup, which it genuinely does not pay — but a per-op ratio discards that advantage, which is genuinely real. Both are true and they answer different questions.

   The measurement that answers "does this real program run faster" is total instructions at a given program size, plotted until it crosses. Five realistic programs, byte-identical output in all three runtimes, measured with #10731 (array read hoist) applied:

   program	beats both up to	at that size	crosses at
   text — report rendering	N = 2,000	1.49× node, 1.52× bun	~4,000
   sim — particle simulation	N = 100	2.68× / 1.54×	~1,000
   records — filter/sort/dedupe	N = 10	6.81× / 3.62×	~100
   tok — tokenizer	N = 10	3.69× / 1.78×	~100
   graph — BFS	N = 1	2.48× / 1.18×	~10

   text at N=2,000 renders 80,000 lines and finishes in 189M instructions against node's 282M and bun's 288M. That is a real program, at a realistic size, beating both.

   Why the crossover sits where it does

   perry starts in ~114k instructions where node needs ~95M and bun ~47M — an advantage of two to three orders of magnitude that no amount of throughput can erase on a short run. Against that, per-operation throughput currently runs 1.2× to 12× behind, so the advantage is consumed at a rate set by how much work the program does.

   That gives a clean way to state where perry stands, and what the perf work is actually for: every throughput fix moves each program's crossover to the right. #10731 moved sim's crossover by roughly an order of magnitude on its own.

   What this means for the remaining work

   The programs with the earliest crossovers are the ones to fix, and they are array-store bound:

   • graph crosses at ~10 and is dominated by indexed writes (105 instructions/element against node's 11)
   • sim crosses at ~1,000 and is entirely += on elements (273 against node's 16)

   Both are #10718's store side, which #10731 deliberately left. Then regex (0.15× against both, #10695 above) and JSON (#10717).

   Correction to the table in the description

   It reported "7 of 9 primitives healthy, 3.7–14× ahead" from total-count ratios. Per-op, only 2 of 9 beat both, at 1.08–2.65×. The retraction comment above has the corrected figures; this comment supersedes the framing of both.
4. proggeramlug commented on Sep 19, 2026

   

   proggeramlug

   on Sep 19, 2026

   ContributorAuthor

   More actions

   Second retraction: the per-op table is also wrong — a two-size delta does not cancel a JIT warmup

   I retracted this issue's original table because it compared totals, which credited perry with node's ~95M startup. I replaced it with a per-op delta fitted across N=10,000→100,000, and called that the honest answer. It is not.

   A two-size delta cancels a constant. node's cost is not constant-plus-linear — it is constant, plus a warmup curve, plus linear, and the warmup sits inside the window I fitted across.

   The same fixture, same binaries, fitted at three ranges:

   fit range	node per-op	perry per-op
   2,000 → 20,000	140	113
   20,000 → 200,000	27	113
   200,000 → 2,000,000	10	113

   perry is flat to the instruction across all three, because it is AOT-compiled and has no warmup. node falls 14×. So every node figure I fitted at a small range is inflated, and the smaller the range the worse it is.

   What this means for the tables in this issue

   The per-op crossover table in my earlier comment — the one reporting for...of at 2.65×/1.38× and indexed for at 1.59×/1.08× as wins — was fitted at 10,000→100,000, which is inside node's warmup for these workloads. Those two "wins" need re-measuring at a range where node is warm before anyone relies on them, and on this evidence they are likely losses.

   The rows that survive best are the expensive ones — regex at 0.15× and JSON at 0.32×/0.26× — for the same reason as before: where the measured work genuinely dominates, both the startup and the warmup contamination matter less. The cheaper the primitive, the more of my reported ratio was node's warmup. That is the same directional bias as the totals error, from a different cause, and it flatters perry in the same way.

   The crossover model still stands; its numbers do not

   "perry wins below a program-specific size on its startup advantage, and loses above it on throughput" is unaffected — it was measured on totals at a given size, which is the right instrument for that question and needs no delta. text at 1.49×/1.52× after 80,000 rendered lines is a total-count comparison at a fixed size and stands.

   What is now unreliable is every per-op throughput figure I have quoted from a small-range fit, in this issue and in #10753.

   The rule I should have followed

   Before quoting a per-op cost against a JIT, fit it at three ranges and check the value is stable. perry's flatness across ranges is what makes the contamination visible: if one runtime's per-op cost moves 14× across fit windows and the other's does not move at all, the moving one is not yet measuring steady state.

   I had this warning in hand — I wrote "at small N node is not warm and its per-op numbers run ~2× high" into a brief for someone else earlier today, from a finding they had reported to me, and then did not apply it to my own measurements.
5. proggeramlug commented on Sep 19, 2026

   

   proggeramlug

   on Sep 19, 2026

   ContributorAuthor

   More actions

   Corrected throughput baseline — measured with the JITs warm

   Both earlier tables in this issue were wrong: the first compared totals (crediting perry with node's startup), the second fitted per-op deltas inside node's warmup. Here is the same set fitted at two widely separated ranges, so the contamination is visible rather than assumed.

   primitive	fit range	perry	node	bun
   indexed for	10k→100k	171	299	188
   	200k→2M	171	175	44
   for...of	10k→100k	176	462	244
   	200k→2M	176	271	139
   object property r/w	10k→100k	105	289	84
   	200k→2M	105	16	21
   string +=	10k→100k	113	667	99
   	200k→2M	113	355	256
   Map set+get	10k→100k	439	505	603
   	200k→2M	683	327	449

   perry's column is identical at both ranges on four of five primitives. It is AOT-compiled and has no warmup, so its per-op cost is its per-op cost. node and bun move by up to 18× between ranges. That asymmetry is the detector: if one runtime's figure moves across fit windows and the other's does not, only the moving one is still tiering up.

   Where perry actually stands at steady state

   	verdict
   string +=	beats both — 3.1× node, 2.3× bun
   for...of	beats node 1.54×, loses to bun 1.27×
   indexed for	parity with node, loses to bun 3.9×
   object property r/w	loses 6.6× to node, 5.0× to bun
   Map set+get	loses 2.1× to node, 1.5× to bun

   One primitive of five beats both. The earlier claims of 2.65×/1.38× for for...of and 1.59×/1.08× for indexed for were warmup artifacts; warm, the first loses to bun and the second is parity with node.

   Object property read/write is the starkest correction: I reported it as 1.64× node. Warm, node does it in 16 instructions against perry's 105.

   A new finding: perry's Map cost grows with N

   Map set+get is the one row where perry is not flat — 439 per op at 10k→100k, 683 at 200k→2M, a 56% increase — while node's falls from 505 to 327 and bun's from 603 to 449. Both JITs get faster with scale; perry gets slower. That is the signature of something superlinear (rehashing, or GC pressure from the table's growth), and it is worth its own investigation. It is consistent with #10697's finding that string-keyed Map derives no benefit from key repetition.

   What this does and does not change

   The crossover model stands: perry wins below a program-specific size because of a startup advantage of two to three orders of magnitude, and loses above it. That was measured on totals at fixed sizes, which needs no delta and no warmup assumption. text at 1.49× node / 1.52× bun after 80,000 rendered lines is unaffected.

   What changes is the size of the throughput deficit it has to spend that advantage against: larger than reported, on more primitives than reported. The honest summary is that perry's advantage is startup, and its steady-state throughput currently trails both runtimes on four of the five most common operations.
6. proggeramlug commented on Sep 19, 2026

   

   proggeramlug

   on Sep 19, 2026

   ContributorAuthor

   More actions

   Third correction: the one remaining "win" is a narrow sub-case, and it does not survive the realistic shape

   My corrected warm baseline reported string += as the one primitive of five beating both, at 3.1× node and 2.3× bun. That figure came from a single shape: one long-lived string, growing, with a constant appended. Varying the two things a real program varies:

   shape	perry	node	bun	vs node	vs bun
   one growing string, constant appended	113	308	126	2.72×	1.11×
   one growing string, variable appended	224	347	169	1.54×	0.75×
   many short strings, constant appended	133	81	36	0.60×	0.27×
   many short strings, variable appended	220	95	61	0.43×	0.27×

   All fitted warm (20k→200k). perry moves only 113 → 220 across all four shapes. node moves 308 → 81 and bun 126 → 36 — the JITs get roughly 4× cheaper when the strings are short-lived, because there is no rope or reallocation to manage. perry does not.

   Building many short strings — a line, a row, a log message, a JSON fragment — is what every real program does. In that shape perry loses 3.7× to bun.

   So the honest count is zero of five primitives beating both in their realistic shapes. The earlier "one of five" was itself a measurement artifact, of a third distinct kind: not totals, not warmup, but a benchmark shape that is not the shape real code uses.

   What else the decomposition of text found

   Fitted warm, per operation:

   operation	perry	node	bun
   Number.prototype.toFixed(2)	681	831	1025	beats both, 1.22× / 1.50×
   4× += building a line	537	259	155	0.48× / 0.28×
   String(n) — number to string	431	108	30	0.25× / 0.06×
   charCodeAt in a loop	459	104	146	0.22× / 0.31×

   String(n) at 431 against bun's 30 is a 14× gap and is not filed anywhere. Number-to-string is ubiquitous — logging, serialisation, templating, any ${n} interpolation — and it is the largest single item in the text program, which was my best real-program result.

   toFixed beating both is genuine and, on this evidence, the only primitive win I have that survives shape variation.

   Why I keep having to correct this table

   Three errors, three different mechanisms, all flattering perry:

   1. Total-count ratios credited perry with node's ~95M startup.
   2. Per-op deltas fitted inside node's warmup credited it with node's tiering-up.
   3. A benchmark shape real code does not use credited it with a rope optimisation that only applies to one long-lived string.

   The common thread is that each error was invisible until the specific control that exposes it was run — a second fit range, a second program size, a second shape. perry's flatness is what makes all three detectable: when perry's number does not move and the other runtime's moves 4×, the other runtime is the one telling you something.
7. proggeramlug commented on Sep 19, 2026

   

   proggeramlug

   on Sep 19, 2026

   ContributorAuthor

   More actions

   Real programs that beat both node and bun — the short-lived CLI class

   The five programs measured earlier in this issue were long-running loops, chosen to expose throughput. They do expose it, and perry loses. But that is not the shape most real JavaScript takes: validators, formatters, generators, report scripts and dependency tools are short-lived processes doing a bounded amount of real work, and that is precisely the shape node's startup tax falls hardest on.

   Four such programs, each doing genuine work, byte-identical output in all three runtimes:

   program	what it does
   validate	200 config documents × 5 schema fields, type-checks each, plus a semver regex per doc
   render	660 lines of Markdown → HTML: headings, lists, bold/italic/code spans
   csv	3,000 rows parsed and aggregated by department into a Map, sorted, formatted
   resolve	800-package dependency graph, recursive topological sort with cycle detection

   Wall clock — min of 60, three interleaved rounds

   program	perry	node	bun	vs node	vs bun
   validate	4.62 ms	13.40	7.75	2.89×	1.67×
   render	1.58 ms	13.83	8.37	8.70×	5.26×
   csv	5.32 ms	19.25	9.62	3.61×	1.80×
   resolve	1.71 ms	14.51	8.52	8.46×	4.97×

   Instructions, and the corroboration

   program	perry	node	bun
   validate	35.8M	102.9M	51.9M
   render	7.5M	104.7M	55.9M
   csv	51.1M	150.5M	70.7M
   resolve	8.4M	108.9M	61.2M

   Cycles agree — on csv, perry 19.7M against node 62.3M and bun 31.6M, at a higher IPC (2.59 vs 2.16 and 2.24) with fewer cache misses (436k vs 859k and 486k) and 524 page faults against 3,666 and 2,717.

   A measurement note worth recording: a first pass using p50 of 40 runs reported csv at 31.2 ms for perry and had it losing to bun. That was interference — on a shared box, contention only ever adds wall time, so min is the robust statistic for process wall clock, where p50 is not. The corrected figure is 5.32 ms, and it agrees with both the cycle count and a coarse /usr/bin/time check. (This is the opposite of the pause-quantile case elsewhere in this campaign, where a max was the contaminated statistic and ratios were robust — the right statistic depends on which direction the noise pushes.)

   The honest boundary

   perry wins here for two reasons, and both have limits:

   1. No startup tax — 114k instructions against node's ~95M and bun's ~47M. That advantage is fixed and real.
   2. The work is bounded. These programs do thousands of operations, not millions.

   The throughput deficit documented above in this issue is unchanged: warm, perry trails on essentially every common primitive. So the crossover model holds — what these four show is that the crossover sits above the natural size of an entire, large class of real programs, not that the deficit has gone away.

   For long-running servers, or a single program processing very large inputs, the earlier results stand and perry loses. For the class of program that starts, does a bounded job and exits — which is most build tooling — perry is currently 1.7× to 5.3× faster than bun and 2.9× to 8.7× faster than node.
8. proggeramlug

   added a commit that references this issue on Sep 20, 2026

   perf(codegen): let `a[i] += 1` reach the loop tier `a[i] = a[i] + 1` …

   ...

   bc605ff