Search 200 variants of pure noise, keep the luckiest one, and you get this:
n=900 WR 53% meanR +0.111 PF 1.26 t +2.77
Profit factor 1.26 over 900 trades. It looks like an edge. There isn't one — those trades were drawn with a mean of exactly zero.
edgelab is the harness that says so out loud:
G1_n PASS 900
G2_pf PASS 1.259
G3_t FAIL 2.77
G4_folds PASS [58.0, 13.1, 13.4, 15.4]
G5_bootstrap PASS 0.9966
VERDICT: REJECT
Four of five checks pass and the verdict is still REJECT. That is the whole idea: any single statistic can be searched against until it looks good, but a battery of pre-registered ones is much harder to fool. Reproduce it in about ten seconds — no market data, no worker, no configuration:
uv run --extra sci python examples/edgelab_demo.pyThe same battery on a rule with a genuine +0.19R edge returns PROMOTE. Telling those two apart is the entire problem.
Two pieces of infrastructure, usable independently.
simfarm/ ships Python jobs from a laptop to a workstation and gets
results back, so simulations run on the machine with the cores instead of
choking the one you're typing on. A thin, self-owned alternative to Ray/Dask
for a two-machine setup.
simfarm/edgelab/ is the statistics above: permutation nulls with a
finite-population correction, day-clustered t-statistics, block bootstrap,
synthetic no-edge markets, and a pre-registered promotion gate.
client ──submit(fn, args)──▶ worker (CPU pool + GPU lane)
◀──result / logs────
import simfarm
c = simfarm.connect("workstation") # token from SIMFARM_TOKEN
job = c.submit(my_sim, n_paths=1_000_000, gpu=False)
value = job.result() # blocks; RemoteError on failure
results = c.map(backtest, param_grid) # parallel sweep, order preservedsubmit()serialises the function — closures and lambdas included — with cloudpickle and sends it over HTTP.- The worker runs each job in its own subprocess: CPU jobs on a pool sized to
all cores,
gpu=Truejobs on a dedicated lane that falls back to CPU when no card is present. - Return values and worker-side
printoutput come back to the client. Remote exceptions surface asRemoteErrorwith the full traceback. check_env()reports client/worker version skew before it bites. This is the main failure mode of a two-machine setup — cloudpickle ships bytecode, so a Python minor-version gap between the two sides crashes the worker on unpickle.
- submit / result / status over HTTP, token-gated
- per-job processes with CPU + GPU lanes and CPU fallback
-
map()parameter sweeps (cancels stragglers when one item raises) - worker-side log capture, remote-exception propagation
- env fingerprint + skew detection
- live log streaming —
job.stream_logs()/simfarm logs <id> -f - job cancellation —
job.cancel()/simfarm cancel <id> -
simfarm ps— job table with real queued-vs-running state - result persistence across a worker restart
Jobs are spooled to $SIMFARM_SPOOL (default ~/.simfarm/jobs): metadata,
logs, and the result payload as separate files, written atomically. If the
worker machine reboots mid-sweep, finished results are still fetchable by job
id afterwards, and jobs that were in flight come back as interrupted rather
than vanishing or being reported as successes. Reattach from a fresh client
with c.job("<id>").
Queued jobs are not resumed across a restart — their payloads are held in memory only, so a job that never started is honestly marked interrupted rather than silently re-run.
job = c.submit(big_sweep, grid)
for chunk in job.stream_logs(): # yields output as the job produces it
print(chunk, end="")Log reads are offset-based rather than a held-open WebSocket, so a laptop sleep or a network blip resumes from the byte offset instead of dropping the stream.
Backtesting is easy to do and easy to fool yourself with. A strategy backtest with a few hundred trades and a twenty-configuration search behind it will produce something that looks profitable essentially every time. The interesting engineering problem is not finding a pattern — it's building a harness that tells you when the pattern is noise, before money is committed.
edgelab ships no rules and no strategy parameters. You supply the rule; it returns a per-trade DataFrame, and everything downstream is instrument-agnostic.
from simfarm.edgelab import day_matrix, run_gate, print_report, stats, synth
M, dates = day_matrix(bars_1m) # 1m OHLCV -> (day, bar, ohlcv)
trades = my_rule(M, dates) # -> DataFrame: day_idx, date, r
stats.basic(trades.r.values) # n, win rate, meanR, PF, t
stats.t_day_clustered(r, day_idx) # honest t when >1 trade/day
stats.filter_permutation_p(r_all, keep) # is a filter better than a RANDOM
# subset of the same size?
stats.min_perm_for_alpha(0.05 / n_trials) # draws needed to resolve an alpha
stats.day_bootstrap_p(r, day_idx) # P(total R > 0), day-clustered
print_report(run_gate(trades), name="my_rule")One battery, pre-registered thresholds, one verdict. All of it configurable via
GateConfig — the defaults are a starting point, not a claim.
| check | default | |
|---|---|---|
| G1 | trade count | n >= 600 |
| G2 | profit factor | PF >= 1.25 |
| G3 | t-statistic (day-clustered when >1 trade shares a day) | t >= 3.0 |
| G4 | all time folds positive | 4 folds |
| G5 | day-clustered bootstrap P(total R > 0) |
>= 95%, 10k draws |
| G6 | luck: beats the best PF of the same rule on synthetic no-edge markets | supplied |
| G7 | transfer: still positive on a second instrument | t >= 1.0 |
| G8 | friction: survives 3× cost | PF >= 1.10, t >= 1.5 |
| G9 | structural: only per-trade R streams are accepted (enforced by shape) | always |
G1–G5 are hard requirements. G6–G8 need inputs you must supply and are reported
as SKIPPED without them — and a skipped check is not a pass. PROMOTE
requires every supplied check to pass with none of G6–G8 skipped; LEAD means
the hard checks passed but the evidence is incomplete.
Two bugs found in this machinery, both of which silently changed verdicts, are the reason the primitives are written the way they are:
- A permutation test whose null was drawn with replacement, making it a
bootstrap rather than a same-size-subset test. The correct null carries the
finite-population correction
sqrt((n-k)/(n-1))and is narrower — by 1.15× at a 25% keep fraction and 2.03× at 76%. The uncorrected version was widest exactly where real filters operate, inflating p and turning genuine filters into recorded nulls. - The same primitive run at
n_perm=4000against a Bonferroni alpha of 0.00078, where the estimate carried 57% relative error — a measurement coarser than the threshold it was being compared to. Hencemin_perm_for_alpha().
examples/stats_selftest.py checks the permutation null is calibrated and
fails on the pre-fix implementation. It needs no market data.
Worker — see scripts/windows-setup.md if the
worker lives in WSL2 behind Windows NAT:
uv sync --extra sci
export SIMFARM_TOKEN="$(openssl rand -hex 16)" # copy this to the client
uv run simfarm worker # binds 0.0.0.0:8765Client:
uv sync --extra sci
export SIMFARM_TOKEN="<same token>"
uv run simfarm ping <worker-hostname>
uv run python examples/monte_carlo.py <worker-hostname>edgelab, no worker and no data required:
uv run --extra sci python examples/edgelab_demo.py # gate vs noise vs real edge
uv run --extra sci python examples/stats_selftest.py # permutation null calibrationThe worker executes arbitrary pickled callables sent by anything holding the
shared token. That is the whole point, and it means the worker is exactly as
privileged as the client. Run it on a private network (a tailnet, a LAN you
control) — never expose port 8765 to the internet, and treat SIMFARM_TOKEN
like an SSH key.
uv.lock is committed on purpose. cloudpickle ships function bytecode, so
client and worker must run the same Python minor version — a 3.13 client
against a 3.14 worker crashes the worker subprocess on unpickle. Install both
sides the same way: uv sync --extra sci.
No market data is included. edgelab reads 1-minute OHLCV as a pandas frame
with open/high/low/close/volume columns and a UTC DatetimeIndex; bring your
own from whichever vendor you license.
uv run --extra dev pytest tests/ -qsimfarm/ client, worker, protocol, CLI
simfarm/edgelab/ stats, gate, day-matrix loaders, synthetic no-edge markets
examples/ runnable demos — no market data required
scripts/ worker bootstrap, WSL2/Tailscale setup notes
- Optional result cache (skip re-running an identical payload)
- Auto-ship the client's local source package to the worker (kill env skew)
- Multiple workers / basic load balancing
- Spool GC / retention policy (
simfarm rm, age-based purge)
MIT — see LICENSE.