Add replayable portfolio accounting and SQLite experiments

.github/workflows/tests.yml

@@ -11,4 +11,8 @@ jobs:
           cache: pip
       - run: pip install -e ".[dev]"
       - run: python -m unittest discover -s tests -v
-      - run: ruff check engine tests run.py run_cross_sectional.py
+      - run: ruff check engine tests run.py run_cross_sectional.py run_offline.py run_experiments.py benchmark.py
+      - run: python run_offline.py > /tmp/offline.txt
+      - run: diff -u fixtures/synthetic/expected.txt /tmp/offline.txt
+      - run: python run_experiments.py --db /tmp/experiments.sqlite demo
+      - run: python benchmark.py --symbols 3 --bars 90 --repeats 1

.gitignore

@@ -11,3 +11,7 @@ data/*.png
 data/**/*.csv
 data/**/*.png
 !data/.gitkeep
+
+.venv-runtime/
+*.sqlite
+*.sqlite-*

README.md

@@ -30,6 +30,40 @@ pip install -e ".[dev]"
 python -m unittest discover -s tests -v
 ```
 
+## Offline accounting and stored replay
+
+After installation, run from this repository's root:
+
+```bash
+python run_offline.py
+python run_experiments.py --db experiments.sqlite demo
+```
+
+The first command prints a versioned synthetic OHLC fixture's expected ledgers
+and timeline. The second stores four cost scenarios in SQLite and verifies exact
+replay from stored configuration and prices. No credentials or network are used.
+
+```text
+Jan 1 close: signal A at 10.00; B rejected for insufficient cash
+Jan 2 open:  buy 8 A at 12.012; commission 1.00; cash 2.904
+Jan 2 close: 8 A marked at 13.00; equity 106.904
+Jan 3 open:  exit A; final cash and equity 113.792
+```
+
+Today's close signal cannot receive today's price. The invented example is an
+accounting demonstration, not a historical trading result. See the
+[fixture and regeneration command](fixtures/synthetic/README.md),
+[storage, architecture and data policies](docs/EXPERIMENTS.md), and
+[measured benchmark](docs/PERFORMANCE.md).
+
+`compare RUN_ID_A RUN_ID_B`, `costs RUN_ID`, `replay RUN_ID` and `plan RUN_ID`
+are subcommands of `run_experiments.py`. Comparison rejects mismatched data,
+timing or costs; the separate cost report varies only slippage. The original
+`Backtest.run()` ledger API is preserved. Strict calendar validation is available;
+the default retains the documented common-calendar policy.
+
+## Historical examples
+
 Run the SPY example (this step downloads market data):
 
 ```bash

benchmark.py

@@ -0,0 +1,166 @@
+"""Seeded offline benchmark; correctness checks run before any measurements."""
+
+import argparse
+import cProfile
+import json
+import os
+import platform
+import resource
+import statistics
+import subprocess
+import tempfile
+import time
+import tracemalloc
+from pathlib import Path
+
+import numpy as np
+import pandas as pd
+
+from engine.backtest import Backtest
+from engine.reproducibility import digest, ledger_records, versions
+from engine.storage import SQLiteStore, snapshot
+from engine.strategy import MovingAverageCross
+
+
+def generate_data(symbols, bars, seed):
+    if symbols < 1 or bars < 31:
+        raise ValueError("benchmark requires at least one symbol and 31 bars")
+    rng = np.random.default_rng(seed)
+    dates = pd.bdate_range("2000-01-03", periods=bars)
+    closes = 100 * np.exp(np.cumsum(rng.normal(0, 0.01, (bars, symbols)), axis=0))
+    opens = np.vstack([closes[0], closes[:-1]]) * np.exp(rng.normal(0, 0.002, (bars, symbols)))
+    return {
+        f"S{i:03}": pd.DataFrame({"Open": opens[:, i], "Close": closes[:, i]}, index=dates)
+        for i in range(symbols)
+    }
+
+
+def make_backtest(data, seed):
+    return Backtest(
+        list(data),
+        price_data=data,
+        strategy_cls=MovingAverageCross,
+        strategy_kwargs={"short": 5, "long": 30},
+        initial_capital=1_000_000,
+        calendar_policy="strict",
+        provenance={"kind": "synthetic", "seed": seed, "source": "benchmark.generate_data v1"},
+    )
+
+
+def check_accounting(result, initial):
+    fills = result["fills"]
+    if not fills.empty:
+        if not (fills.fill_dt > fills.submitted_dt).all():
+            raise AssertionError("same-bar fill")
+        flows = fills.groupby("fill_dt").cash_flow.sum()
+    else:
+        flows = pd.Series(dtype=float)
+    cash = result["cash"].set_index("dt")
+    expected_cash = initial + flows.reindex(cash.index, fill_value=0).cumsum()
+    np.testing.assert_allclose(cash.cash, expected_cash, rtol=1e-12, atol=1e-7)
+    marked = result["positions"].groupby("dt").market_value.sum()
+    np.testing.assert_allclose(cash.equity, cash.cash + marked, rtol=1e-12, atol=1e-7)
+    if cash.cash.min() < -1e-8 or result["positions"].quantity.min() < 0:
+        raise AssertionError("negative cash or short position")
+
+
+def main():
+    parser = argparse.ArgumentParser(description=__doc__)
+    parser.add_argument("--symbols", type=int, default=20)
+    parser.add_argument("--bars", type=int, default=2520)
+    parser.add_argument("--seed", type=int, default=17)
+    parser.add_argument("--repeats", type=int, default=3)
+    parser.add_argument("--profile", type=Path)
+    args = parser.parse_args()
+    if args.repeats < 1:
+        parser.error("repeats must be positive")
+    data = generate_data(args.symbols, args.bars, args.seed)
+
+    # Untimed correctness run precedes throughput, allocation and profiling runs.
+    baseline = make_backtest(data, args.seed)
+    result = baseline.run()
+    check_accounting(result, 1_000_000)
+    expected = digest(ledger_records(result))
+    event_count = len(baseline.event_records)
+    ledger_rows = sum(len(frame) for frame in result.values())
+    del baseline, result
+    seconds = []
+    for _ in range(args.repeats):
+        started = time.perf_counter()
+        bt = make_backtest(data, args.seed)
+        result = bt.run()
+        seconds.append(time.perf_counter() - started)
+        if digest(ledger_records(result)) != expected:
+            raise AssertionError("benchmark replay mismatch")
+        del bt, result
+
+    # Separate allocation measurement so tracemalloc does not distort throughput.
+    tracemalloc.start()
+    bt = make_backtest(data, args.seed)
+    result = bt.run()
+    _, peak = tracemalloc.get_traced_memory()
+    tracemalloc.stop()
+    started = time.perf_counter()
+    experiment = snapshot(bt, result)
+    snapshot_seconds = time.perf_counter() - started
+    with tempfile.TemporaryDirectory() as directory:
+        path = Path(directory) / "benchmark.sqlite"
+        with SQLiteStore(path) as store:
+            started = time.perf_counter()
+            store.save(experiment)
+            save_seconds = time.perf_counter() - started
+            started = time.perf_counter()
+            store.load(bt.run_id)
+            load_seconds = time.perf_counter() - started
+            plan = store.query_plan(bt.run_id)
+        db_bytes = path.stat().st_size
+    if args.profile:
+        args.profile.parent.mkdir(parents=True, exist_ok=True)
+        profiler = cProfile.Profile()
+        profiler.enable()
+        profiled = make_backtest(data, args.seed)
+        profiled.run()
+        profiler.disable()
+        profiler.dump_stats(args.profile)
+    cpu = platform.processor() or platform.machine()
+    if platform.system() == "Darwin":
+        try:
+            cpu = subprocess.check_output(
+                ["sysctl", "-n", "machdep.cpu.brand_string"], text=True, stderr=subprocess.DEVNULL
+            ).strip()
+        except (OSError, subprocess.CalledProcessError):
+            cpu += " (CPU model unavailable in sandbox)"
+    rss = resource.getrusage(resource.RUSAGE_SELF).ru_maxrss
+    rss_bytes = rss if platform.system() == "Darwin" else rss * 1024
+    report = {
+        "kind": "synthetic",
+        "seed": args.seed,
+        "symbols": args.symbols,
+        "bars_per_symbol": args.bars,
+        "input_rows": args.symbols * args.bars,
+        "consumed_columns": ["Open", "Close"],
+        "events": event_count,
+        "ledger_rows": ledger_rows,
+        "repeats": args.repeats,
+        "seconds": seconds,
+        "median_seconds": statistics.median(seconds),
+        "events_per_second": event_count / statistics.median(seconds),
+        "tracemalloc_peak_bytes": peak,
+        "process_peak_rss_bytes": rss_bytes,
+        "snapshot_seconds": snapshot_seconds,
+        "sqlite_save_seconds": save_seconds,
+        "sqlite_load_seconds": load_seconds,
+        "database_bytes": db_bytes,
+        "ledger_hash": expected,
+        "versions": versions(),
+        "hardware": {"os": platform.platform(), "cpu": cpu, "logical_cpus": os.cpu_count()},
+        "query_plan": plan,
+        "measurement_scope": "Throughput includes Backtest construction, input hashing and run; excludes "
+        "data generation, validation, persistence, profiling and allocation tracing. Allocation peak "
+        "is a separate constructor/run; RSS is process lifetime and includes other passes.",
+    }
+    print(json.dumps(report, indent=2))
+
+
+if __name__ == "__main__":
+    main()

docs/DESIGN.md

@@ -19,3 +19,10 @@ An order submitted on the final bar is cancelled with `end_of_data`; it is never
 ## Intentionally excluded
 
 Version 1 does not add more strategies. Its goal is a reliable engine boundary, auditable ledgers, and honest evaluation—not a strategy catalogue.
+
+## Experiment persistence
+
+The completed run can now be saved behind a save/load interface, with immutable
+audit events, content identities and exact configuration replay. See
+[replayable experiments](EXPERIMENTS.md) for storage architecture, SQL reports,
+compatibility details and data policies.

docs/EXPERIMENTS.md

@@ -0,0 +1,132 @@
+# Replayable experiments
+
+## Architecture and public API
+
+`Backtest(...).run()` still returns the original seven pandas ledgers. Existing
+strategy classes and injected DataFrames continue to work. `event_records`,
+`run_id`, `config`, `input_snapshot` and `metadata` are additional attributes.
+Instances are single-use; create a fresh instance for each run.
+
+The queue retains MarketEvent, SignalEvent, OrderEvent and FillEvent. The audit
+stream uses immutable records for signals, accepted orders, rejected quantities,
+booked fills and valuations. Accepted records contain the order and total cash
+reservation; broker fill proposals become booked fill events only after actual
+cash/position checks. This distinction prevents replay from booking a rejected
+or reduced quantity as if it filled. Sequence numbers start at zero and order
+all records within a run; timestamps alone cannot order events on the same day.
+A final cancellation releases reservations and emits a terminal valuation at the
+last timestamp, without adding another return observation.
+
+The portfolio remains the sole accounting implementation. The store receives a
+completed experiment through `ExperimentStore.save` and retrieves it through
+`load`; neither implementation computes trades. MemoryStore provides isolated
+copies for tests. SQLiteStore implements the same boundary with explicit SQL.
+There is no service, ORM, broker integration or production database connection.
+
+## Identity and reproducibility
+
+SHA-256 run IDs address canonical JSON containing the configuration, normalized
+input hash, engine source hash, strategy source hash, dependency versions, timing,
+provenance and excluded dates. Ordered symbols are part of configuration because
+competing orders consume cash in that order. Configuration and data also have
+separate hashes. Identical inputs in the identical environment reproduce the
+same ID. Different costs, versions, source or provenance produce different IDs;
+an ID is a reproducible experiment identity, not a timestamped execution attempt.
+
+The database retains every normalized Open/Close observation before calendar
+intersection, plus the excluded-date list and provenance. Replay uses these
+stored observations, never a fresh download or the current fixture file. High,
+Low and unused input columns are not part of the consumed dataset. Preserve an
+external raw-data artifact if those columns are needed for separate research.
+
+Use `provenance={"kind": "synthetic", "source": "...", "seed": 17}` for synthetic
+inputs and identify source, adjustment conventions and dates for historical
+inputs. An unspecified injected frame is labeled `injected`, not claimed to be
+historical. The downloader uses adjusted historical prices. No live feed is
+implemented. Callers are responsible for truthful source labels.
+
+`replay` requires matching engine source, strategy source and Python/pandas/NumPy/
+SQLite versions, then checks every event and ledger against the saved manifest.
+The CLI permits only named built-in strategies; the Python API can accept an
+explicit registry for custom strategies. Database text is never imported or
+executed. Interactive classes without inspectable source may run but cannot be
+exactly replayed. Configuration and provenance must be JSON-serializable.
+Custom strategy determinism remains the caller's responsibility: source hashing
+does not capture arbitrary external files, global state or helper dependencies.
+
+## Storage and queries
+
+`engine/sql/001_initial.sql` creates runs, events and ledgers, keyed by run ID.
+Foreign keys, event-kind checks, JSON validity, nonnegative accounting fields,
+unique sequence/row keys and projection-to-JSON checks constrain storage. SQL
+report columns are checked against their complete JSON records to avoid divergent
+copies. Indexes cover run comparison groups, slippage groups, event kinds and
+ledger timelines. `PRAGMA user_version` tracks the migration. Initialization and
+whole-run saves use explicit transactions; failure rolls back the entire run.
+Newer unknown schema versions fail. A repeated identical save is idempotent;
+different output for the same ID fails. There is no partial-run resume support.
+
+`summary.sql` computes drawdown with a running MAX window and sums actual fill
+commissions and slippage. Tests reconcile these with `engine.metrics.performance`
+and pandas fill totals, including a declining equity curve. Queries bind run IDs
+and comparison hashes as parameters. `plan` runs SQLite's real EXPLAIN QUERY PLAN;
+see [measured performance](PERFORMANCE.md) for an observed plan.
+
+```bash
+python run_experiments.py --db experiments.sqlite demo
+python run_experiments.py --db experiments.sqlite replay RUN_ID
+python run_experiments.py --db experiments.sqlite compare RUN_ID_A RUN_ID_B
+python run_experiments.py --db experiments.sqlite costs RUN_ID
+python run_experiments.py --db experiments.sqlite plan RUN_ID
+```
+
+Copy IDs printed by `demo`. It stores and verifies replay for 0, 1, 5 and 10 bps.
+`compare` requires identical inputs, timing, initial capital, costs, source labels,
+calendar policy and engine/dependency versions; only strategy/configuration may
+vary. `costs` holds strategy/configuration constant and permits only slippage to
+vary. It reports separate reruns, not a post-hoc fee subtraction from one ledger.
+Comparisons are accounting reports, not selection or untouched evaluation claims.
+
+## Data policy and limits
+
+- `calendar_policy="common"` preserves the original intersection policy. A next
+  bar means the next date shared by every symbol, potentially several days later.
+  Excluded dates are retained in metadata. This can truncate a delisted symbol's
+  tail for the whole portfolio and is unsuitable for delisting research.
+- `calendar_policy="strict"` requires identical dates and is used by the stored
+  demo and benchmark. Missing internal bars or unequal tails fail before trading.
+  Neither policy forward-fills or invents an open. There is no exchange calendar:
+  a date missing from every symbol cannot be detected automatically. Daily
+  timestamps are session labels; timezone information is stripped for legacy
+  compatibility. Callers must align session labels across markets before injection.
+- Duplicate/invalid timestamps, nonfinite/missing prices and nonpositive prices
+  fail. Explicit nonzero `Stale`, `Delisted`, `Dividends` or `Stock Splits` columns
+  fail as unsupported. Repeated prices alone cannot establish staleness. Supply
+  quality flags or validate freshness upstream. Unflagged corporate actions
+  cannot be inferred reliably from price changes. The engine does not process
+  action cash flows, share adjustments or delisting recoveries.
+- A last-bar order is cancelled; positions already held remain marked at the
+  last observed close. There is no invented terminal liquidation.
+- Buy affordability uses commission on actual filled quantity. A reduced fill
+  still pays the configured minimum. A sale whose proceeds plus cash cannot
+  cover commission is rejected and the position remains held. These fix edge
+  cases where the earlier implementation prorated away minimum fees or allowed
+  negative cash on a collapsed-price sale.
+- Accounting uses floating-point currency and integer shares; tests use explicit
+  tolerances. No shorting, financing, borrow, margin, partial liquidity or taxes.
+- Completed runs, events, snapshots and ledgers remain in memory. This is a daily
+  research engine, not an unbounded streaming storage system. Database manifests
+  detect accidental corruption; they are not signatures against malicious edits.
+
+## Validation and references
+
+The original offline tests remain in CI, alongside the golden fixture, event
+reduction, replay, constraints, rollback, missing/stale-bar, fee and comparison
+tests. Benchmarking follows these correctness tests and has no machine-specific
+speed threshold. Installation requires dependency downloads once; all demos,
+replay, tests and benchmarks then work without network access or credentials.
+
+Implementation references: [Python 3.12 sqlite3 transactions and parameters](https://docs.python.org/3.12/library/sqlite3.html),
+[SQLite window functions](https://www.sqlite.org/windowfunctions.html),
+[pandas 3.0 DataFrame records](https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.to_dict.html),
+and [NumPy Generator](https://numpy.org/doc/stable/reference/random/generator.html).