Benchmark results and public-data reproductions

[initial-d]

This is the public thread for sharing benchmark results, public-data reproductions, and setup feedback for ml-quant-trading.

What would be most useful

• CPU/GPU benchmark results for tensor factor primitives.
• Public-data reproductions using yfinance, Baostock, or another accessible data source.
• Factor-engine edge cases around missing data, masks, halts, limit-up / limit-down events, and cross-sectional ranking.
• Notebook or setup issues that make the project harder to try.

Useful links

• Repository: https://github.com/initial-d/ml-quant-trading
• v0.1.0 release: https://github.com/initial-d/ml-quant-trading/releases/tag/v0.1.0
• Paper: https://arxiv.org/abs/2507.07107
• Benchmark issue: Call for benchmarks and public-data reproductions #12
• Community outreach notes: https://github.com/initial-d/ml-quant-trading/blob/main/docs/community_outreach.md

Suggested benchmark report shape

Hardware:
OS:
Python:
PyTorch:
Command:
Panel size:
Device:
Runtime:
Notes:

This project is research software, not financial advice or a live trading recommendation. Please avoid posting proprietary datasets, secrets, or broker/account information.

[initial-d]

Update: the first maintainer CPU benchmark is now on the benchmark board:

https://github.com/initial-d/ml-quant-trading/blob/main/docs/benchmark_board.md

Current baseline: MacBook Air / Apple M5 / macOS / PyTorch 2.8.0 / CPU-only. More useful comparisons: CUDA GPU, Linux CPU, larger panels, and public-data reproduction notes.

[initial-d]

Update: a public-data mini reproduction note is now available:

https://github.com/initial-d/ml-quant-trading/blob/main/docs/public_data_mini_reproduction.md

It uses yfinance, 10 liquid US equities, a six-factor subset, and one-day forward rank IC. More useful next reports: larger public universes, ETFs, A-share public-provider runs, and backtest variants with explicit costs.

[initial-d]

Maintainer update: added a larger public-data validation benchmark in 9cc65cd.

New docs:

• Public-data validation benchmark: https://github.com/initial-d/ml-quant-trading/blob/main/docs/public_data_validation.md
• Reality Check and Validation Status: https://github.com/initial-d/ml-quant-trading/blob/main/docs/reality_check.md

What changed:

• Added scripts/public_data_validation.py with yfinance presets for 100 US large caps, ETF-50, and mixed stock/ETF universes.
• Added walk-forward validation with 504-day train / 63-day test / 63-day step defaults.
• Added transaction cost + slippage reporting, turnover, drawdown, gross/net metrics, and active metrics versus equal weight.
• Added baseline comparisons: equal weight, 20-day momentum, Alpha101 mean score, MLP on Alpha101 features, Transformer on Alpha101 features.
• Recorded a 100-stock yfinance reference run from 2021-01-01 to 2025-01-01.

Important result from the maintainer run: equal weight was the strongest net baseline in this public-data reference run. That is useful negative evidence and a reminder that this benchmark is a validation diagnostic, not a claim of deployable alpha.

CI is green on Python 3.9, 3.10, and 3.11: https://github.com/initial-d/ml-quant-trading/actions/runs/28733020426

Community benchmark runs on different universes, GPUs, and data providers are very welcome.

[initial-d]

Maintainer update: validation reports are now easier to share.

Commit: 4b39b7b

python scripts/public_data_validation.py ... now writes:

• summary.md for human-readable results
• summary.csv for spreadsheets
• summary.json for future aggregation scripts
• metadata.json with command, environment, CUDA, data coverage, and config
• submission.md as a copy-ready GitHub issue body

I also added a dedicated Public-data validation report issue template so community runs can include the same command, environment, coverage, and result fields.

CI is green on Python 3.9, 3.10, and 3.11: https://github.com/initial-d/ml-quant-trading/actions/runs/28733879832

[initial-d]

Maintainer update: added validation report aggregation.

Commit: 507fae1

New helper:

python scripts/aggregate_validation_reports.py artifacts/public_data_validation

It scans one or many summary.json files from public-data validation runs and writes a compact Markdown/CSV leaderboard with source, preset, date range, panel size, strategy, return, Sharpe, drawdown, turnover, coverage, and environment fields.

This should make future community reports easier to compare without treating the table as a trading-performance claim.

CI is green on Python 3.9, 3.10, and 3.11: https://github.com/initial-d/ml-quant-trading/actions/runs/28734048384

[initial-d]

Maintainer update: added a validation report audit gate.

Commit: dd2c151

New helper:

python scripts/audit_validation_report.py artifacts/public_data_validation/summary.json

The audit checks missing metadata, missing result rows, low data coverage, tickers with no data, missing equal-weight baseline, non-finite metrics, negative cost settings, unusual turnover/drawdown, and non-positive final equity.

This is meant to improve reproducibility hygiene for community reports. It is not a claim that any strategy is profitable.

CI is green on Python 3.9, 3.10, and 3.11: https://github.com/initial-d/ml-quant-trading/actions/runs/28734249377

[initial-d]

Small update for reproducibility reports: the public-data validation harness now supports effective-cost sensitivity grids.

Example:

python scripts/public_data_validation.py \
  --source yfinance \
  --preset us-large-100 \
  --cost-grid-bps 0,7,15,30

This writes cost_sensitivity.md, cost_sensitivity.csv, and cost_sensitivity.json, and also embeds the rows in summary.json under cost_sensitivity. The intent is to make reports easier to compare when turnover-heavy strategies only work under one narrow cost/slippage assumption.

CI is passing on Python 3.9, 3.10, and 3.11 for commit 3549ce9.

[initial-d]

Maintainer update: v0.2.0 is now released.

Release: https://github.com/initial-d/ml-quant-trading/releases/tag/v0.2.0
Validation digest: https://github.com/initial-d/ml-quant-trading/blob/main/docs/validation_digest_20260720.md

This release collects the public validation surface added after v0.1.0:

• synthetic validation baseline
• Baostock A-share validation report
• yfinance HTTP 429 troubleshooting and smoke-test workflow
• contributor pairing path for validation and benchmark reports
• English factor handbook and Chinese README summaries

The next most useful community reports are still CUDA GPU benchmarks, Linux CPU benchmarks, and public-data reruns with clear data provenance. Negative results and provider failures are welcome if the command, environment, and failure mode are documented.

[initial-d]

Maintainer update: current discovery surface and benchmark ask.

The project now has a few stable public entry points beyond the repository itself:

• arXiv paper: https://arxiv.org/abs/2507.07107
• Hugging Face Papers page: https://huggingface.co/papers/2507.07107
• Awesome Quant listing under Factor Analysis: https://github.com/wilsonfreitas/awesome-quant#factor-analysis
• Awesome AI Trading Research listing under B1 Factor Investing: https://github.com/ohselab/awesome-ai-trading-research/blob/main/papers.md#b1-factor-investing-23
• project site: https://initial-d.github.io/ml-quant-trading/

GitHub traffic over the current rolling 14-day window is also showing real discovery rather than only maintainer activity:

• 1,206 views / 246 unique visitors
• 465 clones / 249 unique cloners
• visible referrers include GitHub, Google, Bing, arXiv, ChatGPT, Zhihu, Baidu, and DuckDuckGo

The most useful next contributions are still evidence-producing reports rather than feature requests:

1. CUDA GPU benchmark reports for tensor factor primitives.
2. Linux CPU benchmark reports, especially on ordinary non-workstation hardware.
3. Public-data reruns with clear source, date range, universe, costs, and provider failure notes.
4. Factor-engine edge cases around masks, missing values, limit-up / limit-down, halts, and cross-sectional ranking.

Negative results are welcome. Provider failures are welcome. Slow hardware reports are welcome. The goal is to make the public validation surface more honest and easier to compare, not to curate only flattering numbers.

Useful starting points:

• Start Here: https://github.com/initial-d/ml-quant-trading/blob/main/docs/start_here.md
• Research Card: https://github.com/initial-d/ml-quant-trading/blob/main/docs/research_card.md
• Public-data validation docs: https://github.com/initial-d/ml-quant-trading/blob/main/docs/public_data_validation.md
• Benchmark issue: Collect community CPU/GPU benchmark results #7
• Pairing issue: Pairing invitation: co-author a public-data validation or benchmark PR #22

[initial-d]

Maintainer update: I just fixed the Colab/Baostock demo bootstrap path so it clones the canonical repository URL directly:

https://colab.research.google.com/github/initial-d/ml-quant-trading/blob/main/demo_baostock.ipynb

If you try the notebook from a fresh Colab runtime, I would still appreciate short reproduction notes here: OS/runtime, whether Baostock login works, command/notebook section reached, and any dependency or data-access failure. Public-data failures are useful too as long as the report includes the blocker rather than an empty benchmark result.

[initial-d]

Maintainer update: I did another small outreach pass focused on useful technical threads rather than general promotion.

The themes were all directly related to this project's validation boundary:

• suspended/missing-price handling in A-share backtests
• rebalance-calendar semantics
• public-data provenance and yfinance/EDGAR edge cases
• Polars/dataframe migration benchmark contracts
• SPX/options data-contract feasibility
• replay-safe structure/factor attribution
• research/paper/live boundaries for AI quant systems

If you arrive here from one of those threads, the most useful contribution is still a small reproduction note: platform, data source, command/notebook path, whether the run completed, and any data-access or rate-limit blocker. Failed public-data runs are useful too when the blocker is explicit.

[initial-d]

Maintainer update: I ran a new-project outreach pass today, deliberately avoiding projects already contacted in the previous rounds.

This pass focused on adjacent validation problems rather than broad promotion:

• mode contracts for backtest / simulation / live execution
• fixture-vs-real-data provenance and deployed-output drift
• process handoff provenance for live/optimization systems
• risk-aware lending allocation and auto-compounding cost sensitivity
• A-share and portfolio repos with no healthy issue entry point were checked but not spammed

If you are arriving from one of those threads, the best contribution here is still a concise reproduction note: data source, command/notebook path, environment, output artifact, and any blocker such as missing data, rate limits, or dependency issues.

[initial-d]

Maintainer update: another small healthy outreach pass, this time on fresh projects only.

The useful themes were:

• multiple-comparison accounting for autonomous research engines
• Qlib-vs-custom-engine reconciliation for A-share backtests
• regime-classifier recall improvements with protection for existing crisis detection

I am still trying to route new visitors toward reproducible evidence rather than stars. If you test the repo, please leave the command/notebook path, data source, environment, and any blocker so the result can be compared instead of just admired.

[initial-d]

Maintainer update — v0.2.3 validation dashboard is now live.

Recent changes since the last update:

• Added the AkShare CSI 300 daily 213-factor public-data validation report.
• Promoted the latest maintained snapshot into a validation dashboard.
• Added cost-aware reporting for gross/net return, turnover, cost drag, and equal-weight comparison.
• Added an equity_curves.csv export path in the full pipeline so future runs can generate daily equity-curve charts.
• Added an AkShare panel coverage gate: low-coverage public-data runs now fail closed instead of producing misleading validation artifacts.

Current useful reproduction target:

python scripts/akshare_csi300_full_pipeline.py \
  --factor-set all \
  --max-tickers 300 \
  --start 2021-01-01 \
  --end 2025-01-01 \
  --rebalance-step 1 \
  --cost-grid-bps 0,7,15,30

I am especially interested in independent public-data reruns that can produce a clean equity_curves.csv with high panel coverage. If AkShare is unstable from your network, please report the blocker rather than forcing a partial-data result.

Dashboard: https://github.com/initial-d/ml-quant-trading/blob/main/docs/validation_dashboard.md
Release: https://github.com/initial-d/ml-quant-trading/releases/tag/v0.2.3