Question about test results

[evg3nin]

Hi! Great work!

But, in my opinion, there is a problem in the current code:

$ python train.py --config configs/experiments/itransformer_hybrid_origin.py
2026-04-24 20:13:40 | Experiment : itransformer_hybrid
2026-04-24 20:13:40 | Device     : cpu
2026-04-24 20:13:40 | Backbone   : itransformer
2026-04-24 20:13:40 | Loss       : hybrid
2026-04-24 20:13:40 | Output     : output/itransformer_hybrid
2026-04-24 20:13:40 | PolicyNetwork created — backbone=itransformer  params=52.4K
2026-04-24 20:13:40 | ====================================================
2026-04-24 20:13:40 |   SANITY CHECKS
2026-04-24 20:13:40 | ====================================================
2026-04-24 20:13:40 |   PASS  gradient_flow         all params receive gradient
2026-04-24 20:13:45 |   PASS  long_bias             mean_position=+0.1503  expected_sign=+
2026-04-24 20:13:50 |   PASS  short_bias            mean_position=-0.1933  expected_sign=-
2026-04-24 20:13:50 |   ALL PASSED
2026-04-24 20:13:50 | ====================================================
2026-04-24 20:13:50 | Building dataset [diffquant_f1ceafda6404]
ts shape: (2629440,)
ts[:5]: [1609459200000 1609459260000 1609459320000 1609459380000 1609459440000]
2026-04-24 20:13:50 | Source: btcusdt_1min_2021_2025.npz  bars=2,629,440  2021-01-01 00:00 → 2025-12-31 23:59
2026-04-24 20:13:51 | Aggregated to 30-min: 87,648 bars
ts shape: (87648,)
ts[:5]: [1609461 1609462 1609464 1609466 1609468]
2026-04-24 20:13:51 | Features: shape=(87647, 6)  channels=['open', 'high', 'low', 'close', 'volume', 'rolling_vol']
2026-04-24 20:13:51 | Slicing: accepted=0  skipped_gap=0  skipped_align=87,528
2026-04-24 20:13:51 | Samples: 0  stride=24
2026-04-24 20:13:51 |   train       0 samples  [2024-01-01 → 2025-03-31]
2026-04-24 20:13:51 |   val         0 samples  [2025-03-31 → 2025-06-30]
2026-04-24 20:13:51 |   test        0 samples  [2025-06-30 → 2025-09-30]
2026-04-24 20:13:51 |   backtest    0 samples  [2025-09-30 → 2025-12-31]
2026-04-24 20:13:51 | Cached → data_cache/diffquant_f1ceafda6404.npz
2026-04-24 20:13:51 | Dataset  — train: 0 samples | val raw: 0 bars
2026-04-24 20:13:52 | PolicyNetwork created — backbone=itransformer  params=52.4K
2026-04-24 20:13:52 | PolicyNetwork: 52,354 params | backbone=itransformer | loss=hybrid | n_features=6
Traceback (most recent call last):
  File "/home/ec2-user/diffquant/train.py", line 133, in <module>
    main()
  File "/home/ec2-user/diffquant/train.py", line 126, in main
    trainer.train(
  File "/home/ec2-user/diffquant/training/trainer.py", line 263, in train
    loader = DataLoader(
             ^^^^^^^^^^^
  File "/home/ec2-user/.local/lib/python3.12/site-packages/torch/utils/data/dataloader.py", line 394, in __init__
    sampler = RandomSampler(dataset, generator=generator)  # type: ignore[arg-type]
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ec2-user/.local/lib/python3.12/site-packages/torch/utils/data/sampler.py", line 149, in __init__
    raise ValueError(
ValueError: num_samples should be a positive integer value, but got num_samples=0

which was resolved after I made the following fix:

diff --git a/data/aggregator.py b/data/aggregator.py
-    ts_out = (agg.index.view(np.int64) // 1_000_000).astype(np.int64)
+    ts_out = (agg.index.view(np.int64)).astype(np.int64)

Unfortunately, I was unable to achieve the same results as in your article: https://habr.com/en/articles/1022254/

Is this the correct approach to reproduce the results:

1. Search for model hyperparameters and edit configs/experiments/itransformer_hybrid.py to set the obtained parameters
   python optimize.py --config configs/experiments/itransformer_hybrid.py --trials 50

2. Train the model
   python train.py --config configs/experiments/itransformer_hybrid.py

3. Optimize thresholds and edit configs/experiments/itransformer_hybrid.py to set the obtained parameters
   python optimize_thresholds.py --config configs/experiments/itransformer_hybrid.py --trials 50 --objective sharpe

4. Run test and backtest

python evaluate.py --config configs/experiments/itransformer_hybrid.py --mode test

python evaluate.py --config configs/experiments/itransformer_hybrid.py --mode backtest

[YuriyKolesnikov]

Hi! Thanks for the detailed report.
I tested the pipeline again on my side and it works correctly in my environment.
The issue you observed is most likely related to timestamp handling differences across pandas/numpy versions. In your log, aggregated timestamps became too small:

[1609461, 1609462, ...]

while they should remain Unix milliseconds:

[1609461000000, ...]

A robust fix is to replace timestamp extraction in data/aggregator.py with explicit UTC millisecond conversion:

idx = agg.index
if getattr(idx, "tz", None) is not None:
    idx = idx.tz_convert("UTC").tz_localize(None)
ts_out = idx.to_numpy(dtype="datetime64[ms]").astype(np.int64)

Also, after a failed build, clear the cache before rerunning. Otherwise the pipeline may load the previously cached broken dataset:

rm -rf data_cache/

Please share your pandas and numpy versions. This will help make the pipeline more robust across environments.

[evg3nin]

my version:

>>> print(pd.__version__)  # Версия pandas
3.0.1
>>> print(np.__version__)  # Версия numpy
2.4.2

idx = agg.index
if getattr(idx, "tz", None) is not None:
    idx = idx.tz_convert("UTC").tz_localize(None)
ts_out = idx.to_numpy(dtype="datetime64[ms]").astype(np.int64)

yes, it worked for me. Thanks you!

my results after train

2026-04-28 18:04:37 | Experiment : itransformer_hybrid
2026-04-28 18:04:37 | Checkpoint : output/itransformer_hybrid/models/best.pth
2026-04-28 18:04:37 | Mode       : both
2026-04-28 18:04:38 | PolicyNetwork created — backbone=itransformer  params=52.4K
2026-04-28 18:04:38 | Best val Sharpe from training: +1.4000
2026-04-28 18:04:38 | Cache hit [diffquant_f1ceafda6404]
2026-04-28 18:04:39 | Running test evaluation...
2026-04-28 18:04:41 |
================================================================
  TEST REPORT  |  2025-07-02 → 2025-09-29
================================================================

  [Performance]
  :      total commission =    +0.15%
  :      avg commission   =  +0.0085%
  :      max bar loss     =   -0.390%
  :      max bar profit   =   +0.184%
  :      total return     =    -1.55%
  :      exp day return   =   -0.017%
  :      max drawdown     =    +1.80%

  [Risk Metrics]
  :      sharpe  (ann)    =   -2.025
  :      sortino (ann)    =   -1.608
  :      calmar           =   -0.861

  [Time]
  :      total days       =       90
  :      profit days      =   17  (18.9%)
  :      active days      =   44  (48.9%)

  [Position]
  :      flat fraction    =    0.586
  :      long fraction    =    0.068
  :      short fraction   =    0.346
  :      mean |position|  =    0.051
  :      turnover / bar   =   0.0005
  :      rebalances       =       18

  [Directional Accuracy]
  :      long  correct    =  140/ 295  (47.5%)
  :      short correct    =  761/1494  (50.9%)
  :      direction acc    =    50.36%
  :      correct avg ret  =  -0.0103%
  :      incorrect avg    =  +0.0089%

================================================================
2026-04-28 18:04:41 | Daily log → output/itransformer_hybrid/test_daily.log
2026-04-28 18:04:41 | Rebalance log → output/itransformer_hybrid/test_rebalances.log  (18 events)
2026-04-28 18:04:42 | Saved → output/itransformer_hybrid/plots/test_equity.png
2026-04-28 18:04:42 | Saved → output/itransformer_hybrid/plots/test_positions.png
2026-04-28 18:04:42 | [test] Sharpe=-2.025  Return=-1.55%  MaxDD=1.80%  Rebalances=18
2026-04-28 18:04:42 | Running backtest evaluation...
2026-04-28 18:04:45 |
================================================================
  BACKTEST REPORT  |  2025-10-02 → 2025-12-30
================================================================

  [Performance]
  :      total commission =    +0.22%
  :      avg commission   =  +0.0103%
  :      max bar loss     =   -0.404%
  :      max bar profit   =   +0.298%
  :      total return     =    +0.07%
  :      exp day return   =   +0.001%
  :      max drawdown     =    +1.81%

  [Risk Metrics]
  :      sharpe  (ann)    =    0.083
  :      sortino (ann)    =    0.071
  :      calmar           =    0.040

  [Time]
  :      total days       =       90
  :      profit days      =   23  (25.6%)
  :      active days      =   46  (51.1%)

  [Position]
  :      flat fraction    =    0.599
  :      long fraction    =    0.001
  :      short fraction   =    0.400
  :      mean |position|  =    0.046
  :      turnover / bar   =   0.0007
  :      rebalances       =       21

  [Directional Accuracy]
  :      long  correct    =    1/   6  (16.7%)
  :      short correct    =  866/1728  (50.1%)
  :      direction acc    =    50.00%
  :      correct avg ret  =  -0.0275%
  :      incorrect avg    =  +0.0279%

================================================================
2026-04-28 18:04:45 | Daily log → output/itransformer_hybrid/backtest_daily.log
2026-04-28 18:04:45 | Rebalance log → output/itransformer_hybrid/backtest_rebalances.log  (21 events)
2026-04-28 18:04:46 | Saved → output/itransformer_hybrid/plots/backtest_equity.png
2026-04-28 18:04:46 | Saved → output/itransformer_hybrid/plots/backtest_positions.png
2026-04-28 18:04:46 | [backtest] Sharpe=+0.083  Return=+0.07%  MaxDD=1.81%  Rebalances=21

results after thresholds optimization:

python optimize_thresholds.py --config configs/experiments/itransformer_hybrid.py --trials 100 --objective sharpe
2026-04-28 18:04:57 | ============================================================
2026-04-28 18:04:57 |   THRESHOLD OPTIMISATION
2026-04-28 18:04:57 |   Experiment  : itransformer_hybrid
2026-04-28 18:04:57 |   Checkpoint  : output/itransformer_hybrid/models/best.pth
2026-04-28 18:04:57 |   Objective   : sharpe
2026-04-28 18:04:57 |   Trials      : 100
2026-04-28 18:04:57 |   Split       : val  (test/backtest untouched)
2026-04-28 18:04:57 | ============================================================
2026-04-28 18:04:57 | PolicyNetwork created — backbone=itransformer  params=52.4K
2026-04-28 18:04:57 | Model loaded — 52,354 params | best_val_sharpe_during_training=+1.4000
2026-04-28 18:04:58 | Cache hit [diffquant_f1ceafda6404]
2026-04-28 18:04:58 | Val split loaded — 4,368 bars  [2025-03-31 → 2025-06-29]
2026-04-28 18:04:58 | Starting threshold search (100 trials)...
Best trial: 36. Best value: -0.274218:  83%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████                                 | 83/100 [01:39<00:36,  2.15s/it]


Best trial: 36. Best value: -0.274218: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 100/100 [02:03<00:00,  1.23s/it]


python evaluate.py --config configs/experiments/itransformer_hybrid.py
2026-04-28 18:07:36 | Experiment : itransformer_hybrid
2026-04-28 18:07:36 | Checkpoint : output/itransformer_hybrid/models/best.pth
2026-04-28 18:07:36 | Mode       : both
2026-04-28 18:07:37 | PolicyNetwork created — backbone=itransformer  params=52.4K
2026-04-28 18:07:37 | Best val Sharpe from training: +1.4000
2026-04-28 18:07:37 | Cache hit [diffquant_f1ceafda6404]
2026-04-28 18:07:37 | Running test evaluation...
2026-04-28 18:07:39 |
================================================================
  TEST REPORT  |  2025-07-02 → 2025-09-29
================================================================

  [Performance]
  :      total commission =    +0.05%
  :      avg commission   =  +0.0095%
  :      max bar loss     =   -0.042%
  :      max bar profit   =   +0.035%
  :      total return     =    -0.28%
  :      exp day return   =   -0.003%
  :      max drawdown     =    +0.28%

  [Risk Metrics]
  :      sharpe  (ann)    =   -3.553
  :      sortino (ann)    =   -0.589
  :      calmar           =   -1.000

  [Time]
  :      total days       =       90
  :      profit days      =    0  (0.0%)
  :      active days      =    4  (4.4%)

  [Position]
  :      flat fraction    =    0.993
  :      long fraction    =    0.000
  :      short fraction   =    0.007
  :      mean |position|  =    0.001
  :      turnover / bar   =   0.0002
  :      rebalances       =        5

  [Directional Accuracy]
  :      long  correct    =    0/   0  (0.0%)
  :      short correct    =   18/  31  (58.1%)
  :      direction acc    =    58.06%
  :      correct avg ret  =  -0.0230%
  :      incorrect avg    =  +0.0139%

================================================================
2026-04-28 18:07:39 | Daily log → output/itransformer_hybrid/test_daily.log
2026-04-28 18:07:39 | Rebalance log → output/itransformer_hybrid/test_rebalances.log  (5 events)
2026-04-28 18:07:40 | Saved → output/itransformer_hybrid/plots/test_equity.png
2026-04-28 18:07:41 | Saved → output/itransformer_hybrid/plots/test_positions.png
2026-04-28 18:07:41 | [test] Sharpe=-3.553  Return=-0.28%  MaxDD=0.28%  Rebalances=5
2026-04-28 18:07:41 | Running backtest evaluation...
2026-04-28 18:07:43 |
================================================================
  BACKTEST REPORT  |  2025-10-02 → 2025-12-30
================================================================

  [Performance]
  :      total commission =    +0.09%
  :      avg commission   =  +0.0095%
  :      max bar loss     =   -0.106%
  :      max bar profit   =   +0.149%
  :      total return     =    +0.10%
  :      exp day return   =   +0.001%
  :      max drawdown     =    +0.27%

  [Risk Metrics]
  :      sharpe  (ann)    =    0.519
  :      sortino (ann)    =    0.080
  :      calmar           =    0.357

  [Time]
  :      total days       =       90
  :      profit days      =    3  (3.3%)
  :      active days      =    6  (6.7%)

  [Position]
  :      flat fraction    =    0.992
  :      long fraction    =    0.000
  :      short fraction   =    0.008
  :      mean |position|  =    0.001
  :      turnover / bar   =   0.0003
  :      rebalances       =       10

  [Directional Accuracy]
  :      long  correct    =    0/   0  (0.0%)
  :      short correct    =   15/  34  (44.1%)
  :      direction acc    =    44.12%
  :      correct avg ret  =  -0.0400%
  :      incorrect avg    =  +0.0417%

================================================================

Please tell me which version pandas/numpy you are using?

[YuriyKolesnikov]

Thanks for the detailed logs and for testing this carefully.
I just re-ran the full pipeline on my side and reproduced the original numbers exactly.
My log:

2026-04-28 19:41:33 | Experiment : itransformer_hybrid
2026-04-28 19:41:33 | Device     : cpu
2026-04-28 19:41:33 | Backbone   : itransformer
2026-04-28 19:41:33 | Loss       : hybrid
2026-04-28 19:41:33 | Output     : output/itransformer_hybrid
2026-04-28 19:41:33 | Sanity checks skipped.
2026-04-28 19:41:33 | Cache hit [diffquant_f1ceafda6404]
2026-04-28 19:41:33 | Dataset  — train: 910 samples | val raw: 4,368 bars
2026-04-28 19:41:34 | PolicyNetwork created — backbone=itransformer  params=52.4K
2026-04-28 19:41:34 | PolicyNetwork: 52,354 params | backbone=itransformer | loss=hybrid | n_features=6
2026-04-28 19:41:34 | Training: itransformer_hybrid | epochs=30 | batch=64 | train_samples=910
2026-04-28 19:42:08 | [   2/30]  loss=+0.1503  tr_sharpe=-0.1010  |  val_sharpe=-6.4914  val_ret=-14.12%  dd=15.629  flat=0.016  turn=0.0335  ★ best
2026-04-28 19:42:38 | [   4/30]  loss=+0.0678  tr_sharpe=-0.0432  |  val_sharpe=-2.4507  val_ret=-2.84%  dd=4.224  flat=0.143  turn=0.0003  ★ best
2026-04-28 19:43:12 | [   6/30]  loss=+0.0551  tr_sharpe=-0.0323  |  val_sharpe=-2.6344  val_ret=-2.72%  dd=3.661  flat=0.348  turn=0.0002
2026-04-28 19:43:41 | [   8/30]  loss=+0.0468  tr_sharpe=-0.0193  |  val_sharpe=-0.6439  val_ret=-0.39%  dd=1.120  flat=0.775  turn=0.0003  ★ best
2026-04-28 19:44:09 | [  10/30]  loss=+0.0401  tr_sharpe=-0.0118  |  val_sharpe=-0.7165  val_ret=-1.11%  dd=2.285  flat=0.568  turn=0.0019
2026-04-28 19:44:43 | [  12/30]  loss=+0.0335  tr_sharpe=-0.0028  |  val_sharpe=+0.4573  val_ret=+0.94%  dd=3.314  flat=0.177  turn=0.0035  ★ best
2026-04-28 19:45:17 | [  14/30]  loss=+0.0213  tr_sharpe=+0.0085  |  val_sharpe=+0.3038  val_ret=+0.80%  dd=4.301  flat=0.240  turn=0.0079
2026-04-28 19:45:45 | [  16/30]  loss=+0.0135  tr_sharpe=+0.0155  |  val_sharpe=+0.1345  val_ret=+0.30%  dd=6.118  flat=0.079  turn=0.0063
2026-04-28 19:46:13 | [  18/30]  loss=+0.0004  tr_sharpe=+0.0269  |  val_sharpe=+1.0079  val_ret=+4.08%  dd=6.467  flat=0.205  turn=0.0106  ★ best
2026-04-28 19:46:43 | [  20/30]  loss=-0.0035  tr_sharpe=+0.0315  |  val_sharpe=+1.2131  val_ret=+5.17%  dd=4.413  flat=0.154  turn=0.0101  ★ best
2026-04-28 19:47:11 | [  22/30]  loss=-0.0041  tr_sharpe=+0.0343  |  val_sharpe=+0.8277  val_ret=+3.54%  dd=8.042  flat=0.211  turn=0.0133
2026-04-28 19:47:40 | [  24/30]  loss=-0.0014  tr_sharpe=+0.0367  |  val_sharpe=+0.6180  val_ret=+2.56%  dd=5.159  flat=0.183  turn=0.0108
2026-04-28 19:48:09 | [  26/30]  loss=-0.0132  tr_sharpe=+0.0410  |  val_sharpe=+0.9540  val_ret=+4.31%  dd=7.110  flat=0.139  turn=0.0134
2026-04-28 19:48:38 | [  28/30]  loss=-0.0092  tr_sharpe=+0.0415  |  val_sharpe=+1.2096  val_ret=+5.55%  dd=6.513  flat=0.138  turn=0.0136
2026-04-28 19:49:08 | [  30/30]  loss=-0.0083  tr_sharpe=+0.0412  |  val_sharpe=+1.2541  val_ret=+5.75%  dd=7.333  flat=0.132  turn=0.0135  ★ best
2026-04-28 19:49:08 | Saved → output/itransformer_hybrid/plots/train_loss.png
2026-04-28 19:49:09 | Saved → output/itransformer_hybrid/plots/val_sharpe.png
2026-04-28 19:49:09 | Saved → output/itransformer_hybrid/plots/val_sortino.png
2026-04-28 19:49:09 | Saved → output/itransformer_hybrid/plots/val_max_drawdown.png
2026-04-28 19:49:10 | Saved → output/itransformer_hybrid/plots/val_turnover.png
2026-04-28 19:49:10 | Saved → output/itransformer_hybrid/plots/val_flat_frac.png
2026-04-28 19:49:10 | Training complete. Best val Sharpe: +1.2541  → output/itransformer_hybrid/models/best.pth
2026-04-28 19:51:29 | Experiment : itransformer_hybrid
2026-04-28 19:51:29 | Checkpoint : output/itransformer_hybrid/models/best.pth
2026-04-28 19:51:29 | Mode       : both
2026-04-28 19:51:30 | PolicyNetwork created — backbone=itransformer  params=52.4K
2026-04-28 19:51:30 | Best val Sharpe from training: +1.2541
2026-04-28 19:51:30 | Cache hit [diffquant_f1ceafda6404]
2026-04-28 19:51:30 | Running test evaluation...
2026-04-28 19:51:37 | 
================================================================
  TEST REPORT  |  2025-07-02 → 2025-09-29
================================================================

  [Performance]
  :      total commission =    +2.50%
  :      avg commission   =  +0.0316%
  :      max bar loss     =   -1.387%
  :      max bar profit   =   +1.290%
  :      total return     =    +8.22%
  :      exp day return   =   +0.091%
  :      max drawdown     =    +6.10%

  [Risk Metrics]
  :      sharpe  (ann)    =    1.735
  :      sortino (ann)    =    2.173
  :      calmar           =    1.346

  [Time]
  :      total days       =       90
  :      profit days      =   46  (51.1%)
  :      active days      =   83  (92.2%)

  [Position]
  :      flat fraction    =    0.158
  :      long fraction    =    0.669
  :      short fraction   =    0.173
  :      mean |position|  =    0.445
  :      turnover / bar   =   0.0083
  :      rebalances       =       79

  [Directional Accuracy]
  :      long  correct    = 1448/2888  (50.1%)
  :      short correct    =  393/ 747  (52.6%)
  :      direction acc    =    50.65%
  :      correct avg ret  =  +0.0656%
  :      incorrect avg    =  -0.0613%

================================================================
2026-04-28 19:51:37 | Daily log → output/itransformer_hybrid/test_daily.log
2026-04-28 19:51:37 | Rebalance log → output/itransformer_hybrid/test_rebalances.log  (79 events)
2026-04-28 19:51:39 | Saved → output/itransformer_hybrid/plots/test_equity.png
2026-04-28 19:51:40 | Saved → output/itransformer_hybrid/plots/test_positions.png
2026-04-28 19:51:40 | [test] Sharpe=+1.735  Return=+8.22%  MaxDD=6.10%  Rebalances=79
2026-04-28 19:51:40 | Running backtest evaluation...
2026-04-28 19:51:47 | 
================================================================
  BACKTEST REPORT  |  2025-10-02 → 2025-12-30
================================================================

  [Performance]
  :      total commission =    +2.60%
  :      avg commission   =  +0.0342%
  :      max bar loss     =   -1.464%
  :      max bar profit   =   +1.585%
  :      total return     =    +6.91%
  :      exp day return   =   +0.080%
  :      max drawdown     =    +7.91%

  [Risk Metrics]
  :      sharpe  (ann)    =    1.152
  :      sortino (ann)    =    1.250
  :      calmar           =    0.874

  [Time]
  :      total days       =       90
  :      profit days      =   43  (47.8%)
  :      active days      =   76  (84.4%)

  [Position]
  :      flat fraction    =    0.258
  :      long fraction    =    0.533
  :      short fraction   =    0.209
  :      mean |position|  =    0.357
  :      turnover / bar   =   0.0086
  :      rebalances       =       76

  [Directional Accuracy]
  :      long  correct    = 1179/2301  (51.2%)
  :      short correct    =  441/ 903  (48.8%)
  :      direction acc    =    50.56%
  :      correct avg ret  =  +0.0753%
  :      incorrect avg    =  -0.0708%

================================================================
2026-04-28 19:51:47 | Daily log → output/itransformer_hybrid/backtest_daily.log
2026-04-28 19:51:47 | Rebalance log → output/itransformer_hybrid/backtest_rebalances.log  (76 events)
2026-04-28 19:51:49 | Saved → output/itransformer_hybrid/plots/backtest_equity.png
2026-04-28 19:51:49 | Saved → output/itransformer_hybrid/plots/backtest_positions.png
2026-04-28 19:51:49 | [backtest] Sharpe=+1.152  Return=+6.91%  MaxDD=7.91%  Rebalances=76

My environment for reference:

torch        2.0.0
numpy        1.26.4
pandas       2.3.3
pydantic     2.13.3
matplotlib   3.10.9
optuna       4.8.0

Looking at your logs, the training itself worked well (val Sharpe +1.40). The divergence in final results is most likely caused by optimize_thresholds.py. It found thresholds that pushed the model into near-permanent flat (flat=0.993, only 5 rebalances), which collapsed the evaluation metrics.
For clean reproduction, skip threshold optimization and run:

python train.py --config configs/experiments/itransformer_hybrid.py
python evaluate.py --config configs/experiments/itransformer_hybrid.py

Hope this helps, and thanks again for taking the time to test this carefully.

[evg3nin]

I tried to reproduce it with the same versions as you

print(pd.version)
2.3.3
print(np.version)
1.26.4
print(torch.__version__)
2.0.0+cu117

but unfortunately the result is different

2026-04-29 12:04:54 | Experiment : itransformer_hybrid
2026-04-29 12:04:54 | Device     : cpu
2026-04-29 12:04:54 | Backbone   : itransformer
2026-04-29 12:04:54 | Loss       : hybrid
2026-04-29 12:04:54 | Output     : output/itransformer_hybrid
2026-04-29 12:04:54 | PolicyNetwork created — backbone=itransformer  params=52.4K
2026-04-29 12:04:54 | ====================================================
2026-04-29 12:04:54 |   SANITY CHECKS
2026-04-29 12:04:54 | ====================================================
2026-04-29 12:04:54 |   PASS  gradient_flow         all params receive gradient
2026-04-29 12:05:06 |   PASS  long_bias             mean_position=+0.1601  expected_sign=+
2026-04-29 12:05:18 |   PASS  short_bias            mean_position=-0.1836  expected_sign=-
2026-04-29 12:05:18 |   ALL PASSED
2026-04-29 12:05:18 | ====================================================
2026-04-29 12:05:18 | Building dataset [diffquant_f1ceafda6404]
2026-04-29 12:05:19 | Source: btcusdt_1min_2021_2025.npz  bars=2,629,440  2021-01-01 00:00 → 2025-12-31 23:59
2026-04-29 12:05:19 | Aggregated to 30-min: 87,648 bars
2026-04-29 12:05:21 | Features: shape=(87647, 6)  channels=['open', 'high', 'low', 'close', 'volume', 'rolling_vol']
2026-04-29 12:05:21 | Slicing: accepted=3,647  skipped_gap=0  skipped_align=0
2026-04-29 12:05:21 | Samples: 3,647  stride=24
2026-04-29 12:05:21 |   train       910 samples  [2024-01-01 → 2025-03-31]
2026-04-29 12:05:21 |   val         182 samples  [2025-03-31 → 2025-06-30]
2026-04-29 12:05:21 |   test        184 samples  [2025-06-30 → 2025-09-30]
2026-04-29 12:05:21 |   backtest    181 samples  [2025-09-30 → 2025-12-31]
2026-04-29 12:05:21 | Cached → data_cache/diffquant_f1ceafda6404.npz
2026-04-29 12:05:21 | Dataset  — train: 910 samples | val raw: 4,368 bars
2026-04-29 12:05:21 | PolicyNetwork created — backbone=itransformer  params=52.4K
2026-04-29 12:05:21 | PolicyNetwork: 52,354 params | backbone=itransformer | loss=hybrid | n_features=6
2026-04-29 12:05:21 | Training: itransformer_hybrid | epochs=30 | batch=64 | train_samples=910
2026-04-29 12:05:41 | [   2/30]  loss=+0.1401  tr_sharpe=-0.0833  |  val_sharpe=-6.0778  val_ret=-12.28%  dd=13.792  flat=0.016  turn=0.0274  ★ best
2026-04-29 12:06:00 | [   4/30]  loss=+0.0693  tr_sharpe=-0.0439  |  val_sharpe=-2.1556  val_ret=-2.49%  dd=4.086  flat=0.107  turn=0.0002  ★ best
2026-04-29 12:06:19 | [   6/30]  loss=+0.0522  tr_sharpe=-0.0294  |  val_sharpe=-2.7023  val_ret=-3.05%  dd=4.185  flat=0.194  turn=0.0002
2026-04-29 12:06:39 | [   8/30]  loss=+0.0482  tr_sharpe=-0.0235  |  val_sharpe=-0.9408  val_ret=-1.03%  dd=2.442  flat=0.373  turn=0.0003  ★ best
2026-04-29 12:06:58 | [  10/30]  loss=+0.0407  tr_sharpe=-0.0168  |  val_sharpe=-2.3841  val_ret=-2.85%  dd=4.058  flat=0.195  turn=0.0007
2026-04-29 12:07:18 | [  12/30]  loss=+0.0339  tr_sharpe=-0.0055  |  val_sharpe=+0.6942  val_ret=+1.32%  dd=2.790  flat=0.258  turn=0.0077  ★ best
2026-04-29 12:07:38 | [  14/30]  loss=+0.0252  tr_sharpe=+0.0099  |  val_sharpe=-0.7508  val_ret=-1.65%  dd=3.466  flat=0.149  turn=0.0064
2026-04-29 12:07:59 | [  16/30]  loss=+0.0075  tr_sharpe=+0.0248  |  val_sharpe=-0.7596  val_ret=-2.38%  dd=4.973  flat=0.146  turn=0.0131
2026-04-29 12:08:24 | [  18/30]  loss=+0.0029  tr_sharpe=+0.0307  |  val_sharpe=-1.2132  val_ret=-4.21%  dd=7.189  flat=0.218  turn=0.0156
2026-04-29 12:08:43 | [  20/30]  loss=-0.0028  tr_sharpe=+0.0373  |  val_sharpe=-2.2357  val_ret=-6.37%  dd=8.768  flat=0.217  turn=0.0156
2026-04-29 12:09:07 | [  22/30]  loss=-0.0021  tr_sharpe=+0.0451  |  val_sharpe=-2.6846  val_ret=-9.01%  dd=11.333  flat=0.186  turn=0.0166
2026-04-29 12:09:26 | [  24/30]  loss=-0.0150  tr_sharpe=+0.0456  |  val_sharpe=-2.3949  val_ret=-8.67%  dd=11.951  flat=0.189  turn=0.0181
2026-04-29 12:09:48 | [  26/30]  loss=-0.0081  tr_sharpe=+0.0487  |  val_sharpe=-2.7628  val_ret=-10.09%  dd=12.551  flat=0.188  turn=0.0182
2026-04-29 12:10:10 | [  28/30]  loss=-0.0109  tr_sharpe=+0.0507  |  val_sharpe=-2.7444  val_ret=-9.94%  dd=12.587  flat=0.185  turn=0.0183
2026-04-29 12:10:31 | [  30/30]  loss=-0.0105  tr_sharpe=+0.0509  |  val_sharpe=-2.6672  val_ret=-9.83%  dd=12.617  flat=0.169  turn=0.0192
2026-04-29 12:10:32 | Saved → output/itransformer_hybrid/plots/train_loss.png
2026-04-29 12:10:32 | Saved → output/itransformer_hybrid/plots/val_sharpe.png
2026-04-29 12:10:32 | Saved → output/itransformer_hybrid/plots/val_sortino.png
2026-04-29 12:10:33 | Saved → output/itransformer_hybrid/plots/val_max_drawdown.png
2026-04-29 12:10:33 | Saved → output/itransformer_hybrid/plots/val_turnover.png
2026-04-29 12:10:33 | Saved → output/itransformer_hybrid/plots/val_flat_frac.png
2026-04-29 12:10:33 | Training complete. Best val Sharpe: +0.6942  → output/itransformer_hybrid/models/best.pth
2026-04-29 12:10:58 | Experiment : itransformer_hybrid
2026-04-29 12:10:58 | Checkpoint : output/itransformer_hybrid/models/best.pth
2026-04-29 12:10:58 | Mode       : both
2026-04-29 12:10:59 | PolicyNetwork created — backbone=itransformer  params=52.4K
2026-04-29 12:10:59 | Best val Sharpe from training: +0.6942
2026-04-29 12:10:59 | Cache hit [diffquant_f1ceafda6404]
2026-04-29 12:11:00 | Running test evaluation...
2026-04-29 12:11:06 | 
================================================================
  TEST REPORT  |  2025-07-02 → 2025-09-29
================================================================

  [Performance]
  :      total commission =    +2.69%
  :      avg commission   =  +0.0153%
  :      max bar loss     =   -0.713%
  :      max bar profit   =   +0.473%
  :      total return     =    -5.32%
  :      exp day return   =   -0.060%
  :      max drawdown     =    +5.32%

  [Risk Metrics]
  :      sharpe  (ann)    =   -3.970
  :      sortino (ann)    =   -4.081
  :      calmar           =   -1.000

  [Time]
  :      total days       =       90
  :      profit days      =   32  (35.6%)
  :      active days      =   87  (96.7%)

  [Position]
  :      flat fraction    =    0.284
  :      long fraction    =    0.183
  :      short fraction   =    0.532
  :      mean |position|  =    0.113
  :      turnover / bar   =   0.0089
  :      rebalances       =      176

  [Directional Accuracy]
  :      long  correct    =  370/ 791  (46.8%)
  :      short correct    = 1196/2299  (52.0%)
  :      direction acc    =    50.68%
  :      correct avg ret  =  -0.0059%
  :      incorrect avg    =  +0.0043%

================================================================
2026-04-29 12:11:06 | Daily log → output/itransformer_hybrid/test_daily.log
2026-04-29 12:11:06 | Rebalance log → output/itransformer_hybrid/test_rebalances.log  (176 events)
2026-04-29 12:11:08 | Saved → output/itransformer_hybrid/plots/test_equity.png
2026-04-29 12:11:09 | Saved → output/itransformer_hybrid/plots/test_positions.png
2026-04-29 12:11:09 | [test] Sharpe=-3.970  Return=-5.32%  MaxDD=5.32%  Rebalances=176
2026-04-29 12:11:09 | Running backtest evaluation...
2026-04-29 12:11:16 | 
================================================================
  BACKTEST REPORT  |  2025-10-02 → 2025-12-30
================================================================

  [Performance]
  :      total commission =    +1.84%
  :      avg commission   =  +0.0149%
  :      max bar loss     =   -0.613%
  :      max bar profit   =   +0.751%
  :      total return     =    +3.89%
  :      exp day return   =   +0.043%
  :      max drawdown     =    +2.13%

  [Risk Metrics]
  :      sharpe  (ann)    =    1.822
  :      sortino (ann)    =    2.144
  :      calmar           =    1.831

  [Time]
  :      total days       =       90
  :      profit days      =   45  (50.0%)
  :      active days      =   86  (95.6%)

  [Position]
  :      flat fraction    =    0.232
  :      long fraction    =    0.119
  :      short fraction   =    0.648
  :      mean |position|  =    0.114
  :      turnover / bar   =   0.0061
  :      rebalances       =      123

  [Directional Accuracy]
  :      long  correct    =  262/ 516  (50.8%)
  :      short correct    = 1390/2799  (49.7%)
  :      direction acc    =    49.83%
  :      correct avg ret  =  -0.0159%
  :      incorrect avg    =  +0.0193%

================================================================
2026-04-29 12:11:16 | Daily log → output/itransformer_hybrid/backtest_daily.log
2026-04-29 12:11:16 | Rebalance log → output/itransformer_hybrid/backtest_rebalances.log  (123 events)
2026-04-29 12:11:17 | Saved → output/itransformer_hybrid/plots/backtest_equity.png
2026-04-29 12:11:18 | Saved → output/itransformer_hybrid/plots/backtest_positions.png
2026-04-29 12:11:18 | [backtest] Sharpe=+1.822  Return=+3.89%  MaxDD=2.13%  Rebalances=123

I ran these commands:

python train.py --config configs/experiments/itransformer_hybrid.py
python evaluate.py --config configs/experiments/itransformer_hybrid.py

[YuriyKolesnikov]

Thanks for checking the versions.

Even if the library versions look the same, there may still be backend-level differences: CPU vs CUDA build, kernel implementation, floating-point order, or other PyTorch internals. DiffQuant is quite sensitive to these small numerical differences because the final result depends on validation checkpoint selection, continuous positions, thresholding, turnover, and walk-forward PnL.

As the next diagnostic step, let’s try rolling back to a conservative CPU-only PyTorch build and check whether the training trajectory itself starts to converge similarly. Before comparing final test/backtest numbers, please first run only the training stage and compare the validation curve / best validation Sharpe.

Please remove the current PyTorch installation and install torch 1.13.1+cpu:

pip uninstall -y torch torchvision torchaudio

pip install torch==1.13.1+cpu torchvision==0.14.1+cpu torchaudio==0.13.1+cpu --extra-index-url https://download.pytorch.org/whl/cpu

Then clear the cached dataset and previous outputs:

rm -rf data_cache/
rm -rf output/itransformer_hybrid/

After that, rerun the same config:

python train.py --config configs/experiments/itransformer_hybrid.py

[evg3nin]

after reinstall pytorch:

>>> print(torch.__version__)
1.13.1+cpu

the result is the same as before

2026-04-29 15:11:04 | Experiment : itransformer_hybrid
2026-04-29 15:11:04 | Device     : cpu
2026-04-29 15:11:04 | Backbone   : itransformer
2026-04-29 15:11:04 | Loss       : hybrid
2026-04-29 15:11:04 | Output     : output/itransformer_hybrid
2026-04-29 15:11:04 | PolicyNetwork created — backbone=itransformer  params=52.4K
2026-04-29 15:11:04 | ====================================================
2026-04-29 15:11:04 |   SANITY CHECKS
2026-04-29 15:11:04 | ====================================================
2026-04-29 15:11:05 |   PASS  gradient_flow         all params receive gradient
2026-04-29 15:11:14 |   PASS  long_bias             mean_position=+0.1601  expected_sign=+
2026-04-29 15:11:23 |   PASS  short_bias            mean_position=-0.1836  expected_sign=-
2026-04-29 15:11:23 |   ALL PASSED
2026-04-29 15:11:23 | ====================================================
2026-04-29 15:11:23 | Building dataset [diffquant_f1ceafda6404]
2026-04-29 15:11:24 | Source: btcusdt_1min_2021_2025.npz  bars=2,629,440  2021-01-01 00:00 → 2025-12-31 23:59
2026-04-29 15:11:24 | Aggregated to 30-min: 87,648 bars
2026-04-29 15:11:26 | Features: shape=(87647, 6)  channels=['open', 'high', 'low', 'close', 'volume', 'rolling_vol']
2026-04-29 15:11:26 | Slicing: accepted=3,647  skipped_gap=0  skipped_align=0
2026-04-29 15:11:26 | Samples: 3,647  stride=24
2026-04-29 15:11:26 |   train       910 samples  [2024-01-01 → 2025-03-31]
2026-04-29 15:11:26 |   val         182 samples  [2025-03-31 → 2025-06-30]
2026-04-29 15:11:26 |   test        184 samples  [2025-06-30 → 2025-09-30]
2026-04-29 15:11:26 |   backtest    181 samples  [2025-09-30 → 2025-12-31]
2026-04-29 15:11:26 | Cached → data_cache/diffquant_f1ceafda6404.npz
2026-04-29 15:11:26 | Dataset  — train: 910 samples | val raw: 4,368 bars
2026-04-29 15:11:26 | PolicyNetwork created — backbone=itransformer  params=52.4K
2026-04-29 15:11:26 | PolicyNetwork: 52,354 params | backbone=itransformer | loss=hybrid | n_features=6
2026-04-29 15:11:26 | Training: itransformer_hybrid | epochs=30 | batch=64 | train_samples=910
2026-04-29 15:11:43 | [   2/30]  loss=+0.1401  tr_sharpe=-0.0833  |  val_sharpe=-6.0778  val_ret=-12.28%  dd=13.792  flat=0.016  turn=0.0274  ★ best
2026-04-29 15:11:59 | [   4/30]  loss=+0.0693  tr_sharpe=-0.0439  |  val_sharpe=-2.1556  val_ret=-2.49%  dd=4.086  flat=0.107  turn=0.0002  ★ best
2026-04-29 15:12:15 | [   6/30]  loss=+0.0522  tr_sharpe=-0.0294  |  val_sharpe=-2.7023  val_ret=-3.05%  dd=4.185  flat=0.194  turn=0.0002
2026-04-29 15:12:33 | [   8/30]  loss=+0.0482  tr_sharpe=-0.0235  |  val_sharpe=-0.9408  val_ret=-1.03%  dd=2.442  flat=0.373  turn=0.0003  ★ best
2026-04-29 15:12:48 | [  10/30]  loss=+0.0407  tr_sharpe=-0.0168  |  val_sharpe=-2.3841  val_ret=-2.85%  dd=4.058  flat=0.195  turn=0.0007
2026-04-29 15:13:04 | [  12/30]  loss=+0.0339  tr_sharpe=-0.0055  |  val_sharpe=+0.6942  val_ret=+1.32%  dd=2.790  flat=0.258  turn=0.0077  ★ best
2026-04-29 15:13:21 | [  14/30]  loss=+0.0252  tr_sharpe=+0.0099  |  val_sharpe=-0.7508  val_ret=-1.65%  dd=3.466  flat=0.149  turn=0.0064
2026-04-29 15:13:37 | [  16/30]  loss=+0.0075  tr_sharpe=+0.0248  |  val_sharpe=-0.7596  val_ret=-2.38%  dd=4.973  flat=0.146  turn=0.0131
2026-04-29 15:13:54 | [  18/30]  loss=+0.0029  tr_sharpe=+0.0307  |  val_sharpe=-1.2132  val_ret=-4.21%  dd=7.189  flat=0.218  turn=0.0156
2026-04-29 15:14:11 | [  20/30]  loss=-0.0028  tr_sharpe=+0.0373  |  val_sharpe=-2.2358  val_ret=-6.37%  dd=8.769  flat=0.217  turn=0.0156
2026-04-29 15:14:29 | [  22/30]  loss=-0.0021  tr_sharpe=+0.0451  |  val_sharpe=-2.6847  val_ret=-9.01%  dd=11.333  flat=0.186  turn=0.0166
2026-04-29 15:14:45 | [  24/30]  loss=-0.0150  tr_sharpe=+0.0456  |  val_sharpe=-2.3952  val_ret=-8.67%  dd=11.951  flat=0.189  turn=0.0181
2026-04-29 15:15:03 | [  26/30]  loss=-0.0081  tr_sharpe=+0.0487  |  val_sharpe=-2.7630  val_ret=-10.09%  dd=12.551  flat=0.188  turn=0.0182
2026-04-29 15:15:20 | [  28/30]  loss=-0.0109  tr_sharpe=+0.0507  |  val_sharpe=-2.7448  val_ret=-9.94%  dd=12.587  flat=0.185  turn=0.0183
2026-04-29 15:15:37 | [  30/30]  loss=-0.0105  tr_sharpe=+0.0509  |  val_sharpe=-2.6671  val_ret=-9.83%  dd=12.617  flat=0.169  turn=0.0192
2026-04-29 15:15:38 | Saved → output/itransformer_hybrid/plots/train_loss.png
2026-04-29 15:15:38 | Saved → output/itransformer_hybrid/plots/val_sharpe.png
2026-04-29 15:15:38 | Saved → output/itransformer_hybrid/plots/val_sortino.png
2026-04-29 15:15:39 | Saved → output/itransformer_hybrid/plots/val_max_drawdown.png
2026-04-29 15:15:39 | Saved → output/itransformer_hybrid/plots/val_turnover.png
2026-04-29 15:15:39 | Saved → output/itransformer_hybrid/plots/val_flat_frac.png
2026-04-29 15:15:39 | Training complete. Best val Sharpe: +0.6942  → output/itransformer_hybrid/models/best.pth

[YuriyKolesnikov]

Thanks for checking.

At this point, I suggest testing from a completely clean conda environment. I also re-tested the project in a fresh environment, and the pipeline worked correctly there.

conda create -n new_env python=3.10.6 -y
conda activate new_env

pip install "numpy<2" "pandas>=2.0" "pydantic>=2.0.0" "matplotlib>=3.8.0" "optuna>=3.6.0"

pip install torch==1.13.1+cpu torchvision==0.14.1+cpu torchaudio==0.13.1+cpu \
  --extra-index-url https://download.pytorch.org/whl/cpu

pip show torch numpy pandas pydantic matplotlib optuna | grep -E "^(Name|Version)"

Then clear cached artifacts and rerun training:

rm -rf data_cache/
rm -rf output/itransformer_hybrid/

python train.py --config configs/experiments/itransformer_hybrid.py

[evg3nin]

Unfortunately, nothing has changed. I'm attaching the full action log.

output.log

[YuriyKolesnikov]

Thanks for the detailed testing.
To separate training variance from evaluation reproducibility, please try evaluating the reference checkpoint used for the article.
I will provide best.pth.
best.zip

Please place it here:

output/itransformer_hybrid/models/best.pth

Then run:

python evaluate.py --config configs/experiments/itransformer_hybrid.py

I also recommend trying more conservative inference thresholds:

cfg.eval.enter_long_thr  = 0.2
cfg.eval.enter_short_thr = 0.2

In practice, stronger entry thresholds can filter weak signals and often produce cleaner walk-forward behaviour.

If the evaluation matches the article result from the same checkpoint, then the dataset, feature pipeline, and evaluation path are consistent. Any remaining difference is related to the training trajectory itself, which can be sensitive in this type of differentiable trading setup: initialization, batch order, CPU/backend numerical differences, and validation checkpoint selection may lead to different trained policies.

DiffQuant is an experimental research project, and I am continuing to improve training stability, reproducibility, and future portfolio-level extensions.

Thanks again for helping test the project.

[evg3nin]

with this model https://github.com/user-attachments/files/27234150/best.zip my results:

(new_env) ec2-user@ip-172-31-96-4:~/diffquant$ unzip best.zip
Archive:  best.zip
  inflating: best.pth
(new_env) ec2-user@ip-172-31-96-4:~/diffquant$ mv best.pth output/itransformer_hybrid/models/best.pth
(new_env) ec2-user@ip-172-31-96-4:~/diffquant$ python evaluate.py --config configs/experiments/itransformer_hybrid.py
2026-04-30 13:16:48 | Experiment : itransformer_hybrid
2026-04-30 13:16:48 | Checkpoint : output/itransformer_hybrid/models/best.pth
2026-04-30 13:16:48 | Mode       : both
2026-04-30 13:16:49 | PolicyNetwork created — backbone=itransformer  params=52.4K
2026-04-30 13:16:49 | Best val Sharpe from training: +1.2541
2026-04-30 13:16:50 | Cache hit [diffquant_ad649bda54d9]
2026-04-30 13:16:50 | Running test evaluation...
2026-04-30 13:16:58 |
================================================================
  TEST REPORT  |  2025-07-02 → 2025-09-29
================================================================

  [Performance]
  :      total commission =    +4.90%
  :      avg commission   =  +0.0308%
  :      max bar loss     =   -2.114%
  :      max bar profit   =   +1.450%
  :      total return     =    +0.02%
  :      exp day return   =   +0.008%
  :      max drawdown     =   +17.52%

  [Risk Metrics]
  :      sharpe  (ann)    =    0.101
  :      sortino (ann)    =    0.114
  :      calmar           =    0.001

  [Time]
  :      total days       =       90
  :      profit days      =   37  (41.1%)
  :      active days      =   86  (95.6%)

  [Position]
  :      flat fraction    =    0.157
  :      long fraction    =    0.412
  :      short fraction   =    0.431
  :      mean |position|  =    0.317
  :      turnover / bar   =   0.0162
  :      rebalances       =      159

  [Directional Accuracy]
  :      long  correct    =  903/1779  (50.8%)
  :      short correct    =  966/1863  (51.9%)
  :      direction acc    =    51.32%
  :      correct avg ret  =  +0.0628%
  :      incorrect avg    =  -0.0631%

================================================================
2026-04-30 13:16:58 | Daily log → output/itransformer_hybrid/test_daily.log
2026-04-30 13:16:58 | Rebalance log → output/itransformer_hybrid/test_rebalances.log  (159 events)
2026-04-30 13:17:00 | Saved → output/itransformer_hybrid/plots/test_equity.png
2026-04-30 13:17:01 | Saved → output/itransformer_hybrid/plots/test_positions.png
2026-04-30 13:17:01 | [test] Sharpe=+0.101  Return=+0.02%  MaxDD=17.52%  Rebalances=159
2026-04-30 13:17:01 | Running backtest evaluation...
2026-04-30 13:17:02 |
================================================================
  BACKTEST REPORT  |  2025-10-02 → 2025-10-11
================================================================

  [Performance]
  :      total commission =    +0.62%
  :      avg commission   =  +0.0325%
  :      max bar loss     =   -3.589%
  :      max bar profit   =   +4.953%
  :      total return     =    -5.32%
  :      exp day return   =   -0.518%
  :      max drawdown     =   +10.27%

  [Risk Metrics]
  :      sharpe  (ann)    =   -3.549
  :      sortino (ann)    =   -3.480
  :      calmar           =   -0.518

  [Time]
  :      total days       =       10
  :      profit days      =    5  (50.0%)
  :      active days      =   10  (100.0%)

  [Position]
  :      flat fraction    =    0.235
  :      long fraction    =    0.473
  :      short fraction   =    0.292
  :      mean |position|  =    0.261
  :      turnover / bar   =   0.0194
  :      rebalances       =       19

  [Directional Accuracy]
  :      long  correct    =  111/ 214  (51.9%)
  :      short correct    =   66/ 133  (49.6%)
  :      direction acc    =    51.01%
  :      correct avg ret  =  +0.1033%
  :      incorrect avg    =  -0.1345%

================================================================
2026-04-30 13:17:02 | Daily log → output/itransformer_hybrid/backtest_daily.log
2026-04-30 13:17:02 | Rebalance log → output/itransformer_hybrid/backtest_rebalances.log  (19 events)
2026-04-30 13:17:03 | Saved → output/itransformer_hybrid/plots/backtest_equity.png
2026-04-30 13:17:04 | Saved → output/itransformer_hybrid/plots/backtest_positions.png
2026-04-30 13:17:04 | [backtest] Sharpe=-3.549  Return=-5.32%  MaxDD=10.27%  Rebalances=19

after change thresholds:

(new_env) ec2-user@ip-172-31-96-4:~/diffquant$ cat configs/experiments/itransformer_hybrid.py | grep enter_short_thr
cfg.eval.enter_short_thr        = 0.2  # 0.2
(new_env) ec2-user@ip-172-31-96-4:~/diffquant$ cat configs/experiments/itransformer_hybrid.py | grep enter_long_thr
cfg.eval.enter_long_thr         = 0.2  # 0.2
(new_env) ec2-user@ip-172-31-96-4:~/diffquant$ python evaluate.py --config configs/experiments/itransformer_hybrid.py
2026-04-30 13:18:15 | Experiment : itransformer_hybrid
2026-04-30 13:18:15 | Checkpoint : output/itransformer_hybrid/models/best.pth
2026-04-30 13:18:15 | Mode       : both
2026-04-30 13:18:16 | PolicyNetwork created — backbone=itransformer  params=52.4K
2026-04-30 13:18:16 | Best val Sharpe from training: +1.2541
2026-04-30 13:18:17 | Cache hit [diffquant_ad649bda54d9]
2026-04-30 13:18:17 | Running test evaluation...
2026-04-30 13:18:26 |
================================================================
  TEST REPORT  |  2025-07-02 → 2025-09-29
================================================================

  [Performance]
  :      total commission =    +3.07%
  :      avg commission   =  +0.0370%
  :      max bar loss     =   -2.117%
  :      max bar profit   =   +4.785%
  :      total return     =   +11.80%
  :      exp day return   =   +0.137%
  :      max drawdown     =   +15.06%

  [Risk Metrics]
  :      sharpe  (ann)    =    1.362
  :      sortino (ann)    =    1.701
  :      calmar           =    0.783

  [Time]
  :      total days       =       90
  :      profit days      =   34  (37.8%)
  :      active days      =   80  (88.9%)

  [Position]
  :      flat fraction    =    0.265
  :      long fraction    =    0.585
  :      short fraction   =    0.150
  :      mean |position|  =    0.416
  :      turnover / bar   =   0.0102
  :      rebalances       =       83

  [Directional Accuracy]
  :      long  correct    = 1284/2528  (50.8%)
  :      short correct    =  346/ 648  (53.4%)
  :      direction acc    =    51.32%
  :      correct avg ret  =  +0.1377%
  :      incorrect avg    =  -0.1352%

================================================================
2026-04-30 13:18:26 | Daily log → output/itransformer_hybrid/test_daily.log
2026-04-30 13:18:26 | Rebalance log → output/itransformer_hybrid/test_rebalances.log  (83 events)
2026-04-30 13:18:28 | Saved → output/itransformer_hybrid/plots/test_equity.png
2026-04-30 13:18:29 | Saved → output/itransformer_hybrid/plots/test_positions.png
2026-04-30 13:18:29 | [test] Sharpe=+1.362  Return=+11.80%  MaxDD=15.06%  Rebalances=83
2026-04-30 13:18:29 | Running backtest evaluation...
2026-04-30 13:18:30 |
================================================================
  BACKTEST REPORT  |  2025-10-02 → 2025-10-11
================================================================

  [Performance]
  :      total commission =    +0.47%
  :      avg commission   =  +0.0336%
  :      max bar loss     =   -3.589%
  :      max bar profit   =   +4.953%
  :      total return     =    -5.83%
  :      exp day return   =   -0.572%
  :      max drawdown     =   +10.86%

  [Risk Metrics]
  :      sharpe  (ann)    =   -3.881
  :      sortino (ann)    =   -3.644
  :      calmar           =   -0.536

  [Time]
  :      total days       =       10
  :      profit days      =    5  (50.0%)
  :      active days      =   10  (100.0%)

  [Position]
  :      flat fraction    =    0.308
  :      long fraction    =    0.563
  :      short fraction   =    0.130
  :      mean |position|  =    0.272
  :      turnover / bar   =   0.0148
  :      rebalances       =       14

  [Directional Accuracy]
  :      long  correct    =  132/ 255  (51.8%)
  :      short correct    =   28/  59  (47.5%)
  :      direction acc    =    50.96%
  :      correct avg ret  =  +0.1375%
  :      incorrect avg    =  -0.1769%

================================================================
2026-04-30 13:18:30 | Daily log → output/itransformer_hybrid/backtest_daily.log
2026-04-30 13:18:30 | Rebalance log → output/itransformer_hybrid/backtest_rebalances.log  (14 events)
2026-04-30 13:18:31 | Saved → output/itransformer_hybrid/plots/backtest_equity.png
2026-04-30 13:18:32 | Saved → output/itransformer_hybrid/plots/backtest_positions.png
2026-04-30 13:18:32 | [backtest] Sharpe=-3.881  Return=-5.83%  MaxDD=10.86%  Rebalances=14

maybe there is some problem with the dataset? I got it in the following way:
hf download ResearchRL/diffquant-data --local-dir data_source/ --repo-type dataset

[YuriyKolesnikov]

Thanks, the reference checkpoint seems to be loaded correctly - the reported best validation Sharpe matches the checkpoint.
The remaining mismatch looks like a data/cache issue. In your run the cache hash is different, and the backtest is only 10 days, while the reference run uses the full hold-out period.

Please clean generated artifacts first:

rm -rf data_cache/
rm -rf output/itransformer_hybrid/

Also please use the stronger entry thresholds:

cfg.eval.enter_long_thr  = 0.2
cfg.eval.enter_short_thr = 0.2

Then rebuild from a clean run.

python train.py --config configs/experiments/itransformer_hybrid.py

next - replace the generated best.pth with the reference checkpoint, and run evaluation again.

python evaluate.py --config configs/experiments/itransformer_hybrid.py

Before comparing metrics, please check that the cache hash [diffquant_f1ceafda6404] and test/backtest periods match the reference run.

[evg3nin]

I understood what the problem with mismatch in cache hash.

I had a problem initially with my versions pandas/numpy

>>> print(pd.__version__)  
3.0.1
>>> print(np.__version__) 
2.4.2

2026-04-24 20:13:51 |   train       0 samples  [2024-01-01 → 2025-03-31]
2026-04-24 20:13:51 |   val         0 samples  [2025-03-31 → 2025-06-30]
2026-04-24 20:13:51 |   test        0 samples  [2025-06-30 → 2025-09-30]
2026-04-24 20:13:51 |   backtest    0 samples  [2025-09-30 → 2025-12-31]
2026-04-24 20:13:51 | Cached → data_cache/diffquant_f1ceafda6404.npz
2026-04-24 20:13:51 | Dataset  — train: 0 samples | val raw: 0 bars
2026-04-24 20:13:52 | PolicyNetwork created — backbone=itransformer  params=52.4K
2026-04-24 20:13:52 | PolicyNetwork: 52,354 params | backbone=itransformer | loss=hybrid | n_features=6
Traceback (most recent call last):
  File "/home/ec2-user/diffquant/train.py", line 133, in <module>
    main()
  File "/home/ec2-user/diffquant/train.py", line 126, in main
    trainer.train(
  File "/home/ec2-user/diffquant/training/trainer.py", line 263, in train
    loader = DataLoader(
             ^^^^^^^^^^^
  File "/home/ec2-user/.local/lib/python3.12/site-packages/torch/utils/data/dataloader.py", line 394, in __init__
    sampler = RandomSampler(dataset, generator=generator)  # type: ignore[arg-type]
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ec2-user/.local/lib/python3.12/site-packages/torch/utils/data/sampler.py", line 149, in __init__
    raise ValueError(
ValueError: num_samples should be a positive integer value, but got num_samples=0

which I corrected like this

diff --git a/data/aggregator.py b/data/aggregator.py
-    ts_out = (agg.index.view(np.int64) // 1_000_000).astype(np.int64)
+    ts_out = (agg.index.view(np.int64)).astype(np.int64)

and then like this

idx = agg.index
if getattr(idx, "tz", None) is not None:
    idx = idx.tz_convert("UTC").tz_localize(None)
ts_out = idx.to_numpy(dtype="datetime64[ms]").astype(np.int64)

Now i'm using conda ENV with original code https://github.com/YuriyKolesnikov/diffquant:

conda create -n new_env python=3.10.6 -y
conda activate new_env

pip install "numpy<2" "pandas>=2.0" "pydantic>=2.0.0" "matplotlib>=3.8.0" "optuna>=3.6.0"

pip install torch==1.13.1+cpu torchvision==0.14.1+cpu torchaudio==0.13.1+cpu \
  --extra-index-url https://download.pytorch.org/whl/cpu

and the mismatch in cache hash is no longer observed. But train/evaluate resutls have not improved:

(new_env) ec2-user@ip-172-31-96-4:~/diffquant$ cat configs/experiments/itransformer_hybrid.py | grep enter_long_thr
cfg.eval.enter_long_thr         = 0.2  # 0.2
(new_env) ec2-user@ip-172-31-96-4:~/diffquant$ cat configs/experiments/itransformer_hybrid.py | grep enter_short_thr
cfg.eval.enter_short_thr        = 0.2  # 0.2
(new_env) ec2-user@ip-172-31-96-4:~/diffquant$ python train.py --config configs/experiments/itransformer_hybrid.py
2026-04-30 15:26:45 | Experiment : itransformer_hybrid
2026-04-30 15:26:45 | Device     : cpu
2026-04-30 15:26:45 | Backbone   : itransformer
2026-04-30 15:26:45 | Loss       : hybrid
2026-04-30 15:26:45 | Output     : output/itransformer_hybrid
2026-04-30 15:26:45 | PolicyNetwork created — backbone=itransformer  params=52.4K
2026-04-30 15:26:45 | ====================================================
2026-04-30 15:26:45 |   SANITY CHECKS
2026-04-30 15:26:45 | ====================================================
2026-04-30 15:26:45 |   PASS  gradient_flow         all params receive gradient
2026-04-30 15:26:56 |   PASS  long_bias             mean_position=+0.1601  expected_sign=+
2026-04-30 15:27:08 |   PASS  short_bias            mean_position=-0.1836  expected_sign=-
2026-04-30 15:27:08 |   ALL PASSED
2026-04-30 15:27:08 | ====================================================
2026-04-30 15:27:08 | Building dataset [diffquant_f1ceafda6404]
2026-04-30 15:27:09 | Source: btcusdt_1min_2021_2025.npz  bars=2,629,440  2021-01-01 00:00 → 2025-12-31 23:59
2026-04-30 15:27:09 | Aggregated to 30-min: 87,648 bars
2026-04-30 15:27:11 | Features: shape=(87647, 6)  channels=['open', 'high', 'low', 'close', 'volume', 'rolling_vol']
2026-04-30 15:27:11 | Slicing: accepted=3,647  skipped_gap=0  skipped_align=0
2026-04-30 15:27:11 | Samples: 3,647  stride=24
2026-04-30 15:27:11 |   train       910 samples  [2024-01-01 → 2025-03-31]
2026-04-30 15:27:11 |   val         182 samples  [2025-03-31 → 2025-06-30]
2026-04-30 15:27:11 |   test        184 samples  [2025-06-30 → 2025-09-30]
2026-04-30 15:27:11 |   backtest    181 samples  [2025-09-30 → 2025-12-31]
2026-04-30 15:27:11 | Cached → data_cache/diffquant_f1ceafda6404.npz
2026-04-30 15:27:11 | Dataset  — train: 910 samples | val raw: 4,368 bars
2026-04-30 15:27:12 | PolicyNetwork created — backbone=itransformer  params=52.4K
2026-04-30 15:27:12 | PolicyNetwork: 52,354 params | backbone=itransformer | loss=hybrid | n_features=6
2026-04-30 15:27:12 | Training: itransformer_hybrid | epochs=30 | batch=64 | train_samples=910
[1/30]  loss=+0.1928  sharpe=-0.1649:   3%|██████▌                                                                                                                                                                                              | 1/30 [00:07<03:37,  7.51s/it]/home/ec2-user/miniconda3/envs/new_env/lib/python3.10/site-packages/torch/optim/lr_scheduler.py:163: UserWarning: The epoch parameter in `scheduler.step()` was not necessary and is being deprecated where possible. Please use `scheduler.step()` to step the scheduler. During the deprecation, if epoch is different from None, the closed form is used instead of the new chainable form, where available. Please open an issue if you are unable to replicate your use case: https://github.com/pytorch/pytorch/issues/new/choose.
  warnings.warn(EPOCH_DEPRECATION_WARNING, UserWarning)
[2/30]  loss=+0.1401  sharpe=-0.0833  |  val_sharpe=-2.9214  val_ret=-7.22%  ★ best:   3%|█████                                                                                                                                                 | 1/30 [00:22<03:37,  7.51s/it]2026-04-30 15:27:34 | [   2/30]  loss=+0.1401  tr_sharpe=-0.0833  |  val_sharpe=-2.9214  val_ret=-7.22%  dd=9.751  flat=0.098  turn=0.0058  ★ best
[4/30]  loss=+0.0693  sharpe=-0.0439  |  val_sharpe=+0.0000  val_ret=+0.00%  ★ best:  10%|███████████████                                                                                                                                       | 3/30 [00:41<04:08,  9.22s/it]2026-04-30 15:27:53 | [   4/30]  loss=+0.0693  tr_sharpe=-0.0439  |  val_sharpe=+0.0000  val_ret=+0.00%  dd=0.000  flat=1.000  turn=0.0000  ★ best
[6/30]  loss=+0.0522  sharpe=-0.0294  |  val_sharpe=+0.0000  val_ret=+0.00%:  17%|██████████████████████████▎                                                                                                                                   | 5/30 [01:01<03:51,  9.27s/it]2026-04-30 15:28:13 | [   6/30]  loss=+0.0522  tr_sharpe=-0.0294  |  val_sharpe=+0.0000  val_ret=+0.00%  dd=0.000  flat=1.000  turn=0.0000
[8/30]  loss=+0.0482  sharpe=-0.0235  |  val_sharpe=+0.0000  val_ret=+0.00%:  23%|████████████████████████████████████▊                                                                                                                         | 7/30 [01:26<03:41,  9.61s/it]2026-04-30 15:28:38 | [   8/30]  loss=+0.0482  tr_sharpe=-0.0235  |  val_sharpe=+0.0000  val_ret=+0.00%  dd=0.000  flat=1.000  turn=0.0000
[10/30]  loss=+0.0407  sharpe=-0.0168  |  val_sharpe=+0.0000  val_ret=+0.00%:  30%|███████████████████████████████████████████████                                                                                                              | 9/30 [01:44<03:33, 10.16s/it]2026-04-30 15:28:56 | [  10/30]  loss=+0.0407  tr_sharpe=-0.0168  |  val_sharpe=+0.0000  val_ret=+0.00%  dd=0.000  flat=1.000  turn=0.0000
[12/30]  loss=+0.0339  sharpe=-0.0055  |  val_sharpe=+1.6524  val_ret=+4.03%  ★ best:  37%|██████████████████████████████████████████████████████▎                                                                                             | 11/30 [02:04<02:56,  9.28s/it]2026-04-30 15:29:16 | [  12/30]  loss=+0.0339  tr_sharpe=-0.0055  |  val_sharpe=+1.6524  val_ret=+4.03%  dd=2.253  flat=0.620  turn=0.0044  ★ best
[14/30]  loss=+0.0252  sharpe=+0.0099  |  val_sharpe=+0.3194  val_ret=+0.98%:  43%|███████████████████████████████████████████████████████████████████▌                                                                                        | 13/30 [02:24<02:36,  9.18s/it]2026-04-30 15:29:36 | [  14/30]  loss=+0.0252  tr_sharpe=+0.0099  |  val_sharpe=+0.3194  val_ret=+0.98%  dd=4.808  flat=0.428  turn=0.0038
[16/30]  loss=+0.0075  sharpe=+0.0248  |  val_sharpe=+1.1772  val_ret=+5.09%:  50%|██████████████████████████████████████████████████████████████████████████████                                                                              | 15/30 [02:46<02:25,  9.70s/it]2026-04-30 15:29:58 | [  16/30]  loss=+0.0075  tr_sharpe=+0.0248  |  val_sharpe=+1.1772  val_ret=+5.09%  dd=5.123  flat=0.250  turn=0.0082
[18/30]  loss=+0.0029  sharpe=+0.0307  |  val_sharpe=+1.7310  val_ret=+8.50%  ★ best:  57%|███████████████████████████████████████████████████████████████████████████████████▊                                                                | 17/30 [03:12<02:15, 10.45s/it]2026-04-30 15:30:24 | [  18/30]  loss=+0.0029  tr_sharpe=+0.0307  |  val_sharpe=+1.7310  val_ret=+8.50%  dd=5.986  flat=0.283  turn=0.0111  ★ best
[20/30]  loss=-0.0028  sharpe=+0.0373  |  val_sharpe=-0.4221  val_ret=-1.91%:  63%|██████████████████████████████████████████████████████████████████████████████████████████████████▊                                                         | 19/30 [03:36<02:03, 11.20s/it]2026-04-30 15:30:48 | [  20/30]  loss=-0.0028  tr_sharpe=+0.0373  |  val_sharpe=-0.4221  val_ret=-1.91%  dd=8.647  flat=0.342  turn=0.0115
[22/30]  loss=-0.0021  sharpe=+0.0451  |  val_sharpe=-0.4193  val_ret=-2.20%:  70%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████▏                                              | 21/30 [04:02<01:37, 10.82s/it]2026-04-30 15:31:14 | [  22/30]  loss=-0.0021  tr_sharpe=+0.0451  |  val_sharpe=-0.4193  val_ret=-2.20%  dd=9.470  flat=0.341  turn=0.0132
[24/30]  loss=-0.0150  sharpe=+0.0456  |  val_sharpe=-1.0105  val_ret=-4.74%:  77%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▌                                    | 23/30 [04:27<01:23, 11.98s/it]2026-04-30 15:31:39 | [  24/30]  loss=-0.0150  tr_sharpe=+0.0456  |  val_sharpe=-1.0105  val_ret=-4.74%  dd=9.941  flat=0.292  turn=0.0147
[26/30]  loss=-0.0081  sharpe=+0.0487  |  val_sharpe=-0.8088  val_ret=-3.92%:  83%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████                          | 25/30 [04:53<00:59, 11.84s/it]2026-04-30 15:32:05 | [  26/30]  loss=-0.0081  tr_sharpe=+0.0487  |  val_sharpe=-0.8088  val_ret=-3.92%  dd=9.480  flat=0.285  turn=0.0149
[28/30]  loss=-0.0109  sharpe=+0.0507  |  val_sharpe=-0.9982  val_ret=-4.77%:  90%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▍               | 27/30 [05:15<00:36, 12.05s/it]2026-04-30 15:32:27 | [  28/30]  loss=-0.0109  tr_sharpe=+0.0507  |  val_sharpe=-0.9982  val_ret=-4.77%  dd=9.815  flat=0.281  turn=0.0150
[30/30]  loss=-0.0105  sharpe=+0.0509  |  val_sharpe=-1.1735  val_ret=-5.21%:  97%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▊     | 29/30 [05:38<00:10, 10.85s/it]2026-04-30 15:32:50 | [  30/30]  loss=-0.0105  tr_sharpe=+0.0509  |  val_sharpe=-1.1735  val_ret=-5.21%  dd=11.036  flat=0.272  turn=0.0151
[30/30]  loss=-0.0105  sharpe=+0.0509  |  val_sharpe=-1.1735  val_ret=-5.21%: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 30/30 [05:38<00:00, 11.29s/it]
2026-04-30 15:32:51 | Saved → output/itransformer_hybrid/plots/train_loss.png
2026-04-30 15:32:51 | Saved → output/itransformer_hybrid/plots/val_sharpe.png
2026-04-30 15:32:51 | Saved → output/itransformer_hybrid/plots/val_sortino.png
2026-04-30 15:32:52 | Saved → output/itransformer_hybrid/plots/val_max_drawdown.png
2026-04-30 15:32:52 | Saved → output/itransformer_hybrid/plots/val_turnover.png
2026-04-30 15:32:52 | Saved → output/itransformer_hybrid/plots/val_flat_frac.png
2026-04-30 15:32:52 | Training complete. Best val Sharpe: +1.7310  → output/itransformer_hybrid/models/best.pth
(new_env) ec2-user@ip-172-31-96-4:~/diffquant$ python evaluate.py --config configs/experiments/itransformer_hybrid.py
2026-04-30 15:34:43 | Experiment : itransformer_hybrid
2026-04-30 15:34:43 | Checkpoint : output/itransformer_hybrid/models/best.pth
2026-04-30 15:34:43 | Mode       : both
2026-04-30 15:34:44 | PolicyNetwork created — backbone=itransformer  params=52.4K
2026-04-30 15:34:44 | Best val Sharpe from training: +1.7310
2026-04-30 15:34:44 | Cache hit [diffquant_f1ceafda6404]
2026-04-30 15:34:45 | Running test evaluation...
2026-04-30 15:34:52 |
================================================================
  TEST REPORT  |  2025-07-02 → 2025-09-29
================================================================

  [Performance]
  :      total commission =    +3.59%
  :      avg commission   =  +0.0307%
  :      max bar loss     =   -1.357%
  :      max bar profit   =   +0.848%
  :      total return     =    -4.78%
  :      exp day return   =   -0.053%
  :      max drawdown     =    +7.78%

  [Risk Metrics]
  :      sharpe  (ann)    =   -1.454
  :      sortino (ann)    =   -1.546
  :      calmar           =   -0.614

  [Time]
  :      total days       =       90
  :      profit days      =   35  (38.9%)
  :      active days      =   82  (91.1%)

  [Position]
  :      flat fraction    =    0.299
  :      long fraction    =    0.384
  :      short fraction   =    0.317
  :      mean |position|  =    0.279
  :      turnover / bar   =   0.0119
  :      rebalances       =      117

  [Directional Accuracy]
  :      long  correct    =  820/1658  (49.5%)
  :      short correct    =  705/1369  (51.5%)
  :      direction acc    =    50.38%
  :      correct avg ret  =  +0.0183%
  :      incorrect avg    =  -0.0193%

================================================================
2026-04-30 15:34:52 | Daily log → output/itransformer_hybrid/test_daily.log
2026-04-30 15:34:52 | Rebalance log → output/itransformer_hybrid/test_rebalances.log  (117 events)
2026-04-30 15:34:53 | Saved → output/itransformer_hybrid/plots/test_equity.png
2026-04-30 15:34:54 | Saved → output/itransformer_hybrid/plots/test_positions.png
2026-04-30 15:34:54 | [test] Sharpe=-1.454  Return=-4.78%  MaxDD=7.78%  Rebalances=117
2026-04-30 15:34:54 | Running backtest evaluation...
2026-04-30 15:35:03 |
================================================================
  BACKTEST REPORT  |  2025-10-02 → 2025-12-30
================================================================

  [Performance]
  :      total commission =    +2.86%
  :      avg commission   =  +0.0298%
  :      max bar loss     =   -1.338%
  :      max bar profit   =   +2.152%
  :      total return     =    +0.76%
  :      exp day return   =   +0.013%
  :      max drawdown     =   +12.40%

  [Risk Metrics]
  :      sharpe  (ann)    =    0.212
  :      sortino (ann)    =    0.227
  :      calmar           =    0.061

  [Time]
  :      total days       =       90
  :      profit days      =   41  (45.6%)
  :      active days      =   79  (87.8%)

  [Position]
  :      flat fraction    =    0.280
  :      long fraction    =    0.400
  :      short fraction   =    0.320
  :      mean |position|  =    0.273
  :      turnover / bar   =   0.0095
  :      rebalances       =       96

  [Directional Accuracy]
  :      long  correct    =  872/1728  (50.5%)
  :      short correct    =  673/1383  (48.7%)
  :      direction acc    =    49.66%
  :      correct avg ret  =  +0.0338%
  :      incorrect avg    =  -0.0308%

================================================================
2026-04-30 15:35:03 | Daily log → output/itransformer_hybrid/backtest_daily.log
2026-04-30 15:35:03 | Rebalance log → output/itransformer_hybrid/backtest_rebalances.log  (96 events)
2026-04-30 15:35:05 | Saved → output/itransformer_hybrid/plots/backtest_equity.png
2026-04-30 15:35:05 | Saved → output/itransformer_hybrid/plots/backtest_positions.png
2026-04-30 15:35:05 | [backtest] Sharpe=+0.212  Return=+0.76%  MaxDD=12.40%  Rebalances=96

evaluate result with you best model

(new_env) ec2-user@ip-172-31-96-4:~/diffquant$ rm -r output/itransformer_hybrid; rm -rf data_cache/*
(new_env) ec2-user@ip-172-31-96-4:~/diffquant$ mkdir -p output/itransformer_hybrid/models/
(new_env) ec2-user@ip-172-31-96-4:~/diffquant$ unzip best.zip
Archive:  best.zip
  inflating: best.pth
(new_env) ec2-user@ip-172-31-96-4:~/diffquant$ mv best.pth output/itransformer_hybrid/models/
(new_env) ec2-user@ip-172-31-96-4:~/diffquant$ python evaluate.py --config configs/experiments/itransformer_hybrid.py
2026-04-30 15:48:58 | Experiment : itransformer_hybrid
2026-04-30 15:48:58 | Checkpoint : output/itransformer_hybrid/models/best.pth
2026-04-30 15:48:58 | Mode       : both
2026-04-30 15:48:59 | PolicyNetwork created — backbone=itransformer  params=52.4K
2026-04-30 15:48:59 | Best val Sharpe from training: +1.2541
2026-04-30 15:48:59 | Building dataset [diffquant_f1ceafda6404]
2026-04-30 15:49:00 | Source: btcusdt_1min_2021_2025.npz  bars=2,629,440  2021-01-01 00:00 → 2025-12-31 23:59
2026-04-30 15:49:00 | Aggregated to 30-min: 87,648 bars
2026-04-30 15:49:02 | Features: shape=(87647, 6)  channels=['open', 'high', 'low', 'close', 'volume', 'rolling_vol']
2026-04-30 15:49:02 | Slicing: accepted=3,647  skipped_gap=0  skipped_align=0
2026-04-30 15:49:02 | Samples: 3,647  stride=24
2026-04-30 15:49:02 |   train       910 samples  [2024-01-01 → 2025-03-31]
2026-04-30 15:49:02 |   val         182 samples  [2025-03-31 → 2025-06-30]
2026-04-30 15:49:02 |   test        184 samples  [2025-06-30 → 2025-09-30]
2026-04-30 15:49:02 |   backtest    181 samples  [2025-09-30 → 2025-12-31]
2026-04-30 15:49:02 | Cached → data_cache/diffquant_f1ceafda6404.npz
2026-04-30 15:49:02 | Running test evaluation...
2026-04-30 15:49:10 |
================================================================
  TEST REPORT  |  2025-07-02 → 2025-09-29
================================================================

  [Performance]
  :      total commission =    +4.19%
  :      avg commission   =  +0.0283%
  :      max bar loss     =   -1.332%
  :      max bar profit   =   +1.231%
  :      total return     =    +4.00%
  :      exp day return   =   +0.045%
  :      max drawdown     =    +4.77%

  [Risk Metrics]
  :      sharpe  (ann)    =    1.208
  :      sortino (ann)    =    1.501
  :      calmar           =    0.838

  [Time]
  :      total days       =       90
  :      profit days      =   48  (53.3%)
  :      active days      =   89  (98.9%)

  [Position]
  :      flat fraction    =    0.097
  :      long fraction    =    0.443
  :      short fraction   =    0.460
  :      mean |position|  =    0.292
  :      turnover / bar   =   0.0138
  :      rebalances       =      148

  [Directional Accuracy]
  :      long  correct    =  961/1915  (50.2%)
  :      short correct    =  999/1985  (50.3%)
  :      direction acc    =    50.26%
  :      correct avg ret  =  +0.0279%
  :      incorrect avg    =  -0.0239%

================================================================
2026-04-30 15:49:10 | Daily log → output/itransformer_hybrid/test_daily.log
2026-04-30 15:49:10 | Rebalance log → output/itransformer_hybrid/test_rebalances.log  (148 events)
2026-04-30 15:49:11 | Saved → output/itransformer_hybrid/plots/test_equity.png
2026-04-30 15:49:12 | Saved → output/itransformer_hybrid/plots/test_positions.png
2026-04-30 15:49:12 | [test] Sharpe=+1.208  Return=+4.00%  MaxDD=4.77%  Rebalances=148
2026-04-30 15:49:12 | Running backtest evaluation...
2026-04-30 15:49:19 |
================================================================
  BACKTEST REPORT  |  2025-10-02 → 2025-12-30
================================================================

  [Performance]
  :      total commission =    +3.87%
  :      avg commission   =  +0.0296%
  :      max bar loss     =   -1.466%
  :      max bar profit   =   +1.583%
  :      total return     =    -9.60%
  :      exp day return   =   -0.108%
  :      max drawdown     =   +10.95%

  [Risk Metrics]
  :      sharpe  (ann)    =   -1.942
  :      sortino (ann)    =   -2.097
  :      calmar           =   -0.877

  [Time]
  :      total days       =       90
  :      profit days      =   39  (43.3%)
  :      active days      =   88  (97.8%)

  [Position]
  :      flat fraction    =    0.125
  :      long fraction    =    0.338
  :      short fraction   =    0.537
  :      mean |position|  =    0.256
  :      turnover / bar   =   0.0128
  :      rebalances       =      131

  [Directional Accuracy]
  :      long  correct    =  720/1460  (49.3%)
  :      short correct    = 1173/2321  (50.5%)
  :      direction acc    =    50.07%
  :      correct avg ret  =  +0.0206%
  :      incorrect avg    =  -0.0238%

================================================================
2026-04-30 15:49:19 | Daily log → output/itransformer_hybrid/backtest_daily.log
2026-04-30 15:49:19 | Rebalance log → output/itransformer_hybrid/backtest_rebalances.log  (131 events)
2026-04-30 15:49:21 | Saved → output/itransformer_hybrid/plots/backtest_equity.png
2026-04-30 15:49:21 | Saved → output/itransformer_hybrid/plots/backtest_positions.png
2026-04-30 15:49:21 | [backtest] Sharpe=-1.942  Return=-9.60%  MaxDD=10.95%  Rebalances=131

But if use your best model with parameters

(new_env) ec2-user@ip-172-31-96-4:~/diffquant$ cat configs/experiments/itransformer_hybrid.py | grep enter_long_thr
cfg.eval.enter_long_thr         = 0.1  # 0.2
(new_env) ec2-user@ip-172-31-96-4:~/diffquant$ cat configs/experiments/itransformer_hybrid.py | grep enter_short_thr
cfg.eval.enter_short_thr        = 0.1  # 0.2

then everything is fine:

(new_env) ec2-user@ip-172-31-96-4:~/diffquant$ python evaluate.py --config configs/experiments/itransformer_hybrid.py
2026-04-30 15:50:29 | Experiment : itransformer_hybrid
2026-04-30 15:50:29 | Checkpoint : output/itransformer_hybrid/models/best.pth
2026-04-30 15:50:29 | Mode       : both
2026-04-30 15:50:30 | PolicyNetwork created — backbone=itransformer  params=52.4K
2026-04-30 15:50:30 | Best val Sharpe from training: +1.2541
2026-04-30 15:50:30 | Cache hit [diffquant_f1ceafda6404]
2026-04-30 15:50:30 | Running test evaluation...
2026-04-30 15:50:37 |
================================================================
  TEST REPORT  |  2025-07-02 → 2025-09-29
================================================================

  [Performance]
  :      total commission =    +2.50%
  :      avg commission   =  +0.0316%
  :      max bar loss     =   -1.387%
  :      max bar profit   =   +1.290%
  :      total return     =    +8.22%
  :      exp day return   =   +0.091%
  :      max drawdown     =    +6.10%

  [Risk Metrics]
  :      sharpe  (ann)    =    1.735
  :      sortino (ann)    =    2.173
  :      calmar           =    1.346

  [Time]
  :      total days       =       90
  :      profit days      =   46  (51.1%)
  :      active days      =   83  (92.2%)

  [Position]
  :      flat fraction    =    0.158
  :      long fraction    =    0.669
  :      short fraction   =    0.173
  :      mean |position|  =    0.445
  :      turnover / bar   =   0.0083
  :      rebalances       =       79

  [Directional Accuracy]
  :      long  correct    = 1448/2888  (50.1%)
  :      short correct    =  393/ 747  (52.6%)
  :      direction acc    =    50.65%
  :      correct avg ret  =  +0.0656%
  :      incorrect avg    =  -0.0613%

================================================================
2026-04-30 15:50:37 | Daily log → output/itransformer_hybrid/test_daily.log
2026-04-30 15:50:37 | Rebalance log → output/itransformer_hybrid/test_rebalances.log  (79 events)
2026-04-30 15:50:39 | Saved → output/itransformer_hybrid/plots/test_equity.png
2026-04-30 15:50:40 | Saved → output/itransformer_hybrid/plots/test_positions.png
2026-04-30 15:50:40 | [test] Sharpe=+1.735  Return=+8.22%  MaxDD=6.10%  Rebalances=79
2026-04-30 15:50:40 | Running backtest evaluation...
2026-04-30 15:50:48 |
================================================================
  BACKTEST REPORT  |  2025-10-02 → 2025-12-30
================================================================

  [Performance]
  :      total commission =    +2.60%
  :      avg commission   =  +0.0342%
  :      max bar loss     =   -1.464%
  :      max bar profit   =   +1.585%
  :      total return     =    +6.91%
  :      exp day return   =   +0.080%
  :      max drawdown     =    +7.91%

  [Risk Metrics]
  :      sharpe  (ann)    =    1.152
  :      sortino (ann)    =    1.250
  :      calmar           =    0.874

  [Time]
  :      total days       =       90
  :      profit days      =   43  (47.8%)
  :      active days      =   76  (84.4%)

  [Position]
  :      flat fraction    =    0.258
  :      long fraction    =    0.533
  :      short fraction   =    0.209
  :      mean |position|  =    0.357
  :      turnover / bar   =   0.0086
  :      rebalances       =       76

  [Directional Accuracy]
  :      long  correct    = 1179/2301  (51.2%)
  :      short correct    =  441/ 903  (48.8%)
  :      direction acc    =    50.56%
  :      correct avg ret  =  +0.0753%
  :      incorrect avg    =  -0.0708%

================================================================
2026-04-30 15:50:48 | Daily log → output/itransformer_hybrid/backtest_daily.log
2026-04-30 15:50:48 | Rebalance log → output/itransformer_hybrid/backtest_rebalances.log  (76 events)
2026-04-30 15:50:50 | Saved → output/itransformer_hybrid/plots/backtest_equity.png
2026-04-30 15:50:51 | Saved → output/itransformer_hybrid/plots/backtest_positions.png
2026-04-30 15:50:51 | [backtest] Sharpe=+1.152  Return=+6.91%  MaxDD=7.91%  Rebalances=76