LLM agents propose forecasting tasks for a relational database in
RelBench v3 format. Each task is a RelBench task
manifest whose DuckDB SQL computes, at a timestamp t, a label per entity from the window
(t, t + timedelta]. A fixed harness validates each task; a separate agent reviews what the
harness keeps. Humans edit the instructions (generate.md, review.md); agents edit only their
workspace.
$ pixi install
$ pixi run taskgen init rel-f1Then start an agent (e.g. Claude Code) in this repository and prompt:
Read generate.md and let's generate tasks for workspace rel-f1. Target: 2 approved tasks.
Approved tasks load like any RelBench task:
import relbench
dataset = relbench.load_dataset("workspace/rel-f1/dataset")
task = dataset.load_task("driver-inactivity-180d")
train = task.get_table("train")init takes a local folder in RelBench v3 layout (manifest.yaml + db/*.parquet) or any
Hugging Face dataset as org/repo[/subdir] (log in with huggingface-cli login for private repos):
$ pixi run taskgen init /path/to/my-dataset --name mydb
$ pixi run taskgen init some-org/some-repo/my-dataset --name mydb --val 2020-01-01 --test 2021-01-01--name defaults to the manifest's name; --val/--test are needed only if the manifest has
no cutoffs.
taskgen evaluate runs the checks in order and stops at the first that fails; each reason
starts with its check's name:
| check | rule |
|---|---|
manifest |
a forecast manifest (binary or regression) with the keys generate.md shows |
splits |
RelBench materializes train/val/test; at most MAX_TRAIN_TIMESTAMPS train timestamps |
output |
three columns, no NULLs, existing entities, one row per (timestamp, entity), valid labels |
split sizes |
enough rows and non-majority labels per split |
label window |
rows use only the strict past (< timestamp) and the label window (timestamp, timestamp + timedelta] |
timestamp independence |
a timestamp's rows don't depend on the other timestamps |
novelty |
not a near-duplicate of a kept task |
learnability |
a predictor on RDBLearn features beats constants and the entity's own label history on val + test |
The harness checks mechanics, not meaning, so a fresh reviewer agent then reads each kept
task, its labels and the database (never the generator's notes) and writes approve,
revise or reject to reviews/<task>.json. Only approved tasks count.
Thresholds are in taskgen/config.py, and so is the predictor that learnability fits on RDBLearn's
features of the database before each timestamp (taskgen/featurizer.py): PREDICTOR picks
LightGBM (the default), XGBoost, TabICLv2 or NVIDIA's Kumo Tabular. Each check is a class in
taskgen/checks/; to add one, subclass Check and list it in taskgen/checks/__init__.py.
.
├── generate.md instructions for the agent that proposes tasks
├── review.md instructions for the agent that reviews them
├── taskgen/ the harness (thresholds in config.py, one class per check in checks/)
├── tests/ harness tests on a synthetic database
└── workspace/<name>/ one per database, created by `taskgen init` (gitignored)
├── dataset/ a RelBench v3 dataset
│ ├── manifest.yaml tables, keys and the val/test cutoffs
│ ├── db/ → the source tables (parquet)
+ │ │ ┏━━ generated tasks, in RelBench v3 format ━━━━━━━━━━━━━━━━━━━━━━━━┓
+ │ └──╂─ tasks/<task>/ ┃
+ │ ┃ ├── manifest.yaml the task's SQL and fields ┃
+ │ ┃ ├── {train,val,test}.parquet its labels ┃
+ │ ┃ └── taskgen.json the harness's report ┃
+ │ ┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┛
├── candidates/<task>/ the generator's manifest.yaml, plus evaluate's report.json
├── reviews/<task>.json the reviewer's verdict: approve, revise or reject
├── results.tsv one row per evaluation
└── notes/ scratch/ logs/ rejected/ the generator's working files$ pixi run test # run the tests
$ pixi run fmt # format with ruff
$ pixi run pre-commit install # once per clone: run ruff on every commitpixi.toml pins relbench==3.0.1, fastdfs==1.1, rdblearn==1.1, tabicl==2.2.0 and NVIDIA's
structured-data-models==0.2.1 (Kumo Tabular), with
an override for rdblearn's relbench==2.1.2 pin (only its unused from_relbench helper imports
relbench). PyTorch comes from conda-forge so that it shares one OpenMP runtime with LightGBM and
XGBoost, and the environment fixes PYTHONHASHSEED because TabICL's predictions depend on it.
Only osx-arm64 is locked.
This repository is an improved version of the code used to generate the forecasting tasks in The Join, the pretraining data of RT-J. Tasks follow the RelBench format.
@inproceedings{ranjan2026rtj,
title={{RT-J}: Large-Scale Pretraining of Relational Transformers for Context-Efficient Predictions},
author={Rishabh Ranjan and Vignesh Kothapalli and Harshvardhan Agarwal and Charilaos Kanatsoulis and Roshan Upendra and Tom Palczewski and Carlos Guestrin and Jure Leskovec},
booktitle={The Fortieth Annual Conference on Neural Information Processing Systems},
year={2026}
}
@inproceedings{relbenchv2,
title={RelBench v2: A Large-Scale Benchmark and Repository for Relational Data},
author={Gu, Justin and Ranjan, Rishabh and Kanatsoulis, Charilaos and Tang, Haiming and Jurkovic, Martin and Hudovernik, Valter and Znidar, Mark and Chaturvedi, Pranshu and Shroff, Parth and Li, Fengyu and Leskovec, Jure},
booktitle={3rd Workshop on Navigating and Addressing Data Problems for Foundation Models (DATA-FM) at ICLR 2026},
year={2026},
note={arXiv:2602.12606 [cs.LG]},
url={https://arxiv.org/abs/2602.12606}
}Apache-2.0, © 2026 stanford-star.