Skip to content

About

Agentic task generation for relational databases

Resources

Stars

8 stars

Watchers

0 watching

Forks

Repository files navigation

Agentic task generation for relational databases

Python 3.12 RelBench v3 License: Apache-2.0

LLM agents propose forecasting tasks for a relational database in RelBench v3 format. Each task is a RelBench task manifest whose DuckDB SQL computes, at a timestamp t, a label per entity from the window (t, t + timedelta]. A fixed harness validates each task; a separate agent reviews what the harness keeps. Humans edit the instructions (generate.md, review.md); agents edit only their workspace.

Quick start

$ pixi install
$ pixi run taskgen init rel-f1

Then start an agent (e.g. Claude Code) in this repository and prompt:

Read generate.md and let's generate tasks for workspace rel-f1. Target: 2 approved tasks.

Approved tasks load like any RelBench task:

import relbench

dataset = relbench.load_dataset("workspace/rel-f1/dataset")
task = dataset.load_task("driver-inactivity-180d")
train = task.get_table("train")

Your own dataset

init takes a local folder in RelBench v3 layout (manifest.yaml + db/*.parquet) or any Hugging Face dataset as org/repo[/subdir] (log in with huggingface-cli login for private repos):

$ pixi run taskgen init /path/to/my-dataset --name mydb
$ pixi run taskgen init some-org/some-repo/my-dataset --name mydb --val 2020-01-01 --test 2021-01-01

--name defaults to the manifest's name; --val/--test are needed only if the manifest has no cutoffs.

How it works

The generator writes a candidate; taskgen evaluate runs eight checks and discards or keeps it; a fresh reviewer approves, rejects or sends it back for revision.

taskgen evaluate runs the checks in order and stops at the first that fails; each reason starts with its check's name:

check rule
manifest a forecast manifest (binary or regression) with the keys generate.md shows
splits RelBench materializes train/val/test; at most MAX_TRAIN_TIMESTAMPS train timestamps
output three columns, no NULLs, existing entities, one row per (timestamp, entity), valid labels
split sizes enough rows and non-majority labels per split
label window rows use only the strict past (< timestamp) and the label window (timestamp, timestamp + timedelta]
timestamp independence a timestamp's rows don't depend on the other timestamps
novelty not a near-duplicate of a kept task
learnability a predictor on RDBLearn features beats constants and the entity's own label history on val + test

The harness checks mechanics, not meaning, so a fresh reviewer agent then reads each kept task, its labels and the database (never the generator's notes) and writes approve, revise or reject to reviews/<task>.json. Only approved tasks count.

Thresholds are in taskgen/config.py, and so is the predictor that learnability fits on RDBLearn's features of the database before each timestamp (taskgen/featurizer.py): PREDICTOR picks LightGBM (the default), XGBoost, TabICLv2 or NVIDIA's Kumo Tabular. Each check is a class in taskgen/checks/; to add one, subclass Check and list it in taskgen/checks/__init__.py.

Layout

 .
 ├── generate.md               instructions for the agent that proposes tasks
 ├── review.md                 instructions for the agent that reviews them
 ├── taskgen/                  the harness (thresholds in config.py, one class per check in checks/)
 ├── tests/                    harness tests on a synthetic database
 └── workspace/<name>/         one per database, created by `taskgen init` (gitignored)
     ├── dataset/              a RelBench v3 dataset
     │   ├── manifest.yaml     tables, keys and the val/test cutoffs
     │   ├── db/               → the source tables (parquet)
+    │   │  ┏━━ generated tasks, in RelBench v3 format ━━━━━━━━━━━━━━━━━━━━━━━━┓
+    │   └──╂─ tasks/<task>/                                                   ┃
+    │      ┃    ├── manifest.yaml              the task's SQL and fields      ┃
+    │      ┃    ├── {train,val,test}.parquet   its labels                     ┃
+    │      ┃    └── taskgen.json               the harness's report           ┃
+    │      ┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┛
     ├── candidates/<task>/    the generator's manifest.yaml, plus evaluate's report.json
     ├── reviews/<task>.json   the reviewer's verdict: approve, revise or reject
     ├── results.tsv           one row per evaluation
     └── notes/ scratch/ logs/ rejected/    the generator's working files

Development

$ pixi run test                  # run the tests
$ pixi run fmt                   # format with ruff
$ pixi run pre-commit install    # once per clone: run ruff on every commit

pixi.toml pins relbench==3.0.1, fastdfs==1.1, rdblearn==1.1, tabicl==2.2.0 and NVIDIA's structured-data-models==0.2.1 (Kumo Tabular), with an override for rdblearn's relbench==2.1.2 pin (only its unused from_relbench helper imports relbench). PyTorch comes from conda-forge so that it shares one OpenMP runtime with LightGBM and XGBoost, and the environment fixes PYTHONHASHSEED because TabICL's predictions depend on it. Only osx-arm64 is locked.

Related

This repository is an improved version of the code used to generate the forecasting tasks in The Join, the pretraining data of RT-J. Tasks follow the RelBench format.

Citation

@inproceedings{ranjan2026rtj,
    title={{RT-J}: Large-Scale Pretraining of Relational Transformers for Context-Efficient Predictions},
    author={Rishabh Ranjan and Vignesh Kothapalli and Harshvardhan Agarwal and Charilaos Kanatsoulis and Roshan Upendra and Tom Palczewski and Carlos Guestrin and Jure Leskovec},
    booktitle={The Fortieth Annual Conference on Neural Information Processing Systems},
    year={2026}
}

@inproceedings{relbenchv2,
  title={RelBench v2: A Large-Scale Benchmark and Repository for Relational Data},
  author={Gu, Justin and Ranjan, Rishabh and Kanatsoulis, Charilaos and Tang, Haiming and Jurkovic, Martin and Hudovernik, Valter and Znidar, Mark and Chaturvedi, Pranshu and Shroff, Parth and Li, Fengyu and Leskovec, Jure},
  booktitle={3rd Workshop on Navigating and Addressing Data Problems for Foundation Models (DATA-FM) at ICLR 2026},
  year={2026},
  note={arXiv:2602.12606 [cs.LG]},
  url={https://arxiv.org/abs/2602.12606}
}

License

Apache-2.0, © 2026 stanford-star.

About

Agentic task generation for relational databases

Resources

Stars

8 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages