Skip to content

PluRel v2 into main, toward v2-alpha - #169

Merged
kvignesh1420 merged 94 commits into
mainfrom
v2
Nov 16, 2026
Merged

kvignesh1420 merged 94 commits into
mainfrom
v2

Conversation

@kvignesh1420

Copy link
Copy Markdown
Collaborator

Merges the v2 branch into main. This is a step toward the v2-alpha release, not a release: the version stays 2.0.0.dev0, PyPI stays at 1.1.0, and the README says PluRel v2 is under active development toward v2-alpha. The v1 generator remains at the v1.1.0 and v1.0.0 tags.

What changes on main

  • The v1 modules (scm, dag, config, dataset, ts, bipartite, transforms, utils) are replaced by the v2 API: declarative Table/Schema, realize/sample, SchemaPrior/TablePrior, relbench output, and describe/explain.
  • Docs: README.md, CHANGELOG.md, COMPATIBILITY.md, plus concept scripts, domain recipes, analyses and benchmarks under examples/ and benchmarks/.
  • CI covers lint, pyright, tests on Linux and macOS with per-platform draw fingerprints, the README blocks, the recipes, a clean wheel install on 3.12/3.13, and a benchmark smoke run.

Pre-merge review

Known and not blocking

  • The linux-x86_64-X86_V3 fingerprints predate v2: fixes from the review before merging into main #166. If GitHub assigns a V3 runner, its log prints the digests to paste.
  • A static database's manifest records val_timestamp: NaT, which relational-transformer's date parser rejects. SchemaPrior always has events, so corpora are unaffected.
  • main fast-forwards to v2. A fast-forward keeps per-PR history, including older commits with Claude co-author lines; a squash merge does not.

kvignesh1420 and others added 30 commits September 19, 2026 00:28
Delete scm.py (torch propagation), transforms.py, ts.py, dataset.py,
schema.py (random + SQL schema builders), config.py, dag.py, utils.py,
their tests, the generation scripts and the SQL notebook. bipartite.py
stays as the FK link sampler. Drop torch, pytorch-frame, sqlalchemy and
tqdm from the dependencies and re-lock; version becomes 2.0.0.dev0.
* Add the v2 distributions module

random.py fixes the RNG contract (Array, Seed, generator). distributions.py
holds the Distribution protocol and the source families: Normal, Uniform,
Beta, LogNormal, Exponential, Pareto, Poisson, Mixture, TimeSeries composed
of Trend, Cycle and AutoRegressive noise, and Calendar for weekday- and
hour-weighted timestamps as epoch seconds. All sample(n, rng) -> (n,) float.

* Fold pytest.ini into pyproject and drop the torch-era env option

* Drop the Array alias; use np.ndarray directly

* Keep module constants at the top

* Sample calendar slots with replacement; make Trend a pure power law

* Trim distribution tests to contract, edge cases, and composition

* Simplify the AR(1) recursion

* Shorten Mixture and Calendar sampling
* Add the v2 mechanisms core: terms, noise, and the additive family

The Mechanism protocol fixes the (n, d) node contract and the structure
an oracle reads: dim, parents, mean_parents, noise_parents and
interaction_pairs. Effect terms (Effect, LookupEffect, ProductEffect,
LookupScaleEffect, TransformedProductEffect) compose into Additive,
Aggregate and Multiplicative, which also expose per-term contributions.
Noise draws from any Distribution and may scale with parents. Root draws
a d-dimensional block from a Distribution.

* Share key and the term contract through a Term base class

* One base class per family: Mechanism and Effect

* Fold the additive family into Combine with a REDUCTIONS registry

* Name effect subclasses as effects

* Drop the redundant std from Noise; Normal accepts a zero std

* Root draws its block through Noise

* Keep Mechanism to the sampling contract; structure lives on Combine

* Place Mechanism beside its subclasses

* Mechanism owns its Noise and sample_noise

* Default dim and parents on Mechanism; inline Combine.terms

* Any effect may scale the noise

* Combine keeps to parents, contributions and evaluate

* Noise is a Distribution; heteroscedastic scale moves onto Combine

* Drop scale effects; heteroscedasticity is a product with an exogenous node

* Reduce over effects directly; drop contributions

* Effects are unary; interactions are product nodes
…exp (#39)

* Add block-valued effects, Softmax and the concat/logsumexp reductions

* Derive Combine.dim from the effects instead of storing it
* Add the MLP, tree, Fourier and quadratic effects

* Default MLP activations to the identity

* Name the identity transform identity
* Add the single-table SCM sampler

* Sort the SCM with graphlib instead of networkx
* Add Column views and DataFrame sampling

* Give the timedelta a unit

* Call column categories categories

* Make Column observations functions of the latent

* Require columns when building an SCM

* Call node values latents

* Model missingness as a rate or an indicator node

* Name the node result latent and export the default calendar
* Add the link models and drop networkx

* Draw cluster sizes from a distribution

* Validate link sizes and parameters

* Name the self-referential link HierarchyLink

* Call the cluster counts counts

* Name the self-referential link TreeLink

* Harden the HSBM sampler against zero draws and non-finite weights

* Raise on sparse link configurations instead of falling back

* Fix cluster bound rounding and reject fully excluded parents

* Split the HSBM sampler into one method per step

* Test the HSBM sampler against its closed form and fuzz its invariants

* Draw HSBM affinities from distributions

* Note that Pareto is the Lomax form

* Call per-parent weights popularity

* Name the block affinity method block_affinity

* Name the block affinity variable too

* Tidy the package exports
* Add FK, Port and the cross-table Schema sampler

* Split sampling into links, propagate and observe with per-node streams

* Validate intervened nodes and report shape mismatches clearly

* Key links by table and column and pin order independence and chained influence

* Pin that ports read through the key they name

* Raise on unmatched rows without a fill, on non-finite latents and on invalid links

* Call link arrays indices

* Keep constants together
* Add the relbench database, write and read

* Name primary keys per table

* Declare primary key and time column on the table

* Call the table declarations pkey_column and time_column

* Make the primary key a key column of the table

* Declare timestamp columns by kind so child events can follow parent events

* Declare and enforce time order between timestamp columns

* Check frames against their table declarations

* Reject self-ordering, unordered split timestamps and silent overwrites

* Name the io functions, fix empty calendars and bins, pin timestamp resolution, and fuzz the stack

* Tolerate a half-written dataset on overwrite
* Add the random DAG layouts

* Give the Cauchy layout TabICL's density offset

* Match networkx's Watts-Strogatz rewiring and pin the Prufer decoder
* Add the table prior

* Draw integer ranges exactly and inclusively

* Split the draw ranges into four single-expression classes

* Build effects through a registry and filter eligible families

* Name the integer ranges IntegersRange and LogIntegersRange

* Draw hyperparameters hierarchically as TabICL does

* Pin the between-table diversity that meta sampling adds

* Prefix the table prior knobs by what they control

* Document the table prior knobs

* Document the prior knobs in Google style

* Give binned columns their own bin count knob

* Name every count, width, rate and std for what it is
* Add the schema prior

* Derive time nodes from the table instead of a magic name

* Keep self-referential gathers acyclic and draw sources without replacement

* Observe childless aggregates as missing and draw HSBM cluster sizes

* Restore the follow and aggregate checks to their own blocks

* Draw HSBM affinities and widen the table layouts

* Split the schema seed into structure and row lanes

SchemaPrior.realize and SchemaPrior.rows both started from the same stream,
so a table's row count was a near-deterministic function of the table count
(correlation 0.98 over 300 seeds). Each now takes its own spawned lane.

TablePrior rejects an effect_families choice that cannot change width at
construction instead of failing on the first block or width-changing edge.

* Draw the warp location on the open unit interval

rng.random() can return exactly 0.0, which made a Beta shape zero and the
warped draw raise. The location now comes from the same grid without its
zero point, so both shapes are positive by construction.
* Order temporal tables by time and make events wait for the events they reference

relbench cuts every temporal table at test_timestamp and reads keys as row
positions, which only survives the cut when rows are in time order and no key
points past the cut. Schema.observe now sorts temporal tables by observed time
with missing times last, resets keys to row positions, remaps every foreign key
through the permutation and permutes latents alongside; create_database refuses
a temporal table out of time order.

The prior makes every activity table temporal, propagates temporality down
foreign keys, and lets a temporal table wait for all events it references: its
time is the maximum of the referenced times, and of its own schedule unless it
is tied to them, plus a drawn delay. Nullable keys keep their own schedule with
an absent parent imposing no bound. Self-referential trees stay on tables that
do not wait, where calendar roots are drawn in order, so a tree parent is the
earlier row. SCM.timestamp_nodes includes the ancestors of timestamp nodes so
nothing measured in seconds is gathered or aggregated.

* Aggregate only static children into their parents

A parent row summarizing events that happen after it leaks the future at any
cutoff, the same flaw as snapshot counters on real entity tables. Aggregates
now go only on keys whose child table is static, which by propagation makes
the parent static too, so the summary is a timeless fact. Gathers stay on
every key because the child waits for what it reads.

* Keep time columns on event tables only

Referenced tables are static, as in v1, so no key can point into the future
and the waiting mechanism, its schedule root and its two knobs go away.
Cutting a database at any time trivially leaves every key valid.

* Order temporal tables when creating the database instead of when sampling

create_database puts temporal tables in time order with keys as row
positions and remaps every foreign key, so Schema stays free of the relbench
contract and the ordering is a no-op for the prior, whose calendar roots are
drawn in order. write_database still refuses an unordered hand-built database.

* Add a time column on request instead of by probability

The schema prior decides time columns by table role, so time_probability only
served single-table use, where the caller can ask for one explicitly.

* Drop the timestamp filter from aggregate, whose child is always static

* Keep self-referential trees on static tables

With trees only on static tables no key involves an event table on the parent
side, so cut safety needs no reasoning about time order and gather no longer
filters time nodes on the parent side.
scripts/synthetic_gen.py writes one relbench database per seed through a
process pool: SchemaPrior -> realize -> rows -> sample -> create_database ->
write_database, named <db_prefix>-<seed> under --out_dir, and rejects a
non-positive database count. split_timestamps places the validation and test
boundaries at quantiles of the pooled time columns so a chosen share of
timestamped rows falls in each window.
* Fold Root and Softmax into Combine

One node type: a Combine without effects is a root whose value is its noise,
dim wide, and a Combine with onehot emits the one-hot argmax of its scores,
which with Gumbel noise samples the class from their softmax. bias is a
per-dimension offset any node may carry. dim is derived from the effects or
the bias and checked against a given value instead of being inferred silently.

* Drop the root branch from _consume, every consumer is a Combine

* Call the one node type Node

With no sibling types, Combine said what the node did rather than what it is.
The one-entry mechanism registry goes with it, and the prior's node knobs
take the node_ prefix the others already use.
* Rename Effect to Edge

An edge carries one parent's contribution to a child; the families are
LinearEdge, MatrixEdge, MLPEdge, TreeEdge, FourierEdge, QuadraticEdge,
NearestEdge and LookupEdge, registered as EDGES. Node.effects is Node.edges
and the prior's edge_families knob follows.

* Edges cross keys; Port and Mechanism are gone

An edge's tail is a node name, a Foreign(key, node) read from the row the key
points at, or a Summary(table, key, node, how) aggregated over the rows that
point back. The schema resolves crossing tails while it evaluates, from the
parent's single latent array, and derives the cross-table order from them.
A table with crossing tails or key-derived missingness samples only through a
Schema. FK carries the fill edges read on null keys; Summary carries the fill
for childless parents under mean, max or min. Missing values are applied at
observation: Orphan(key) and Childless(table, key) markers on a column. Node
absorbs the noise contract, SCM.mechanisms is SCM.nodes, and the prior
expresses gathers as crossing edges and aggregates as summary nodes.

* Name the edge tests after edges

* Name the last two edge tests after edges

* Feed summaries into existing parent nodes instead of new columns

An aggregation of child rows shapes a drawn parent node the way a gather
shapes a drawn child node; the prior no longer makes a column of it. Summaries
are wired before any key is read and keys read only nodes with no summary
among their ancestors, so the cross-table graph stays acyclic.

* Drop the Location alias

* Propagate in one loop over a flat topological order

generations and the schema-level evaluate are gone: every node of every table
is evaluated once, in topological order, with its own noise stream, and the
tails that cross keys are resolved as the loop reaches the node.

* Drop the Orphan and Childless markers

Masking an orphan's looked-up attribute or a childless parent's summary is a
post-processing step on the frames; the library keeps its finite fills and
leaves the NA to the caller.

* Make Schema the only executor

SCM describes a table: its nodes, columns, order and the tails that cross
keys. Schema executes: links, propagate, observe, sample. The per-node step,
intervention or edges plus noise then the shape and finiteness check, lives
inline in propagate, and a single table samples as a one-table schema.

* Finish the node vocabulary in the prior and tests

* Finish the node vocabulary in the integration test

* Key the table priors by table name in realize

* Gather the graph vocabulary in graph.py and the relational one in schema.py

graph.py holds Node, Edge and its families, the crossing tails Foreign and
Summary, and the aggregations; schema.py holds SCM, FK and Schema, the only
executor. mechanisms.py and scm.py are gone.

* Import the tails and aggregations from graph.py

* Fold the single-table tests into the schema tests

scm.py is gone, so its tests live with the module that now holds SCM.

* Wire gathers and aggregates through one helper

Both draw sources without replacement, a consumer per source, and add an edge
built by the table's prior; only the tail differs. The parent-side time filter
and the child-side summary filter were redundant under the static-parent rule.

* Split the per-node bindings in propagate onto two lines
An SCM is now purely local, the plan of one table, and refuses edges that read
across a key. The Schema holds the connectivity: keys and crossings, the edges
whose tails are Foreign or Summary, mapped from a (table, node); their
contribution is added to the node's own edges and noise when the Schema
realizes every node in topological order. The prior builds every table
locally first, then decides the crossings on the built tables where every
node name and width is known in both directions, summaries first, then reads,
then trees, giving each key's tails to a drawn share of the reading table's
nodes with at least one reader, and hands tables, keys and crossings to the
Schema. gather, aggregate and the replace-based rewiring are gone.
* Prior draws every mechanism the library has

Edge families lookup and nearest, reductions min and concat, roots Beta,
Exponential, Poisson and time series on temporal tables, nested
categoricals, empirical binning, indicator-driven missingness, four more
node layouts, duplicate keys and summaries of wide nodes.

* A concat node is as wide as its parents together
A numeric node, an indicator node and a categorical node without a
nested edge standardize the reduction plus crossing terms before noise
and bias, so the noise knob, the biases and the nested masks keep their
meaning at any depth.
* Prior ranges and presets reach v1's

Wider table, link level, class, bin and noise ranges; lognormal, pareto,
skewed beta and sparse poisson roots; five more transforms; deeper and
rescaled MLP edges; zero-inflated and outlier marginals; two more node
layouts; per-table calendars.

* A drawn share of the keys between static tables summarizes the child into the parent

* Nested logits floor at the most negative finite value, so no score lifts a disallowed class

* Nested categoricals refine a parent's class through a choice node and a product of indicators

* Remove nested categoricals

* The share of keys that summarize is drawn per database

* Every share is a range, warped per database or table and drawn per use
* Sample HSBM links per label group, calendars arithmetically and AR(1) noise as a scan

Children with the same cluster labels share one distribution over parents,
so the link sampler draws one inverse CDF per group instead of a parents by
children matrix. The calendar computes weekday and hour from epoch seconds
instead of pandas timedeltas. The AR(1) recursion is a blocked cumulative
sum instead of a Python loop.

* Calendars thin uniform draws by rejection; tree and quadratic edges avoid large temporaries

Rejection sampling replaces the five-fold candidate pool, which left 14% of
rows sharing a timestamp. A tree edge accumulates one tree at a time and a
quadratic edge is one matrix product and one elementwise product.

* A calendar span without weight raises instead of looping
* README for PluRel 2.0: generating, building and drawing databases

* README walks from one table to a corpus in five steps

* Shorter overview

* Overview without the paper sentence

* Overview without the pretraining note

* Getting Started sections instead of numbered steps

* Terser Getting Started
)

Two mechanisms a downstream single-table user needed could only be written as hidden helper
nodes, which a reader of the graph then had to recognise by convention. Both are now one node:

- "compound" joins REDUCTIONS: the exponential of the clipped sum, centred at zero, so a
  compounding mechanism (multiplicative risk factors, growth) is Node(edges, op="compound").
  The table prior draws it beside the other reductions.
- Node.scale, edges whose exp'd clipped sum multiplies the noise draw: heteroscedastic noise,
  the spread following the parents those edges read; they are parents like any other, so the
  topological order and the interventions see them.

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
…ns; the prior draws them for roots, node noise and marginals (#61)

Real columns and noises have symmetric heavy tails (Laplace, Student-t), skewed positive shapes
(Gamma, which also covers chi-square), overdispersed counts (negative binomial, which covers
the geometric) and mirrored skews. Affine puts a location and scale on any distribution; a
negative scale mirrors it. The table prior draws the new families for root noise and column
marginals, and a numeric node's noise is now a drawn unit-scale shape (normal, Laplace,
Student-t or a centred gamma) times node_noise_std instead of a normal alone.

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* Add runnable examples for generation concepts

* Address review feedback on generation concept examples
…platform draw differences (#156)

* v2: sort a rank map's latent once, breaking ties only when it has them

* v2: benchmark ladders, seeds apart from repeats, and a Linux measurement workflow

* v2: synthetic single-table and hand-authored-plus-synthetic workloads; v1 migration table

* v2: latency and memory in the README, Linux benchmarks, and how far platforms' draws differ

* v2: rank map re-sorts only tied rows; draws saved with levels and digests; compare reports blanks, ranks and missing draws

* v2: compatibility: draws differ by processor on Linux runners, measured

* v2: fingerprints keyed by NumPy's vector level; clearer benchmark figures, one panel per kind of schema

* v2: larger, darker figure text; labels in a column beside the data; phases for two workloads of each kind

* v2: figures beside their results, workloads named by what they are

* v2: time and memory figures in one row of panels, legends beneath

* v2: a simple benchmark: one random table and random databases of 2 to 128 tables, at exact row totals

* v2: drop the rank-map optimization, keeping the library lean

* v2: results of the simple benchmark on the M4 and the Linux runner, and the README section

* v2: benchmark the prior flow as users run it, without an extra calibration; save draws by platform; scaled memory ticks

* v2: benchmark results without the extra calibration, on both machines

* v2: README: which database is slowest, as measured

* v2: README: the benchmark's cost follows hidden values per row, not table count

* v2: README: per-table costs at small sizes

* v2: README: per-table costs, measured
* v2: examples/analysis/diversity.py, how diverse the random prior's databases are

* v2: diversity: data-seed spread pooled over 8 generators, missing measures left out, mechanism edge kinds only, each generator realized once
)

* v2: the table prior warps per database, and tables warp around it

* v2: Linux fingerprints for the nested warp, a test that warps nest, changelog

* v2: benchmark structures from seed 1, the first whose workloads all sample at every size

* Revert "v2: benchmark structures from seed 1, the first whose workloads all sample at every size"

This reverts commit 26e1bac.

* v2: nested warps stay near the database's preference; nested priors warp generically; a TablePrior fingerprint

* v2: Linux fingerprints for the reworked nesting and the TablePrior draws
* v2: benchmark a sample of tables and databases from the default priors, reporting distributions

* v2: results of 200 draws from the priors on both machines, and the README section

* v2: benchmark: record the plurel commit measured, warp table row counts as the prior does, guard empty and zero-time draws

* v2: benchmark results at the merged prior (287fef8) on both machines; full history for the plurel commit
… TablePrior (#149) (#158)

* v2: diversity measures graph depth, drawn parameters, time coverage and leaks, collapsed categoricals, and TablePrior alone

* v2: diversity imports Schema from plurel
…#161)

* v2: examples/analysis/difficulty.py, how learnable and how hard the random priors' targets are

* v2: difficulty: analysis-only asserts, exact and near copies from the data, parent and time-block bootstrap, quick mode for recipes

* v2: difficulty: time blocks over the test period, looser control check in the quick run
#159)

examples/reference/realism.py profiles rel-f1 and 60 draws of SchemaPrior at rel-f1's row counts
with diversity.py's measures plus marginal shape, category counts, children per parent and time
coverage; checks the rules the generator authors; and reports rel-f1's percentile among the
draws per measure, marking the size-dependent ones. Integer codes count as categorical. It
downloads the dataset, so pixi run recipes runs only its --self-test, a generated stand-in
exported through create_database. The measures both scripts share move unchanged to
examples/analysis/_profile.py, which run_recipes.py skips.
* v2: a shorter README: quickstart first, one short section per task, details in examples/

* v2: README: scored categoricals realize first, quickstart writes its own path, marginals, processors, summary fills, held latents
* v2: decision boundaries of TablePrior tables in the README

examples/figures/decision_boundaries.py holds two numeric causes of a categorical column on a
grid, other sources at their medians, and draws the class each cell gets most often over 25 data
seeds, for eight table seeds. The README shows the figure under Random tables and databases.

* v2: decision boundaries: typical levels, ranked colours, recipes check

Categorical sources are held at their most common level instead of an all-zeros median, levels
are the union over draws, and the six largest classes get colours with the rest grey. matplotlib
joins the pixi environment as a PyPI dependency (numpy unchanged), recipes draw one table in quick
mode, and the package job installs matplotlib for it. README wording and runtimes corrected.

* v2: decision boundaries: one BLAS thread, clearer names, comments

The figure script runs under threadpool_limits(1), which brings the default run back to about a
minute on one core; the committed PNG is unchanged. run_recipes and the CI package job say that
figures run in quick mode and need matplotlib.
…165)

* v2: note that PluRel v2 is under active development toward v2-alpha

* v2: the note in the README's words
* v2: fixes from the review before merging into main

A static table may not point at events; categorical levels are strings; TablePrior columns
read distinct slots; structured missingness blanks at its rate; a tiny autocorrelation is its
noise; blanks keep a ByLevel column's values; ranged causes draw in key order; a negated
shaped Range mirrors its shape; clearer errors for zero-weight edge families and Derived
shapes; realize handles given positions into a counted table.

* v2: Linux fingerprints for the review fixes
Relational Transformer's preprocessor reads val_timestamp as a %Y-%m-%d date and panicked on
the full quantile times v2 wrote. split_timestamps now returns the midnight before each
quantile, and the manifest records a midnight as a date, as the published corpus does.
@kvignesh1420
kvignesh1420 merged commit 90458b8 into main Nov 16, 2026
10 checks passed

This branch was previously deployed

1 inactive deployment
github-pages — 90458b83 Deployed Nov 16, 2026 by kvignesh1420 via deploy #75
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant