Skip to content

Make the model the source of truth for bounds+priors; steer TOML to experiment-only overrides (docs + guardrails) #369

Description

@vsbuffalo

Summary

Bounds and priors for an estimated parameter can be declared in two places:

  • the model parameters {} block — p : rate in [lo, hi] ~ dist(...), and
  • the fit TOML [estimate].<p>bounds = [...], prior = { ... }.

In practice this two-layer setup is a footgun, and the docs/examples steer users
(and coding agents) toward the TOML layer, which is the wrong default. Proposal:
make the model file the canonical source of truth for bounds + priors, document
it explicitly, and reserve the TOML [estimate] block for experiment-specific
overrides
(a narrowed bound for a sensitivity run, a swapped prior for one
experiment) — not for defining priors/bounds on every basic fit. Plus a couple of
guardrails so the divergence can't stay silent.

Where this bit us (Garki DMT methods test, real session)

Concrete costs, all from priors/bounds living in the TOML instead of the model:

  1. init = "from_prior" was silently unusable. It reads priors from the model
    ~ declarations only (per docs/inference.md), not from [estimate].prior.
    Because our priors were in the TOML, the boundary-safe init couldn't see them,
    so we were pushed onto uniform_unconstrained — which repeatedly started chains
    in this model's large 2-observation-stream extinction -inf region, where
    they got stuck for the entire warmup (2/4 chains at ll = -inf, 0% acceptance,

    1000 steps). Moving the priors into the model made from_prior usable. Nothing
    warned that the TOML priors were being ignored by the init.

  2. Silent bounds divergence. The model declared g : rate in [0.001, 1.0]
    while the TOML overrode bounds = [0.01, 0.5]. Two disagreeing sources, no
    diagnostic; easy to lose track of which is authoritative (and which the init /
    transform actually uses).

  3. Prior/bounds coupling surfaces late. A beta prior requires the param's
    bounds to be exactly [0, 1]; a probability param narrowed to [0.02, 0.99]

    • beta fails only at fit-run time:
      error: parameter 'qgam': beta prior requires bounds [0, 1], got [0.02, 0.99].
      Correct as a rule, but it's a late, interface-level trap that a model-first
      convention (declare the beta-prior param once, on [0,1]) would make obvious.

The through-line: a coding agent (me) copied the [estimate].prior pattern from
the fit-toml examples, and inherited all three problems. The model-first path
would have avoided them.

Requests

Docs (primary). State the convention explicitly in docs/fit-toml.md and
docs/inference.md:

A parameter's bounds and prior belong in the model (p : kind in [lo,hi] ~ dist(...)) — that is the definition used by every fit. The fit TOML
[estimate] block should only name which params are estimated, and is for
experiment-specific overrides (a tighter bound for a sensitivity sweep, a
different prior for one experiment) — not for (re)defining priors/bounds on
ordinary fits.

Right now the fit-toml examples feature [estimate].<p>.prior prominently, which
teaches the TOML-first pattern by example. Add the model-first steer, and ideally
recast the canonical example so [estimate] is just g = {} / a name list with
the bounds+prior in the model.

Guardrails (secondary, optional but high-value):

  • Warn when a TOML [estimate] entry redefines a bound/prior already declared in
    the model — surface the divergence and which one wins.
  • Warn loudly when init = "from_prior" is active and any estimated param's prior
    exists only in the TOML (currently silent — the init just falls back to
    bounds-uniform for that param). At minimum, name the params it couldn't find a
    model prior for.
  • (Maybe) let from_prior fall back to the TOML prior, or document firmly that it
    never will.

Assessment (requested)

Agree — this would materially help the workflow. It matches the "the model is
the program" design (the model should be the self-contained source of truth), and
it removes an entire class of two-layer confusion we hit repeatedly in one session
(the from_prior foot-gun above cost the most). Model-first also makes the models
reproducible and portable on their own, and keeps the TOML to what it's actually
good at: naming an experiment and overriding one thing for it.

Filed from the camdl-garki methods test.

Metadata

Metadata

Assignees

No one assigned

    Labels

    area/cliCLI, output/progress UX, CAS run identity, orchestrationarea/compilerOCaml DSL → IR compiler surfaceeffort/LLarge: multi-day, cross-cutting, or needs an RFCkind/designRFC / proposal-stage decision

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions