A Simulation Study of Dirichlet Process Mixture Priors and Goal-Specific Posterior Summaries in Bayesian IRT

Read the book: https://joonho112.github.io/dpmirt-simulation-study/

A preregistered Monte Carlo study of Bayesian semiparametric item response theory, published as a Quarto book with the frozen analysis data and the code that regenerates every figure, table and reported number.

The study crosses two model families (Rasch, 2PL), three latent shapes, five reliability tiers on a calibrated ladder, and four sample sizes into 120 conditions; 2,400 frozen datasets are fitted under three prior arms (Gaussian and two Dirichlet process mixture elicitations), summarized three ways (posterior mean, constrained Bayes, triple-goal), and scored on five loss families, for 7,200 fits and 115,200 loss values. The book is organized around the question a practitioner faces — which prior and which summary, for which goal, at which level of measurement information — rather than around the hypothesis list, and it separates the registered record from a labeled post-registration exploratory synthesis.

This is the third reporting edition. The preceding edition was sent for external review, which reproduced the confirmatory estimates and identified reporting defects; this edition repairs them and adds the verification gates that keep them from returning. Appendix D records what was found and what changed. The review correspondence and the design notes are working material and are not distributed with the book.

The three volumes

This project is published as three linked volumes. Each stands on its own; read together they separate what is true of a model, what is true in a study, and what changes on a real test.

Volume	What it owns
Theory	The mathematics and its sources: the measurement models, identification, reliability, shrinkage, DP mixture priors, and why three inferential goals imply three posterior summaries
Simulation study (this repository)	Realized outcomes under known truth: the preregistered design grid, the confirmatory record, Monte Carlo error, and the gate verdicts
Case studies	What changes on real tests: thirteen Item Response Warehouse cases under both item models, reported as consequence rather than correctness

How to cite

Lee, J. (2026). A Simulation Study of Dirichlet Process Mixture Priors and Goal-Specific Posterior Summaries in Bayesian IRT. https://joonho112.github.io/dpmirt-simulation-study/

Machine-readable metadata is in CITATION.cff. The two methods papers the study builds on are Lee (2025), arXiv:2512.16012 (reliability targeting) and Lee (2026), arXiv:2602.06301 (prior elicitation).

Licence

Text, figures, tables and data are released under CC BY 4.0; the code that builds the book is released under the MIT Licence. See LICENSE and LICENSE-CONTENT.

What v3 preserves and repairs

• The frozen production row-sets, confirmatory estimates, and results-first chapter structure are unchanged from v2.
• 35 reader-visible figures: 25 result figure divs (26 image bases because the Rasch/2PL levels tabs share one cross-reference), 8 concept figures, and 2 Mermaid diagrams. Generated images use PNG + PDF + caption sidecars.
• Exploratory analyses beyond the preregistered family remain first-class evidence: winner maps, lever decomposition (summary vs prior), crossed-lever competition, H6 shape decomposition, the item-count companion model, rank-flatness quantification.
• The external review's findings are remediated in the reliability and design account, the condensed theory, the registration history, result provenance, figure semantics, and the scope of the practitioner guidance.
• The reliability factor is documented as the implemented inverse-information MSEM design coefficient, not as exact posterior/EAP or observed-score reliability. Its frozen ladder combines nested test lengths with a cell-specific global discrimination refinement (c_star); v3 does not interpret it as a pure length intervention.
• Confirmatory records and post-registration exploratory synthesis are explicitly separated; exploration is not forced back into the preregistered family.
• Method provenance is visible at the point of reliance: the reliability-targeting method is cited to its arXiv preprint (arXiv:2512.16012), and the parent DPMirt manuscript supplies earlier simulation evidence rather than a pilot conjecture.
• Prose follows Rabe-Hesketh and Skrondal (2006) as its style benchmark.

Layout

docs/            the published site (GitHub Pages serves this directory)
book/            Quarto project (renders to ../docs)
  _common.R      facts loader + formatting helpers, sourced by every chapter
  chapters/      23 chapters (index.qmd holds the front matter)
  figures/       copies of figures/ that Quarto can resolve
code/R/          all generation code
  00-paths.R     layout resolution; discovers the repo root, no machine paths
  01-style.R     theme, palettes, save_figure (PNG+PDF+md sidecar)
  02-data.R      typed loaders for the frozen rowsets
  03-derive.R    derived layer -> data/derived/*.rds
  03b-exemplars.R  exemplar extraction from the V3 stores (loss-verified)
  04-facts.R     the facts object -> data/derived/book-facts.rds
  05-tables.R    gt tables -> tables/T-*.rds (+ csv mirrors)
  06-semantic-qa.R  figure and caption semantic gate
  07-release-qa.R   numerical release gate
  08-publication-qa.R  pre-publication hygiene gate
  fig-*.R        figure generators -> figures/{concepts,results}
code/tikz/       TikZ sources (estimand map; compiled with dvilualatex + dvisvgm)
data/tidy/       frozen production row-sets, the study's evidence (12 files, 15 MB;
                 the two largest ship gzip-compressed and are read transparently)
data/design/     design extracts + s2 ladder realization CSVs + tail-floor incidence
data/derived/    plotting frames, facts and QA results (rebuilt by 03/03b/04)
figures/, tables/  master outputs (book/figures mirrors figures/)

Rebuild

Run the following from the repository root, in order. The semantic QA step verifies that each master PNG/PDF/caption sidecar has an identical book/ copy. Quarto's automatic freeze does not track every upstream RDS or copied sidecar dependency, so the final two cache directories are removed before rendering.

Rscript code/R/03-derive.R
Rscript code/R/03b-exemplars.R      # optional; see the note below
Rscript code/R/04-facts.R
Rscript code/R/05-tables.R
for f in code/R/fig-*.R; do Rscript "$f"; done
cp -f figures/results/* book/figures/results/
cp -f figures/concepts/* book/figures/concepts/
Rscript code/R/06-semantic-qa.R
Rscript code/R/07-release-qa.R
rm -rf book/_freeze book/.quarto
cd book && quarto render

Then, before publishing, Rscript code/R/08-publication-qa.R.

R 4.6.0, Quarto 1.9.x; packages data.table, ggplot2, patchwork, ggdist, scales, ragg, gt, lme4, lmerTest, viridisLite, jsonlite, yaml, digest.

The repository root is discovered from code/R/00-paths.R's own location, so a clone builds wherever it is placed; DPMIRT_BOOK_ROOT overrides it if needed. Step 2 re-extracts the exemplar figures from the production fit stores, which are not distributed with this repository (they are 174 GB); the extracted objects are committed, so skipping that step reproduces the book exactly. Set DPMIRT_V3_SRC if you do have the stores and they live elsewhere.

Two environment notes carried from the build machine: data.table is pinned to one thread because multithreaded fread hangs there, and PDFs are written through the quartz device because cairo is unavailable. Neither is required elsewhere. The TikZ diagram needs the dvisvgm class option and the DVI route (dvilualatex then dvisvgm --exact).

Confirmatory record and exploratory synthesis

The locked estimands and decisions remain intact in the confirmatory record, while v3 keeps post-registration analyses and synthesis available as explicitly exploratory evidence. For H4, Chapter 12 gives the complete seven-cell table, intervals, and the post-outcome timing of Addendum 006. Summaries and discussion retain the invariant warning---seven bimodal reliability-0.5/0.6 cells had 5--14% higher MSEL---and point back to Chapter 12 instead of repeating the table at every mention. Corrected safety ratios only (1.052--1.138); convergence numbers from log 060 (0 exclusions); the 41-of-57 imprecision split from v1 is superseded by the explicit 49 / 5 / 3 interval decomposition computed here.

Release validation

The release candidate passed a complete clean rebuild and render:

• 32/32 numerical release checks, including the four frozen-row-set hashes, H1--H7, evidence and safety censuses, convergence, truncation, and the false-win diagnostic;
• 19/19 semantic figure checks covering 35 reader-visible figures;
• 102/102 PNG, PDF, and caption-sidecar master/book pairs byte-identical;
• 13/13 publication hygiene checks: no machine-specific path in any build source, no credential-shaped string, no file above 50 MB, licence and citation metadata present, and the published site shaped for GitHub Pages;
• 24/24 Quarto sources rendered to docs/index.html with no unresolved cross-references.

Machine-readable results are in data/derived/release_qa.csv, data/derived/figure_semantic_qa.csv and data/derived/publication_qa.csv. The only non-self-contained rebuild dependency is the read access 03b-exemplars.R needs to the frozen V3 fit stores; the committed exemplar objects mean the book itself rebuilds without them.

Publishing

The site in docs/ is what GitHub Pages serves (Settings → Pages → Deploy from a branch → main / /docs). docs/.nojekyll is required so that Pages does not strip the underscore-prefixed asset directories Quarto generates. Re-rendering rewrites docs/, so a publication commit normally contains both the source change and the regenerated site.