diff --git a/ARCHITECTURE.md b/ARCHITECTURE.md index 6ddd063..72d7e66 100644 --- a/ARCHITECTURE.md +++ b/ARCHITECTURE.md @@ -1,10 +1,11 @@ # kaefa Architecture -Last updated: 2026-02-14 +Last updated: 2026-08-25 ## Purpose `kaefa` is an R package for automated exploratory factor analysis (AEFA). +Accepted decisions already true on this branch are recorded in `docs/adr/`. It provides: - core AEFA execution (`aefa`, `engineAEFA`), diff --git a/README.Rmd b/README.Rmd index f646ed6..d1519e8 100644 --- a/README.Rmd +++ b/README.Rmd @@ -15,143 +15,147 @@ knitr::opts_chunk$set( # kaefa -The goal of kaefa is to improve researchers' ability to identify unexplained factor structures in complex, cross-classified multilevel data in R. It uses an automated exploratory factor analysis (aefa) framework. +[![Ask DeepWiki](https://deepwiki.com/badge.svg)](https://deepwiki.com/ContextualWisdomLab/kaefa) -## Algorithm +**Automated exploratory IRT/factor-model search for complex assessment data in +R.** -The automated exploratory factor analysis (aefa) framework implements a **greedy search algorithm** to efficiently explore the model space and find improved model configurations. The algorithm iteratively: +kaefa helps researchers explore uncertain factor structures without manually +rebuilding every candidate model. Its AEFA workflow coordinates candidate +estimation, information-criterion comparison, item-fit review, and iterative +refinement while keeping the fitted `mirt` model artifacts available for +scientific inspection. -1. Evaluates multiple model candidates with different factor structures and item response models -2. Selects the best model using AIC by default, with AICc, BIC, and sample-size-adjusted BIC available. DIC is used only when a fitted model actually supplies posterior DIC; it is never approximated with AIC. -3. Assesses item fit and removes poorly fitting items one at a time -4. Re-estimates the model until convergence to a locally optimal solution +## Where kaefa fits -This greedy approach enables efficient exploration of the model space while seeking improved solutions through iterative refinement. The method aligns with model selection and exploratory factor analysis research (Preacher, Zhang, Kim, & Mels, 2013; Jennrich & Bentler, 2011). +Use kaefa when the factor structure is not fully known in advance and you need a +repeatable exploratory search over supported IRT/factor-model candidates. The +package is aimed at psychometric and applied measurement work, including data +with multilevel or cross-classified structure where supported by the underlying +model path. -**References:** +kaefa does not turn an exploratory search into confirmatory evidence. Factor +interpretation, model adequacy, recovery, sampling assumptions, and downstream +score-use decisions still require domain review and appropriate validation. -- Preacher, K. J., Zhang, G., Kim, C., & Mels, G. (2013). Choosing the optimal number of factors in exploratory factor analysis: A model selection perspective. Multivariate Behavioral Research, 48(1), 28-56. https://doi.org/10.1080/00273171.2012.710386 -- Jennrich, R. I., & Bentler, P. M. (2011). Exploratory bi-factor analysis. Psychometrika, 76(4), 537-549. https://doi.org/10.1007/s11336-011-9218-4 -- Hurvich, C. M., & Tsai, C.-L. (1989). Regression and time series model selection in small samples. Biometrika, 76(2), 297-307. https://doi.org/10.1093/biomet/76.2.297 -- Spiegelhalter, D. J., Best, N. G., Carlin, B. P., & van der Linde, A. (2002). Bayesian measures of model complexity and fit. Journal of the Royal Statistical Society: Series B, 64(4), 583-639. https://doi.org/10.1111/1467-9868.00353 +## Core workflow -## Installation +1. Provide an item-response data set or supported model inputs. +2. Run `aefa()` with the candidate/search settings appropriate to the study. +3. kaefa evaluates candidate models through `engineAEFA()` and the configured + fit/search policy. +4. Review the selected model, item-fit evidence, convergence diagnostics, and + retained search history before interpreting the result. -You can install kaefa from github with: +A minimal scientific API example is: -```{r gh-installation, eval = FALSE} -# install.packages("devtools") -devtools::install_github("seonghobae/kaefa") +```{r example, eval = FALSE} +library(kaefa) + +result <- kaefa::aefa(mirt::Science) +result ``` -## Example +The current numerical estimation boundary is `mirt`; kaefa orchestrates model +search and supporting diagnostics rather than owning an independent IRT +likelihood engine. -This is a basic example which shows you how to solve a common problem: +## Model-search behavior -```{r example, eval = FALSE} -## basic example code -library('kaefa') -mod1 <- kaefa::aefa(mirt::Science) -mod1 -``` +The AEFA loop uses a greedy candidate-search strategy. AIC is the default +selection criterion, with AICc, BIC, and sample-size-adjusted BIC available where +the fitted model supplies the required quantities. DIC is used only when a +fitted model actually supplies posterior DIC; it is not approximated with AIC. -## Remote Execution (Optional) +Item-fit/search behavior and accepted `mirt` fit-search decisions are documented +in [`docs/adr/`](docs/adr/). The research/provenance index is maintained in +[`docs/papers/README.md`](docs/papers/README.md). -You can preconfigure remote hosts and SSH keys for `aefaInit()`: - -```{r remote-ssh, eval = FALSE} -options(kaefaServers = c("node1", "node2")) -ssh_keys <- c( - normalizePath("~/.ssh/kaefa_node1"), - normalizePath("~/.ssh/kaefa_node2") -) -init <- aefaInit(sshKeyPath = ssh_keys) -``` +## Product surfaces -Security checklist: +kaefa currently remains one R package with three explicit product boundaries: -- Use absolute paths (expand `~` with `normalizePath()`). -- Restrict key permissions (for example, `chmod 600 ~/.ssh/kaefa_node1`). -- Store keys in encrypted storage or a secrets manager; never commit them. -- Rotate keys regularly (for example, quarterly) and limit access to required users or groups. +- **kaefa-core** — `aefa()`, `engineAEFA()`, model selection, item-fit evaluation, + theta-prior utilities, and scientific/recovery evidence. +- **kaefa-studio** — the optional bundled Shiny interface launched with + `launchAEFA()`. +- **kaefa-runner** — the future hosted/container/remote execution boundary; it + is not a separately released product today. -## Local vs Remote Execution Sizing Guide +Optional remote execution is configured through `aefaInit()`. Local execution +remains the default. SSH keys and other credentials must stay outside the +repository and be supplied through appropriately protected local/hosted +configuration. -Use these default thresholds when deciding whether to run `aefa()` on a local -workstation or a remote server. +## Evaluate the source -| Workload profile | Suggested runtime | Recommended environment | -| --- | --- | --- | -| Up to ~5,000 respondents and up to ~60 items | Usually under 30 minutes | Local machine (8+ CPU threads, 16GB+ RAM) | -| ~5,000-20,000 respondents or ~60-150 items | About 30-120 minutes | Remote VM/cluster node (16+ CPU threads, 32GB+ RAM) | -| Over ~20,000 respondents or over ~150 items | Often over 2 hours | Remote cluster/HPC (32+ CPU threads, 64GB+ RAM) | +The repository does not currently publish a GitHub Release. Source metadata is +`0.1.428`; that value and a passing development check are not immutable release +or support evidence. -These are empirical guidelines. Runtime and memory can vary by hardware, -`aefa()` options (for example, rotation/estimation choices), and parallel job -count. +For maintainers working in an isolated dependency/license-review environment, +run the repository package checks against the checkout rather than relying on +historical `seonghobae/kaefa`, Travis CI, or AppVeyor instructions: -Operational notes: +```bash +R CMD build . +R CMD check --no-manual kaefa_*.tar.gz +``` -- Prefer local runs for exploratory tuning and small pilot datasets. -- Prefer remote runs when model-search cycles are long, memory usage spikes, or - multiple analyses must run in parallel. -- If you see repeated slow convergence, monitor RAM/CPU and move the workload - to remote infrastructure before increasing model complexity. +The exact supported R/dependency environment and hosted evidence are defined by +current repository workflows and package metadata, not by historical badges in +this README. -## Interactive Shiny Interface +## Interactive studio -For applied psychologists who prefer a point-and-click interface without writing code, kaefa now includes an interactive Shiny web application: +The Shiny surface exposes the same underlying package responsibility for users +who prefer an interactive workflow: ```{r shiny-launch, eval = FALSE} -# Launch the interactive interface -library('kaefa') +library(kaefa) launchAEFA() ``` -The Shiny interface provides: - -- **Easy data upload**: Upload your item response data in CSV or RDS format -- **Simple configuration**: Configure factor extraction, rotation methods, and model selection criteria through dropdown menus -- **Visual results**: View factor loadings, item fit statistics, and model fit indices in an organized interface -- **Export results**: Download complete results and summary reports - -This makes kaefa accessible to researchers without programming experience while maintaining all the powerful automated factor analysis capabilities. - -## New Feature: fitdistrplus Integration for Theta Priors +Treat UI output as analysis evidence to review, not as an automatic scientific +or operational decision. -kaefa now supports setting theta priors based on empirical raw score distributions using the `fitdistrplus` package. This feature allows you to: +## Architecture and verification -1. Fit distributions to raw scores to inform theta priors -2. Test if calibration works for non-nominal models -3. Validate model calibration against empirical distributions +The current runtime flow is: -Example usage: +`aefa()` -> candidate search -> `engineAEFA()` / `mirt` estimation -> model +comparison -> item-fit/refinement -> selected model + diagnostics. -```{r fitdistrplus-example, eval = FALSE} -# Fit a distribution to raw scores -fit <- fitThetaPrior(mirt::Science, dist = "norm") - -# Test calibration with distribution fit -testResult <- testThetaPriorCalibration(mirt::Science, dist = "norm") - -# Apply theta prior during calibration -model <- applyThetaPrior(mirt::Science, fit, minExtraction = 1, maxExtraction = 1) -``` +See [ARCHITECTURE.md](ARCHITECTURE.md) for repository layout, runtime flow, +remote-execution boundaries, and quality gates. Scientific changes should carry +focused regression/recovery evidence; documentation-only checks do not prove +parameter recovery or model validity. -For more examples and detailed documentation, see the `examples/` directory. +## Commercial licensing status -## Software Quality Information +**Not currently cleared for ContextualWisdomLab commercial +intake/distribution.** -## Continuous Integration (Ubuntu, macOS, Windows) -[![R-CMD-check](https://github.com/seonghobae/kaefa/actions/workflows/R-CMD-check.yaml/badge.svg)](https://github.com/seonghobae/kaefa/actions/workflows/R-CMD-check.yaml) +`DESCRIPTION` declares kaefa itself as `GPL-3` and directly imports `mirt`, a +GPL-family runtime dependency, plus additional R dependencies that require a +complete current license inventory. There is no root Apache-2.0/MIT `LICENSE` +file that overrides those terms. -### Ubuntu and Mac environment -[![Travis-CI Build Status](https://travis-ci.org/seonghobae/kaefa.svg?branch=master)](https://travis-ci.org/seonghobae/kaefa) +Issue #83 owns the first-party provenance/relicensing review and replacement of +the GPL-family runtime/build dependency boundary while preserving intentionally +supported scientific behavior. Until that work is integrated and verified, do +not present kaefa as Apache-2.0/MIT-cleared or commercially policy-compliant. -### windows environment -[![AppVeyor Build Status](https://ci.appveyor.com/api/projects/status/github/seonghobae/kaefa?branch=master&svg=true)](https://ci.appveyor.com/project/seonghobae/kaefa) +## Documentation and contribution - - +- [Architecture](ARCHITECTURE.md) — product/runtime ownership boundaries. +- [Architecture decisions](docs/adr/README.md) — accepted fit/search decisions. +- [Research sources](docs/papers/README.md) — formula and literature provenance. +- [Contributor Code of Conduct](CONDUCT.md) — collaboration expectations. +- [Ask DeepWiki](https://deepwiki.com/ContextualWisdomLab/kaefa) — + repository-aware navigation. -[Contributor Code of Conduct](CONDUCT.md) +For scientific behavior changes, include the estimand/model assumption being +changed and reproducible recovery/regression evidence. For documentation or +operability changes, keep user-facing claims tied to current protected source +and exact-head checks. diff --git a/README.md b/README.md index 76d0386..6a11b39 100644 --- a/README.md +++ b/README.md @@ -5,183 +5,147 @@ # kaefa -The goal of kaefa is to improve researchers’ ability to identify -unexplained factor structures in complex, cross-classified multilevel -data in R. It uses an automated exploratory factor analysis (aefa) -framework. - -## Algorithm - -The automated exploratory factor analysis (aefa) framework implements a -**greedy search algorithm** to efficiently explore the model space and -find improved model configurations. The algorithm iteratively: - -1. Evaluates multiple model candidates with different factor structures - and item response models -2. Selects the best model using AIC by default, with AICc, BIC, and - sample-size-adjusted BIC available. DIC is used only when a fitted model - actually supplies posterior DIC; it is never approximated with AIC. -3. Assesses item fit and removes poorly fitting items one at a time -4. Re-estimates the model until convergence to a locally optimal - solution - -This greedy approach enables efficient exploration of the model space -while seeking improved solutions through iterative refinement. The -method aligns with model selection and exploratory factor analysis -research (Preacher, Zhang, Kim, & Mels, 2013; Jennrich & Bentler, 2011). - -**References:** - -- Preacher, K. J., Zhang, G., Kim, C., & Mels, G. (2013). Choosing the - optimal number of factors in exploratory factor analysis: A model - selection perspective. Multivariate Behavioral Research, 48(1), 28-56. - -- Jennrich, R. I., & Bentler, P. M. (2011). Exploratory bi-factor - analysis. Psychometrika, 76(4), 537-549. - -- Hurvich, C. M., & Tsai, C.-L. (1989). Regression and time series model - selection in small samples. Biometrika, 76(2), 297-307. - -- Spiegelhalter, D. J., Best, N. G., Carlin, B. P., & van der Linde, A. - (2002). Bayesian measures of model complexity and fit. Journal of the - Royal Statistical Society: Series B, 64(4), 583-639. - - -## Installation - -You can install kaefa from github with: - -``` r -# install.packages("devtools") -devtools::install_github("seonghobae/kaefa") -``` +[![Ask DeepWiki](https://deepwiki.com/badge.svg)](https://deepwiki.com/ContextualWisdomLab/kaefa) -## Example +**Automated exploratory IRT/factor-model search for complex assessment data in +R.** -This is a basic example which shows you how to solve a common problem: +kaefa helps researchers explore uncertain factor structures without manually +rebuilding every candidate model. Its AEFA workflow coordinates candidate +estimation, information-criterion comparison, item-fit review, and iterative +refinement while keeping the fitted `mirt` model artifacts available for +scientific inspection. -``` r -## basic example code -library('kaefa') -mod1 <- kaefa::aefa(mirt::Science) -mod1 -``` +## Where kaefa fits -## Remote Execution (Optional) +Use kaefa when the factor structure is not fully known in advance and you need a +repeatable exploratory search over supported IRT/factor-model candidates. The +package is aimed at psychometric and applied measurement work, including data +with multilevel or cross-classified structure where supported by the underlying +model path. -You can preconfigure remote hosts and SSH keys for `aefaInit()`: +kaefa does not turn an exploratory search into confirmatory evidence. Factor +interpretation, model adequacy, recovery, sampling assumptions, and downstream +score-use decisions still require domain review and appropriate validation. -``` r -options(kaefaServers = c("node1", "node2")) -ssh_keys <- c( - normalizePath("~/.ssh/kaefa_node1"), - normalizePath("~/.ssh/kaefa_node2") -) -init <- aefaInit(sshKeyPath = ssh_keys) -``` +## Core workflow -Security checklist: +1. Provide an item-response data set or supported model inputs. +2. Run `aefa()` with the candidate/search settings appropriate to the study. +3. kaefa evaluates candidate models through `engineAEFA()` and the configured + fit/search policy. +4. Review the selected model, item-fit evidence, convergence diagnostics, and + retained search history before interpreting the result. -- Use absolute paths (expand `~` with `normalizePath()`). -- Restrict key permissions (for example, - `chmod 600 ~/.ssh/kaefa_node1`). -- Store keys in encrypted storage or a secrets manager; never commit - them. -- Rotate keys regularly (for example, quarterly) and limit access to - required users or groups. +A minimal scientific API example is: -## Local vs Remote Execution Sizing Guide +```r +library(kaefa) -Use these default thresholds when deciding whether to run `aefa()` on a -local workstation or a remote server. - -| Workload profile | Suggested runtime | Recommended environment | -|----|----|----| -| Up to ~5,000 respondents and up to ~60 items | Usually under 30 minutes | Local machine (8+ CPU threads, 16GB+ RAM) | -| ~5,000-20,000 respondents or ~60-150 items | About 30-120 minutes | Remote VM/cluster node (16+ CPU threads, 32GB+ RAM) | -| Over ~20,000 respondents or over ~150 items | Often over 2 hours | Remote cluster/HPC (32+ CPU threads, 64GB+ RAM) | +result <- kaefa::aefa(mirt::Science) +result +``` -These are empirical guidelines. Runtime and memory can vary by hardware, -`aefa()` options (for example, rotation/estimation choices), and -parallel job count. +The current numerical estimation boundary is `mirt`; kaefa orchestrates model +search and supporting diagnostics rather than owning an independent IRT +likelihood engine. -Operational notes: +## Model-search behavior -- Prefer local runs for exploratory tuning and small pilot datasets. -- Prefer remote runs when model-search cycles are long, memory usage - spikes, or multiple analyses must run in parallel. -- If you see repeated slow convergence, monitor RAM/CPU and move the - workload to remote infrastructure before increasing model complexity. +The AEFA loop uses a greedy candidate-search strategy. AIC is the default +selection criterion, with AICc, BIC, and sample-size-adjusted BIC available where +the fitted model supplies the required quantities. DIC is used only when a +fitted model actually supplies posterior DIC; it is not approximated with AIC. -## Interactive Shiny Interface +Item-fit/search behavior and accepted `mirt` fit-search decisions are documented +in [`docs/adr/`](docs/adr/). The research/provenance index is maintained in +[`docs/papers/README.md`](docs/papers/README.md). -For applied psychologists who prefer a point-and-click interface without -writing code, kaefa now includes an interactive Shiny web application: +## Product surfaces -``` r -# Launch the interactive interface -library('kaefa') -launchAEFA() -``` +kaefa currently remains one R package with three explicit product boundaries: -The Shiny interface provides: +- **kaefa-core** — `aefa()`, `engineAEFA()`, model selection, item-fit evaluation, + theta-prior utilities, and scientific/recovery evidence. +- **kaefa-studio** — the optional bundled Shiny interface launched with + `launchAEFA()`. +- **kaefa-runner** — the future hosted/container/remote execution boundary; it + is not a separately released product today. -- **Easy data upload**: Upload your item response data in CSV or RDS - format -- **Simple configuration**: Configure factor extraction, rotation - methods, and model selection criteria through dropdown menus -- **Visual results**: View factor loadings, item fit statistics, and - model fit indices in an organized interface -- **Export results**: Download complete results and summary reports +Optional remote execution is configured through `aefaInit()`. Local execution +remains the default. SSH keys and other credentials must stay outside the +repository and be supplied through appropriately protected local/hosted +configuration. -This makes kaefa accessible to researchers without programming -experience while maintaining all the powerful automated factor analysis -capabilities. +## Evaluate the source -## New Feature: fitdistrplus Integration for Theta Priors +The repository does not currently publish a GitHub Release. Source metadata is +`0.1.428`; that value and a passing development check are not immutable release +or support evidence. -kaefa now supports setting theta priors based on empirical raw score -distributions using the `fitdistrplus` package. This feature allows you -to: +For maintainers working in an isolated dependency/license-review environment, +run the repository package checks against the checkout rather than relying on +historical `seonghobae/kaefa`, Travis CI, or AppVeyor instructions: -1. Fit distributions to raw scores to inform theta priors -2. Test if calibration works for non-nominal models -3. Validate model calibration against empirical distributions +```bash +R CMD build . +R CMD check --no-manual kaefa_*.tar.gz +``` -Example usage: +The exact supported R/dependency environment and hosted evidence are defined by +current repository workflows and package metadata, not by historical badges in +this README. -``` r -# Fit a distribution to raw scores -fit <- fitThetaPrior(mirt::Science, dist = "norm") +## Interactive studio -# Test calibration with distribution fit -testResult <- testThetaPriorCalibration(mirt::Science, dist = "norm") +The Shiny surface exposes the same underlying package responsibility for users +who prefer an interactive workflow: -# Apply theta prior during calibration -model <- applyThetaPrior(mirt::Science, fit, minExtraction = 1, maxExtraction = 1) +```r +library(kaefa) +launchAEFA() ``` -For more examples and detailed documentation, see the `examples/` -directory. +Treat UI output as analysis evidence to review, not as an automatic scientific +or operational decision. + +## Architecture and verification -## Software Quality Information +The current runtime flow is: -## Continuous Integration (Ubuntu, macOS, Windows) +`aefa()` -> candidate search -> `engineAEFA()` / `mirt` estimation -> model +comparison -> item-fit/refinement -> selected model + diagnostics. -[![R-CMD-check](https://github.com/seonghobae/kaefa/actions/workflows/R-CMD-check.yaml/badge.svg)](https://github.com/seonghobae/kaefa/actions/workflows/R-CMD-check.yaml) +See [ARCHITECTURE.md](ARCHITECTURE.md) for repository layout, runtime flow, +remote-execution boundaries, and quality gates. Scientific changes should carry +focused regression/recovery evidence; documentation-only checks do not prove +parameter recovery or model validity. -### Ubuntu and Mac environment +## Commercial licensing status -[![Travis-CI Build -Status](https://travis-ci.org/seonghobae/kaefa.svg?branch=master)](https://travis-ci.org/seonghobae/kaefa) +**Not currently cleared for ContextualWisdomLab commercial +intake/distribution.** -### windows environment +`DESCRIPTION` declares kaefa itself as `GPL-3` and directly imports `mirt`, a +GPL-family runtime dependency, plus additional R dependencies that require a +complete current license inventory. There is no root Apache-2.0/MIT `LICENSE` +file that overrides those terms. -[![AppVeyor Build -Status](https://ci.appveyor.com/api/projects/status/github/seonghobae/kaefa?branch=master&svg=true)](https://ci.appveyor.com/project/seonghobae/kaefa) +Issue #83 owns the first-party provenance/relicensing review and replacement of +the GPL-family runtime/build dependency boundary while preserving intentionally +supported scientific behavior. Until that work is integrated and verified, do +not present kaefa as Apache-2.0/MIT-cleared or commercially policy-compliant. - +## Documentation and contribution - +- [Architecture](ARCHITECTURE.md) — product/runtime ownership boundaries. +- [Architecture decisions](docs/adr/README.md) — accepted fit/search decisions. +- [Research sources](docs/papers/README.md) — formula and literature provenance. +- [Contributor Code of Conduct](CONDUCT.md) — collaboration expectations. +- [Ask DeepWiki](https://deepwiki.com/ContextualWisdomLab/kaefa) — + repository-aware navigation. -[Contributor Code of Conduct](CONDUCT.md) +For scientific behavior changes, include the estimand/model assumption being +changed and reproducible recovery/regression evidence. For documentation or +operability changes, keep user-facing claims tied to current protected source +and exact-head checks. diff --git a/SECURITY.md b/SECURITY.md index a4e53f2..3fa99cb 100644 --- a/SECURITY.md +++ b/SECURITY.md @@ -9,12 +9,19 @@ Older snapshots are accepted on a best-effort basis. If you discover a potential security issue: -1. Open a private security advisory at: - [GitHub security advisories](https://github.com/seonghobae/kaefa/security/advisories/new). -2. Do not create a public issue before we review and coordinate. - -Include: affected version/commit, impact details, reproduction steps, -and any proof-of-concept details. +1. Use this repository's private GitHub Security Advisory workflow when it is + available: + [GitHub security advisories](https://github.com/ContextualWisdomLab/kaefa/security/advisories/new). +2. If that private repository feature is unavailable to you, contact the + ContextualWisdomLab repository maintainers through an established private + channel. +3. Do not create a public issue, pull-request comment, or discussion before we + review and coordinate. + +Include the affected version or commit, impact details, reproduction steps, +and only the minimum proof-of-concept material needed to reproduce safely. +Do not include secrets, production credentials, or private data in a public +channel. ## Scope diff --git a/docs/adr/0001-mirt-estimation-delegation.md b/docs/adr/0001-mirt-estimation-delegation.md new file mode 100644 index 0000000..589fcdd --- /dev/null +++ b/docs/adr/0001-mirt-estimation-delegation.md @@ -0,0 +1,39 @@ +# ADR 0001: Delegate IRT/EFA estimation and item-fit to mirt + +- Status: Accepted +- Date: 2026-08-25 + +## Context + +kaefa searches unexplained factor structures by fitting candidate IRT/EFA +models and pruning poorly fitting items. The heavy estimation (MML-EM, +rotation, item-fit statistics) is already implemented and validated in +`mirt`. Re-implementing those internals in kaefa would duplicate a +maintained package and drift from its published definitions. + +## Decision + +IRT/EFA estimation and item-fit statistics are delegated to `mirt` +(Chalmers, 2012). kaefa owns the search loop and the decision rules on +top: `engineAEFA()` estimates candidates through `.mirt` / `.mixedmirt` +wrappers, `evaluateItemFit()` calls `mirt::itemfit()`, and `aefa()` +selects and prunes from that output. + +kaefa does not re-implement `P(theta)`, the MML-EM E-/M-step, `S-X2`, +`infit`, or `outfit`. Those remain `mirt`'s responsibility. + +## Consequences + +- Fit numbers consumed by the search (`Zh`, `S-X2`, `infit`/`outfit`) + come from `mirt` and stay subject to `mirt`'s validation. +- Package-local rules (cutoffs, AICc reconstruction, the DIC boundary) + are recorded in later ADRs and pinned in `docs/papers/README.md`. +- A `mirt` version change can change numeric output without a kaefa + formula change. + +## References + +Chalmers, R. P. (2012). mirt: A multidimensional item response theory +package for the R environment. *Journal of Statistical Software, 48*(6), +1–29. +[https://doi.org/10.18637/jss.v048.i06](https://doi.org/10.18637/jss.v048.i06) diff --git a/docs/adr/0002-zh-misfit-decision-rule.md b/docs/adr/0002-zh-misfit-decision-rule.md new file mode 100644 index 0000000..f693198 --- /dev/null +++ b/docs/adr/0002-zh-misfit-decision-rule.md @@ -0,0 +1,47 @@ +# ADR 0002: Standardised log-likelihood misfit Zh decision rule + +- Status: Accepted +- Date: 2026-08-25 + +## Context + +The search must decide when an item is misfitting so it can be removed +and the candidate re-estimated. `mirt::itemfit(fit_stats = "Zh")` +supplies the standardised log-likelihood statistic of Drasgow, Levine, +and Williams (1985). kaefa applies a local cutoff; it does not +re-implement `mirt`'s `Zh` internals. + +## Decision + +The published statistic, pinned in `docs/papers/README.md`, is + + Zh = (l0 - E[l0]) / sqrt(Var[l0]) + +An item is flagged as misfitting when + + Zh + qnorm(0.975) / sqrt(n) < qnorm(fitIndicesCutOff / 2) + +That is a one-sided lower-tail test at level `fitIndicesCutOff / 2` with +the small-sample correction `qnorm(0.975) / sqrt(n)`. With the default +`fitIndicesCutOff = 0.005` the threshold is `qnorm(0.0025) = -2.807`. + +The arithmetic lives in `.zhMisfitCount()` in `R/kaefa.R` and is applied +at the three search sites (rotation scan, best-candidate check, and the +final `ZhCond` gate). Those three sites must stay identical. + +## Consequences + +- Drift between the three sites changes which rotations and items the + search treats as misfitting. A 2019 debug commit dropped `/sqrt(n)` + from one site; the shared helper exists to prevent that class of + inconsistency. +- `Zh` itself remains a `mirt` computation. Audits compare the local + cutoff to the pinned equation, not a re-derived `Zh`. + +## References + +Drasgow, F., Levine, M. V., & Williams, E. A. (1985). Appropriateness +measurement with polychotomous item response models and standardized +indices. *British Journal of Mathematical and Statistical Psychology, +38*(1), 67–86. +[https://doi.org/10.1111/j.2044-8317.1985.tb00817.x](https://doi.org/10.1111/j.2044-8317.1985.tb00817.x) diff --git a/docs/adr/0003-sx2-rmsea-misfit-gates.md b/docs/adr/0003-sx2-rmsea-misfit-gates.md new file mode 100644 index 0000000..37c3258 --- /dev/null +++ b/docs/adr/0003-sx2-rmsea-misfit-gates.md @@ -0,0 +1,43 @@ +# ADR 0003: S-X2 and RMSEA.S_X2 misfit gates + +- Status: Accepted +- Date: 2026-08-25 + +## Context + +`Zh` is not the only item-fit gate in the search. When `mirt::itemfit()` +returns Orlando and Thissen (2000) `S-X2` columns, kaefa also uses the +limited-information p-value and the RMSEA computed from that statistic. + +## Decision + +kaefa flags misfit from `S-X2` when either of these holds +(`R/kaefa.R`): + +- `p.S_X2 < fitIndicesCutOff` +- `round(RMSEA.S_X2, 2) >= .05` + +`S-X2` is computed by `mirt` (`fit_stats = "S_X2"`). The 0.05 RMSEA +close-fit threshold for limited-information item fit follows +Maydeu-Olivares and Joe (2014). kaefa does not re-implement `S-X2`. + +## Consequences + +- Items can be pruned for a significant `S-X2` p-value or for rounded + RMSEA at or above 0.05 even when `Zh` is acceptable. +- Rounding RMSEA to two decimals is part of the accepted gate, not an + informal display choice. +- The statistic and its RMSEA remain `mirt` output; only the gates are + package-local. + +## References + +Orlando, M., & Thissen, D. (2000). Likelihood-based item-fit indices for +dichotomous item response theory models. *Applied Psychological +Measurement, 24*(1), 50–64. +[https://doi.org/10.1177/01466216000241003](https://doi.org/10.1177/01466216000241003) + +Maydeu-Olivares, A., & Joe, H. (2014). Assessing approximate fit in +categorical data analysis. *Multivariate Behavioral Research, 49*(4), +305–328. +[https://doi.org/10.1080/00273171.2014.911075](https://doi.org/10.1080/00273171.2014.911075) diff --git a/docs/adr/0004-rasch-infit-outfit.md b/docs/adr/0004-rasch-infit-outfit.md new file mode 100644 index 0000000..f88bbe1 --- /dev/null +++ b/docs/adr/0004-rasch-infit-outfit.md @@ -0,0 +1,32 @@ +# ADR 0004: infit and outfit only for Rasch unidimensional models + +- Status: Accepted +- Date: 2026-08-25 + +## Context + +Wright and Masters (1982) mean-square `infit` and `outfit` are Rasch +fit statistics. `mirt::itemfit(fit_stats = "infit")` can return them, +but they are not defined for the general multidimensional, non-Rasch +candidates `engineAEFA()` explores. + +## Decision + +`evaluateItemFit()` requests `infit` (which also returns `outfit`) only +when the fitted model has at least one Rasch item type and +`nfact == 1`. Other models skip that `itemfit` call. + +kaefa does not re-implement the mean-square residuals. Computation stays +in `mirt`. + +## Consequences + +- Multidimensional or non-Rasch candidates are judged by `Zh` and, when + available, `S-X2` / `RMSEA.S_X2`, not by `infit`/`outfit`. +- A later request for `infit` on a 2PL or multifactor model would be a + new decision, not an extension of this one. + +## References + +Wright, B. D., & Masters, G. N. (1982). *Rating scale analysis*. MESA +Press. diff --git a/docs/adr/0005-aicc-dic-criteria.md b/docs/adr/0005-aicc-dic-criteria.md new file mode 100644 index 0000000..5b6fdf4 --- /dev/null +++ b/docs/adr/0005-aicc-dic-criteria.md @@ -0,0 +1,47 @@ +# ADR 0005: AICc reconstruction and the DIC boundary + +- Status: Accepted +- Date: 2026-08-25 + +## Context + +`aefa()` selects among candidates by an information criterion. Current +`mirt` fits supply `AIC` and `logLik` but do not consistently expose +`AICc`. The maximum-likelihood / MAP models from current `mirt` +versions also do not expose the posterior deviance quantities needed to +reconstruct DIC. + +## Decision + +AICc is reconstructed only from the Hurvich and Tsai (1989) +small-sample correction. When the fit does not already supply a finite +`AICc`, kaefa recovers `k = (AIC + 2 * logLik) / 2` and applies + + AICc = AIC + 2 * k * (k + 1) / (n - k - 1) + +The statistic is undefined when `n <= k + 1`; kaefa reports that reason +rather than returning a fabricated finite score. + +DIC is accepted only when the fitted model supplies a finite posterior +DIC (Spiegelhalter et al., 2002). DIC is never reconstructed from AIC +and is never relabelled from AIC. `CAIC` is not treated as an alias for +`AICc`. + +## Consequences + +- Default search can use AIC, AICc, BIC, or saBIC without a posterior + sample. +- Requesting DIC on an ML/MAP `mirt` fit that lacks a finite DIC is an + error, not a silent fall-back to AIC. +- Sequential DIF selection also refuses to substitute AIC for DIC. + +## References + +Hurvich, C. M., & Tsai, C.-L. (1989). Regression and time series model +selection in small samples. *Biometrika, 76*(2), 297–307. +[https://doi.org/10.1093/biomet/76.2.297](https://doi.org/10.1093/biomet/76.2.297) + +Spiegelhalter, D. J., Best, N. G., Carlin, B. P., & van der Linde, A. +(2002). Bayesian measures of model complexity and fit. *Journal of the +Royal Statistical Society: Series B, 64*(4), 583–639. +[https://doi.org/10.1111/1467-9868.00353](https://doi.org/10.1111/1467-9868.00353) diff --git a/docs/adr/0006-local-default-independent-package.md b/docs/adr/0006-local-default-independent-package.md new file mode 100644 index 0000000..57c1b5b --- /dev/null +++ b/docs/adr/0006-local-default-independent-package.md @@ -0,0 +1,37 @@ +# ADR 0006: Local default and independent R package + +- Status: Accepted +- Date: 2026-08-25 + +## Context + +`ARCHITECTURE.md` describes kaefa as an R package whose runtime path is +`aefa()` / `engineAEFA()`, with `aefaInit()` available for optional +worker setup. In the ContextualWisdomLab ecosystem kaefa is a leaf: +other components may call it, but it must run without those components. + +## Decision + +Local execution is the default. `aefaInit()` remote workers are +optional. Hosts may be preconfigured with `options(kaefaServers = ...)`, +but nothing in the search requires a remote node. + +kaefa remains an independent R package (MSA leaf): it is runnable +without naruon and callable as a dependency. This repository does not +add sibling-repo checkouts or git submodules for ecosystem components. + +## Consequences + +- Users can run `aefa()` on a local workstation with the package + installed; remote SSH workers are an opt-in. +- Security-sensitive values (keys, tokens) stay out of git history, as + already stated in `ARCHITECTURE.md`. +- Ecosystem wiring to naruon or sibling repositories is out of scope + for this package's default path. + +## References + +No additional paper. This decision is already stated in +`ARCHITECTURE.md` (local default; optional `aefaInit()`) and in the +package `DESCRIPTION` (standalone R package). The allowed citation list +for this ADR set does not include an execution-topology paper. diff --git a/docs/adr/README.md b/docs/adr/README.md new file mode 100644 index 0000000..b183d29 --- /dev/null +++ b/docs/adr/README.md @@ -0,0 +1,17 @@ +# Architecture decision records + +This directory records accepted decisions that are already true on `develop`. +It does not invent product behaviour. Formula provenance and the pinned +equations remain in [`docs/papers/README.md`](../papers/README.md). + +Status dates are 2026-08-25. Citations were live-checked on that date; see +each ADR's References for the required locator. + +| ADR | Decision | Status | +| --- | --- | --- | +| [0001](0001-mirt-estimation-delegation.md) | IRT/EFA estimation and item-fit statistics are delegated to `mirt`; kaefa owns search and decision rules | Accepted | +| [0002](0002-zh-misfit-decision-rule.md) | Standardised log-likelihood misfit `Zh` decision rule | Accepted | +| [0003](0003-sx2-rmsea-misfit-gates.md) | `S-X2` and `RMSEA.S_X2` misfit gates | Accepted | +| [0004](0004-rasch-infit-outfit.md) | `infit`/`outfit` requested only for Rasch unidimensional models | Accepted | +| [0005](0005-aicc-dic-criteria.md) | AICc reconstructed via Hurvich and Tsai; DIC accepted only when `mirt` supplies a finite posterior DIC | Accepted | +| [0006](0006-local-default-independent-package.md) | Local execution is the default; kaefa remains an independent R package | Accepted | diff --git a/docs/papers/README.md b/docs/papers/README.md index cc2caa8..0530744 100644 --- a/docs/papers/README.md +++ b/docs/papers/README.md @@ -9,6 +9,7 @@ reference against which `R/kaefa.R` and `R/utils.R` are audited. The source articles are copyrighted and cannot be redistributed here, so each is cited with its DOI. Open-access / preprint links are noted where available. +Decision records for these rules live in `docs/adr/`. ## 1. Standardised log-likelihood fit statistic `Zh`