diff --git a/README.md b/README.md index b2403d0..dbe465f 100644 --- a/README.md +++ b/README.md @@ -1,31 +1,111 @@ # nonnest2 -nonnest2 provides functionality for comparing non-nested models' fit and distinguishability, relying on theory from Vuong (1989). The authors acknowledge support from NSF grant SES-1061334. The contents of this package are those of the authors and do not reflect the views of the National Science Foundation. +[![Ask DeepWiki](https://deepwiki.com/badge.svg)](https://deepwiki.com/ContextualWisdomLab/nonnest2) -The package is intended to work automatically for models of many classes, including `lavaan`, `mirt`, and `glm`. It can also be applied to models of new, unseen classes, so long as the user provides functions to compute the model's casewise log-likelihoods and casewise first derivatives. A notable example is the [*merDeriv* package](https://github.com/nctingwang/merDeriv), which provides those functions for many models estimated via *lme4*. +**Evidence-based comparison of nested and non-nested statistical models in R.** +`nonnest2` helps analysts answer two questions that ordinary fit indices do not answer by themselves: whether two fitted models are empirically distinguishable, and which model is better supported when they are distinguishable. The package implements tests based on Vuong (1989) and also provides confidence intervals for differences in AIC and BIC. -# Example +The package works with several established R model classes and exposes a small adapter boundary for additional classes. It is a statistical comparison library; it does not choose a substantive model for you, establish causal validity, or turn information criteria into proof of scientific truth. -Consider the two factor analysis models below, estimated in `lavaan`. The models are not nested due to `x4` and `x7`. +## When to use it -```r -library("lavaan") +Use `nonnest2` when you have two fitted models whose casewise quantities refer to the **same dependent variable(s), the same observations, and the same observation row order**. For `lavaan` objects, the modeled variables must likewise correspond. The package does not currently verify row alignment for you, so a reordered or mismatched analysis sample can produce a numerically valid-looking but scientifically invalid comparison. + +Typical jobs include: + +- testing whether two models are distinguishable from the observed data; +- comparing relative model fit using Vuong-style tests; +- obtaining interval estimates for differences in AIC and BIC; and +- extending the comparison machinery to another model class that can provide the required casewise quantities. + +Built-in `llcont()` methods currently cover model classes from ecosystems including `lavaan`, `mirt`, `OpenMx`, base/generalized linear models, and several other regression-model families. The exact exported public API is `vuongtest()`, `icci()`, and `llcont()`. + +## Install from this source tree -m1 <- ' visual =~ x1 + x2 + x3 + x4 - textual =~ x4 + x5 + x6 - speed =~ x7 + x8 + x9 ' -fit1 <- cfa(m1, data=HolzingerSwineford1939) +The package metadata requires R 3.0.0 or later. From a checked-out revision: -m2 <- ' visual =~ x1 + x2 + x3 - textual =~ x4 + x5 + x6 + x7 - speed =~ x7 + x8 + x9 ' -fit2 <- cfa(m2, data=HolzingerSwineford1939) +```bash +R CMD INSTALL . ``` -We can use *nonnest2* to compare the models via Vuong tests and to obtain interval estimates of differences in the models' AIC and BIC values. +For development or release verification, use the repository's R package workflow rather than inferring quality from source presence alone: + +```bash +R CMD build . +R CMD check nonnest2_*.tar.gz +``` + +The repository also contains an `R-CMD-check` GitHub Actions workflow. Its current result should be verified on the exact revision you intend to use. + +## Quick start + +The example below compares two factor models that are non-nested because the indicator assignments differ. Fit both models to the same rows in the same order. ```r -vuongtest(fit1, fit2) -icci(fit1, fit2) +library(lavaan) +library(nonnest2) + +m1 <- ' + visual =~ x1 + x2 + x3 + x4 + textual =~ x4 + x5 + x6 + speed =~ x7 + x8 + x9 +' +fit1 <- cfa(m1, data = HolzingerSwineford1939) + +m2 <- ' + visual =~ x1 + x2 + x3 + textual =~ x4 + x5 + x6 + x7 + speed =~ x7 + x8 + x9 +' +fit2 <- cfa(m2, data = HolzingerSwineford1939) + +comparison <- vuongtest(fit1, fit2) +comparison ``` + +This particular Holzinger–Swineford pair is the package vignette's **indistinguishable-model** example, so it is useful for learning the first-stage variance/distinguishability test but it is **not** a valid `icci()` example. Interpret the distinguishability result before treating relative-fit evidence as meaningful. + +`icci()` is intended for **non-nested models that are distinguishable according to the `vuongtest()` variance/distinguishability test**. If models are nested or indistinguishable, the intervals returned by `icci()` are incorrect for their intended interpretation. Use `icci()` only after a comparison satisfies that gate; the vignette's Political Democracy example demonstrates a distinguishable non-nested pair. + +Use the inferential threshold and model assumptions appropriate to the analysis rather than treating a package default as a scientific decision rule. + +## Integration model + +`nonnest2` keeps the public comparison layer small, but its adapter responsibilities are distinct: + +- `llcont(model)` returns a numeric vector of **casewise log-likelihood contributions** in observation order; +- `vuongtest()` separately obtains **casewise score contributions in the same observation order** through `score1` / `score2` when supplied, or through the package's class-specific/default score path such as `sandwich::estfun()` or the supported `mirt` score path; and +- `vc1` / `vc2` supply **parameter covariance matrices whose row/column ordering must match the score parameter ordering used by `calcAB()`**; they are not observation-level arrays. + +A new model adapter must therefore preserve observation-row alignment between likelihood and score contributions, preserve parameter-order alignment between scores and covariance matrices, and provide each statistical quantity through the correct interface. `llcont()` is not a combined likelihood-and-derivative return contract. + +For the exact supported S3 methods, see [`NAMESPACE`](NAMESPACE). Package history is recorded in [`NEWS`](NEWS), and longer worked material lives under [`vignettes/`](vignettes/). + +## Statistical basis + +The package metadata cites: + +> Vuong, Q. H. (1989). Likelihood ratio tests for model selection and non-nested hypotheses. *Econometrica, 57*(2), 307–333. https://doi.org/10.2307/1912557 + +The software implements statistical procedures based on that theory; using the software does not remove the need to check identification, estimator assumptions, data quality, model misspecification, observation alignment, and the substantive meaning of the compared models. + +## Project status and verification + +Current package metadata declares version `0.5-9` dated 2026-03-31. Treat that as source/package metadata, not by itself as evidence of a particular published artifact, deployment, benchmark, or certification. For any consequential analysis, bind results to the exact package revision and preserve the fitted-model inputs and software environment needed to reproduce the comparison. + +Maintainers can track current scientific, integration, release, and licensing gaps in [`docs/product-technical-gap-baseline.md`](docs/product-technical-gap-baseline.md). That dated ledger is not a substitute for fresh exact-head CI/review evidence. + +## Contributing + +Keep changes focused on the statistical contract and supported model adapters. New adapters should include realistic tests for casewise likelihood contributions, score/covariance integration, row alignment assumptions, and comparison behavior. Changes to numerical formulas, supported model semantics, or public return values should be documented and verified through the repository's ordinary R package checks. + +## License and commercial-use boundary + +The authoritative package metadata declares **`GPL-2 | GPL-3`** and identifies upstream/external authors and contributors. This repository therefore does **not** claim a new MIT or Apache-2.0 grant for the inherited package source. + +GPL licenses permit commercial use under their terms, but GPL-family source is not an approved default inbound component under ContextualWisdomLab's current commercial ecosystem policy. Accordingly, this repository should not be presented as approved for incorporation into a ContextualWisdomLab commercial product unless the exact provenance, copyright-holder rights, distribution model, and copyleft obligations are resolved through an explicitly approved repository-level path. + +See [`docs/commercial-license-boundary.md`](docs/commercial-license-boundary.md) for the evidence and closure conditions. Dependency licenses are separate from the license of `nonnest2` itself and do not relicense this package. + +The original NSF acknowledgement remains part of the package history: material in this package is partially based on work supported by NSF grant SES-1061334, and the package contents do not represent NSF views. diff --git a/docs/commercial-license-boundary.md b/docs/commercial-license-boundary.md new file mode 100644 index 0000000..18c7d53 --- /dev/null +++ b/docs/commercial-license-boundary.md @@ -0,0 +1,59 @@ +# nonnest2 commercial-license boundary + +## Purpose + +This note records why the current `nonnest2` source cannot be silently converted to the normal ContextualWisdomLab Apache-2.0/MIT repository baseline, and what evidence would be required before the package could be treated as an approved commercial-ecosystem component. + +It is a provenance and engineering-control record, not legal advice and not a change to the package's existing license. + +## Current evidence + +This record was verified on **2026-09-02** against the repository default branch at immutable revision `master@807e9405f8c32faafdf186f977a24d0b23358b43` and re-read on this documentation lineage. The `DESCRIPTION` bytes on that source state: + +- `Package: nonnest2`; +- `Version: 0.5-9`; +- `License: GPL-2 | GPL-3`; +- Edgar Merkle and Dongjun You are package authors, with additional contributors including Lennart Schneider, Mauricio Garnier-Villarreal, Seongho Bae, and Phil Chalmers; and +- the package URL is `https://github.com/qpsy/nonnest2`. + +The commit history reachable from `master@807e9405f8c32faafdf186f977a24d0b23358b43` was also inspected on 2026-09-02 and contains ContextualWisdomLab maintenance contributions. That bounded maintenance evidence does not establish ownership of every inherited copyright interest and therefore does not establish unilateral relicensing authority over the complete package. + +This document intentionally does not claim a branch-protection state that has not been established as licensing provenance. The immutable source revision and package metadata—not a mutable branch label—are the evidence used here. + +No root MIT/Apache license is added by this documentation lane. The R package `DESCRIPTION` license remains the authoritative source-license declaration unless and until an evidence-backed licensing change is approved by the relevant rights holders. + +## ContextualWisdomLab policy boundary + +GPL licenses permit commercial use under their terms. The blocker here is not a claim that GPL is noncommercial; it is that ContextualWisdomLab's current inbound policy does not accept GPL/LGPL/AGPL-family source as the normal incorporation baseline for its intended commercial distribution model. + +Therefore: + +- do not describe this repository as MIT-, Apache-2.0-, or otherwise permissively licensed; +- do not treat hosting or maintaining this repository as proof of relicensing authority; +- do not present the package as an approved dependency for incorporation into another ContextualWisdomLab commercial product under the normal permissive baseline; +- do not use dependency licenses to infer a different license for `nonnest2` source; and +- do not remove copyright, attribution, or source-license obligations in order to make the repository appear policy-clean. + +## Closure paths + +The commercial-policy blocker is resolved only by an evidence-backed repository-level outcome such as one of the following: + +1. **Rights-backed permissive relicensing.** Obtain authoritative provenance and copyright-holder/contributor rights sufficient to relicense all relevant source under a commercially compatible permissive license, then update package metadata, root licensing/NOTICE material, source headers where applicable, and release evidence together. +2. **Independent compatible replacement.** Replace GPL-covered implementation with independently authored, commercially compatible source without copying or deriving from GPL implementation details that would carry derivative obligations. Preserve the validated public statistical contract through black-box/reference testing and independent implementation evidence. +3. **Explicit approved GPL distribution model.** Adopt a repository-specific commercial distribution model that is demonstrably compatible with the exact GPL obligations and explicitly approved as an exception to the organization baseline. + +A README disclaimer, a new dependency, a process boundary, or the absence of a root `LICENSE` file is not sufficient closure. + +## Scientific continuity required by any replacement + +A licensing repair must not silently change the statistical product. At minimum, the surviving implementation must re-establish evidence for: + +- `vuongtest()` distinguishability and relative-fit behavior; +- `icci()` AIC/BIC difference interval behavior for non-nested, distinguishable models; +- `llcont()` casewise log-likelihood contribution semantics; +- the separate score-function and covariance-matrix contracts used by `vuongtest()`; +- exact observation/sample ordering assumptions across compared models and adapter outputs; +- supported adapter behavior for the model classes the repository continues to advertise; and +- edge cases, numerical tolerances, and failure behavior required by existing tests and documented examples. + +Release or incorporation claims should be bound to one exact source revision and ordinary package/security verification rather than inferred from this documentation record. diff --git a/docs/index.md b/docs/index.md new file mode 100644 index 0000000..573cfdb --- /dev/null +++ b/docs/index.md @@ -0,0 +1,50 @@ +# nonnest2 + +[![Ask DeepWiki](https://deepwiki.com/badge.svg)](https://deepwiki.com/ContextualWisdomLab/nonnest2) + +`nonnest2` is an R package for evidence-based comparison of nested and non-nested statistical models. It implements Vuong-style distinguishability and relative-fit procedures and provides confidence intervals for differences in AIC and BIC. + +## Start here + +The task-first [README](https://github.com/ContextualWisdomLab/nonnest2/blob/master/README.md) covers installation, a worked example, model-alignment requirements, supported adapter contracts, verification, statistical interpretation, and contribution guidance. + +The public API is intentionally small: + +- `vuongtest()` — distinguishability and relative-fit comparison; +- `icci()` — information-criterion difference intervals for non-nested models that satisfy the distinguishability condition; and +- `llcont()` — model-specific casewise log-likelihood contributions. + +## Statistical boundary + +Compared models must refer to the same dependent variable(s), the same observations, and the same observation order. `nonnest2` does not currently verify row alignment. `llcont()` likelihood contributions and score contributions therefore share the observation ordering, while `vc1` / `vc2` are parameter covariance matrices whose rows and columns must match the score parameter ordering consumed by `calcAB()`; they are not observation-level inputs. + +`icci()` is valid only for non-nested models that are distinguishable according to the `vuongtest()` variance/distinguishability test. Nested or indistinguishable pairs must not be interpreted through its intervals. + +These procedures compare fitted models under their assumptions. They do not establish causal validity, substitute for identification or data-quality checks, or turn information criteria into proof of scientific truth. Consequential analyses should preserve the exact package revision, fitted-model inputs, and software environment needed for reproduction. + +## Verification + +For a checked-out revision: + +```bash +R CMD build . +R CMD check nonnest2_*.tar.gz +``` + +Use current exact-head CI evidence when deciding whether a revision is fit for use; documentation presence is not a release or quality receipt. + +## License and commercial-use boundary + +The inherited package metadata declares `GPL-2 | GPL-3`. ContextualWisdomLab maintenance of this repository does not create a new permissive grant over upstream copyright. GPL permits commercial use under its terms, but GPL-family source is outside ContextualWisdomLab's normal inbound commercial baseline. + +See [commercial-license-boundary.md](commercial-license-boundary.md) for the recorded immutable provenance, constraints, and closure paths. This documentation source must not be read as approval for incorporation into a ContextualWisdomLab commercial product while that boundary remains unresolved. + +## More documentation + +- [README](https://github.com/ContextualWisdomLab/nonnest2/blob/master/README.md) — product overview, usage, adapters, and interpretation +- [Product and technical gap baseline](product-technical-gap-baseline.md) — current scientific, integration, release, and licensing gaps +- [Commercial license boundary](commercial-license-boundary.md) — provenance and policy status +- [Vignettes](https://github.com/ContextualWisdomLab/nonnest2/tree/master/vignettes) — longer worked material in the source tree +- [Ask DeepWiki](https://deepwiki.com/ContextualWisdomLab/nonnest2) — repository-aware navigation and questions + +This file is only a publication-ready documentation source. It does not itself establish GitHub Pages publication or a released package artifact. diff --git a/docs/product-technical-gap-baseline.md b/docs/product-technical-gap-baseline.md new file mode 100644 index 0000000..ea4f8b5 --- /dev/null +++ b/docs/product-technical-gap-baseline.md @@ -0,0 +1,103 @@ +# nonnest2 product and technical gap baseline + +**Snapshot:** 2026-09-02 +**Default-branch evidence:** `master@807e9405f8c32faafdf186f977a24d0b23358b43` +**Audience:** maintainers, scientific reviewers, integration reviewers, and licensing/provenance reviewers + +This document records current product responsibility, scientific contracts, provenance constraints, and buyer-visible gaps. It is a dated evidence ledger, not a substitute for re-reading live GitHub PR/check/release state before integration. + +## Product responsibility + +`nonnest2` is an R statistical library for comparison of fitted nested and non-nested models using Vuong-based distinguishability and relative-fit procedures, plus confidence intervals for differences in AIC and BIC. + +The repository owns the comparison procedures and adapter contracts exposed by `vuongtest()`, `icci()`, and `llcont()`. It does not own the fitted-model implementations supplied by `lavaan`, `mirt`, `OpenMx`, base R model classes, or other supported model ecosystems, and it does not turn statistical comparison into causal or substantive model validity. + +## Ubiquitous language and invariants + +- **Casewise log-likelihood contribution:** one numeric contribution per observation, returned by `llcont()` or a caller-supplied `ll1` / `ll2` function. +- **Score contribution:** casewise parameter-score information supplied separately through `score1` / `score2` or the supported class-specific/default score path. +- **Parameter covariance:** covariance evidence supplied through `vc1` / `vc2` and used separately from the likelihood-contribution contract. +- **Distinguishability:** whether the competing models can be statistically distinguished on the observed data under the Vuong procedure. +- **Relative fit:** direction/evidence of model-fit difference after the relevant distinguishability/nesting conditions are respected. +- **Observation alignment invariant:** compared casewise vectors must refer to the same dependent variable(s), same observations, and identical observation row order. Current source documentation explicitly states that the package does not verify this alignment. +- **ICCI applicability invariant:** `icci()` is intended for non-nested models that are distinguishable under the `vuongtest()` variance/distinguishability test; nested or indistinguishable inputs can yield intervals that are invalid for the intended interpretation. + +## Context Map + +```text +fitted-model packages / user models + | + | llcont + score + covariance adapters + v ++-------------------------------+ +| nonnest2 model comparison | +| - distinguishability | +| - relative fit | +| - AIC/BIC difference intervals| ++-------------------------------+ + | + v +analysis/reporting code owned by the caller +``` + +External model packages remain separate authorities for their estimands, fitting behavior, parameterization, likelihoods, scores, and covariance estimators. `nonnest2` consumes those quantities through adapter boundaries; it does not copy their domain/runtime authority. + +## Product flow UML + +```mermaid +sequenceDiagram + participant A as Analyst + participant M1 as Fitted model 1 + participant M2 as Fitted model 2 + participant N as nonnest2 + + A->>N: vuongtest(model1, model2) + N->>M1: casewise likelihood / score / covariance + N->>M2: casewise likelihood / score / covariance + N-->>A: distinguishability + relative-fit evidence + alt non-nested and distinguishable + A->>N: icci(model1, model2) + N-->>A: AIC/BIC difference intervals + else nested or indistinguishable + A-->>A: do not interpret icci intervals as valid + end +``` + +## Data / ERD boundary + +The current package has no repository-owned database or persistent domain schema. A database ERD would therefore invent authority that does not exist and is intentionally **not applicable** to this baseline. Fitted model objects and analysis data remain caller/model-package state. + +## Current implementation evidence + +Default-branch `DESCRIPTION` at `807e9405f8c32faafdf186f977a24d0b23358b43` declares package version `0.5-9`, R `>= 3.0.0`, imports `CompQuadForm`, `mvtnorm`, `lavaan`, `sandwich`, and `methods`, and declares source license `GPL-2 | GPL-3`. It also identifies Edgar Merkle and Dongjun You as authors plus additional contributors and points to the upstream `qpsy/nonnest2` repository. + +`R/vuongtest.R` and `R/icci.R` document the same dependent-variable / modeled-variable / observation-order requirement and explicitly state that current code does not check it. `R/icci.R` also states that nested or variance-test-indistinguishable models make its returned intervals incorrect for the intended interpretation. + +The ContextualWisdomLab GitHub repository currently exposes no GitHub Release. Package/source metadata must therefore not be presented as an immutable organization release artifact. + +## Gap register + +| Priority | Gap | Evidence / risk | Required action | Status | +| --- | --- | --- | --- | --- | +| P0 | Commercial source-license incompatibility with organization default | `DESCRIPTION` declares `GPL-2` \| `GPL-3`; inherited authors/contributors mean maintenance does not prove unilateral relicensing authority | Establish rights-backed permissive relicensing, independently authored compatible replacement with scientific regression evidence, or an explicitly approved repository-specific GPL distribution model | **Blocked / unresolved** | +| P0 | Observation alignment is not executable | Source docs say models must use the same dependent/modeled variables and identical observation order, but current code does not validate that invariant | Add a fail-closed comparison/adaptor contract that proves compatible casewise population/order where technically observable; otherwise require an explicit caller-supplied immutable alignment identity and test mismatch cases | **Open scientific-integrity gap** | +| P1 | Adapter conformance is implicit across multiple quantities | Likelihood contributions, score contributions, and covariance matrices are separate interfaces and can silently disagree in row/parameter semantics | Add executable adapter conformance fixtures for supported classes, including length/order, finite-value, parameter-dimension, missing-data and mismatch failure cases | **Open evidence gap** | +| P1 | No immutable ContextualWisdomLab release | GitHub release inventory is empty; `DESCRIPTION` version is source metadata only | After licensing and scientific gates are resolved, publish one protected-source package/release with reproducible checks, source provenance, checksum/SBOM/license evidence as applicable | **Not release-ready** | +| P1 | Open PR queue contains overlapping security/performance variants | Multiple current PRs target the same exported-input-validation and `llcont.glm` optimization surfaces | Reconcile by verified source-delta carryover into one canonical writer per concern; retire predecessors only after every valid test/fix/documentation delta is proven present | **Active reconciliation required** | +| P2 | Product documentation was legacy/terse | Canonical README PR #118 now owns product-first usage, statistical conditions, provenance, Pages-safe navigation, and this gap ledger | Keep README claims synchronized with protected source and exact release state; do not promote branch evidence to released truth | **In progress on #118** | + +## Current README / documentation integration lane + +PR #118 is the canonical public-surface writer for `README.md`, `docs/index.md`, `docs/commercial-license-boundary.md`, and this gap baseline. Badge-only PR #115 was closed only after its complete two-line Ask DeepWiki delta was verified present in #118. + +At snapshot time, #118 exact current head is `c76d8b91f56f74e238da82257066596b420a92c2` before this baseline commit; every source mutation after that head invalidates those predecessor checks. Live checks/reviews must always be re-read from the final unchanged head. + +## Licensing / provenance action boundary + +See [`commercial-license-boundary.md`](commercial-license-boundary.md). The organization must not add a root Apache-2.0/MIT file or rewrite `DESCRIPTION` merely because the repository is hosted under ContextualWisdomLab. Dependency licenses also do not change the inherited package source license. + +Any independent replacement must preserve the public statistical behavior without copying GPL implementation expression. Scientific acceptance should include reference/black-box parity for distinguishability, relative fit, AIC/BIC interval behavior, supported adapter classes, missingness/alignment edge cases, and numerical tolerances. + +## Verification and update rule + +Update this ledger when a protected-source statistical contract changes, a licensing/provenance fact changes, an immutable release appears, or a gap changes state. PR heads/check run IDs are snapshot evidence only; merge authority always comes from a fresh read of the unchanged candidate head, current base, reviews/threads, checks, and repository governance. diff --git a/vignettes/nonnest2.Rmd b/vignettes/nonnest2.Rmd index 3f72b95..48117e7 100644 --- a/vignettes/nonnest2.Rmd +++ b/vignettes/nonnest2.Rmd @@ -65,13 +65,9 @@ vuongtest(fit1, fit2) ``` We see that we obtain two tests, the *variance test* and the *non-nested likelihood ratio test*. The former test informs us of the models' distinguishability, while the latter test compares the fits of two distinguishable models. Focusing on the variance test here, we obtain a small test statistic and a large *p*-value. These results imply that, as we suspected, the two models are indistinguishable in the focal population. -We can also use Vuong's theory to obtain confidence intervals for the difference in the models' AIC or BIC statistics: -```{r} -icci(fit1, fit2) -``` -Based on this output, we see that the AIC and BIC statistics are close but lower for `fit1` than for `fit2`. The 95% confidence intervals overlap with 0, implying that the model fits are sufficiently close that neither can be preferred over the other. The confidence intervals for AIC and BIC differences are exactly the same because both models have the same numbers of parameters; this would not typically occur for other models. +Do **not** use `icci()` for this pair. The `icci()` function documentation states that its intervals are incorrect when models are nested or when the `vuongtest()` variance test indicates that the models are indistinguishable. The next example shows a non-nested pair that passes the distinguishability gate before information-criterion intervals are computed. -## Distiguishable and Non-Nested models +## Distinguishable and Non-Nested models Now consider the following structural equation models, using Bollen's political democracy data. The first model is the original (classic) model, while the second model estimates different residual covariance parameters. The models are non-nested due to the differing residual covariances. ```{r} @@ -119,7 +115,13 @@ We now apply the Vuong tests to the two models. ```{r} vuongtest(fit1, fit2) ``` -We now see that the variance tests indicates that the models are distinguishable (at least, at $\alpha=.05$). This allows us to move on to the second test, which compares the fits of the two models. Based on this second test, we conclude that the two models have equal fit in the population of interest. Note that *fit* is defined here in terms of Kullback-Leibler distance (between each candidate model and the true model), and we make no assumption about either of the two candidate models being the true model. +We now see that the variance test indicates that the models are distinguishable (at least, at $\alpha=.05$). This allows us to move on to the second test, which compares the fits of the two models. Based on this second test, we conclude that the two models have equal fit in the population of interest. Note that *fit* is defined here in terms of Kullback-Leibler distance (between each candidate model and the true model), and we make no assumption about either of the two candidate models being the true model. + +Because this pair is non-nested and passes the distinguishability gate, `icci()` can now be used for confidence intervals on the AIC and BIC differences: +```{r} +icci(fit1, fit2) +``` +Interpret those intervals together with the distinguishability and relative-fit evidence; they do not make an indistinguishable or nested comparison valid. ## Nested models Finally, `vuongtest()` includes a `nested` argument, using Vuong's theory to compare nested models. The resulting test statistics, which are very similar to the statistics described above, can be viewed as robust versions of the traditional likelihood ratio test (LRT) for nested models. This is because Vuong's theory makes no assumption about either candidate model being the true model, whereas the traditional LRT assumes that the full model is the truth. @@ -155,4 +157,4 @@ Both tests can be viewed as alternatives to the traditional likelihood ratio tes ```{r} anova(fit1, fit3) ``` -which provides similar results. In particular, the Vuong likelihood ratio test statistic is equal to the traditional likelihood ratio test statistic, though the *p*-values differ. This is because the null distributions differ: the traditional LRT uses a chi-square null distribution, whereas the Vuong LRT uses a weighted sum of chi-square distributions. The latter distribution converges to the traditional chi-square distribution when the full model is the true model. \ No newline at end of file +which provides similar results. In particular, the Vuong likelihood ratio test statistic is equal to the traditional likelihood ratio test statistic, though the *p*-values differ. This is because the null distributions differ: the traditional LRT uses a chi-square null distribution, whereas the Vuong LRT uses a weighted sum of chi-square distributions. The latter distribution converges to the traditional chi-square distribution when the full model is the true model.