Controlled benchmarks for latent factor structure in images: recovering a known transformation factor from a representation, transferring it across classes, separating it from unrelated factors, and using it to classify. Factors include shape or class identity, color, stroke thickness, rotation, scale, position jitter, and localized outline roughness.
- Benchmarks use controlled synthetic data with known factors, paired generator interventions, and matched-random null comparisons. Real handwritten digits (MNIST, with an offline 8x8 fallback) are used for factor discovery without factor labels.
- Unsupervised factor recovery is not identifiable in general. Recovery is scored against known factors on synthetic data and validated by visualization and identity preservation on real digits, not by a cosine to a ground-truth direction.
- Representations compared: raw pixels, grayscale, Fourier-radial energy, Haar and Daubechies wavelet energy, 2D scattering (kymatio), random projection, and a small CNN encoder. Factor directions are estimated from paired interventions or from within-class differences. Classifiers are nearest-class-mean, logistic regression, or Mahalanobis (LDA) distance.
Style factors recovered from within-class differences (the method of experiments 15-16), using class labels only and no factor labels, traversed on one MNIST digit. Each row is a discovered factor; columns move from -2 to +2 standard deviations along it. Rows correspond to slant, thickness, and width changes.
De-confounded held-out accuracy as a function of how much of the discovered style subspace is kept (experiment 19). Alpha 0 is full removal, alpha 1 keeps the subspace untouched. The optimum is interior (near 0.5), so partial down-weighting beats both full removal and no removal. The dashed lines are Mahalanobis/LDA distance, which uses class labels only; the clean-regime line reaches a higher accuracy than any explicit-subspace setting, and the confounded-regime line is matched by the best interior setting.
Method comparison across five datasets and both regimes (experiment 21). Each panel is a dataset; bars are mean test accuracy over seeds with standard-deviation error bars. A weighted metric (LDA, learned diagonal) beats plain nearest-mean on every dataset, including CIFAR cat/dog/horse on frozen resnet18 features, but logistic regression on the same features is the strongest or near-strongest method throughout.
The steering-vector audit (experiments 24-26): the transfer residual of a constant steering vector fitted on training examples and applied to held-out ones, for eight controlled factors across a pixel control and three frozen encoders (supervised resnet18, contrastive CLIP, self-supervised DINOv2). A residual below 0.25 would mean the factor steers; no cell reaches it, including the pre-registered CLIP-steers-hue prediction (0.98) and real-digit thickness and slant (Experiment 26). Where a factor is represented at all, it is represented as a low-dimensional curve, not an editable direction. The full write-up is docs/paper_draft.md.
The first two figures regenerate with docs/make_preview_figures.py. The benchmark figure is
produced by the experiment 21 runner and copied into place with
docs/make_preview_figures.py --only benchmark. The audit figure regenerates with
docs/make_audit_figure.py (add --recorded to use the note values without re-running
the encoder passes).
| Experiment | Method | Result |
|---|---|---|
| 01 Synthetic factor inventory | Statistical audit of the generator (Cramer's V, pairing match rate). | Separable shape-color association 0.083, correlated regime 0.80, broken-pairing match rate 0.83 down to 0.04. Generator passes readiness checks. |
| 02 Representation anatomy | Linear probes, sensitivity, and leakage vs a matched-random null for raw, Fourier, wavelet, scattering. | Scattering leads on scale, shape, thickness, rotation. Trivial baselines lead on color and jitter. A structured method beats the null on 5 of 6 factors. |
| 03 Factor-change operators and transfer | Leave-one-shape-out mean-delta operator applied to a held-out shape. | Thickness, color, jitter, scale transfer (cosine 0.84 to 0.95). Rotation does not (0.35). |
| 04 Correlated-factor transfer stress | Operators learned under a shape-color correlation, tested on the broken split. | About 69 percent of cells stay robust under the broken split. Per-pair differencing cancels the co-occurring nuisance. |
| 05 Operator-algebra diagnostics | Equivariance-residual ladder, delta-subspace spectrum, principal angles, bootstrap stability. | Structured representations make a factor a low-rank shift or diagonal action (rank-90 1 to 13). Raw pixels make the same factor a high-rank linear map (35 to 160). |
| 05R Operator-algebra validation | Known-answer controls, independent recompute, dimension matching, null calibration. | Result holds under dimension matching: participation ratio scattering 1.9 vs raw 11, with raw still 8.6 at 64 dims. Report dimension-robust measures, not raw rank. |
| 06 Invariance mechanism (translation) | Translation nuisance sweep, scattering J-sweep, decodability guardrail. | Not scattering-specific. Fourier-magnitude and DWT-energy stay flat and keep the factor decodable. Scattering invariance is bounded by 2^J. |
| 07 Deformation discriminator | Smooth-diffeomorphism nuisance; scattering vs a learned CNN encoder vs a radial-energy summary. | No structured advantage on amount factors. An 8-bin radial-FFT-energy summary is simpler, equally decodable, and transfers thickness to an unseen shape better (worst-shape residual 0.09 vs encoder 0.59, scattering 0.87). |
| 08 Local-style transfer | Localized same-energy roughness, add-style operator, nearest-neighbour on-manifold judge. | A style-aware encoder transfers the style across class (worst-shape 1.0). A style-blind encoder reaches 0.5; scattering and the energy or raw baselines reach 0.0 to 0.48, and the energy baselines place the style in the wrong location. |
| 09A Local style factor atlas | Cluster intervention deltas with location labels hidden (ARI, NMI, purity). | Scattering clusters style location without labels (ARI 0.45 to 0.67, leave-one-shape-out 1.0). Energy summaries fail on location. |
| 10 Held-out combination generalization | Biased shape-nuisance training, unseen-combination test, project out a discovered nuisance subspace. | Projecting out the nuisance recovers held-out accuracy (raw with color 0.47 to 0.94, FFT with thickness 0.00 to 0.76). A learned encoder resists linear removal. |
| 11 Robustness and factor geometry | (A) correlation sweep, (B) counterfactual augmentation during training, (C) factor count. | (A) the shortcut appears above about 0.85 correlation. (B) augmentation moves the encoder 0.37 to 1.00. (C) 2 of 6 factor pairs are near-orthogonal (>60 degrees). |
| 12 Factor decomposition atlas | Visual: color direction as an image, PCA before and after color removal, borrow-and-subtract. | The color direction is the union of the donor outlines. Removing the color subspace organizes the embedding by shape. Borrow-and-subtract residual 0.67. |
| 13 Per-example factor relevance | Apply each known factor to an example, measure the prediction-flip rate. | A color-confounded classifier flips on recolor for confusable polygons (0.60) but not the distinctive triangle (0.00). A balanced classifier flips 0.01. 73 percent of checks are prunable. |
| 14 ICA vs known-factor baseline | Unsupervised FastICA compared to a probe handed the factor directions. | FastICA recovers color at cosine 0.84 and shape at 0.43 to 0.76 without labels. The handed-the-directions probe scores 1.0. |
| 15 Within-class-difference discovery | Difference same-class examples, decompose, using class labels only. | Color recovery cosine 0.84 vs raw ICA 0.73. The gap grows as more factors are added. |
| 16 Factor discovery on MNIST | Within-class differences on real handwritten digits, no factor labels. | Recovers interpretable style factors (slant, thickness, width). The identity-preservation metric saturates (within-class and ICA both 1.00). |
| 17 Remove, sweep, spectrum | (A) factor-count sweep, (B) remove-or-hurt under a blur confound, (C) within-class variance spectrum. | (A) the advantage reverses when a high-dimensional factor is added. (B) full removal lowers out-of-distribution accuracy 0.908 to 0.874. (C) about 18 effective within-class factors, a gradual spectrum. |
| 18 Factor-weighted distance | Raw mean distance vs hard removal vs Mahalanobis/LDA, with confidence coverage. | Weighting 0.827 > raw 0.804 > removal 0.756. At the top 20 percent by confidence all three reach about 0.98. |
| 19 Down-weight sweep and isolation | Sweep the style-subspace keep fraction; confounded vs clean training on a de-confounded test. | Interior optimum near 0.5 to 0.75 beats both removal (0.756) and keep (0.804). The LDA gain is larger without the confound (0.089 vs 0.022), so it is general. |
| 20 Learned metric weighting | Prototype-distance classifier with a learned metric on the discovered style subspace; train (confounded) vs validation (de-confounded) cross-entropy across capacities, against LDA. | A single learned weight reproduces the swept alpha (0.52 vs 0.50). The validation cross-entropy rise grows with capacity (1 param 0.014, 40 params 0.018, 1600 params 0.046); the unregularized full metric overfits where LDA (regularized) does not. De-confounded test accuracy saturates near 0.83 for the weighted methods vs nearest-mean 0.773. |
| 21 Multi-dataset benchmark | The Experiment 20 comparison plus logistic regression and k-NN, across five datasets (digits, MNIST, Fashion-MNIST, KMNIST, and CIFAR cat/dog/horse with frozen resnet18 features), 5 seeds, confounded and clean regimes. | A weighted metric beats nearest-mean on 5 of 5 datasets in both regimes; on CIFAR (CNN features) weighting adds 0.05 to 0.06 over nearest-mean (0.83 clean). Learned alpha tracks LDA within 0.035. The 1600-parameter full metric overfits (validation cross-entropy rise up to 0.74 on digits) where low-capacity metrics do not. Logistic regression on the same features is the strongest or near-strongest method throughout, and the weighting family does not consistently beat it. |
| 22 A-priori alpha | Three closed-form predictors of the down-weight alpha (variance matching, Ledoit-Wolf, Fisher ratio) from training data only, against a fine swept optimum; 10 seeds, digits and synthetic shapes, both regimes. | Best-predictor regret 0.015 to 0.049, never within 0.01. The optimum is protocol-sensitive. LDA (no alpha) matches or beats the best swept alpha in 3 of 4 cells. Open question closed in the negative. |
| 23 Orbit structure vs linear rank | Intrinsic dimension (TwoNN, Levina-Bickel, shuffle null), per-orbit rank, and a kernel rung on controlled rotation, subpixel translation, and brightness orbits in four representations. | All 12 cells land in their predicted class (five curved, three invariant, two direction_global, two per-example). Rotation and translation in pixel-like representations are curved: rank90 11 to 19 vs intrinsic dimension 0.2 to 2.3, and the steering vector fails (shift residual 0.95 to 1.01). High linear rank conflated curvature with diffuseness. |
| 24 Factor orbits in frozen latents | The Experiment 23 battery plus a held-out-bases transfer split on three frozen encoders (supervised resnet18, contrastive CLIP, self-supervised DINOv2) vs a pixel control. | All three encoders turn no factor orbit into a transferable direction: the transfer shift residual stays 0.98 to 1.00 (rotation, translation) and 0.49 to 0.59 (brightness), none below the 0.25 steerable threshold; no cell is direction_global. The translation diffuse reading (intrinsic dimension 4 to 6) is a fine-scale floor reading, not confirmed structure (a noise-floor control reproduces it). Pretraining objective does not change the reading: the calibrated negative for steering arithmetic on these factors across all three. |
| 25 Appearance factors in frozen latents | The Experiment 24 battery on three factors the encoders plausibly represent (roughness, thickness, hue) rather than the augmentation nuisances of Experiment 24, with a pre-registered prediction that CLIP would steer hue. | The prediction is falsified and the negative generalizes: no appearance cell steers in any encoder (transfer shift residual 0.77 to 1.05, hue in CLIP 0.98), and the pixel control holds (0.89 to 1.05). Hue is a curved color-circle arc everywhere; CLIP carries the most transferable hue structure (kernel transfer 0.49) but as a curve, not a steering vector. Across six factors and three pretraining paradigms, a constant steering vector never transfers. |
| 26 Real-digit external check | The same battery on controlled thickness and slant orbits of real MNIST digits (the Experiment 16 style axes), the external check on real image statistics, seed-checked with saturation and shear-wrap controls. | The negative extends to real images: no cell steers (transfer shift residual 0.60 to 1.02, none below 0.25), every cell curved. The 8x8 fallback near-miss (thickness 0.38) did not survive at 28x28 (0.60), and the encoders move thickness to 0.99, away from steerable. |
| 27 Trained latents and steerability | The 24-26 battery on three trained latent spaces (autoencoder, VAE, factor-supervised) against a pixel control, adding probe R2 (linear readability of the factor); seeds 0-2. | The audit's first steerable cell: brightness in a plain autoencoder is a transferable global direction (transfer shift residual 0.03, seed-stable 0.02 to 0.06), where frozen encoders left it per-example (0.49 to 0.59). Reconstruction pressure, not linear-readability supervision, creates it: a factor-supervised latent makes brightness perfectly readable (probe R2 1.00) yet unsteerable (0.66), and rotation and hue stay non-steerable in every trained latent. Linear readability and steerability dissociate. The write-up of Experiments 23 to 27 is in docs/paper_draft.md. |
- On amount factors (thickness, scale, color), low-dimensional energy summaries match or exceed scattering and a small CNN on simplicity, decodability, and cross-class transfer (Exp 02, 06, 07).
- A localized same-energy style transfers across class only when the representation isolates it. A style-aware encoder does this. A fixed wavelet or scattering prior and position-pooling energy summaries do not (Exp 08, 09A).
- Within-class differences recover style factors from class labels alone, both on synthetic shapes (color cosine 0.84) and as interpretable slant, thickness, and width axes on MNIST (Exp 15, 16).
- Removing a discovered nuisance subspace helps held-out-combination accuracy under a real confound but lowers it otherwise. Partial down-weighting beats full removal, and the accuracy effect coincides with Mahalanobis/LDA whitening, which needs only class labels (Exp 10, 17, 18, 19).
- The distinctive contribution of explicit factor discovery is interpretability (a named, visualizable axis). Classification accuracy is on par with, not above, the within-class covariance (Exp 18, 19).
- The weighting result replicates across five datasets and both a confounded and a clean regime, including natural images with frozen resnet18 features: a weighted metric beats nearest-mean on all five, but a plain logistic regression on the same features is the strongest or near-strongest method throughout, and the weighting family does not consistently beat it (Exp 21).
The explicit factor work is a diagnostic and debiasing instrument, not a classification-accuracy method. The accuracy gain from weighting is the within-class-covariance (LDA) effect, which needs only class labels, and a standard discriminative classifier matches or beats the entire metric family. The durable value is interpretability (named, visualizable within-class factors), shortcut detection (per-example counterfactual relevance), and a label-light way to find and down-weight a nuisance. An objective end-to-end review, a techniques glossary, the failures, a novelty assessment, and the open weak-supervision direction are in docs/review_and_lessons.md. The follow-on program (Experiments 22 to 27: the alpha closure, orbit structure vs linear rank, and the steering audit, which finds that frozen encoders never steer across pose, appearance, and real-digit factors but a trained autoencoder does steer a global factor where linear-readability supervision does not) is in docs/future_directions.md, and its write-up as a unit is docs/paper_draft.md.
- Benchmarks use controlled synthetic factors and small real-digit subsets. Methods are mostly linear in pixel or representation space.
- Recovery on real digits is validated by visualization and identity preservation, not by cosine to a ground-truth direction, because no ground-truth factor labels exist there.
- Several experiments run a single seed in quick mode. Numbers shift with sample size and with the chosen subspace dimension.
- Uniform subspace scaling (the alpha sweep in Exp 19) scales class signal and nuisance together along a mixed direction. Whether the factor transform can be decomposed to keep only the class-preserving part is unresolved. Setting the fraction a priori was tested in Experiment 22 and closed in the negative: closed-form predictors miss the swept optimum, which is itself protocol-sensitive, and LDA needs no fraction at all.
- Whether discovered within-class factors reduce human labeling effort (active learning with fewer oracle queries, or anomaly and mislabel detection) is untested. A label-efficiency comparison against uncertainty sampling and random selection would test it. See docs/review_and_lessons.md.
- Whether frozen foundation-model encoders turn the curved factor orbits of Experiment 23 into transferable directions was tested in Experiments 24 and 25 across three encoders (supervised resnet18, contrastive CLIP, self-supervised DINOv2) and six factors and answered in the negative throughout: no factor orbit became a transferable direction (transfer shift residual 0.49 to 1.05, none below the 0.25 steerable threshold), the pretraining objective did not change the reading, the negative held even for appearance factors the encoders represent (roughness, thickness, hue), and it extended to real MNIST digits under controlled thickness and slant transforms (Experiment 26). The fully observational variant (a thin-versus-thick group-difference vector scored on actually-thick digits) is the remaining external check. The write-up is in docs/paper_draft.md; the forward program is in docs/future_directions.md.
- Whether trained latent spaces, rather than frozen discriminative encoders, steer where the frozen ones do not was tested in Experiment 27: a plain autoencoder makes brightness a transferable steering direction (the audit's first steerable cell), while explicit linear-readability supervision makes the factor readable but not steerable, so reconstruction pressure creates steerability and a linear probe does not predict it. Whether the curved factors (rotation, hue) ever become constant-vector steerable under any training pressure, rather than only kernel-mappable, remains open (the branch plan is in docs/phase9_plan.md).
structured-transform-discovery/
├── README.md
├── requirements.txt # numpy, scikit-learn, scipy, PyWavelets; torch (CPU) and kymatio optional
├── docs/
│ ├── conventions.md # writing and figure conventions for this project
│ ├── paper_draft.md # the Exp 23-26 write-up: the steering-vector audit
│ ├── make_preview_figures.py # regenerates the README preview figures into docs/figures/
│ ├── make_audit_figure.py # regenerates the steering-audit figure (Exp 24-26)
│ ├── figures/ # committed preview figures
│ ├── retrospective.md # the experiment arc: questions, methods, findings
│ ├── future_directions.md # the follow-on program (Exp 22-27 and beyond)
│ ├── experiment_summary.md # per-experiment index and central narrative
│ ├── project_plan.md # phased roadmap
│ ├── research_roadmap.md # operator-algebra view and forward plan
│ ├── data_inventory.md # generator, factor table, paired interventions
│ └── update_map.md # which docs to touch after each experiment
├── src/structured_transforms/
│ ├── data/ # synthetic_factors.py, local_style.py (generators + interventions)
│ ├── representation/ # raw, fourier, wavelet, scattering, random, learned CNN encoder
│ ├── transforms/ # mean and local delta operators + matched nulls
│ └── evaluation/ # probes, geometry, leakage, transfer, operator-algebra, factor relevance, combination generalization, weighted distance
├── experiments/
│ ├── 01_synthetic_factor_inventory/ 02_representation_anatomy/
│ ├── 03_factor_transfer/ 04_correlation_stress/
│ ├── 05_operator_algebra/ 05_operator_algebra_validation/
│ ├── 06_invariance_mechanism/ 07_deformation_discriminator/
│ ├── 08_local_style_transfer/ 09A_local_style_factor_atlas/
│ ├── 10_combination_generalization/ 11_robustness_factor_geometry/
│ ├── 12_factor_decomposition_atlas/ 13_factor_relevance/
│ ├── 14_ica_vs_known/ 15_within_class_discovery/
│ ├── 16_real_digit_factors/ 17_remove_sweep_spectrum/
│ ├── 18_weighted_distance/ 19_downweight_sweep/
│ ├── 20_learned_weighting/ 21_robust_benchmark/
│ ├── 22_apriori_alpha/ 23_orbit_structure/
│ ├── 24_foundation_latents/ 25_appearance_factors/
│ ├── 26_real_digit_check/
│ └── 27_trained_latents/ # the first steerable cell: a trained autoencoder steers brightness
├── results/{tables,figures}/ # generated CSVs, debug reports, figures (gitignored)
└── tests/
Each experiment directory holds a runner (run_*.py), a note (*.md) describing method,
results, and caveats, and is covered by a smoke or known-answer test under tests/.
# one-time setup
py -m venv .venv
.\.venv\Scripts\python.exe -m pip install -r requirements.txt
# regenerate the README preview figures (downloads MNIST on first use; --dataset digits for offline 8x8)
$env:PYTHONPATH='src'; .\.venv\Scripts\python.exe -B docs/make_preview_figures.py
# Experiment 19 -- down-weight sweep and confound isolation
$env:PYTHONPATH='src'; .\.venv\Scripts\python.exe -B experiments\19_downweight_sweep\run_downweight_sweep.py --quick
# Experiment 23 -- orbit structure vs linear rank (fully offline, numpy only)
$env:PYTHONPATH='src'; .\.venv\Scripts\python.exe -B experiments\23_orbit_structure\run_orbit_structure.py --quick
# Experiment 16 -- factor discovery on real MNIST digits
$env:PYTHONPATH='src'; .\.venv\Scripts\python.exe -B experiments\16_real_digit_factors\run_real_digit_factors.py --quick
# Experiment 10 -- held-out combination generalization
$env:PYTHONPATH='src'; .\.venv\Scripts\python.exe -B experiments\10_combination_generalization\run_combination_generalization.py --quick
# Experiment 08 -- local-style transfer (writes a demo figure)
$env:PYTHONPATH='src'; .\.venv\Scripts\python.exe -B experiments\08_local_style_transfer\run_local_style_transfer.py --quick
# run the tests
.\.venv\Scripts\python.exe -m unittest discover -s testsEvery runner accepts --quick or --standard (sample size), --dataset mnist or
--dataset digits (digit experiments), and --skip-plots. Use --dataset digits for a
fully offline run. Optional dependencies (torch, kymatio) are skipped with
--skip-encoder or --skip-scattering where they apply.
See docs/experiment_summary.md for the per-experiment
index and docs/retrospective.md for the full arc.