Reproducibility package for the manuscript
Prediction-powered inference for AI-based maize height phenotyping from field images Haoming Wang, Chong Wang, Peng Liu — prepared for The Plant Phenome Journal (Original Article)
Repository: https://github.com/ChongWangStat/PPImaizeimage
Submission snapshot: release v1.0.1 (revision 10.4).
The public package contains the rights-cleared plant-level measurement,
prediction, covariate, sampling, and design tables needed to reproduce the
PPI analyses, simulations, sensitivity analyses, tables, and nonphotographic
figures. The representative field-image panels embedded in the manuscript are
authorized for publication and are included as a frozen figure. The complete
field-image archive and CVAT annotation masks cannot be redistributed because
of confidentiality and third-party rights restrictions; consequently,
training or re-running the detector from the study images and regenerating the
photographic panels are outside the public reproducibility scope. See
REPRODUCIBILITY_SCOPE.md for an exact crosswalk
and DATA_DICTIONARY.md for metadata.
pip install -r requirements-lock.txt # exact versions used for the article
bash verify.sh # ~1 minute: unit tests + 37 manuscript-value checks
bash rebuild_public.sh # fast rebuild from released result tables
bash run_all.sh # ~8 hours: regenerates the analysisverify.sh is the fast check: it runs the unit tests for the estimator
identities and 20_check_numbers.py, which checks 37 key numerical claims in
the article against the result tables and fails on any mismatch. requirements.txt
gives loose lower bounds instead, if you would rather use your own versions.
Re-running the detector is optional and needs requirements-detector.txt,
whose CPU build of PyTorch comes from a separate index:
pip install -r requirements-detector.txt \
--extra-index-url https://download.pytorch.org/whl/cpurun_all.sh begins by running the unit tests in tests/, which check the
algebraic identities the article states -- the difference-of-fits form, the
two-channel bias decomposition, the calibration invariance, the gamma
reparameterization of the mean, and the cluster sandwich against an explicit
loop -- so a disagreement between the article and the code fails loudly and
early. They take a few seconds and can be run on their own with
python3 tests/run_tests.py (pytest is not required, but pytest tests also
works).
Runtime is roughly 8 hours on two cores, dominated by the simulations
(04, 05b, 08, 14, 19, 24, 28) and the 9,999-replicate wild bootstraps in 02, 03,
10, 11, 15, 16 and 25. COVERAGE_REPS=200 MISSPEC_REPS=200 SAMEACC_REPS=100 bash run_all.sh cuts the longest steps for a quick pass. Results
land in results/tables/, figures in manuscript/figures/, and the two PDFs
in manuscript/. The statistical analysis starts from the released detector
prediction tables in data/raw/. Here, raw means unmodified source tables
for the statistical workflow; it does not mean raw field imagery.
Re-running the detector from the study images is not part of the public
workflow.
Reviewer-facing documentation is separated by purpose:
REPRODUCIBILITY_SCOPE.mdmaps claims and artifacts to their public inputs and scripts, and states the restricted upstream boundary.DATA_DICTIONARY.mddocuments the released source, derived, and result tables.MODEL_CARD.mdrecords the detector configuration, intended use, internal diagnostics, and lack of independent external validation.examples/demo_mean_ppi.pyis a small generated-data demonstration of the correction, independent of the study data.CITATION.cffprovides machine-readable citation metadata.
The scientific goal is to estimate the mean true (meter-stick) maize plant height over the imaged plants, and how it relates to two genotype-level covariates over the plants that carry them, using image-derived height predictions corrected by a small sample of reference (meter-stick) measurements — prediction-powered inference (PPI). The primary date is 2021-07-12 (2,533 plants, 517 rows); the same predictor is then applied on 2021-07-30 and 2021-08-06.
Three things about the data are essential to understanding the code.
1. Row identifiers were re-derived. Image names follow the pattern
GENOTYPE.21-C-004_YYYY-MM-DD-HH-MM_CAMERA.JPG. The pipeline's own parser took
the first "C+digits" as the row, which for genotypes such as C123, CML228 or
CI91B is the genotype; 46 of 517 images on 07-12 were mislabeled and 37
pairs of rows collapsed onto one id. 01_build_analysis_data.py derives the
row from the unambiguous .21-C-NNN_ token and joins covariates from the
field map (genotype_and_weather_2021.xlsx); the file-name genotype then
matches the field map for 516/517 images.
2. The labeled subsample is built from the full ground-truth table.
data/raw/plant_height_data_2021.csv holds meter-stick heights for 300 plants
(three per row in 100 rows) on six dates, identified by seed position 1–6
counted from the right of the camera view. Two definitions were recorded:
ft (topmost point of the plant along a stick held on the stalk — the
extended leaf, and after tasseling the flag leaf; tassel excluded) and ct
(topmost fully visible collar). ft is one consistent definition on every
date and is the reference measurement throughout; ct is reported as an alternative
for the later dates. On 07-12 the reference measurement is interpolated:
Y = (6/7)·ft_0711 + (1/7)·ft_0718; on 07-30 and 08-06 it is measured on the
imaging day.
In data/raw/maize_ppi_analysis_ready_with_manual.csv, by contrast, no record
carries both a manual height and a YOLOv8-pose prediction: the 132 records
with manual heights are paired with a different predictor (the CVAT
crop-matching procedure) on non-imaging dates. They are kept as
legacy_labeled.csv only to demonstrate (Section 3.10, Table S6) what goes
wrong when a rectifier is built from the wrong predictor.
3. Detections and measured plants had to be reconciled, and the detector's
training rows excluded where they overlap. The detector indexes plants from
the right; seed positions are also counted from the right; they disagree
whenever a seed position is empty. 01_build_analysis_data.py infers seed
positions from root x-gaps and accepts a row only when the inferred positions
close at 6. The detector was trained on 30 ground-truth rows' images from
07-30/08-06 (data/raw/detector_training_rows.json); those rows are excluded
from the later-date labeled sets and their exclusion on 07-12 is a sensitivity
analysis.
data/
raw/ unmodified tabular/statistical inputs (not raw imagery)
plant_height_data_2021.csv ground truth: 300 plants x 6 dates, ft and ct
maize_ppi_analysis_ready_with_manual.csv plant-level analysis file (YOLO predictions
for 2021-07-12 + the 132 legacy records)
heights_compare_2021.csv crop-matching heights on 12 hand-identified rows
yolo_predictions_2021-07-12.csv YOLOv8-pose inference output (original run)
yolo_predictions_2021-07-30.csv, ..._2021-08-06.csv same weights/settings, later dates
yolo_predictions_2021-07-12_rerun.csv 07-12 re-run in the same environment (check only)
yolo_inference_summary_*.csv per-run inference summaries
detector_training_rows.json rows used to train / evaluate the detector
genotype_and_weather_2021.xlsx genotype table + weather station
derived/ built by 01_build_analysis_data.py
target_all.csv N = 2,533 imaged plants, 517 rows (mean estimand)
target_population.csv N = 1,659 plants with covariates, 341 rows (regression)
labeled_all.csv n = 141 reference plants, 48 rows
labeled_subsample.csv n = 84 reference plants with covariates, 28 rows
later_dates/ the same four files per later date (labeled sets
exclude detector-training rows; *_incl_training_rows.csv kept)
legacy_labeled.csv the 132 crop-matching records, for the contrast
build_summary.json counts, accuracy summaries, slot-assignment outcomes, row-id correction
code/
ppi_lib.py estimators (difference-of-fits PPI/PPI++), cluster-robust
covariance for coefficients and linear contrasts,
selection-aware studentized wild bootstrap
simlib.py variance-component calibration + fast NumPy estimators
00_check_pipeline_rerun.py compare the released original-run and re-run tables -> table 0
01_build_analysis_data.py build target populations and labeled sets (three dates)
02_main_analysis.py primary PPI analysis: post-stratified mean + regression
-> tables 1, 1b, 2, 2b
03_sensitivity.py sensitivity analyses S1-S6 -> tables S1-S6
04_simulation.py design simulation -> table 3
05b_selection_aware_check.py naive vs selection-aware interval coverage -> table 4b
08_why_ppi_works.py real-data N curve; gain vs rho and N/n -> tables 5, 6
09_predictor_recalibration.py can the predictor be improved without retraining? -> table 7
10_later_dates.py the same predictor on 2021-07-30 and 2021-08-06 -> tables 8, 8b-d
06_figures.py the article's and the supplement's matplotlib figures
07_supp_tables.py LaTeX tables for the supplement
28_same_accuracy.py six predictors matched on accuracy, differing in
inferential damage -> table 26
30_planning.py planning calculator: correlation x rows x plants per row
-> tables 27, 27b-d
pipeline/ the upstream computer-vision pipeline incl. best.pt (see below)
tests/
test_estimators.py unit tests for the algebraic identities the article states
run_tests.py runs them without pytest (pytest tests also works)
results/tables/ all numerical results as CSV/JSON, plus run logs
manuscript/
main.tex, supplementary.tex, references.bib
figures/fig_flow.tex, fig1_combined.tex TikZ sources for the attrition diagram and
the combined Figure 1
figures/ the six article figures and four supplemental figures as PDF
tables/ generated LaTeX tables
main.pdf, supplementary.pdf
code/ppi_lib.py implements:
- PPI / PPI++ in the difference-of-fits form of Angelopoulos et al. (2023):
β(λ) = OLS_L(Y − λŶ ~ X) + λ·OLS_U(Ŷ ~ X); for the mean,μ(λ) = mean_L(Y − λŶ) + λ·mean_U(Ŷ). λ = 0 is manual-only, λ = 1 is original PPI, λ tuned per coefficient over [0, 3] is PPI++. The grid extends past 1 because the variance-optimal λ exceeds 1 when the predictor compresses variation relative to the truth, as it does here for the hybrids. Why this form: the M-estimation form[λA_U+(1−λ)A_L]⁻¹[...]projects the rectifier through the target Gram matrix; with a miscalibrated predictor and few labeled plants the Gram-matrix mismatch leaks into the estimate as variance (and, under covariate shift, bias). The difference-of-fits form attains the closed-form efficiency1 − ρ²/(1 + n/(N−n)); both facts are verified by simulation in08_why_ppi_works.py. - Post-stratified PPI for the mean of a population with known strata:
X= stratum indicators, estimand = contrastc'βwithc= population stratum shares. Every routine accepts either a coefficient index or a contrast vector (contrast_vector). - Cluster-robust covariance from per-row influence totals in which a row's labeled and target contributions are summed before squaring (labeled units are a subset of the target population).
- Selection-aware studentized wild cluster bootstrap: one Rademacher
weight per field row multiplies the residuals of both least-squares fits,
the estimator and its variance are recomputed at the refitted coefficients,
and λ is re-tuned inside every replicate. With ≤ 14 labeled rows the 2^G
sign vectors are enumerated exhaustively; otherwise 9,999 draws.
05b_selection_aware_check.pyshows this raises PPI++ coverage from 0.81 to 0.89 at 11 rows and from 0.88 to 0.91 at 20. - Two-phase control-variate estimator (Chen & Chen, 2000), and the two
imputation comparators (imputation-only: raw predictions treated as
measurements; calibrated imputation: regression-calibrated predictions
with a naive cluster-robust SE) in
02_main_analysis.py. - Support restriction and inverse-propensity weighting.
Seeds are fixed (SEED = 20260909; 777 in 05b_selection_aware_check.py)
and results are deterministic.
code/pipeline/ contains the scripts that produced the height predictions,
in execution order, plus the trained weights best.pt (6.4 MB):
extract_multiclass_mask.py— CVAT masks → cropped plant / negative samplesrename_out_inplace.py— standardise crop naming (right-to-left ordering)crop_match_height.py— match crops back to source images; recover bounding box, top/root keypoints and pixel heightregress_px_to_truth.py— fit the pixel→cm calibration (ŷ = −1.7734 + 0.42633·h_px)convert_to_yolo.py,split_yolo_dataset.py— build the YOLOv8-pose dataset- training:
yolo pose train data=yolo_dataset/dataset.yaml model=yolov8n-pose.pt epochs=300 batch=8(run namepose_top_root_mps1024_b8) run_maize_height_inference_windows.py— inference with geometric post-filters; this is the script that produced every prediction table indata/raw/, run aspython run_maize_height_inference_windows.py --model best.pt --source <images.zip> --out_dir <dir> --imgsz 640 --conf 0.05 --iou 0.50 --device cpu(predict_top_root.pyandREADME_inference_settings.mdare the student's later variant and are kept for reference)build_manual_id_map_template.py,apply_manual_id_map.py,merge_pred_with_truth.py,apply_manual_exclusion.py,apply_px_to_cm.py— the student's plant-ID reconciliation and unit conversion (superseded by01_build_analysis_data.py)
The complete field-image archive (534 + 350 + 392 JPEGs, approximately 700 MB)
and original CVAT annotation masks cannot be redistributed because of
confidentiality and third-party rights restrictions. The released frozen
prediction tables are therefore the public starting point for the statistical
analysis. The code and trained weights document the upstream algorithm, and
the manuscript reports a frozen same-environment fidelity check, but the study
images required to repeat that check are not public. The detector metrics based
on the 30 training images are internal, training-set diagnostics rather than
external validation; 22_detector_metrics.py retains the supplied result table
when the restricted masks are absent. Production-format figure export
additionally requires the Poppler commands pdftoppm and pdftops;
run_all.sh skips that export when they are not installed.
The original paper (Angelopoulos et al. 2023, Sec. 1.2 and App. B) assumes a
prediction rule that is fixed before the analysis and independent of the
labeled data, and a target population in which all predictions are observed
and the outcome is observed for a subset. Here the detector was trained by
the imaging group on 30 annotated images from ground-truth rows on 07-30 and
08-06 before this analysis existed, then run unchanged over every date.
Because those 30 rows would be their own training images on the later dates,
they are excluded from the later-date labeled sets (in_detector_training);
on 07-12 the overlap is in rows but not images and a sensitivity analysis
excludes them. The ground-truth rows were chosen by the field crew rather
than at random, which is why the analysis post-stratifies, restricts to the
labeled covariate support, and reports IPW and transport diagnostics.
cd manuscript/figures && pdflatex fig1_workflow.tex && cd ..
latexmk -pdf main.tex
latexmk -pdf supplementary.texStandard TeX Live packages only: geometry, inputenc, fontenc, times,
setspace, lineno (article), amsmath, amssymb, graphicx, booktabs,
multirow, array, longtable, natbib with apalike, hyperref,
caption, microtype, algorithm, algpseudocode (article), and xr-hyper
(supplement). Rebuilding the optional TikZ figure source also uses standalone
and tikz; the compiled figure PDF is supplied. These packages ship in
texlive-latex-extra or smaller.
The Plant Phenome Journal distributes no LaTeX class file, so the manuscript
uses a plain article layout that follows the journal's section order and
produces the double-spaced, line-numbered PDF it asks for at review. Tables and
figures are embedded at first mention, as the instructions request. Compile
main.tex before supplementary.tex: the Supplemental Material cites the
article's section, equation and figure numbers through the xr package, which
reads main.aux.
Figure 1 is a two-panel composite: manuscript/figures/fig1_workflow.tex draws
the workflow and fig_flow.tex the attrition diagram, both in TikZ, and
fig1_combined.tex stacks them into the figure the article includes.
- The reference rows were chosen before this analysis was planned, and about half of the imaged ground-truth rows can be reconciled with the detections under a rule that refuses to guess seed positions. A prospective design should record plant identities at planting.
- Essentially every ground-truth row was used at some stage to train, validate or calibrate the detector; the 30 training rows are excluded from the later-date labeled sets. A prospective design should annotate rows that carry no ground truth (or use cross-fitting).
- Per-measurement noise of the meter-stick heights is about 12 cm. After tasseling the crew recorded a single flag-leaf value in both height columns for 18–31% of labeled plants, so the later-date reference measurement is less clean than the primary one.
- The YOLO top keypoint marks the attachment of the highest leaf; the reference
standard is the topmost point of the plant. The resulting offset (−23 cm on
07-12, −48 cm after tasseling) is exactly the kind of systematic error PPI
removes; it also caps the predictor's correlation with the truth (0.77 on
07-12; 0.44–0.58 later), which is what limits the efficiency gain.
09_predictor_recalibration.pyshows the cap is set by the detector, not by the pixel-to-cm calibration.
| Script | What it does |
|---|---|
00_check_pipeline_rerun.py |
compares the released original-run and same-environment re-run prediction tables (the restricted images are not read) |
01_build_analysis_data.py |
builds the analysis tables: row-id correction, covariate join, reference interpolation, seed-position reconciliation |
02_main_analysis.py |
primary estimates: mean and regression, five analyses, selection-aware bootstrap |
03_sensitivity.py |
analysis-set, reference-construction, weighting and mismatched-correction sensitivity |
04_simulation.py |
design simulation (rows x plants per row x sampling scheme) |
05b_selection_aware_check.py |
naive versus selection-aware intervals with few labeled rows |
08_why_ppi_works.py |
efficiency against rho and N/n, closed form versus simulation |
09_predictor_recalibration.py |
can the pixel-to-cm calibration be improved without retraining? |
10_later_dates.py |
the same predictor on 2021-07-30 and 2021-08-06 |
11_predictor_independence.py |
provenance, invariance to the calibration, calibration-free analysis, cross-fitting, never-seen rows |
12_attrition_selection.py |
flow from planted rows to analyzed plants, selection diagnostics, identification sensitivity |
13_image_figure.py |
the field-image figure with detections and keypoints |
14_coverage_realised.py |
interval coverage at the designs this study actually has |
15_design_adjusted.py |
block-adjusted regression and informative cluster size |
16_value.py |
measurement-equivalent rows and the tercile decision endpoint |
17_detector_report.py |
detection counts and error by date, stratum and seed position |
18_main_tables.py |
renders the article and supplemental LaTeX tables from the CSVs |
19_estimator_form.py |
difference-of-fits versus M-estimation form |
20_check_numbers.py |
checks 37 key numerical claims in the article against the CSVs |
21_export_figures.py |
figure export for submission |
22_detector_metrics.py |
optional detector diagnostics against restricted CVAT masks; retains the frozen table when masks are absent |
23_missingness.py |
measured height of the plants that did and did not enter the analysis |
24_misspecification.py |
departures from the structure the correction assumes |
25_calibration_comparator.py |
calibrated imputation with an honest bootstrap interval; simultaneous inference |
26_transport_check.py |
stability of the transported error-covariate relations |
27_make_scholarone_main.py |
the Word file ScholarOne requires in the main-document slot |
28_same_accuracy.py |
six predictors matched on accuracy, differing in inferential damage |
30_planning.py |
the planning calculator: correlation x rows x plants per row |
tests/run_tests.py |
unit tests for the estimator identities stated in the article |
The manuscript sources are in manuscript/ (main.tex, supplementary.tex).
The canonical public input to the PPI analyses is the frozen plant-level
prediction table for each date in data/raw/yolo_predictions_2021-*.csv,
together with the released reference-measurement, covariate, cluster, stratum,
and sampling tables. The trained detector (code/pipeline/best.pt) and its
inference script are released to document the upstream algorithm.
00_check_pipeline_rerun.py does not run the detector: it compares the released
original-run and same-environment re-run tables. The comparison found 2,460 of
2,533 published plants within 5 px and 2,453 of those with height within 2 px;
this is a frozen fidelity diagnostic, not a bit-for-bit public reproduction.
All inferential analyses start from the released prediction tables and do not
depend on access to the restricted images.
PPI assumes the predictor is fixed before the analysis and independent of the
reference measurements used for the correction. The detector here is trained on
image annotations and never sees a manual height, but the pixel-to-centimeter
calibration was fitted on manual heights. Two facts, proved in Section 2.8 of
the article and verified in 11_predictor_independence.py, bound what that can
do: the estimator is exactly invariant to the calibration intercept, and the
calibration slope enters only through the product lambda * b, so the whole
analysis can be run on raw pixel heights — a predictor no manual measurement
touched — and returns the same answers.