Pattern robustness drops from 100% (deterministic) to ~53% under EoT eval

[imjohsep]

The new EoT eval mode (added in #) reveals that test_plate_pattern.png evades the detector in only about half of augmented views, despite scoring 100% under the deterministic single-pass eval.

Repro

scarecrow eval test_plate.jpg --pattern test_plate_pattern.png --eot-samples 32 --eot-seed 42

Observed

Baseline being 0/32 confirms the augmentation distribution itself is not what's causing detections, so the pattern genuinely fails to suppress the detector under ~half of realistic camera transforms.

Why this matters

The deterministic eval only exercises one view. In deployment, an ALPR camera never sees the plate from exactly that view; angle, distance, blur, lighting, and lens distortion all vary. A pattern that only works at a single canonical view is not useful in the real world; the EoT rate is the honest metric.

Likely causes

in rough order of likelihood

1. Under-sampled training-time EoT. Config.eot_samples defaults to 4 (generate.py:38). At only 4 augmented views per step, gradient direction is noisy and the LSE worst-case objective can chase a shallow optimum. Bumping to 8–16 should tighten the distribution.
2. TAU tuning. TAU = 3.0 (generate.py:14) softens the max across EoT samples. Higher TAU pushes harder on the hardest sample but destabilizes past ~10. Worth sweeping 3, 5, 8.
3. Augmentation range mismatch. The training range may be milder than what the eval samples. It is worth logging per-step worst-case conf during training and comparing to eval scores.
4. Overfit to the target plate. The pattern is optimized against exactly one image; a broader dataset (or repeated random crops of the same plate) may help.

Proposed next steps

• Add training-time EoT eval every N steps so we can see the robustness curve, not just final-step loss
• Sweep eot_samples ∈ {4, 8, 16, 32} × TAU ∈ {3, 5, 8} on test_plate.jpg and log final EoT evasion rate at seed 42
• Report EoT rate (not deterministic rate) as the headline number in the README

Proposed Acceptance

Target: ≥ 90% EoT evasion at 32 samples, mean conf ≤ 0.10, without regressing the clean-baseline 0/32.