WD2 Controller Gate

Research stack for learning a supervised driving policy for Watch Dogs 2 streamed through NVIDIA GeForce NOW on macOS. It includes recording, data audits, training, offline shadow mode, and a deliberately constrained macOS live harness. The project is not an unattended-control system. The current V3 candidate remains blocked by its corrected deployment-parity result.

Current state

Implemented now:

• native Swift recorder for a selected GFN window;
• synchronized JPEG frames or optional HEVC chunks, keyboard events, action labels, timestamps, visual hashes, focus/window validity, and session-end summary;
• strict per-session validator, manifest builder, whole-session train/validation/test splitter, and offset-estimation aid;
• dual-stream temporal baseline: scene image + minimap crop + previous actions -> GRU state -> action heads;
• staged experiment runner for balancing, event labels, spatial representation, temporal history, input ablations, latency, and three-seed confirmation;
• frozen V3 candidate (denser_history, seed 23) and a completed three-session clean-holdout evaluation;
• deterministic ActionGate that fails closed, plus a deliberately limited human-gated macOS flight-test harness;
• shadow replay that writes predictions to JSONL and cannot control the game;
• corrected component-aware ActionGate confidence, event-target parity replay, session-aware stability scoring and a five-threshold development calibration;
• a corrected V0–V6 diagnostic suite showing that V3 gated_self remains degenerate even at the least restrictive tested threshold.
• selector diagnostics for directional label prevalence, candidate ordering and route-group separability; the current targeted-right evidence remains below random and selector output is abstention-only;
• an onset/brake-aware policy checkpoint selector and bounded no-history V7/V8 development candidates with mandatory gated_self replay.

Not implemented yet:

• the DINOv3 foundation-encoder policy family;
• offline Depth Anything / RAFT pseudo-label preparation and trained auxiliary motion, geometry, path and risk heads;
• a validated route-state command extractor;
• an independent GuardNet (deliberately deferred from the current manually supervised research mode).

The next policy candidate must not use raw previous-action input. The current first deliverable is a frozen DINOv3 ConvNeXt Tiny scene encoder with a causal temporal policy, evaluated against V8 on the existing grouped split. Depth Anything V2 Small and RAFT Small are offline teachers only: the deployed policy will receive RGB history plus a high-level minimap route command, never teacher outputs. The complete gated work order is in docs/ROADMAP_2026-07-26.md.

The active checkpoint is artifacts/releases/v3-candidate-2026-07-15/. It achieved steering macro-F1 0.7314 on the sealed three-session clean holdout, versus 0.7197 for repeat-last-action (+0.0117). This is a modest positive result, not an autonomy-release result. See docs/PROJECT_STATUS.md for the active plan.

The P0 deployment audit is now resolved as decision B: no confidence threshold is promoted and V3 does not proceed to a live smoke test. At the diagnostic lower bound 0.50, V3 gated_self was 99.73% neutral, switched steering only 0.40/min versus the event-target rate 57.09/min, and had brake F1 0. The raw/post-gate audit is in artifacts/parity/gate-calibration-v4/REPORT.md.

Target architecture

The near-term deployed target is deliberately compact: one stateful RGB policy, an explicit minimap route command and deterministic safeguards. It borrows the useful parts of larger autonomy stacks—pretrained visual features, temporal perception, auxiliary training tasks and a recovery data flywheel—without trying to reproduce a fleet-scale multi-camera transformer on a 24 GB fanless MacBook Air. An independent guard model is deferred until extended operation is under consideration.

GFN frame stream (15 Hz base, 30 Hz action bursts)
  ├─ minimap route-state extractor -> left / straight / right / arrival / uncertain
  └─ DrivePolicy
       ├─ frozen DINOv3 ConvNeXt Tiny scene encoder
       ├─ spatial adapter + causal temporal memory
       ├─ RGB history + explicit high-level route context
       └─ steer / throttle / brake + auxiliary motion/path/risk outputs
                 ↓
          deterministic focus, freshness, takeover and ActionGate safeguards
                 ↓
        bounded human-gated macOS research executor

There is no direct speedometer signal. We will not infer fake km/h. Depth Anything V2 Small and RAFT Small will generate offline, confidence-masked relative-geometry and motion targets. The RGB policy learns those concepts via auxiliary heads but will not run either teacher during inference.

The present CNN+GRU is a suitable compute baseline, not a proven final architecture. Its original global average pooling discarded road geometry, its ten-step output trained only step zero, and its auxiliary heads are unsupervised. The next experiment replaces only the scene encoder with frozen DINOv3 spatial features; it is a controlled foundation-model test, not a switch to a larger generic transformer or world model.

Recorder

The recorder saves a stable 15 FPS base stream and internally samples at 30 FPS. Each non-repeat key down/up opens a 300 ms burst window, so short steering taps and their immediate visual response receive denser coverage without storing 30 FPS for the full session. It records W/A/S/D/Space globally and needs Screen Recording and Input Monitoring permissions.

cd /Users/josefjara/Documents/projekty/wd2-controller-gate
./scripts/wd2_recorder.sh check-permissions
./scripts/wd2_recorder.sh list-windows

Record a short pilot session using the current GFN window ID:

./scripts/wd2_recorder.sh record \
  --window-id YOUR_GFN_WINDOW_ID \
  --fps 15 \
  --stream-fps 30 \
  --burst-ms 300 \
  --width 1280 \
  --height 720 \
  --output datasets

Stop with Ctrl + Option + Command + R or Ctrl-C.

For longer collection, use opt-in HEVC chunks. They use the local ffmpeg VideoToolbox encoder and remain separately decodable at the selected chunk boundary. JPEG remains the safe default.

./scripts/wd2_recorder.sh record \
  --window-id YOUR_GFN_WINDOW_ID \
  --route-id downtown-loop-a \
  --run-id dev-2026-07-19-a \
  --storage hevc \
  --chunk-seconds 60 \
  --fps 15 --stream-fps 30 --width 1280 --height 720 \
  --output datasets

For development collection, --route-id is required project discipline even though the recorder keeps it optional for ad-hoc pilots. Sessions sharing a route ID are held in the same split group, preventing route leakage between training and evaluation.

Each session now has:

datasets/<session_id>/
  session.json             # configuration and selected window
  session_end.json         # final capture counters and effective FPS
  key_events.jsonl         # raw keyboard transitions
  labels.jsonl             # frame/action pairs plus flags and visual hash
  frames/frame_000000.jpg    # JPEG mode
  # or chunks/chunk_00000.mov # HEVC mode
  diagnostics/

focus_valid is true only while the GFN owning process is foreground. window_valid verifies the selected window still exists and is onscreen. duplicate_candidate is based on a sparse visual hash. These flags make a frame ineligible for normal training by default; they are retained for auditability.

Validate each session before using it:

./scripts/validate_keyboard_session.py datasets/<session_id> --strict

The pilot gate is: no missing frames, no failed writes, p95 frame gap at or below 1.75× the target frame interval, and no unreviewed invalid-focus/menu material.

HEVC is an archival capture format, not direct trainer input. Validate it first, then materialize a derived JPEG session; the source chunk session remains unchanged:

PYTHONPATH=src .venv/bin/python scripts/validate_keyboard_session.py \
  datasets/<hevc-session> --strict
PYTHONPATH=src .venv/bin/python scripts/materialize_hevc_session.py \
  datasets/<hevc-session> --output datasets/<derived-jpeg-session>
PYTHONPATH=src .venv/bin/python scripts/validate_keyboard_session.py \
  datasets/<derived-jpeg-session> --strict

Run a short capture/materialization check on each new machine or ffmpeg build before trusting HEVC mode for a long collection.

ML environment

Use Python 3.12 rather than the system Python 3.14. The package bound intentionally keeps the training environment below Python 3.14 until every required ML wheel is verified on this Mac.

cd /Users/josefjara/Documents/projekty/wd2-controller-gate
python3.12 -m venv .venv
source .venv/bin/activate
python -m pip install --upgrade pip
python -m pip install -e '.[dev,vision,export,foundation]'

PyTorch uses the MPS backend on Apple Silicon when available; the trainer selects it automatically. Confirm it before a long run:

python -c 'import torch; print(torch.backends.mps.is_available())'

Foundation-model preparation

The DINOv3 policy uses a gated Hugging Face checkpoint. Accept its terms on the model page once, authenticate locally, then download the three approved assets. Depth Anything and RAFT are cached now for future offline teacher work but are not inputs to the first policy run.

.venv/bin/hf auth login
./scripts/prepare_perception_models.sh
PYTHONPATH=src .venv/bin/python scripts/verify_foundation_models.py

The first new training run is intentionally not started by the preparation command. After the cache verifies, run:

./scripts/run_v9_dinov3_foundation.sh

Dataset workflow

Never split random neighboring frames across train and test. Build a manifest with an explicit label offset, then split by complete sessions:

mkdir -p manifests splits

# Begin at 0 ms; calibrate and change only after a shadow-mode check.
python scripts/build_manifest.py datasets --output manifests/all.jsonl --label-offset-ms 0
python scripts/dataset_report.py manifests/all.jsonl
python scripts/make_splits.py manifests/all.jsonl --output-dir splits
python scripts/evaluate_baselines.py splits/val.jsonl

The offset helper is a coarse starting point based on visual motion and pedal changes. It is not ground truth and must be confirmed in shadow mode:

python scripts/estimate_label_offset.py datasets/<session_id>

For the first usable dataset, collect at least two hours of clean, tagged driving with a fixed chase camera and stable HUD. Include normal lane following, turns, traffic braking, hill starts, low-speed recovery, and a small amount of reverse/handbrake behavior. Keep a set of entire routes and conditions reserved exclusively for test.

Train and shadow replay

The custom mobile-style encoder remains the reproducible baseline. The first new policy family will use frozen DINOv3 ConvNeXt Tiny scene features while keeping the same causal data split and replay contract. The lower-left minimap crop (0–16 % of frame width, 67–100 % of frame height) is retained, but its future role is an explicit route command rather than an unvalidated direct steering shortcut.

source .venv/bin/activate

python -m wd2_controller.train \
  --train-manifest splits/train.jsonl \
  --val-manifest splits/val.jsonl \
  --output artifacts/baseline-v0

Monitor a running training job in a second terminal:

while true; do clear; .venv/bin/python scripts/training_status.py artifacts/v1; sleep 3; done

Prepared second run

After collecting the next data batch, prepare (but do not start) the reproducible v1 split:

./scripts/prepare_v1_training.sh

When ready to train, run this in one terminal and monitor it in another with training_status.py:

./scripts/run_v1_training.sh

The script evaluates the selected macro-F1 checkpoint on the held-out test sessions after training.

To prevent idle sleep during a v2 run and then put the Mac to sleep only after successful training and test evaluation, run ./scripts/run_v2_then_sleep.sh.

v2: event-aware turning and braking

The v2 policy intentionally has no handbrake output. It learns only steering, throttle and brake. Short A/D/S key-down events from action_pulse_next_window are retained as targets, and the train sampler oversamples turning and braking clips. This prevents long straight/W-held sections from dominating every batch.

The first v2 experiment showed why sampling and loss weighting must be ablated separately: inverse class weights plus aggressive oversampling predicted a turn on roughly 55% of test clips although turns formed roughly 20% of the targets. Its steering macro-F1 was 0.5034, below the repeat-last-action baseline 0.6749. The data remain useful; the training recipe is rejected.

v3 systematic experiment suite

The systematic suite is complete: it ran 14 train/validation screening runs and three confirmation seeds. The winning denser_history recipe is frozen as the current V3 candidate; do not reopen the old V2 benchmark to choose another recipe.

The V3 experiment plan is historical evidence. The current foundation-policy plan supersedes its forward-looking milestones; see docs/ROADMAP_2026-07-26.md.

The future Human-Gated DAgger intervention and clean-replay design is prepared in docs/INTERVENTION_REPLAY_PIPELINE.md. Its state machine, simulator, passive sidecar/merger and manifest tooling are available but deferred until the foundation-policy and route-command gates pass; it adds no intervention-specific game control.

1. balancing: natural distribution, square-root weights, effective-number weights, mild sampling, and a weak combination;
2. target labels: event-aware pulses versus captured held states;
3. representation: control, spatial grids, compact state, no minimap, no previous action, shorter history, and denser history;
4. confirmation: the one winning recipe is trained from scratch with seeds 23, 41, and 73 for six epochs.

Screening never opens the v2 benchmark split. Every run reports validation steering macro-F1, gain over repeat-last-action, action-transition F1, false-turn rate, brake F1, and p95 inference latency. Only the unchanged three-seed confirmation recipe is evaluated on that benchmark, including mean and variation across seeds. Interrupted suites can be run again; completed experiments are skipped.

The existing V2 test sessions influenced earlier decisions and are not a clean release test. The V3 clean-triplet holdout has now been consumed once; keep it sealed. Any replacement candidate needs new, disjoint development sessions and a newly sealed release holdout.

Inspect the complete plan without training:

.venv/bin/python scripts/run_experiment_suite.py --dry-run

Run everything, keep macOS awake, and sleep only after every training and final test succeeds:

./scripts/run_v3_experiments_then_sleep.sh

Monitor it from a second terminal:

while true; do clear; .venv/bin/python scripts/experiment_status.py; sleep 5; done

Or start the read-only local dashboard, which opens Safari and refreshes every two seconds:

./scripts/v3_dashboard.sh

Stop only the dashboard with Ctrl-C; the trainer continues independently.

Constrained live harness — currently blocked for V3

The repository retains a deliberately limited 30-second, 7.5 Hz live harness. It checks foreground focus before every key transition, emits only W/A/S/D, has no handbrake, releases all synthetic keys on focus loss/error/exit, and lets F8 or Ctrl-C immediately return control to the human. Do not run it with V3 now. The corrected parity calibration promoted no gate threshold and V3 collapses under cold-start gated_self history.

Grant the terminal Screen Recording and Accessibility permissions, then install the macOS bridge once:

.venv/bin/python -m pip install -e '.[live]'

The following command is retained for a future candidate only after docs/PROJECT_STATUS.md explicitly authorizes a smoke test. The script waits three seconds, then uses the focused GFN game window; an optional window ID pins a specific window.

./scripts/run_live_v3_flight_test.sh
# optional: ./scripts/run_live_v3_flight_test.sh YOUR_GFN_WINDOW_ID

If the HTTP dashboard is unreliable while MPS training is under load, use the server-free live report instead. Run it in a second terminal, then open the generated file in Safari; it refreshes itself every four seconds and includes the current run, overall ETA, stage map, trainer CPU/RAM, validation metrics, leaderboard, and live log:

.venv/bin/python scripts/v3_file_report.py
open -a Safari artifacts/experiments/v3-systematic/live_report.html

To pause the suite deliberately, press Ctrl-Z in the terminal that started run_v3_experiments_then_sleep.sh. This suspends both the runner and the current training process without discarding its artifacts. Resume in that same terminal with fg; do not start a second suite runner. A long pause can expire the sudo authorization used only for the final automatic sleep, so run sudo -v again near the end if that final sleep matters.

Results are written under artifacts/experiments/v3-systematic/, including leaderboard.md and final_summary.json.

Replay a recorded session through the checkpoint:

python -m wd2_controller.shadow datasets/<session_id> \
  --checkpoint artifacts/releases/v3-candidate-2026-07-15/model.pt \
  --config artifacts/releases/v3-candidate-2026-07-15/resolved_config.yaml

It writes diagnostics/shadow_predictions.jsonl. Shadow mode does not create CGEvents or otherwise control the game.

Promotion gates

Each model must pass all earlier gates before moving forward:

1. Recorder gate: data integrity, valid flags, measured frame timing.
2. Data gate: session-level train/validation/test split; no leakage; action distributions reported.
3. Open-loop gate: beat majority and repeat-last-action baselines on balanced steering F1, action-onset timing, and calibration—not merely raw accuracy.
4. Shadow gate: stable 20 Hz local inference, p95 local latency below 50 ms, sensible confidence on menus and unusual scenes.
5. Bounded human-gated gate: focus/freshness checks, ActionGate, manual takeover and releases within 100 ms; this follows a safe offline candidate and explicit user authorization.
6. Extended-operation gate: an independent GuardNet, fixed unseen-route matrix, route completion, collision/intervention rate, invalid-screen false actions, and regression suite all improve together.

Immediate next work

1. Keep the P0 baseline reproducible: standard tests/lint/config contract are green; stale and completed runs are listed in docs/EXPERIMENT_REGISTRY.md.
2. Implement the frozen DINOv3 ConvNeXt Tiny action-only policy and compare it against V8 using the existing grouped development split and gated_self.
3. Add Depth Anything or RAFT only after a passing DINO baseline, one confidence-masked auxiliary target family at a time.
4. Keep the route/screen selector offline and abstaining. The route-state extractor must prove its own grouped feature quality before becoming a high-level command.
5. Do not reopen consumed holdouts, add a world model, or start unattended live control. GuardNet is intentionally deferred from the current phase.

Safety boundary

This repository includes a constrained, opt-in 30-second live test harness; it is not unattended driving. F8 enters persistent human-only takeover mode and releases all synthetic keys; F10 is the separate explicit AI-resume command, which clears temporal history and waits for a fresh context. The current V3 candidate is not authorized to use it. Any future activation must remain bounded by focus checks, ActionGate, manual takeover and automatic key release after the offline and bounded human-gated gates above have demonstrably passed. GuardNet is required before extended or unattended operation, not before the manually supervised research mode.