Skip to content

agent_context.md

Latest commit

 

History

History
101 lines (88 loc) · 9.89 KB

agent_context.md

File metadata and controls

101 lines (88 loc) · 9.89 KB
Raw

RL-RTHS Agent Context

Project Goal

Train a PPO agent (SB3) for Atari tasks (Ms. Pacman, Amidar, Qbert) with a recovery mechanism:

  • Detect when policy behavior is stuck in a local loop.
  • Temporarily switch to heuristic recovery (RTHS-style action fallback).
  • Return control to PPO once recovery is reached.

Core Representation Model

World-model style representation learning inspired by PPGS:

  • Encoder h_theta: observation s_t -> 16D hyperspherical latent z_t.
  • Forward model f_phi: predicts z_{t+1} from (z_t, a_t).
  • Inverse model p_omega: predicts a_t from (z_t, z_{t+1}) to reduce latent collapse.

Current Research Focus: Dynamic Distractor Invariance

Atari environments include moving distractors even when agent position is unchanged.
Main hypothesis: enforce NOOP invariance in latent space.

  • Standard margin-style separation can overreact to distractor motion.
  • Modified behavior: when action is NOOP, push ||z_{t+1} - z_t||^2 -> 0.
  • Intended effect: latent features track controllable state changes, not distractor-only motion.

Runtime Workflow (High-Level)

  1. PPO acts normally.
  2. Latent trajectory is monitored for short-horizon stagnation/oscillation ("stuck").
  3. If stuck, fallback policy iterates through ranked PPO action alternatives until progress.
  4. Resume PPO after recovery signal.

RTHS evaluation scripts (compare_rths.py, run_rths_policy.py)

  • compare_rths.py: Compares subopt vs main policies, vanilla vs RTHS, using raw episodic return from info["raw_ep_return"] (unclipped ALE score), not SB3 evaluate_policy clipped rewards. Supports --render (passes render_mode="human" through make_eval_env, same pattern as games/*/eval.py).
  • Warmup / false positives: For the first N env steps per episode (--start-tracking-step), RTHS does not override the policy and does not update LatentStateTracker. If the flag is omitted, N defaults to the same counts as manual-play post-reset warmup — DEFAULT_WARMUP_NOOPS_BY_GAME in rths/env/wrappers.py: qbert 39, pacman 66, amidar 12. Use 0 for immediate RTHS (legacy).
  • run_rths_policy.py: Same semantics as compare_rths.py for --start-tracking-step.

Manual play / demo warmup (--warmup-noops, unset)

DEFAULT_WARMUP_NOOPS_BY_GAME in rths/env/wrappers.py: qbert 39, pacman 66, amidar 12. Used by debug/manual_play_atari.py, debug/demo_action_sequence.py, and (when omitted) --start-tracking-step in compare_rths.py / run_rths_policy.py. Ms. Pac-Man: 66 post-reset NOOPs so the 67th env step is the first where directional input lines up with intent.

Q*bert human-play timing (empirical)

Observed with make_eval_env + SDL manual control (debug/manual_play_atari.py); full detail: debug/qbert_manual_play_timing.md.

  • ~39 NOOP env.steps after reset for Q*bert: playfield / sprites / HUD tend to fully appear and inputs feel trustworthy (same as default warmup above).
  • Trailing NOOPs after each key (Q*bert): debug/manual_play_atari.py defaults to --noop-after-action 10 for --game qbert (action first, then 10 NOOPs — a cooldown so the next key lines up with what you see; Q*bert feels sluggish / desynced without it). Use --noop-after-action 0 to try raw one-step-per-key.
  • Pac-Man / Amidar: default --noop-after-action 0. Control feels immediate on the action step; extra trailing NOOPs are not needed for that Q*bert-style alignment. (Each NOOP is still neutral input and advances the env; on Pac-Man you often still see motion during those steps because the game keeps moving along corridors until a wall or turn—not because the API repeats the last action.)

Trailing --noop-after-action (per key, Q*bert-only default) is not the same thing as RTHS delay or post-reset warmup. Not a change to PPO’s per-step action semantics—those remain one action per step.

Notes for Future Agents

  • Keep this file concise and durable (what/why/current direction).
  • Move ephemeral experiment details to the tracker below instead of expanding sections above.

Tracker Entries

  • [2026-03-21] Type: Change Env/Setup: Repo layout. What changed: Moved manual_play_atari.py, demo_action_sequence.py, manual_play_log.jsonl, and notes/qbert_manual_play_timing.md → debug/ (debug/README.md). Path logic in debug/manual_play_atari.py resolves repo root whether the script lives at repo root or under debug/. agent_context.md and rths/env/wrappers.py comment updated. Observation: Run as uv run python debug/manual_play_atari.py from repo root. Next action: —
  • [2026-03-20] Type: Change Env/Setup: RTHS eval (compare_rths.py, run_rths_policy.py). What changed: DEFAULT_WARMUP_NOOPS_BY_GAME in rths/env/wrappers.py (qbert 39, pacman 66, amidar 12). Manual play / demo import it; --start-tracking-step defaults to None and resolves to that map per --game when omitted. README.md / agent_context.md updated. Observation: RTHS “kick-in” step count aligned with empirical post-reset readiness (same as --warmup-noops defaults). Next action: Override per experiment with explicit --start-tracking-step if needed.
  • [2026-03-20] Type: Result Env/Setup: Ms. Pac-Man manual SDL; make_eval_env stack. What changed: Default --warmup-noops for --game pacman set to 66 in debug/manual_play_atari.py and debug/demo_action_sequence.py (responsive from cumulative step 67; was briefly 150 / 67 during tuning). agent_context.md — “Manual play / demo warmup” section updated. Observation: 66 warmup NOOPs → first trustworthy directional step at 67. Next action: Tune if a different ROM/wrapper combo needs a different count.
  • [2026-03-20] Type: Result Env/Setup: Q*bert manual SDL play; make_eval_env stack. What changed: Added debug/qbert_manual_play_timing.md (warmup ~39; Q*bert default --noop-after-action 10 in debug/manual_play_atari.py; later moved with other harnesses under debug/). Linked under “Q*bert human-play timing”. Observation: Human-facing boot/spacing differs from PPO’s one-action-per-step training. Next action: Keep RTHS start-tracking-step separate unless experiments tie them.
  • [2026-03-20] Type: Result Env/Setup: RTHS eval UX and early-episode false positives (esp. Q*bert). What changed: compare_rths.py — added --render; added --start-tracking-step (default 5) and gated RTHS override + record_action until step >= N per episode (aligned with run_rths_policy.py). run_rths_policy.py — default --start-tracking-step changed from 0 to 5, help text expanded. README.md — documented warmup, compare_rths --render / --start-tracking-step, and run_rths_policy warmup note. Observation: Prior compare_rths path recorded latents from step 0, so spurious “loops” at spawn could force a bad second-choice action; warmup defers RTHS until a few policy steps have executed. Next action: Tune N per game if needed; consider documenting raw vs clipped reward distinction next to games/*/eval.py if confusion recurs.
  • [2026-03-04] Type: Attempt Env/Setup: Context doc restructuring. What changed: Added persistent tracker for bugs/issues/attempts related to distractor-invariant encoder work. Observation: No experiment result recorded yet. Next action: Add concrete entries after each training/eval run (metrics + failure mode).
  • [2026-03-04] Type: Result Env/Setup: Refactor and tooling updates across Pacman/Amidar/Qbert latent workflow. What changed: Added offline dataset generation (generate_data.py) with PPO rollouts, 8 parallel env support, tqdm progress, NOOP/random action mixing (5%/5%), and reward logging in dataset. Reworked all train_latent.py scripts to train from .pkl datasets (no rollout collection), added pre-training visual dataset sanity checks (5 samples, NOOP-forced final sample), and added training progress bars. Updated save paths so checkpoints land in both models/latent/ and model-specific folders (models/encoder, models/forward, models/inverse). Added legacy checkpoint compatibility shim (impala.py) so old PPO zips load after module moves. Observation: Legacy ModuleNotFoundError: impala issue is resolved; latent checkpoints now appear in per-model folders as expected. Next action: Run comparative latent training sweeps and evaluate loop-recovery behavior using the updated offline dataset pipeline.
  • [2026-03-16] Type: Attempt Env/Setup: Hyperparameter study for loop detection false-negative reduction. What changed: Created run_hyperparam_study.py — full Optuna-based HPO pipeline that sweeps latent model training params (lr, margin, loss weights, latent_dim, hidden dims, batch size, epochs, updates_per_epoch) and jointly evaluates loop-detection thresholds. Uses NOOP reidentification rate as primary false-negative proxy, combined with inverse accuracy, forward MSE, and margin satisfaction into a composite score weighted 40% toward recall. Monitoring: Optuna SQLite + optuna-dashboard (local web UI), TensorBoard per-trial curves, JSON per-trial logs. Added optuna to pyproject.toml deps; optuna-dashboard is optional ([dashboard] extra). Observation: Script imports clean, CLI tested, no lint errors. Awaiting dataset generation before first run. Next action: Generate dataset with generate_data.py, run run_hyperparam_study.py, analyze best config and false-negative rates. Later: add orientation-aware loop detection.