Non-invasive perception + reinforcement learning agent for generals.io.
Built on generals-bots (JAX simulator).
| Component | Status | Owner |
|---|---|---|
| Decision (Part B1) — Padded23UNetPolicy | ✅ Implemented | Anqiao Fu |
| SL Training — Behaviour cloning from human replays | ✅ Implemented | Anqiao Fu |
| PPO Training — Self-play with curriculum | ✅ Implemented | Anqiao Fu |
| Perception (Part A1) — Screen capture & grid detection | ✅ Implemented | GuZhi Xun |
| Execution (Part A2) — Mouse controller | ✅ Implemented | GuZhi Xun |
If You are TA: 我们在提交之前看到了助教对于git仓库的要求，请您留意该repo的内容与实际报告中的有细微差异：我们报告中的版本在一位同学的线上租借的容器中（由于当时还需要整理代码没有提交到这个repo），而由于该服务器租借用的子账号，后来同学用完了这个账号里的现金无法继续充值，这些内容暂时没有以代码的形式呈现。不过好在我们这里还有另一个可用的版本，只不过是基于Generals的一个模拟器进行视觉识别而非直接对屏幕截图识别，原理相同且实现路径也大体一致，但是我们担心助教误解于是希望能够在此说明。我们也在寻找这个问题的解决方法，对于带来的问题我们深表歉意。 由于组内的同学对于相关的AI训练很感兴趣，我们并没有打算到此为止，如果您需要审阅作业可以以此版本为准，谢谢！
Fixed-size 23×23 architecture designed for stable 48h training convergence:
Input:
spatial: (22, 23, 23) — 22 hand-crafted spatial channels
scalar: (16,) — 16 scalar features (army/land counts, timestep, etc.)
Backbone: U-Net ResNet with GroupNorm, SiLU, ScalarFiLM
Encoder: ResBlocks + downsample (23→12→6→3)
Bottleneck: ResBlock + FiLM conditioning
Decoder: Upsample (3→6→12→23) + skip connections + ResBlocks
Heads:
action_type: (2,) — pass / move
move: (4,2,23,23) — direction × split × grid
value: scalar — state-value estimate (GAP + scalar MLP)
aux: (23,23) — opponent general belief (Conv1x1)
Action space: 4233 discrete actions (4232 move + 1 pass).
Parameters: ~2M–4M.
Human Replays (HuggingFace)
│
▼
┌───────────────────┐
│ SL (50 epochs) │ ← Behaviour cloning from ~18,803 replays
│ 3–4h │
└────────┬──────────┘
│ pretrained weights
▼
┌───────────────────┐
│ PPO (1000 iters) │ ← Self-play with curriculum
│ 6–8h │ Random → Expander → Self-Play
└────────┬──────────┘
│ final model
▼
evaluate & deploy
Key innovations:
- Fixed 23×23 shape — avoids JIT recompilation across varied grid sizes (was a major bottleneck)
- U-Net backbone — better long-range spatial reasoning than pure CNN
- 22-channel spatial encoding — rich feature set including frontier, local advantage, distance fields
- Hierarchical policy head — separates pass/move from spatial move selection
- Phase-aware reward — early-game expansion → late-game city capture
- Belief state — opponent trace + general heatmap inference through fog of war
┌──────────────┐
│ Generals.io │
│ (Web Game) │
└──────┬───────┘
┌───────────────┼───────────────┐
▼ ▼ ▼
┌──────────────┐ ┌──────────────┐ ┌──────────────┐
│ Perception │ │ Decision │ │ Execution │
│ (Part A1) │ │ (Part B1) │ │ (Part A2) │
│ Screen cap │ │ U-Net CNN │ │ Mouse Ctrl │
│ Grid detect │ │ PPO / SL │ │ Coord Mapper│
│ OCR digits │ │ Belief │ │ │
└──────────────┘ └──────────────┘ └──────────────┘
⏳ stubs ✅ done ⏳ stubs
Prerequisites:
- Python 3.11 or 3.12 (JAX is not yet fully compatible with 3.13)
- CUDA-capable GPU (recommended) — the JAX simulator benefits significantly from GPU
- The
generals-botssubmodule installed
# Setup
uv sync
pip install -e generals-bots
# Single GUI game — expander vs random, 10x10 grid
python run.py
# Same as above, short form
python run.py --mode gui --agent expander --opponent random --grid 10x10All functionality via python run.py [OPTIONS].
| Flag | Description |
|---|---|
--mode gui |
Single game with real-time pygame display (default) |
--mode headless |
Batch games via jax.vmap, periodic stats, no GUI |
--mode human |
Play vs AI yourself (WASD + mouse controls) |
--mode evaluate |
Batch evaluate a trained model vs opponent |
python run.py --mode headless --games 500 --stats-interval 50
python run.py --mode human --opponent expander
python run.py --mode evaluate --agent model --model-path checkpoints/final.eqx --games 200--agent / --opponent |
Description |
|---|---|
expander |
Greedy border-expansion heuristic (strong baseline) |
random |
Random valid moves |
placeholder |
Untrained CNN (random weights) |
model |
Trained checkpoint (requires --model-path) |
# Placeholder CNN vs expander
python run.py --agent placeholder --opponent expander
# Trained model
python run.py --agent model --model-path checkpoints/final.eqx| Flag | Type | Default | Description |
|---|---|---|---|
--grid |
HxW |
10x10 |
Grid dimensions |
--seed |
int | random | RNG seed |
--mountain-density |
X.X-Y.Y |
0.18-0.26 |
Mountain fraction |
--num-cities |
X-Y |
9-11 |
City count |
--min-generals-distance |
int | 3 |
Min BFS distance between generals |
--castle-values |
X-Y |
40-51 |
City army value range |
--truncation |
int | 500 |
Max ticks before forced draw |
python run.py --grid 20x20 --seed 42 --num-cities 5-8
python run.py --grid 8x8 --min-generals-distance 2# PPO training (defaults: 10x10, vs random)
python train.py
# SL pretraining from human replays
python train.py --sl --sl-epochs 50 --sl-batch-size 256
# Full SL → PPO pipeline
bash train_full_pipeline.sh
# PPO with YAML config
python train.py --config experiments/my_exp.yaml
# Resume training from checkpoint
python train.py --resume runs/20260528_143021
# PPO initialised from SL weights
python train.py --init-from checkpoints/sl_pretrained.eqx --min-grid 17 --max-grid 23Training output structure (runs/YYYYMMDD_HHMMSS[_tag]/):
runs/
└── 20260528_143021/
├── config.yaml # Final merged config
├── checkpoints/ # best.eqx, iter_*.eqx, final.eqx
└── training_log.jsonl # Running metrics
The supervised learning pipeline:
- Loads human replays from HuggingFace (
strakammm/generals_io_replays) - Simulates each game using the JAX game engine
- Extracts (observation, action) pairs with the 22-channel / 16-scalar encoding
- Caches processed data on disk for fast re-loading (
sl_data_cache/) - Trains via hierarchical cross-entropy loss (action_type + move)
python train.py --sl \
--replay-dir /path/to/replays \
--sl-epochs 50 \
--sl-batch-size 256 \
--sl-lr 3e-4 \
--seed 42| Key | Action |
|---|---|
Q |
Quit |
Left arrow |
Slow down |
Right arrow |
Speed up |
Space |
Pause / Resume (replay only) |
# Show tile type labels
python run.py --show-tile-types
# Higher FPS, shorter games
python run.py --fps 30 --truncation 100
# Suppress non-essential output
python run.py --mode headless --quietGeneralsBot/
├── run.py # Main CLI entry point
├── train.py # SL + PPO training entry point
├── train_full_pipeline.sh # Full SL→PPO orchestration script
├── eval_checkpoint.py # *Legacy* old-CNN checkpoint evaluator
├── eval_pretrained.py # *Legacy* pretrained model evaluator
├── analyze_replay_data.py # Replay dataset & cache analysis tool
├── pyproject.toml # Project metadata & dependencies
├── REQUIREMENTS
│
├── src/generals_bot/ # Core source
│ ├── config.py # Config dataclass
│ ├── orchestrator.py # Pipeline orchestrator (4 modes)
│ ├── rewards.py # Reward functions (composite, phase-aware)
│ │
│ ├── perception/ # Part A1 — Information Extraction
│ │ ├── base.py # Abstract interface
│ │ ├── simulated.py # Engine observation → state cache
│ │ ├── live.py # Screen capture + grid detect + OCR (STUBS)
│ │ └── trainer.py # Training stubs
│ │
│ ├── decision/ # Part B1 — Decision Model
│ │ ├── model.py # Padded23UNetPolicy (U-Net ResNet + FiLM)
│ │ ├── encoding.py # 22-channel spatial + 16-scalar encoding
│ │ ├── action_space.py # 5-element & 4233 action utilities
│ │ ├── agent.py # DecisionAgent wrapper (4 modes)
│ │ ├── belief.py # Belief state (opponent trace + heatmap)
│ │ ├── replay_dataset.py # HuggingFace replay → training pairs
│ │ └── trainer.py # PPO / SL training loops
│ │
│ └── execution/ # Part A2 — Action Execution
│ ├── base.py # Abstract interface
│ ├── simulated.py # Direct env.step()
│ ├── live.py # Mouse controller (STUBS)
│ └── trainer.py # Training stubs
│
├── generals-bots/ # Git submodule — JAX game engine
├── docs/ # Architecture & implementation plans
│ ├── NEW_ARCHITECTURE.md # Padded-23 U-Net design doc
│ ├── MULTI_AGENT_IMPLEMENTATION_PLAN.md # Implementation breakdown
│ ├── last-plan.md # Latest training plan
│ ├── ref.md # Reference notes
│ └── Non-Invasive Perception...pdf # Full project paper
│
├── tests/
│ └── test_pipeline.py # 11 smoke tests
├── sl_data_cache/ # Cached processed replay data
└── checkpoints/ # Model weight files (.eqx)
python -m pytest tests/test_pipeline.py -vExpected: 11 passed.
- Python ≥ 3.11
jax,jaxlib(JIT-compiled simulator, CUDA 12)equinox(neural network layers as PyTrees)optax(optimisation)pygame(GUI)numpypython-socketio(live game client)pyyaml(config serialisation)datasets(HuggingFace replay loading)generals-bots(submodule, installed viapip install -e generals-bots)
| Parameter | Default | Note |
|---|---|---|
num_steps |
256 | steps per env per rollout |
num_epochs |
4 | PPO epochs per rollout |
minibatch_size |
512 | per gradient step |
clip_epsilon |
0.2 | PPO clipping range |
learning_rate |
3e-4 | Adam |
entropy_coef |
0.02 (config) / 0.01 (CLI) | ⚠ Config & CLI defaults differ |
value_coef |
0.5 | value loss weight |
max_grad_norm |
0.5 | global gradient clipping |
gamma |
0.99 | discount factor |
gae_lambda |
0.95 | GAE parameter |
num_envs |
256 | parallel environments |
reward_phase |
True | phase-aware reward shaping |
use_belief |
True | 2-channel belief state |
curriculum |
random → expander → self | SL skips random phase |
- Phase 1: RandomAgent → advance at 75% win rate
- Phase 2: ExpanderAgent → advance at 75% win rate
- Phase 3: Self-play vs past checkpoints (rolling window of 5)
Phase-aware (default, reward_phase=True): interpolates weights over game time.
R = base_reward (generals captured)
+ ratio_weight × Δ(army_ratio_log) # early 0.5 → late 0.2
+ city_weight × Δ(cities) # early 0.15 → late 0.5
+ ratio_weight × Δ(land_ratio_log) # same as army ratio
where α = clamp((t - 50) / 150, 0, 1), linear interpolation.
At game end (general captured): R = base_reward only — no shaping bonus.
Composite (non-phase-aware, when --entropy-coef 0.02): fixed weights city=0.4, ratio=0.3.
| Metric | Value |
|---|---|
| Parallel environments | 256–512 |
| Steps/second | 50k–100k+ (on GPU) |
| PPO iterations to converge (10×10) | ~200–400 |
| SL 50 epochs | ~3–4h |
| SL checkpoint | ~2 MB (eqx) |
| PPO 1000 iterations | ~6–8h |
| Total pipeline (SL + PPO) | ~9–12h |