A human-like companion AI that learns to follow and play with a player in Minecraft, trained with Behavioral Cloning + PPO reinforcement learning on Apple Silicon.
一个在 Minecraft 中学习跟随并陪伴玩家的类人智能体，基于行为克隆 + PPO 强化学习，运行于 Apple Silicon。
After 500K steps of PPO fine-tuning on top of behavioral cloning pre-training, the agent maintains 3–5 block follow distance across flat terrain and moderate obstacles, with smooth head movement and no mechanical snapping.
经过行为克隆预训练 + 500K 步 PPO 微调后，智能体能在平坦地形和一般障碍物环境中将跟随距离稳定保持在 3–5 格，头部转动自然，无机械式瞬间转向。
| Metric / 指标 | Result / 结果 |
|---|---|
| BC validation action accuracy / BC 验证集动作准确率 | 81.5% |
| PPO training steps / PPO 训练步数 | 500,000 |
| Stable follow distance / 稳定跟随距离 | ~3.6–3.9 blocks / 格 |
| Control loop frequency / 控制循环频率 | 20 Hz |
| End-to-end inference latency / 端到端推理延迟 | < 50 ms |
| Hardware / 硬件 | Mac mini M4 (16–24 GB unified memory) |
This is not a scripted pathfinding bot. The goal is to build a human-like digital companion that:
这不是一个脚本寻路机器人。目标是构建一个类人的数字伙伴，能够：
- Follow naturally — navigate complex 3D terrain (forests, hills, caves) without hard-coded rules. 自然跟随 — 无硬编码规则地在复杂 3D 地形中导航。
- Move like a human — learned from real player recordings to produce organic movement, not grid-snapping. 像人一样移动 — 从真实玩家录制数据中学习，产生有机运动而非格子式位移。
- Run on consumer hardware — entirely on a Mac mini M4 at 20 Hz, with headroom for future LLM slow-loop integration. 在消费级硬件上运行 — 全程在 Mac mini M4 上以 20 Hz 运行，并为未来接入 LLM 慢环预留空间。
┌─────────────────────────────────────────────────────────────┐
│ Mac mini M4 │
│ │
│ ┌─────────────────┐ ZMQ PUB :5555 ┌────────────────┐ │
│ │ Mineflayer Bot │ ──────────────────► │ Python Agent │ │
│ │ (Node.js) │ │ │ │
│ │ Headless │ ◄────────────────── │ PPO Policy │ │
│ │ bridge/bot.js │ ZMQ REP :5556 │ PyTorch MPS │ │
│ └────────┬────────┘ (action idx) └────────────────┘ │
│ │ │
│ ┌────────▼────────┐ │
│ │ Paper 1.20.1 │ ◄── WandererBot (PPO training target) │
│ │ Server │ ◄── Fabric Mod (BC data recording) │
│ └─────────────────┘ │
└─────────────────────────────────────────────────────────────┘
Key design choice / 核心设计选择: No vision/CNN. The agent perceives the world through structured data only — a 9×9×9 block ID grid + kinematics + target vector. This keeps inference under 50 ms on M4 Metal (MPS) while the Minecraft server runs on the same machine.
无视觉/CNN。智能体仅通过结构化数据感知世界——9×9×9 方块 ID 网格 + 运动学 + 目标向量。这使推理在 M4 Metal (MPS) 上保持在 50ms 以内，同时 Minecraft 服务器也运行在同一台机器上。
Phase 1: Data Collection / 第一阶段：数据采集
Single-player (Fabric Mod) or
Multiplayer observer (bridge/recorder.js)
│
▼
data/recordings/session_*.jsonl
│
▼ convert_data.py (filters idle frames)
│
data/recordings/dataset.h5
│
Phase 2: Behavioral Cloning / 第二阶段：行为克隆
▼ bc_train.py (~5 min, 20 epochs)
│
models/bc_pretrained.pt (val_acc = 81.5%)
│
Phase 3: PPO Fine-tuning / 第三阶段：PPO 微调
▼ ppo.py --load bc_pretrained.pt (~hours)
│
models/ppo_NNNNNNN.pt
│
▼ eval.py (inference only)
Live agent in-game / 游戏内实时运行
| Field / 字段 | Shape | Description / 说明 |
|---|---|---|
grid |
[729] int16 | 9×9×9 block type IDs centered on bot, world-axis-aligned / 以 bot 为中心的 9×9×9 方块类型 ID，世界轴对齐 |
kinematics |
[6] float32 | pitch/yaw (÷π), vx/vy/vz (÷10, clipped ±1), on_ground / 俯仰角/偏航角（÷π）, 三轴速度（÷10，截断至 ±1）, 是否落地 |
target |
[3] float32 | ΔX/ΔY/ΔZ to follow target, clipped ±1 (raw ÷50) / 到跟随目标的 ΔX/ΔY/ΔZ，截断至 ±1（原始值 ÷50） |
Block type IDs use the same registry (Registries.BLOCK.getRawId() in Fabric, block.type in Mineflayer) — max ~1057 in 1.20.1, with N_BLOCK_TYPES=1200 for safety.
方块类型 ID 使用相同的注册表（Fabric 中 Registries.BLOCK.getRawId()，Mineflayer 中 block.type）——1.20.1 中最大约 1057，为保险起见设 N_BLOCK_TYPES=1200。
| ID | Action / 动作 |
|---|---|
| 0 | forward / 前进 |
| 1 | back / 后退 |
| 2 | left / 左平移 |
| 3 | right / 右平移 |
| 4 | jump / 跳跃 |
| 5 | forward + sprint / 前进 + 疾跑 |
| 6 | forward + left / 前进 + 左 |
| 7 | forward + right / 前进 + 右 |
| 8 | yaw −0.3 rad / 偏航 −0.3 弧度（左转头） |
| 9 | yaw +0.3 rad / 偏航 +0.3 弧度（右转头） |
| 10 | pitch −0.2 rad / 俯仰 −0.2 弧度（抬头） |
| 11 | pitch +0.2 rad / 俯仰 +0.2 弧度（低头） |
Input / 输入:
grid [B, 729] → Embedding(1200, 16) → flatten → [B, 11664]
kinematics [B, 6]
target [B, 3]
concat → [B, 11673]
Backbone / 主干:
Linear(11673→512) + LayerNorm + ReLU
Linear(512→256) + LayerNorm + ReLU
Linear(256→128) + ReLU
Output heads / 输出头:
Actor: Linear(128→12) → softmax → action probabilities
Critic: Linear(128→1) → state value
Total parameters / 总参数量: ~6.15M
Device / 设备: MPS (Apple Silicon Metal)
| Component / 组件 | Weight / 权重 | Description / 说明 |
|---|---|---|
| Follow distance reward / 跟随距离奖励 | ×1.0 | Sweet spot 2–5 blocks (land), 3–8 blocks (water) / 甜区陆地 2–5 格，水中 3–8 格 |
| PBRS shaping / 势能塑形奖励 | ×0.3 | Potential-based reward shaping toward target / 基于势能的向目标奖励塑形 |
| Stuck penalty / 卡住惩罚 | ×1.0 | Penalize low velocity near target / 靠近目标时速度过低给予惩罚 |
| View smoothness penalty / 视角平滑度惩罚 | ×1.0 | Penalize oscillating head movement / 惩罚视角振荡 |
| Cliff penalty / 坠落惩罚 | ×1.0 | Penalize airborne without jump input / 惩罚无跳跃输入的离地状态 |
Rewards are normalized online with Welford algorithm and clipped to ±2.0. PPO uses RewardNormalizer (Welford), clip ±10 on normalized values.
奖励使用 Welford 在线算法归一化并截断至 ±2.0。PPO 的 RewardNormalizer（Welford）对归一化值截断 ±10。
| Hyperparameter / 超参数 | Value / 值 |
|---|---|
| total_timesteps | 500,000 |
| n_steps | 512 |
| batch_size | 64 |
| n_epochs | 4 |
| gamma | 0.99 |
| gae_lambda | 0.95 |
| clip_range | 0.2 |
| ent_coef | 0.01 |
| learning_rate | 3e-4 (linear decay) |
| save_every | 50,000 steps |
minecraft-agent/
├── agent/
│ ├── policy.py # PolicyNetwork (PPO) + BCPolicyNetwork (BC)
│ ├── ppo.py # CleanRL-style PPO trainer with RewardNormalizer
│ ├── bc_train.py # Behavioral cloning trainer
│ ├── bc_eval.py # BC model evaluation
│ ├── convert_data.py # JSONL → HDF5 converter (filters idle frames)
│ ├── env.py # Gymnasium environment, 8-component reward
│ └── eval.py # Inference-only script (no training)
│
├── bridge/
│ ├── bot.js # RL agent bot — ZMQ PUB:5555 + REP:5556
│ ├── wanderer.js # Wandering bot — PPO training target
│ ├── recorder.js # Multiplayer observer recorder (per-player JSONL)
│ ├── run_bots.sh # Launch bot.js + wanderer.js together
│ └── sensors/
│ └── grid.js # getGrid9x9x9 / getTargetDelta / getKinematics
│
├── fabric-mod/
│ └── src/main/java/com/mcagent/recorder/
│ └── DataRecorderMod.java # Single-player tick recorder → JSONL
│
├── mr_server/
│ ├── server.properties # offline-mode, view-distance=4, no monsters
│ ├── ops.json # AgentBot, WandererBot OP（人类玩家请本地自行 /op 或编辑 ops）
│ └── spigot.yml # moved-too-quickly-multiplier=100.0 (anti-kick)
│
├── data/recordings/ # session_*.jsonl + dataset.h5
├── models/ # bc_pretrained.pt + ppo_NNNNNNN.pt checkpoints
├── runs/ # TensorBoard event files (bc/ and ppo/)
│
├── start_server.sh # Start Paper 1.20.1 server
├── start_bot.sh # Start bot.js（默认跟随目标：WandererBot，可用 MC_TARGET 覆盖）
├── run_training.sh # Auto-resume PPO: finds latest checkpoint, restarts on failure
├── server_setup.sh # Full Ubuntu/Debian deployment with systemd
└── test_zmq.py # ZMQ diagnostic script
bot.js ──PUB:5555──► Python agent (env.py)
every physicsTick (50ms / 20Hz)
payload: { t, position, kinematics, grid, target_delta, health, food }
Python agent (env.py) ──REP:5556──► bot.js
discrete action index 0–11
reply: { ok: true }
SUB socket: zmq.CONFLATE=1 — only the latest frame is kept, stale frames dropped.
IPC file: /tmp/agentbot_pos.json — written every tick by bot.js,
read by Fabric Mod to compute target_delta in single-player mode.
- macOS with Apple Silicon (M1/M2/M3/M4)
- Java 17+ (for Fabric mod build and Paper server)
- Node.js 20 LTS
- Python 3.11
- Prism Launcher (for running Fabric client with the recorder mod)
bash start_server.sh
# Paper 1.20.1 starts on localhost:25565 (offline mode)python3.11 -m venv .venv
source .venv/bin/activate
pip install torch torchvision # PyTorch with MPS support
pip install gymnasium pyzmq h5py numpycd bridge
npm installcd fabric-mod
./gradlew build
# Copy build/libs/data-recorder-1.0.0.jar to Prism Launcher mods folder
# 默认写入路径由 Fabric Mod 固定为 ~/project/minecraft/data/recordings/session_*.jsonl（见 DataRecorderMod.java）；与仓库目录可不同，需自行同步或改源码路径Play Minecraft with the Fabric mod installed. Each session writes a .jsonl file automatically. Aim for at least 1–2 hours of follow/walk gameplay.
安装 Fabric Mod 后正常游玩 Minecraft，每次会话自动写入 .jsonl 文件。建议采集至少 1–2 小时的跟随/行走游戏数据。
# Alternatively, record from multiplayer observer perspective:
# 或者从多人模式旁观视角录制：
cd bridge && node recorder.jssource .venv/bin/activate
python agent/convert_data.py data/recordings/session_*.jsonl
# Output: data/recordings/dataset.h5
# Idle frames (no key pressed) are filtered automatically.
# 输出：data/recordings/dataset.h5（自动过滤无按键的空闲帧）python agent/bc_train.py --data data/recordings/dataset.h5
# ~5 minutes on M4. Saves to models/bc_pretrained.pt
# 在 M4 上约需 5 分钟。保存至 models/bc_pretrained.pt# Terminal 1: RL agent bot + wanderer bot
# 终端 1：RL 智能体机器人 + 漫游目标机器人
cd bridge && bash run_bots.sh# Terminal 2:
source .venv/bin/activate
# Fresh start from BC weights:
# 从 BC 权重开始：
python agent/ppo.py --load models/bc_pretrained.pt
# Or auto-resume from latest checkpoint (recommended for overnight runs):
# 或自动从最新检查点续训（推荐用于无人值守过夜训练）：
bash run_training.shtensorboard --logdir ./runs
# Open http://localhost:6006 in browser# Terminal 2 (instead of ppo.py):
# 终端 2（替换 ppo.py）：
source .venv/bin/activate
python agent/eval.py --load models/ppo_0500000.pt- BC key order / BC 键序：Fabric mod writes
[W, S, A, D, Jump, Sprint].bc_train.pyunpacks asW, S, A, D, J, SP— note S before A. / Fabric Mod 写入顺序为[W, S, A, D, Jump, Sprint]，bc_train.py按W, S, A, D, J, SP解包——注意 S 在 A 前面。 - Pitch/Yaw units / 俯仰/偏航单位：Fabric mod converts Java
getYaw()/getPitch()(degrees) to radians before writing. Mineflayer provides radians natively. Both consistent. / Fabric Mod 将 Java 的度数单位转换为弧度后写入，Mineflayer 原生提供弧度。两者一致。 - Anti-cheat / 反作弊：AgentBot runs in creative mode.
spigot.ymlsetsmoved-too-quickly-multiplier: 100.0to prevent kick on fast RL moves. / AgentBot 以创意模式运行，spigot.yml设置moved-too-quickly-multiplier: 100.0防止因 RL 快速移动被踢。 - State freshness / 状态新鲜度：
zmq.CONFLATE=1on the SUB socket ensures only the latest game state is processed, dropping stale frames. / SUB socket 设置zmq.CONFLATE=1，确保只处理最新的游戏状态，丢弃过时帧。 - Yaw wrap-around / 偏航角回绕：Angular differences computed as
(d + π) % 2π − πto handle the ±π boundary correctly. / 角度差计算为(d + π) % 2π − π以正确处理 ±π 边界。
| # | Bug | Fix / 修复 |
|---|---|---|
| 1 | Block ID used state ID (~26000+) instead of type ID | Registries.BLOCK.getRawId() instead of getRawIdFromState() / 改用类型 ID |
| 2 | Pitch/Yaw sent in degrees instead of radians | Multiply by Math.PI/180 on Java side / Java 侧乘以 Math.PI/180 |
| 3 | N_BLOCK_TYPES too small (512) |
Raised to 1200 / 提升至 1200 |
| 4 | view_delta computed across multi-file boundaries |
Compute per-file, then concat / 逐文件计算再拼接 |
| 5 | Yaw wrap-around at ±π caused incorrect gradients | (d+π) % 2π − π normalization / 角度归一化 |
| 6 | Idle frames (no keypress) defaulted to action 0 | Filter idle frames in convert_data.py / 过滤空闲帧 |
| 7 | A/S key order swapped in BC unpacking | W, S, A, D order (S before A) / S 在 A 前面 |
| 8 | Fabric mod writer not flushed on shutdown | Registered shutdown hook on instance, not static / 实例注册 shutdown hook |
| Component / 组件 | Memory / 内存 |
|---|---|
| macOS + background / 系统后台 | ~3.0 GB |
| Paper server / Paper 服务器 | ~1.5 GB |
| Mineflayer + Node.js | ~0.3 GB |
| PyTorch MPS (training) / PyTorch MPS（训练） | ~1.0 GB |
| Policy network inference / 策略网络推理 | < 0.2 GB |
| Total / 合计 | ~6 GB |
Comfortably fits in 16 GB unified memory, leaving ample headroom for future LLM slow-loop integration via API.
在 16 GB 统一内存中绰绰有余，为未来通过 API 接入 LLM 慢环预留了充足空间。
- LLM slow loop / LLM 慢环：Integrate Claude/GPT API for high-level intent reasoning ("help the player chop trees", "avoid the creeper"). / 接入 Claude/GPT API 实现高层意图推理（"帮玩家砍树"、"避开苦力怕"）。
- More action diversity / 更多动作多样性：Add sneak, attack, use-item actions for richer interaction. / 添加潜行、攻击、使用物品动作以实现更丰富的交互。
- Terrain generalization / 地形泛化：Train on procedurally varied terrains (caves, oceans, mountains). / 在程序生成的多样地形（洞穴、海洋、山地）上训练。
- Multi-agent / 多智能体：Have multiple bots coordinate — one follows, one scouts ahead. / 多个 bot 协同——一个跟随，一个前方探路。
| 会上传（源码与模板） | 不上传（.gitignore，仅本地） |
|---|---|
agent/、bridge/（含 package-lock.json）、fabric-mod/、paper-plugin/ 源码 |
bridge/node_modules/、.venv/ |
mr_server/ 下配置模板（server.properties、spigot.yml、ops.json 等） |
mr_server/world*、libraries/、cache/、*.jar、logs/ |
data/recordings/.gitkeep、models/.gitkeep |
data/recordings/*.jsonl、dataset.h5、models/*.pt、runs/ |
mr_server/plugins/bStats/config.yml.example |
mr_server/usercache.json（含玩家名与 UUID）、mr_server/plugins/bStats/config.yml（含 bStats serverUuid） |
| 脚本与文档 | .env / .env.*、.claude/settings.local.json、Gradle build/ 与 .gradle/ |
仓库内的 ops.json 仅保留训练用机器人 OP；真人游戏账号请在本地用 /op <名字> 或编辑 ops.json，勿把含好友游戏 ID 的 ops.json / usercache.json 提交到公共仓库。
克隆后需要本地准备：创建 Python venv 并安装依赖；cd bridge && npm install；将 Paper JAR 放入 mr_server/ 并首次启动生成世界；录制与权重放在 data/recordings/、models/（上述目录默认不提交）。
MIT License. See LICENSE for details.
Built on: Mineflayer · Paper MC · Fabric · PyTorch · ZeroMQ · CleanRL