🤖 Shadow Hand AI Controller | 灵巧手智能仿真系统

A High-Fidelity Robotic Hand Simulation System driven by LLM & Physics Engine

基于大语言模型(LLM)与实时物理引擎的机器人灵巧手控制平台

---

📖 Introduction / 项目简介

English: This project bridges the gap between natural language and robotic control. It allows users to control a 24-DoF Shadow Dexterous Hand using voice or text commands. The system uses a local LLM (via LM Studio) to interpret user intent and PyBullet to simulate realistic physics and PID control in real-time.

中文： 本项目将生成式 AI 与传统机器人控制相结合，实现了从“自然语言指令”到“机械手动作”的端到端映射。系统利用本地部署的大语言模型作为“大脑”理解意图，并通过 PyBullet 物理引擎进行高保真的运动学仿真。

---

🆚 Comparison / 核心特性对比

Feature / 特性	Traditional Simulation / 传统仿真	Shadow Hand AI (This Project)
Command Input 指令输入	Complex Code / Joint Parameters 复杂的代码或关节参数	Natural Language (Voice/Text) 自然语言 (语音/文本)
Intelligence 智能水平	Hard-coded Logic 硬编码逻辑	LLM Reasoning & Auto-Correction 大模型自动推断与纠错
Control Logic 控制逻辑	Static Trajectory 僵硬的预设轨迹	Physics-based PID Control 基于物理的 PID 实时解算
Interaction 交互体验	Command Line 命令行	Sci-Fi GUI + Voice Interaction 科幻风 GUI + 语音交互

---

📐 Architecture / 系统架构

graph TD
    User["User Input (Voice/Text)"] --> |SpeechRecognition| ASR[ASR Engine]
    ASR --> |Prompt Engineering| LLM["LM Studio (Local LLM)"]
    
    subgraph "AI Brain / 认知层"
    LLM --> |Intent Recog| Intent[Intent Parsing]
    LLM --> |Correction| Correct[Typo Correction]
    LLM --> |JSON Output| Command[Control Command]
    end
    
    Command --> |Parse| Controller[Python Controller]
    
    subgraph "Physics Body / 物理层 (PyBullet)"
    Controller --> |Target Angle| Motor[Joint Motors]
    Motor --> |PID Calc| Physics[Physics Engine]
    Physics --> |Feedback| Feedback[Error Monitor]
    end
    
    Physics --> |Render| GUI[PyQt6 Dashboard]
    Feedback --> |Log| Log[Real-time Logs]

Loading

---

🚀 Quick Start / 快速开始

1. Prerequisites / 环境准备

• Python 3.8+
• LM Studio: Running Local Server on port 1234.

Install Dependencies / 安装依赖:

pip install -r requirements.txt

Note for Windows Users: If pyaudio fails to install, use pipwin:

pip install pipwin
pipwin install pyaudio

2. Fix Model Paths / 模型路径修复

Before the first run, execute this script to ensure the URDF can find the mesh files: 首次运行前，请务必执行此脚本以修复模型贴图路径：

python fix_urdf.py

3. Run / 启动系统

Start the Controller / 启动主程序:

python gui_main.py

---

🛠️ Modules / 功能模块详解

1. Perception & Interaction (感知与交互)

• Voice Wake-up: Integrated Google Speech Recognition for bilingual input.
• GUI Dashboard: Real-time heatmap of joint angles, current action status, and AI thinking logs.

2. AI Decision Core (AI 决策核心)

• Intent Understanding: Translates abstract requests (e.g., "Spell Apple") into action sequences (A-P-P-L-E).
• Robustness: Filters punctuation and auto-corrects typos (e.g., "Helo" -> "HELLO").

3. Physics Simulation (物理仿真)

• URDF Loading: Loads my_shadow_hand.urdf with full 24-DoF dynamics.
• PID Controller:
• Force: 100.0 (Max Torque)
• Velocity: 8.0 (Servo Speed)
• PositionGain: 0.1 (Kp)

4. Automated Testing (自动化测试)

Located in gui_qt_auto_test.py, designed for validation:

• A-Z Traversal: Validates all 26 alphabet sign language poses.
• Stress Test: Randomly executes 50+ actions to test stability and memory usage.

---

📂 File Structure / 文件结构

ShadowHand-AI/
├── gui_main.py           # [Entry] Main Application (GUI + Logic)
├── gui_qt_auto_test.py   # [Test] Automated Stress Testing Suite
├── my_shadow_hand.urdf   # [Model] Robot Description File
├── fix_urdf.py           # [Tool] Path Fixer Utility
├── requirements.txt      # [Config] Python Dependencies
└── meshes/               # [Asset] 3D Models (.dae/.stl)

---

📊 Performance / 性能表现

Tested on Standard Laptop (i7 CPU + Integrated GPU):

• Simulation FPS: 60+ FPS (PyBullet OpenGL2)
• AI Latency: < 1.5s (Qwen 1.5B int4 model)
• Convergence Time: ~0.3s per action
• Stability: Passed 1-hour continuous stress test.